{"thread":{"id":"54245","subject":"[PATCH 00/13] reftable library","startedAt":"2020-09-16T19:10:49Z","lastAt":"2021-07-22T08:31:17Z","messageCount":251,"participants":["Han-Wen Nienhuys via GitGitGadget","Johannes Schindelin via GitGitGadget","Junio C Hamano","Han-Wen Nienhuys","Jeff King","Ramsay Jones","Jonathan Nieder","Johannes Schindelin","Jonathan Tan","Josh Steadmon","Emily Shaffer","Patrick Steinhardt","Ævar Arnfjörð Bjarmason","SZEDER Gábor via GitGitGadget","Felipe Contreras","Stefan Beller","Eric Sunshine","Derrick Stolee","brian m. carlson","Andrzej Hunt"],"isPatch":true,"patchVersion":1,"patchTotal":13},"messages":[{"id":"405656","messageId":"pull.847.git.git.1600283416.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":null,"subject":"[PATCH 00/13] reftable library","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-16T19:10:03Z","receivedAt":"2020-09-16T19:10:49Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"This splits the giant commit from \nhttps://github.com/gitgitgadget/git/pull/539 into a series of smaller\ncommits, which build and have unittests.\n\nThe final commit should also be split up, but I want to wait until we have\nconsensus that the bottom commits look good.\n\nHan-Wen Nienhuys (12):\n reftable: add LICENSE\n reftable: define the public API\n reftable: add a barebones unittest framework\n reftable: utility functions\n reftable: (de)serialization for the polymorphic record type.\n reftable: reading/writing blocks\n reftable: a generic binary tree implementation\n reftable: write reftable files\n reftable: read reftable files\n reftable: file level tests\n reftable: rest of library\n reftable: \"test-tool dump-reftable\" command.\n\nJohannes Schindelin (1):\n vcxproj: adjust for the reftable changes\n\n Makefile | 46 +-\n config.mak.uname | 2 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n reftable/.gitattributes | 1 +\n reftable/LICENSE | 31 +\n reftable/VERSION | 1 +\n reftable/basics.c | 131 +++\n reftable/basics.h | 48 +\n reftable/block.c | 443 +++++++\n reftable/block.h | 129 +++\n reftable/block_test.c | 158 +++\n reftable/blocksource.c | 148 +++\n reftable/blocksource.h | 22 +\n reftable/compat.c | 110 ++\n reftable/compat.h | 48 +\n reftable/constants.h | 21 +\n reftable/dump.c | 212 ++++\n reftable/iter.c | 242 ++++\n reftable/iter.h | 72 ++\n reftable/merged.c | 358 ++++++\n reftable/merged.h | 39 +\n reftable/merged_test.c | 331 ++++++\n reftable/pq.c | 115 ++\n reftable/pq.h | 34 +\n reftable/publicbasics.c | 100 ++\n reftable/reader.c | 733 ++++++++++++\n reftable/reader.h | 78 ++\n reftable/record.c | 1114 ++++++++++++++++++\n reftable/record.h | 143 +++\n reftable/record_test.c | 410 +++++++\n reftable/refname.c | 209 ++++\n reftable/refname.h | 38 +\n reftable/refname_test.c | 100 ++\n reftable/reftable-tests.h | 22 +\n reftable/reftable.c | 104 ++\n reftable/reftable.h | 585 ++++++++++\n reftable/reftable_test.c | 585 ++++++++++\n reftable/stack.c | 1224 ++++++++++++++++++++\n reftable/stack.h | 50 +\n reftable/stack_test.c | 787 +++++++++++++\n reftable/strbuf.c | 142 +++\n reftable/strbuf.h | 80 ++\n reftable/strbuf_test.c | 37 +\n reftable/system.h | 51 +\n reftable/test_framework.c | 68 ++\n reftable/test_framework.h | 60 +\n reftable/tree.c | 63 +\n reftable/tree.h | 34 +\n reftable/tree_test.c | 62 +\n reftable/update.sh | 22 +\n reftable/writer.c | 664 +++++++++++\n reftable/writer.h | 60 +\n reftable/zlib-compat.c | 92 ++\n t/helper/test-reftable.c | 20 +\n t/helper/test-tool.c | 2 +\n t/helper/test-tool.h | 2 +\n 56 files changed, 10489 insertions(+), 5 deletions(-)\n create mode 100644 reftable/.gitattributes\n create mode 100644 reftable/LICENSE\n create mode 100644 reftable/VERSION\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/compat.c\n create mode 100644 reftable/compat.h\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/dump.c\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/reftable.c\n create mode 100644 reftable/reftable.h\n create mode 100644 reftable/reftable_test.c\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n create mode 100644 reftable/strbuf.c\n create mode 100644 reftable/strbuf.h\n create mode 100644 reftable/strbuf_test.c\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n create mode 100755 reftable/update.sh\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n create mode 100644 reftable/zlib-compat.c\n create mode 100644 t/helper/test-reftable.c\n\n\nbase-commit: 54e85e7af1ac9e9a92888060d6811ae767fea1bc\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-847%2Fhanwen%2Flibreftable-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-847/hanwen/libreftable-v1\nPull-Request: https://github.com/git/git/pull/847\n-- \ngitgitgadget\n"},{"id":"405657","messageId":"f697eac2286a37ff8d01e94102af3ce869fba3ff.1600283416.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.git.git.1600283416.gitgitgadget@gmail.com","subject":"[PATCH 01/13] reftable: add LICENSE","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-16T19:10:04Z","receivedAt":"2020-09-16T19:10:58Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nTODO: relicense?\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/LICENSE | 31 +++++++++++++++++++++++++++++++\n 1 file changed, 31 insertions(+)\n create mode 100644 reftable/LICENSE\n\ndiff --git a/reftable/LICENSE b/reftable/LICENSE\nnew file mode 100644\nindex 0000000000..402e0f9356\n--- /dev/null\n+++ b/reftable/LICENSE\n@@ -0,0 +1,31 @@\n+BSD License\n+\n+Copyright (c) 2020, Google LLC\n+All rights reserved.\n+\n+Redistribution and use in source and binary forms, with or without\n+modification, are permitted provided that the following conditions are\n+met:\n+\n+* Redistributions of source code must retain the above copyright notice,\n+this list of conditions and the following disclaimer.\n+\n+* Redistributions in binary form must reproduce the above copyright\n+notice, this list of conditions and the following disclaimer in the\n+documentation and/or other materials provided with the distribution.\n+\n+* Neither the name of Google LLC nor the names of its contributors may\n+be used to endorse or promote products derived from this software\n+without specific prior written permission.\n+\n+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n+\"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n-- \ngitgitgadget\n\n"},{"id":"405658","messageId":"25259b683bc35b4f6741c1b2b07b41b99019c088.1600283416.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.git.git.1600283416.gitgitgadget@gmail.com","subject":"[PATCH 10/13] reftable: read reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-16T19:10:13Z","receivedAt":"2020-09-16T19:11:09Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 3 +\n reftable/iter.c | 242 +++++++++++++++\n reftable/iter.h | 72 +++++\n reftable/reader.c | 733 ++++++++++++++++++++++++++++++++++++++++++++\n reftable/reader.h | 78 +++++\n reftable/reftable.c | 104 +++++++\n 6 files changed, 1232 insertions(+)\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/reftable.c\n\ndiff --git a/Makefile b/Makefile\nindex a200633945..275fc4227c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2349,7 +2349,10 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/compat.o\n+REFTABLE_OBJS += reftable/iter.o\n+REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/reftable.o\n REFTABLE_OBJS += reftable/strbuf.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\ndiff --git a/reftable/iter.c b/reftable/iter.c\nnew file mode 100644\nindex 0000000000..6631408ef8\n--- /dev/null\n+++ b/reftable/iter.c\n@@ -0,0 +1,242 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"iter.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"reftable.h\"\n+\n+int iterator_is_null(struct reftable_iterator *it)\n+{\n+\treturn it->ops == NULL;\n+}\n+\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n+{\n+\treturn 1;\n+}\n+\n+static void empty_iterator_close(void *arg)\n+{\n+}\n+\n+struct reftable_iterator_vtable empty_vtable = {\n+\t.next = &empty_iterator_next,\n+\t.close = &empty_iterator_close,\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = NULL;\n+\tit->ops = &empty_vtable;\n+}\n+\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n+{\n+\treturn it->ops->next(it->iter_arg, rec);\n+}\n+\n+void reftable_iterator_destroy(struct reftable_iterator *it)\n+{\n+\tif (it->ops == NULL) {\n+\t\treturn;\n+\t}\n+\tit->ops->close(it->iter_arg);\n+\tit->ops = NULL;\n+\tFREE_AND_NULL(it->iter_arg);\n+}\n+\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { 0 };\n+\treftable_record_from_ref(&rec, ref);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { 0 };\n+\treftable_record_from_log(&rec, log);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+static void filtering_ref_iterator_close(void *iter_arg)\n+{\n+\tstruct filtering_ref_iterator *fri =\n+\t\t(struct filtering_ref_iterator *)iter_arg;\n+\tstrbuf_release(&fri->oid);\n+\treftable_iterator_destroy(&fri->it);\n+}\n+\n+static int filtering_ref_iterator_next(void *iter_arg,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct filtering_ref_iterator *fri =\n+\t\t(struct filtering_ref_iterator *)iter_arg;\n+\tstruct reftable_ref_record *ref =\n+\t\t(struct reftable_ref_record *)rec->data;\n+\tint err = 0;\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&fri->it, ref);\n+\t\tif (err != 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (fri->double_check) {\n+\t\t\tstruct reftable_iterator it = { 0 };\n+\n+\t\t\terr = reftable_table_seek_ref(&fri->tab, &it,\n+\t\t\t\t\t\t ref->refname);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = reftable_iterator_next_ref(&it, ref);\n+\t\t\t}\n+\n+\t\t\treftable_iterator_destroy(&it);\n+\n+\t\t\tif (err < 0) {\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (err > 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif ((ref->target_value != NULL &&\n+\t\t !memcmp(fri->oid.buf, ref->target_value, fri->oid.len)) ||\n+\t\t (ref->value != NULL &&\n+\t\t !memcmp(fri->oid.buf, ref->value, fri->oid.len))) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treftable_ref_record_clear(ref);\n+\treturn err;\n+}\n+\n+struct reftable_iterator_vtable filtering_ref_iterator_vtable = {\n+\t.next = &filtering_ref_iterator_next,\n+\t.close = &filtering_ref_iterator_close,\n+};\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *it,\n+\t\t\t\t\t struct filtering_ref_iterator *fri)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = fri;\n+\tit->ops = &filtering_ref_iterator_vtable;\n+}\n+\n+static void indexed_table_ref_iter_close(void *p)\n+{\n+\tstruct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p;\n+\tblock_iter_close(&it->cur);\n+\treftable_block_done(&it->block_reader.block);\n+\tstrbuf_release(&it->oid);\n+}\n+\n+static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n+{\n+\tuint64_t off;\n+\tint err = 0;\n+\tif (it->offset_idx == it->offset_len) {\n+\t\tit->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\n+\treftable_block_done(&it->block_reader.block);\n+\n+\toff = it->offsets[it->offset_idx++];\n+\terr = reader_init_block_reader(it->r, &it->block_reader, off,\n+\t\t\t\t BLOCK_TYPE_REF);\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\tif (err > 0) {\n+\t\t/* indexed block does not exist. */\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\t}\n+\tblock_reader_start(&it->block_reader, &it->cur);\n+\treturn 0;\n+}\n+\n+static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\n+{\n+\tstruct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p;\n+\tstruct reftable_ref_record *ref =\n+\t\t(struct reftable_ref_record *)rec->data;\n+\n+\twhile (1) {\n+\t\tint err = block_iter_next(&it->cur, rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\terr = indexed_table_ref_iter_next_block(it);\n+\t\t\tif (err < 0) {\n+\t\t\t\treturn err;\n+\t\t\t}\n+\n+\t\t\tif (it->is_finished) {\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!memcmp(it->oid.buf, ref->target_value, it->oid.len) ||\n+\t\t !memcmp(it->oid.buf, ref->value, it->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+}\n+\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len)\n+{\n+\tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n+\tstruct indexed_table_ref_iter *itr =\n+\t\treftable_calloc(sizeof(struct indexed_table_ref_iter));\n+\tint err = 0;\n+\n+\t*itr = empty;\n+\titr->r = r;\n+\tstrbuf_add(&itr->oid, oid, oid_len);\n+\n+\titr->offsets = offsets;\n+\titr->offset_len = offset_len;\n+\n+\terr = indexed_table_ref_iter_next_block(itr);\n+\tif (err < 0) {\n+\t\treftable_free(itr);\n+\t} else {\n+\t\t*dest = itr;\n+\t}\n+\treturn err;\n+}\n+\n+struct reftable_iterator_vtable indexed_table_ref_iter_vtable = {\n+\t.next = &indexed_table_ref_iter_next,\n+\t.close = &indexed_table_ref_iter_close,\n+};\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = itr;\n+\tit->ops = &indexed_table_ref_iter_vtable;\n+}\ndiff --git a/reftable/iter.h b/reftable/iter.h\nnew file mode 100644\nindex 0000000000..c92a178773\n--- /dev/null\n+++ b/reftable/iter.h\n@@ -0,0 +1,72 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef ITER_H\n+#define ITER_H\n+\n+#include \"block.h\"\n+#include \"record.h\"\n+#include \"strbuf.h\"\n+\n+struct reftable_iterator_vtable {\n+\tint (*next)(void *iter_arg, struct reftable_record *rec);\n+\tvoid (*close)(void *iter_arg);\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it);\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec);\n+\n+/* Returns true for a zeroed out iterator, such as the one returned from\n+ iterator_destroy. */\n+int iterator_is_null(struct reftable_iterator *it);\n+\n+/* iterator that produces only ref records that point to `oid` */\n+struct filtering_ref_iterator {\n+\tint double_check;\n+\tstruct reftable_table tab;\n+\tstruct strbuf oid;\n+\tstruct reftable_iterator it;\n+};\n+#define FILTERING_REF_ITERATOR_INIT \\\n+\t{ \\\n+\t\t.oid = STRBUF_INIT \\\n+\t}\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *,\n+\t\t\t\t\t struct filtering_ref_iterator *);\n+\n+/* iterator that produces only ref records that point to `oid`,\n+ but using the object index.\n+ */\n+struct indexed_table_ref_iter {\n+\tstruct reftable_reader *r;\n+\tstruct strbuf oid;\n+\n+\t/* mutable */\n+\tuint64_t *offsets;\n+\n+\t/* Points to the next offset to read. */\n+\tint offset_idx;\n+\tint offset_len;\n+\tstruct block_reader block_reader;\n+\tstruct block_iter cur;\n+\tint is_finished;\n+};\n+\n+#define INDEXED_TABLE_REF_ITER_INIT \\\n+\t{ \\\n+\t\t.cur = { .last_key = STRBUF_INIT }, .oid = STRBUF_INIT, \\\n+\t}\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr);\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len);\n+\n+#endif\ndiff --git a/reftable/reader.c b/reftable/reader.c\nnew file mode 100644\nindex 0000000000..fae2dbb64e\n--- /dev/null\n+++ b/reftable/reader.c\n@@ -0,0 +1,733 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reader.h\"\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"iter.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+#include \"tree.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source)\n+{\n+\treturn source->ops->size(source->arg);\n+}\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tint result = source->ops->read_block(source->arg, dest, off, size);\n+\tdest->source = *source;\n+\treturn result;\n+}\n+\n+void block_source_close(struct reftable_block_source *source)\n+{\n+\tif (source->ops == NULL) {\n+\t\treturn;\n+\t}\n+\n+\tsource->ops->close(source->arg);\n+\tsource->ops = NULL;\n+}\n+\n+static struct reftable_reader_offsets *\n+reader_offsets_for(struct reftable_reader *r, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\t\treturn &r->ref_offsets;\n+\tcase BLOCK_TYPE_LOG:\n+\t\treturn &r->log_offsets;\n+\tcase BLOCK_TYPE_OBJ:\n+\t\treturn &r->obj_offsets;\n+\t}\n+\tabort();\n+}\n+\n+static int reader_get_block(struct reftable_reader *r,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t sz)\n+{\n+\tif (off >= r->size)\n+\t\treturn 0;\n+\n+\tif (off + sz > r->size) {\n+\t\tsz = r->size - off;\n+\t}\n+\n+\treturn block_source_read_block(&r->source, dest, off, sz);\n+}\n+\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r)\n+{\n+\treturn r->hash_id;\n+}\n+\n+const char *reader_name(struct reftable_reader *r)\n+{\n+\treturn r->name;\n+}\n+\n+static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n+\t\t\tuint8_t *header)\n+{\n+\tuint8_t *f = footer;\n+\tuint8_t first_block_typ;\n+\tint err = 0;\n+\tuint32_t computed_crc;\n+\tuint32_t file_crc;\n+\n+\tif (memcmp(f, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tf += 4;\n+\n+\tif (memcmp(footer, header, header_size(r->version))) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tf++;\n+\tr->block_size = get_be24(f);\n+\n+\tf += 3;\n+\tr->min_update_index = get_be64(f);\n+\tf += 8;\n+\tr->max_update_index = get_be64(f);\n+\tf += 8;\n+\n+\tif (r->version == 1) {\n+\t\tr->hash_id = SHA1_ID;\n+\t} else {\n+\t\tr->hash_id = get_be32(f);\n+\t\tswitch (r->hash_id) {\n+\t\tcase SHA1_ID:\n+\t\t\tbreak;\n+\t\tcase SHA256_ID:\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tf += 4;\n+\t}\n+\n+\tr->ref_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->obj_offsets.offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1);\n+\tr->obj_offsets.offset >>= 5;\n+\n+\tr->obj_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tcomputed_crc = crc32(0, footer, f - footer);\n+\tfile_crc = get_be32(f);\n+\tf += 4;\n+\tif (computed_crc != file_crc) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tfirst_block_typ = header[header_size(r->version)];\n+\tr->ref_offsets.is_present = (first_block_typ == BLOCK_TYPE_REF);\n+\tr->ref_offsets.offset = 0;\n+\tr->log_offsets.is_present = (first_block_typ == BLOCK_TYPE_LOG ||\n+\t\t\t\t r->log_offsets.offset > 0);\n+\tr->obj_offsets.is_present = r->obj_offsets.offset > 0;\n+\terr = 0;\n+done:\n+\treturn err;\n+}\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name)\n+{\n+\tstruct reftable_block footer = { 0 };\n+\tstruct reftable_block header = { 0 };\n+\tint err = 0;\n+\n+\tmemset(r, 0, sizeof(struct reftable_reader));\n+\n+\t/* Need +1 to read type of first block. */\n+\terr = block_source_read_block(source, &header, 0, header_size(2) + 1);\n+\tif (err != header_size(2) + 1) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tif (memcmp(header.data, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tr->version = header.data[4];\n+\tif (r->version != 1 && r->version != 2) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tr->size = block_source_size(source) - footer_size(r->version);\n+\tr->source = *source;\n+\tr->name = xstrdup(name);\n+\tr->hash_id = 0;\n+\n+\terr = block_source_read_block(source, &footer, r->size,\n+\t\t\t\t footer_size(r->version));\n+\tif (err != footer_size(r->version)) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = parse_footer(r, footer.data, header.data);\n+done:\n+\treftable_block_done(&footer);\n+\treftable_block_done(&header);\n+\treturn err;\n+}\n+\n+struct table_iter {\n+\tstruct reftable_reader *r;\n+\tuint8_t typ;\n+\tuint64_t block_off;\n+\tstruct block_iter bi;\n+\tint is_finished;\n+};\n+#define TABLE_ITER_INIT \\\n+\t{ \\\n+\t\t.bi = {.last_key = STRBUF_INIT } \\\n+\t}\n+\n+static void table_iter_copy_from(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = src->block_off;\n+\tdest->is_finished = src->is_finished;\n+\tblock_iter_copy_from(&dest->bi, &src->bi);\n+}\n+\n+static int table_iter_next_in_block(struct table_iter *ti,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tint res = block_iter_next(&ti->bi, rec);\n+\tif (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {\n+\t\t((struct reftable_ref_record *)rec->data)->update_index +=\n+\t\t\tti->r->min_update_index;\n+\t}\n+\n+\treturn res;\n+}\n+\n+static void table_iter_block_done(struct table_iter *ti)\n+{\n+\tif (ti->bi.br == NULL) {\n+\t\treturn;\n+\t}\n+\treftable_block_done(&ti->bi.br->block);\n+\tFREE_AND_NULL(ti->bi.br);\n+\n+\tti->bi.last_key.len = 0;\n+\tti->bi.next_off = 0;\n+}\n+\n+static int32_t extract_block_size(uint8_t *data, uint8_t *typ, uint64_t off,\n+\t\t\t\t int version)\n+{\n+\tint32_t result = 0;\n+\n+\tif (off == 0) {\n+\t\tdata += header_size(version);\n+\t}\n+\n+\t*typ = data[0];\n+\tif (reftable_is_block_type(*typ)) {\n+\t\tresult = get_be24(data + 1);\n+\t}\n+\treturn result;\n+}\n+\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ)\n+{\n+\tint32_t guess_block_size = r->block_size ? r->block_size :\n+\t\t\t\t\t\t\t DEFAULT_BLOCK_SIZE;\n+\tstruct reftable_block block = { 0 };\n+\tuint8_t block_typ = 0;\n+\tint err = 0;\n+\tuint32_t header_off = next_off ? 0 : header_size(r->version);\n+\tint32_t block_size = 0;\n+\n+\tif (next_off >= r->size)\n+\t\treturn 1;\n+\n+\terr = reader_get_block(r, &block, next_off, guess_block_size);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tblock_size = extract_block_size(block.data, &block_typ, next_off,\n+\t\t\t\t\tr->version);\n+\tif (block_size < 0)\n+\t\treturn block_size;\n+\n+\tif (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {\n+\t\treftable_block_done(&block);\n+\t\treturn 1;\n+\t}\n+\n+\tif (block_size > guess_block_size) {\n+\t\treftable_block_done(&block);\n+\t\terr = reader_get_block(r, &block, next_off, block_size);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\treturn block_reader_init(br, &block, header_off, r->block_size,\n+\t\t\t\t hash_size(r->hash_id));\n+}\n+\n+static int table_iter_next_block(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tuint64_t next_block_off = src->block_off + src->bi.br->full_block_size;\n+\tstruct block_reader br = { 0 };\n+\tint err = 0;\n+\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = next_block_off;\n+\n+\terr = reader_init_block_reader(src->r, &br, next_block_off, src->typ);\n+\tif (err > 0) {\n+\t\tdest->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\tif (err != 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct block_reader *brp =\n+\t\t\treftable_malloc(sizeof(struct block_reader));\n+\t\t*brp = br;\n+\n+\t\tdest->is_finished = 0;\n+\t\tblock_reader_start(brp, &dest->bi);\n+\t}\n+\treturn 0;\n+}\n+\n+static int table_iter_next(struct table_iter *ti, struct reftable_record *rec)\n+{\n+\tif (reftable_record_type(rec) != ti->typ)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\twhile (1) {\n+\t\tstruct table_iter next = TABLE_ITER_INIT;\n+\t\tint err = 0;\n+\t\tif (ti->is_finished) {\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\terr = table_iter_next_in_block(ti, rec);\n+\t\tif (err <= 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err != 0) {\n+\t\t\tti->is_finished = 1;\n+\t\t}\n+\t\ttable_iter_block_done(ti);\n+\t\tif (err != 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\ttable_iter_copy_from(ti, &next);\n+\t\tblock_iter_close(&next.bi);\n+\t}\n+}\n+\n+static int table_iter_next_void(void *ti, struct reftable_record *rec)\n+{\n+\treturn table_iter_next((struct table_iter *)ti, rec);\n+}\n+\n+static void table_iter_close(void *p)\n+{\n+\tstruct table_iter *ti = (struct table_iter *)p;\n+\ttable_iter_block_done(ti);\n+\tblock_iter_close(&ti->bi);\n+}\n+\n+struct reftable_iterator_vtable table_iter_vtable = {\n+\t.next = &table_iter_next_void,\n+\t.close = &table_iter_close,\n+};\n+\n+static void iterator_from_table_iter(struct reftable_iterator *it,\n+\t\t\t\t struct table_iter *ti)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = ti;\n+\tit->ops = &table_iter_vtable;\n+}\n+\n+static int reader_table_iter_at(struct reftable_reader *r,\n+\t\t\t\tstruct table_iter *ti, uint64_t off,\n+\t\t\t\tuint8_t typ)\n+{\n+\tstruct block_reader br = { 0 };\n+\tstruct block_reader *brp = NULL;\n+\n+\tint err = reader_init_block_reader(r, &br, off, typ);\n+\tif (err != 0)\n+\t\treturn err;\n+\n+\tbrp = reftable_malloc(sizeof(struct block_reader));\n+\t*brp = br;\n+\tti->r = r;\n+\tti->typ = block_reader_type(brp);\n+\tti->block_off = off;\n+\tblock_reader_start(brp, &ti->bi);\n+\treturn 0;\n+}\n+\n+static int reader_start(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\tuint8_t typ, int index)\n+{\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tuint64_t off = offs->offset;\n+\tif (index) {\n+\t\toff = offs->index_offset;\n+\t\tif (off == 0) {\n+\t\t\treturn 1;\n+\t\t}\n+\t\ttyp = BLOCK_TYPE_INDEX;\n+\t}\n+\n+\treturn reader_table_iter_at(r, ti, off, typ);\n+}\n+\n+static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\t struct reftable_record *want)\n+{\n+\tstruct reftable_record rec =\n+\t\treftable_new_record(reftable_record_type(want));\n+\tstruct strbuf want_key = STRBUF_INIT;\n+\tstruct strbuf got_key = STRBUF_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = -1;\n+\n+\treftable_record_key(want, &want_key);\n+\n+\twhile (1) {\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = block_reader_first_key(next.bi.br, &got_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (strbuf_cmp(&got_key, &want_key) > 0) {\n+\t\t\ttable_iter_block_done(&next);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_block_done(ti);\n+\t\ttable_iter_copy_from(ti, &next);\n+\t}\n+\n+\terr = block_iter_seek(&ti->bi, &want_key);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = 0;\n+\n+done:\n+\tblock_iter_close(&next.bi);\n+\treftable_record_destroy(&rec);\n+\tstrbuf_release(&want_key);\n+\tstrbuf_release(&got_key);\n+\treturn err;\n+}\n+\n+static int reader_seek_indexed(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_index_record want_index = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record want_index_rec = { 0 };\n+\tstruct reftable_index_record index_result = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record index_result_rec = { 0 };\n+\tstruct table_iter index_iter = TABLE_ITER_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = 0;\n+\n+\treftable_record_key(rec, &want_index.last_key);\n+\treftable_record_from_index(&want_index_rec, &want_index);\n+\treftable_record_from_index(&index_result_rec, &index_result);\n+\n+\terr = reader_start(r, &index_iter, reftable_record_type(rec), 1);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reader_seek_linear(r, &index_iter, &want_index_rec);\n+\twhile (1) {\n+\t\terr = table_iter_next(&index_iter, &index_result_rec);\n+\t\ttable_iter_block_done(&index_iter);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = reader_table_iter_at(r, &next, index_result.offset, 0);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = block_iter_seek(&next.bi, &want_index.last_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (next.typ == reftable_record_type(rec)) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (next.typ != BLOCK_TYPE_INDEX) {\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_copy_from(&index_iter, &next);\n+\t}\n+\n+\tif (err == 0) {\n+\t\tstruct table_iter empty = TABLE_ITER_INIT;\n+\t\tstruct table_iter *malloced =\n+\t\t\treftable_calloc(sizeof(struct table_iter));\n+\t\t*malloced = empty;\n+\t\ttable_iter_copy_from(malloced, &next);\n+\t\titerator_from_table_iter(it, malloced);\n+\t}\n+done:\n+\tblock_iter_close(&next.bi);\n+\ttable_iter_close(&index_iter);\n+\treftable_record_clear(&want_index_rec);\n+\treftable_record_clear(&index_result_rec);\n+\treturn err;\n+}\n+\n+static int reader_seek_internal(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it,\n+\t\t\t\tstruct reftable_record *rec)\n+{\n+\tstruct reftable_reader_offsets *offs =\n+\t\treader_offsets_for(r, reftable_record_type(rec));\n+\tuint64_t idx = offs->index_offset;\n+\tstruct table_iter ti = TABLE_ITER_INIT;\n+\tint err = 0;\n+\tif (idx > 0)\n+\t\treturn reader_seek_indexed(r, it, rec);\n+\n+\terr = reader_start(r, &ti, reftable_record_type(rec), 0);\n+\tif (err < 0)\n+\t\treturn err;\n+\terr = reader_seek_linear(r, &ti, rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct table_iter *p =\n+\t\t\treftable_malloc(sizeof(struct table_iter));\n+\t\t*p = ti;\n+\t\titerator_from_table_iter(it, p);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec)\n+{\n+\tuint8_t typ = reftable_record_type(rec);\n+\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tif (!offs->is_present) {\n+\t\titerator_set_empty(it);\n+\t\treturn 0;\n+\t}\n+\n+\treturn reader_seek_internal(r, it, rec);\n+}\n+\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { 0 };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { 0 };\n+\treftable_record_from_log(&rec, &log);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_reader_seek_log_at(r, it, name, max);\n+}\n+\n+void reader_close(struct reftable_reader *r)\n+{\n+\tblock_source_close(&r->source);\n+\tFREE_AND_NULL(r->name);\n+}\n+\n+int reftable_new_reader(struct reftable_reader **p,\n+\t\t\tstruct reftable_block_source *src, char const *name)\n+{\n+\tstruct reftable_reader *rd =\n+\t\treftable_calloc(sizeof(struct reftable_reader));\n+\tint err = init_reader(rd, src, name);\n+\tif (err == 0) {\n+\t\t*p = rd;\n+\t} else {\n+\t\tblock_source_close(src);\n+\t\treftable_free(rd);\n+\t}\n+\treturn err;\n+}\n+\n+void reftable_reader_free(struct reftable_reader *r)\n+{\n+\treader_close(r);\n+\treftable_free(r);\n+}\n+\n+static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct reftable_obj_record want = {\n+\t\t.hash_prefix = oid,\n+\t\t.hash_prefix_len = r->object_id_len,\n+\t};\n+\tstruct reftable_record want_rec = { 0 };\n+\tstruct reftable_iterator oit = { 0 };\n+\tstruct reftable_obj_record got = { 0 };\n+\tstruct reftable_record got_rec = { 0 };\n+\tint err = 0;\n+\tstruct indexed_table_ref_iter *itr = NULL;\n+\n+\t/* Look through the reverse index. */\n+\treftable_record_from_obj(&want_rec, &want);\n+\terr = reader_seek(r, &oit, &want_rec);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\t/* read out the reftable_obj_record */\n+\treftable_record_from_obj(&got_rec, &got);\n+\terr = iterator_next(&oit, &got_rec);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 0 ||\n+\t memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {\n+\t\t/* didn't find it; return empty iterator */\n+\t\titerator_set_empty(it);\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),\n+\t\t\t\t\t got.offsets, got.offset_len);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tgot.offsets = NULL;\n+\titerator_from_indexed_table_ref_iter(it, itr);\n+\n+done:\n+\treftable_iterator_destroy(&oit);\n+\treftable_record_clear(&got_rec);\n+\treturn err;\n+}\n+\n+static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct table_iter ti_empty = TABLE_ITER_INIT;\n+\tstruct table_iter *ti = reftable_calloc(sizeof(struct table_iter));\n+\tstruct filtering_ref_iterator *filter = NULL;\n+\tstruct filtering_ref_iterator empty = FILTERING_REF_ITERATOR_INIT;\n+\tint oid_len = hash_size(r->hash_id);\n+\tint err;\n+\n+\t*ti = ti_empty;\n+\terr = reader_start(r, ti, BLOCK_TYPE_REF, 0);\n+\tif (err < 0) {\n+\t\treftable_free(ti);\n+\t\treturn err;\n+\t}\n+\n+\tfilter = reftable_malloc(sizeof(struct filtering_ref_iterator));\n+\t*filter = empty;\n+\n+\tstrbuf_add(&filter->oid, oid, oid_len);\n+\treftable_table_from_reader(&filter->tab, r);\n+\tfilter->double_check = 0;\n+\titerator_from_table_iter(&filter->it, ti);\n+\n+\titerator_from_filtering_ref_iterator(it, filter);\n+\treturn 0;\n+}\n+\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid)\n+{\n+\tif (r->obj_offsets.is_present)\n+\t\treturn reftable_reader_refs_for_indexed(r, it, oid);\n+\treturn reftable_reader_refs_for_unindexed(r, it, oid);\n+}\n+\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r)\n+{\n+\treturn r->max_update_index;\n+}\n+\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r)\n+{\n+\treturn r->min_update_index;\n+}\ndiff --git a/reftable/reader.h b/reftable/reader.h\nnew file mode 100644\nindex 0000000000..8170258881\n--- /dev/null\n+++ b/reftable/reader.h\n@@ -0,0 +1,78 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef READER_H\n+#define READER_H\n+\n+#include \"block.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source);\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size);\n+void block_source_close(struct reftable_block_source *source);\n+\n+/* metadata for a block type */\n+struct reftable_reader_offsets {\n+\tint is_present;\n+\tuint64_t offset;\n+\tuint64_t index_offset;\n+};\n+\n+/* The state for reading a reftable file. */\n+struct reftable_reader {\n+\t/* for convience, associate a name with the instance. */\n+\tchar *name;\n+\tstruct reftable_block_source source;\n+\n+\t/* Size of the file, excluding the footer. */\n+\tuint64_t size;\n+\n+\t/* 'sha1' for SHA1, 's256' for SHA-256 */\n+\tuint32_t hash_id;\n+\n+\tuint32_t block_size;\n+\tuint64_t min_update_index;\n+\tuint64_t max_update_index;\n+\t/* Length of the OID keys in the 'o' section */\n+\tint object_id_len;\n+\tint version;\n+\n+\tstruct reftable_reader_offsets ref_offsets;\n+\tstruct reftable_reader_offsets obj_offsets;\n+\tstruct reftable_reader_offsets log_offsets;\n+};\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name);\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec);\n+void reader_close(struct reftable_reader *r);\n+const char *reader_name(struct reftable_reader *r);\n+\n+/* initialize a block reader to read from `r` */\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ);\n+\n+/* generic interface to reftables */\n+struct reftable_table_vtable {\n+\tint (*seek_record)(void *tab, struct reftable_iterator *it,\n+\t\t\t struct reftable_record *);\n+\tuint32_t (*hash_id)(void *tab);\n+\tuint64_t (*min_update_index)(void *tab);\n+\tuint64_t (*max_update_index)(void *tab);\n+};\n+\n+int reftable_table_seek_record(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t struct reftable_record *rec);\n+\n+#endif\ndiff --git a/reftable/reftable.c b/reftable/reftable.c\nnew file mode 100644\nindex 0000000000..1425aef6f2\n--- /dev/null\n+++ b/reftable/reftable.c\n@@ -0,0 +1,104 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable.h\"\n+#include \"record.h\"\n+#include \"reader.h\"\n+\n+static int reftable_reader_seek_void(void *tab, struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn reader_seek((struct reftable_reader *)tab, it, rec);\n+}\n+\n+static uint32_t reftable_reader_hash_id_void(void *tab)\n+{\n+\treturn reftable_reader_hash_id((struct reftable_reader *)tab);\n+}\n+\n+static uint64_t reftable_reader_min_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_min_update_index((struct reftable_reader *)tab);\n+}\n+\n+static uint64_t reftable_reader_max_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_max_update_index((struct reftable_reader *)tab);\n+}\n+\n+static struct reftable_table_vtable reader_vtable = {\n+\t.seek_record = reftable_reader_seek_void,\n+\t.hash_id = reftable_reader_hash_id_void,\n+\t.min_update_index = reftable_reader_min_update_index_void,\n+\t.max_update_index = reftable_reader_max_update_index_void,\n+};\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { 0 };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader)\n+{\n+\tassert(tab->ops == NULL);\n+\ttab->ops = &reader_vtable;\n+\ttab->table_arg = reader;\n+}\n+\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_iterator it = { 0 };\n+\tint err = reftable_table_seek_ref(tab, &it, name);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_ref(&it, ref);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref->refname, name) ||\n+\t reftable_ref_record_is_deletion(ref)) {\n+\t\treftable_ref_record_clear(ref);\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+int reftable_table_seek_record(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t struct reftable_record *rec)\n+{\n+\treturn tab->ops->seek_record(tab->table_arg, it, rec);\n+}\n+\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->max_update_index(tab->table_arg);\n+}\n+\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->min_update_index(tab->table_arg);\n+}\n+\n+uint32_t reftable_table_hash_id(struct reftable_table *tab)\n+{\n+\treturn tab->ops->hash_id(tab->table_arg);\n+}\n-- \ngitgitgadget\n\n"},{"id":"405659","messageId":"64d98e60b2609f6b4d27f69545d16bd00c8578ba.1600283416.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.git.git.1600283416.gitgitgadget@gmail.com","subject":"[PATCH 12/13] reftable: rest of library","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-16T19:10:15Z","receivedAt":"2020-09-16T19:11:23Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis will be further split up once preceding commits have passed review.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 8 +-\n reftable/VERSION | 1 +\n reftable/dump.c | 212 +++++++\n reftable/merged.c | 358 +++++++++++\n reftable/merged.h | 39 ++\n reftable/merged_test.c | 331 +++++++++++\n reftable/pq.c | 115 ++++\n reftable/pq.h | 34 ++\n reftable/refname.c | 209 +++++++\n reftable/refname.h | 38 ++\n reftable/refname_test.c | 100 ++++\n reftable/stack.c | 1224 ++++++++++++++++++++++++++++++++++++++\n reftable/stack.h | 50 ++\n reftable/stack_test.c | 787 ++++++++++++++++++++++++\n reftable/update.sh | 22 +\n t/helper/test-reftable.c | 3 +\n 16 files changed, 3530 insertions(+), 1 deletion(-)\n create mode 100644 reftable/VERSION\n create mode 100644 reftable/dump.c\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n create mode 100755 reftable/update.sh\n\ndiff --git a/Makefile b/Makefile\nindex 22dd31e7ff..6652259baf 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2347,12 +2347,16 @@ XDIFF_OBJS += xdiff/xutils.o\n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n-REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/compat.o\n REFTABLE_OBJS += reftable/iter.o\n+REFTABLE_OBJS += reftable/merged.o\n+REFTABLE_OBJS += reftable/pq.o\n+REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/refname.o\n REFTABLE_OBJS += reftable/reftable.o\n+REFTABLE_OBJS += reftable/stack.o\n REFTABLE_OBJS += reftable/strbuf.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n@@ -2360,7 +2364,9 @@ REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n+REFTABLE_TEST_OBJS += reftable/refname_test.o\n REFTABLE_TEST_OBJS += reftable/reftable_test.o\n+REFTABLE_TEST_OBJS += reftable/stack_test.o\n REFTABLE_TEST_OBJS += reftable/strbuf_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\ndiff --git a/reftable/VERSION b/reftable/VERSION\nnew file mode 100644\nindex 0000000000..7553c395eb\n--- /dev/null\n+++ b/reftable/VERSION\n@@ -0,0 +1 @@\n+b8ec0f74c74cb6752eb2033ad8e755a9c19aad15 C: add missing header\ndiff --git a/reftable/dump.c b/reftable/dump.c\nnew file mode 100644\nindex 0000000000..00b444e8c9\n--- /dev/null\n+++ b/reftable/dump.c\n@@ -0,0 +1,212 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+\n+#include \"reftable.h\"\n+#include \"reftable-tests.h\"\n+\n+static uint32_t hash_id;\n+\n+static int dump_table(const char *tablename)\n+{\n+\tstruct reftable_block_source src = { 0 };\n+\tint err = reftable_block_source_from_file(&src, tablename);\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_ref_record ref = { 0 };\n+\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_reader *r = NULL;\n+\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\terr = reftable_new_reader(&r, &src, tablename);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\terr = reftable_reader_seek_ref(r, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_ref_record_print(&ref, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_clear(&ref);\n+\n+\terr = reftable_reader_seek_log(r, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_log_record_print(&log, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_clear(&log);\n+\n+\treftable_reader_free(r);\n+\treturn 0;\n+}\n+\n+static int compact_stack(const char *stackdir)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = {};\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_stack_compact_all(stack, NULL);\n+\tif (err < 0)\n+\t\tgoto done;\n+done:\n+\tif (stack != NULL) {\n+\t\treftable_stack_destroy(stack);\n+\t}\n+\treturn err;\n+}\n+\n+static int dump_stack(const char *stackdir)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = {};\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_ref_record ref = { 0 };\n+\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_merged_table *merged = NULL;\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tmerged = reftable_stack_merged_table(stack);\n+\n+\terr = reftable_merged_table_seek_ref(merged, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_ref_record_print(&ref, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_clear(&ref);\n+\n+\terr = reftable_merged_table_seek_log(merged, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_log_record_print(&log, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_clear(&log);\n+\n+\treftable_stack_destroy(stack);\n+\treturn 0;\n+}\n+\n+static void print_help(void)\n+{\n+\tprintf(\"usage: dump [-cst] arg\\n\\n\"\n+\t \"options: \\n\"\n+\t \" -c compact\\n\"\n+\t \" -t dump table\\n\"\n+\t \" -s dump stack\\n\"\n+\t \" -h this help\\n\"\n+\t \" -2 use SHA256\\n\"\n+\t \"\\n\");\n+}\n+\n+int reftable_dump_main(int argc, char *const *argv)\n+{\n+\tint err = 0;\n+\tint opt;\n+\tint opt_dump_table = 0;\n+\tint opt_dump_stack = 0;\n+\tint opt_compact = 0;\n+\tconst char *arg = NULL;\n+\twhile ((opt = getopt(argc, argv, \"2chts\")) != -1) {\n+\t\tswitch (opt) {\n+\t\tcase '2':\n+\t\t\thash_id = 0x73323536;\n+\t\t\tbreak;\n+\t\tcase 't':\n+\t\t\topt_dump_table = 1;\n+\t\t\tbreak;\n+\t\tcase 's':\n+\t\t\topt_dump_stack = 1;\n+\t\t\tbreak;\n+\t\tcase 'c':\n+\t\t\topt_compact = 1;\n+\t\t\tbreak;\n+\t\tcase '?':\n+\t\tcase 'h':\n+\t\t\tprint_help();\n+\t\t\treturn 2;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (argv[optind] == NULL) {\n+\t\tfprintf(stderr, \"need argument\\n\");\n+\t\tprint_help();\n+\t\treturn 2;\n+\t}\n+\n+\targ = argv[optind];\n+\n+\tif (opt_dump_table) {\n+\t\terr = dump_table(arg);\n+\t} else if (opt_dump_stack) {\n+\t\terr = dump_stack(arg);\n+\t} else if (opt_compact) {\n+\t\terr = compact_stack(arg);\n+\t}\n+\n+\tif (err < 0) {\n+\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv[0], arg,\n+\t\t\treftable_error_str(err));\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\ndiff --git a/reftable/merged.c b/reftable/merged.c\nnew file mode 100644\nindex 0000000000..22b071cb5d\n--- /dev/null\n+++ b/reftable/merged.c\n@@ -0,0 +1,358 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"constants.h\"\n+#include \"iter.h\"\n+#include \"pq.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+#include \"system.h\"\n+\n+static int merged_iter_init(struct merged_iter *mi)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\t\tint err = iterator_next(&mi->stack[i], &rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t\t\treftable_record_destroy(&rec);\n+\t\t} else {\n+\t\t\tstruct pq_entry e = {\n+\t\t\t\t.rec = rec,\n+\t\t\t\t.index = i,\n+\t\t\t};\n+\t\t\tmerged_iter_pqueue_add(&mi->pq, e);\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void merged_iter_close(void *p)\n+{\n+\tstruct merged_iter *mi = (struct merged_iter *)p;\n+\tint i = 0;\n+\tmerged_iter_pqueue_clear(&mi->pq);\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t}\n+\treftable_free(mi->stack);\n+}\n+\n+static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,\n+\t\t\t\t\t size_t idx)\n+{\n+\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\tstruct pq_entry e = {\n+\t\t.rec = rec,\n+\t\t.index = idx,\n+\t};\n+\tint err = iterator_next(&mi->stack[idx], &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (err > 0) {\n+\t\treftable_iterator_destroy(&mi->stack[idx]);\n+\t\treftable_record_destroy(&rec);\n+\t\treturn 0;\n+\t}\n+\n+\tmerged_iter_pqueue_add(&mi->pq, e);\n+\treturn 0;\n+}\n+\n+static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)\n+{\n+\tif (iterator_is_null(&mi->stack[idx]))\n+\t\treturn 0;\n+\treturn merged_iter_advance_nonnull_subiter(mi, idx);\n+}\n+\n+static int merged_iter_next_entry(struct merged_iter *mi,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct strbuf entry_key = STRBUF_INIT;\n+\tstruct pq_entry entry = { 0 };\n+\tint err = 0;\n+\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\tentry = merged_iter_pqueue_remove(&mi->pq);\n+\terr = merged_iter_advance_subiter(mi, entry.index);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\t/*\n+\t One can also use reftable as datacenter-local storage, where the ref\n+\t database is maintained in globally consistent database (eg.\n+\t CockroachDB or Spanner). In this scenario, replication delays together\n+\t with compaction may cause newer tables to contain older entries. In\n+\t such a deployment, the loop below must be changed to collect all\n+\t entries for the same key, and return new the newest one.\n+\t*/\n+\treftable_record_key(&entry.rec, &entry_key);\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq)) {\n+\t\tstruct pq_entry top = merged_iter_pqueue_top(mi->pq);\n+\t\tstruct strbuf k = STRBUF_INIT;\n+\t\tint err = 0, cmp = 0;\n+\n+\t\treftable_record_key(&top.rec, &k);\n+\n+\t\tcmp = strbuf_cmp(&k, &entry_key);\n+\t\tstrbuf_release(&k);\n+\n+\t\tif (cmp > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\t\terr = merged_iter_advance_subiter(mi, top.index);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_record_destroy(&top.rec);\n+\t}\n+\n+\treftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));\n+\treftable_record_destroy(&entry.rec);\n+\tstrbuf_release(&entry_key);\n+\treturn 0;\n+}\n+\n+static int merged_iter_next(struct merged_iter *mi, struct reftable_record *rec)\n+{\n+\twhile (1) {\n+\t\tint err = merged_iter_next_entry(mi, rec);\n+\t\tif (err == 0 && mi->suppress_deletions &&\n+\t\t reftable_record_is_deletion(rec)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\treturn err;\n+\t}\n+}\n+\n+static int merged_iter_next_void(void *p, struct reftable_record *rec)\n+{\n+\tstruct merged_iter *mi = (struct merged_iter *)p;\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\treturn merged_iter_next(mi, rec);\n+}\n+\n+struct reftable_iterator_vtable merged_iter_vtable = {\n+\t.next = &merged_iter_next_void,\n+\t.close = &merged_iter_close,\n+};\n+\n+static void iterator_from_merged_iter(struct reftable_iterator *it,\n+\t\t\t\t struct merged_iter *mi)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = mi;\n+\tit->ops = &merged_iter_vtable;\n+}\n+\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id)\n+{\n+\tstruct reftable_merged_table *m = NULL;\n+\tuint64_t last_max = 0;\n+\tuint64_t first_min = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t min = reftable_table_min_update_index(&stack[i]);\n+\t\tuint64_t max = reftable_table_max_update_index(&stack[i]);\n+\n+\t\tif (reftable_table_hash_id(&stack[i]) != hash_id) {\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\t\t}\n+\t\tif (i == 0 || min < first_min) {\n+\t\t\tfirst_min = min;\n+\t\t}\n+\t\tif (i == 0 || max > last_max) {\n+\t\t\tlast_max = max;\n+\t\t}\n+\t}\n+\n+\tm = (struct reftable_merged_table *)reftable_calloc(\n+\t\tsizeof(struct reftable_merged_table));\n+\tm->stack = stack;\n+\tm->stack_len = n;\n+\tm->min = first_min;\n+\tm->max = last_max;\n+\tm->hash_id = hash_id;\n+\t*dest = m;\n+\treturn 0;\n+}\n+\n+/* clears the list of subtable, without affecting the readers themselves. */\n+void merged_table_clear(struct reftable_merged_table *mt)\n+{\n+\tFREE_AND_NULL(mt->stack);\n+\tmt->stack_len = 0;\n+}\n+\n+void reftable_merged_table_free(struct reftable_merged_table *mt)\n+{\n+\tif (mt == NULL) {\n+\t\treturn;\n+\t}\n+\tmerged_table_clear(mt);\n+\treftable_free(mt);\n+}\n+\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->max;\n+}\n+\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->min;\n+}\n+\n+int merged_table_seek_record(struct reftable_merged_table *mt,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_iterator *iters = reftable_calloc(\n+\t\tsizeof(struct reftable_iterator) * mt->stack_len);\n+\tstruct merged_iter merged = {\n+\t\t.stack = iters,\n+\t\t.typ = reftable_record_type(rec),\n+\t\t.hash_id = mt->hash_id,\n+\t\t.suppress_deletions = mt->suppress_deletions,\n+\t};\n+\tint n = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < mt->stack_len && err == 0; i++) {\n+\t\tint e = reftable_table_seek_record(&mt->stack[i], &iters[n],\n+\t\t\t\t\t\t rec);\n+\t\tif (e < 0) {\n+\t\t\terr = e;\n+\t\t}\n+\t\tif (e == 0) {\n+\t\t\tn++;\n+\t\t}\n+\t}\n+\tif (err < 0) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < n; i++) {\n+\t\t\treftable_iterator_destroy(&iters[i]);\n+\t\t}\n+\t\treftable_free(iters);\n+\t\treturn err;\n+\t}\n+\n+\tmerged.stack_len = n;\n+\terr = merged_iter_init(&merged);\n+\tif (err < 0) {\n+\t\tmerged_iter_close(&merged);\n+\t\treturn err;\n+\t} else {\n+\t\tstruct merged_iter *p =\n+\t\t\treftable_malloc(sizeof(struct merged_iter));\n+\t\t*p = merged;\n+\t\titerator_from_merged_iter(it, p);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { 0 };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { 0 };\n+\treftable_record_from_log(&rec, &log);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_merged_table_seek_log_at(mt, it, name, max);\n+}\n+\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\n+{\n+\treturn mt->hash_id;\n+}\n+\n+static int reftable_merged_table_seek_void(void *tab,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn merged_table_seek_record((struct reftable_merged_table *)tab, it,\n+\t\t\t\t\trec);\n+}\n+\n+static uint32_t reftable_merged_table_hash_id_void(void *tab)\n+{\n+\treturn reftable_merged_table_hash_id(\n+\t\t(struct reftable_merged_table *)tab);\n+}\n+\n+static uint64_t reftable_merged_table_min_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_min_update_index(\n+\t\t(struct reftable_merged_table *)tab);\n+}\n+\n+static uint64_t reftable_merged_table_max_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_max_update_index(\n+\t\t(struct reftable_merged_table *)tab);\n+}\n+\n+static struct reftable_table_vtable merged_table_vtable = {\n+\t.seek_record = reftable_merged_table_seek_void,\n+\t.hash_id = reftable_merged_table_hash_id_void,\n+\t.min_update_index = reftable_merged_table_min_update_index_void,\n+\t.max_update_index = reftable_merged_table_max_update_index_void,\n+};\n+\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *merged)\n+{\n+\tassert(tab->ops == NULL);\n+\ttab->ops = &merged_table_vtable;\n+\ttab->table_arg = merged;\n+}\ndiff --git a/reftable/merged.h b/reftable/merged.h\nnew file mode 100644\nindex 0000000000..66a1d71830\n--- /dev/null\n+++ b/reftable/merged.h\n@@ -0,0 +1,39 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef MERGED_H\n+#define MERGED_H\n+\n+#include \"pq.h\"\n+#include \"reftable.h\"\n+\n+struct reftable_merged_table {\n+\tstruct reftable_table *stack;\n+\tsize_t stack_len;\n+\tuint32_t hash_id;\n+\tint suppress_deletions;\n+\n+\tuint64_t min;\n+\tuint64_t max;\n+};\n+\n+struct merged_iter {\n+\tstruct reftable_iterator *stack;\n+\tuint32_t hash_id;\n+\tsize_t stack_len;\n+\tuint8_t typ;\n+\tint suppress_deletions;\n+\tstruct merged_iter_pqueue pq;\n+};\n+\n+void merged_table_clear(struct reftable_merged_table *mt);\n+int merged_table_seek_record(struct reftable_merged_table *mt,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t struct reftable_record *rec);\n+\n+#endif\ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nnew file mode 100644\nindex 0000000000..18a0fe1dac\n--- /dev/null\n+++ b/reftable/merged_test.c\n@@ -0,0 +1,331 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"pq.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_pq(void)\n+{\n+\tchar *names[54] = { 0 };\n+\tint N = ARRAY_SIZE(names) - 1;\n+\n+\tstruct merged_iter_pqueue pq = { 0 };\n+\tconst char *last = NULL;\n+\n+\tint i = 0;\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tsnprintf(name, sizeof(name), \"%02d\", i);\n+\t\tnames[i] = xstrdup(name);\n+\t}\n+\n+\ti = 1;\n+\tdo {\n+\t\tstruct reftable_record rec =\n+\t\t\treftable_new_record(BLOCK_TYPE_REF);\n+\t\tstruct pq_entry e = { 0 };\n+\n+\t\treftable_record_as_ref(&rec)->refname = names[i];\n+\t\te.rec = rec;\n+\t\tmerged_iter_pqueue_add(&pq, e);\n+\t\tmerged_iter_pqueue_check(pq);\n+\t\ti = (i * 7) % N;\n+\t} while (i != 1);\n+\n+\twhile (!merged_iter_pqueue_is_empty(pq)) {\n+\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct reftable_ref_record *ref =\n+\t\t\treftable_record_as_ref(&e.rec);\n+\n+\t\tmerged_iter_pqueue_check(pq);\n+\n+\t\tif (last != NULL) {\n+\t\t\tassert(strcmp(last, ref->refname) < 0);\n+\t\t}\n+\t\tlast = ref->refname;\n+\t\tref->refname = NULL;\n+\t\treftable_free(ref);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\n+\tmerged_iter_pqueue_clear(&pq);\n+}\n+\n+static void write_test_table(struct strbuf *buf,\n+\t\t\t struct reftable_ref_record refs[], int n)\n+{\n+\tint min = 0xffffffff;\n+\tint max = 0;\n+\tint i = 0;\n+\tint err;\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_writer *w = NULL;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t ui = refs[i].update_index;\n+\t\tif (ui > max) {\n+\t\t\tmax = ui;\n+\t\t}\n+\t\tif (ui < min) {\n+\t\t\tmin = ui;\n+\t\t}\n+\t}\n+\n+\tw = reftable_new_writer(&strbuf_add_void, buf, &opts);\n+\treftable_writer_set_limits(w, min, max);\n+\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t before = refs[i].update_index;\n+\t\tint n = reftable_writer_add_ref(w, &refs[i]);\n+\t\tassert(n == 0);\n+\t\tassert(before == refs[i].update_index);\n+\t}\n+\n+\terr = reftable_writer_close(w);\n+\tassert_err(err);\n+\n+\treftable_writer_free(w);\n+}\n+\n+static struct reftable_merged_table *\n+merged_table_from_records(struct reftable_ref_record **refs,\n+\t\t\t struct reftable_block_source **source,\n+\t\t\t struct reftable_reader ***readers, int *sizes,\n+\t\t\t struct strbuf *buf, int n)\n+{\n+\tint i = 0;\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err;\n+\tstruct reftable_table *tabs =\n+\t\treftable_calloc(n * sizeof(struct reftable_table));\n+\t*readers = reftable_calloc(n * sizeof(struct reftable_reader *));\n+\t*source = reftable_calloc(n * sizeof(**source));\n+\tfor (i = 0; i < n; i++) {\n+\t\twrite_test_table(&buf[i], refs[i], sizes[i]);\n+\t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n+\n+\t\terr = reftable_new_reader(&(*readers)[i], &(*source)[i],\n+\t\t\t\t\t \"name\");\n+\t\tassert_err(err);\n+\t\treftable_table_from_reader(&tabs[i], (*readers)[i]);\n+\t}\n+\n+\terr = reftable_new_merged_table(&mt, tabs, n, SHA1_ID);\n+\tassert_err(err);\n+\treturn mt;\n+}\n+\n+static void readers_destroy(struct reftable_reader **readers, size_t n)\n+{\n+\tint i = 0;\n+\tfor (; i < n; i++)\n+\t\treftable_reader_free(readers[i]);\n+\treftable_free(readers);\n+}\n+\n+static void test_merged_between(void)\n+{\n+\tuint8_t hash1[SHA1_SIZE] = { 1, 2, 3, 0 };\n+\n+\tstruct reftable_ref_record r1[] = { {\n+\t\t.refname = \"b\",\n+\t\t.update_index = 1,\n+\t\t.value = hash1,\n+\t} };\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t} };\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2 };\n+\tint sizes[] = { 1, 1 };\n+\tstruct strbuf bufs[2] = { STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 2);\n+\tint i;\n+\tstruct reftable_ref_record ref = { 0 };\n+\tstruct reftable_iterator it = { 0 };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tassert_err(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tassert_err(err);\n+\tassert(ref.update_index == 2);\n+\treftable_ref_record_clear(&ref);\n+\treftable_iterator_destroy(&it);\n+\treaders_destroy(readers, 2);\n+\treftable_merged_table_free(mt);\n+\tfor (i = 0; i < ARRAY_SIZE(bufs); i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treftable_free(bs);\n+}\n+\n+static void test_merged(void)\n+{\n+\tuint8_t hash1[SHA1_SIZE] = { 1 };\n+\tuint8_t hash2[SHA1_SIZE] = { 2 };\n+\tstruct reftable_ref_record r1[] = { {\n+\t\t\t\t\t\t .refname = \"a\",\n+\t\t\t\t\t\t .update_index = 1,\n+\t\t\t\t\t\t .value = hash1,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .refname = \"b\",\n+\t\t\t\t\t\t .update_index = 1,\n+\t\t\t\t\t\t .value = hash1,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .refname = \"c\",\n+\t\t\t\t\t\t .update_index = 1,\n+\t\t\t\t\t\t .value = hash1,\n+\t\t\t\t\t } };\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t} };\n+\tstruct reftable_ref_record r3[] = {\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value = hash2,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"d\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value = hash1,\n+\t\t},\n+\t};\n+\n+\tstruct reftable_ref_record want[] = {\n+\t\tr2[0],\n+\t\tr1[1],\n+\t\tr3[0],\n+\t\tr3[1],\n+\t};\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2, r3 };\n+\tint sizes[3] = { 3, 1, 2 };\n+\tstruct strbuf bufs[3] = { STRBUF_INIT, STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 3);\n+\n+\tstruct reftable_iterator it = { 0 };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tstruct reftable_ref_record *out = NULL;\n+\tsize_t len = 0;\n+\tsize_t cap = 0;\n+\tint i = 0;\n+\n+\tassert_err(err);\n+\twhile (len < 100) { /* cap loops/recursion. */\n+\t\tstruct reftable_ref_record ref = { 0 };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tout = reftable_realloc(\n+\t\t\t\tout, sizeof(struct reftable_ref_record) * cap);\n+\t\t}\n+\t\tout[len++] = ref;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\tassert(ARRAY_SIZE(want) == len);\n+\tfor (i = 0; i < len; i++) {\n+\t\tassert(reftable_ref_record_equal(&want[i], &out[i], SHA1_SIZE));\n+\t}\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_clear(&out[i]);\n+\t}\n+\treftable_free(out);\n+\n+\tfor (i = 0; i < 3; i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treaders_destroy(readers, 3);\n+\treftable_merged_table_free(mt);\n+\treftable_free(bs);\n+}\n+\n+static void test_default_write_opts(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"master\",\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tstruct reftable_block_source source = { 0 };\n+\tstruct reftable_table *tab = reftable_calloc(sizeof(*tab) * 1);\n+\tuint32_t hash_id;\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_merged_table *merged = NULL;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tassert_err(err);\n+\n+\terr = reftable_writer_close(w);\n+\tassert_err(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tassert_err(err);\n+\n+\thash_id = reftable_reader_hash_id(rd);\n+\tassert(hash_id == SHA1_ID);\n+\n+\treftable_table_from_reader(&tab[0], rd);\n+\terr = reftable_new_merged_table(&merged, tab, 1, SHA1_ID);\n+\tassert_err(err);\n+\n+\treftable_reader_free(rd);\n+\treftable_merged_table_free(merged);\n+\tstrbuf_release(&buf);\n+}\n+\n+/* XXX test refs_for(oid) */\n+\n+int merged_test_main(int argc, const char *argv[])\n+{\n+\tadd_test_case(\"test_merged_between\", &test_merged_between);\n+\tadd_test_case(\"test_pq\", &test_pq);\n+\tadd_test_case(\"test_merged\", &test_merged);\n+\tadd_test_case(\"test_default_write_opts\", &test_default_write_opts);\n+\treturn test_main(argc, argv);\n+}\ndiff --git a/reftable/pq.c b/reftable/pq.c\nnew file mode 100644\nindex 0000000000..5fa1b5f437\n--- /dev/null\n+++ b/reftable/pq.c\n@@ -0,0 +1,115 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"pq.h\"\n+\n+#include \"reftable.h\"\n+#include \"system.h\"\n+#include \"basics.h\"\n+\n+int pq_less(struct pq_entry a, struct pq_entry b)\n+{\n+\tstruct strbuf ak = STRBUF_INIT;\n+\tstruct strbuf bk = STRBUF_INIT;\n+\tint cmp = 0;\n+\treftable_record_key(&a.rec, &ak);\n+\treftable_record_key(&b.rec, &bk);\n+\n+\tcmp = strbuf_cmp(&ak, &bk);\n+\n+\tstrbuf_release(&ak);\n+\tstrbuf_release(&bk);\n+\n+\tif (cmp == 0)\n+\t\treturn a.index > b.index;\n+\n+\treturn cmp < 0;\n+}\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.heap[0];\n+}\n+\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.len == 0;\n+}\n+\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq)\n+{\n+\tint i = 0;\n+\tfor (i = 1; i < pq.len; i++) {\n+\t\tint parent = (i - 1) / 2;\n+\n+\t\tassert(pq_less(pq.heap[parent], pq.heap[i]));\n+\t}\n+}\n+\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tstruct pq_entry e = pq->heap[0];\n+\tpq->heap[0] = pq->heap[pq->len - 1];\n+\tpq->len--;\n+\n+\ti = 0;\n+\twhile (i < pq->len) {\n+\t\tint min = i;\n+\t\tint j = 2 * i + 1;\n+\t\tint k = 2 * i + 2;\n+\t\tif (j < pq->len && pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tmin = j;\n+\t\t}\n+\t\tif (k < pq->len && pq_less(pq->heap[k], pq->heap[min])) {\n+\t\t\tmin = k;\n+\t\t}\n+\n+\t\tif (min == i) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[i], pq->heap[min]);\n+\t\ti = min;\n+\t}\n+\n+\treturn e;\n+}\n+\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)\n+{\n+\tint i = 0;\n+\tif (pq->len == pq->cap) {\n+\t\tpq->cap = 2 * pq->cap + 1;\n+\t\tpq->heap = reftable_realloc(pq->heap,\n+\t\t\t\t\t pq->cap * sizeof(struct pq_entry));\n+\t}\n+\n+\tpq->heap[pq->len++] = e;\n+\ti = pq->len - 1;\n+\twhile (i > 0) {\n+\t\tint j = (i - 1) / 2;\n+\t\tif (pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[j], pq->heap[i]);\n+\n+\t\ti = j;\n+\t}\n+}\n+\n+void merged_iter_pqueue_clear(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < pq->len; i++) {\n+\t\treftable_record_destroy(&pq->heap[i].rec);\n+\t}\n+\tFREE_AND_NULL(pq->heap);\n+\tpq->len = pq->cap = 0;\n+}\ndiff --git a/reftable/pq.h b/reftable/pq.h\nnew file mode 100644\nindex 0000000000..918ac3990d\n--- /dev/null\n+++ b/reftable/pq.h\n@@ -0,0 +1,34 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef PQ_H\n+#define PQ_H\n+\n+#include \"record.h\"\n+\n+struct pq_entry {\n+\tint index;\n+\tstruct reftable_record rec;\n+};\n+\n+int pq_less(struct pq_entry a, struct pq_entry b);\n+\n+struct merged_iter_pqueue {\n+\tstruct pq_entry *heap;\n+\tsize_t len;\n+\tsize_t cap;\n+};\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq);\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq);\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq);\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e);\n+void merged_iter_pqueue_clear(struct merged_iter_pqueue *pq);\n+\n+#endif\ndiff --git a/reftable/refname.c b/reftable/refname.c\nnew file mode 100644\nindex 0000000000..423e2b06bc\n--- /dev/null\n+++ b/reftable/refname.c\n@@ -0,0 +1,209 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"reftable.h\"\n+#include \"basics.h\"\n+#include \"refname.h\"\n+#include \"strbuf.h\"\n+\n+struct find_arg {\n+\tchar **names;\n+\tconst char *want;\n+};\n+\n+static int find_name(size_t k, void *arg)\n+{\n+\tstruct find_arg *f_arg = (struct find_arg *)arg;\n+\treturn strcmp(f_arg->names[k], f_arg->want) >= 0;\n+}\n+\n+int modification_has_ref(struct modification *mod, const char *name)\n+{\n+\tstruct reftable_ref_record ref = { 0 };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len && !strcmp(mod->add[idx], name)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tif (mod->del_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->del,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\tif (idx < mod->del_len && !strcmp(mod->del[idx], name)) {\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\terr = reftable_table_read_ref(&mod->tab, name, &ref);\n+\treftable_ref_record_clear(&ref);\n+\treturn err;\n+}\n+\n+static void modification_clear(struct modification *mod)\n+{\n+\t/* don't delete the strings themselves; they're owned by ref records.\n+\t */\n+\tFREE_AND_NULL(mod->add);\n+\tFREE_AND_NULL(mod->del);\n+\tmod->add_len = 0;\n+\tmod->del_len = 0;\n+}\n+\n+int modification_has_ref_with_prefix(struct modification *mod,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = prefix,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len &&\n+\t\t !strncmp(prefix, mod->add[idx], strlen(prefix)))\n+\t\t\tgoto done;\n+\t}\n+\terr = reftable_table_seek_ref(&mod->tab, &it, prefix);\n+\tif (err)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\n+\t\tif (mod->del_len > 0) {\n+\t\t\tstruct find_arg arg = {\n+\t\t\t\t.names = mod->del,\n+\t\t\t\t.want = ref.refname,\n+\t\t\t};\n+\t\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\t\tif (idx < mod->del_len &&\n+\t\t\t !strcmp(ref.refname, mod->del[idx])) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (strncmp(ref.refname, prefix, strlen(prefix))) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_ref_record_clear(&ref);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+int validate_refname(const char *name)\n+{\n+\twhile (1) {\n+\t\tchar *next = strchr(name, '/');\n+\t\tif (!*name) {\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\t}\n+\t\tif (!next) {\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (next - name == 0 || (next - name == 1 && *name == '.') ||\n+\t\t (next - name == 2 && name[0] == '.' && name[1] == '.'))\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\tname = next + 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz)\n+{\n+\tstruct modification mod = {\n+\t\t.tab = tab,\n+\t\t.add = reftable_calloc(sizeof(char *) * sz),\n+\t\t.del = reftable_calloc(sizeof(char *) * sz),\n+\t};\n+\tint i = 0;\n+\tint err = 0;\n+\tfor (; i < sz; i++) {\n+\t\tif (reftable_ref_record_is_deletion(&recs[i])) {\n+\t\t\tmod.del[mod.del_len++] = recs[i].refname;\n+\t\t} else {\n+\t\t\tmod.add[mod.add_len++] = recs[i].refname;\n+\t\t}\n+\t}\n+\n+\terr = modification_validate(&mod);\n+\tmodification_clear(&mod);\n+\treturn err;\n+}\n+\n+static void strbuf_trim_component(struct strbuf *sl)\n+{\n+\twhile (sl->len > 0) {\n+\t\tint is_slash = (sl->buf[sl->len - 1] == '/');\n+\t\tstrbuf_setlen(sl, sl->len - 1);\n+\t\tif (is_slash)\n+\t\t\tbreak;\n+\t}\n+}\n+\n+int modification_validate(struct modification *mod)\n+{\n+\tstruct strbuf slashed = STRBUF_INIT;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (; i < mod->add_len; i++) {\n+\t\terr = validate_refname(mod->add[i]);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\tstrbuf_addstr(&slashed, \"/\");\n+\n+\t\terr = modification_has_ref_with_prefix(mod, slashed.buf);\n+\t\tif (err == 0) {\n+\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\twhile (slashed.len) {\n+\t\t\tstrbuf_trim_component(&slashed);\n+\t\t\terr = modification_has_ref(mod, slashed.buf);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\terr = 0;\n+done:\n+\tstrbuf_release(&slashed);\n+\treturn err;\n+}\ndiff --git a/reftable/refname.h b/reftable/refname.h\nnew file mode 100644\nindex 0000000000..13bf1e6861\n--- /dev/null\n+++ b/reftable/refname.h\n@@ -0,0 +1,38 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+#ifndef REFNAME_H\n+#define REFNAME_H\n+\n+#include \"reftable.h\"\n+\n+struct modification {\n+\tstruct reftable_table tab;\n+\n+\tchar **add;\n+\tsize_t add_len;\n+\n+\tchar **del;\n+\tsize_t del_len;\n+};\n+\n+// -1 = error, 0 = found, 1 = not found\n+int modification_has_ref(struct modification *mod, const char *name);\n+\n+// -1 = error, 0 = found, 1 = not found.\n+int modification_has_ref_with_prefix(struct modification *mod,\n+\t\t\t\t const char *prefix);\n+\n+// 0 = OK.\n+int validate_refname(const char *name);\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz);\n+\n+int modification_validate(struct modification *mod);\n+\n+#endif\ndiff --git a/reftable/refname_test.c b/reftable/refname_test.c\nnew file mode 100644\nindex 0000000000..f3dcaade2a\n--- /dev/null\n+++ b/reftable/refname_test.c\n@@ -0,0 +1,100 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable.h\"\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"refname.h\"\n+#include \"system.h\"\n+\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct testcase {\n+\tchar *add;\n+\tchar *del;\n+\tint error_code;\n+};\n+\n+static void test_conflict(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"a/b\",\n+\t\t.target = \"destination\", /* make sure it's not a symref. */\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tint i;\n+\tstruct reftable_block_source source = { 0 };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct testcase cases[] = {\n+\t\t{ \"a/b/c\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"b\", NULL, 0 },\n+\t\t{ \"a\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"a\", \"a/b\", 0 },\n+\n+\t\t{ \"p/\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p//q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/./q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/../q\", NULL, REFTABLE_REFNAME_ERROR },\n+\n+\t\t{ \"a/b/c\", \"a/b\", 0 },\n+\t\t{ NULL, \"a//b\", 0 },\n+\t};\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tassert_err(err);\n+\n+\terr = reftable_writer_close(w);\n+\tassert_err(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tassert_err(err);\n+\n+\treftable_table_from_reader(&tab, rd);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct modification mod = {\n+\t\t\t.tab = tab,\n+\t\t};\n+\n+\t\tif (cases[i].add != NULL) {\n+\t\t\tmod.add = &cases[i].add;\n+\t\t\tmod.add_len = 1;\n+\t\t}\n+\t\tif (cases[i].del != NULL) {\n+\t\t\tmod.del = &cases[i].del;\n+\t\t\tmod.del_len = 1;\n+\t\t}\n+\n+\t\terr = modification_validate(&mod);\n+\t\tassert(err == cases[i].error_code);\n+\t}\n+\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+int refname_test_main(int argc, const char *argv[])\n+{\n+\tadd_test_case(\"test_conflict\", &test_conflict);\n+\treturn test_main(argc, argv);\n+}\ndiff --git a/reftable/stack.c b/reftable/stack.c\nnew file mode 100644\nindex 0000000000..cf8a8e854b\n--- /dev/null\n+++ b/reftable/stack.c\n@@ -0,0 +1,1224 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+#include \"merged.h\"\n+#include \"reader.h\"\n+#include \"refname.h\"\n+#include \"reftable.h\"\n+#include \"writer.h\"\n+\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config)\n+{\n+\tstruct reftable_stack *p =\n+\t\treftable_calloc(sizeof(struct reftable_stack));\n+\tstruct strbuf list_file_name = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tif (config.hash_id == 0) {\n+\t\tconfig.hash_id = SHA1_ID;\n+\t}\n+\n+\t*dest = NULL;\n+\n+\tstrbuf_reset(&list_file_name);\n+\tstrbuf_addstr(&list_file_name, dir);\n+\tstrbuf_addstr(&list_file_name, \"/tables.list\");\n+\n+\tp->list_file = strbuf_detach(&list_file_name, NULL);\n+\tp->reftable_dir = xstrdup(dir);\n+\tp->config = config;\n+\n+\terr = reftable_stack_reload_maybe_reuse(p, 1);\n+\tif (err < 0) {\n+\t\treftable_stack_destroy(p);\n+\t} else {\n+\t\t*dest = p;\n+\t}\n+\treturn err;\n+}\n+\n+static int fd_read_lines(int fd, char ***namesp)\n+{\n+\toff_t size = lseek(fd, 0, SEEK_END);\n+\tchar *buf = NULL;\n+\tint err = 0;\n+\tif (size < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\terr = lseek(fd, 0, SEEK_SET);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tbuf = reftable_malloc(size + 1);\n+\tif (read(fd, buf, size) != size) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\tbuf[size] = 0;\n+\n+\tparse_names(buf, size, namesp);\n+\n+done:\n+\treftable_free(buf);\n+\treturn err;\n+}\n+\n+int read_lines(const char *filename, char ***namesp)\n+{\n+\tint fd = open(filename, O_RDONLY, 0644);\n+\tint err = 0;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\t*namesp = reftable_calloc(sizeof(char *));\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\treturn REFTABLE_IO_ERROR;\n+\t}\n+\terr = fd_read_lines(fd, namesp);\n+\tclose(fd);\n+\treturn err;\n+}\n+\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st)\n+{\n+\treturn st->merged;\n+}\n+\n+/* Close and free the stack */\n+void reftable_stack_destroy(struct reftable_stack *st)\n+{\n+\tif (st->merged != NULL) {\n+\t\treftable_merged_table_free(st->merged);\n+\t\tst->merged = NULL;\n+\t}\n+\n+\tif (st->readers != NULL) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < st->readers_len; i++) {\n+\t\t\treftable_reader_free(st->readers[i]);\n+\t\t}\n+\t\tst->readers_len = 0;\n+\t\tFREE_AND_NULL(st->readers);\n+\t}\n+\tFREE_AND_NULL(st->list_file);\n+\tFREE_AND_NULL(st->reftable_dir);\n+\treftable_free(st);\n+}\n+\n+static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n+\t\t\t\t\t\t int cur_len)\n+{\n+\tstruct reftable_reader **cur =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * cur_len);\n+\tint i = 0;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tcur[i] = st->readers[i];\n+\t}\n+\treturn cur;\n+}\n+\n+static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n+\t\t\t\t int reuse_open)\n+{\n+\tint cur_len = st->merged == NULL ? 0 : st->merged->stack_len;\n+\tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n+\tint err = 0;\n+\tint names_len = names_length(names);\n+\tstruct reftable_reader **new_readers =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * names_len);\n+\tstruct reftable_table *new_tables =\n+\t\treftable_calloc(sizeof(struct reftable_table) * names_len);\n+\tint new_readers_len = 0;\n+\tstruct reftable_merged_table *new_merged = NULL;\n+\tint i;\n+\n+\twhile (*names) {\n+\t\tstruct reftable_reader *rd = NULL;\n+\t\tchar *name = *names++;\n+\n+\t\t/* this is linear; we assume compaction keeps the number of\n+\t\t tables under control so this is not quadratic. */\n+\t\tint j = 0;\n+\t\tfor (j = 0; reuse_open && j < cur_len; j++) {\n+\t\t\tif (cur[j] != NULL && 0 == strcmp(cur[j]->name, name)) {\n+\t\t\t\trd = cur[j];\n+\t\t\t\tcur[j] = NULL;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (rd == NULL) {\n+\t\t\tstruct reftable_block_source src = { 0 };\n+\t\t\tstruct strbuf table_path = STRBUF_INIT;\n+\t\t\tstrbuf_addstr(&table_path, st->reftable_dir);\n+\t\t\tstrbuf_addstr(&table_path, \"/\");\n+\t\t\tstrbuf_addstr(&table_path, name);\n+\n+\t\t\terr = reftable_block_source_from_file(&src,\n+\t\t\t\t\t\t\t table_path.buf);\n+\t\t\tstrbuf_release(&table_path);\n+\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\n+\t\t\terr = reftable_new_reader(&rd, &src, name);\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\n+\t\tnew_readers[new_readers_len] = rd;\n+\t\treftable_table_from_reader(&new_tables[new_readers_len], rd);\n+\t\tnew_readers_len++;\n+\t}\n+\n+\t/* success! */\n+\terr = reftable_new_merged_table(&new_merged, new_tables,\n+\t\t\t\t\tnew_readers_len, st->config.hash_id);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tnew_tables = NULL;\n+\tst->readers_len = new_readers_len;\n+\tif (st->merged != NULL) {\n+\t\tmerged_table_clear(st->merged);\n+\t\treftable_merged_table_free(st->merged);\n+\t}\n+\tif (st->readers != NULL) {\n+\t\treftable_free(st->readers);\n+\t}\n+\tst->readers = new_readers;\n+\tnew_readers = NULL;\n+\tnew_readers_len = 0;\n+\n+\tnew_merged->suppress_deletions = 1;\n+\tst->merged = new_merged;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tif (cur[i] != NULL) {\n+\t\t\treader_close(cur[i]);\n+\t\t\treftable_reader_free(cur[i]);\n+\t\t}\n+\t}\n+\n+done:\n+\tfor (i = 0; i < new_readers_len; i++) {\n+\t\treader_close(new_readers[i]);\n+\t\treftable_reader_free(new_readers[i]);\n+\t}\n+\treftable_free(new_readers);\n+\treftable_free(new_tables);\n+\treftable_free(cur);\n+\treturn err;\n+}\n+\n+/* return negative if a before b. */\n+static int tv_cmp(struct timeval *a, struct timeval *b)\n+{\n+\ttime_t diff = a->tv_sec - b->tv_sec;\n+\tint udiff = a->tv_usec - b->tv_usec;\n+\n+\tif (diff != 0)\n+\t\treturn diff;\n+\n+\treturn udiff;\n+}\n+\n+int reftable_stack_reload_maybe_reuse(struct reftable_stack *st, int reuse_open)\n+{\n+\tstruct timeval deadline = { 0 };\n+\tint err = gettimeofday(&deadline, NULL);\n+\tint64_t delay = 0;\n+\tint tries = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tdeadline.tv_sec += 3;\n+\twhile (1) {\n+\t\tchar **names = NULL;\n+\t\tchar **names_after = NULL;\n+\t\tstruct timeval now = { 0 };\n+\t\tint err = gettimeofday(&now, NULL);\n+\t\tint err2 = 0;\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* Only look at deadlines after the first few times. This\n+\t\t simplifies debugging in GDB */\n+\t\ttries++;\n+\t\tif (tries > 3 && tv_cmp(&now, &deadline) >= 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = read_lines(st->list_file, &names);\n+\t\tif (err < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\t\terr = reftable_stack_reload_once(st, names, reuse_open);\n+\t\tif (err == 0) {\n+\t\t\tfree_names(names);\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err != REFTABLE_NOT_EXIST_ERROR) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* err == REFTABLE_NOT_EXIST_ERROR can be caused by a concurrent\n+\t\t writer. Check if there was one by checking if the name list\n+\t\t changed.\n+\t\t*/\n+\t\terr2 = read_lines(st->list_file, &names_after);\n+\t\tif (err2 < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err2;\n+\t\t}\n+\n+\t\tif (names_equal(names_after, names)) {\n+\t\t\tfree_names(names);\n+\t\t\tfree_names(names_after);\n+\t\t\treturn err;\n+\t\t}\n+\t\tfree_names(names);\n+\t\tfree_names(names_after);\n+\n+\t\tdelay = delay + (delay * rand()) / RAND_MAX + 1;\n+\t\tsleep_millisec(delay);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/* -1 = error\n+ 0 = up to date\n+ 1 = changed. */\n+static int stack_uptodate(struct reftable_stack *st)\n+{\n+\tchar **names = NULL;\n+\tint err = read_lines(st->list_file, &names);\n+\tint i = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tfor (i = 0; i < st->readers_len; i++) {\n+\t\tif (names[i] == NULL) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (strcmp(st->readers[i]->name, names[i])) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (names[st->merged->stack_len] != NULL) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tfree_names(names);\n+\treturn err;\n+}\n+\n+int reftable_stack_reload(struct reftable_stack *st)\n+{\n+\tint err = stack_uptodate(st);\n+\tif (err > 0)\n+\t\treturn reftable_stack_reload_maybe_reuse(st, 1);\n+\treturn err;\n+}\n+\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write)(struct reftable_writer *wr, void *arg),\n+\t\t void *arg)\n+{\n+\tint err = stack_try_add(st, write, arg);\n+\tif (err < 0) {\n+\t\tif (err == REFTABLE_LOCK_ERROR) {\n+\t\t\t/* Ignore error return, we want to propagate\n+\t\t\t REFTABLE_LOCK_ERROR.\n+\t\t\t*/\n+\t\t\treftable_stack_reload(st);\n+\t\t}\n+\t\treturn err;\n+\t}\n+\n+\tif (!st->disable_auto_compact)\n+\t\treturn reftable_stack_auto_compact(st);\n+\n+\treturn 0;\n+}\n+\n+static void format_name(struct strbuf *dest, uint64_t min, uint64_t max)\n+{\n+\tchar buf[100];\n+\tsnprintf(buf, sizeof(buf), \"0x%012\" PRIx64 \"-0x%012\" PRIx64, min, max);\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, buf);\n+}\n+\n+struct reftable_addition {\n+\tint lock_file_fd;\n+\tstruct strbuf lock_file_name;\n+\tstruct reftable_stack *stack;\n+\tchar **names;\n+\tchar **new_tables;\n+\tint new_tables_len;\n+\tuint64_t next_update_index;\n+};\n+\n+#define REFTABLE_ADDITION_INIT \\\n+\t{ \\\n+\t\t.lock_file_name = STRBUF_INIT \\\n+\t}\n+\n+static int reftable_stack_init_addition(struct reftable_addition *add,\n+\t\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tadd->stack = st;\n+\n+\tstrbuf_reset(&add->lock_file_name);\n+\tstrbuf_addstr(&add->lock_file_name, st->list_file);\n+\tstrbuf_addstr(&add->lock_file_name, \".lock\");\n+\n+\tadd->lock_file_fd = open(add->lock_file_name.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (add->lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = REFTABLE_LOCK_ERROR;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\terr = stack_uptodate(st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 1) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->next_update_index = reftable_stack_next_update_index(st);\n+done:\n+\tif (err) {\n+\t\treftable_addition_close(add);\n+\t}\n+\treturn err;\n+}\n+\n+void reftable_addition_close(struct reftable_addition *add)\n+{\n+\tint i = 0;\n+\tstruct strbuf nm = STRBUF_INIT;\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstrbuf_reset(&nm);\n+\t\tstrbuf_addstr(&nm, add->stack->list_file);\n+\t\tstrbuf_addstr(&nm, \"/\");\n+\t\tstrbuf_addstr(&nm, add->new_tables[i]);\n+\t\tunlink(nm.buf);\n+\t\treftable_free(add->new_tables[i]);\n+\t\tadd->new_tables[i] = NULL;\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\tif (add->lock_file_fd > 0) {\n+\t\tclose(add->lock_file_fd);\n+\t\tadd->lock_file_fd = 0;\n+\t}\n+\tif (add->lock_file_name.len > 0) {\n+\t\tunlink(add->lock_file_name.buf);\n+\t\tstrbuf_release(&add->lock_file_name);\n+\t}\n+\n+\tfree_names(add->names);\n+\tadd->names = NULL;\n+\tstrbuf_release(&nm);\n+}\n+\n+void reftable_addition_destroy(struct reftable_addition *add)\n+{\n+\tif (add == NULL) {\n+\t\treturn;\n+\t}\n+\treftable_addition_close(add);\n+\treftable_free(add);\n+}\n+\n+int reftable_addition_commit(struct reftable_addition *add)\n+{\n+\tstruct strbuf table_list = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\tif (add->new_tables_len == 0)\n+\t\tgoto done;\n+\n+\tfor (i = 0; i < add->stack->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->stack->readers[i]->name);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->new_tables[i]);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\n+\terr = write(add->lock_file_fd, table_list.buf, table_list.len);\n+\tstrbuf_release(&table_list);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = close(add->lock_file_fd);\n+\tadd->lock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(add->lock_file_name.buf, add->stack->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(add->stack);\n+\n+done:\n+\treftable_addition_close(add);\n+\treturn err;\n+}\n+\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n+\t*dest = reftable_calloc(sizeof(**dest));\n+\t**dest = empty;\n+\terr = reftable_stack_init_addition(*dest, st);\n+\tif (err) {\n+\t\treftable_free(*dest);\n+\t\t*dest = NULL;\n+\t}\n+\treturn err;\n+}\n+\n+int stack_try_add(struct reftable_stack *st,\n+\t\t int (*write_table)(struct reftable_writer *wr, void *arg),\n+\t\t void *arg)\n+{\n+\tstruct reftable_addition add = REFTABLE_ADDITION_INIT;\n+\tint err = reftable_stack_init_addition(&add, st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tif (err > 0) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_addition_add(&add, write_table, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_addition_commit(&add);\n+done:\n+\treftable_addition_close(&add);\n+\treturn err;\n+}\n+\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf tab_file_name = STRBUF_INIT;\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\tint tab_fd = 0;\n+\n+\tstrbuf_reset(&next_name);\n+\tformat_name(&next_name, add->next_update_index, add->next_update_index);\n+\n+\tstrbuf_addstr(&temp_tab_file_name, add->stack->reftable_dir);\n+\tstrbuf_addstr(&temp_tab_file_name, \"/\");\n+\tstrbuf_addbuf(&temp_tab_file_name, &next_name);\n+\tstrbuf_addstr(&temp_tab_file_name, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab_file_name.buf);\n+\tif (tab_fd < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd,\n+\t\t\t\t &add->stack->config);\n+\terr = write_table(wr, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_writer_close(wr);\n+\tif (err == REFTABLE_EMPTY_TABLE_ERROR) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = stack_check_addition(add->stack, temp_tab_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (wr->min_update_index < add->next_update_index) {\n+\t\terr = REFTABLE_API_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tformat_name(&next_name, wr->min_update_index, wr->max_update_index);\n+\tstrbuf_addstr(&next_name, \".ref\");\n+\n+\tstrbuf_addstr(&tab_file_name, add->stack->reftable_dir);\n+\tstrbuf_addstr(&tab_file_name, \"/\");\n+\tstrbuf_addbuf(&tab_file_name, &next_name);\n+\n+\t/* TODO: should check destination out of paranoia */\n+\terr = rename(temp_tab_file_name.buf, tab_file_name.buf);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->new_tables = reftable_realloc(add->new_tables,\n+\t\t\t\t\t sizeof(*add->new_tables) *\n+\t\t\t\t\t\t (add->new_tables_len + 1));\n+\tadd->new_tables[add->new_tables_len] = strbuf_detach(&next_name, NULL);\n+\tadd->new_tables_len++;\n+done:\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (temp_tab_file_name.len > 0) {\n+\t\tunlink(temp_tab_file_name.buf);\n+\t}\n+\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&tab_file_name);\n+\tstrbuf_release(&next_name);\n+\treftable_writer_free(wr);\n+\treturn err;\n+}\n+\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st)\n+{\n+\tint sz = st->merged->stack_len;\n+\tif (sz > 0)\n+\t\treturn reftable_reader_max_update_index(st->readers[sz - 1]) +\n+\t\t 1;\n+\treturn 1;\n+}\n+\n+static int stack_compact_locked(struct reftable_stack *st, int first, int last,\n+\t\t\t\tstruct strbuf *temp_tab,\n+\t\t\t\tstruct reftable_log_expiry_config *config)\n+{\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tint tab_fd = -1;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\n+\tformat_name(&next_name,\n+\t\t reftable_reader_min_update_index(st->readers[first]),\n+\t\t reftable_reader_max_update_index(st->readers[first]));\n+\n+\tstrbuf_reset(temp_tab);\n+\tstrbuf_addstr(temp_tab, st->reftable_dir);\n+\tstrbuf_addstr(temp_tab, \"/\");\n+\tstrbuf_addbuf(temp_tab, &next_name);\n+\tstrbuf_addstr(temp_tab, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab->buf);\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd, &st->config);\n+\n+\terr = stack_write_compact(st, wr, first, last, config);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = reftable_writer_close(wr);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\n+done:\n+\treftable_writer_free(wr);\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (err != 0 && temp_tab->len > 0) {\n+\t\tunlink(temp_tab->buf);\n+\t\tstrbuf_release(temp_tab);\n+\t}\n+\tstrbuf_release(&next_name);\n+\treturn err;\n+}\n+\n+int stack_write_compact(struct reftable_stack *st, struct reftable_writer *wr,\n+\t\t\tint first, int last,\n+\t\t\tstruct reftable_log_expiry_config *config)\n+{\n+\tint subtabs_len = last - first + 1;\n+\tstruct reftable_table *subtabs = reftable_calloc(\n+\t\tsizeof(struct reftable_table) * (last - first + 1));\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_ref_record ref = { 0 };\n+\tstruct reftable_log_record log = { 0 };\n+\n+\tuint64_t entries = 0;\n+\n+\tint i = 0, j = 0;\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct reftable_reader *t = st->readers[i];\n+\t\treftable_table_from_reader(&subtabs[j++], t);\n+\t\tst->stats.bytes += t->size;\n+\t}\n+\treftable_writer_set_limits(wr, st->readers[first]->min_update_index,\n+\t\t\t\t st->readers[last]->max_update_index);\n+\n+\terr = reftable_new_merged_table(&mt, subtabs, subtabs_len,\n+\t\t\t\t\tst->config.hash_id);\n+\tif (err < 0) {\n+\t\treftable_free(subtabs);\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_merged_table_seek_ref(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (first == 0 && reftable_ref_record_is_deletion(&ref)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_ref(wr, &ref);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_merged_table_seek_log(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (first == 0 && reftable_log_record_is_deletion(&log)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config != NULL && config->time > 0 &&\n+\t\t log.time < config->time) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config != NULL && config->min_update_index > 0 &&\n+\t\t log.update_index < config->min_update_index) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_log(wr, &log);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\tif (mt != NULL) {\n+\t\tmerged_table_clear(mt);\n+\t\treftable_merged_table_free(mt);\n+\t}\n+\treftable_ref_record_clear(&ref);\n+\treftable_log_record_clear(&log);\n+\tst->stats.entries_written += entries;\n+\treturn err;\n+}\n+\n+/* < 0: error. 0 == OK, > 0 attempt failed; could retry. */\n+static int stack_compact_range(struct reftable_stack *st, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *expiry)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf new_table_name = STRBUF_INIT;\n+\tstruct strbuf lock_file_name = STRBUF_INIT;\n+\tstruct strbuf ref_list_contents = STRBUF_INIT;\n+\tstruct strbuf new_table_path = STRBUF_INIT;\n+\tint err = 0;\n+\tint have_lock = 0;\n+\tint lock_file_fd = 0;\n+\tint compact_count = last - first + 1;\n+\tchar **listp = NULL;\n+\tchar **delete_on_success =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tchar **subtable_locks =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tint i = 0;\n+\tint j = 0;\n+\tint is_empty_table = 0;\n+\n+\tif (first > last || (expiry == NULL && first == last)) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\tst->stats.attempts++;\n+\n+\tstrbuf_reset(&lock_file_name);\n+\tstrbuf_addstr(&lock_file_name, st->list_file);\n+\tstrbuf_addstr(&lock_file_name, \".lock\");\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\t/* Don't want to write to the lock for now. */\n+\tclose(lock_file_fd);\n+\tlock_file_fd = 0;\n+\n+\thave_lock = 1;\n+\terr = stack_uptodate(st);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct strbuf subtab_file_name = STRBUF_INIT;\n+\t\tstruct strbuf subtab_lock = STRBUF_INIT;\n+\t\tint sublock_file_fd = -1;\n+\n+\t\tstrbuf_addstr(&subtab_file_name, st->reftable_dir);\n+\t\tstrbuf_addstr(&subtab_file_name, \"/\");\n+\t\tstrbuf_addstr(&subtab_file_name, reader_name(st->readers[i]));\n+\n+\t\tstrbuf_reset(&subtab_lock);\n+\t\tstrbuf_addbuf(&subtab_lock, &subtab_file_name);\n+\t\tstrbuf_addstr(&subtab_lock, \".lock\");\n+\n+\t\tsublock_file_fd = open(subtab_lock.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\t\tif (sublock_file_fd > 0) {\n+\t\t\tclose(sublock_file_fd);\n+\t\t} else if (sublock_file_fd < 0) {\n+\t\t\tif (errno == EEXIST) {\n+\t\t\t\terr = 1;\n+\t\t\t} else {\n+\t\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\t}\n+\t\t}\n+\n+\t\tsubtable_locks[j] = subtab_lock.buf;\n+\t\tdelete_on_success[j] = subtab_file_name.buf;\n+\t\tj++;\n+\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\terr = unlink(lock_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\thave_lock = 0;\n+\n+\terr = stack_compact_locked(st, first, last, &temp_tab_file_name,\n+\t\t\t\t expiry);\n+\t/* Compaction + tombstones can create an empty table out of non-empty\n+\t * tables. */\n+\tis_empty_table = (err == REFTABLE_EMPTY_TABLE_ERROR);\n+\tif (is_empty_table) {\n+\t\terr = 0;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\thave_lock = 1;\n+\n+\tformat_name(&new_table_name, st->readers[first]->min_update_index,\n+\t\t st->readers[last]->max_update_index);\n+\tstrbuf_addstr(&new_table_name, \".ref\");\n+\n+\tstrbuf_reset(&new_table_path);\n+\tstrbuf_addstr(&new_table_path, st->reftable_dir);\n+\tstrbuf_addstr(&new_table_path, \"/\");\n+\tstrbuf_addbuf(&new_table_path, &new_table_name);\n+\n+\tif (!is_empty_table) {\n+\t\terr = rename(temp_tab_file_name.buf, new_table_path.buf);\n+\t\tif (err < 0) {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < first; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tif (!is_empty_table) {\n+\t\tstrbuf_addbuf(&ref_list_contents, &new_table_name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tfor (i = last + 1; i < st->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\n+\terr = write(lock_file_fd, ref_list_contents.buf, ref_list_contents.len);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\terr = close(lock_file_fd);\n+\tlock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(lock_file_name.buf, st->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\thave_lock = 0;\n+\n+\t/* Reload the stack before deleting. On windows, we can only delete the\n+\t files after we closed them.\n+\t*/\n+\terr = reftable_stack_reload_maybe_reuse(st, first < last);\n+\n+\tlistp = delete_on_success;\n+\twhile (*listp) {\n+\t\tif (strcmp(*listp, new_table_path.buf)) {\n+\t\t\tunlink(*listp);\n+\t\t}\n+\t\tlistp++;\n+\t}\n+\n+done:\n+\tfree_names(delete_on_success);\n+\n+\tlistp = subtable_locks;\n+\twhile (*listp) {\n+\t\tunlink(*listp);\n+\t\tlistp++;\n+\t}\n+\tfree_names(subtable_locks);\n+\tif (lock_file_fd > 0) {\n+\t\tclose(lock_file_fd);\n+\t\tlock_file_fd = 0;\n+\t}\n+\tif (have_lock) {\n+\t\tunlink(lock_file_name.buf);\n+\t}\n+\tstrbuf_release(&new_table_name);\n+\tstrbuf_release(&new_table_path);\n+\tstrbuf_release(&ref_list_contents);\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&lock_file_name);\n+\treturn err;\n+}\n+\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\treturn stack_compact_range(st, 0, st->merged->stack_len - 1, config);\n+}\n+\n+static int stack_compact_range_stats(struct reftable_stack *st, int first,\n+\t\t\t\t int last,\n+\t\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint err = stack_compact_range(st, first, last, config);\n+\tif (err > 0) {\n+\t\tst->stats.failures++;\n+\t}\n+\treturn err;\n+}\n+\n+static int segment_size(struct segment *s)\n+{\n+\treturn s->end - s->start;\n+}\n+\n+int fastlog2(uint64_t sz)\n+{\n+\tint l = 0;\n+\tif (sz == 0)\n+\t\treturn 0;\n+\tfor (; sz; sz /= 2) {\n+\t\tl++;\n+\t}\n+\treturn l - 1;\n+}\n+\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n+{\n+\tstruct segment *segs = reftable_calloc(sizeof(struct segment) * n);\n+\tint next = 0;\n+\tstruct segment cur = { 0 };\n+\tint i = 0;\n+\n+\tif (n == 0) {\n+\t\t*seglen = 0;\n+\t\treturn segs;\n+\t}\n+\tfor (i = 0; i < n; i++) {\n+\t\tint log = fastlog2(sizes[i]);\n+\t\tif (cur.log != log && cur.bytes > 0) {\n+\t\t\tstruct segment fresh = {\n+\t\t\t\t.start = i,\n+\t\t\t};\n+\n+\t\t\tsegs[next++] = cur;\n+\t\t\tcur = fresh;\n+\t\t}\n+\n+\t\tcur.log = log;\n+\t\tcur.end = i + 1;\n+\t\tcur.bytes += sizes[i];\n+\t}\n+\tsegs[next++] = cur;\n+\t*seglen = next;\n+\treturn segs;\n+}\n+\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, sizes, n);\n+\tstruct segment min_seg = {\n+\t\t.log = 64,\n+\t};\n+\tint i = 0;\n+\tfor (i = 0; i < seglen; i++) {\n+\t\tif (segment_size(&segs[i]) == 1) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (segs[i].log < min_seg.log) {\n+\t\t\tmin_seg = segs[i];\n+\t\t}\n+\t}\n+\n+\twhile (min_seg.start > 0) {\n+\t\tint prev = min_seg.start - 1;\n+\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmin_seg.start = prev;\n+\t\tmin_seg.bytes += sizes[prev];\n+\t}\n+\n+\treftable_free(segs);\n+\treturn min_seg;\n+}\n+\n+static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes =\n+\t\treftable_calloc(sizeof(uint64_t) * st->merged->stack_len);\n+\tint version = (st->config.hash_id == SHA1_ID) ? 1 : 2;\n+\tint overhead = header_size(version) - 1;\n+\tint i = 0;\n+\tfor (i = 0; i < st->merged->stack_len; i++) {\n+\t\tsizes[i] = st->readers[i]->size - overhead;\n+\t}\n+\treturn sizes;\n+}\n+\n+int reftable_stack_auto_compact(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n+\tstruct segment seg =\n+\t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n+\treftable_free(sizes);\n+\tif (segment_size(&seg) > 0)\n+\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n+\t\t\t\t\t\t NULL);\n+\n+\treturn 0;\n+}\n+\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st)\n+{\n+\treturn &st->stats;\n+}\n+\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_table tab = { NULL };\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\treturn reftable_table_read_ref(&tab, refname, ref);\n+}\n+\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(st);\n+\tint err = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_log(&it, log);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(log->refname, refname) ||\n+\t reftable_log_record_is_deletion(log)) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tif (err) {\n+\t\treftable_log_record_clear(log);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+int stack_check_addition(struct reftable_stack *st, const char *new_tab_name)\n+{\n+\tint err = 0;\n+\tstruct reftable_block_source src = { 0 };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct reftable_ref_record *refs = NULL;\n+\tstruct reftable_iterator it = { NULL };\n+\tint cap = 0;\n+\tint len = 0;\n+\tint i = 0;\n+\n+\tif (st->config.skip_name_check)\n+\t\treturn 0;\n+\n+\terr = reftable_block_source_from_file(&src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&rd, &src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tif (err > 0) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { 0 };\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (len >= cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\trefs = reftable_realloc(refs, cap * sizeof(refs[0]));\n+\t\t}\n+\n+\t\trefs[len++] = ref;\n+\t}\n+\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\n+\terr = validate_ref_record_addition(tab, refs, len);\n+\n+done:\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_clear(&refs[i]);\n+\t}\n+\n+\tfree(refs);\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\treturn err;\n+}\ndiff --git a/reftable/stack.h b/reftable/stack.h\nnew file mode 100644\nindex 0000000000..5ef138b025\n--- /dev/null\n+++ b/reftable/stack.h\n@@ -0,0 +1,50 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef STACK_H\n+#define STACK_H\n+\n+#include \"reftable.h\"\n+#include \"system.h\"\n+\n+struct reftable_stack {\n+\tchar *list_file;\n+\tchar *reftable_dir;\n+\tint disable_auto_compact;\n+\n+\tstruct reftable_write_options config;\n+\n+\tstruct reftable_reader **readers;\n+\tsize_t readers_len;\n+\tstruct reftable_merged_table *merged;\n+\tstruct reftable_compaction_stats stats;\n+};\n+\n+int read_lines(const char *filename, char ***lines);\n+int stack_try_add(struct reftable_stack *st,\n+\t\t int (*write_table)(struct reftable_writer *wr, void *arg),\n+\t\t void *arg);\n+int stack_write_compact(struct reftable_stack *st, struct reftable_writer *wr,\n+\t\t\tint first, int last,\n+\t\t\tstruct reftable_log_expiry_config *config);\n+int fastlog2(uint64_t sz);\n+int stack_check_addition(struct reftable_stack *st, const char *new_tab_name);\n+void reftable_addition_close(struct reftable_addition *add);\n+int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t int reuse_open);\n+\n+struct segment {\n+\tint start, end;\n+\tint log;\n+\tuint64_t bytes;\n+};\n+\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n);\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n);\n+\n+#endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nnew file mode 100644\nindex 0000000000..de806441dc\n--- /dev/null\n+++ b/reftable/stack_test.c\n@@ -0,0 +1,787 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+\n+#include \"merged.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+#include \n+#include \n+\n+static void test_read_file(void)\n+{\n+\tchar fn[256] = \"/tmp/stack.test_read_file.XXXXXX\";\n+\tint fd = mkstemp(fn);\n+\tchar out[1024] = \"line1\\n\\nline2\\nline3\";\n+\tint n, err;\n+\tchar **names = NULL;\n+\tchar *want[] = { \"line1\", \"line2\", \"line3\" };\n+\tint i = 0;\n+\n+\tassert(fd > 0);\n+\tn = write(fd, out, strlen(out));\n+\tassert(n == strlen(out));\n+\terr = close(fd);\n+\tassert(err >= 0);\n+\n+\terr = read_lines(fn, &names);\n+\tassert_err(err);\n+\n+\tfor (i = 0; names[i] != NULL; i++) {\n+\t\tassert(0 == strcmp(want[i], names[i]));\n+\t}\n+\tfree_names(names);\n+\tremove(fn);\n+}\n+\n+static void test_parse_names(void)\n+{\n+\tchar buf[] = \"line\\n\";\n+\tchar **names = NULL;\n+\tparse_names(buf, strlen(buf), &names);\n+\n+\tassert(NULL != names[0]);\n+\tassert(0 == strcmp(names[0], \"line\"));\n+\tassert(NULL == names[1]);\n+\tfree_names(names);\n+}\n+\n+static void test_names_equal(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", \"c\", NULL };\n+\tchar *b[] = { \"a\", \"b\", \"d\", NULL };\n+\tchar *c[] = { \"a\", \"b\", NULL };\n+\n+\tassert(names_equal(a, a));\n+\tassert(!names_equal(a, b));\n+\tassert(!names_equal(a, c));\n+}\n+\n+static int write_test_ref(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct reftable_ref_record *ref = arg;\n+\treftable_writer_set_limits(wr, ref->update_index, ref->update_index);\n+\treturn reftable_writer_add_ref(wr, ref);\n+}\n+\n+struct write_log_arg {\n+\tstruct reftable_log_record *log;\n+\tuint64_t update_index;\n+};\n+\n+static int write_test_log(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct write_log_arg *wla = arg;\n+\n+\treftable_writer_set_limits(wr, wla->update_index, wla->update_index);\n+\treturn reftable_writer_add_log(wr, wla->log);\n+}\n+\n+static void test_reftable_stack_add_one(void)\n+{\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.target = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { 0 };\n+\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tassert_err(err);\n+\tassert(0 == strcmp(\"master\", dest.target));\n+\n+\treftable_ref_record_clear(&dest);\n+\treftable_stack_destroy(st);\n+\treftable_clear_dir(dir);\n+}\n+\n+static void test_reftable_stack_uptodate(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL;\n+\tstruct reftable_stack *st2 = NULL;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.target = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"branch2\",\n+\t\t.update_index = 2,\n+\t\t.target = \"master\",\n+\t};\n+\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tassert_err(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_add(st1, &write_test_ref, &ref1);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tassert(err == REFTABLE_LOCK_ERROR);\n+\n+\terr = reftable_stack_reload(st2);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tassert_err(err);\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\treftable_clear_dir(dir);\n+}\n+\n+static void test_reftable_stack_transaction_api(void)\n+{\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_addition *add = NULL;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.target = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { 0 };\n+\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_new_addition(&add, st);\n+\tassert_err(err);\n+\n+\terr = reftable_addition_add(add, &write_test_ref, &ref);\n+\tassert_err(err);\n+\n+\terr = reftable_addition_commit(add);\n+\tassert_err(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tassert_err(err);\n+\tassert(0 == strcmp(\"master\", dest.target));\n+\n+\treftable_ref_record_clear(&dest);\n+\treftable_stack_destroy(st);\n+\treftable_clear_dir(dir);\n+}\n+\n+static void test_reftable_stack_validate_refname(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tint i;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"a/b\",\n+\t\t.update_index = 1,\n+\t\t.target = \"master\",\n+\t};\n+\tchar *additions[] = { \"a\", \"a/b/c\" };\n+\n+\tassert(mkdtemp(dir));\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tassert_err(err);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(additions); i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = additions[i],\n+\t\t\t.update_index = 1,\n+\t\t\t.target = \"master\",\n+\t\t};\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tassert(err == REFTABLE_NAME_CONFLICT);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\treftable_clear_dir(dir);\n+}\n+\n+static int write_error(struct reftable_writer *wr, void *arg)\n+{\n+\treturn *((int *)arg);\n+}\n+\n+static void test_reftable_stack_update_index_check(void)\n+{\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"name1\",\n+\t\t.update_index = 1,\n+\t\t.target = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"name2\",\n+\t\t.update_index = 1,\n+\t\t.target = \"master\",\n+\t};\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref1);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref2);\n+\tassert(err == REFTABLE_API_ERROR);\n+\treftable_stack_destroy(st);\n+\treftable_clear_dir(dir);\n+}\n+\n+static void test_reftable_stack_lock_failure(void)\n+{\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err, i;\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\tfor (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) {\n+\t\terr = reftable_stack_add(st, &write_error, &i);\n+\t\tassert(err == i);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\treftable_clear_dir(dir);\n+}\n+\n+static void test_reftable_stack_add(void)\n+{\n+\tint i = 0;\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t.exact_log_message = 1,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_ref_record refs[2] = { { 0 } };\n+\tstruct reftable_log_record logs[2] = { { 0 } };\n+\tint N = ARRAY_SIZE(refs);\n+\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\tst->disable_auto_compact = 1;\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].value = reftable_malloc(SHA1_SIZE);\n+\t\trefs[i].update_index = i + 1;\n+\t\tset_test_hash(refs[i].value, i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = N + i + 1;\n+\t\tlogs[i].new_hash = reftable_malloc(SHA1_SIZE);\n+\t\tlogs[i].email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].new_hash, i);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tassert_err(err);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tassert_err(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tassert_err(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_ref_record dest = { 0 };\n+\t\tint err = reftable_stack_read_ref(st, refs[i].refname, &dest);\n+\t\tassert_err(err);\n+\t\tassert(reftable_ref_record_equal(&dest, refs + i, SHA1_SIZE));\n+\t\treftable_ref_record_clear(&dest);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_log_record dest = { 0 };\n+\t\tint err = reftable_stack_read_log(st, refs[i].refname, &dest);\n+\t\tassert_err(err);\n+\t\tassert(reftable_log_record_equal(&dest, logs + i, SHA1_SIZE));\n+\t\treftable_log_record_clear(&dest);\n+\t}\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_clear(&refs[i]);\n+\t\treftable_log_record_clear(&logs[i]);\n+\t}\n+\treftable_clear_dir(dir);\n+}\n+\n+static void test_reftable_stack_log_normalize(void)\n+{\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t0,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\n+\tuint8_t h1[SHA1_SIZE] = { 0x01 }, h2[SHA1_SIZE] = { 0x02 };\n+\n+\tstruct reftable_log_record input = {\n+\t\t.refname = \"branch\",\n+\t\t.update_index = 1,\n+\t\t.new_hash = h1,\n+\t\t.old_hash = h2,\n+\t};\n+\tstruct reftable_log_record dest = {\n+\t\t.update_index = 0,\n+\t};\n+\tstruct write_log_arg arg = {\n+\t\t.log = &input,\n+\t\t.update_index = 1,\n+\t};\n+\n+\tassert(mkdtemp(dir));\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\tinput.message = \"one\\ntwo\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tassert(err == REFTABLE_API_ERROR);\n+\n+\tinput.message = \"one\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tassert_err(err);\n+\tassert(0 == strcmp(dest.message, \"one\\n\"));\n+\n+\tinput.message = \"two\\n\";\n+\targ.update_index = 2;\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tassert_err(err);\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tassert_err(err);\n+\tassert(0 == strcmp(dest.message, \"two\\n\"));\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\treftable_log_record_clear(&dest);\n+\treftable_clear_dir(dir);\n+}\n+\n+static void test_reftable_stack_tombstone(void)\n+{\n+\tint i = 0;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record refs[2] = { { 0 } };\n+\tstruct reftable_log_record logs[2] = { { 0 } };\n+\tint N = ARRAY_SIZE(refs);\n+\tstruct reftable_ref_record dest = { 0 };\n+\tstruct reftable_log_record log_dest = { 0 };\n+\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tconst char *buf = \"branch\";\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\tif (i % 2 == 0) {\n+\t\t\trefs[i].value = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_test_hash(refs[i].value, i);\n+\t\t}\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\t/* update_index is part of the key. */\n+\t\tlogs[i].update_index = 42;\n+\t\tif (i % 2 == 0) {\n+\t\t\tlogs[i].new_hash = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_test_hash(logs[i].new_hash, i);\n+\t\t\tlogs[i].email = xstrdup(\"identity@invalid\");\n+\t\t}\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tassert_err(err);\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tassert_err(err);\n+\t}\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tassert(err == 1);\n+\treftable_ref_record_clear(&dest);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tassert(err == 1);\n+\treftable_log_record_clear(&log_dest);\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tassert(err == 1);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tassert(err == 1);\n+\treftable_ref_record_clear(&dest);\n+\treftable_log_record_clear(&log_dest);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_clear(&refs[i]);\n+\t\treftable_log_record_clear(&logs[i]);\n+\t}\n+\treftable_clear_dir(dir);\n+}\n+\n+static void test_reftable_stack_hash_id(void)\n+{\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"master\",\n+\t\t.target = \"target\",\n+\t\t.update_index = 1,\n+\t};\n+\tstruct reftable_write_options cfg32 = { .hash_id = SHA256_ID };\n+\tstruct reftable_stack *st32 = NULL;\n+\tstruct reftable_write_options cfg_default = { 0 };\n+\tstruct reftable_stack *st_default = NULL;\n+\tstruct reftable_ref_record dest = { 0 };\n+\n+\tassert(mkdtemp(dir));\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tassert_err(err);\n+\n+\t/* can't read it with the wrong hash ID. */\n+\terr = reftable_new_stack(&st32, dir, cfg32);\n+\tassert(err == REFTABLE_FORMAT_ERROR);\n+\n+\t/* check that we can read it back with default config too. */\n+\terr = reftable_new_stack(&st_default, dir, cfg_default);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n+\tassert_err(err);\n+\n+\tassert(!strcmp(dest.target, ref.target));\n+\treftable_ref_record_clear(&dest);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st_default);\n+\treftable_clear_dir(dir);\n+}\n+\n+static void test_log2(void)\n+{\n+\tassert(1 == fastlog2(3));\n+\tassert(2 == fastlog2(4));\n+\tassert(2 == fastlog2(5));\n+}\n+\n+static void test_sizes_to_segments(void)\n+{\n+\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n+\t/* .................0 1 2 3 4 5 */\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tassert(segs[2].log == 3);\n+\tassert(segs[2].start == 5);\n+\tassert(segs[2].end == 6);\n+\n+\tassert(segs[1].log == 2);\n+\tassert(segs[1].start == 2);\n+\tassert(segs[1].end == 5);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_empty(void)\n+{\n+\tuint64_t sizes[0];\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tassert(seglen == 0);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_all_equal(void)\n+{\n+\tuint64_t sizes[] = { 5, 5 };\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tassert(seglen == 1);\n+\tassert(segs[0].start == 0);\n+\tassert(segs[0].end == 2);\n+\treftable_free(segs);\n+}\n+\n+static void test_suggest_compaction_segment(void)\n+{\n+\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n+\t/* .................0 1 2 3 4 5 6 */\n+\tstruct segment min =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tassert(min.start == 2);\n+\tassert(min.end == 7);\n+}\n+\n+static void test_suggest_compaction_segment_nothing(void)\n+{\n+\tuint64_t sizes[] = { 64, 32, 16, 8, 4, 2 };\n+\tstruct segment result =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tassert(result.start == result.end);\n+}\n+\n+static void test_reflog_expire(void)\n+{\n+\tchar dir[256] = \"/tmp/stack.test_reflog_expire.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tstruct reftable_log_record logs[20] = { { 0 } };\n+\tint N = ARRAY_SIZE(logs) - 1;\n+\tint i = 0;\n+\tint err;\n+\tstruct reftable_log_expiry_config expiry = {\n+\t\t.time = 10,\n+\t};\n+\tstruct reftable_log_record log = { 0 };\n+\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = i;\n+\t\tlogs[i].time = i;\n+\t\tlogs[i].new_hash = reftable_malloc(SHA1_SIZE);\n+\t\tlogs[i].email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].new_hash, i);\n+\t}\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tassert_err(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_read_log(st, logs[9].refname, &log);\n+\tassert(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[11].refname, &log);\n+\tassert_err(err);\n+\n+\texpiry.min_update_index = 15;\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_read_log(st, logs[14].refname, &log);\n+\tassert(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[16].refname, &log);\n+\tassert_err(err);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i <= N; i++) {\n+\t\treftable_log_record_clear(&logs[i]);\n+\t}\n+\treftable_clear_dir(dir);\n+\treftable_log_record_clear(&log);\n+}\n+\n+static int write_nothing(struct reftable_writer *wr, void *arg)\n+{\n+\treftable_writer_set_limits(wr, 1, 1);\n+\treturn 0;\n+}\n+\n+static void test_empty_add(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_stack *st2 = NULL;\n+\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_add(st, &write_nothing, NULL);\n+\tassert_err(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tassert_err(err);\n+\treftable_clear_dir(dir);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st2);\n+}\n+\n+static void test_reftable_stack_auto_compaction(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tint err, i;\n+\tint N = 100;\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t\t.target = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tassert_err(err);\n+\n+\t\tassert(i < 3 || st->merged->stack_len < 2 * fastlog2(i));\n+\t}\n+\n+\tassert(reftable_stack_compaction_stats(st)->entries_written <\n+\t (uint64_t)(N * fastlog2(N)));\n+\n+\treftable_stack_destroy(st);\n+\treftable_clear_dir(dir);\n+}\n+\n+int stack_test_main(int argc, const char *argv[])\n+{\n+\tadd_test_case(\"test_reftable_stack_uptodate\",\n+\t\t &test_reftable_stack_uptodate);\n+\tadd_test_case(\"test_reftable_stack_transaction_api\",\n+\t\t &test_reftable_stack_transaction_api);\n+\tadd_test_case(\"test_reftable_stack_hash_id\",\n+\t\t &test_reftable_stack_hash_id);\n+\tadd_test_case(\"test_sizes_to_segments_all_equal\",\n+\t\t &test_sizes_to_segments_all_equal);\n+\tadd_test_case(\"test_reftable_stack_auto_compaction\",\n+\t\t &test_reftable_stack_auto_compaction);\n+\tadd_test_case(\"test_reftable_stack_validate_refname\",\n+\t\t &test_reftable_stack_validate_refname);\n+\tadd_test_case(\"test_reftable_stack_update_index_check\",\n+\t\t &test_reftable_stack_update_index_check);\n+\tadd_test_case(\"test_reftable_stack_lock_failure\",\n+\t\t &test_reftable_stack_lock_failure);\n+\tadd_test_case(\"test_reftable_stack_log_normalize\",\n+\t\t &test_reftable_stack_log_normalize);\n+\tadd_test_case(\"test_reftable_stack_tombstone\",\n+\t\t &test_reftable_stack_tombstone);\n+\tadd_test_case(\"test_reftable_stack_add_one\",\n+\t\t &test_reftable_stack_add_one);\n+\tadd_test_case(\"test_empty_add\", test_empty_add);\n+\tadd_test_case(\"test_reflog_expire\", test_reflog_expire);\n+\tadd_test_case(\"test_suggest_compaction_segment\",\n+\t\t &test_suggest_compaction_segment);\n+\tadd_test_case(\"test_suggest_compaction_segment_nothing\",\n+\t\t &test_suggest_compaction_segment_nothing);\n+\tadd_test_case(\"test_sizes_to_segments\", &test_sizes_to_segments);\n+\tadd_test_case(\"test_sizes_to_segments_empty\",\n+\t\t &test_sizes_to_segments_empty);\n+\tadd_test_case(\"test_log2\", &test_log2);\n+\tadd_test_case(\"test_parse_names\", &test_parse_names);\n+\tadd_test_case(\"test_read_file\", &test_read_file);\n+\tadd_test_case(\"test_names_equal\", &test_names_equal);\n+\tadd_test_case(\"test_reftable_stack_add\", &test_reftable_stack_add);\n+\treturn test_main(argc, argv);\n+}\ndiff --git a/reftable/update.sh b/reftable/update.sh\nnew file mode 100755\nindex 0000000000..4dadd875f4\n--- /dev/null\n+++ b/reftable/update.sh\n@@ -0,0 +1,22 @@\n+#!/bin/sh\n+\n+set -eu\n+\n+# Override this to import from somewhere else, say \"../reftable\".\n+SRC=${SRC:-origin}\n+BRANCH=${BRANCH:-master}\n+\n+((git --git-dir reftable-repo/.git fetch -f ${SRC} ${BRANCH}:import && cd reftable-repo && git checkout -f $(git rev-parse import) ) ||\n+ git clone https://github.com/google/reftable reftable-repo)\n+\n+cp reftable-repo/c/*.[ch] reftable/\n+cp reftable-repo/c/include/*.[ch] reftable/\n+cp reftable-repo/LICENSE reftable/\n+\n+git --git-dir reftable-repo/.git show --no-patch --format=oneline HEAD \\\n+ > reftable/VERSION\n+\n+mv reftable/system.h reftable/system.h~\n+sed 's|if REFTABLE_IN_GITCORE|if 1 /* REFTABLE_IN_GITCORE */|' < reftable/system.h~ > reftable/system.h\n+\n+git add reftable/*.[ch] reftable/LICENSE reftable/VERSION\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 2273fdfe39..def8883439 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,9 +4,12 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tblock_test_main(argc, argv);\n+\tmerged_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\trefname_test_main(argc, argv);\n \treftable_test_main(argc, argv);\n \tstrbuf_test_main(argc, argv);\n+\tstack_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"405660","messageId":"4e40fc3b40087abb0e104f707f290d58a1f55d95.1600283416.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.git.git.1600283416.gitgitgadget@gmail.com","subject":"[PATCH 09/13] reftable: write reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-16T19:10:12Z","receivedAt":"2020-09-16T19:11:28Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/writer.c | 664 ++++++++++++++++++++++++++++++++++++++++++++++\n reftable/writer.h | 60 +++++\n 3 files changed, 725 insertions(+)\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n\ndiff --git a/Makefile b/Makefile\nindex b96c235087..a200633945 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2352,6 +2352,7 @@ REFTABLE_OBJS += reftable/compat.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/strbuf.o\n REFTABLE_OBJS += reftable/tree.o\n+REFTABLE_OBJS += reftable/writer.o\n REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/block_test.o\ndiff --git a/reftable/writer.c b/reftable/writer.c\nnew file mode 100644\nindex 0000000000..44ddcc6757\n--- /dev/null\n+++ b/reftable/writer.c\n@@ -0,0 +1,664 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"writer.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+#include \"tree.h\"\n+\n+static struct reftable_block_stats *\n+writer_reftable_block_stats(struct reftable_writer *w, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase 'r':\n+\t\treturn &w->stats.ref_stats;\n+\tcase 'o':\n+\t\treturn &w->stats.obj_stats;\n+\tcase 'i':\n+\t\treturn &w->stats.idx_stats;\n+\tcase 'g':\n+\t\treturn &w->stats.log_stats;\n+\t}\n+\tabort();\n+\treturn NULL;\n+}\n+\n+/* write data, queuing the padding for the next write. Returns negative for\n+ * error. */\n+static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n+\t\t\tint padding)\n+{\n+\tint n = 0;\n+\tif (w->pending_padding > 0) {\n+\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding);\n+\t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n+\t\tif (n < 0)\n+\t\t\treturn n;\n+\n+\t\tw->pending_padding = 0;\n+\t\treftable_free(zeroed);\n+\t}\n+\n+\tw->pending_padding = padding;\n+\tn = w->write(w->write_arg, data, len);\n+\tif (n < 0)\n+\t\treturn n;\n+\tn += padding;\n+\treturn 0;\n+}\n+\n+static void options_set_defaults(struct reftable_write_options *opts)\n+{\n+\tif (opts->restart_interval == 0) {\n+\t\topts->restart_interval = 16;\n+\t}\n+\n+\tif (opts->hash_id == 0) {\n+\t\topts->hash_id = SHA1_ID;\n+\t}\n+\tif (opts->block_size == 0) {\n+\t\topts->block_size = DEFAULT_BLOCK_SIZE;\n+\t}\n+}\n+\n+static int writer_version(struct reftable_writer *w)\n+{\n+\treturn (w->opts.hash_id == 0 || w->opts.hash_id == SHA1_ID) ? 1 : 2;\n+}\n+\n+static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n+{\n+\tmemcpy((char *)dest, \"REFT\", 4);\n+\n+\tdest[4] = writer_version(w);\n+\n+\tput_be24(dest + 5, w->opts.block_size);\n+\tput_be64(dest + 8, w->min_update_index);\n+\tput_be64(dest + 16, w->max_update_index);\n+\tif (writer_version(w) == 2) {\n+\t\tput_be32(dest + 24, w->opts.hash_id);\n+\t}\n+\treturn header_size(writer_version(w));\n+}\n+\n+static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n+{\n+\tint block_start = 0;\n+\tif (w->next == 0) {\n+\t\tblock_start = header_size(writer_version(w));\n+\t}\n+\n+\tstrbuf_release(&w->last_key);\n+\tblock_writer_init(&w->block_writer_data, typ, w->block,\n+\t\t\t w->opts.block_size, block_start,\n+\t\t\t hash_size(w->opts.hash_id));\n+\tw->block_writer = &w->block_writer_data;\n+\tw->block_writer->restart_interval = w->opts.restart_interval;\n+}\n+\n+struct reftable_writer *\n+reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts)\n+{\n+\tstruct reftable_writer *wp =\n+\t\treftable_calloc(sizeof(struct reftable_writer));\n+\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n+\toptions_set_defaults(opts);\n+\tif (opts->block_size >= (1 << 24)) {\n+\t\t/* TODO - error return? */\n+\t\tabort();\n+\t}\n+\twp->last_key = reftable_empty_strbuf;\n+\twp->block = reftable_calloc(opts->block_size);\n+\twp->write = writer_func;\n+\twp->write_arg = writer_arg;\n+\twp->opts = *opts;\n+\twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n+\n+\treturn wp;\n+}\n+\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max)\n+{\n+\tw->min_update_index = min;\n+\tw->max_update_index = max;\n+}\n+\n+void reftable_writer_free(struct reftable_writer *w)\n+{\n+\treftable_free(w->block);\n+\treftable_free(w);\n+}\n+\n+struct obj_index_tree_node {\n+\tstruct strbuf hash;\n+\tuint64_t *offsets;\n+\tsize_t offset_len;\n+\tsize_t offset_cap;\n+};\n+\n+#define OBJ_INDEX_TREE_NODE_INIT \\\n+\t{ \\\n+\t\t.hash = STRBUF_INIT \\\n+\t}\n+\n+static int obj_index_tree_node_compare(const void *a, const void *b)\n+{\n+\treturn strbuf_cmp(&((const struct obj_index_tree_node *)a)->hash,\n+\t\t\t &((const struct obj_index_tree_node *)b)->hash);\n+}\n+\n+static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n+{\n+\tuint64_t off = w->next;\n+\n+\tstruct obj_index_tree_node want = { .hash = *hash };\n+\n+\tstruct tree_node *node = tree_search(&want, &w->obj_index_tree,\n+\t\t\t\t\t &obj_index_tree_node_compare, 0);\n+\tstruct obj_index_tree_node *key = NULL;\n+\tif (node == NULL) {\n+\t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n+\t\tkey = reftable_malloc(sizeof(struct obj_index_tree_node));\n+\t\t*key = empty;\n+\n+\t\tstrbuf_reset(&key->hash);\n+\t\tstrbuf_addbuf(&key->hash, hash);\n+\t\ttree_search((void *)key, &w->obj_index_tree,\n+\t\t\t &obj_index_tree_node_compare, 1);\n+\t} else {\n+\t\tkey = node->key;\n+\t}\n+\n+\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off) {\n+\t\treturn;\n+\t}\n+\n+\tif (key->offset_len == key->offset_cap) {\n+\t\tkey->offset_cap = 2 * key->offset_cap + 1;\n+\t\tkey->offsets = reftable_realloc(\n+\t\t\tkey->offsets, sizeof(uint64_t) * key->offset_cap);\n+\t}\n+\n+\tkey->offsets[key->offset_len++] = off;\n+}\n+\n+static int writer_add_record(struct reftable_writer *w,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tint result = -1;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = 0;\n+\treftable_record_key(rec, &key);\n+\tif (strbuf_cmp(&w->last_key, &key) >= 0)\n+\t\tgoto done;\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, &key);\n+\tif (w->block_writer == NULL) {\n+\t\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\t}\n+\n+\tassert(block_writer_type(w->block_writer) == reftable_record_type(rec));\n+\n+\tif (block_writer_add(w->block_writer, rec) == 0) {\n+\t\tresult = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0) {\n+\t\tresult = err;\n+\t\tgoto done;\n+\t}\n+\n+\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\terr = block_writer_add(w->block_writer, rec);\n+\tif (err < 0) {\n+\t\tresult = err;\n+\t\tgoto done;\n+\t}\n+\n+\tresult = 0;\n+done:\n+\tstrbuf_release(&key);\n+\treturn result;\n+}\n+\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { 0 };\n+\tstruct reftable_ref_record copy = *ref;\n+\tint err = 0;\n+\n+\tif (ref->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\tif (ref->update_index < w->min_update_index ||\n+\t ref->update_index > w->max_update_index)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\treftable_record_from_ref(&rec, ©);\n+\tcopy.update_index -= w->min_update_index;\n+\terr = writer_add_record(w, &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (!w->opts.skip_index_objects && ref->value != NULL) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, (char *)ref->value, hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\n+\tif (!w->opts.skip_index_objects && ref->target_value != NULL) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, ref->target_value, hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(refs, n, reftable_ref_record_compare_name);\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_ref(w, &refs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { 0 };\n+\tchar *input_log_message = log->message;\n+\tstruct strbuf cleaned_message = STRBUF_INIT;\n+\tint err;\n+\tif (log->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\tif (w->block_writer != NULL &&\n+\t block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {\n+\t\tint err = writer_finish_public_section(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tif (!w->opts.exact_log_message && log->message != NULL) {\n+\t\tstrbuf_addstr(&cleaned_message, log->message);\n+\t\twhile (cleaned_message.len &&\n+\t\t cleaned_message.buf[cleaned_message.len - 1] == '\\n')\n+\t\t\tstrbuf_setlen(&cleaned_message,\n+\t\t\t\t cleaned_message.len - 1);\n+\t\tif (strchr(cleaned_message.buf, '\\n')) {\n+\t\t\t// multiple lines not allowed.\n+\t\t\terr = REFTABLE_API_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tstrbuf_addstr(&cleaned_message, \"\\n\");\n+\t\tlog->message = cleaned_message.buf;\n+\t}\n+\n+\tw->next -= w->pending_padding;\n+\tw->pending_padding = 0;\n+\n+\treftable_record_from_log(&rec, log);\n+\terr = writer_add_record(w, &rec);\n+\n+done:\n+\tlog->message = input_log_message;\n+\tstrbuf_release(&cleaned_message);\n+\treturn err;\n+}\n+\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(logs, n, reftable_log_record_compare_key);\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_log(w, &logs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+static int writer_finish_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tuint64_t index_start = 0;\n+\tint max_level = 0;\n+\tint threshold = w->opts.unpadded ? 1 : 3;\n+\tint before_blocks = w->stats.idx_stats.blocks;\n+\tint err = writer_flush_block(w);\n+\tint i = 0;\n+\tstruct reftable_block_stats *bstats = NULL;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\twhile (w->index_len > threshold) {\n+\t\tstruct reftable_index_record *idx = NULL;\n+\t\tint idx_len = 0;\n+\n+\t\tmax_level++;\n+\t\tindex_start = w->next;\n+\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\tidx = w->index;\n+\t\tidx_len = w->index_len;\n+\n+\t\tw->index = NULL;\n+\t\tw->index_len = 0;\n+\t\tw->index_cap = 0;\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstruct reftable_record rec = { 0 };\n+\t\t\treftable_record_from_index(&rec, idx + i);\n+\t\t\tif (block_writer_add(w->block_writer, &rec) == 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\terr = writer_flush_block(w);\n+\t\t\tif (err < 0)\n+\t\t\t\treturn err;\n+\n+\t\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\t\terr = block_writer_add(w->block_writer, &rec);\n+\t\t\tif (err != 0) {\n+\t\t\t\t/* write into fresh block should always succeed\n+\t\t\t\t */\n+\t\t\t\tabort();\n+\t\t\t}\n+\t\t}\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstrbuf_release(&idx[i].last_key);\n+\t\t}\n+\t\treftable_free(idx);\n+\t}\n+\n+\twriter_clear_index(w);\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tbstats = writer_reftable_block_stats(w, typ);\n+\tbstats->index_blocks = w->stats.idx_stats.blocks - before_blocks;\n+\tbstats->index_offset = index_start;\n+\tbstats->max_index_level = max_level;\n+\n+\t/* Reinit lastKey, as the next section can start with any key. */\n+\tw->last_key.len = 0;\n+\n+\treturn 0;\n+}\n+\n+struct common_prefix_arg {\n+\tstruct strbuf *last;\n+\tint max;\n+};\n+\n+static void update_common(void *void_arg, void *key)\n+{\n+\tstruct common_prefix_arg *arg = (struct common_prefix_arg *)void_arg;\n+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n+\tif (arg->last != NULL) {\n+\t\tint n = common_prefix_size(&entry->hash, arg->last);\n+\t\tif (n > arg->max) {\n+\t\t\targ->max = n;\n+\t\t}\n+\t}\n+\targ->last = &entry->hash;\n+}\n+\n+struct write_record_arg {\n+\tstruct reftable_writer *w;\n+\tint err;\n+};\n+\n+static void write_object_record(void *void_arg, void *key)\n+{\n+\tstruct write_record_arg *arg = (struct write_record_arg *)void_arg;\n+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n+\tstruct reftable_obj_record obj_rec = {\n+\t\t.hash_prefix = (uint8_t *)entry->hash.buf,\n+\t\t.hash_prefix_len = arg->w->stats.object_id_len,\n+\t\t.offsets = entry->offsets,\n+\t\t.offset_len = entry->offset_len,\n+\t};\n+\tstruct reftable_record rec = { 0 };\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\treftable_record_from_obj(&rec, &obj_rec);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\n+\targ->err = writer_flush_block(arg->w);\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\twriter_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\tobj_rec.offset_len = 0;\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\n+\t/* Should be able to write into a fresh block. */\n+\tassert(arg->err == 0);\n+\n+done:;\n+}\n+\n+static void object_record_free(void *void_arg, void *key)\n+{\n+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n+\n+\tFREE_AND_NULL(entry->offsets);\n+\tstrbuf_release(&entry->hash);\n+\treftable_free(entry);\n+}\n+\n+static int writer_dump_object_index(struct reftable_writer *w)\n+{\n+\tstruct write_record_arg closure = { .w = w };\n+\tstruct common_prefix_arg common = { 0 };\n+\tif (w->obj_index_tree != NULL) {\n+\t\tinfix_walk(w->obj_index_tree, &update_common, &common);\n+\t}\n+\tw->stats.object_id_len = common.max + 1;\n+\n+\twriter_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\n+\tif (w->obj_index_tree != NULL) {\n+\t\tinfix_walk(w->obj_index_tree, &write_object_record, &closure);\n+\t}\n+\n+\tif (closure.err < 0)\n+\t\treturn closure.err;\n+\treturn writer_finish_section(w);\n+}\n+\n+int writer_finish_public_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = 0;\n+\tint err = 0;\n+\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\n+\ttyp = block_writer_type(w->block_writer);\n+\terr = writer_finish_section(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\tif (typ == BLOCK_TYPE_REF && !w->opts.skip_index_objects &&\n+\t w->stats.ref_stats.index_blocks > 0) {\n+\t\terr = writer_dump_object_index(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tif (w->obj_index_tree != NULL) {\n+\t\tinfix_walk(w->obj_index_tree, &object_record_free, NULL);\n+\t\ttree_free(w->obj_index_tree);\n+\t\tw->obj_index_tree = NULL;\n+\t}\n+\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+int reftable_writer_close(struct reftable_writer *w)\n+{\n+\tuint8_t footer[72];\n+\tuint8_t *p = footer;\n+\tint err = writer_finish_public_section(w);\n+\tint empty_table = w->next == 0;\n+\tif (err != 0)\n+\t\tgoto done;\n+\tw->pending_padding = 0;\n+\tif (empty_table) {\n+\t\t/* Empty tables need a header anyway. */\n+\t\tuint8_t header[28];\n+\t\tint n = writer_write_header(w, header);\n+\t\terr = padded_write(w, header, n, 0);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\tp += writer_write_header(w, footer);\n+\tput_be64(p, w->stats.ref_stats.index_offset);\n+\tp += 8;\n+\tput_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n+\tp += 8;\n+\tput_be64(p, w->stats.obj_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be64(p, w->stats.log_stats.offset);\n+\tp += 8;\n+\tput_be64(p, w->stats.log_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be32(p, crc32(0, footer, p - footer));\n+\tp += 4;\n+\n+\terr = padded_write(w, footer, footer_size(writer_version(w)), 0);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (empty_table) {\n+\t\terr = REFTABLE_EMPTY_TABLE_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\t/* free up memory. */\n+\tblock_writer_clear(&w->block_writer_data);\n+\twriter_clear_index(w);\n+\tstrbuf_release(&w->last_key);\n+\treturn err;\n+}\n+\n+void writer_clear_index(struct reftable_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->index_len; i++) {\n+\t\tstrbuf_release(&w->index[i].last_key);\n+\t}\n+\n+\tFREE_AND_NULL(w->index);\n+\tw->index_len = 0;\n+\tw->index_cap = 0;\n+}\n+\n+const int debug = 0;\n+\n+static int writer_flush_nonempty_block(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tstruct reftable_block_stats *bstats =\n+\t\twriter_reftable_block_stats(w, typ);\n+\tuint64_t block_typ_off = (bstats->blocks == 0) ? w->next : 0;\n+\tint raw_bytes = block_writer_finish(w->block_writer);\n+\tint padding = 0;\n+\tint err = 0;\n+\tstruct reftable_index_record ir = { .last_key = STRBUF_INIT };\n+\tif (raw_bytes < 0)\n+\t\treturn raw_bytes;\n+\n+\tif (!w->opts.unpadded && typ != BLOCK_TYPE_LOG) {\n+\t\tpadding = w->opts.block_size - raw_bytes;\n+\t}\n+\n+\tif (block_typ_off > 0) {\n+\t\tbstats->offset = block_typ_off;\n+\t}\n+\n+\tbstats->entries += w->block_writer->entries;\n+\tbstats->restarts += w->block_writer->restart_len;\n+\tbstats->blocks++;\n+\tw->stats.blocks++;\n+\n+\tif (debug) {\n+\t\tfprintf(stderr, \"block %c off %\" PRIu64 \" sz %d (%d)\\n\", typ,\n+\t\t\tw->next, raw_bytes,\n+\t\t\tget_be24(w->block + w->block_writer->header_off + 1));\n+\t}\n+\n+\tif (w->next == 0) {\n+\t\twriter_write_header(w, w->block);\n+\t}\n+\n+\terr = padded_write(w, w->block, raw_bytes, padding);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (w->index_cap == w->index_len) {\n+\t\tw->index_cap = 2 * w->index_cap + 1;\n+\t\tw->index = reftable_realloc(\n+\t\t\tw->index,\n+\t\t\tsizeof(struct reftable_index_record) * w->index_cap);\n+\t}\n+\n+\tir.offset = w->next;\n+\tstrbuf_reset(&ir.last_key);\n+\tstrbuf_addbuf(&ir.last_key, &w->block_writer->last_key);\n+\tw->index[w->index_len] = ir;\n+\n+\tw->index_len++;\n+\tw->next += padding + raw_bytes;\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+int writer_flush_block(struct reftable_writer *w)\n+{\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\tif (w->block_writer->entries == 0)\n+\t\treturn 0;\n+\treturn writer_flush_nonempty_block(w);\n+}\n+\n+const struct reftable_stats *writer_stats(struct reftable_writer *w)\n+{\n+\treturn &w->stats;\n+}\ndiff --git a/reftable/writer.h b/reftable/writer.h\nnew file mode 100644\nindex 0000000000..c69ecffea0\n--- /dev/null\n+++ b/reftable/writer.h\n@@ -0,0 +1,60 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef WRITER_H\n+#define WRITER_H\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"reftable.h\"\n+#include \"strbuf.h\"\n+#include \"tree.h\"\n+\n+struct reftable_writer {\n+\tint (*write)(void *, const void *, size_t);\n+\tvoid *write_arg;\n+\tint pending_padding;\n+\tstruct strbuf last_key;\n+\n+\t/* offset of next block to write. */\n+\tuint64_t next;\n+\tuint64_t min_update_index, max_update_index;\n+\tstruct reftable_write_options opts;\n+\n+\t/* memory buffer for writing */\n+\tuint8_t *block;\n+\n+\t/* writer for the current section. NULL or points to\n+\t * block_writer_data */\n+\tstruct block_writer *block_writer;\n+\n+\tstruct block_writer block_writer_data;\n+\n+\t/* pending index records for the current section */\n+\tstruct reftable_index_record *index;\n+\tsize_t index_len;\n+\tsize_t index_cap;\n+\n+\t/*\n+\t tree for use with tsearch; used to populate the 'o' inverse OID\n+\t map */\n+\tstruct tree_node *obj_index_tree;\n+\n+\tstruct reftable_stats stats;\n+};\n+\n+/* finishes a block, and writes it to storage */\n+int writer_flush_block(struct reftable_writer *w);\n+\n+/* deallocates memory related to the index */\n+void writer_clear_index(struct reftable_writer *w);\n+\n+/* finishes writing a 'r' (refs) or 'g' (reflogs) section */\n+int writer_finish_public_section(struct reftable_writer *w);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"405661","messageId":"b8729fee9e7609abba6fe21c1f37873fb33e3c48.1600283416.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.git.git.1600283416.gitgitgadget@gmail.com","subject":"[PATCH 13/13] reftable: \"test-tool dump-reftable\" command.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-16T19:10:16Z","receivedAt":"2020-09-16T19:11:32Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis command dumps individual tables or a stack of of tables.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 ++\n t/helper/test-reftable.c | 5 +++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 9 insertions(+)\n\ndiff --git a/Makefile b/Makefile\nindex 6652259baf..ee82ff2915 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2363,6 +2363,8 @@ REFTABLE_OBJS += reftable/writer.o\n REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/dump.o\n+REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/refname_test.o\n REFTABLE_TEST_OBJS += reftable/reftable_test.o\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex def8883439..aff4fbccda 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -13,3 +13,8 @@ int cmd__reftable(int argc, const char **argv)\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n+\n+int cmd__dump_reftable(int argc, const char **argv)\n+{\n+\treturn reftable_dump_main(argc, (char *const *)argv);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 10366b7b76..9e689f9d2b 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -53,6 +53,7 @@ static struct test_cmd cmds[] = {\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"reftable\", cmd__reftable },\n+\t{ \"dump-reftable\", cmd__dump_reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex d52ba2f5e5..bf833e01d4 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -17,6 +17,7 @@ int cmd__dump_cache_tree(int argc, const char **argv);\n int cmd__dump_fsmonitor(int argc, const char **argv);\n int cmd__dump_split_index(int argc, const char **argv);\n int cmd__dump_untracked_cache(int argc, const char **argv);\n+int cmd__dump_reftable(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\n-- \ngitgitgadget\n"},{"id":"405662","messageId":"3448fcf82866602ea41c66160c92571f9736f61e.1600283416.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.git.git.1600283416.gitgitgadget@gmail.com","subject":"[PATCH 11/13] reftable: file level tests","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-16T19:10:14Z","receivedAt":"2020-09-16T19:11:36Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable_test.c | 585 +++++++++++++++++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 3 files changed, 587 insertions(+)\n create mode 100644 reftable/reftable_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 275fc4227c..22dd31e7ff 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2360,6 +2360,7 @@ REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n+REFTABLE_TEST_OBJS += reftable/reftable_test.o\n REFTABLE_TEST_OBJS += reftable/strbuf_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\ndiff --git a/reftable/reftable_test.c b/reftable/reftable_test.c\nnew file mode 100644\nindex 0000000000..73efeaf523\n--- /dev/null\n+++ b/reftable/reftable_test.c\n@@ -0,0 +1,585 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable.h\"\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static const int update_index = 5;\n+\n+static void test_buffer(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_block out = { 0 };\n+\tint n;\n+\tuint8_t in[] = \"hello\";\n+\tstrbuf_add(&buf, in, sizeof(in));\n+\tblock_source_from_strbuf(&source, &buf);\n+\tassert(block_source_size(&source) == 6);\n+\tn = block_source_read_block(&source, &out, 0, sizeof(in));\n+\tassert(n == sizeof(in));\n+\tassert(!memcmp(in, out.data, n));\n+\treftable_block_done(&out);\n+\n+\tn = block_source_read_block(&source, &out, 1, 2);\n+\tassert(n == 2);\n+\tassert(!memcmp(out.data, \"el\", 2));\n+\n+\treftable_block_done(&out);\n+\tblock_source_close(&source);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void write_table(char ***names, struct strbuf *buf, int N,\n+\t\t\tint block_size, uint32_t hash_id)\n+{\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = block_size,\n+\t\t.hash_id = hash_id,\n+\t};\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, buf, &opts);\n+\tstruct reftable_ref_record ref = { 0 };\n+\tint i = 0, n;\n+\tstruct reftable_log_record log = { 0 };\n+\tconst struct reftable_stats *stats = NULL;\n+\t*names = reftable_calloc(sizeof(char *) * (N + 1));\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[SHA256_SIZE] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tref.refname = name;\n+\t\tref.value = hash;\n+\t\tref.update_index = update_index;\n+\t\t(*names)[i] = xstrdup(name);\n+\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tassert(n == 0);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[SHA256_SIZE] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tlog.refname = name;\n+\t\tlog.new_hash = hash;\n+\t\tlog.update_index = update_index;\n+\t\tlog.message = \"message\";\n+\n+\t\tn = reftable_writer_add_log(w, &log);\n+\t\tassert(n == 0);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tassert(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tfor (i = 0; i < stats->ref_stats.blocks; i++) {\n+\t\tint off = i * opts.block_size;\n+\t\tif (off == 0) {\n+\t\t\toff = header_size((hash_id == SHA256_ID) ? 2 : 1);\n+\t\t}\n+\t\tassert(buf->buf[off] == 'r');\n+\t}\n+\n+\tassert(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+}\n+\n+static void test_log_buffer_size(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 4096,\n+\t};\n+\tint err;\n+\tstruct reftable_log_record log = {\n+\t\t.refname = \"refs/heads/master\",\n+\t\t.name = \"Han-Wen Nienhuys\",\n+\t\t.email = \"hanwen@google.com\",\n+\t\t.tz_offset = 100,\n+\t\t.time = 0x5e430672,\n+\t\t.update_index = 0xa,\n+\t\t.message = \"commit: 9\\n\",\n+\t};\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\t/* This tests buffer extension for log compression. Must use a random\n+\t hash, to ensure that the compressed part is larger than the original.\n+\t*/\n+\tuint8_t hash1[SHA1_SIZE], hash2[SHA1_SIZE];\n+\tfor (int i = 0; i < SHA1_SIZE; i++) {\n+\t\thash1[i] = (uint8_t)(rand() % 256);\n+\t\thash2[i] = (uint8_t)(rand() % 256);\n+\t}\n+\tlog.old_hash = hash1;\n+\tlog.new_hash = hash2;\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\terr = reftable_writer_add_log(w, &log);\n+\tassert_err(err);\n+\terr = reftable_writer_close(w);\n+\tassert_err(err);\n+\treftable_writer_free(w);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_log_write_read(void)\n+{\n+\tint N = 2;\n+\tchar **names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint err;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { 0 };\n+\tint i = 0;\n+\tstruct reftable_log_record log = { 0 };\n+\tint n;\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_reader rd = { 0 };\n+\tstruct reftable_block_source source = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tconst struct reftable_stats *stats = NULL;\n+\treftable_writer_set_limits(w, 0, N);\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[256];\n+\t\tstruct reftable_ref_record ref = { 0 };\n+\t\tsnprintf(name, sizeof(name), \"b%02d%0*d\", i, 130, 7);\n+\t\tnames[i] = xstrdup(name);\n+\t\tref.refname = name;\n+\t\tref.update_index = i;\n+\n+\t\terr = reftable_writer_add_ref(w, &ref);\n+\t\tassert_err(err);\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash1[SHA1_SIZE], hash2[SHA1_SIZE];\n+\t\tstruct reftable_log_record log = { 0 };\n+\t\tset_test_hash(hash1, i);\n+\t\tset_test_hash(hash2, i + 1);\n+\n+\t\tlog.refname = names[i];\n+\t\tlog.update_index = i;\n+\t\tlog.old_hash = hash1;\n+\t\tlog.new_hash = hash2;\n+\n+\t\terr = reftable_writer_add_log(w, &log);\n+\t\tassert_err(err);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tassert(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tassert(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tassert_err(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[N - 1]);\n+\tassert_err(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tassert_err(err);\n+\n+\t/* end of iteration. */\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tassert(0 < err);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_clear(&ref);\n+\n+\terr = reftable_reader_seek_log(&rd, &it, \"\");\n+\tassert_err(err);\n+\n+\ti = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tassert_err(err);\n+\t\tassert_streq(names[i], log.refname);\n+\t\tassert(i == log.update_index);\n+\t\ti++;\n+\t\treftable_log_record_clear(&log);\n+\t}\n+\n+\tassert(i == N);\n+\treftable_iterator_destroy(&it);\n+\n+\t/* cleanup. */\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_sequential(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_block_source source = { 0 };\n+\tstruct reftable_reader rd = { 0 };\n+\tint err = 0;\n+\tint j = 0;\n+\n+\twrite_table(&names, &buf, N, 256, SHA1_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tassert_err(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tassert_err(err);\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { 0 };\n+\t\tint r = reftable_iterator_next_ref(&it, &ref);\n+\t\tassert(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tassert(0 == strcmp(names[j], ref.refname));\n+\t\tassert(update_index == ref.update_index);\n+\n+\t\tj++;\n+\t\treftable_ref_record_clear(&ref);\n+\t}\n+\tassert(j == N);\n+\treftable_iterator_destroy(&it);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\n+\treader_close(&rd);\n+}\n+\n+static void test_table_write_small_table(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 1;\n+\twrite_table(&names, &buf, N, 4096, SHA1_ID);\n+\tassert(buf.len < 200);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+}\n+\n+static void test_table_read_api(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { 0 };\n+\tstruct reftable_block_source source = { 0 };\n+\tint err;\n+\tint i;\n+\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_iterator it = { 0 };\n+\n+\twrite_table(&names, &buf, N, 256, SHA1_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tassert_err(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[0]);\n+\tassert_err(err);\n+\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tassert(err == REFTABLE_API_ERROR);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_free(names);\n+\treader_close(&rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_table_read_write_seek(int index, int hash_id)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { 0 };\n+\tstruct reftable_block_source source = { 0 };\n+\tint err;\n+\tint i = 0;\n+\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct strbuf pastLast = STRBUF_INIT;\n+\tstruct reftable_ref_record ref = { 0 };\n+\n+\twrite_table(&names, &buf, N, 256, hash_id);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tassert_err(err);\n+\tassert(hash_id == reftable_reader_hash_id(&rd));\n+\n+\tif (!index) {\n+\t\trd.ref_offsets.index_offset = 0;\n+\t} else {\n+\t\tassert(rd.ref_offsets.index_offset > 0);\n+\t}\n+\n+\tfor (i = 1; i < N; i++) {\n+\t\tint err = reftable_reader_seek_ref(&rd, &it, names[i]);\n+\t\tassert_err(err);\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tassert_err(err);\n+\t\tassert(0 == strcmp(names[i], ref.refname));\n+\t\tassert(i == ref.value[0]);\n+\n+\t\treftable_ref_record_clear(&ref);\n+\t\treftable_iterator_destroy(&it);\n+\t}\n+\n+\tstrbuf_addstr(&pastLast, names[N - 1]);\n+\tstrbuf_addstr(&pastLast, \"/\");\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, pastLast.buf);\n+\tif (err == 0) {\n+\t\tstruct reftable_ref_record ref = { 0 };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tassert(err > 0);\n+\t} else {\n+\t\tassert(err > 0);\n+\t}\n+\n+\tstrbuf_release(&pastLast);\n+\treftable_iterator_destroy(&it);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_free(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_seek_linear(void)\n+{\n+\ttest_table_read_write_seek(0, SHA1_ID);\n+}\n+\n+static void test_table_read_write_seek_linear_sha256(void)\n+{\n+\ttest_table_read_write_seek(0, SHA256_ID);\n+}\n+\n+static void test_table_read_write_seek_index(void)\n+{\n+\ttest_table_read_write_seek(1, SHA1_ID);\n+}\n+\n+static void test_table_refs_for(int indexed)\n+{\n+\tint N = 50;\n+\tchar **want_names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint want_names_len = 0;\n+\tuint8_t want_hash[SHA1_SIZE];\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { 0 };\n+\tint i = 0;\n+\tint n;\n+\tint err;\n+\tstruct reftable_reader rd;\n+\tstruct reftable_block_source source = { 0 };\n+\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_iterator it = { 0 };\n+\tint j;\n+\n+\tset_test_hash(want_hash, 4);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[SHA1_SIZE];\n+\t\tchar fill[51] = { 0 };\n+\t\tchar name[100];\n+\t\tuint8_t hash1[SHA1_SIZE];\n+\t\tuint8_t hash2[SHA1_SIZE];\n+\t\tstruct reftable_ref_record ref = { 0 };\n+\n+\t\tmemset(hash, i, sizeof(hash));\n+\t\tmemset(fill, 'x', 50);\n+\t\t/* Put the variable part in the start */\n+\t\tsnprintf(name, sizeof(name), \"br%02d%s\", i, fill);\n+\t\tname[40] = 0;\n+\t\tref.refname = name;\n+\n+\t\tset_test_hash(hash1, i / 4);\n+\t\tset_test_hash(hash2, 3 + i / 4);\n+\t\tref.value = hash1;\n+\t\tref.target_value = hash2;\n+\n+\t\t/* 80 bytes / entry, so 3 entries per block. Yields 17\n+\t\t */\n+\t\t/* blocks. */\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tassert(n == 0);\n+\n+\t\tif (!memcmp(hash1, want_hash, SHA1_SIZE) ||\n+\t\t !memcmp(hash2, want_hash, SHA1_SIZE)) {\n+\t\t\twant_names[want_names_len++] = xstrdup(name);\n+\t\t}\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tassert(n == 0);\n+\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tassert_err(err);\n+\tif (!indexed) {\n+\t\trd.obj_offsets.is_present = 0;\n+\t}\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tassert_err(err);\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_reader_refs_for(&rd, &it, want_hash);\n+\tassert_err(err);\n+\n+\tj = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tassert(err >= 0);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tassert(j < want_names_len);\n+\t\tassert(0 == strcmp(ref.refname, want_names[j]));\n+\t\tj++;\n+\t\treftable_ref_record_clear(&ref);\n+\t}\n+\tassert(j == want_names_len);\n+\n+\tstrbuf_release(&buf);\n+\tfree_names(want_names);\n+\treftable_iterator_destroy(&it);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_refs_for_no_index(void)\n+{\n+\ttest_table_refs_for(0);\n+}\n+\n+static void test_table_refs_for_obj_index(void)\n+{\n+\ttest_table_refs_for(1);\n+}\n+\n+static void test_table_empty(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_block_source source = { 0 };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_ref_record rec = { 0 };\n+\tstruct reftable_iterator it = { 0 };\n+\tint err;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_close(w);\n+\tassert(err == REFTABLE_EMPTY_TABLE_ERROR);\n+\treftable_writer_free(w);\n+\n+\tassert(buf.len == header_size(1) + footer_size(1));\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tassert_err(err);\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tassert_err(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tassert(err > 0);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+int reftable_test_main(int argc, const char *argv[])\n+{\n+\tadd_test_case(\"test_log_write_read\", test_log_write_read);\n+\tadd_test_case(\"test_table_read_write_seek_linear_sha256\",\n+\t\t &test_table_read_write_seek_linear_sha256);\n+\tadd_test_case(\"test_log_buffer_size\", test_log_buffer_size);\n+\tadd_test_case(\"test_table_write_small_table\",\n+\t\t &test_table_write_small_table);\n+\tadd_test_case(\"test_buffer\", &test_buffer);\n+\tadd_test_case(\"test_table_read_api\", &test_table_read_api);\n+\tadd_test_case(\"test_table_read_write_sequential\",\n+\t\t &test_table_read_write_sequential);\n+\tadd_test_case(\"test_table_read_write_seek_linear\",\n+\t\t &test_table_read_write_seek_linear);\n+\tadd_test_case(\"test_table_read_write_seek_index\",\n+\t\t &test_table_read_write_seek_index);\n+\tadd_test_case(\"test_table_read_write_refs_for_no_index\",\n+\t\t &test_table_refs_for_no_index);\n+\tadd_test_case(\"test_table_read_write_refs_for_obj_index\",\n+\t\t &test_table_refs_for_obj_index);\n+\tadd_test_case(\"test_table_empty\", &test_table_empty);\n+\treturn test_main(argc, argv);\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 9c4e0f42dc..2273fdfe39 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -5,6 +5,7 @@ int cmd__reftable(int argc, const char **argv)\n {\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\treftable_test_main(argc, argv);\n \tstrbuf_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n-- \ngitgitgadget\n\n"},{"id":"405663","messageId":"17fb8d050dae2efd84f090268095da4d04b15c88.1600283416.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.git.git.1600283416.gitgitgadget@gmail.com","subject":"[PATCH 08/13] reftable: a generic binary tree implementation","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-16T19:10:11Z","receivedAt":"2020-09-16T19:11:37Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis is necessary for building a OID => ref map on write\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 ++\n reftable/tree.c | 63 ++++++++++++++++++++++++++++++++++++++++\n reftable/tree.h | 34 ++++++++++++++++++++++\n reftable/tree_test.c | 62 +++++++++++++++++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 162 insertions(+)\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n\ndiff --git a/Makefile b/Makefile\nindex a897818a7e..b96c235087 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2351,12 +2351,14 @@ REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/compat.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/strbuf.o\n+REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/strbuf_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n+REFTABLE_TEST_OBJS += reftable/tree_test.o\n \n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \\\ndiff --git a/reftable/tree.c b/reftable/tree.c\nnew file mode 100644\nindex 0000000000..0061d14e30\n--- /dev/null\n+++ b/reftable/tree.c\n@@ -0,0 +1,63 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert)\n+{\n+\tint res;\n+\tif (*rootp == NULL) {\n+\t\tif (!insert) {\n+\t\t\treturn NULL;\n+\t\t} else {\n+\t\t\tstruct tree_node *n =\n+\t\t\t\treftable_calloc(sizeof(struct tree_node));\n+\t\t\tn->key = key;\n+\t\t\t*rootp = n;\n+\t\t\treturn *rootp;\n+\t\t}\n+\t}\n+\n+\tres = compare(key, (*rootp)->key);\n+\tif (res < 0)\n+\t\treturn tree_search(key, &(*rootp)->left, compare, insert);\n+\telse if (res > 0)\n+\t\treturn tree_search(key, &(*rootp)->right, compare, insert);\n+\treturn *rootp;\n+}\n+\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg)\n+{\n+\tif (t->left != NULL) {\n+\t\tinfix_walk(t->left, action, arg);\n+\t}\n+\taction(arg, t->key);\n+\tif (t->right != NULL) {\n+\t\tinfix_walk(t->right, action, arg);\n+\t}\n+}\n+\n+void tree_free(struct tree_node *t)\n+{\n+\tif (t == NULL) {\n+\t\treturn;\n+\t}\n+\tif (t->left != NULL) {\n+\t\ttree_free(t->left);\n+\t}\n+\tif (t->right != NULL) {\n+\t\ttree_free(t->right);\n+\t}\n+\treftable_free(t);\n+}\ndiff --git a/reftable/tree.h b/reftable/tree.h\nnew file mode 100644\nindex 0000000000..954512e9a3\n--- /dev/null\n+++ b/reftable/tree.h\n@@ -0,0 +1,34 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TREE_H\n+#define TREE_H\n+\n+/* tree_node is a generic binary search tree. */\n+struct tree_node {\n+\tvoid *key;\n+\tstruct tree_node *left, *right;\n+};\n+\n+/* looks for `key` in `rootp` using `compare` as comparison function. If insert\n+ is set, insert the key if it's not found. Else, return NULL.\n+*/\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert);\n+\n+/* performs an infix walk of the tree. */\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg);\n+\n+/*\n+ deallocates the tree nodes recursively. Keys should be deallocated separately\n+ by walking over the tree. */\n+void tree_free(struct tree_node *t);\n+\n+#endif\ndiff --git a/reftable/tree_test.c b/reftable/tree_test.c\nnew file mode 100644\nindex 0000000000..c6d448cbe8\n--- /dev/null\n+++ b/reftable/tree_test.c\n@@ -0,0 +1,62 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static int test_compare(const void *a, const void *b)\n+{\n+\treturn (char *)a - (char *)b;\n+}\n+\n+struct curry {\n+\tvoid *last;\n+};\n+\n+static void check_increasing(void *arg, void *key)\n+{\n+\tstruct curry *c = (struct curry *)arg;\n+\tif (c->last != NULL) {\n+\t\tassert(test_compare(c->last, key) < 0);\n+\t}\n+\tc->last = key;\n+}\n+\n+static void test_tree(void)\n+{\n+\tstruct tree_node *root = NULL;\n+\n+\tvoid *values[11] = { 0 };\n+\tstruct tree_node *nodes[11] = { 0 };\n+\tint i = 1;\n+\tstruct curry c = { 0 };\n+\tdo {\n+\t\tnodes[i] = tree_search(values + i, &root, &test_compare, 1);\n+\t\ti = (i * 7) % 11;\n+\t} while (i != 1);\n+\n+\tfor (i = 1; i < ARRAY_SIZE(nodes); i++) {\n+\t\tassert(values + i == nodes[i]->key);\n+\t\tassert(nodes[i] ==\n+\t\t tree_search(values + i, &root, &test_compare, 0));\n+\t}\n+\n+\tinfix_walk(root, check_increasing, &c);\n+\ttree_free(root);\n+}\n+\n+int tree_test_main(int argc, const char *argv[])\n+{\n+\tadd_test_case(\"test_tree\", &test_tree);\n+\treturn test_main(argc, argv);\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 81a9bd5667..9c4e0f42dc 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,5 +6,6 @@ int cmd__reftable(int argc, const char **argv)\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \tstrbuf_test_main(argc, argv);\n+\ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"405664","messageId":"1ba8e3eb3045b3ab6462febeaa8b90dc14910098.1600283416.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.git.git.1600283416.gitgitgadget@gmail.com","subject":"[PATCH 07/13] reftable: reading/writing blocks","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-16T19:10:10Z","receivedAt":"2020-09-16T19:11:42Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nIncludes a code snippet copied from zlib\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 3 +\n reftable/.gitattributes | 1 +\n reftable/block.c | 443 +++++++++++++++++++++++++++++++++++++++\n reftable/block.h | 129 ++++++++++++\n reftable/block_test.c | 158 ++++++++++++++\n reftable/zlib-compat.c | 92 ++++++++\n t/helper/test-reftable.c | 1 +\n 7 files changed, 827 insertions(+)\n create mode 100644 reftable/.gitattributes\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n create mode 100644 reftable/zlib-compat.c\n\ndiff --git a/Makefile b/Makefile\nindex 0fb0048c34..a897818a7e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2345,12 +2345,15 @@ XDIFF_OBJS += xdiff/xprepare.o\n XDIFF_OBJS += xdiff/xutils.o\n \n REFTABLE_OBJS += reftable/basics.o\n+REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/compat.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/strbuf.o\n+REFTABLE_OBJS += reftable/zlib-compat.o\n \n+REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/strbuf_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\ndiff --git a/reftable/.gitattributes b/reftable/.gitattributes\nnew file mode 100644\nindex 0000000000..f44451a379\n--- /dev/null\n+++ b/reftable/.gitattributes\n@@ -0,0 +1 @@\n+/zlib-compat.c\twhitespace=-indent-with-non-tab,-trailing-space\ndiff --git a/reftable/block.c b/reftable/block.c\nnew file mode 100644\nindex 0000000000..d61faa0fd4\n--- /dev/null\n+++ b/reftable/block.c\n@@ -0,0 +1,443 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+#include \"system.h\"\n+#include \"zlib.h\"\n+\n+int header_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 24;\n+\tcase 2:\n+\t\treturn 28;\n+\t}\n+\tabort();\n+}\n+\n+int footer_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 68;\n+\tcase 2:\n+\t\treturn 72;\n+\t}\n+\tabort();\n+}\n+\n+int block_writer_register_restart(struct block_writer *w, int n, int restart,\n+\t\t\t\t struct strbuf *key);\n+\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n+{\n+\tbw->buf = buf;\n+\tbw->hash_size = hash_size;\n+\tbw->block_size = block_size;\n+\tbw->header_off = header_off;\n+\tbw->buf[header_off] = typ;\n+\tbw->next = header_off + 4;\n+\tbw->restart_interval = 16;\n+\tbw->entries = 0;\n+\tbw->restart_len = 0;\n+\tbw->last_key.len = 0;\n+}\n+\n+uint8_t block_writer_type(struct block_writer *bw)\n+{\n+\treturn bw->buf[bw->header_off];\n+}\n+\n+/* adds the reftable_record to the block. Returns -1 if it does not fit, 0 on\n+ success */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tstruct strbuf last =\n+\t\tw->entries % w->restart_interval == 0 ? empty : w->last_key;\n+\tstruct string_view out = {\n+\t\t.buf = w->buf + w->next,\n+\t\t.len = w->block_size - w->next,\n+\t};\n+\n+\tstruct string_view start = out;\n+\n+\tint is_restart = 0;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint n = 0;\n+\n+\treftable_record_key(rec, &key);\n+\tn = reftable_encode_key(&is_restart, out, last, key,\n+\t\t\t\treftable_record_val_type(rec));\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tn = reftable_record_encode(rec, out, w->hash_size);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tif (block_writer_register_restart(w, start.len - out.len, is_restart,\n+\t\t\t\t\t &key) < 0)\n+\t\tgoto done;\n+\n+\tstrbuf_release(&key);\n+\treturn 0;\n+\n+done:\n+\tstrbuf_release(&key);\n+\treturn -1;\n+}\n+\n+int block_writer_register_restart(struct block_writer *w, int n, int is_restart,\n+\t\t\t\t struct strbuf *key)\n+{\n+\tint rlen = w->restart_len;\n+\tif (rlen >= MAX_RESTARTS) {\n+\t\tis_restart = 0;\n+\t}\n+\n+\tif (is_restart) {\n+\t\trlen++;\n+\t}\n+\tif (2 + 3 * rlen + n > w->block_size - w->next)\n+\t\treturn -1;\n+\tif (is_restart) {\n+\t\tif (w->restart_len == w->restart_cap) {\n+\t\t\tw->restart_cap = w->restart_cap * 2 + 1;\n+\t\t\tw->restarts = reftable_realloc(\n+\t\t\t\tw->restarts, sizeof(uint32_t) * w->restart_cap);\n+\t\t}\n+\n+\t\tw->restarts[w->restart_len++] = w->next;\n+\t}\n+\n+\tw->next += n;\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, key);\n+\tw->entries++;\n+\treturn 0;\n+}\n+\n+int block_writer_finish(struct block_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->restart_len; i++) {\n+\t\tput_be24(w->buf + w->next, w->restarts[i]);\n+\t\tw->next += 3;\n+\t}\n+\n+\tput_be16(w->buf + w->next, w->restart_len);\n+\tw->next += 2;\n+\tput_be24(w->buf + 1 + w->header_off, w->next);\n+\n+\tif (block_writer_type(w) == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + w->header_off;\n+\t\tuint8_t *compressed = NULL;\n+\t\tint zresult = 0;\n+\t\tuLongf src_len = w->next - block_header_skip;\n+\t\tsize_t dest_cap = src_len;\n+\n+\t\tcompressed = reftable_malloc(dest_cap);\n+\t\twhile (1) {\n+\t\t\tuLongf out_dest_len = dest_cap;\n+\n+\t\t\tzresult = compress2(compressed, &out_dest_len,\n+\t\t\t\t\t w->buf + block_header_skip, src_len,\n+\t\t\t\t\t 9);\n+\t\t\tif (zresult == Z_BUF_ERROR) {\n+\t\t\t\tdest_cap *= 2;\n+\t\t\t\tcompressed =\n+\t\t\t\t\treftable_realloc(compressed, dest_cap);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (Z_OK != zresult) {\n+\t\t\t\treftable_free(compressed);\n+\t\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t\t}\n+\n+\t\t\tmemcpy(w->buf + block_header_skip, compressed,\n+\t\t\t out_dest_len);\n+\t\t\tw->next = out_dest_len + block_header_skip;\n+\t\t\treftable_free(compressed);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\treturn w->next;\n+}\n+\n+uint8_t block_reader_type(struct block_reader *r)\n+{\n+\treturn r->block.data[r->header_off];\n+}\n+\n+int block_reader_init(struct block_reader *br, struct reftable_block *block,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size)\n+{\n+\tuint32_t full_block_size = table_block_size;\n+\tuint8_t typ = block->data[header_off];\n+\tuint32_t sz = get_be24(block->data + header_off + 1);\n+\n+\tuint16_t restart_count = 0;\n+\tuint32_t restart_start = 0;\n+\tuint8_t *restart_bytes = NULL;\n+\n+\tif (!reftable_is_block_type(typ))\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tif (typ == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + header_off;\n+\t\tuLongf dst_len = sz - block_header_skip; /* total size of dest\n+\t\t\t\t\t\t\t buffer. */\n+\t\tuLongf src_len = block->len - block_header_skip;\n+\t\t/* Log blocks specify the *uncompressed* size in their header.\n+\t\t */\n+\t\tuint8_t *uncompressed = reftable_malloc(sz);\n+\n+\t\t/* Copy over the block header verbatim. It's not compressed. */\n+\t\tmemcpy(uncompressed, block->data, block_header_skip);\n+\n+\t\t/* Uncompress */\n+\t\tif (Z_OK != uncompress_return_consumed(\n+\t\t\t\t uncompressed + block_header_skip, &dst_len,\n+\t\t\t\t block->data + block_header_skip,\n+\t\t\t\t &src_len)) {\n+\t\t\treftable_free(uncompressed);\n+\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t}\n+\n+\t\tif (dst_len + block_header_skip != sz)\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\t\t/* We're done with the input data. */\n+\t\treftable_block_done(block);\n+\t\tblock->data = uncompressed;\n+\t\tblock->len = sz;\n+\t\tblock->source = malloc_block_source();\n+\t\tfull_block_size = src_len + block_header_skip;\n+\t} else if (full_block_size == 0) {\n+\t\tfull_block_size = sz;\n+\t} else if (sz < full_block_size && sz < block->len &&\n+\t\t block->data[sz] != 0) {\n+\t\t/* If the block is smaller than the full block size, it is\n+\t\t padded (data followed by '\\0') or the next block is\n+\t\t unaligned. */\n+\t\tfull_block_size = sz;\n+\t}\n+\n+\trestart_count = get_be16(block->data + sz - 2);\n+\trestart_start = sz - 2 - 3 * restart_count;\n+\trestart_bytes = block->data + restart_start;\n+\n+\t/* transfer ownership. */\n+\tbr->block = *block;\n+\tblock->data = NULL;\n+\tblock->len = 0;\n+\n+\tbr->hash_size = hash_size;\n+\tbr->block_len = restart_start;\n+\tbr->full_block_size = full_block_size;\n+\tbr->header_off = header_off;\n+\tbr->restart_count = restart_count;\n+\tbr->restart_bytes = restart_bytes;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t block_reader_restart_offset(struct block_reader *br, int i)\n+{\n+\treturn get_be24(br->restart_bytes + 3 * i);\n+}\n+\n+void block_reader_start(struct block_reader *br, struct block_iter *it)\n+{\n+\tit->br = br;\n+\tstrbuf_reset(&it->last_key);\n+\tit->next_off = br->header_off + 4;\n+}\n+\n+struct restart_find_args {\n+\tint error;\n+\tstruct strbuf key;\n+\tstruct block_reader *r;\n+};\n+\n+static int restart_key_less(size_t idx, void *args)\n+{\n+\tstruct restart_find_args *a = (struct restart_find_args *)args;\n+\tuint32_t off = block_reader_restart_offset(a->r, idx);\n+\tstruct string_view in = {\n+\t\t.buf = a->r->block.data + off,\n+\t\t.len = a->r->block_len - off,\n+\t};\n+\n+\t/* the restart key is verbatim in the block, so this could avoid the\n+\t alloc for decoding the key */\n+\tstruct strbuf rkey = STRBUF_INIT;\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tuint8_t unused_extra;\n+\tint n = reftable_decode_key(&rkey, &unused_extra, last_key, in);\n+\tint result;\n+\tif (n < 0) {\n+\t\ta->error = 1;\n+\t\treturn -1;\n+\t}\n+\n+\tresult = strbuf_cmp(&a->key, &rkey);\n+\tstrbuf_release(&rkey);\n+\treturn result;\n+}\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src)\n+{\n+\tdest->br = src->br;\n+\tdest->next_off = src->next_off;\n+\tstrbuf_reset(&dest->last_key);\n+\tstrbuf_addbuf(&dest->last_key, &src->last_key);\n+}\n+\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec)\n+{\n+\tstruct string_view in = {\n+\t\t.buf = it->br->block.data + it->next_off,\n+\t\t.len = it->br->block_len - it->next_off,\n+\t};\n+\tstruct string_view start = in;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tuint8_t extra = 0;\n+\tint n = 0;\n+\n+\tif (it->next_off >= it->br->block_len)\n+\t\treturn 1;\n+\n+\tn = reftable_decode_key(&key, &extra, it->last_key, in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\n+\tstring_view_consume(&in, n);\n+\tn = reftable_record_decode(rec, key, extra, in, it->br->hash_size);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tstrbuf_reset(&it->last_key);\n+\tstrbuf_addbuf(&it->last_key, &key);\n+\tit->next_off += start.len - in.len;\n+\tstrbuf_release(&key);\n+\treturn 0;\n+}\n+\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tint off = br->header_off + 4;\n+\tstruct string_view in = {\n+\t\t.buf = br->block.data + off,\n+\t\t.len = br->block_len - off,\n+\t};\n+\n+\tuint8_t extra = 0;\n+\tint n = reftable_decode_key(key, &extra, empty, in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\treturn 0;\n+}\n+\n+int block_iter_seek(struct block_iter *it, struct strbuf *want)\n+{\n+\treturn block_reader_seek(it->br, it, want);\n+}\n+\n+void block_iter_close(struct block_iter *it)\n+{\n+\tstrbuf_release(&it->last_key);\n+}\n+\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want)\n+{\n+\tstruct restart_find_args args = {\n+\t\t.key = *want,\n+\t\t.r = br,\n+\t};\n+\tstruct reftable_record rec = reftable_new_record(block_reader_type(br));\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = 0;\n+\tstruct block_iter next = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\n+\tint i = binsearch(br->restart_count, &restart_key_less, &args);\n+\tif (args.error) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tit->br = br;\n+\tif (i > 0) {\n+\t\ti--;\n+\t\tit->next_off = block_reader_restart_offset(br, i);\n+\t} else {\n+\t\tit->next_off = br->header_off + 4;\n+\t}\n+\n+\t/* We're looking for the last entry less/equal than the wanted key, so\n+\t we have to go one entry too far and then back up.\n+\t*/\n+\twhile (1) {\n+\t\tblock_iter_copy_from(&next, it);\n+\t\terr = block_iter_next(&next, &rec);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tif (err > 0 || strbuf_cmp(&key, want) >= 0) {\n+\t\t\terr = 0;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tblock_iter_copy_from(it, &next);\n+\t}\n+\n+done:\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&next.last_key);\n+\treftable_record_destroy(&rec);\n+\n+\treturn err;\n+}\n+\n+void block_writer_clear(struct block_writer *bw)\n+{\n+\tFREE_AND_NULL(bw->restarts);\n+\tstrbuf_release(&bw->last_key);\n+\t/* the block is not owned. */\n+}\n+\n+void reftable_block_done(struct reftable_block *blockp)\n+{\n+\tstruct reftable_block_source source = blockp->source;\n+\tif (blockp != NULL && source.ops != NULL)\n+\t\tsource.ops->return_block(source.arg, blockp);\n+\tblockp->data = NULL;\n+\tblockp->len = 0;\n+\tblockp->source.ops = NULL;\n+\tblockp->source.arg = NULL;\n+}\ndiff --git a/reftable/block.h b/reftable/block.h\nnew file mode 100644\nindex 0000000000..14a6f835f4\n--- /dev/null\n+++ b/reftable/block.h\n@@ -0,0 +1,129 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCK_H\n+#define BLOCK_H\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+\n+/*\n+ Writes reftable blocks. The block_writer is reused across blocks to minimize\n+ allocation overhead.\n+*/\n+struct block_writer {\n+\tuint8_t *buf;\n+\tuint32_t block_size;\n+\n+\t/* Offset ofof the global header. Nonzero in the first block only. */\n+\tuint32_t header_off;\n+\n+\t/* How often to restart keys. */\n+\tint restart_interval;\n+\tint hash_size;\n+\n+\t/* Offset of next uint8_t to write. */\n+\tuint32_t next;\n+\tuint32_t *restarts;\n+\tuint32_t restart_len;\n+\tuint32_t restart_cap;\n+\n+\tstruct strbuf last_key;\n+\tint entries;\n+};\n+\n+/*\n+ initializes the blockwriter to write `typ` entries, using `buf` as temporary\n+ storage. `buf` is not owned by the block_writer. */\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n+\n+/*\n+ returns the block type (eg. 'r' for ref records.\n+*/\n+uint8_t block_writer_type(struct block_writer *bw);\n+\n+/* appends the record, or -1 if it doesn't fit. */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec);\n+\n+/* appends the key restarts, and compress the block if necessary. */\n+int block_writer_finish(struct block_writer *w);\n+\n+/* clears out internally allocated block_writer members. */\n+void block_writer_clear(struct block_writer *bw);\n+\n+/* Read a block. */\n+struct block_reader {\n+\t/* offset of the block header; nonzero for the first block in a\n+\t * reftable. */\n+\tuint32_t header_off;\n+\n+\t/* the memory block */\n+\tstruct reftable_block block;\n+\tint hash_size;\n+\n+\t/* size of the data, excluding restart data. */\n+\tuint32_t block_len;\n+\tuint8_t *restart_bytes;\n+\tuint16_t restart_count;\n+\n+\t/* size of the data in the file. For log blocks, this is the compressed\n+\t * size. */\n+\tuint32_t full_block_size;\n+};\n+\n+/* Iterate over entries in a block */\n+struct block_iter {\n+\t/* offset within the block of the next entry to read. */\n+\tuint32_t next_off;\n+\tstruct block_reader *br;\n+\n+\t/* key for last entry we read. */\n+\tstruct strbuf last_key;\n+};\n+\n+/* initializes a block reader. */\n+int block_reader_init(struct block_reader *br, struct reftable_block *bl,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size);\n+\n+/* Position `it` at start of the block */\n+void block_reader_start(struct block_reader *br, struct block_iter *it);\n+\n+/* Position `it` to the `want` key in the block */\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want);\n+\n+/* Returns the block type (eg. 'r' for refs) */\n+uint8_t block_reader_type(struct block_reader *r);\n+\n+/* Decodes the first key in the block */\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key);\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src);\n+\n+/* return < 0 for error, 0 for OK, > 0 for EOF. */\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec);\n+\n+/* Seek to `want` with in the block pointed to by `it` */\n+int block_iter_seek(struct block_iter *it, struct strbuf *want);\n+\n+/* deallocate memory for `it`. The block reader and its block is left intact. */\n+void block_iter_close(struct block_iter *it);\n+\n+/* size of file header, depending on format version */\n+int header_size(int version);\n+\n+/* size of file footer, depending on format version */\n+int footer_size(int version);\n+\n+/* returns a block to its source. */\n+void reftable_block_done(struct reftable_block *ret);\n+\n+#endif\ndiff --git a/reftable/block_test.c b/reftable/block_test.c\nnew file mode 100644\nindex 0000000000..1ca327b3c7\n--- /dev/null\n+++ b/reftable/block_test.c\n@@ -0,0 +1,158 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"system.h\"\n+\n+#include \"blocksource.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct binsearch_args {\n+\tint key;\n+\tint *arr;\n+};\n+\n+static int binsearch_func(size_t i, void *void_args)\n+{\n+\tstruct binsearch_args *args = (struct binsearch_args *)void_args;\n+\n+\treturn args->key < args->arr[i];\n+}\n+\n+static void test_binsearch(void)\n+{\n+\tint arr[] = { 2, 4, 6, 8, 10 };\n+\tsize_t sz = ARRAY_SIZE(arr);\n+\tstruct binsearch_args args = {\n+\t\t.arr = arr,\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 1; i < 11; i++) {\n+\t\tint res;\n+\t\targs.key = i;\n+\t\tres = binsearch(sz, &binsearch_func, &args);\n+\n+\t\tif (res < sz) {\n+\t\t\tassert(args.key < arr[res]);\n+\t\t\tif (res > 0) {\n+\t\t\t\tassert(args.key >= arr[res - 1]);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tassert(args.key == 10 || args.key == 11);\n+\t\t}\n+\t}\n+}\n+\n+static void test_block_read_write(void)\n+{\n+\tconst int header_off = 21; /* random */\n+\tchar *names[30];\n+\tconst int N = ARRAY_SIZE(names);\n+\tconst int block_size = 1024;\n+\tstruct reftable_block block = { 0 };\n+\tstruct block_writer bw = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tstruct reftable_ref_record ref = { 0 };\n+\tstruct reftable_record rec = { 0 };\n+\tint i = 0;\n+\tint n;\n+\tstruct block_reader br = { 0 };\n+\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\tint j = 0;\n+\tstruct strbuf want = STRBUF_INIT;\n+\n+\tblock.data = reftable_calloc(block_size);\n+\tblock.len = block_size;\n+\tblock.source = malloc_block_source();\n+\tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n+\t\t\t header_off, hash_size(SHA1_ID));\n+\treftable_record_from_ref(&rec, &ref);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tuint8_t hash[SHA1_SIZE];\n+\t\tsnprintf(name, sizeof(name), \"branch%02d\", i);\n+\t\tmemset(hash, i, sizeof(hash));\n+\n+\t\tref.refname = name;\n+\t\tref.value = hash;\n+\t\tnames[i] = xstrdup(name);\n+\t\tn = block_writer_add(&bw, &rec);\n+\t\tref.refname = NULL;\n+\t\tref.value = NULL;\n+\t\tassert(n == 0);\n+\t}\n+\n+\tn = block_writer_finish(&bw);\n+\tassert(n > 0);\n+\n+\tblock_writer_clear(&bw);\n+\n+\tblock_reader_init(&br, &block, header_off, block_size, SHA1_SIZE);\n+\n+\tblock_reader_start(&br, &it);\n+\n+\twhile (1) {\n+\t\tint r = block_iter_next(&it, &rec);\n+\t\tassert(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tassert_streq(names[j], ref.refname);\n+\t\tj++;\n+\t}\n+\n+\treftable_record_clear(&rec);\n+\tblock_iter_close(&it);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\t\tstrbuf_reset(&want);\n+\t\tstrbuf_addstr(&want, names[i]);\n+\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tassert(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tassert(n == 0);\n+\n+\t\tassert_streq(names[i], ref.refname);\n+\n+\t\twant.len--;\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tassert(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tassert(n == 0);\n+\t\tassert_streq(names[10 * (i / 10)], ref.refname);\n+\n+\t\tblock_iter_close(&it);\n+\t}\n+\n+\treftable_record_clear(&rec);\n+\treftable_block_done(&br.block);\n+\tstrbuf_release(&want);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+}\n+\n+int block_test_main(int argc, const char *argv[])\n+{\n+\tadd_test_case(\"binsearch\", &test_binsearch);\n+\tadd_test_case(\"block_read_write\", &test_block_read_write);\n+\treturn test_main(argc, argv);\n+}\ndiff --git a/reftable/zlib-compat.c b/reftable/zlib-compat.c\nnew file mode 100644\nindex 0000000000..3e0b0f24f1\n--- /dev/null\n+++ b/reftable/zlib-compat.c\n@@ -0,0 +1,92 @@\n+/* taken from zlib's uncompr.c\n+\n+ commit cacf7f1d4e3d44d871b605da3b647f07d718623f\n+ Author: Mark Adler \n+ Date: Sun Jan 15 09:18:46 2017 -0800\n+\n+ zlib 1.2.11\n+\n+*/\n+\n+/*\n+ * Copyright (C) 1995-2003, 2010, 2014, 2016 Jean-loup Gailly, Mark Adler\n+ * For conditions of distribution and use, see copyright notice in zlib.h\n+ */\n+\n+#include \"system.h\"\n+\n+/* clang-format off */\n+\n+/* ===========================================================================\n+ Decompresses the source buffer into the destination buffer. *sourceLen is\n+ the byte length of the source buffer. Upon entry, *destLen is the total size\n+ of the destination buffer, which must be large enough to hold the entire\n+ uncompressed data. (The size of the uncompressed data must have been saved\n+ previously by the compressor and transmitted to the decompressor by some\n+ mechanism outside the scope of this compression library.) Upon exit,\n+ *destLen is the size of the decompressed data and *sourceLen is the number\n+ of source bytes consumed. Upon return, source + *sourceLen points to the\n+ first unused input byte.\n+\n+ uncompress returns Z_OK if success, Z_MEM_ERROR if there was not enough\n+ memory, Z_BUF_ERROR if there was not enough room in the output buffer, or\n+ Z_DATA_ERROR if the input data was corrupted, including if the input data is\n+ an incomplete zlib stream.\n+*/\n+int ZEXPORT uncompress_return_consumed (\n+ Bytef *dest,\n+ uLongf *destLen,\n+ const Bytef *source,\n+ uLong *sourceLen) {\n+ z_stream stream;\n+ int err;\n+ const uInt max = (uInt)-1;\n+ uLong len, left;\n+ Byte buf[1]; /* for detection of incomplete stream when *destLen == 0 */\n+\n+ len = *sourceLen;\n+ if (*destLen) {\n+ left = *destLen;\n+ *destLen = 0;\n+ }\n+ else {\n+ left = 1;\n+ dest = buf;\n+ }\n+\n+ stream.next_in = (z_const Bytef *)source;\n+ stream.avail_in = 0;\n+ stream.zalloc = (alloc_func)0;\n+ stream.zfree = (free_func)0;\n+ stream.opaque = (voidpf)0;\n+\n+ err = inflateInit(&stream);\n+ if (err != Z_OK) return err;\n+\n+ stream.next_out = dest;\n+ stream.avail_out = 0;\n+\n+ do {\n+ if (stream.avail_out == 0) {\n+ stream.avail_out = left > (uLong)max ? max : (uInt)left;\n+ left -= stream.avail_out;\n+ }\n+ if (stream.avail_in == 0) {\n+ stream.avail_in = len > (uLong)max ? max : (uInt)len;\n+ len -= stream.avail_in;\n+ }\n+ err = inflate(&stream, Z_NO_FLUSH);\n+ } while (err == Z_OK);\n+\n+ *sourceLen -= len + stream.avail_in;\n+ if (dest != buf)\n+ *destLen = stream.total_out;\n+ else if (stream.total_out && err == Z_BUF_ERROR)\n+ left = 1;\n+\n+ inflateEnd(&stream);\n+ return err == Z_STREAM_END ? Z_OK :\n+ err == Z_NEED_DICT ? Z_DATA_ERROR :\n+ err == Z_BUF_ERROR && left + stream.avail_out ? Z_DATA_ERROR :\n+ err;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 9341272089..81a9bd5667 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -3,6 +3,7 @@\n \n int cmd__reftable(int argc, const char **argv)\n {\n+\tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \tstrbuf_test_main(argc, argv);\n \treturn 0;\n-- \ngitgitgadget\n\n"},{"id":"405665","messageId":"570a8c4bcac54295865ee0fc4514073b59725511.1600283416.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.git.git.1600283416.gitgitgadget@gmail.com","subject":"[PATCH 05/13] reftable: utility functions","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-16T19:10:08Z","receivedAt":"2020-09-16T19:11:43Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis commit provides basic utility classes for the reftable library.\n\nSince the reftable library must compile standalone, there may be some overlap\nwith git-core utility functions.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 26 ++++++-\n reftable/basics.c | 131 +++++++++++++++++++++++++++++++++\n reftable/basics.h | 48 +++++++++++++\n reftable/blocksource.c | 148 ++++++++++++++++++++++++++++++++++++++\n reftable/blocksource.h | 22 ++++++\n reftable/compat.c | 110 ++++++++++++++++++++++++++++\n reftable/compat.h | 48 +++++++++++++\n reftable/publicbasics.c | 100 ++++++++++++++++++++++++++\n reftable/reftable-tests.h | 22 ++++++\n reftable/strbuf.c | 142 ++++++++++++++++++++++++++++++++++++\n reftable/strbuf.h | 80 +++++++++++++++++++++\n reftable/strbuf_test.c | 37 ++++++++++\n reftable/system.h | 51 +++++++++++++\n t/helper/test-reftable.c | 8 +++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 16 files changed, 972 insertions(+), 3 deletions(-)\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/compat.c\n create mode 100644 reftable/compat.h\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/strbuf.c\n create mode 100644 reftable/strbuf.h\n create mode 100644 reftable/strbuf_test.c\n create mode 100644 reftable/system.h\n create mode 100644 t/helper/test-reftable.c\n\ndiff --git a/Makefile b/Makefile\nindex 86e5411f39..66158525a9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -720,6 +720,7 @@ TEST_BUILTINS_OBJS += test-read-cache.o\n TEST_BUILTINS_OBJS += test-read-graph.o\n TEST_BUILTINS_OBJS += test-read-midx.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n+TEST_BUILTINS_OBJS += test-reftable.o\n TEST_BUILTINS_OBJS += test-regex.o\n TEST_BUILTINS_OBJS += test-repository.o\n TEST_BUILTINS_OBJS += test-revision-walking.o\n@@ -799,6 +800,8 @@ TEST_SHELL_PATH = $(SHELL_PATH)\n \n LIB_FILE = libgit.a\n XDIFF_LIB = xdiff/lib.a\n+REFTABLE_LIB = reftable/libreftable.a\n+REFTABLE_TEST_LIB = reftable/libreftable_test.a\n \n GENERATED_H += config-list.h\n GENERATED_H += command-list.h\n@@ -1160,7 +1163,7 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n-GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB)\n+GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB)\n EXTLIBS =\n \n GIT_USER_AGENT = git/$(GIT_VERSION)\n@@ -2341,10 +2344,21 @@ XDIFF_OBJS += xdiff/xpatience.o\n XDIFF_OBJS += xdiff/xprepare.o\n XDIFF_OBJS += xdiff/xutils.o\n \n+REFTABLE_OBJS += reftable/basics.o\n+REFTABLE_OBJS += reftable/blocksource.o\n+REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/compat.o\n+REFTABLE_OBJS += reftable/strbuf.o\n+\n+REFTABLE_TEST_OBJS += reftable/strbuf_test.o\n+REFTABLE_TEST_OBJS += reftable/test_framework.o\n+\n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \\\n \t$(XDIFF_OBJS) \\\n \t$(FUZZ_OBJS) \\\n+\t$(REFTABLE_OBJS) \\\n+\t$(REFTABLE_TEST_OBJS) \\\n \tcommon-main.o \\\n \tgit.o\n ifndef NO_CURL\n@@ -2474,6 +2488,12 @@ $(LIB_FILE): $(LIB_OBJS)\n $(XDIFF_LIB): $(XDIFF_OBJS)\n \t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n \n+$(REFTABLE_LIB): $(REFTABLE_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n+$(REFTABLE_TEST_LIB): $(REFTABLE_TEST_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n export DEFAULT_EDITOR DEFAULT_PAGER\n \n Documentation/GIT-EXCLUDED-PROGRAMS: FORCE\n@@ -2752,7 +2772,7 @@ perf: all\n \n t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n \n-t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS)\n+t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) $(REFTABLE_TEST_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n \n check-sha1:: t/helper/test-tool$X\n@@ -3080,7 +3100,7 @@ cocciclean:\n clean: profile-clean coverage-clean cocciclean\n \t$(RM) *.res\n \t$(RM) $(OBJECTS)\n-\t$(RM) $(LIB_FILE) $(XDIFF_LIB)\n+\t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) git$X\n \t$(RM) $(TEST_PROGRAMS)\n \t$(RM) $(FUZZ_PROGRAMS)\ndiff --git a/reftable/basics.c b/reftable/basics.c\nnew file mode 100644\nindex 0000000000..c429055d15\n--- /dev/null\n+++ b/reftable/basics.c\n@@ -0,0 +1,131 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+\n+void put_be24(uint8_t *out, uint32_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 16) & 0xff);\n+\tout[1] = (uint8_t)((i >> 8) & 0xff);\n+\tout[2] = (uint8_t)(i & 0xff);\n+}\n+\n+uint32_t get_be24(uint8_t *in)\n+{\n+\treturn (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 |\n+\t (uint32_t)(in[2]);\n+}\n+\n+void put_be16(uint8_t *out, uint16_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 8) & 0xff);\n+\tout[1] = (uint8_t)(i & 0xff);\n+}\n+\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n+{\n+\tsize_t lo = 0;\n+\tsize_t hi = sz;\n+\n+\t/* invariant: (hi == sz) || f(hi) == true\n+\t (lo == 0 && f(0) == true) || fi(lo) == false\n+\t */\n+\twhile (hi - lo > 1) {\n+\t\tsize_t mid = lo + (hi - lo) / 2;\n+\n+\t\tint val = f(mid, args);\n+\t\tif (val) {\n+\t\t\thi = mid;\n+\t\t} else {\n+\t\t\tlo = mid;\n+\t\t}\n+\t}\n+\n+\tif (lo == 0) {\n+\t\tif (f(0, args)) {\n+\t\t\treturn 0;\n+\t\t} else {\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\treturn hi;\n+}\n+\n+void free_names(char **a)\n+{\n+\tchar **p = a;\n+\tif (p == NULL) {\n+\t\treturn;\n+\t}\n+\twhile (*p) {\n+\t\treftable_free(*p);\n+\t\tp++;\n+\t}\n+\treftable_free(a);\n+}\n+\n+int names_length(char **names)\n+{\n+\tint len = 0;\n+\tchar **p = names;\n+\twhile (*p) {\n+\t\tp++;\n+\t\tlen++;\n+\t}\n+\treturn len;\n+}\n+\n+void parse_names(char *buf, int size, char ***namesp)\n+{\n+\tchar **names = NULL;\n+\tsize_t names_cap = 0;\n+\tsize_t names_len = 0;\n+\n+\tchar *p = buf;\n+\tchar *end = buf + size;\n+\twhile (p < end) {\n+\t\tchar *next = strchr(p, '\\n');\n+\t\tif (next != NULL) {\n+\t\t\t*next = 0;\n+\t\t} else {\n+\t\t\tnext = end;\n+\t\t}\n+\t\tif (p < next) {\n+\t\t\tif (names_len == names_cap) {\n+\t\t\t\tnames_cap = 2 * names_cap + 1;\n+\t\t\t\tnames = reftable_realloc(\n+\t\t\t\t\tnames, names_cap * sizeof(char *));\n+\t\t\t}\n+\t\t\tnames[names_len++] = xstrdup(p);\n+\t\t}\n+\t\tp = next + 1;\n+\t}\n+\n+\tif (names_len == names_cap) {\n+\t\tnames_cap = 2 * names_cap + 1;\n+\t\tnames = reftable_realloc(names, names_cap * sizeof(char *));\n+\t}\n+\n+\tnames[names_len] = NULL;\n+\t*namesp = names;\n+}\n+\n+int names_equal(char **a, char **b)\n+{\n+\twhile (*a && *b) {\n+\t\tif (strcmp(*a, *b)) {\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\ta++;\n+\t\tb++;\n+\t}\n+\n+\treturn *a == *b;\n+}\ndiff --git a/reftable/basics.h b/reftable/basics.h\nnew file mode 100644\nindex 0000000000..90639865a7\n--- /dev/null\n+++ b/reftable/basics.h\n@@ -0,0 +1,48 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BASICS_H\n+#define BASICS_H\n+\n+#include \"system.h\"\n+\n+/* Bigendian en/decoding of integers */\n+\n+void put_be24(uint8_t *out, uint32_t i);\n+uint32_t get_be24(uint8_t *in);\n+void put_be16(uint8_t *out, uint16_t i);\n+\n+/*\n+ find smallest index i in [0, sz) at which f(i) is true, assuming\n+ that f is ascending. Return sz if f(i) is false for all indices.\n+*/\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args);\n+\n+/*\n+ Frees a NULL terminated array of malloced strings. The array itself is also\n+ freed.\n+ */\n+void free_names(char **a);\n+\n+/* parse a newline separated list of names. Empty names are discarded. */\n+void parse_names(char *buf, int size, char ***namesp);\n+\n+/* compares two NULL-terminated arrays of strings. */\n+int names_equal(char **a, char **b);\n+\n+/* returns the array size of a NULL-terminated array of strings. */\n+int names_length(char **names);\n+\n+/* Allocation routines; they invoke the functions set through\n+ * reftable_set_alloc() */\n+void *reftable_malloc(size_t sz);\n+void *reftable_realloc(void *p, size_t sz);\n+void reftable_free(void *p);\n+void *reftable_calloc(size_t sz);\n+\n+#endif\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nnew file mode 100644\nindex 0000000000..f12cea2472\n--- /dev/null\n+++ b/reftable/blocksource.c\n@@ -0,0 +1,148 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"strbuf.h\"\n+#include \"reftable.h\"\n+\n+static void strbuf_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void strbuf_close(void *b)\n+{\n+}\n+\n+static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct strbuf *b = (struct strbuf *)v;\n+\tassert(off + size <= b->len);\n+\tdest->data = reftable_calloc(size);\n+\tmemcpy(dest->data, b->buf + off, size);\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static uint64_t strbuf_size(void *b)\n+{\n+\treturn ((struct strbuf *)b)->len;\n+}\n+\n+struct reftable_block_source_vtable strbuf_vtable = {\n+\t.size = &strbuf_size,\n+\t.read_block = &strbuf_read_block,\n+\t.return_block = &strbuf_return_block,\n+\t.close = &strbuf_close,\n+};\n+\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf)\n+{\n+\tassert(bs->ops == NULL);\n+\tbs->ops = &strbuf_vtable;\n+\tbs->arg = buf;\n+}\n+\n+static void malloc_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+struct reftable_block_source_vtable malloc_vtable = {\n+\t.return_block = &malloc_return_block,\n+};\n+\n+struct reftable_block_source malloc_block_source_instance = {\n+\t.ops = &malloc_vtable,\n+};\n+\n+struct reftable_block_source malloc_block_source(void)\n+{\n+\treturn malloc_block_source_instance;\n+}\n+\n+struct file_block_source {\n+\tint fd;\n+\tuint64_t size;\n+};\n+\n+static uint64_t file_size(void *b)\n+{\n+\treturn ((struct file_block_source *)b)->size;\n+}\n+\n+static void file_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void file_close(void *b)\n+{\n+\tint fd = ((struct file_block_source *)b)->fd;\n+\tif (fd > 0) {\n+\t\tclose(fd);\n+\t\t((struct file_block_source *)b)->fd = 0;\n+\t}\n+\n+\treftable_free(b);\n+}\n+\n+static int file_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct file_block_source *b = (struct file_block_source *)v;\n+\tassert(off + size <= b->size);\n+\tdest->data = reftable_malloc(size);\n+\tif (pread(b->fd, dest->data, size, off) != size)\n+\t\treturn -1;\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+struct reftable_block_source_vtable file_vtable = {\n+\t.size = &file_size,\n+\t.read_block = &file_read_block,\n+\t.return_block = &file_return_block,\n+\t.close = &file_close,\n+};\n+\n+int reftable_block_source_from_file(struct reftable_block_source *bs,\n+\t\t\t\t const char *name)\n+{\n+\tstruct stat st = { 0 };\n+\tint err = 0;\n+\tint fd = open(name, O_RDONLY);\n+\tstruct file_block_source *p = NULL;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\treturn REFTABLE_NOT_EXIST_ERROR;\n+\t\t}\n+\t\treturn -1;\n+\t}\n+\n+\terr = fstat(fd, &st);\n+\tif (err < 0)\n+\t\treturn -1;\n+\n+\tp = reftable_calloc(sizeof(struct file_block_source));\n+\tp->size = st.st_size;\n+\tp->fd = fd;\n+\n+\tassert(bs->ops == NULL);\n+\tbs->ops = &file_vtable;\n+\tbs->arg = p;\n+\treturn 0;\n+}\ndiff --git a/reftable/blocksource.h b/reftable/blocksource.h\nnew file mode 100644\nindex 0000000000..3faf83fa9d\n--- /dev/null\n+++ b/reftable/blocksource.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCKSOURCE_H\n+#define BLOCKSOURCE_H\n+\n+#include \"strbuf.h\"\n+\n+struct reftable_block_source;\n+\n+/* Create an in-memory block source for reading reftables */\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf);\n+\n+struct reftable_block_source malloc_block_source(void);\n+\n+#endif\ndiff --git a/reftable/compat.c b/reftable/compat.c\nnew file mode 100644\nindex 0000000000..a48c5aa5e3\n--- /dev/null\n+++ b/reftable/compat.c\n@@ -0,0 +1,110 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+\n+*/\n+\n+/* compat.c - compatibility functions for standalone compilation */\n+\n+#include \"system.h\"\n+#include \"basics.h\"\n+\n+#ifdef REFTABLE_STANDALONE\n+\n+#include \n+\n+void put_be32(void *p, uint32_t i)\n+{\n+\tuint8_t *out = (uint8_t *)p;\n+\n+\tout[0] = (uint8_t)((i >> 24) & 0xff);\n+\tout[1] = (uint8_t)((i >> 16) & 0xff);\n+\tout[2] = (uint8_t)((i >> 8) & 0xff);\n+\tout[3] = (uint8_t)((i)&0xff);\n+}\n+\n+uint32_t get_be32(uint8_t *in)\n+{\n+\treturn (uint32_t)(in[0]) << 24 | (uint32_t)(in[1]) << 16 |\n+\t (uint32_t)(in[2]) << 8 | (uint32_t)(in[3]);\n+}\n+\n+void put_be64(void *p, uint64_t v)\n+{\n+\tuint8_t *out = (uint8_t *)p;\n+\tint i = sizeof(uint64_t);\n+\twhile (i--) {\n+\t\tout[i] = (uint8_t)(v & 0xff);\n+\t\tv >>= 8;\n+\t}\n+}\n+\n+uint64_t get_be64(void *out)\n+{\n+\tuint8_t *bytes = (uint8_t *)out;\n+\tuint64_t v = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < sizeof(uint64_t); i++) {\n+\t\tv = (v << 8) | (uint8_t)(bytes[i] & 0xff);\n+\t}\n+\treturn v;\n+}\n+\n+uint16_t get_be16(uint8_t *in)\n+{\n+\treturn (uint32_t)(in[0]) << 8 | (uint32_t)(in[1]);\n+}\n+\n+char *xstrdup(const char *s)\n+{\n+\tint l = strlen(s);\n+\tchar *dest = (char *)reftable_malloc(l + 1);\n+\tstrncpy(dest, s, l + 1);\n+\treturn dest;\n+}\n+\n+void sleep_millisec(int millisecs)\n+{\n+\tusleep(millisecs * 1000);\n+}\n+\n+void reftable_clear_dir(const char *dirname)\n+{\n+\tDIR *dir = opendir(dirname);\n+\tstruct dirent *ent = NULL;\n+\tassert(dir);\n+\twhile ((ent = readdir(dir)) != NULL) {\n+\t\tunlinkat(dirfd(dir), ent->d_name, 0);\n+\t}\n+\tclosedir(dir);\n+\trmdir(dirname);\n+}\n+\n+#else\n+\n+#include \"../dir.h\"\n+\n+void reftable_clear_dir(const char *dirname)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstrbuf_addstr(&path, dirname);\n+\tremove_dir_recursively(&path, 0);\n+\tstrbuf_release(&path);\n+}\n+\n+#endif\n+\n+int hash_size(uint32_t id)\n+{\n+\tswitch (id) {\n+\tcase 0:\n+\tcase SHA1_ID:\n+\t\treturn SHA1_SIZE;\n+\tcase SHA256_ID:\n+\t\treturn SHA256_SIZE;\n+\t}\n+\tabort();\n+}\ndiff --git a/reftable/compat.h b/reftable/compat.h\nnew file mode 100644\nindex 0000000000..a765c57e96\n--- /dev/null\n+++ b/reftable/compat.h\n@@ -0,0 +1,48 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef COMPAT_H\n+#define COMPAT_H\n+\n+#include \"system.h\"\n+\n+#ifdef REFTABLE_STANDALONE\n+\n+/* functions that git-core provides, for standalone compilation */\n+#include \n+\n+uint64_t get_be64(void *in);\n+void put_be64(void *out, uint64_t i);\n+\n+void put_be32(void *out, uint32_t i);\n+uint32_t get_be32(uint8_t *in);\n+\n+uint16_t get_be16(uint8_t *in);\n+\n+#define ARRAY_SIZE(a) sizeof((a)) / sizeof((a)[0])\n+#define FREE_AND_NULL(x) \\\n+\tdo { \\\n+\t\treftable_free(x); \\\n+\t\t(x) = NULL; \\\n+\t} while (0)\n+#define QSORT(arr, n, cmp) qsort(arr, n, sizeof(arr[0]), cmp)\n+#define SWAP(a, b) \\\n+\t{ \\\n+\t\tchar tmp[sizeof(a)]; \\\n+\t\tassert(sizeof(a) == sizeof(b)); \\\n+\t\tmemcpy(&tmp[0], &a, sizeof(a)); \\\n+\t\tmemcpy(&a, &b, sizeof(a)); \\\n+\t\tmemcpy(&b, &tmp[0], sizeof(a)); \\\n+\t}\n+\n+char *xstrdup(const char *s);\n+\n+void sleep_millisec(int millisecs);\n+\n+#endif\n+#endif\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\nnew file mode 100644\nindex 0000000000..a31463ff9a\n--- /dev/null\n+++ b/reftable/publicbasics.c\n@@ -0,0 +1,100 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+const char *reftable_error_str(int err)\n+{\n+\tstatic char buf[250];\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn \"I/O error\";\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn \"corrupt reftable file\";\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn \"file does not exist\";\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn \"data is outdated\";\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn \"misuse of the reftable API\";\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn \"zlib failure\";\n+\tcase REFTABLE_NAME_CONFLICT:\n+\t\treturn \"file/directory conflict\";\n+\tcase REFTABLE_REFNAME_ERROR:\n+\t\treturn \"invalid refname\";\n+\tcase -1:\n+\t\treturn \"general error\";\n+\tdefault:\n+\t\tsnprintf(buf, sizeof(buf), \"unknown error code %d\", err);\n+\t\treturn buf;\n+\t}\n+}\n+\n+int reftable_error_to_errno(int err)\n+{\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn EIO;\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn EFAULT;\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn ENOENT;\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn EBUSY;\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn EINVAL;\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn EDOM;\n+\tdefault:\n+\t\treturn ERANGE;\n+\t}\n+}\n+\n+void *(*reftable_malloc_ptr)(size_t sz) = &malloc;\n+void *(*reftable_realloc_ptr)(void *, size_t) = &realloc;\n+void (*reftable_free_ptr)(void *) = &free;\n+\n+void *reftable_malloc(size_t sz)\n+{\n+\treturn (*reftable_malloc_ptr)(sz);\n+}\n+\n+void *reftable_realloc(void *p, size_t sz)\n+{\n+\treturn (*reftable_realloc_ptr)(p, sz);\n+}\n+\n+void reftable_free(void *p)\n+{\n+\treftable_free_ptr(p);\n+}\n+\n+void *reftable_calloc(size_t sz)\n+{\n+\tvoid *p = reftable_malloc(sz);\n+\tmemset(p, 0, sz);\n+\treturn p;\n+}\n+\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n+{\n+\treftable_malloc_ptr = malloc;\n+\treftable_realloc_ptr = realloc;\n+\treftable_free_ptr = free;\n+}\n+\n+int reftable_fd_write(void *arg, const void *data, size_t sz)\n+{\n+\tint *fdp = (int *)arg;\n+\treturn write(*fdp, data, sz);\n+}\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nnew file mode 100644\nindex 0000000000..e38471888f\n--- /dev/null\n+++ b/reftable/reftable-tests.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_TESTS_H\n+#define REFTABLE_TESTS_H\n+\n+int block_test_main(int argc, const char **argv);\n+int merged_test_main(int argc, const char **argv);\n+int record_test_main(int argc, const char **argv);\n+int refname_test_main(int argc, const char **argv);\n+int reftable_test_main(int argc, const char **argv);\n+int strbuf_test_main(int argc, const char **argv);\n+int stack_test_main(int argc, const char **argv);\n+int tree_test_main(int argc, const char **argv);\n+int reftable_dump_main(int argc, char *const *argv);\n+\n+#endif\ndiff --git a/reftable/strbuf.c b/reftable/strbuf.c\nnew file mode 100644\nindex 0000000000..136bf65591\n--- /dev/null\n+++ b/reftable/strbuf.c\n@@ -0,0 +1,142 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"strbuf.h\"\n+\n+#ifdef REFTABLE_STANDALONE\n+\n+void strbuf_init(struct strbuf *s, size_t alloc)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\t*s = empty;\n+}\n+\n+void strbuf_grow(struct strbuf *s, size_t extra)\n+{\n+\tsize_t newcap = s->len + extra + 1;\n+\tif (newcap > s->cap) {\n+\t\ts->buf = reftable_realloc(s->buf, newcap);\n+\t\ts->cap = newcap;\n+\t}\n+}\n+\n+static void strbuf_resize(struct strbuf *s, int l)\n+{\n+\tint zl = l + 1; /* one uint8_t for 0 termination. */\n+\tassert(s->canary == STRBUF_CANARY);\n+\tif (s->cap < zl) {\n+\t\tint c = s->cap * 2;\n+\t\tif (c < zl) {\n+\t\t\tc = zl;\n+\t\t}\n+\t\ts->cap = c;\n+\t\ts->buf = reftable_realloc(s->buf, s->cap);\n+\t}\n+\ts->len = l;\n+\ts->buf[l] = 0;\n+}\n+\n+void strbuf_setlen(struct strbuf *s, size_t l)\n+{\n+\tassert(s->cap >= l + 1);\n+\ts->len = l;\n+\ts->buf[l] = 0;\n+}\n+\n+void strbuf_reset(struct strbuf *s)\n+{\n+\tstrbuf_resize(s, 0);\n+}\n+\n+void strbuf_addstr(struct strbuf *d, const char *s)\n+{\n+\tint l1 = d->len;\n+\tint l2 = strlen(s);\n+\tassert(d->canary == STRBUF_CANARY);\n+\n+\tstrbuf_resize(d, l2 + l1);\n+\tmemcpy(d->buf + l1, s, l2);\n+}\n+\n+void strbuf_addbuf(struct strbuf *s, struct strbuf *a)\n+{\n+\tint end = s->len;\n+\tassert(s->canary == STRBUF_CANARY);\n+\tstrbuf_resize(s, s->len + a->len);\n+\tmemcpy(s->buf + end, a->buf, a->len);\n+}\n+\n+char *strbuf_detach(struct strbuf *s, size_t *sz)\n+{\n+\tchar *p = NULL;\n+\tp = (char *)s->buf;\n+\tif (sz)\n+\t\t*sz = s->len;\n+\ts->buf = NULL;\n+\ts->cap = 0;\n+\ts->len = 0;\n+\treturn p;\n+}\n+\n+void strbuf_release(struct strbuf *s)\n+{\n+\tassert(s->canary == STRBUF_CANARY);\n+\ts->cap = 0;\n+\ts->len = 0;\n+\treftable_free(s->buf);\n+\ts->buf = NULL;\n+}\n+\n+int strbuf_cmp(const struct strbuf *a, const struct strbuf *b)\n+{\n+\tint min = a->len < b->len ? a->len : b->len;\n+\tint res = memcmp(a->buf, b->buf, min);\n+\tassert(a->canary == STRBUF_CANARY);\n+\tassert(b->canary == STRBUF_CANARY);\n+\tif (res != 0)\n+\t\treturn res;\n+\tif (a->len < b->len)\n+\t\treturn -1;\n+\telse if (a->len > b->len)\n+\t\treturn 1;\n+\telse\n+\t\treturn 0;\n+}\n+\n+int strbuf_add(struct strbuf *b, const void *data, size_t sz)\n+{\n+\tassert(b->canary == STRBUF_CANARY);\n+\tstrbuf_grow(b, sz);\n+\tmemcpy(b->buf + b->len, data, sz);\n+\tb->len += sz;\n+\tb->buf[b->len] = 0;\n+\treturn sz;\n+}\n+\n+#endif\n+\n+int strbuf_add_void(void *b, const void *data, size_t sz)\n+{\n+\tstrbuf_add((struct strbuf *)b, data, sz);\n+\treturn sz;\n+}\n+\n+int common_prefix_size(struct strbuf *a, struct strbuf *b)\n+{\n+\tint p = 0;\n+\twhile (p < a->len && p < b->len) {\n+\t\tif (a->buf[p] != b->buf[p]) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tp++;\n+\t}\n+\n+\treturn p;\n+}\n+\n+struct strbuf reftable_empty_strbuf = STRBUF_INIT;\ndiff --git a/reftable/strbuf.h b/reftable/strbuf.h\nnew file mode 100644\nindex 0000000000..c2d7aca8dd\n--- /dev/null\n+++ b/reftable/strbuf.h\n@@ -0,0 +1,80 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef SLICE_H\n+#define SLICE_H\n+\n+#ifdef REFTABLE_STANDALONE\n+\n+#include \"basics.h\"\n+\n+/*\n+ Provides a bounds-checked, growable byte ranges. To use, initialize as \"strbuf\n+ x = STRBUF_INIT;\"\n+ */\n+struct strbuf {\n+\tsize_t len;\n+\tsize_t cap;\n+\tchar *buf;\n+\n+\t/* Used to enforce initialization with STRBUF_INIT */\n+\tuint8_t canary;\n+};\n+\n+#define STRBUF_CANARY 0x42\n+#define STRBUF_INIT \\\n+\t{ \\\n+\t\t0, 0, NULL, STRBUF_CANARY \\\n+\t}\n+\n+void strbuf_addstr(struct strbuf *dest, const char *src);\n+\n+/* Deallocate and clear strbuf */\n+void strbuf_release(struct strbuf *strbuf);\n+\n+/* Set strbuf to 0 length, but retain buffer. */\n+void strbuf_reset(struct strbuf *strbuf);\n+\n+/* Initializes a strbuf. Accepts a strbuf with random garbage. */\n+void strbuf_init(struct strbuf *strbuf, size_t alloc);\n+\n+/* Return `buf`, clearing out `s`. Optionally return len (not cap) in `sz`. */\n+char *strbuf_detach(struct strbuf *s, size_t *sz);\n+\n+/* Set length of the slace to `l`, but don't reallocated. */\n+void strbuf_setlen(struct strbuf *s, size_t l);\n+\n+/* Ensure `l` bytes beyond current length are available */\n+void strbuf_grow(struct strbuf *s, size_t l);\n+\n+/* Signed comparison */\n+int strbuf_cmp(const struct strbuf *a, const struct strbuf *b);\n+\n+/* Append `data` to the `dest` strbuf. */\n+int strbuf_add(struct strbuf *dest, const void *data, size_t sz);\n+\n+/* Append `add` to `dest. */\n+void strbuf_addbuf(struct strbuf *dest, struct strbuf *add);\n+\n+#else\n+\n+#include \"../git-compat-util.h\"\n+#include \"../strbuf.h\"\n+\n+#endif\n+\n+extern struct strbuf reftable_empty_strbuf;\n+\n+/* Like strbuf_add, but suitable for passing to reftable_new_writer\n+ */\n+int strbuf_add_void(void *b, const void *data, size_t sz);\n+\n+/* Find the longest shared prefix size of `a` and `b` */\n+int common_prefix_size(struct strbuf *a, struct strbuf *b);\n+\n+#endif\ndiff --git a/reftable/strbuf_test.c b/reftable/strbuf_test.c\nnew file mode 100644\nindex 0000000000..39f561c81a\n--- /dev/null\n+++ b/reftable/strbuf_test.c\n@@ -0,0 +1,37 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"strbuf.h\"\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_strbuf(void)\n+{\n+\tstruct strbuf s = STRBUF_INIT;\n+\tstruct strbuf t = STRBUF_INIT;\n+\n+\tstrbuf_addstr(&s, \"abc\");\n+\tassert(0 == strcmp(\"abc\", s.buf));\n+\n+\tstrbuf_addstr(&t, \"pqr\");\n+\tstrbuf_addbuf(&s, &t);\n+\tassert(0 == strcmp(\"abcpqr\", s.buf));\n+\n+\tstrbuf_release(&s);\n+\tstrbuf_release(&t);\n+}\n+\n+int strbuf_test_main(int argc, const char *argv[])\n+{\n+\tadd_test_case(\"test_strbuf\", &test_strbuf);\n+\treturn test_main(argc, argv);\n+}\ndiff --git a/reftable/system.h b/reftable/system.h\nnew file mode 100644\nindex 0000000000..567eb8a87d\n--- /dev/null\n+++ b/reftable/system.h\n@@ -0,0 +1,51 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef SYSTEM_H\n+#define SYSTEM_H\n+\n+#ifndef REFTABLE_STANDALONE\n+\n+#include \"git-compat-util.h\"\n+#include \"cache.h\"\n+#include \n+\n+#else\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+\n+#include \"compat.h\"\n+\n+#endif /* REFTABLE_STANDALONE */\n+\n+void reftable_clear_dir(const char *dirname);\n+\n+#define SHA1_ID 0x73686131\n+#define SHA256_ID 0x73323536\n+#define SHA1_SIZE 20\n+#define SHA256_SIZE 32\n+\n+/* This is uncompress2, which is only available in zlib as of 2017.\n+ */\n+int uncompress_return_consumed(Bytef *dest, uLongf *destLen,\n+\t\t\t const Bytef *source, uLong *sourceLen);\n+int hash_size(uint32_t id);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nnew file mode 100644\nindex 0000000000..7d50aa6bcc\n--- /dev/null\n+++ b/t/helper/test-reftable.c\n@@ -0,0 +1,8 @@\n+#include \"reftable/reftable-tests.h\"\n+#include \"test-tool.h\"\n+\n+int cmd__reftable(int argc, const char **argv)\n+{\n+\tstrbuf_test_main(argc, argv);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 590b2efca7..10366b7b76 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -52,6 +52,7 @@ static struct test_cmd cmds[] = {\n \t{ \"read-graph\", cmd__read_graph },\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n+\t{ \"reftable\", cmd__reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex ddc8e990e9..d52ba2f5e5 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -41,6 +41,7 @@ int cmd__read_cache(int argc, const char **argv);\n int cmd__read_graph(int argc, const char **argv);\n int cmd__read_midx(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n+int cmd__reftable(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__repository(int argc, const char **argv);\n int cmd__revision_walking(int argc, const char **argv);\n-- \ngitgitgadget\n\n"},{"id":"405666","messageId":"791f69c000556e93bf5fcfc0ec9304833b12565b.1600283416.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.git.git.1600283416.gitgitgadget@gmail.com","subject":"[PATCH 06/13] reftable: (de)serialization for the polymorphic record type.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-16T19:10:09Z","receivedAt":"2020-09-16T19:11:45Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/constants.h | 21 +\n reftable/record.c | 1114 ++++++++++++++++++++++++++++++++++++++\n reftable/record.h | 143 +++++\n reftable/record_test.c | 410 ++++++++++++++\n t/helper/test-reftable.c | 1 +\n 6 files changed, 1691 insertions(+)\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 66158525a9..0fb0048c34 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2348,8 +2348,10 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/compat.o\n+REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/strbuf.o\n \n+REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/strbuf_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n \ndiff --git a/reftable/constants.h b/reftable/constants.h\nnew file mode 100644\nindex 0000000000..5eee72c4c1\n--- /dev/null\n+++ b/reftable/constants.h\n@@ -0,0 +1,21 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef CONSTANTS_H\n+#define CONSTANTS_H\n+\n+#define BLOCK_TYPE_LOG 'g'\n+#define BLOCK_TYPE_INDEX 'i'\n+#define BLOCK_TYPE_REF 'r'\n+#define BLOCK_TYPE_OBJ 'o'\n+#define BLOCK_TYPE_ANY 0\n+\n+#define MAX_RESTARTS ((1 << 16) - 1)\n+#define DEFAULT_BLOCK_SIZE 4096\n+\n+#endif\ndiff --git a/reftable/record.c b/reftable/record.c\nnew file mode 100644\nindex 0000000000..21c9bba077\n--- /dev/null\n+++ b/reftable/record.c\n@@ -0,0 +1,1114 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+/* record.c - methods for different types of records. */\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"constants.h\"\n+#include \"reftable.h\"\n+#include \"basics.h\"\n+\n+int get_var_int(uint64_t *dest, struct string_view *in)\n+{\n+\tint ptr = 0;\n+\tuint64_t val;\n+\n+\tif (in->len == 0)\n+\t\treturn -1;\n+\tval = in->buf[ptr] & 0x7f;\n+\n+\twhile (in->buf[ptr] & 0x80) {\n+\t\tptr++;\n+\t\tif (ptr > in->len) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tval = (val + 1) << 7 | (uint64_t)(in->buf[ptr] & 0x7f);\n+\t}\n+\n+\t*dest = val;\n+\treturn ptr + 1;\n+}\n+\n+int put_var_int(struct string_view *dest, uint64_t val)\n+{\n+\tuint8_t buf[10] = { 0 };\n+\tint i = 9;\n+\tint n = 0;\n+\tbuf[i] = (uint8_t)(val & 0x7f);\n+\ti--;\n+\twhile (1) {\n+\t\tval >>= 7;\n+\t\tif (!val) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tval--;\n+\t\tbuf[i] = 0x80 | (uint8_t)(val & 0x7f);\n+\t\ti--;\n+\t}\n+\n+\tn = sizeof(buf) - i - 1;\n+\tif (dest->len < n)\n+\t\treturn -1;\n+\tmemcpy(dest->buf, &buf[i + 1], n);\n+\treturn n;\n+}\n+\n+int reftable_is_block_type(uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\tcase BLOCK_TYPE_LOG:\n+\tcase BLOCK_TYPE_OBJ:\n+\tcase BLOCK_TYPE_INDEX:\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+static int decode_string(struct strbuf *dest, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t tsize = 0;\n+\tint n = get_var_int(&tsize, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\tif (in.len < tsize)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, in.buf, tsize);\n+\tstring_view_consume(&in, tsize);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static int encode_string(char *str, struct string_view s)\n+{\n+\tstruct string_view start = s;\n+\tint l = strlen(str);\n+\tint n = put_var_int(&s, l);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\tif (s.len < l)\n+\t\treturn -1;\n+\tmemcpy(s.buf, str, l);\n+\tstring_view_consume(&s, l);\n+\n+\treturn start.len - s.len;\n+}\n+\n+int reftable_encode_key(int *restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra)\n+{\n+\tstruct string_view start = dest;\n+\tint prefix_len = common_prefix_size(&prev_key, &key);\n+\tuint64_t suffix_len = key.len - prefix_len;\n+\tint n = put_var_int(&dest, (uint64_t)prefix_len);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\t*restart = (prefix_len == 0);\n+\n+\tn = put_var_int(&dest, suffix_len << 3 | (uint64_t)extra);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\tif (dest.len < suffix_len)\n+\t\treturn -1;\n+\tmemcpy(dest.buf, key.buf + prefix_len, suffix_len);\n+\tstring_view_consume(&dest, suffix_len);\n+\n+\treturn start.len - dest.len;\n+}\n+\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t prefix_len = 0;\n+\tuint64_t suffix_len = 0;\n+\tint n = get_var_int(&prefix_len, &in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tif (prefix_len > last_key.len)\n+\t\treturn -1;\n+\n+\tn = get_var_int(&suffix_len, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\t*extra = (uint8_t)(suffix_len & 0x7);\n+\tsuffix_len >>= 3;\n+\n+\tif (in.len < suffix_len)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(key);\n+\tstrbuf_add(key, last_key.buf, prefix_len);\n+\tstrbuf_add(key, in.buf, suffix_len);\n+\tstring_view_consume(&in, suffix_len);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static void reftable_ref_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_ref_record *rec =\n+\t\t(const struct reftable_ref_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, rec->refname);\n+}\n+\n+static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *ref = (struct reftable_ref_record *)rec;\n+\tstruct reftable_ref_record *src = (struct reftable_ref_record *)src_rec;\n+\tassert(hash_size > 0);\n+\n+\t/* This is simple and correct, but we could probably reuse the hash\n+\t fields. */\n+\treftable_ref_record_clear(ref);\n+\tif (src->refname != NULL) {\n+\t\tref->refname = xstrdup(src->refname);\n+\t}\n+\n+\tif (src->target != NULL) {\n+\t\tref->target = xstrdup(src->target);\n+\t}\n+\n+\tif (src->target_value != NULL) {\n+\t\tref->target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->target_value, src->target_value, hash_size);\n+\t}\n+\n+\tif (src->value != NULL) {\n+\t\tref->value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value, src->value, hash_size);\n+\t}\n+\tref->update_index = src->update_index;\n+}\n+\n+static char hexdigit(int c)\n+{\n+\tif (c <= 9)\n+\t\treturn '0' + c;\n+\treturn 'a' + (c - 10);\n+}\n+\n+static void hex_format(char *dest, uint8_t *src, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (src != NULL) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < hash_size; i++) {\n+\t\t\tdest[2 * i] = hexdigit(src[i] >> 4);\n+\t\t\tdest[2 * i + 1] = hexdigit(src[i] & 0xf);\n+\t\t}\n+\t\tdest[2 * hash_size] = 0;\n+\t}\n+}\n+\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[SHA256_SIZE + 1] = { 0 };\n+\tprintf(\"ref{%s(%\" PRIu64 \") \", ref->refname, ref->update_index);\n+\tif (ref->value != NULL) {\n+\t\thex_format(hex, ref->value, hash_size(hash_id));\n+\t\tprintf(\"%s\", hex);\n+\t}\n+\tif (ref->target_value != NULL) {\n+\t\thex_format(hex, ref->target_value, hash_size(hash_id));\n+\t\tprintf(\" (T %s)\", hex);\n+\t}\n+\tif (ref->target != NULL) {\n+\t\tprintf(\"=> %s\", ref->target);\n+\t}\n+\tprintf(\"}\\n\");\n+}\n+\n+static void reftable_ref_record_clear_void(void *rec)\n+{\n+\treftable_ref_record_clear((struct reftable_ref_record *)rec);\n+}\n+\n+void reftable_ref_record_clear(struct reftable_ref_record *ref)\n+{\n+\treftable_free(ref->refname);\n+\treftable_free(ref->target);\n+\treftable_free(ref->target_value);\n+\treftable_free(ref->value);\n+\tmemset(ref, 0, sizeof(struct reftable_ref_record));\n+}\n+\n+static uint8_t reftable_ref_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\tif (r->value != NULL) {\n+\t\tif (r->target_value != NULL) {\n+\t\t\treturn 2;\n+\t\t} else {\n+\t\t\treturn 1;\n+\t\t}\n+\t} else if (r->target != NULL)\n+\t\treturn 3;\n+\treturn 0;\n+}\n+\n+static int reftable_ref_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\tstruct string_view start = s;\n+\tint n = put_var_int(&s, r->update_index);\n+\tassert(hash_size > 0);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tif (r->value != NULL) {\n+\t\tif (s.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t}\n+\n+\tif (r->target_value != NULL) {\n+\t\tif (s.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->target_value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t}\n+\n+\tif (r->target != NULL) {\n+\t\tint n = encode_string(r->target, s);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t}\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_ref_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *r = (struct reftable_ref_record *)rec;\n+\tstruct string_view start = in;\n+\tint seen_value = 0;\n+\tint seen_target_value = 0;\n+\tint seen_target = 0;\n+\n+\tint n = get_var_int(&r->update_index, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tassert(hash_size > 0);\n+\n+\tstring_view_consume(&in, n);\n+\n+\tr->refname = reftable_realloc(r->refname, key.len + 1);\n+\tmemcpy(r->refname, key.buf, key.len);\n+\tr->refname[key.len] = 0;\n+\n+\tswitch (val_type) {\n+\tcase 1:\n+\tcase 2:\n+\t\tif (in.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (r->value == NULL) {\n+\t\t\tr->value = reftable_malloc(hash_size);\n+\t\t}\n+\t\tseen_value = 1;\n+\t\tmemcpy(r->value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tif (val_type == 1) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (r->target_value == NULL) {\n+\t\t\tr->target_value = reftable_malloc(hash_size);\n+\t\t}\n+\t\tseen_target_value = 1;\n+\t\tmemcpy(r->target_value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\tcase 3: {\n+\t\tstruct strbuf dest = STRBUF_INIT;\n+\t\tint n = decode_string(&dest, in);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\t\tseen_target = 1;\n+\t\tif (r->target != NULL) {\n+\t\t\treftable_free(r->target);\n+\t\t}\n+\t\tr->target = dest.buf;\n+\t} break;\n+\n+\tcase 0:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t\tbreak;\n+\t}\n+\n+\tif (!seen_target && r->target != NULL) {\n+\t\tFREE_AND_NULL(r->target);\n+\t}\n+\tif (!seen_target_value && r->target_value != NULL) {\n+\t\tFREE_AND_NULL(r->target_value);\n+\t}\n+\tif (!seen_value && r->value != NULL) {\n+\t\tFREE_AND_NULL(r->value);\n+\t}\n+\n+\treturn start.len - in.len;\n+}\n+\n+static int reftable_ref_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_ref_record_is_deletion(\n+\t\t(const struct reftable_ref_record *)p);\n+}\n+\n+struct reftable_record_vtable reftable_ref_record_vtable = {\n+\t.key = &reftable_ref_record_key,\n+\t.type = BLOCK_TYPE_REF,\n+\t.copy_from = &reftable_ref_record_copy_from,\n+\t.val_type = &reftable_ref_record_val_type,\n+\t.encode = &reftable_ref_record_encode,\n+\t.decode = &reftable_ref_record_decode,\n+\t.clear = &reftable_ref_record_clear_void,\n+\t.is_deletion = &reftable_ref_record_is_deletion_void,\n+};\n+\n+static void reftable_obj_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_obj_record *rec =\n+\t\t(const struct reftable_obj_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, rec->hash_prefix, rec->hash_prefix_len);\n+}\n+\n+static void reftable_obj_record_clear(void *rec)\n+{\n+\tstruct reftable_obj_record *obj = (struct reftable_obj_record *)rec;\n+\tFREE_AND_NULL(obj->hash_prefix);\n+\tFREE_AND_NULL(obj->offsets);\n+\tmemset(obj, 0, sizeof(struct reftable_obj_record));\n+}\n+\n+static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_obj_record *obj = (struct reftable_obj_record *)rec;\n+\tconst struct reftable_obj_record *src =\n+\t\t(const struct reftable_obj_record *)src_rec;\n+\tint olen;\n+\n+\treftable_obj_record_clear(obj);\n+\t*obj = *src;\n+\tobj->hash_prefix = reftable_malloc(obj->hash_prefix_len);\n+\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n+\n+\tolen = obj->offset_len * sizeof(uint64_t);\n+\tobj->offsets = reftable_malloc(olen);\n+\tmemcpy(obj->offsets, src->offsets, olen);\n+}\n+\n+static uint8_t reftable_obj_record_val_type(const void *rec)\n+{\n+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n+\tif (r->offset_len > 0 && r->offset_len < 8)\n+\t\treturn r->offset_len;\n+\treturn 0;\n+}\n+\n+static int reftable_obj_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n+\tstruct string_view start = s;\n+\tint i = 0;\n+\tint n = 0;\n+\tuint64_t last = 0;\n+\tif (r->offset_len == 0 || r->offset_len >= 8) {\n+\t\tn = put_var_int(&s, r->offset_len);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t}\n+\tif (r->offset_len == 0)\n+\t\treturn start.len - s.len;\n+\tn = put_var_int(&s, r->offsets[0]);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tlast = r->offsets[0];\n+\tfor (i = 1; i < r->offset_len; i++) {\n+\t\tint n = put_var_int(&s, r->offsets[i] - last);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tlast = r->offsets[i];\n+\t}\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_obj_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n+\tuint64_t count = val_type;\n+\tint n = 0;\n+\tuint64_t last;\n+\tint j;\n+\tr->hash_prefix = reftable_malloc(key.len);\n+\tmemcpy(r->hash_prefix, key.buf, key.len);\n+\tr->hash_prefix_len = key.len;\n+\n+\tif (val_type == 0) {\n+\t\tn = get_var_int(&count, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\n+\t\tstring_view_consume(&in, n);\n+\t}\n+\n+\tr->offsets = NULL;\n+\tr->offset_len = 0;\n+\tif (count == 0)\n+\t\treturn start.len - in.len;\n+\n+\tr->offsets = reftable_malloc(count * sizeof(uint64_t));\n+\tr->offset_len = count;\n+\n+\tn = get_var_int(&r->offsets[0], &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\tlast = r->offsets[0];\n+\tj = 1;\n+\twhile (j < count) {\n+\t\tuint64_t delta = 0;\n+\t\tint n = get_var_int(&delta, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\n+\t\tlast = r->offsets[j] = (delta + last);\n+\t\tj++;\n+\t}\n+\treturn start.len - in.len;\n+}\n+\n+static int not_a_deletion(const void *p)\n+{\n+\treturn 0;\n+}\n+\n+struct reftable_record_vtable reftable_obj_record_vtable = {\n+\t.key = &reftable_obj_record_key,\n+\t.type = BLOCK_TYPE_OBJ,\n+\t.copy_from = &reftable_obj_record_copy_from,\n+\t.val_type = &reftable_obj_record_val_type,\n+\t.encode = &reftable_obj_record_encode,\n+\t.decode = &reftable_obj_record_decode,\n+\t.clear = &reftable_obj_record_clear,\n+\t.is_deletion = not_a_deletion,\n+};\n+\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[SHA256_SIZE + 1] = { 0 };\n+\n+\tprintf(\"log{%s(%\" PRIu64 \") %s <%s> %\" PRIu64 \" %04d\\n\", log->refname,\n+\t log->update_index, log->name, log->email, log->time,\n+\t log->tz_offset);\n+\thex_format(hex, log->old_hash, hash_size(hash_id));\n+\tprintf(\"%s => \", hex);\n+\thex_format(hex, log->new_hash, hash_size(hash_id));\n+\tprintf(\"%s\\n\\n%s\\n}\\n\", hex, log->message);\n+}\n+\n+static void reftable_log_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_log_record *rec =\n+\t\t(const struct reftable_log_record *)r;\n+\tint len = strlen(rec->refname);\n+\tuint8_t i64[8];\n+\tuint64_t ts = 0;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, (uint8_t *)rec->refname, len + 1);\n+\n+\tts = (~ts) - rec->update_index;\n+\tput_be64(&i64[0], ts);\n+\tstrbuf_add(dest, i64, sizeof(i64));\n+}\n+\n+static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_log_record *dst = (struct reftable_log_record *)rec;\n+\tconst struct reftable_log_record *src =\n+\t\t(const struct reftable_log_record *)src_rec;\n+\n+\treftable_log_record_clear(dst);\n+\t*dst = *src;\n+\tif (dst->refname != NULL) {\n+\t\tdst->refname = xstrdup(dst->refname);\n+\t}\n+\tif (dst->email != NULL) {\n+\t\tdst->email = xstrdup(dst->email);\n+\t}\n+\tif (dst->name != NULL) {\n+\t\tdst->name = xstrdup(dst->name);\n+\t}\n+\tif (dst->message != NULL) {\n+\t\tdst->message = xstrdup(dst->message);\n+\t}\n+\n+\tif (dst->new_hash != NULL) {\n+\t\tdst->new_hash = reftable_malloc(hash_size);\n+\t\tmemcpy(dst->new_hash, src->new_hash, hash_size);\n+\t}\n+\tif (dst->old_hash != NULL) {\n+\t\tdst->old_hash = reftable_malloc(hash_size);\n+\t\tmemcpy(dst->old_hash, src->old_hash, hash_size);\n+\t}\n+}\n+\n+static void reftable_log_record_clear_void(void *rec)\n+{\n+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n+\treftable_log_record_clear(r);\n+}\n+\n+void reftable_log_record_clear(struct reftable_log_record *r)\n+{\n+\treftable_free(r->refname);\n+\treftable_free(r->new_hash);\n+\treftable_free(r->old_hash);\n+\treftable_free(r->name);\n+\treftable_free(r->email);\n+\treftable_free(r->message);\n+\tmemset(r, 0, sizeof(struct reftable_log_record));\n+}\n+\n+static uint8_t reftable_log_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_log_record *log =\n+\t\t(const struct reftable_log_record *)rec;\n+\n+\treturn reftable_log_record_is_deletion(log) ? 0 : 1;\n+}\n+\n+static uint8_t zero[SHA256_SIZE] = { 0 };\n+\n+static int reftable_log_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n+\tstruct string_view start = s;\n+\tint n = 0;\n+\tuint8_t *oldh = r->old_hash;\n+\tuint8_t *newh = r->new_hash;\n+\tif (reftable_log_record_is_deletion(r))\n+\t\treturn 0;\n+\n+\tif (oldh == NULL) {\n+\t\toldh = zero;\n+\t}\n+\tif (newh == NULL) {\n+\t\tnewh = zero;\n+\t}\n+\n+\tif (s.len < 2 * hash_size)\n+\t\treturn -1;\n+\n+\tmemcpy(s.buf, oldh, hash_size);\n+\tmemcpy(s.buf + hash_size, newh, hash_size);\n+\tstring_view_consume(&s, 2 * hash_size);\n+\n+\tn = encode_string(r->name ? r->name : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = encode_string(r->email ? r->email : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = put_var_int(&s, r->time);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tif (s.len < 2)\n+\t\treturn -1;\n+\n+\tput_be16(s.buf, r->tz_offset);\n+\tstring_view_consume(&s, 2);\n+\n+\tn = encode_string(r->message ? r->message : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_log_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n+\tuint64_t max = 0;\n+\tuint64_t ts = 0;\n+\tstruct strbuf dest = STRBUF_INIT;\n+\tint n;\n+\n+\tif (key.len <= 9 || key.buf[key.len - 9] != 0)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->refname = reftable_realloc(r->refname, key.len - 8);\n+\tmemcpy(r->refname, key.buf, key.len - 8);\n+\tts = get_be64(key.buf + key.len - 8);\n+\n+\tr->update_index = (~max) - ts;\n+\n+\tif (val_type == 0) {\n+\t\tFREE_AND_NULL(r->old_hash);\n+\t\tFREE_AND_NULL(r->new_hash);\n+\t\tFREE_AND_NULL(r->message);\n+\t\tFREE_AND_NULL(r->email);\n+\t\tFREE_AND_NULL(r->name);\n+\t\treturn 0;\n+\t}\n+\n+\tif (in.len < 2 * hash_size)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->old_hash = reftable_realloc(r->old_hash, hash_size);\n+\tr->new_hash = reftable_realloc(r->new_hash, hash_size);\n+\n+\tmemcpy(r->old_hash, in.buf, hash_size);\n+\tmemcpy(r->new_hash, in.buf + hash_size, hash_size);\n+\n+\tstring_view_consume(&in, 2 * hash_size);\n+\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->name = reftable_realloc(r->name, dest.len + 1);\n+\tmemcpy(r->name, dest.buf, dest.len);\n+\tr->name[dest.len] = 0;\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->email = reftable_realloc(r->email, dest.len + 1);\n+\tmemcpy(r->email, dest.buf, dest.len);\n+\tr->email[dest.len] = 0;\n+\n+\tts = 0;\n+\tn = get_var_int(&ts, &in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\tr->time = ts;\n+\tif (in.len < 2)\n+\t\tgoto done;\n+\n+\tr->tz_offset = get_be16(in.buf);\n+\tstring_view_consume(&in, 2);\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->message = reftable_realloc(r->message, dest.len + 1);\n+\tmemcpy(r->message, dest.buf, dest.len);\n+\tr->message[dest.len] = 0;\n+\n+\tstrbuf_release(&dest);\n+\treturn start.len - in.len;\n+\n+done:\n+\tstrbuf_release(&dest);\n+\treturn REFTABLE_FORMAT_ERROR;\n+}\n+\n+static int null_streq(char *a, char *b)\n+{\n+\tchar *empty = \"\";\n+\tif (a == NULL)\n+\t\ta = empty;\n+\n+\tif (b == NULL)\n+\t\tb = empty;\n+\n+\treturn 0 == strcmp(a, b);\n+}\n+\n+static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)\n+{\n+\tif (a == NULL)\n+\t\ta = zero;\n+\n+\tif (b == NULL)\n+\t\tb = zero;\n+\n+\treturn !memcmp(a, b, sz);\n+}\n+\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size)\n+{\n+\treturn null_streq(a->name, b->name) && null_streq(a->email, b->email) &&\n+\t null_streq(a->message, b->message) &&\n+\t zero_hash_eq(a->old_hash, b->old_hash, hash_size) &&\n+\t zero_hash_eq(a->new_hash, b->new_hash, hash_size) &&\n+\t a->time == b->time && a->tz_offset == b->tz_offset &&\n+\t a->update_index == b->update_index;\n+}\n+\n+static int reftable_log_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_log_record_is_deletion(\n+\t\t(const struct reftable_log_record *)p);\n+}\n+\n+struct reftable_record_vtable reftable_log_record_vtable = {\n+\t.key = &reftable_log_record_key,\n+\t.type = BLOCK_TYPE_LOG,\n+\t.copy_from = &reftable_log_record_copy_from,\n+\t.val_type = &reftable_log_record_val_type,\n+\t.encode = &reftable_log_record_encode,\n+\t.decode = &reftable_log_record_decode,\n+\t.clear = &reftable_log_record_clear_void,\n+\t.is_deletion = &reftable_log_record_is_deletion_void,\n+};\n+\n+struct reftable_record reftable_new_record(uint8_t typ)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF: {\n+\t\tstruct reftable_ref_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_ref_record));\n+\t\treftable_record_from_ref(&rec, r);\n+\t\treturn rec;\n+\t}\n+\n+\tcase BLOCK_TYPE_OBJ: {\n+\t\tstruct reftable_obj_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_obj_record));\n+\t\treftable_record_from_obj(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_LOG: {\n+\t\tstruct reftable_log_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_log_record));\n+\t\treftable_record_from_log(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_INDEX: {\n+\t\tstruct reftable_index_record empty = { .last_key =\n+\t\t\t\t\t\t\t STRBUF_INIT };\n+\t\tstruct reftable_index_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_index_record));\n+\t\t*r = empty;\n+\t\treftable_record_from_index(&rec, r);\n+\t\treturn rec;\n+\t}\n+\t}\n+\tabort();\n+\treturn rec;\n+}\n+\n+void *reftable_record_yield(struct reftable_record *rec)\n+{\n+\tvoid *p = rec->data;\n+\trec->data = NULL;\n+\treturn p;\n+}\n+\n+void reftable_record_destroy(struct reftable_record *rec)\n+{\n+\treftable_record_clear(rec);\n+\treftable_free(reftable_record_yield(rec));\n+}\n+\n+static void reftable_index_record_key(const void *r, struct strbuf *dest)\n+{\n+\tstruct reftable_index_record *rec = (struct reftable_index_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addbuf(dest, &rec->last_key);\n+}\n+\n+static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_index_record *dst = (struct reftable_index_record *)rec;\n+\tstruct reftable_index_record *src =\n+\t\t(struct reftable_index_record *)src_rec;\n+\n+\tstrbuf_reset(&dst->last_key);\n+\tstrbuf_addbuf(&dst->last_key, &src->last_key);\n+\tdst->offset = src->offset;\n+}\n+\n+static void reftable_index_record_clear(void *rec)\n+{\n+\tstruct reftable_index_record *idx = (struct reftable_index_record *)rec;\n+\tstrbuf_release(&idx->last_key);\n+}\n+\n+static uint8_t reftable_index_record_val_type(const void *rec)\n+{\n+\treturn 0;\n+}\n+\n+static int reftable_index_record_encode(const void *rec, struct string_view out,\n+\t\t\t\t\tint hash_size)\n+{\n+\tconst struct reftable_index_record *r =\n+\t\t(const struct reftable_index_record *)rec;\n+\tstruct string_view start = out;\n+\n+\tint n = put_var_int(&out, r->offset);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&out, n);\n+\n+\treturn start.len - out.len;\n+}\n+\n+static int reftable_index_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t\tuint8_t val_type, struct string_view in,\n+\t\t\t\t\tint hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_index_record *r = (struct reftable_index_record *)rec;\n+\tint n = 0;\n+\n+\tstrbuf_reset(&r->last_key);\n+\tstrbuf_addbuf(&r->last_key, &key);\n+\n+\tn = get_var_int(&r->offset, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&in, n);\n+\treturn start.len - in.len;\n+}\n+\n+struct reftable_record_vtable reftable_index_record_vtable = {\n+\t.key = &reftable_index_record_key,\n+\t.type = BLOCK_TYPE_INDEX,\n+\t.copy_from = &reftable_index_record_copy_from,\n+\t.val_type = &reftable_index_record_val_type,\n+\t.encode = &reftable_index_record_encode,\n+\t.decode = &reftable_index_record_decode,\n+\t.clear = &reftable_index_record_clear,\n+\t.is_deletion = ¬_a_deletion,\n+};\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)\n+{\n+\trec->ops->key(rec->data, dest);\n+}\n+\n+uint8_t reftable_record_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->type;\n+}\n+\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size)\n+{\n+\treturn rec->ops->encode(rec->data, dest, hash_size);\n+}\n+\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size)\n+{\n+\tassert(src->ops->type == rec->ops->type);\n+\n+\trec->ops->copy_from(rec->data, src->data, hash_size);\n+}\n+\n+uint8_t reftable_record_val_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->val_type(rec->data);\n+}\n+\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src, int hash_size)\n+{\n+\treturn rec->ops->decode(rec->data, key, extra, src, hash_size);\n+}\n+\n+void reftable_record_clear(struct reftable_record *rec)\n+{\n+\trec->ops->clear(rec->data);\n+}\n+\n+int reftable_record_is_deletion(struct reftable_record *rec)\n+{\n+\treturn rec->ops->is_deletion(rec->data);\n+}\n+\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *ref_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = ref_rec;\n+\trec->ops = &reftable_ref_record_vtable;\n+}\n+\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *obj_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = obj_rec;\n+\trec->ops = &reftable_obj_record_vtable;\n+}\n+\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *index_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = index_rec;\n+\trec->ops = &reftable_index_record_vtable;\n+}\n+\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *log_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = log_rec;\n+\trec->ops = &reftable_log_record_vtable;\n+}\n+\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_REF);\n+\treturn (struct reftable_ref_record *)rec->data;\n+}\n+\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_LOG);\n+\treturn (struct reftable_log_record *)rec->data;\n+}\n+\n+static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)\n+{\n+\tif (a != NULL && b != NULL)\n+\t\treturn !memcmp(a, b, hash_size);\n+\n+\treturn a == b;\n+}\n+\n+static int str_equal(char *a, char *b)\n+{\n+\tif (a != NULL && b != NULL)\n+\t\treturn 0 == strcmp(a, b);\n+\n+\treturn a == b;\n+}\n+\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\treturn 0 == strcmp(a->refname, b->refname) &&\n+\t a->update_index == b->update_index &&\n+\t hash_equal(a->value, b->value, hash_size) &&\n+\t hash_equal(a->target_value, b->target_value, hash_size) &&\n+\t str_equal(a->target, b->target);\n+}\n+\n+int reftable_ref_record_compare_name(const void *a, const void *b)\n+{\n+\treturn strcmp(((struct reftable_ref_record *)a)->refname,\n+\t\t ((struct reftable_ref_record *)b)->refname);\n+}\n+\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref)\n+{\n+\treturn ref->value == NULL && ref->target == NULL &&\n+\t ref->target_value == NULL;\n+}\n+\n+int reftable_log_record_compare_key(const void *a, const void *b)\n+{\n+\tstruct reftable_log_record *la = (struct reftable_log_record *)a;\n+\tstruct reftable_log_record *lb = (struct reftable_log_record *)b;\n+\n+\tint cmp = strcmp(la->refname, lb->refname);\n+\tif (cmp)\n+\t\treturn cmp;\n+\tif (la->update_index > lb->update_index)\n+\t\treturn -1;\n+\treturn (la->update_index < lb->update_index) ? 1 : 0;\n+}\n+\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log)\n+{\n+\treturn (log->new_hash == NULL && log->old_hash == NULL &&\n+\t\tlog->name == NULL && log->email == NULL &&\n+\t\tlog->message == NULL && log->time == 0 && log->tz_offset == 0 &&\n+\t\tlog->message == NULL);\n+}\n+\n+void string_view_consume(struct string_view *s, int n)\n+{\n+\ts->buf += n;\n+\ts->len -= n;\n+}\ndiff --git a/reftable/record.h b/reftable/record.h\nnew file mode 100644\nindex 0000000000..0c4725cd3c\n--- /dev/null\n+++ b/reftable/record.h\n@@ -0,0 +1,143 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef RECORD_H\n+#define RECORD_H\n+\n+#include \"reftable.h\"\n+#include \"strbuf.h\"\n+#include \"system.h\"\n+\n+/*\n+ A substring of existing string data. This structure takes no responsibility\n+ for the lifetime of the data it points to.\n+*/\n+struct string_view {\n+\tuint8_t *buf;\n+\tsize_t len;\n+};\n+\n+/* Advance `s.buf` by `n`, and decrease length. */\n+void string_view_consume(struct string_view *s, int n);\n+\n+/* utilities for de/encoding varints */\n+\n+int get_var_int(uint64_t *dest, struct string_view *in);\n+int put_var_int(struct string_view *dest, uint64_t val);\n+\n+/* Methods for records. */\n+struct reftable_record_vtable {\n+\t/* encode the key of to a uint8_t strbuf. */\n+\tvoid (*key)(const void *rec, struct strbuf *dest);\n+\n+\t/* The record type of ('r' for ref). */\n+\tuint8_t type;\n+\n+\tvoid (*copy_from)(void *dest, const void *src, int hash_size);\n+\n+\t/* a value of [0..7], indicating record subvariants (eg. ref vs. symref\n+\t * vs ref deletion) */\n+\tuint8_t (*val_type)(const void *rec);\n+\n+\t/* encodes rec into dest, returning how much space was used. */\n+\tint (*encode)(const void *rec, struct string_view dest, int hash_size);\n+\n+\t/* decode data from `src` into the record. */\n+\tint (*decode)(void *rec, struct strbuf key, uint8_t extra,\n+\t\t struct string_view src, int hash_size);\n+\n+\t/* deallocate and null the record. */\n+\tvoid (*clear)(void *rec);\n+\n+\t/* is this a tombstone? */\n+\tint (*is_deletion)(const void *rec);\n+};\n+\n+/* record is a generic wrapper for different types of records. */\n+struct reftable_record {\n+\tvoid *data;\n+\tstruct reftable_record_vtable *ops;\n+};\n+\n+/* returns true for recognized block types. Block start with the block type. */\n+int reftable_is_block_type(uint8_t typ);\n+\n+/* creates a malloced record of the given type. Dispose with record_destroy */\n+struct reftable_record reftable_new_record(uint8_t typ);\n+\n+extern struct reftable_record_vtable reftable_ref_record_vtable;\n+\n+/* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns\n+ number of bytes written. */\n+int reftable_encode_key(int *is_restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra);\n+\n+/* Decode into `key` and `extra` from `in` */\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in);\n+\n+/* reftable_index_record are used internally to speed up lookups. */\n+struct reftable_index_record {\n+\tuint64_t offset; /* Offset of block */\n+\tstruct strbuf last_key; /* Last key of the block. */\n+};\n+\n+/* reftable_obj_record stores an object ID => ref mapping. */\n+struct reftable_obj_record {\n+\tuint8_t *hash_prefix; /* leading bytes of the object ID */\n+\tint hash_prefix_len; /* number of leading bytes. Constant\n+\t\t\t * across a single table. */\n+\tuint64_t *offsets; /* a vector of file offsets. */\n+\tint offset_len;\n+};\n+\n+/* see struct record_vtable */\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);\n+uint8_t reftable_record_type(struct reftable_record *rec);\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size);\n+uint8_t reftable_record_val_type(struct reftable_record *rec);\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size);\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src,\n+\t\t\t int hash_size);\n+int reftable_record_is_deletion(struct reftable_record *rec);\n+\n+/* zeroes out the embedded record */\n+void reftable_record_clear(struct reftable_record *rec);\n+\n+/* clear out the record, yielding the reftable_record data that was\n+ * encapsulated. */\n+void *reftable_record_yield(struct reftable_record *rec);\n+\n+/* clear and deallocate embedded record, and zero `rec`. */\n+void reftable_record_destroy(struct reftable_record *rec);\n+\n+/* initialize generic records from concrete records. The generic record should\n+ * be zeroed out. */\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *objrec);\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *idxrec);\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *refrec);\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *logrec);\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *ref);\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *ref);\n+\n+/* for qsort. */\n+int reftable_ref_record_compare_name(const void *a, const void *b);\n+\n+/* for qsort. */\n+int reftable_log_record_compare_key(const void *a, const void *b);\n+\n+#endif\ndiff --git a/reftable/record_test.c b/reftable/record_test.c\nnew file mode 100644\nindex 0000000000..6da80d33cf\n--- /dev/null\n+++ b/reftable/record_test.c\n@@ -0,0 +1,410 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"reftable.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_copy(struct reftable_record *rec)\n+{\n+\tstruct reftable_record copy =\n+\t\treftable_new_record(reftable_record_type(rec));\n+\treftable_record_copy_from(©, rec, SHA1_SIZE);\n+\t/* do it twice to catch memory leaks */\n+\treftable_record_copy_from(©, rec, SHA1_SIZE);\n+\tswitch (reftable_record_type(©)) {\n+\tcase BLOCK_TYPE_REF:\n+\t\tassert(reftable_ref_record_equal(reftable_record_as_ref(©),\n+\t\t\t\t\t\t reftable_record_as_ref(rec),\n+\t\t\t\t\t\t SHA1_SIZE));\n+\t\tbreak;\n+\tcase BLOCK_TYPE_LOG:\n+\t\tassert(reftable_log_record_equal(reftable_record_as_log(©),\n+\t\t\t\t\t\t reftable_record_as_log(rec),\n+\t\t\t\t\t\t SHA1_SIZE));\n+\t\tbreak;\n+\t}\n+\treftable_record_destroy(©);\n+}\n+\n+static void test_varint_roundtrip(void)\n+{\n+\tuint64_t inputs[] = { 0,\n+\t\t\t 1,\n+\t\t\t 27,\n+\t\t\t 127,\n+\t\t\t 128,\n+\t\t\t 257,\n+\t\t\t 4096,\n+\t\t\t ((uint64_t)1 << 63),\n+\t\t\t ((uint64_t)1 << 63) + ((uint64_t)1 << 63) - 1 };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(inputs); i++) {\n+\t\tuint8_t dest[10];\n+\n+\t\tstruct string_view out = {\n+\t\t\t.buf = dest,\n+\t\t\t.len = sizeof(dest),\n+\t\t};\n+\t\tuint64_t in = inputs[i];\n+\t\tint n = put_var_int(&out, in);\n+\t\tuint64_t got = 0;\n+\n+\t\tassert(n > 0);\n+\t\tout.len = n;\n+\t\tn = get_var_int(&got, &out);\n+\t\tassert(n > 0);\n+\n+\t\tassert(got == in);\n+\t}\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct {\n+\t\tconst char *a, *b;\n+\t\tint want;\n+\t} cases[] = {\n+\t\t{ \"abc\", \"ab\", 2 },\n+\t\t{ \"\", \"abc\", 0 },\n+\t\t{ \"abc\", \"abd\", 2 },\n+\t\t{ \"abc\", \"pqr\", 0 },\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct strbuf a = STRBUF_INIT;\n+\t\tstruct strbuf b = STRBUF_INIT;\n+\t\tstrbuf_addstr(&a, cases[i].a);\n+\t\tstrbuf_addstr(&b, cases[i].b);\n+\t\tassert(common_prefix_size(&a, &b) == cases[i].want);\n+\n+\t\tstrbuf_release(&a);\n+\t\tstrbuf_release(&b);\n+\t}\n+}\n+\n+static void set_hash(uint8_t *h, int j)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < hash_size(SHA1_ID); i++) {\n+\t\th[i] = (j >> i) & 0xff;\n+\t}\n+}\n+\n+static void test_reftable_ref_record_roundtrip(void)\n+{\n+\tint i = 0;\n+\n+\tfor (i = 0; i <= 3; i++) {\n+\t\tstruct reftable_ref_record in = { 0 };\n+\t\tstruct reftable_ref_record out = {\n+\t\t\t.refname = xstrdup(\"old name\"),\n+\t\t\t.value = reftable_calloc(SHA1_SIZE),\n+\t\t\t.target_value = reftable_calloc(SHA1_SIZE),\n+\t\t\t.target = xstrdup(\"old value\"),\n+\t\t};\n+\t\tstruct reftable_record rec_out = { 0 };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_record rec = { 0 };\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\n+\t\tint n, m;\n+\n+\t\tswitch (i) {\n+\t\tcase 0:\n+\t\t\tbreak;\n+\t\tcase 1:\n+\t\t\tin.value = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_hash(in.value, 1);\n+\t\t\tbreak;\n+\t\tcase 2:\n+\t\t\tin.value = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_hash(in.value, 1);\n+\t\t\tin.target_value = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_hash(in.target_value, 2);\n+\t\t\tbreak;\n+\t\tcase 3:\n+\t\t\tin.target = xstrdup(\"target\");\n+\t\t\tbreak;\n+\t\t}\n+\t\tin.refname = xstrdup(\"refs/heads/master\");\n+\n+\t\treftable_record_from_ref(&rec, &in);\n+\t\ttest_copy(&rec);\n+\n+\t\tassert(reftable_record_val_type(&rec) == i);\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\t\tassert(n > 0);\n+\n+\t\t/* decode into a non-zero reftable_record to test for leaks. */\n+\n+\t\treftable_record_from_ref(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, i, dest, SHA1_SIZE);\n+\t\tassert(n == m);\n+\n+\t\tassert((out.value != NULL) == (in.value != NULL));\n+\t\tassert((out.target_value != NULL) == (in.target_value != NULL));\n+\t\tassert((out.target != NULL) == (in.target != NULL));\n+\t\treftable_record_clear(&rec_out);\n+\n+\t\tstrbuf_release(&key);\n+\t\treftable_ref_record_clear(&in);\n+\t}\n+}\n+\n+static void test_reftable_log_record_equal(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 42,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t}\n+\t};\n+\n+\tassert(!reftable_log_record_equal(&in[0], &in[1], SHA1_SIZE));\n+\tin[1].update_index = in[0].update_index;\n+\tassert(reftable_log_record_equal(&in[0], &in[1], SHA1_SIZE));\n+\treftable_log_record_clear(&in[0]);\n+\treftable_log_record_clear(&in[1]);\n+}\n+\n+static void test_reftable_log_record_roundtrip(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.old_hash = reftable_malloc(SHA1_SIZE),\n+\t\t\t.new_hash = reftable_malloc(SHA1_SIZE),\n+\t\t\t.name = xstrdup(\"han-wen\"),\n+\t\t\t.email = xstrdup(\"hanwen@google.com\"),\n+\t\t\t.message = xstrdup(\"test\"),\n+\t\t\t.update_index = 42,\n+\t\t\t.time = 1577123507,\n+\t\t\t.tz_offset = 100,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t}\n+\t};\n+\tset_test_hash(in[0].new_hash, 1);\n+\tset_test_hash(in[0].old_hash, 2);\n+\tfor (int i = 0; i < ARRAY_SIZE(in); i++) {\n+\t\tstruct reftable_record rec = { 0 };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\t/* populate out, to check for leaks. */\n+\t\tstruct reftable_log_record out = {\n+\t\t\t.refname = xstrdup(\"old name\"),\n+\t\t\t.new_hash = reftable_calloc(SHA1_SIZE),\n+\t\t\t.old_hash = reftable_calloc(SHA1_SIZE),\n+\t\t\t.name = xstrdup(\"old name\"),\n+\t\t\t.email = xstrdup(\"old@email\"),\n+\t\t\t.message = xstrdup(\"old message\"),\n+\t\t};\n+\t\tstruct reftable_record rec_out = { 0 };\n+\t\tint n, m, valtype;\n+\n+\t\treftable_record_from_log(&rec, &in[i]);\n+\n+\t\ttest_copy(&rec);\n+\n+\t\treftable_record_key(&rec, &key);\n+\n+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\t\tassert(n >= 0);\n+\t\treftable_record_from_log(&rec_out, &out);\n+\t\tvaltype = reftable_record_val_type(&rec);\n+\t\tm = reftable_record_decode(&rec_out, key, valtype, dest,\n+\t\t\t\t\t SHA1_SIZE);\n+\t\tassert(n == m);\n+\n+\t\tassert(reftable_log_record_equal(&in[i], &out, SHA1_SIZE));\n+\t\treftable_log_record_clear(&in[i]);\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_clear(&rec_out);\n+\t}\n+}\n+\n+static void test_u24_roundtrip(void)\n+{\n+\tuint32_t in = 0x112233;\n+\tuint8_t dest[3];\n+\tuint32_t out;\n+\tput_be24(dest, in);\n+\tout = get_be24(dest);\n+\tassert(in == out);\n+}\n+\n+static void test_key_roundtrip(void)\n+{\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct strbuf roundtrip = STRBUF_INIT;\n+\tint restart;\n+\tuint8_t extra;\n+\tint n, m;\n+\tuint8_t rt_extra;\n+\n+\tstrbuf_addstr(&last_key, \"refs/heads/master\");\n+\tstrbuf_addstr(&key, \"refs/tags/bla\");\n+\textra = 6;\n+\tn = reftable_encode_key(&restart, dest, last_key, key, extra);\n+\tassert(!restart);\n+\tassert(n > 0);\n+\n+\tm = reftable_decode_key(&roundtrip, &rt_extra, last_key, dest);\n+\tassert(n == m);\n+\tassert(0 == strbuf_cmp(&key, &roundtrip));\n+\tassert(rt_extra == extra);\n+\n+\tstrbuf_release(&last_key);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&roundtrip);\n+}\n+\n+static void test_reftable_obj_record_roundtrip(void)\n+{\n+\tuint8_t testHash1[SHA1_SIZE] = { 1, 2, 3, 4, 0 };\n+\tuint64_t till9[] = { 1, 2, 3, 4, 500, 600, 700, 800, 9000 };\n+\tstruct reftable_obj_record recs[3] = { {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 3,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 9,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t } };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(recs); i++) {\n+\t\tstruct reftable_obj_record in = recs[i];\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\tstruct reftable_record rec = { 0 };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_obj_record out = { 0 };\n+\t\tstruct reftable_record rec_out = { 0 };\n+\t\tint n, m;\n+\t\tuint8_t extra;\n+\n+\t\treftable_record_from_obj(&rec, &in);\n+\t\ttest_copy(&rec);\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\t\tassert(n > 0);\n+\t\textra = reftable_record_val_type(&rec);\n+\t\treftable_record_from_obj(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, extra, dest,\n+\t\t\t\t\t SHA1_SIZE);\n+\t\tassert(n == m);\n+\n+\t\tassert(in.hash_prefix_len == out.hash_prefix_len);\n+\t\tassert(in.offset_len == out.offset_len);\n+\n+\t\tassert(!memcmp(in.hash_prefix, out.hash_prefix,\n+\t\t\t in.hash_prefix_len));\n+\t\tassert(0 == memcmp(in.offsets, out.offsets,\n+\t\t\t\t sizeof(uint64_t) * in.offset_len));\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_clear(&rec_out);\n+\t}\n+}\n+\n+static void test_reftable_index_record_roundtrip(void)\n+{\n+\tstruct reftable_index_record in = {\n+\t\t.offset = 42,\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct reftable_record rec = { 0 };\n+\tstruct reftable_index_record out = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record out_rec = { NULL };\n+\tint n, m;\n+\tuint8_t extra;\n+\n+\tstrbuf_addstr(&in.last_key, \"refs/heads/master\");\n+\treftable_record_from_index(&rec, &in);\n+\treftable_record_key(&rec, &key);\n+\ttest_copy(&rec);\n+\n+\tassert(0 == strbuf_cmp(&key, &in.last_key));\n+\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\tassert(n > 0);\n+\n+\textra = reftable_record_val_type(&rec);\n+\treftable_record_from_index(&out_rec, &out);\n+\tm = reftable_record_decode(&out_rec, key, extra, dest, SHA1_SIZE);\n+\tassert(m == n);\n+\n+\tassert(in.offset == out.offset);\n+\n+\treftable_record_clear(&out_rec);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&in.last_key);\n+}\n+\n+int record_test_main(int argc, const char *argv[])\n+{\n+\tadd_test_case(\"test_reftable_log_record_equal\",\n+\t\t &test_reftable_log_record_equal);\n+\tadd_test_case(\"test_reftable_log_record_roundtrip\",\n+\t\t &test_reftable_log_record_roundtrip);\n+\tadd_test_case(\"test_reftable_ref_record_roundtrip\",\n+\t\t &test_reftable_ref_record_roundtrip);\n+\tadd_test_case(\"test_varint_roundtrip\", &test_varint_roundtrip);\n+\tadd_test_case(\"test_key_roundtrip\", &test_key_roundtrip);\n+\tadd_test_case(\"test_common_prefix\", &test_common_prefix);\n+\tadd_test_case(\"test_reftable_obj_record_roundtrip\",\n+\t\t &test_reftable_obj_record_roundtrip);\n+\tadd_test_case(\"test_reftable_index_record_roundtrip\",\n+\t\t &test_reftable_index_record_roundtrip);\n+\tadd_test_case(\"test_u24_roundtrip\", &test_u24_roundtrip);\n+\treturn test_main(argc, argv);\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 7d50aa6bcc..9341272089 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -3,6 +3,7 @@\n \n int cmd__reftable(int argc, const char **argv)\n {\n+\trecord_test_main(argc, argv);\n \tstrbuf_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"405667","messageId":"7bf59d491d75b72b4046661ec94c5e1d6a52047c.1600283416.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.git.git.1600283416.gitgitgadget@gmail.com","subject":"[PATCH 04/13] reftable: add a barebones unittest framework","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-16T19:10:07Z","receivedAt":"2020-09-16T19:11:59Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/test_framework.c | 68 +++++++++++++++++++++++++++++++++++++++\n reftable/test_framework.h | 60 ++++++++++++++++++++++++++++++++++\n 2 files changed, 128 insertions(+)\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n\ndiff --git a/reftable/test_framework.c b/reftable/test_framework.c\nnew file mode 100644\nindex 0000000000..f304a2773a\n--- /dev/null\n+++ b/reftable/test_framework.c\n@@ -0,0 +1,68 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"test_framework.h\"\n+\n+#include \"system.h\"\n+#include \"basics.h\"\n+\n+struct test_case **test_cases;\n+int test_case_len;\n+int test_case_cap;\n+\n+struct test_case *new_test_case(const char *name, void (*testfunc)(void))\n+{\n+\tstruct test_case *tc = reftable_malloc(sizeof(struct test_case));\n+\ttc->name = name;\n+\ttc->testfunc = testfunc;\n+\treturn tc;\n+}\n+\n+struct test_case *add_test_case(const char *name, void (*testfunc)(void))\n+{\n+\tstruct test_case *tc = new_test_case(name, testfunc);\n+\tif (test_case_len == test_case_cap) {\n+\t\ttest_case_cap = 2 * test_case_cap + 1;\n+\t\ttest_cases = reftable_realloc(\n+\t\t\ttest_cases, sizeof(struct test_case) * test_case_cap);\n+\t}\n+\n+\ttest_cases[test_case_len++] = tc;\n+\treturn tc;\n+}\n+\n+int test_main(int argc, const char *argv[])\n+{\n+\tconst char *filter = NULL;\n+\tint i = 0;\n+\tif (argc > 1) {\n+\t\tfilter = argv[1];\n+\t}\n+\n+\tfor (i = 0; i < test_case_len; i++) {\n+\t\tconst char *name = test_cases[i]->name;\n+\t\tif (filter == NULL || strstr(name, filter) != NULL) {\n+\t\t\tprintf(\"case %s\\n\", name);\n+\t\t\ttest_cases[i]->testfunc();\n+\t\t} else {\n+\t\t\tprintf(\"skip %s\\n\", name);\n+\t\t}\n+\n+\t\treftable_free(test_cases[i]);\n+\t}\n+\treftable_free(test_cases);\n+\ttest_cases = 0;\n+\ttest_case_len = 0;\n+\ttest_case_cap = 0;\n+\treturn 0;\n+}\n+\n+void set_test_hash(uint8_t *p, int i)\n+{\n+\tmemset(p, (uint8_t)i, hash_size(SHA1_ID));\n+}\ndiff --git a/reftable/test_framework.h b/reftable/test_framework.h\nnew file mode 100644\nindex 0000000000..3c74d287aa\n--- /dev/null\n+++ b/reftable/test_framework.h\n@@ -0,0 +1,60 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TEST_FRAMEWORK_H\n+#define TEST_FRAMEWORK_H\n+\n+#include \"system.h\"\n+\n+#ifdef NDEBUG\n+#undef NDEBUG\n+#endif\n+\n+#ifdef assert\n+#undef assert\n+#endif\n+\n+#define assert_err(c) \\\n+\tif (c != 0) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: error == %d (%s), want 0\\n\", \\\n+\t\t\t__FILE__, __LINE__, c, reftable_error_str(c)); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define assert_streq(a, b) \\\n+\tif (strcmp(a, b)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s:%d: %s (%s) != %s (%s)\\n\", __FILE__, \\\n+\t\t\t__LINE__, #a, a, #b, b); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define assert(c) \\\n+\tif (!(c)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: failed assertion %s\\n\", __FILE__, \\\n+\t\t\t__LINE__, #c); \\\n+\t\tabort(); \\\n+\t}\n+\n+struct test_case {\n+\tconst char *name;\n+\tvoid (*testfunc)(void);\n+};\n+\n+struct test_case *new_test_case(const char *name, void (*testfunc)(void));\n+struct test_case *add_test_case(const char *name, void (*testfunc)(void));\n+int test_main(int argc, const char *argv[]);\n+\n+void set_test_hash(uint8_t *p, int i);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"405668","messageId":"0f6efc065c9a50966fdc2f43a6c3c1d526a56364.1600283416.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.git.git.1600283416.gitgitgadget@gmail.com","subject":"[PATCH 03/13] vcxproj: adjust for the reftable changes","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-16T19:10:06Z","receivedAt":"2020-09-16T19:11:59Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis allows Git to be compiled via Visual Studio again after integrating\nthe `hn/reftable` branch.\n\nSigned-off-by: Johannes Schindelin \n---\n config.mak.uname | 2 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 ++++++++++-\n 2 files changed, 11 insertions(+), 2 deletions(-)\n\ndiff --git a/config.mak.uname b/config.mak.uname\nindex c7eba69e54..ae4e25a1a4 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -709,7 +709,7 @@ vcxproj:\n \t# Make .vcxproj files and add them\n \tunset QUIET_GEN QUIET_BUILT_IN; \\\n \tperl contrib/buildsystems/generate -g Vcxproj\n-\tgit add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj\n+\tgit add -f git.sln {*,*/lib,*/libreftable,t/helper/*}/*.vcxproj\n \n \t# Generate the LinkOrCopyBuiltins.targets and LinkOrCopyRemoteHttp.targets file\n \t(echo '' && \\\ndiff --git a/contrib/buildsystems/Generators/Vcxproj.pm b/contrib/buildsystems/Generators/Vcxproj.pm\nindex d2584450ba..1a25789d28 100644\n--- a/contrib/buildsystems/Generators/Vcxproj.pm\n+++ b/contrib/buildsystems/Generators/Vcxproj.pm\n@@ -77,7 +77,7 @@ sub createProject {\n my $libs_release = \"\\n \";\n my $libs_debug = \"\\n \";\n if (!$static_library) {\n- $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n+ $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib|reftable\\/libreftable\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n $libs_debug = $libs_release;\n $libs_debug =~ s/zlib\\.lib/zlibd\\.lib/g;\n $libs_debug =~ s/libexpat\\.lib/libexpatd\\.lib/g;\n@@ -232,6 +232,7 @@ sub createProject {\n EOM\n if (!$static_library || $target =~ 'vcs-svn' || $target =~ 'xdiff') {\n my $uuid_libgit = $$build_structure{\"LIBS_libgit_GUID\"};\n+ my $uuid_libreftable = $$build_structure{\"LIBS_reftable/libreftable_GUID\"};\n my $uuid_xdiff_lib = $$build_structure{\"LIBS_xdiff/lib_GUID\"};\n \n print F << \"EOM\";\n@@ -241,6 +242,14 @@ sub createProject {\n false\n \n EOM\n+ if (!($name =~ /xdiff|libreftable/)) {\n+ print F << \"EOM\";\n+ \n+ $uuid_libreftable\n+ false\n+ \n+EOM\n+ }\n if (!($name =~ 'xdiff')) {\n print F << \"EOM\";\n \n-- \ngitgitgadget\n\n"},{"id":"405669","messageId":"e2dbc84833ba04110640de03452c5c1195b5d3a7.1600283416.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.git.git.1600283416.gitgitgadget@gmail.com","subject":"[PATCH 02/13] reftable: define the public API","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-16T19:10:05Z","receivedAt":"2020-09-16T19:12:01Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/reftable.h | 585 ++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 585 insertions(+)\n create mode 100644 reftable/reftable.h\n\ndiff --git a/reftable/reftable.h b/reftable/reftable.h\nnew file mode 100644\nindex 0000000000..5f8ebf5540\n--- /dev/null\n+++ b/reftable/reftable.h\n@@ -0,0 +1,585 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_H\n+#define REFTABLE_H\n+\n+#include \n+#include \n+\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n+\n+/****************************************************************\n+ Basic data types\n+\n+ Reftables store the state of each ref in struct reftable_ref_record, and they\n+ store a sequence of reflog updates in struct reftable_log_record.\n+ ****************************************************************/\n+\n+/* reftable_ref_record holds a ref database entry target_value */\n+struct reftable_ref_record {\n+\tchar *refname; /* Name of the ref, malloced. */\n+\tuint64_t update_index; /* Logical timestamp at which this value is\n+\t\t\t\t written */\n+\tuint8_t *value; /* SHA1, or NULL. malloced. */\n+\tuint8_t *target_value; /* peeled annotated tag, or NULL. malloced. */\n+\tchar *target; /* symref, or NULL. malloced. */\n+};\n+\n+/* returns whether 'ref' represents a deletion */\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);\n+\n+/* prints a reftable_ref_record onto stdout */\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id);\n+\n+/* frees and nulls all pointer values. */\n+void reftable_ref_record_clear(struct reftable_ref_record *ref);\n+\n+/* returns whether two reftable_ref_records are the same */\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size);\n+\n+/* reftable_log_record holds a reflog entry */\n+struct reftable_log_record {\n+\tchar *refname;\n+\tuint64_t update_index; /* logical timestamp of a transactional update.\n+\t\t\t\t*/\n+\tuint8_t *new_hash;\n+\tuint8_t *old_hash;\n+\tchar *name;\n+\tchar *email;\n+\tuint64_t time;\n+\tint16_t tz_offset;\n+\tchar *message;\n+};\n+\n+/* returns whether 'ref' represents the deletion of a log record. */\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log);\n+\n+/* frees and nulls all pointer values. */\n+void reftable_log_record_clear(struct reftable_log_record *log);\n+\n+/* returns whether two records are equal. */\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size);\n+\n+/* dumps a reftable_log_record on stdout, for debugging/testing. */\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id);\n+\n+/****************************************************************\n+ Error handling\n+\n+ Error are signaled with negative integer return values. 0 means success.\n+ ****************************************************************/\n+\n+/* different types of errors */\n+enum reftable_error {\n+\t/* Unexpected file system behavior */\n+\tREFTABLE_IO_ERROR = -2,\n+\n+\t/* Format inconsistency on reading data\n+\t */\n+\tREFTABLE_FORMAT_ERROR = -3,\n+\n+\t/* File does not exist. Returned from block_source_from_file(), because\n+\t it needs special handling in stack.\n+\t*/\n+\tREFTABLE_NOT_EXIST_ERROR = -4,\n+\n+\t/* Trying to write out-of-date data. */\n+\tREFTABLE_LOCK_ERROR = -5,\n+\n+\t/* Misuse of the API:\n+\t - on writing a record with NULL refname.\n+\t - on writing a reftable_ref_record outside the table limits\n+\t - on writing a ref or log record before the stack's next_update_index\n+\t - on writing a log record with multiline message with\n+\t exact_log_message unset\n+\t - on reading a reftable_ref_record from log iterator, or vice versa.\n+\t*/\n+\tREFTABLE_API_ERROR = -6,\n+\n+\t/* Decompression error */\n+\tREFTABLE_ZLIB_ERROR = -7,\n+\n+\t/* Wrote a table without blocks. */\n+\tREFTABLE_EMPTY_TABLE_ERROR = -8,\n+\n+\t/* Dir/file conflict. */\n+\tREFTABLE_NAME_CONFLICT = -9,\n+\n+\t/* Illegal ref name. */\n+\tREFTABLE_REFNAME_ERROR = -10,\n+};\n+\n+/* convert the numeric error code to a string. The string should not be\n+ * deallocated. */\n+const char *reftable_error_str(int err);\n+\n+/*\n+ * Convert the numeric error code to an equivalent errno code.\n+ */\n+int reftable_error_to_errno(int err);\n+\n+/****************************************************************\n+ Writing\n+\n+ Writing single reftables\n+ ****************************************************************/\n+\n+/* reftable_write_options sets options for writing a single reftable. */\n+struct reftable_write_options {\n+\t/* boolean: do not pad out blocks to block size. */\n+\tint unpadded;\n+\n+\t/* the blocksize. Should be less than 2^24. */\n+\tuint32_t block_size;\n+\n+\t/* boolean: do not generate a SHA1 => ref index. */\n+\tint skip_index_objects;\n+\n+\t/* how often to write complete keys in each block. */\n+\tint restart_interval;\n+\n+\t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n+\t * Defaults to SHA1 if unset\n+\t */\n+\tuint32_t hash_id;\n+\n+\t/* boolean: do not check ref names for validity or dir/file conflicts.\n+\t */\n+\tint skip_name_check;\n+\n+\t/* boolean: copy log messages exactly. If unset, check that the message\n+\t * is a single line, and add '\\n' if missing.\n+\t */\n+\tint exact_log_message;\n+};\n+\n+/* reftable_block_stats holds statistics for a single block type */\n+struct reftable_block_stats {\n+\t/* total number of entries written */\n+\tint entries;\n+\t/* total number of key restarts */\n+\tint restarts;\n+\t/* total number of blocks */\n+\tint blocks;\n+\t/* total number of index blocks */\n+\tint index_blocks;\n+\t/* depth of the index */\n+\tint max_index_level;\n+\n+\t/* offset of the first block for this type */\n+\tuint64_t offset;\n+\t/* offset of the top level index block for this type, or 0 if not\n+\t * present */\n+\tuint64_t index_offset;\n+};\n+\n+/* stats holds overall statistics for a single reftable */\n+struct reftable_stats {\n+\t/* total number of blocks written. */\n+\tint blocks;\n+\t/* stats for ref data */\n+\tstruct reftable_block_stats ref_stats;\n+\t/* stats for the SHA1 to ref map. */\n+\tstruct reftable_block_stats obj_stats;\n+\t/* stats for index blocks */\n+\tstruct reftable_block_stats idx_stats;\n+\t/* stats for log blocks */\n+\tstruct reftable_block_stats log_stats;\n+\n+\t/* disambiguation length of shortened object IDs. */\n+\tint object_id_len;\n+};\n+\n+/* reftable_new_writer creates a new writer */\n+struct reftable_writer *\n+reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts);\n+\n+/* write to a file descriptor. fdp should be an int* pointing to the fd. */\n+int reftable_fd_write(void *fdp, const void *data, size_t size);\n+\n+/* Set the range of update indices for the records we will add. When\n+ writing a table into a stack, the min should be at least\n+ reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned.\n+\n+ For transactional updates, typically min==max. When converting an existing\n+ ref database into a single reftable, this would be a range of update-index\n+ timestamps.\n+ */\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max);\n+\n+/* adds a reftable_ref_record. Must be called in ascending\n+ order. The update_index must be within the limits set by\n+ reftable_writer_set_limits(), or REFTABLE_API_ERROR is returned.\n+\n+ It is an error to write a ref record after a log record.\n+ */\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* Convenience function to add multiple refs. Will sort the refs by\n+ name before adding. */\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n);\n+\n+/* adds a reftable_log_record. Must be called in ascending order (with more\n+ recent log entries first.)\n+ */\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* Convenience function to add multiple logs. Will sort the records by\n+ key before adding. */\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n);\n+\n+/* reftable_writer_close finalizes the reftable. The writer is retained so\n+ * statistics can be inspected. */\n+int reftable_writer_close(struct reftable_writer *w);\n+\n+/* writer_stats returns the statistics on the reftable being written.\n+\n+ This struct becomes invalid when the writer is freed.\n+ */\n+const struct reftable_stats *writer_stats(struct reftable_writer *w);\n+\n+/* reftable_writer_free deallocates memory for the writer */\n+void reftable_writer_free(struct reftable_writer *w);\n+\n+/****************************************************************\n+ * ITERATING\n+ ****************************************************************/\n+\n+/* iterator is the generic interface for walking over data stored in a\n+ reftable. It is generally passed around by value.\n+*/\n+struct reftable_iterator {\n+\tstruct reftable_iterator_vtable *ops;\n+\tvoid *iter_arg;\n+};\n+\n+/* reads the next reftable_ref_record. Returns < 0 for error, 0 for OK and > 0:\n+ end of iteration.\n+*/\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* reads the next reftable_log_record. Returns < 0 for error, 0 for OK and > 0:\n+ end of iteration.\n+*/\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* releases resources associated with an iterator. */\n+void reftable_iterator_destroy(struct reftable_iterator *it);\n+\n+/****************************************************************\n+ Reading single tables\n+\n+ The follow routines are for reading single files. For an application-level\n+ interface, skip ahead to struct reftable_merged_table and struct\n+ reftable_stack.\n+ ****************************************************************/\n+\n+/* block_source is a generic wrapper for a seekable readable file.\n+ It is generally passed around by value.\n+ */\n+struct reftable_block_source {\n+\tstruct reftable_block_source_vtable *ops;\n+\tvoid *arg;\n+};\n+\n+/* a contiguous segment of bytes. It keeps track of its generating block_source\n+ so it can return itself into the pool.\n+*/\n+struct reftable_block {\n+\tuint8_t *data;\n+\tint len;\n+\tstruct reftable_block_source source;\n+};\n+\n+/* block_source_vtable are the operations that make up block_source */\n+struct reftable_block_source_vtable {\n+\t/* returns the size of a block source */\n+\tuint64_t (*size)(void *source);\n+\n+\t/* reads a segment from the block source. It is an error to read\n+\t beyond the end of the block */\n+\tint (*read_block)(void *source, struct reftable_block *dest,\n+\t\t\t uint64_t off, uint32_t size);\n+\t/* mark the block as read; may return the data back to malloc */\n+\tvoid (*return_block)(void *source, struct reftable_block *blockp);\n+\n+\t/* release all resources associated with the block source */\n+\tvoid (*close)(void *source);\n+};\n+\n+/* opens a file on the file system as a block_source */\n+int reftable_block_source_from_file(struct reftable_block_source *block_src,\n+\t\t\t\t const char *name);\n+\n+/* The reader struct is a handle to an open reftable file. */\n+struct reftable_reader;\n+\n+/* reftable_new_reader opens a reftable for reading. If successful, returns 0\n+ * code and sets pp. The name is used for creating a stack. Typically, it is the\n+ * basename of the file. The block source `src` is owned by the reader, and is\n+ * closed on calling reftable_reader_destroy().\n+ */\n+int reftable_new_reader(struct reftable_reader **pp,\n+\t\t\tstruct reftable_block_source *src, const char *name);\n+\n+/* reftable_reader_seek_ref returns an iterator where 'name' would be inserted\n+ in the table. To seek to the start of the table, use name = \"\".\n+\n+ example:\n+\n+ struct reftable_reader *r = NULL;\n+ int err = reftable_new_reader(&r, &src, \"filename\");\n+ if (err < 0) { ... }\n+ struct reftable_iterator it = {0};\n+ err = reftable_reader_seek_ref(r, &it, \"refs/heads/master\");\n+ if (err < 0) { ... }\n+ struct reftable_ref_record ref = {0};\n+ while (1) {\n+ err = reftable_iterator_next_ref(&it, &ref);\n+ if (err > 0) {\n+ break;\n+ }\n+ if (err < 0) {\n+ ..error handling..\n+ }\n+ ..found..\n+ }\n+ reftable_iterator_destroy(&it);\n+ reftable_ref_record_clear(&ref);\n+ */\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* returns the hash ID used in this table. */\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r);\n+\n+/* seek to logs for the given name, older than update_index. To seek to the\n+ start of the table, use name = \"\".\n+ */\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index);\n+\n+/* seek to newest log entry for given name. */\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* closes and deallocates a reader. */\n+void reftable_reader_free(struct reftable_reader *);\n+\n+/* return an iterator for the refs pointing to `oid`. */\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid);\n+\n+/* return the max_update_index for a table */\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r);\n+\n+/* return the min_update_index for a table */\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r);\n+\n+/****************************************************************\n+ Merged tables\n+\n+ A ref database kept in a sequence of table files. The merged_table presents a\n+ unified view to reading (seeking, iterating) a sequence of immutable tables.\n+ ****************************************************************/\n+\n+/* A merged table is implements seeking/iterating over a stack of tables. */\n+struct reftable_merged_table;\n+\n+/* A generic reftable; see below. */\n+struct reftable_table;\n+\n+/* reftable_new_merged_table creates a new merged table. It takes ownership of\n+ the stack array.\n+*/\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id);\n+\n+/* returns an iterator positioned just before 'name' */\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns an iterator for log entry, at given update_index */\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index);\n+\n+/* like reftable_merged_table_seek_log_at but look for the newest entry. */\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns the max update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt);\n+\n+/* returns the min update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt);\n+\n+/* releases memory for the merged_table */\n+void reftable_merged_table_free(struct reftable_merged_table *m);\n+\n+/* return the hash ID of the merged table. */\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m);\n+\n+/****************************************************************\n+ Generic tables\n+\n+ A unified API for reading tables, either merged tables, or single readers.\n+ ****************************************************************/\n+\n+struct reftable_table {\n+\tstruct reftable_table_vtable *ops;\n+\tvoid *table_arg;\n+};\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader);\n+\n+/* returns the hash ID from a generic reftable_table */\n+uint32_t reftable_table_hash_id(struct reftable_table *tab);\n+\n+/* create a generic table from reftable_merged_table */\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *table);\n+\n+/* returns the max update_index covered by this table. */\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab);\n+\n+/* returns the min update_index covered by this table. */\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0\n+ for success, and 1 if ref not found. */\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/****************************************************************\n+ Mutable ref database\n+\n+ The stack presents an interface to a mutable sequence of reftables.\n+ ****************************************************************/\n+\n+/* a stack is a stack of reftables, which can be mutated by pushing a table to\n+ * the top of the stack */\n+struct reftable_stack;\n+\n+/* open a new reftable stack. The tables along with the table list will be\n+ stored in 'dir'. Typically, this should be .git/reftables.\n+*/\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config);\n+\n+/* returns the update_index at which a next table should be written. */\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\n+\n+/* holds a transaction to add tables at the top of a stack. */\n+struct reftable_addition;\n+\n+/*\n+ returns a new transaction to add reftables to the given stack. As a side\n+ effect, the ref database is locked.\n+*/\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st);\n+\n+/* Adds a reftable to transaction. */\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+\n+/* Commits the transaction, releasing the lock. */\n+int reftable_addition_commit(struct reftable_addition *add);\n+\n+/* Release all non-committed data from the transaction, and deallocate the\n+ transaction. Releases the lock if held. */\n+void reftable_addition_destroy(struct reftable_addition *add);\n+\n+/* add a new table to the stack. The write_table function must call\n+ reftable_writer_set_limits, add refs and return an error value. */\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *write_arg),\n+\t\t void *write_arg);\n+\n+/* returns the merged_table for seeking. This table is valid until the\n+ next write or reload, and should not be closed or deleted.\n+*/\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st);\n+\n+/* frees all resources associated with the stack. */\n+void reftable_stack_destroy(struct reftable_stack *st);\n+\n+/* Reloads the stack if necessary. This is very cheap to run if the stack was up\n+ * to date */\n+int reftable_stack_reload(struct reftable_stack *st);\n+\n+/* Policy for expiring reflog entries. */\n+struct reftable_log_expiry_config {\n+\t/* Drop entries older than this timestamp */\n+\tuint64_t time;\n+\n+\t/* Drop older entries */\n+\tuint64_t min_update_index;\n+};\n+\n+/* compacts all reftables into a giant table. Expire reflog entries if config is\n+ * non-NULL */\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config);\n+\n+/* heuristically compact unbalanced table stack. */\n+int reftable_stack_auto_compact(struct reftable_stack *st);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0\n+ for success, and 1 if ref not found. */\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* convenience function to read a single log. Returns < 0 for error, 0\n+ for success, and 1 if ref not found. */\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* statistics on past compactions. */\n+struct reftable_compaction_stats {\n+\tuint64_t bytes; /* total number of bytes written */\n+\tuint64_t entries_written; /* total number of entries written, including\n+\t\t\t\t failures. */\n+\tint attempts; /* how often we tried to compact */\n+\tint failures; /* failures happen on concurrent updates */\n+};\n+\n+/* return statistics for compaction up till now. */\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"405961","messageId":"xmqqlfh5i7nk.fsf@gitster.c.googlers.com","threadId":"54245","inReplyTo":"791f69c000556e93bf5fcfc0ec9304833b12565b.1600283416.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 06/13] reftable: (de)serialization for the polymorphic record type.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-20T01:00:47Z","receivedAt":"2020-09-20T01:01:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Han-Wen Nienhuys via GitGitGadget\" writes:\n\n> +static void reftable_log_record_key(const void *r, struct strbuf *dest)\n> +{\n> +\tconst struct reftable_log_record *rec =\n> +\t\t(const struct reftable_log_record *)r;\n> +\tint len = strlen(rec->refname);\n> +\tuint8_t i64[8];\n> +\tuint64_t ts = 0;\n> +\tstrbuf_reset(dest);\n> +\tstrbuf_add(dest, (uint8_t *)rec->refname, len + 1);\n> +\n> +\tts = (~ts) - rec->update_index;\n> +\tput_be64(&i64[0], ts);\n> +\tstrbuf_add(dest, i64, sizeof(i64));\n> +}\n\nWe seem to be getting\n\nreftable/record.c: In function 'reftable_log_record_key':\nreftable/record.c:578:2: error: dereferencing type-punned pointer will break strict-aliasing rules [-Werror=strict-aliasing]\n put_be64(&i64[0], ts);\n ^\n CC reftable/refname.o\n\nwhen this series is merged to 'seen'.\n\ncf. e.g. https://travis-ci.org/github/git/git/jobs/728655368\n\n\nThanks.\n\n\n\n"},{"id":"406041","messageId":"CAFQ2z_M9uBc+VArOVXg-hwTY8bu-gQQSL9JC6TJ5MuTCtxx=EQ@mail.gmail.com","threadId":"54245","inReplyTo":"xmqqlfh5i7nk.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 06/13] reftable: (de)serialization for the polymorphic record type.","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2020-09-21T13:13:05Z","receivedAt":"2020-09-21T13:13:25Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Sun, Sep 20, 2020 at 3:00 AM Junio C Hamano wrote:\n> > +static void reftable_log_record_key(const void *r, struct strbuf *dest)\n> > +{\n> > + const struct reftable_log_record *rec =\n> > + (const struct reftable_log_record *)r;\n> > + int len = strlen(rec->refname);\n> > + uint8_t i64[8];\n> > + uint64_t ts = 0;\n> > + strbuf_reset(dest);\n> > + strbuf_add(dest, (uint8_t *)rec->refname, len + 1);\n> > +\n> > + ts = (~ts) - rec->update_index;\n> > + put_be64(&i64[0], ts);\n> > + strbuf_add(dest, i64, sizeof(i64));\n> > +}\n>\n> We seem to be getting\n>\n> reftable/record.c: In function 'reftable_log_record_key':\n> reftable/record.c:578:2: error: dereferencing type-punned pointer will break strict-aliasing rules [-Werror=strict-aliasing]\n> put_be64(&i64[0], ts);\n> ^\n> CC reftable/refname.o\n>\n> when this series is merged to 'seen'.\n\nThanks for bringing this up. I did see this before, but I've been\nunable to reproduce this locally, and I forgot about it.\n\nThe problem is actually triggered by the Git-provided put_be64()\n(which appears unused in the Git source code). The definition\n\n #define put_be64(p, v) do { *(uint64_t *)(p) = htonll(v); } while (0)\n\nis trying to reinterpret a char* as a uint64_t* , which is illegal in\nstrict aliasing rules? I originally had\n\n+void put_be64(uint8_t *out, uint64_t v)\n+{\n+ int i = sizeof(uint64_t);\n+ while (i--) {\n+ out[i] = (uint8_t)(v & 0xff);\n+ v >>= 8;\n+ }\n+}\n\nin my reftable library, which is portable. Is there a reason for the\nmagic with htonll and friends?\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\nRegistergericht und -nummer: Hamburg, HRB 86891\nSitz der Gesellschaft: Hamburg\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"406242","messageId":"20200924072151.GC1851751@coredump.intra.peff.net","threadId":"54245","inReplyTo":"CAFQ2z_M9uBc+VArOVXg-hwTY8bu-gQQSL9JC6TJ5MuTCtxx=EQ@mail.gmail.com","subject":"Re: [PATCH 06/13] reftable: (de)serialization for the polymorphic record type.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-09-24T07:21:51Z","receivedAt":"2020-09-24T07:21:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Sep 21, 2020 at 03:13:05PM +0200, Han-Wen Nienhuys wrote:\n\n> The problem is actually triggered by the Git-provided put_be64()\n> (which appears unused in the Git source code). The definition\n> \n> #define put_be64(p, v) do { *(uint64_t *)(p) = htonll(v); } while (0)\n> \n> is trying to reinterpret a char* as a uint64_t* , which is illegal in\n> strict aliasing rules?\n\nAh, thanks for finding that. I think this resolves the mystery I had\naround aliasing warnings with put_be32() in f39ad38410 (fast-export: use\nlocal array to store anonymized oid, 2020-06-25). I got confused because\nwhen I looked at the implementation of put_be32(), I stupidly looked at\nthe NO_ALIGNED_LOADS fallback code, which indeed does not alias. But the\none we use on x86 does.\n\nI thought this would be OK because C allows aliasing through \"char *\"\npointers. But I think it may only go the other way. I.e., you can\ntype-pun a uint64_t as a char, but not the other way around. But then\nwhy doesn't basically every use of put_be32() and put_be64() complain?\n\n> I originally had\n> \n> +void put_be64(uint8_t *out, uint64_t v)\n> +{\n> + int i = sizeof(uint64_t);\n> + while (i--) {\n> + out[i] = (uint8_t)(v & 0xff);\n> + v >>= 8;\n> + }\n> +}\n> \n> in my reftable library, which is portable. Is there a reason for the\n> magic with htonll and friends?\n\nPresumably it was thought to be faster. This comes originally from the\nblock-sha1 code in 660231aa97 (block-sha1: support for architectures\nwith memory alignment restrictions, 2009-08-12). I don't know how it\ncompares in practice, and especially these days.\n\nOur fallback routines are similar to an unrolled version of what you\nwrote above.\n\n-Peff\n"},{"id":"406243","messageId":"20200924073125.GD1851751@coredump.intra.peff.net","threadId":"54245","inReplyTo":"20200924072151.GC1851751@coredump.intra.peff.net","subject":"Re: [PATCH 06/13] reftable: (de)serialization for the polymorphic record type.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-09-24T07:31:25Z","receivedAt":"2020-09-24T07:31:28Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Sep 24, 2020 at 03:21:51AM -0400, Jeff King wrote:\n\n> > I originally had\n> > \n> > +void put_be64(uint8_t *out, uint64_t v)\n> > +{\n> > + int i = sizeof(uint64_t);\n> > + while (i--) {\n> > + out[i] = (uint8_t)(v & 0xff);\n> > + v >>= 8;\n> > + }\n> > +}\n> > \n> > in my reftable library, which is portable. Is there a reason for the\n> > magic with htonll and friends?\n> \n> Presumably it was thought to be faster. This comes originally from the\n> block-sha1 code in 660231aa97 (block-sha1: support for architectures\n> with memory alignment restrictions, 2009-08-12). I don't know how it\n> compares in practice, and especially these days.\n> \n> Our fallback routines are similar to an unrolled version of what you\n> wrote above.\n\nWe should be able to measure it pretty easily, since block-sha1 uses a\nlot of get_be32/put_be32. I generated a 4GB random file, built with\nBLK_SHA1=Yes and -O2, and timed:\n\n t/helper/test-tool sha1 writes:\n\n> Then I did the same, but building with -DNO_UNALIGNED_LOADS. The latter\n> actually ran faster, by a small margin. Here are the hyperfine results:\n>\n> [stock]\n> Time (mean ± σ): 6.638 s ± 0.081 s [User: 6.269 s, System: 0.368 s]\n> Range (min … max): 6.550 s … 6.841 s 10 runs\n>\n> [-DNO_UNALIGNED_LOADS]\n> Time (mean ± σ): 6.418 s ± 0.015 s [User: 6.058 s, System: 0.360 s]\n> Range (min … max): 6.394 s … 6.447 s 10 runs\n>\n> For casual use as in reftables I doubt the difference is even\n> measurable. But this result implies that perhaps we ought to just be\n> using the fallback version all the time.\n\nI like that one. One less configurable knob that makes us execute\ndifferent codepaths is one less thing to be worried about.\n\n"},{"id":"406768","messageId":"6228103b4a6dab23ce3a4498eada8ff649909ff0.1601568663.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v2.git.git.1601568663.gitgitgadget@gmail.com","subject":"[PATCH v2 01/13] reftable: add LICENSE","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-01T16:10:51Z","receivedAt":"2020-10-01T16:11:10Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nTODO: relicense?\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/LICENSE | 31 +++++++++++++++++++++++++++++++\n 1 file changed, 31 insertions(+)\n create mode 100644 reftable/LICENSE\n\ndiff --git a/reftable/LICENSE b/reftable/LICENSE\nnew file mode 100644\nindex 0000000000..402e0f9356\n--- /dev/null\n+++ b/reftable/LICENSE\n@@ -0,0 +1,31 @@\n+BSD License\n+\n+Copyright (c) 2020, Google LLC\n+All rights reserved.\n+\n+Redistribution and use in source and binary forms, with or without\n+modification, are permitted provided that the following conditions are\n+met:\n+\n+* Redistributions of source code must retain the above copyright notice,\n+this list of conditions and the following disclaimer.\n+\n+* Redistributions in binary form must reproduce the above copyright\n+notice, this list of conditions and the following disclaimer in the\n+documentation and/or other materials provided with the distribution.\n+\n+* Neither the name of Google LLC nor the names of its contributors may\n+be used to endorse or promote products derived from this software\n+without specific prior written permission.\n+\n+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n+\"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n-- \ngitgitgadget\n\n"},{"id":"406769","messageId":"5d1b946ab5473504e9599d90c0feda407a179a05.1601568663.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v2.git.git.1601568663.gitgitgadget@gmail.com","subject":"[PATCH v2 02/13] reftable: define the public API","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-01T16:10:52Z","receivedAt":"2020-10-01T16:11:12Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/reftable.h | 585 ++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 585 insertions(+)\n create mode 100644 reftable/reftable.h\n\ndiff --git a/reftable/reftable.h b/reftable/reftable.h\nnew file mode 100644\nindex 0000000000..5f8ebf5540\n--- /dev/null\n+++ b/reftable/reftable.h\n@@ -0,0 +1,585 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_H\n+#define REFTABLE_H\n+\n+#include \n+#include \n+\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n+\n+/****************************************************************\n+ Basic data types\n+\n+ Reftables store the state of each ref in struct reftable_ref_record, and they\n+ store a sequence of reflog updates in struct reftable_log_record.\n+ ****************************************************************/\n+\n+/* reftable_ref_record holds a ref database entry target_value */\n+struct reftable_ref_record {\n+\tchar *refname; /* Name of the ref, malloced. */\n+\tuint64_t update_index; /* Logical timestamp at which this value is\n+\t\t\t\t written */\n+\tuint8_t *value; /* SHA1, or NULL. malloced. */\n+\tuint8_t *target_value; /* peeled annotated tag, or NULL. malloced. */\n+\tchar *target; /* symref, or NULL. malloced. */\n+};\n+\n+/* returns whether 'ref' represents a deletion */\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);\n+\n+/* prints a reftable_ref_record onto stdout */\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id);\n+\n+/* frees and nulls all pointer values. */\n+void reftable_ref_record_clear(struct reftable_ref_record *ref);\n+\n+/* returns whether two reftable_ref_records are the same */\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size);\n+\n+/* reftable_log_record holds a reflog entry */\n+struct reftable_log_record {\n+\tchar *refname;\n+\tuint64_t update_index; /* logical timestamp of a transactional update.\n+\t\t\t\t*/\n+\tuint8_t *new_hash;\n+\tuint8_t *old_hash;\n+\tchar *name;\n+\tchar *email;\n+\tuint64_t time;\n+\tint16_t tz_offset;\n+\tchar *message;\n+};\n+\n+/* returns whether 'ref' represents the deletion of a log record. */\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log);\n+\n+/* frees and nulls all pointer values. */\n+void reftable_log_record_clear(struct reftable_log_record *log);\n+\n+/* returns whether two records are equal. */\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size);\n+\n+/* dumps a reftable_log_record on stdout, for debugging/testing. */\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id);\n+\n+/****************************************************************\n+ Error handling\n+\n+ Error are signaled with negative integer return values. 0 means success.\n+ ****************************************************************/\n+\n+/* different types of errors */\n+enum reftable_error {\n+\t/* Unexpected file system behavior */\n+\tREFTABLE_IO_ERROR = -2,\n+\n+\t/* Format inconsistency on reading data\n+\t */\n+\tREFTABLE_FORMAT_ERROR = -3,\n+\n+\t/* File does not exist. Returned from block_source_from_file(), because\n+\t it needs special handling in stack.\n+\t*/\n+\tREFTABLE_NOT_EXIST_ERROR = -4,\n+\n+\t/* Trying to write out-of-date data. */\n+\tREFTABLE_LOCK_ERROR = -5,\n+\n+\t/* Misuse of the API:\n+\t - on writing a record with NULL refname.\n+\t - on writing a reftable_ref_record outside the table limits\n+\t - on writing a ref or log record before the stack's next_update_index\n+\t - on writing a log record with multiline message with\n+\t exact_log_message unset\n+\t - on reading a reftable_ref_record from log iterator, or vice versa.\n+\t*/\n+\tREFTABLE_API_ERROR = -6,\n+\n+\t/* Decompression error */\n+\tREFTABLE_ZLIB_ERROR = -7,\n+\n+\t/* Wrote a table without blocks. */\n+\tREFTABLE_EMPTY_TABLE_ERROR = -8,\n+\n+\t/* Dir/file conflict. */\n+\tREFTABLE_NAME_CONFLICT = -9,\n+\n+\t/* Illegal ref name. */\n+\tREFTABLE_REFNAME_ERROR = -10,\n+};\n+\n+/* convert the numeric error code to a string. The string should not be\n+ * deallocated. */\n+const char *reftable_error_str(int err);\n+\n+/*\n+ * Convert the numeric error code to an equivalent errno code.\n+ */\n+int reftable_error_to_errno(int err);\n+\n+/****************************************************************\n+ Writing\n+\n+ Writing single reftables\n+ ****************************************************************/\n+\n+/* reftable_write_options sets options for writing a single reftable. */\n+struct reftable_write_options {\n+\t/* boolean: do not pad out blocks to block size. */\n+\tint unpadded;\n+\n+\t/* the blocksize. Should be less than 2^24. */\n+\tuint32_t block_size;\n+\n+\t/* boolean: do not generate a SHA1 => ref index. */\n+\tint skip_index_objects;\n+\n+\t/* how often to write complete keys in each block. */\n+\tint restart_interval;\n+\n+\t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n+\t * Defaults to SHA1 if unset\n+\t */\n+\tuint32_t hash_id;\n+\n+\t/* boolean: do not check ref names for validity or dir/file conflicts.\n+\t */\n+\tint skip_name_check;\n+\n+\t/* boolean: copy log messages exactly. If unset, check that the message\n+\t * is a single line, and add '\\n' if missing.\n+\t */\n+\tint exact_log_message;\n+};\n+\n+/* reftable_block_stats holds statistics for a single block type */\n+struct reftable_block_stats {\n+\t/* total number of entries written */\n+\tint entries;\n+\t/* total number of key restarts */\n+\tint restarts;\n+\t/* total number of blocks */\n+\tint blocks;\n+\t/* total number of index blocks */\n+\tint index_blocks;\n+\t/* depth of the index */\n+\tint max_index_level;\n+\n+\t/* offset of the first block for this type */\n+\tuint64_t offset;\n+\t/* offset of the top level index block for this type, or 0 if not\n+\t * present */\n+\tuint64_t index_offset;\n+};\n+\n+/* stats holds overall statistics for a single reftable */\n+struct reftable_stats {\n+\t/* total number of blocks written. */\n+\tint blocks;\n+\t/* stats for ref data */\n+\tstruct reftable_block_stats ref_stats;\n+\t/* stats for the SHA1 to ref map. */\n+\tstruct reftable_block_stats obj_stats;\n+\t/* stats for index blocks */\n+\tstruct reftable_block_stats idx_stats;\n+\t/* stats for log blocks */\n+\tstruct reftable_block_stats log_stats;\n+\n+\t/* disambiguation length of shortened object IDs. */\n+\tint object_id_len;\n+};\n+\n+/* reftable_new_writer creates a new writer */\n+struct reftable_writer *\n+reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts);\n+\n+/* write to a file descriptor. fdp should be an int* pointing to the fd. */\n+int reftable_fd_write(void *fdp, const void *data, size_t size);\n+\n+/* Set the range of update indices for the records we will add. When\n+ writing a table into a stack, the min should be at least\n+ reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned.\n+\n+ For transactional updates, typically min==max. When converting an existing\n+ ref database into a single reftable, this would be a range of update-index\n+ timestamps.\n+ */\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max);\n+\n+/* adds a reftable_ref_record. Must be called in ascending\n+ order. The update_index must be within the limits set by\n+ reftable_writer_set_limits(), or REFTABLE_API_ERROR is returned.\n+\n+ It is an error to write a ref record after a log record.\n+ */\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* Convenience function to add multiple refs. Will sort the refs by\n+ name before adding. */\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n);\n+\n+/* adds a reftable_log_record. Must be called in ascending order (with more\n+ recent log entries first.)\n+ */\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* Convenience function to add multiple logs. Will sort the records by\n+ key before adding. */\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n);\n+\n+/* reftable_writer_close finalizes the reftable. The writer is retained so\n+ * statistics can be inspected. */\n+int reftable_writer_close(struct reftable_writer *w);\n+\n+/* writer_stats returns the statistics on the reftable being written.\n+\n+ This struct becomes invalid when the writer is freed.\n+ */\n+const struct reftable_stats *writer_stats(struct reftable_writer *w);\n+\n+/* reftable_writer_free deallocates memory for the writer */\n+void reftable_writer_free(struct reftable_writer *w);\n+\n+/****************************************************************\n+ * ITERATING\n+ ****************************************************************/\n+\n+/* iterator is the generic interface for walking over data stored in a\n+ reftable. It is generally passed around by value.\n+*/\n+struct reftable_iterator {\n+\tstruct reftable_iterator_vtable *ops;\n+\tvoid *iter_arg;\n+};\n+\n+/* reads the next reftable_ref_record. Returns < 0 for error, 0 for OK and > 0:\n+ end of iteration.\n+*/\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* reads the next reftable_log_record. Returns < 0 for error, 0 for OK and > 0:\n+ end of iteration.\n+*/\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* releases resources associated with an iterator. */\n+void reftable_iterator_destroy(struct reftable_iterator *it);\n+\n+/****************************************************************\n+ Reading single tables\n+\n+ The follow routines are for reading single files. For an application-level\n+ interface, skip ahead to struct reftable_merged_table and struct\n+ reftable_stack.\n+ ****************************************************************/\n+\n+/* block_source is a generic wrapper for a seekable readable file.\n+ It is generally passed around by value.\n+ */\n+struct reftable_block_source {\n+\tstruct reftable_block_source_vtable *ops;\n+\tvoid *arg;\n+};\n+\n+/* a contiguous segment of bytes. It keeps track of its generating block_source\n+ so it can return itself into the pool.\n+*/\n+struct reftable_block {\n+\tuint8_t *data;\n+\tint len;\n+\tstruct reftable_block_source source;\n+};\n+\n+/* block_source_vtable are the operations that make up block_source */\n+struct reftable_block_source_vtable {\n+\t/* returns the size of a block source */\n+\tuint64_t (*size)(void *source);\n+\n+\t/* reads a segment from the block source. It is an error to read\n+\t beyond the end of the block */\n+\tint (*read_block)(void *source, struct reftable_block *dest,\n+\t\t\t uint64_t off, uint32_t size);\n+\t/* mark the block as read; may return the data back to malloc */\n+\tvoid (*return_block)(void *source, struct reftable_block *blockp);\n+\n+\t/* release all resources associated with the block source */\n+\tvoid (*close)(void *source);\n+};\n+\n+/* opens a file on the file system as a block_source */\n+int reftable_block_source_from_file(struct reftable_block_source *block_src,\n+\t\t\t\t const char *name);\n+\n+/* The reader struct is a handle to an open reftable file. */\n+struct reftable_reader;\n+\n+/* reftable_new_reader opens a reftable for reading. If successful, returns 0\n+ * code and sets pp. The name is used for creating a stack. Typically, it is the\n+ * basename of the file. The block source `src` is owned by the reader, and is\n+ * closed on calling reftable_reader_destroy().\n+ */\n+int reftable_new_reader(struct reftable_reader **pp,\n+\t\t\tstruct reftable_block_source *src, const char *name);\n+\n+/* reftable_reader_seek_ref returns an iterator where 'name' would be inserted\n+ in the table. To seek to the start of the table, use name = \"\".\n+\n+ example:\n+\n+ struct reftable_reader *r = NULL;\n+ int err = reftable_new_reader(&r, &src, \"filename\");\n+ if (err < 0) { ... }\n+ struct reftable_iterator it = {0};\n+ err = reftable_reader_seek_ref(r, &it, \"refs/heads/master\");\n+ if (err < 0) { ... }\n+ struct reftable_ref_record ref = {0};\n+ while (1) {\n+ err = reftable_iterator_next_ref(&it, &ref);\n+ if (err > 0) {\n+ break;\n+ }\n+ if (err < 0) {\n+ ..error handling..\n+ }\n+ ..found..\n+ }\n+ reftable_iterator_destroy(&it);\n+ reftable_ref_record_clear(&ref);\n+ */\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* returns the hash ID used in this table. */\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r);\n+\n+/* seek to logs for the given name, older than update_index. To seek to the\n+ start of the table, use name = \"\".\n+ */\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index);\n+\n+/* seek to newest log entry for given name. */\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* closes and deallocates a reader. */\n+void reftable_reader_free(struct reftable_reader *);\n+\n+/* return an iterator for the refs pointing to `oid`. */\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid);\n+\n+/* return the max_update_index for a table */\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r);\n+\n+/* return the min_update_index for a table */\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r);\n+\n+/****************************************************************\n+ Merged tables\n+\n+ A ref database kept in a sequence of table files. The merged_table presents a\n+ unified view to reading (seeking, iterating) a sequence of immutable tables.\n+ ****************************************************************/\n+\n+/* A merged table is implements seeking/iterating over a stack of tables. */\n+struct reftable_merged_table;\n+\n+/* A generic reftable; see below. */\n+struct reftable_table;\n+\n+/* reftable_new_merged_table creates a new merged table. It takes ownership of\n+ the stack array.\n+*/\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id);\n+\n+/* returns an iterator positioned just before 'name' */\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns an iterator for log entry, at given update_index */\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index);\n+\n+/* like reftable_merged_table_seek_log_at but look for the newest entry. */\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns the max update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt);\n+\n+/* returns the min update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt);\n+\n+/* releases memory for the merged_table */\n+void reftable_merged_table_free(struct reftable_merged_table *m);\n+\n+/* return the hash ID of the merged table. */\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m);\n+\n+/****************************************************************\n+ Generic tables\n+\n+ A unified API for reading tables, either merged tables, or single readers.\n+ ****************************************************************/\n+\n+struct reftable_table {\n+\tstruct reftable_table_vtable *ops;\n+\tvoid *table_arg;\n+};\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader);\n+\n+/* returns the hash ID from a generic reftable_table */\n+uint32_t reftable_table_hash_id(struct reftable_table *tab);\n+\n+/* create a generic table from reftable_merged_table */\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *table);\n+\n+/* returns the max update_index covered by this table. */\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab);\n+\n+/* returns the min update_index covered by this table. */\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0\n+ for success, and 1 if ref not found. */\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/****************************************************************\n+ Mutable ref database\n+\n+ The stack presents an interface to a mutable sequence of reftables.\n+ ****************************************************************/\n+\n+/* a stack is a stack of reftables, which can be mutated by pushing a table to\n+ * the top of the stack */\n+struct reftable_stack;\n+\n+/* open a new reftable stack. The tables along with the table list will be\n+ stored in 'dir'. Typically, this should be .git/reftables.\n+*/\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config);\n+\n+/* returns the update_index at which a next table should be written. */\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\n+\n+/* holds a transaction to add tables at the top of a stack. */\n+struct reftable_addition;\n+\n+/*\n+ returns a new transaction to add reftables to the given stack. As a side\n+ effect, the ref database is locked.\n+*/\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st);\n+\n+/* Adds a reftable to transaction. */\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+\n+/* Commits the transaction, releasing the lock. */\n+int reftable_addition_commit(struct reftable_addition *add);\n+\n+/* Release all non-committed data from the transaction, and deallocate the\n+ transaction. Releases the lock if held. */\n+void reftable_addition_destroy(struct reftable_addition *add);\n+\n+/* add a new table to the stack. The write_table function must call\n+ reftable_writer_set_limits, add refs and return an error value. */\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *write_arg),\n+\t\t void *write_arg);\n+\n+/* returns the merged_table for seeking. This table is valid until the\n+ next write or reload, and should not be closed or deleted.\n+*/\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st);\n+\n+/* frees all resources associated with the stack. */\n+void reftable_stack_destroy(struct reftable_stack *st);\n+\n+/* Reloads the stack if necessary. This is very cheap to run if the stack was up\n+ * to date */\n+int reftable_stack_reload(struct reftable_stack *st);\n+\n+/* Policy for expiring reflog entries. */\n+struct reftable_log_expiry_config {\n+\t/* Drop entries older than this timestamp */\n+\tuint64_t time;\n+\n+\t/* Drop older entries */\n+\tuint64_t min_update_index;\n+};\n+\n+/* compacts all reftables into a giant table. Expire reflog entries if config is\n+ * non-NULL */\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config);\n+\n+/* heuristically compact unbalanced table stack. */\n+int reftable_stack_auto_compact(struct reftable_stack *st);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0\n+ for success, and 1 if ref not found. */\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* convenience function to read a single log. Returns < 0 for error, 0\n+ for success, and 1 if ref not found. */\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* statistics on past compactions. */\n+struct reftable_compaction_stats {\n+\tuint64_t bytes; /* total number of bytes written */\n+\tuint64_t entries_written; /* total number of entries written, including\n+\t\t\t\t failures. */\n+\tint attempts; /* how often we tried to compact */\n+\tint failures; /* failures happen on concurrent updates */\n+};\n+\n+/* return statistics for compaction up till now. */\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"406770","messageId":"b94c5f5c6120347fd97bcc2dcc187fc86a802dff.1601568663.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v2.git.git.1601568663.gitgitgadget@gmail.com","subject":"[PATCH v2 04/13] reftable: add a barebones unittest framework","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-01T16:10:54Z","receivedAt":"2020-10-01T16:11:13Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/test_framework.c | 68 +++++++++++++++++++++++++++++++++++++++\n reftable/test_framework.h | 59 +++++++++++++++++++++++++++++++++\n 2 files changed, 127 insertions(+)\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n\ndiff --git a/reftable/test_framework.c b/reftable/test_framework.c\nnew file mode 100644\nindex 0000000000..8d718f2f06\n--- /dev/null\n+++ b/reftable/test_framework.c\n@@ -0,0 +1,68 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"test_framework.h\"\n+\n+#include \"system.h\"\n+#include \"basics.h\"\n+\n+static struct test_case **test_cases;\n+static int test_case_len;\n+static int test_case_cap;\n+\n+static struct test_case *new_test_case(const char *name, void (*testfunc)(void))\n+{\n+\tstruct test_case *tc = reftable_malloc(sizeof(struct test_case));\n+\ttc->name = name;\n+\ttc->testfunc = testfunc;\n+\treturn tc;\n+}\n+\n+struct test_case *add_test_case(const char *name, void (*testfunc)(void))\n+{\n+\tstruct test_case *tc = new_test_case(name, testfunc);\n+\tif (test_case_len == test_case_cap) {\n+\t\ttest_case_cap = 2 * test_case_cap + 1;\n+\t\ttest_cases = reftable_realloc(\n+\t\t\ttest_cases, sizeof(struct test_case) * test_case_cap);\n+\t}\n+\n+\ttest_cases[test_case_len++] = tc;\n+\treturn tc;\n+}\n+\n+int test_main(int argc, const char *argv[])\n+{\n+\tconst char *filter = NULL;\n+\tint i = 0;\n+\tif (argc > 1) {\n+\t\tfilter = argv[1];\n+\t}\n+\n+\tfor (i = 0; i < test_case_len; i++) {\n+\t\tconst char *name = test_cases[i]->name;\n+\t\tif (filter == NULL || strstr(name, filter) != NULL) {\n+\t\t\tprintf(\"case %s\\n\", name);\n+\t\t\ttest_cases[i]->testfunc();\n+\t\t} else {\n+\t\t\tprintf(\"skip %s\\n\", name);\n+\t\t}\n+\n+\t\treftable_free(test_cases[i]);\n+\t}\n+\treftable_free(test_cases);\n+\ttest_cases = NULL;\n+\ttest_case_len = 0;\n+\ttest_case_cap = 0;\n+\treturn 0;\n+}\n+\n+void set_test_hash(uint8_t *p, int i)\n+{\n+\tmemset(p, (uint8_t)i, hash_size(SHA1_ID));\n+}\ndiff --git a/reftable/test_framework.h b/reftable/test_framework.h\nnew file mode 100644\nindex 0000000000..f0a208e880\n--- /dev/null\n+++ b/reftable/test_framework.h\n@@ -0,0 +1,59 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TEST_FRAMEWORK_H\n+#define TEST_FRAMEWORK_H\n+\n+#include \"system.h\"\n+\n+#ifdef NDEBUG\n+#undef NDEBUG\n+#endif\n+\n+#ifdef assert\n+#undef assert\n+#endif\n+\n+#define assert_err(c) \\\n+\tif (c != 0) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: error == %d (%s), want 0\\n\", \\\n+\t\t\t__FILE__, __LINE__, c, reftable_error_str(c)); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define assert_streq(a, b) \\\n+\tif (strcmp(a, b)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s:%d: %s (%s) != %s (%s)\\n\", __FILE__, \\\n+\t\t\t__LINE__, #a, a, #b, b); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define assert(c) \\\n+\tif (!(c)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: failed assertion %s\\n\", __FILE__, \\\n+\t\t\t__LINE__, #c); \\\n+\t\tabort(); \\\n+\t}\n+\n+struct test_case {\n+\tconst char *name;\n+\tvoid (*testfunc)(void);\n+};\n+\n+struct test_case *add_test_case(const char *name, void (*testfunc)(void));\n+int test_main(int argc, const char *argv[]);\n+\n+void set_test_hash(uint8_t *p, int i);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"406771","messageId":"pull.847.v2.git.git.1601568663.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.git.git.1600283416.gitgitgadget@gmail.com","subject":"[PATCH v2 00/13] reftable library","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-01T16:10:50Z","receivedAt":"2020-10-01T16:11:14Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"This splits the giant commit from \nhttps://github.com/gitgitgadget/git/pull/539 into a series of smaller\ncommits, which build and have unittests.\n\nThe final commit should also be split up, but I want to wait until we have\nconsensus that the bottom commits look good.\n\nHan-Wen Nienhuys (12):\n reftable: add LICENSE\n reftable: define the public API\n reftable: add a barebones unittest framework\n reftable: utility functions\n reftable: (de)serialization for the polymorphic record type.\n reftable: reading/writing blocks\n reftable: a generic binary tree implementation\n reftable: write reftable files\n reftable: read reftable files\n reftable: file level tests\n reftable: rest of library\n reftable: \"test-tool dump-reftable\" command.\n\nJohannes Schindelin (1):\n vcxproj: adjust for the reftable changes\n\n Makefile | 46 +-\n config.mak.uname | 2 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n reftable/.gitattributes | 1 +\n reftable/LICENSE | 31 +\n reftable/VERSION | 1 +\n reftable/basics.c | 131 +++\n reftable/basics.h | 48 +\n reftable/block.c | 441 +++++++\n reftable/block.h | 129 ++\n reftable/block_test.c | 158 +++\n reftable/blocksource.c | 148 +++\n reftable/blocksource.h | 22 +\n reftable/compat.c | 110 ++\n reftable/compat.h | 48 +\n reftable/constants.h | 21 +\n reftable/dump.c | 212 ++++\n reftable/iter.c | 242 ++++\n reftable/iter.h | 72 ++\n reftable/merged.c | 358 ++++++\n reftable/merged.h | 36 +\n reftable/merged_test.c | 331 ++++++\n reftable/pq.c | 115 ++\n reftable/pq.h | 32 +\n reftable/publicbasics.c | 100 ++\n reftable/reader.c | 733 ++++++++++++\n reftable/reader.h | 78 ++\n reftable/record.c | 1116 ++++++++++++++++++\n reftable/record.h | 137 +++\n reftable/record_test.c | 410 +++++++\n reftable/refname.c | 209 ++++\n reftable/refname.h | 28 +\n reftable/refname_test.c | 100 ++\n reftable/reftable-tests.h | 22 +\n reftable/reftable.c | 104 ++\n reftable/reftable.h | 585 +++++++++\n reftable/reftable_test.c | 585 +++++++++\n reftable/stack.c | 1240 ++++++++++++++++++++\n reftable/stack.h | 40 +\n reftable/stack_test.c | 788 +++++++++++++\n reftable/strbuf.c | 142 +++\n reftable/strbuf.h | 80 ++\n reftable/strbuf_test.c | 37 +\n reftable/system.h | 51 +\n reftable/test_framework.c | 68 ++\n reftable/test_framework.h | 59 +\n reftable/tree.c | 63 +\n reftable/tree.h | 34 +\n reftable/tree_test.c | 62 +\n reftable/update.sh | 22 +\n reftable/writer.c | 673 +++++++++++\n reftable/writer.h | 51 +\n reftable/zlib-compat.c | 92 ++\n t/helper/test-reftable.c | 20 +\n t/helper/test-tool.c | 2 +\n t/helper/test-tool.h | 2 +\n 56 files changed, 10474 insertions(+), 5 deletions(-)\n create mode 100644 reftable/.gitattributes\n create mode 100644 reftable/LICENSE\n create mode 100644 reftable/VERSION\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/compat.c\n create mode 100644 reftable/compat.h\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/dump.c\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/reftable.c\n create mode 100644 reftable/reftable.h\n create mode 100644 reftable/reftable_test.c\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n create mode 100644 reftable/strbuf.c\n create mode 100644 reftable/strbuf.h\n create mode 100644 reftable/strbuf_test.c\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n create mode 100755 reftable/update.sh\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n create mode 100644 reftable/zlib-compat.c\n create mode 100644 t/helper/test-reftable.c\n\n\nbase-commit: 306ee63a703ad67c54ba1209dc11dd9ea500dc1f\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-847%2Fhanwen%2Flibreftable-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-847/hanwen/libreftable-v2\nPull-Request: https://github.com/git/git/pull/847\n\nRange-diff vs v1:\n\n 1: f697eac228 = 1: 6228103b4a reftable: add LICENSE\n 2: e2dbc84833 = 2: 5d1b946ab5 reftable: define the public API\n 3: 0f6efc065c = 3: 01a669a731 vcxproj: adjust for the reftable changes\n 4: 7bf59d491d ! 4: b94c5f5c61 reftable: add a barebones unittest framework\n @@ reftable/test_framework.c (new)\n +#include \"system.h\"\n +#include \"basics.h\"\n +\n -+struct test_case **test_cases;\n -+int test_case_len;\n -+int test_case_cap;\n ++static struct test_case **test_cases;\n ++static int test_case_len;\n ++static int test_case_cap;\n +\n -+struct test_case *new_test_case(const char *name, void (*testfunc)(void))\n ++static struct test_case *new_test_case(const char *name, void (*testfunc)(void))\n +{\n +\tstruct test_case *tc = reftable_malloc(sizeof(struct test_case));\n +\ttc->name = name;\n @@ reftable/test_framework.c (new)\n +\t\treftable_free(test_cases[i]);\n +\t}\n +\treftable_free(test_cases);\n -+\ttest_cases = 0;\n ++\ttest_cases = NULL;\n +\ttest_case_len = 0;\n +\ttest_case_cap = 0;\n +\treturn 0;\n @@ reftable/test_framework.h (new)\n +\tvoid (*testfunc)(void);\n +};\n +\n -+struct test_case *new_test_case(const char *name, void (*testfunc)(void));\n +struct test_case *add_test_case(const char *name, void (*testfunc)(void));\n +int test_main(int argc, const char *argv[]);\n +\n 5: 570a8c4bca ! 5: 4190da597e reftable: utility functions\n @@ reftable/blocksource.c (new)\n +\treturn ((struct strbuf *)b)->len;\n +}\n +\n -+struct reftable_block_source_vtable strbuf_vtable = {\n ++static struct reftable_block_source_vtable strbuf_vtable = {\n +\t.size = &strbuf_size,\n +\t.read_block = &strbuf_read_block,\n +\t.return_block = &strbuf_return_block,\n @@ reftable/blocksource.c (new)\n +\treftable_free(dest->data);\n +}\n +\n -+struct reftable_block_source_vtable malloc_vtable = {\n ++static struct reftable_block_source_vtable malloc_vtable = {\n +\t.return_block = &malloc_return_block,\n +};\n +\n -+struct reftable_block_source malloc_block_source_instance = {\n ++static struct reftable_block_source malloc_block_source_instance = {\n +\t.ops = &malloc_vtable,\n +};\n +\n @@ reftable/blocksource.c (new)\n +\treturn size;\n +}\n +\n -+struct reftable_block_source_vtable file_vtable = {\n ++static struct reftable_block_source_vtable file_vtable = {\n +\t.size = &file_size,\n +\t.read_block = &file_read_block,\n +\t.return_block = &file_return_block,\n @@ reftable/publicbasics.c (new)\n +\t}\n +}\n +\n -+void *(*reftable_malloc_ptr)(size_t sz) = &malloc;\n -+void *(*reftable_realloc_ptr)(void *, size_t) = &realloc;\n -+void (*reftable_free_ptr)(void *) = &free;\n ++static void *(*reftable_malloc_ptr)(size_t sz) = &malloc;\n ++static void *(*reftable_realloc_ptr)(void *, size_t) = &realloc;\n ++static void (*reftable_free_ptr)(void *) = &free;\n +\n +void *reftable_malloc(size_t sz)\n +{\n 6: 791f69c000 ! 6: 8eb944ea9b reftable: (de)serialization for the polymorphic record type.\n @@ reftable/record.c (new)\n +\t\t(const struct reftable_ref_record *)p);\n +}\n +\n -+struct reftable_record_vtable reftable_ref_record_vtable = {\n ++static struct reftable_record_vtable reftable_ref_record_vtable = {\n +\t.key = &reftable_ref_record_key,\n +\t.type = BLOCK_TYPE_REF,\n +\t.copy_from = &reftable_ref_record_copy_from,\n @@ reftable/record.c (new)\n +\treturn 0;\n +}\n +\n -+struct reftable_record_vtable reftable_obj_record_vtable = {\n ++static struct reftable_record_vtable reftable_obj_record_vtable = {\n +\t.key = &reftable_obj_record_key,\n +\t.type = BLOCK_TYPE_OBJ,\n +\t.copy_from = &reftable_obj_record_copy_from,\n @@ reftable/record.c (new)\n +\t\t(const struct reftable_log_record *)p);\n +}\n +\n -+struct reftable_record_vtable reftable_log_record_vtable = {\n ++static struct reftable_record_vtable reftable_log_record_vtable = {\n +\t.key = &reftable_log_record_key,\n +\t.type = BLOCK_TYPE_LOG,\n +\t.copy_from = &reftable_log_record_copy_from,\n @@ reftable/record.c (new)\n +\treturn rec;\n +}\n +\n -+void *reftable_record_yield(struct reftable_record *rec)\n ++/* clear out the record, yielding the reftable_record data that was\n ++ * encapsulated. */\n ++static void *reftable_record_yield(struct reftable_record *rec)\n +{\n +\tvoid *p = rec->data;\n +\trec->data = NULL;\n @@ reftable/record.c (new)\n +\treturn start.len - in.len;\n +}\n +\n -+struct reftable_record_vtable reftable_index_record_vtable = {\n ++static struct reftable_record_vtable reftable_index_record_vtable = {\n +\t.key = &reftable_index_record_key,\n +\t.type = BLOCK_TYPE_INDEX,\n +\t.copy_from = &reftable_index_record_copy_from,\n @@ reftable/record.h (new)\n +/* creates a malloced record of the given type. Dispose with record_destroy */\n +struct reftable_record reftable_new_record(uint8_t typ);\n +\n -+extern struct reftable_record_vtable reftable_ref_record_vtable;\n -+\n +/* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns\n + number of bytes written. */\n +int reftable_encode_key(int *is_restart, struct string_view dest,\n @@ reftable/record.h (new)\n +/* zeroes out the embedded record */\n +void reftable_record_clear(struct reftable_record *rec);\n +\n -+/* clear out the record, yielding the reftable_record data that was\n -+ * encapsulated. */\n -+void *reftable_record_yield(struct reftable_record *rec);\n -+\n +/* clear and deallocate embedded record, and zero `rec`. */\n +void reftable_record_destroy(struct reftable_record *rec);\n +\n @@ reftable/record_test.c (new)\n +\tint i = 0;\n +\n +\tfor (i = 0; i <= 3; i++) {\n -+\t\tstruct reftable_ref_record in = { 0 };\n ++\t\tstruct reftable_ref_record in = { NULL };\n +\t\tstruct reftable_ref_record out = {\n +\t\t\t.refname = xstrdup(\"old name\"),\n +\t\t\t.value = reftable_calloc(SHA1_SIZE),\n +\t\t\t.target_value = reftable_calloc(SHA1_SIZE),\n +\t\t\t.target = xstrdup(\"old value\"),\n +\t\t};\n -+\t\tstruct reftable_record rec_out = { 0 };\n ++\t\tstruct reftable_record rec_out = { NULL };\n +\t\tstruct strbuf key = STRBUF_INIT;\n -+\t\tstruct reftable_record rec = { 0 };\n ++\t\tstruct reftable_record rec = { NULL };\n +\t\tuint8_t buffer[1024] = { 0 };\n +\t\tstruct string_view dest = {\n +\t\t\t.buf = buffer,\n @@ reftable/record_test.c (new)\n +\tset_test_hash(in[0].new_hash, 1);\n +\tset_test_hash(in[0].old_hash, 2);\n +\tfor (int i = 0; i < ARRAY_SIZE(in); i++) {\n -+\t\tstruct reftable_record rec = { 0 };\n ++\t\tstruct reftable_record rec = { NULL };\n +\t\tstruct strbuf key = STRBUF_INIT;\n +\t\tuint8_t buffer[1024] = { 0 };\n +\t\tstruct string_view dest = {\n @@ reftable/record_test.c (new)\n +\t\t\t.email = xstrdup(\"old@email\"),\n +\t\t\t.message = xstrdup(\"old message\"),\n +\t\t};\n -+\t\tstruct reftable_record rec_out = { 0 };\n ++\t\tstruct reftable_record rec_out = { NULL };\n +\t\tint n, m, valtype;\n +\n +\t\treftable_record_from_log(&rec, &in[i]);\n @@ reftable/record_test.c (new)\n +\t\t\t.buf = buffer,\n +\t\t\t.len = sizeof(buffer),\n +\t\t};\n -+\t\tstruct reftable_record rec = { 0 };\n ++\t\tstruct reftable_record rec = { NULL };\n +\t\tstruct strbuf key = STRBUF_INIT;\n -+\t\tstruct reftable_obj_record out = { 0 };\n -+\t\tstruct reftable_record rec_out = { 0 };\n ++\t\tstruct reftable_obj_record out = { NULL };\n ++\t\tstruct reftable_record rec_out = { NULL };\n +\t\tint n, m;\n +\t\tuint8_t extra;\n +\n @@ reftable/record_test.c (new)\n +\t\t.len = sizeof(buffer),\n +\t};\n +\tstruct strbuf key = STRBUF_INIT;\n -+\tstruct reftable_record rec = { 0 };\n ++\tstruct reftable_record rec = { NULL };\n +\tstruct reftable_index_record out = { .last_key = STRBUF_INIT };\n +\tstruct reftable_record out_rec = { NULL };\n +\tint n, m;\n 7: 1ba8e3eb30 ! 7: 757dd30fe2 reftable: reading/writing blocks\n @@ reftable/block.c (new)\n +\tabort();\n +}\n +\n -+int block_writer_register_restart(struct block_writer *w, int n, int restart,\n -+\t\t\t\t struct strbuf *key);\n ++static int block_writer_register_restart(struct block_writer *w, int n,\n ++\t\t\t\t\t int is_restart, struct strbuf *key)\n ++{\n ++\tint rlen = w->restart_len;\n ++\tif (rlen >= MAX_RESTARTS) {\n ++\t\tis_restart = 0;\n ++\t}\n ++\n ++\tif (is_restart) {\n ++\t\trlen++;\n ++\t}\n ++\tif (2 + 3 * rlen + n > w->block_size - w->next)\n ++\t\treturn -1;\n ++\tif (is_restart) {\n ++\t\tif (w->restart_len == w->restart_cap) {\n ++\t\t\tw->restart_cap = w->restart_cap * 2 + 1;\n ++\t\t\tw->restarts = reftable_realloc(\n ++\t\t\t\tw->restarts, sizeof(uint32_t) * w->restart_cap);\n ++\t\t}\n ++\n ++\t\tw->restarts[w->restart_len++] = w->next;\n ++\t}\n ++\n ++\tw->next += n;\n ++\n ++\tstrbuf_reset(&w->last_key);\n ++\tstrbuf_addbuf(&w->last_key, key);\n ++\tw->entries++;\n ++\treturn 0;\n ++}\n +\n +void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n +\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n @@ reftable/block.c (new)\n +\treturn -1;\n +}\n +\n -+int block_writer_register_restart(struct block_writer *w, int n, int is_restart,\n -+\t\t\t\t struct strbuf *key)\n -+{\n -+\tint rlen = w->restart_len;\n -+\tif (rlen >= MAX_RESTARTS) {\n -+\t\tis_restart = 0;\n -+\t}\n -+\n -+\tif (is_restart) {\n -+\t\trlen++;\n -+\t}\n -+\tif (2 + 3 * rlen + n > w->block_size - w->next)\n -+\t\treturn -1;\n -+\tif (is_restart) {\n -+\t\tif (w->restart_len == w->restart_cap) {\n -+\t\t\tw->restart_cap = w->restart_cap * 2 + 1;\n -+\t\t\tw->restarts = reftable_realloc(\n -+\t\t\t\tw->restarts, sizeof(uint32_t) * w->restart_cap);\n -+\t\t}\n -+\n -+\t\tw->restarts[w->restart_len++] = w->next;\n -+\t}\n -+\n -+\tw->next += n;\n -+\n -+\tstrbuf_reset(&w->last_key);\n -+\tstrbuf_addbuf(&w->last_key, key);\n -+\tw->entries++;\n -+\treturn 0;\n -+}\n +\n +int block_writer_finish(struct block_writer *w)\n +{\n @@ reftable/block_test.c (new)\n +\tchar *names[30];\n +\tconst int N = ARRAY_SIZE(names);\n +\tconst int block_size = 1024;\n -+\tstruct reftable_block block = { 0 };\n ++\tstruct reftable_block block = { NULL };\n +\tstruct block_writer bw = {\n +\t\t.last_key = STRBUF_INIT,\n +\t};\n -+\tstruct reftable_ref_record ref = { 0 };\n -+\tstruct reftable_record rec = { 0 };\n ++\tstruct reftable_ref_record ref = { NULL };\n ++\tstruct reftable_record rec = { NULL };\n +\tint i = 0;\n +\tint n;\n +\tstruct block_reader br = { 0 };\n 8: 17fb8d050d ! 8: e30a7e0281 reftable: a generic binary tree implementation\n @@ reftable/tree_test.c (new)\n +{\n +\tstruct tree_node *root = NULL;\n +\n -+\tvoid *values[11] = { 0 };\n -+\tstruct tree_node *nodes[11] = { 0 };\n ++\tvoid *values[11] = { NULL };\n ++\tstruct tree_node *nodes[11] = { NULL };\n +\tint i = 1;\n -+\tstruct curry c = { 0 };\n ++\tstruct curry c = { NULL };\n +\tdo {\n +\t\tnodes[i] = tree_search(values + i, &root, &test_compare, 1);\n +\t\ti = (i * 7) % 11;\n 9: 4e40fc3b40 ! 9: 68aee16e60 reftable: write reftable files\n @@ reftable/writer.c (new)\n +#include \"reftable.h\"\n +#include \"tree.h\"\n +\n ++/* finishes a block, and writes it to storage */\n ++static int writer_flush_block(struct reftable_writer *w);\n ++\n ++/* deallocates memory related to the index */\n ++static void writer_clear_index(struct reftable_writer *w);\n ++\n ++/* finishes writing a 'r' (refs) or 'g' (reflogs) section */\n ++static int writer_finish_public_section(struct reftable_writer *w);\n ++\n +static struct reftable_block_stats *\n +writer_reftable_block_stats(struct reftable_writer *w, uint8_t typ)\n +{\n @@ reftable/writer.c (new)\n +int reftable_writer_add_ref(struct reftable_writer *w,\n +\t\t\t struct reftable_ref_record *ref)\n +{\n -+\tstruct reftable_record rec = { 0 };\n ++\tstruct reftable_record rec = { NULL };\n +\tstruct reftable_ref_record copy = *ref;\n +\tint err = 0;\n +\n @@ reftable/writer.c (new)\n +int reftable_writer_add_log(struct reftable_writer *w,\n +\t\t\t struct reftable_log_record *log)\n +{\n -+\tstruct reftable_record rec = { 0 };\n ++\tstruct reftable_record rec = { NULL };\n +\tchar *input_log_message = log->message;\n +\tstruct strbuf cleaned_message = STRBUF_INIT;\n +\tint err;\n @@ reftable/writer.c (new)\n +\t\tw->index_len = 0;\n +\t\tw->index_cap = 0;\n +\t\tfor (i = 0; i < idx_len; i++) {\n -+\t\t\tstruct reftable_record rec = { 0 };\n ++\t\t\tstruct reftable_record rec = { NULL };\n +\t\t\treftable_record_from_index(&rec, idx + i);\n +\t\t\tif (block_writer_add(w->block_writer, &rec) == 0) {\n +\t\t\t\tcontinue;\n @@ reftable/writer.c (new)\n +\t\t.offsets = entry->offsets,\n +\t\t.offset_len = entry->offset_len,\n +\t};\n -+\tstruct reftable_record rec = { 0 };\n ++\tstruct reftable_record rec = { NULL };\n +\tif (arg->err < 0)\n +\t\tgoto done;\n +\n @@ reftable/writer.c (new)\n +static int writer_dump_object_index(struct reftable_writer *w)\n +{\n +\tstruct write_record_arg closure = { .w = w };\n -+\tstruct common_prefix_arg common = { 0 };\n ++\tstruct common_prefix_arg common = { NULL };\n +\tif (w->obj_index_tree != NULL) {\n +\t\tinfix_walk(w->obj_index_tree, &update_common, &common);\n +\t}\n @@ reftable/writer.c (new)\n +\treturn writer_finish_section(w);\n +}\n +\n -+int writer_finish_public_section(struct reftable_writer *w)\n ++static int writer_finish_public_section(struct reftable_writer *w)\n +{\n +\tuint8_t typ = 0;\n +\tint err = 0;\n @@ reftable/writer.c (new)\n +\treturn err;\n +}\n +\n -+void writer_clear_index(struct reftable_writer *w)\n ++static void writer_clear_index(struct reftable_writer *w)\n +{\n +\tint i = 0;\n +\tfor (i = 0; i < w->index_len; i++) {\n @@ reftable/writer.c (new)\n +\tw->index_cap = 0;\n +}\n +\n -+const int debug = 0;\n ++static const int debug = 0;\n +\n +static int writer_flush_nonempty_block(struct reftable_writer *w)\n +{\n @@ reftable/writer.c (new)\n +\treturn 0;\n +}\n +\n -+int writer_flush_block(struct reftable_writer *w)\n ++static int writer_flush_block(struct reftable_writer *w)\n +{\n +\tif (w->block_writer == NULL)\n +\t\treturn 0;\n @@ reftable/writer.h (new)\n +\tstruct reftable_stats stats;\n +};\n +\n -+/* finishes a block, and writes it to storage */\n -+int writer_flush_block(struct reftable_writer *w);\n -+\n -+/* deallocates memory related to the index */\n -+void writer_clear_index(struct reftable_writer *w);\n -+\n -+/* finishes writing a 'r' (refs) or 'g' (reflogs) section */\n -+int writer_finish_public_section(struct reftable_writer *w);\n -+\n +#endif\n 10: 25259b683b ! 10: c196de7f06 reftable: read reftable files\n @@ reftable/iter.c (new)\n +{\n +}\n +\n -+struct reftable_iterator_vtable empty_vtable = {\n ++static struct reftable_iterator_vtable empty_vtable = {\n +\t.next = &empty_iterator_next,\n +\t.close = &empty_iterator_close,\n +};\n @@ reftable/iter.c (new)\n +\treturn err;\n +}\n +\n -+struct reftable_iterator_vtable filtering_ref_iterator_vtable = {\n ++static struct reftable_iterator_vtable filtering_ref_iterator_vtable = {\n +\t.next = &filtering_ref_iterator_next,\n +\t.close = &filtering_ref_iterator_close,\n +};\n @@ reftable/iter.c (new)\n +\treturn err;\n +}\n +\n -+struct reftable_iterator_vtable indexed_table_ref_iter_vtable = {\n ++static struct reftable_iterator_vtable indexed_table_ref_iter_vtable = {\n +\t.next = &indexed_table_ref_iter_next,\n +\t.close = &indexed_table_ref_iter_close,\n +};\n @@ reftable/reader.c (new)\n +\tblock_iter_close(&ti->bi);\n +}\n +\n -+struct reftable_iterator_vtable table_iter_vtable = {\n ++static struct reftable_iterator_vtable table_iter_vtable = {\n +\t.next = &table_iter_next_void,\n +\t.close = &table_iter_close,\n +};\n 11: 3448fcf828 ! 11: bf6b929b86 reftable: file level tests\n @@ reftable/reftable_test.c (new)\n +{\n +\tstruct strbuf buf = STRBUF_INIT;\n +\tstruct reftable_block_source source = { NULL };\n -+\tstruct reftable_block out = { 0 };\n ++\tstruct reftable_block out = { NULL };\n +\tint n;\n +\tuint8_t in[] = \"hello\";\n +\tstrbuf_add(&buf, in, sizeof(in));\n @@ reftable/reftable_test.c (new)\n +\t};\n +\tstruct reftable_writer *w =\n +\t\treftable_new_writer(&strbuf_add_void, buf, &opts);\n -+\tstruct reftable_ref_record ref = { 0 };\n ++\tstruct reftable_ref_record ref = { NULL };\n +\tint i = 0, n;\n -+\tstruct reftable_log_record log = { 0 };\n ++\tstruct reftable_log_record log = { NULL };\n +\tconst struct reftable_stats *stats = NULL;\n +\t*names = reftable_calloc(sizeof(char *) * (N + 1));\n +\treftable_writer_set_limits(w, update_index, update_index);\n @@ reftable/reftable_test.c (new)\n +\tstruct reftable_write_options opts = {\n +\t\t.block_size = 256,\n +\t};\n -+\tstruct reftable_ref_record ref = { 0 };\n ++\tstruct reftable_ref_record ref = { NULL };\n +\tint i = 0;\n -+\tstruct reftable_log_record log = { 0 };\n ++\tstruct reftable_log_record log = { NULL };\n +\tint n;\n -+\tstruct reftable_iterator it = { 0 };\n -+\tstruct reftable_reader rd = { 0 };\n -+\tstruct reftable_block_source source = { 0 };\n ++\tstruct reftable_iterator it = { NULL };\n ++\tstruct reftable_reader rd = { NULL };\n ++\tstruct reftable_block_source source = { NULL };\n +\tstruct strbuf buf = STRBUF_INIT;\n +\tstruct reftable_writer *w =\n +\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n @@ reftable/reftable_test.c (new)\n +\treftable_writer_set_limits(w, 0, N);\n +\tfor (i = 0; i < N; i++) {\n +\t\tchar name[256];\n -+\t\tstruct reftable_ref_record ref = { 0 };\n ++\t\tstruct reftable_ref_record ref = { NULL };\n +\t\tsnprintf(name, sizeof(name), \"b%02d%0*d\", i, 130, 7);\n +\t\tnames[i] = xstrdup(name);\n +\t\tref.refname = name;\n @@ reftable/reftable_test.c (new)\n +\t}\n +\tfor (i = 0; i < N; i++) {\n +\t\tuint8_t hash1[SHA1_SIZE], hash2[SHA1_SIZE];\n -+\t\tstruct reftable_log_record log = { 0 };\n ++\t\tstruct reftable_log_record log = { NULL };\n +\t\tset_test_hash(hash1, i);\n +\t\tset_test_hash(hash2, i + 1);\n +\n @@ reftable/reftable_test.c (new)\n +\tchar **names;\n +\tstruct strbuf buf = STRBUF_INIT;\n +\tint N = 50;\n -+\tstruct reftable_iterator it = { 0 };\n -+\tstruct reftable_block_source source = { 0 };\n -+\tstruct reftable_reader rd = { 0 };\n ++\tstruct reftable_iterator it = { NULL };\n ++\tstruct reftable_block_source source = { NULL };\n ++\tstruct reftable_reader rd = { NULL };\n +\tint err = 0;\n +\tint j = 0;\n +\n @@ reftable/reftable_test.c (new)\n +\tassert_err(err);\n +\n +\twhile (1) {\n -+\t\tstruct reftable_ref_record ref = { 0 };\n ++\t\tstruct reftable_ref_record ref = { NULL };\n +\t\tint r = reftable_iterator_next_ref(&it, &ref);\n +\t\tassert(r >= 0);\n +\t\tif (r > 0) {\n @@ reftable/reftable_test.c (new)\n +\tchar **names;\n +\tstruct strbuf buf = STRBUF_INIT;\n +\tint N = 50;\n -+\tstruct reftable_reader rd = { 0 };\n -+\tstruct reftable_block_source source = { 0 };\n ++\tstruct reftable_reader rd = { NULL };\n ++\tstruct reftable_block_source source = { NULL };\n +\tint err;\n +\tint i;\n -+\tstruct reftable_log_record log = { 0 };\n -+\tstruct reftable_iterator it = { 0 };\n ++\tstruct reftable_log_record log = { NULL };\n ++\tstruct reftable_iterator it = { NULL };\n +\n +\twrite_table(&names, &buf, N, 256, SHA1_ID);\n +\n @@ reftable/reftable_test.c (new)\n +\tchar **names;\n +\tstruct strbuf buf = STRBUF_INIT;\n +\tint N = 50;\n -+\tstruct reftable_reader rd = { 0 };\n -+\tstruct reftable_block_source source = { 0 };\n ++\tstruct reftable_reader rd = { NULL };\n ++\tstruct reftable_block_source source = { NULL };\n +\tint err;\n +\tint i = 0;\n +\n -+\tstruct reftable_iterator it = { 0 };\n ++\tstruct reftable_iterator it = { NULL };\n +\tstruct strbuf pastLast = STRBUF_INIT;\n -+\tstruct reftable_ref_record ref = { 0 };\n ++\tstruct reftable_ref_record ref = { NULL };\n +\n +\twrite_table(&names, &buf, N, 256, hash_id);\n +\n @@ reftable/reftable_test.c (new)\n +\n +\terr = reftable_reader_seek_ref(&rd, &it, pastLast.buf);\n +\tif (err == 0) {\n -+\t\tstruct reftable_ref_record ref = { 0 };\n ++\t\tstruct reftable_ref_record ref = { NULL };\n +\t\tint err = reftable_iterator_next_ref(&it, &ref);\n +\t\tassert(err > 0);\n +\t} else {\n @@ reftable/reftable_test.c (new)\n +\tstruct reftable_write_options opts = {\n +\t\t.block_size = 256,\n +\t};\n -+\tstruct reftable_ref_record ref = { 0 };\n ++\tstruct reftable_ref_record ref = { NULL };\n +\tint i = 0;\n +\tint n;\n +\tint err;\n +\tstruct reftable_reader rd;\n -+\tstruct reftable_block_source source = { 0 };\n ++\tstruct reftable_block_source source = { NULL };\n +\n +\tstruct strbuf buf = STRBUF_INIT;\n +\tstruct reftable_writer *w =\n +\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n +\n -+\tstruct reftable_iterator it = { 0 };\n ++\tstruct reftable_iterator it = { NULL };\n +\tint j;\n +\n +\tset_test_hash(want_hash, 4);\n @@ reftable/reftable_test.c (new)\n +\t\tchar name[100];\n +\t\tuint8_t hash1[SHA1_SIZE];\n +\t\tuint8_t hash2[SHA1_SIZE];\n -+\t\tstruct reftable_ref_record ref = { 0 };\n ++\t\tstruct reftable_ref_record ref = { NULL };\n +\n +\t\tmemset(hash, i, sizeof(hash));\n +\t\tmemset(fill, 'x', 50);\n @@ reftable/reftable_test.c (new)\n +\tstruct strbuf buf = STRBUF_INIT;\n +\tstruct reftable_writer *w =\n +\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n -+\tstruct reftable_block_source source = { 0 };\n ++\tstruct reftable_block_source source = { NULL };\n +\tstruct reftable_reader *rd = NULL;\n -+\tstruct reftable_ref_record rec = { 0 };\n -+\tstruct reftable_iterator it = { 0 };\n ++\tstruct reftable_ref_record rec = { NULL };\n ++\tstruct reftable_iterator it = { NULL };\n +\tint err;\n +\n +\treftable_writer_set_limits(w, 1, 1);\n 12: 64d98e60b2 ! 12: 4e38db7f48 reftable: rest of library\n @@ Makefile: REFTABLE_OBJS += reftable/zlib-compat.o\n \n ## reftable/VERSION (new) ##\n @@\n -+b8ec0f74c74cb6752eb2033ad8e755a9c19aad15 C: add missing header\n ++7134eb9f8171a9759800f4187f9e6dde997335e7 C: NULL iso 0 for init\n \n ## reftable/dump.c (new) ##\n @@\n @@ reftable/dump.c (new)\n +\n +static int dump_table(const char *tablename)\n +{\n -+\tstruct reftable_block_source src = { 0 };\n ++\tstruct reftable_block_source src = { NULL };\n +\tint err = reftable_block_source_from_file(&src, tablename);\n -+\tstruct reftable_iterator it = { 0 };\n -+\tstruct reftable_ref_record ref = { 0 };\n -+\tstruct reftable_log_record log = { 0 };\n ++\tstruct reftable_iterator it = { NULL };\n ++\tstruct reftable_ref_record ref = { NULL };\n ++\tstruct reftable_log_record log = { NULL };\n +\tstruct reftable_reader *r = NULL;\n +\n +\tif (err < 0)\n @@ reftable/dump.c (new)\n +{\n +\tstruct reftable_stack *stack = NULL;\n +\tstruct reftable_write_options cfg = {};\n -+\tstruct reftable_iterator it = { 0 };\n -+\tstruct reftable_ref_record ref = { 0 };\n -+\tstruct reftable_log_record log = { 0 };\n ++\tstruct reftable_iterator it = { NULL };\n ++\tstruct reftable_ref_record ref = { NULL };\n ++\tstruct reftable_log_record log = { NULL };\n +\tstruct reftable_merged_table *merged = NULL;\n +\n +\tint err = reftable_new_stack(&stack, stackdir, cfg);\n @@ reftable/merged.c (new)\n +\treturn merged_iter_next(mi, rec);\n +}\n +\n -+struct reftable_iterator_vtable merged_iter_vtable = {\n ++static struct reftable_iterator_vtable merged_iter_vtable = {\n +\t.next = &merged_iter_next_void,\n +\t.close = &merged_iter_close,\n +};\n @@ reftable/merged.c (new)\n +\treturn mt->min;\n +}\n +\n -+int merged_table_seek_record(struct reftable_merged_table *mt,\n -+\t\t\t struct reftable_iterator *it,\n -+\t\t\t struct reftable_record *rec)\n ++static int merged_table_seek_record(struct reftable_merged_table *mt,\n ++\t\t\t\t struct reftable_iterator *it,\n ++\t\t\t\t struct reftable_record *rec)\n +{\n +\tstruct reftable_iterator *iters = reftable_calloc(\n +\t\tsizeof(struct reftable_iterator) * mt->stack_len);\n @@ reftable/merged.c (new)\n +\tstruct reftable_ref_record ref = {\n +\t\t.refname = (char *)name,\n +\t};\n -+\tstruct reftable_record rec = { 0 };\n ++\tstruct reftable_record rec = { NULL };\n +\treftable_record_from_ref(&rec, &ref);\n +\treturn merged_table_seek_record(mt, it, &rec);\n +}\n @@ reftable/merged.c (new)\n +\t\t.refname = (char *)name,\n +\t\t.update_index = update_index,\n +\t};\n -+\tstruct reftable_record rec = { 0 };\n ++\tstruct reftable_record rec = { NULL };\n +\treftable_record_from_log(&rec, &log);\n +\treturn merged_table_seek_record(mt, it, &rec);\n +}\n @@ reftable/merged.h (new)\n +};\n +\n +void merged_table_clear(struct reftable_merged_table *mt);\n -+int merged_table_seek_record(struct reftable_merged_table *mt,\n -+\t\t\t struct reftable_iterator *it,\n -+\t\t\t struct reftable_record *rec);\n +\n +#endif\n \n @@ reftable/merged_test.c (new)\n +\n +static void test_pq(void)\n +{\n -+\tchar *names[54] = { 0 };\n ++\tchar *names[54] = { NULL };\n +\tint N = ARRAY_SIZE(names) - 1;\n +\n -+\tstruct merged_iter_pqueue pq = { 0 };\n ++\tstruct merged_iter_pqueue pq = { NULL };\n +\tconst char *last = NULL;\n +\n +\tint i = 0;\n @@ reftable/merged_test.c (new)\n +\tstruct reftable_merged_table *mt =\n +\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 2);\n +\tint i;\n -+\tstruct reftable_ref_record ref = { 0 };\n -+\tstruct reftable_iterator it = { 0 };\n ++\tstruct reftable_ref_record ref = { NULL };\n ++\tstruct reftable_iterator it = { NULL };\n +\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n +\tassert_err(err);\n +\n @@ reftable/merged_test.c (new)\n +\tstruct reftable_merged_table *mt =\n +\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 3);\n +\n -+\tstruct reftable_iterator it = { 0 };\n ++\tstruct reftable_iterator it = { NULL };\n +\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n +\tstruct reftable_ref_record *out = NULL;\n +\tsize_t len = 0;\n @@ reftable/merged_test.c (new)\n +\n +\tassert_err(err);\n +\twhile (len < 100) { /* cap loops/recursion. */\n -+\t\tstruct reftable_ref_record ref = { 0 };\n ++\t\tstruct reftable_ref_record ref = { NULL };\n +\t\tint err = reftable_iterator_next_ref(&it, &ref);\n +\t\tif (err > 0) {\n +\t\t\tbreak;\n @@ reftable/merged_test.c (new)\n +\t\t.update_index = 1,\n +\t};\n +\tint err;\n -+\tstruct reftable_block_source source = { 0 };\n ++\tstruct reftable_block_source source = { NULL };\n +\tstruct reftable_table *tab = reftable_calloc(sizeof(*tab) * 1);\n +\tuint32_t hash_id;\n +\tstruct reftable_reader *rd = NULL;\n @@ reftable/pq.c (new)\n +#include \"system.h\"\n +#include \"basics.h\"\n +\n -+int pq_less(struct pq_entry a, struct pq_entry b)\n ++static int pq_less(struct pq_entry a, struct pq_entry b)\n +{\n +\tstruct strbuf ak = STRBUF_INIT;\n +\tstruct strbuf bk = STRBUF_INIT;\n @@ reftable/pq.h (new)\n +\tstruct reftable_record rec;\n +};\n +\n -+int pq_less(struct pq_entry a, struct pq_entry b);\n -+\n +struct merged_iter_pqueue {\n +\tstruct pq_entry *heap;\n +\tsize_t len;\n @@ reftable/refname.c (new)\n +\treturn strcmp(f_arg->names[k], f_arg->want) >= 0;\n +}\n +\n -+int modification_has_ref(struct modification *mod, const char *name)\n ++static int modification_has_ref(struct modification *mod, const char *name)\n +{\n -+\tstruct reftable_ref_record ref = { 0 };\n ++\tstruct reftable_ref_record ref = { NULL };\n +\tint err = 0;\n +\n +\tif (mod->add_len > 0) {\n @@ reftable/refname.c (new)\n +\tmod->del_len = 0;\n +}\n +\n -+int modification_has_ref_with_prefix(struct modification *mod,\n -+\t\t\t\t const char *prefix)\n ++static int modification_has_ref_with_prefix(struct modification *mod,\n ++\t\t\t\t\t const char *prefix)\n +{\n +\tstruct reftable_iterator it = { NULL };\n +\tstruct reftable_ref_record ref = { NULL };\n @@ reftable/refname.c (new)\n +\treturn err;\n +}\n +\n -+int validate_refname(const char *name)\n ++static int validate_refname(const char *name)\n +{\n +\twhile (1) {\n +\t\tchar *next = strchr(name, '/');\n @@ reftable/refname.h (new)\n +\tsize_t del_len;\n +};\n +\n -+// -1 = error, 0 = found, 1 = not found\n -+int modification_has_ref(struct modification *mod, const char *name);\n -+\n -+// -1 = error, 0 = found, 1 = not found.\n -+int modification_has_ref_with_prefix(struct modification *mod,\n -+\t\t\t\t const char *prefix);\n -+\n -+// 0 = OK.\n -+int validate_refname(const char *name);\n -+\n +int validate_ref_record_addition(struct reftable_table tab,\n +\t\t\t\t struct reftable_ref_record *recs, size_t sz);\n +\n @@ reftable/refname_test.c (new)\n +\t};\n +\tint err;\n +\tint i;\n -+\tstruct reftable_block_source source = { 0 };\n ++\tstruct reftable_block_source source = { NULL };\n +\tstruct reftable_reader *rd = NULL;\n +\tstruct reftable_table tab = { NULL };\n +\tstruct testcase cases[] = {\n @@ reftable/refname_test.c (new)\n +\treturn test_main(argc, argv);\n +}\n \n + ## reftable/reftable.c ##\n +@@ reftable/reftable.c: int reftable_table_seek_ref(struct reftable_table *tab,\n + \tstruct reftable_ref_record ref = {\n + \t\t.refname = (char *)name,\n + \t};\n +-\tstruct reftable_record rec = { 0 };\n ++\tstruct reftable_record rec = { NULL };\n + \treftable_record_from_ref(&rec, &ref);\n + \treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n + }\n +@@ reftable/reftable.c: void reftable_table_from_reader(struct reftable_table *tab,\n + int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n + \t\t\t struct reftable_ref_record *ref)\n + {\n +-\tstruct reftable_iterator it = { 0 };\n ++\tstruct reftable_iterator it = { NULL };\n + \tint err = reftable_table_seek_ref(tab, &it, name);\n + \tif (err)\n + \t\tgoto done;\n +\n ## reftable/stack.c (new) ##\n @@\n +/*\n @@ reftable/stack.c (new)\n +#include \"reftable.h\"\n +#include \"writer.h\"\n +\n ++static int stack_try_add(struct reftable_stack *st,\n ++\t\t\t int (*write_table)(struct reftable_writer *wr,\n ++\t\t\t\t\t void *arg),\n ++\t\t\t void *arg);\n ++static int stack_write_compact(struct reftable_stack *st,\n ++\t\t\t struct reftable_writer *wr, int first, int last,\n ++\t\t\t struct reftable_log_expiry_config *config);\n ++static int stack_check_addition(struct reftable_stack *st,\n ++\t\t\t\tconst char *new_tab_name);\n ++static void reftable_addition_close(struct reftable_addition *add);\n ++static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n ++\t\t\t\t\t int reuse_open);\n ++\n +int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n +\t\t struct reftable_write_options config)\n +{\n @@ reftable/stack.c (new)\n +\t\t}\n +\n +\t\tif (rd == NULL) {\n -+\t\t\tstruct reftable_block_source src = { 0 };\n ++\t\t\tstruct reftable_block_source src = { NULL };\n +\t\t\tstruct strbuf table_path = STRBUF_INIT;\n +\t\t\tstrbuf_addstr(&table_path, st->reftable_dir);\n +\t\t\tstrbuf_addstr(&table_path, \"/\");\n @@ reftable/stack.c (new)\n +\treturn udiff;\n +}\n +\n -+int reftable_stack_reload_maybe_reuse(struct reftable_stack *st, int reuse_open)\n ++static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n ++\t\t\t\t\t int reuse_open)\n +{\n +\tstruct timeval deadline = { 0 };\n +\tint err = gettimeofday(&deadline, NULL);\n @@ reftable/stack.c (new)\n +\treturn err;\n +}\n +\n -+void reftable_addition_close(struct reftable_addition *add)\n ++static void reftable_addition_close(struct reftable_addition *add)\n +{\n +\tint i = 0;\n +\tstruct strbuf nm = STRBUF_INIT;\n @@ reftable/stack.c (new)\n +\treturn err;\n +}\n +\n -+int stack_try_add(struct reftable_stack *st,\n -+\t\t int (*write_table)(struct reftable_writer *wr, void *arg),\n -+\t\t void *arg)\n ++static int stack_try_add(struct reftable_stack *st,\n ++\t\t\t int (*write_table)(struct reftable_writer *wr,\n ++\t\t\t\t\t void *arg),\n ++\t\t\t void *arg)\n +{\n +\tstruct reftable_addition add = REFTABLE_ADDITION_INIT;\n +\tint err = reftable_stack_init_addition(&add, st);\n @@ reftable/stack.c (new)\n +\treturn err;\n +}\n +\n -+int stack_write_compact(struct reftable_stack *st, struct reftable_writer *wr,\n -+\t\t\tint first, int last,\n -+\t\t\tstruct reftable_log_expiry_config *config)\n ++static int stack_write_compact(struct reftable_stack *st,\n ++\t\t\t struct reftable_writer *wr, int first, int last,\n ++\t\t\t struct reftable_log_expiry_config *config)\n +{\n +\tint subtabs_len = last - first + 1;\n +\tstruct reftable_table *subtabs = reftable_calloc(\n +\t\tsizeof(struct reftable_table) * (last - first + 1));\n +\tstruct reftable_merged_table *mt = NULL;\n +\tint err = 0;\n -+\tstruct reftable_iterator it = { 0 };\n -+\tstruct reftable_ref_record ref = { 0 };\n -+\tstruct reftable_log_record log = { 0 };\n ++\tstruct reftable_iterator it = { NULL };\n ++\tstruct reftable_ref_record ref = { NULL };\n ++\tstruct reftable_log_record log = { NULL };\n +\n +\tuint64_t entries = 0;\n +\n @@ reftable/stack.c (new)\n +int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n +\t\t\t struct reftable_log_record *log)\n +{\n -+\tstruct reftable_iterator it = { 0 };\n ++\tstruct reftable_iterator it = { NULL };\n +\tstruct reftable_merged_table *mt = reftable_stack_merged_table(st);\n +\tint err = reftable_merged_table_seek_log(mt, &it, refname);\n +\tif (err)\n @@ reftable/stack.c (new)\n +\treturn err;\n +}\n +\n -+int stack_check_addition(struct reftable_stack *st, const char *new_tab_name)\n ++static int stack_check_addition(struct reftable_stack *st,\n ++\t\t\t\tconst char *new_tab_name)\n +{\n +\tint err = 0;\n -+\tstruct reftable_block_source src = { 0 };\n ++\tstruct reftable_block_source src = { NULL };\n +\tstruct reftable_reader *rd = NULL;\n +\tstruct reftable_table tab = { NULL };\n +\tstruct reftable_ref_record *refs = NULL;\n @@ reftable/stack.c (new)\n +\t\tgoto done;\n +\n +\twhile (1) {\n -+\t\tstruct reftable_ref_record ref = { 0 };\n ++\t\tstruct reftable_ref_record ref = { NULL };\n +\t\terr = reftable_iterator_next_ref(&it, &ref);\n +\t\tif (err > 0) {\n +\t\t\tbreak;\n @@ reftable/stack.h (new)\n +};\n +\n +int read_lines(const char *filename, char ***lines);\n -+int stack_try_add(struct reftable_stack *st,\n -+\t\t int (*write_table)(struct reftable_writer *wr, void *arg),\n -+\t\t void *arg);\n -+int stack_write_compact(struct reftable_stack *st, struct reftable_writer *wr,\n -+\t\t\tint first, int last,\n -+\t\t\tstruct reftable_log_expiry_config *config);\n -+int fastlog2(uint64_t sz);\n -+int stack_check_addition(struct reftable_stack *st, const char *new_tab_name);\n -+void reftable_addition_close(struct reftable_addition *add);\n -+int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n -+\t\t\t\t int reuse_open);\n +\n +struct segment {\n +\tint start, end;\n @@ reftable/stack.h (new)\n +\tuint64_t bytes;\n +};\n +\n ++int fastlog2(uint64_t sz);\n +struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n);\n +struct segment suggest_compaction_segment(uint64_t *sizes, int n);\n +\n @@ reftable/stack_test.c (new)\n +\t\t.update_index = 1,\n +\t\t.target = \"master\",\n +\t};\n -+\tstruct reftable_ref_record dest = { 0 };\n ++\tstruct reftable_ref_record dest = { NULL };\n +\n +\tassert(mkdtemp(dir));\n +\n @@ reftable/stack_test.c (new)\n +\t\t.update_index = 1,\n +\t\t.target = \"master\",\n +\t};\n -+\tstruct reftable_ref_record dest = { 0 };\n ++\tstruct reftable_ref_record dest = { NULL };\n +\n +\tassert(mkdtemp(dir));\n +\n @@ reftable/stack_test.c (new)\n +\t};\n +\tstruct reftable_stack *st = NULL;\n +\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n -+\tstruct reftable_ref_record refs[2] = { { 0 } };\n -+\tstruct reftable_log_record logs[2] = { { 0 } };\n ++\tstruct reftable_ref_record refs[2] = { { NULL } };\n ++\tstruct reftable_log_record logs[2] = { { NULL } };\n +\tint N = ARRAY_SIZE(refs);\n +\n +\tassert(mkdtemp(dir));\n @@ reftable/stack_test.c (new)\n +\tassert_err(err);\n +\n +\tfor (i = 0; i < N; i++) {\n -+\t\tstruct reftable_ref_record dest = { 0 };\n ++\t\tstruct reftable_ref_record dest = { NULL };\n ++\n +\t\tint err = reftable_stack_read_ref(st, refs[i].refname, &dest);\n +\t\tassert_err(err);\n +\t\tassert(reftable_ref_record_equal(&dest, refs + i, SHA1_SIZE));\n @@ reftable/stack_test.c (new)\n +\t}\n +\n +\tfor (i = 0; i < N; i++) {\n -+\t\tstruct reftable_log_record dest = { 0 };\n ++\t\tstruct reftable_log_record dest = { NULL };\n +\t\tint err = reftable_stack_read_log(st, refs[i].refname, &dest);\n +\t\tassert_err(err);\n +\t\tassert(reftable_log_record_equal(&dest, logs + i, SHA1_SIZE));\n @@ reftable/stack_test.c (new)\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tint err;\n -+\tstruct reftable_ref_record refs[2] = { { 0 } };\n -+\tstruct reftable_log_record logs[2] = { { 0 } };\n ++\tstruct reftable_ref_record refs[2] = { { NULL } };\n ++\tstruct reftable_log_record logs[2] = { { NULL } };\n +\tint N = ARRAY_SIZE(refs);\n -+\tstruct reftable_ref_record dest = { 0 };\n -+\tstruct reftable_log_record log_dest = { 0 };\n ++\tstruct reftable_ref_record dest = { NULL };\n ++\tstruct reftable_log_record log_dest = { NULL };\n +\n +\tassert(mkdtemp(dir));\n +\n @@ reftable/stack_test.c (new)\n +\tstruct reftable_stack *st32 = NULL;\n +\tstruct reftable_write_options cfg_default = { 0 };\n +\tstruct reftable_stack *st_default = NULL;\n -+\tstruct reftable_ref_record dest = { 0 };\n ++\tstruct reftable_ref_record dest = { NULL };\n +\n +\tassert(mkdtemp(dir));\n +\terr = reftable_new_stack(&st, dir, cfg);\n @@ reftable/stack_test.c (new)\n +\tchar dir[256] = \"/tmp/stack.test_reflog_expire.XXXXXX\";\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n -+\tstruct reftable_log_record logs[20] = { { 0 } };\n ++\tstruct reftable_log_record logs[20] = { { NULL } };\n +\tint N = ARRAY_SIZE(logs) - 1;\n +\tint i = 0;\n +\tint err;\n +\tstruct reftable_log_expiry_config expiry = {\n +\t\t.time = 10,\n +\t};\n -+\tstruct reftable_log_record log = { 0 };\n ++\tstruct reftable_log_record log = { NULL };\n +\n +\tassert(mkdtemp(dir));\n +\n 13: b8729fee9e ! 13: c535f838d6 reftable: \"test-tool dump-reftable\" command.\n @@ Makefile: REFTABLE_OBJS += reftable/writer.o\n REFTABLE_TEST_OBJS += reftable/refname_test.o\n REFTABLE_TEST_OBJS += reftable/reftable_test.o\n \n + ## reftable/iter.c ##\n +@@ reftable/iter.c: void reftable_iterator_destroy(struct reftable_iterator *it)\n + int reftable_iterator_next_ref(struct reftable_iterator *it,\n + \t\t\t struct reftable_ref_record *ref)\n + {\n +-\tstruct reftable_record rec = { 0 };\n ++\tstruct reftable_record rec = { NULL };\n + \treftable_record_from_ref(&rec, ref);\n + \treturn iterator_next(it, &rec);\n + }\n +@@ reftable/iter.c: int reftable_iterator_next_ref(struct reftable_iterator *it,\n + int reftable_iterator_next_log(struct reftable_iterator *it,\n + \t\t\t struct reftable_log_record *log)\n + {\n +-\tstruct reftable_record rec = { 0 };\n ++\tstruct reftable_record rec = { NULL };\n + \treftable_record_from_log(&rec, log);\n + \treturn iterator_next(it, &rec);\n + }\n +@@ reftable/iter.c: static int filtering_ref_iterator_next(void *iter_arg,\n + \t\t}\n + \n + \t\tif (fri->double_check) {\n +-\t\t\tstruct reftable_iterator it = { 0 };\n ++\t\t\tstruct reftable_iterator it = { NULL };\n + \n + \t\t\terr = reftable_table_seek_ref(&fri->tab, &it,\n + \t\t\t\t\t\t ref->refname);\n +\n + ## reftable/reader.c ##\n +@@ reftable/reader.c: static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n + int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n + \t\tconst char *name)\n + {\n +-\tstruct reftable_block footer = { 0 };\n +-\tstruct reftable_block header = { 0 };\n ++\tstruct reftable_block footer = { NULL };\n ++\tstruct reftable_block header = { NULL };\n + \tint err = 0;\n + \n + \tmemset(r, 0, sizeof(struct reftable_reader));\n +@@ reftable/reader.c: int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n + {\n + \tint32_t guess_block_size = r->block_size ? r->block_size :\n + \t\t\t\t\t\t\t DEFAULT_BLOCK_SIZE;\n +-\tstruct reftable_block block = { 0 };\n ++\tstruct reftable_block block = { NULL };\n + \tuint8_t block_typ = 0;\n + \tint err = 0;\n + \tuint32_t header_off = next_off ? 0 : header_size(r->version);\n +@@ reftable/reader.c: static int reader_seek_indexed(struct reftable_reader *r,\n + \t\t\t struct reftable_record *rec)\n + {\n + \tstruct reftable_index_record want_index = { .last_key = STRBUF_INIT };\n +-\tstruct reftable_record want_index_rec = { 0 };\n ++\tstruct reftable_record want_index_rec = { NULL };\n + \tstruct reftable_index_record index_result = { .last_key = STRBUF_INIT };\n +-\tstruct reftable_record index_result_rec = { 0 };\n ++\tstruct reftable_record index_result_rec = { NULL };\n + \tstruct table_iter index_iter = TABLE_ITER_INIT;\n + \tstruct table_iter next = TABLE_ITER_INIT;\n + \tint err = 0;\n +@@ reftable/reader.c: int reftable_reader_seek_ref(struct reftable_reader *r,\n + \tstruct reftable_ref_record ref = {\n + \t\t.refname = (char *)name,\n + \t};\n +-\tstruct reftable_record rec = { 0 };\n ++\tstruct reftable_record rec = { NULL };\n + \treftable_record_from_ref(&rec, &ref);\n + \treturn reader_seek(r, it, &rec);\n + }\n +@@ reftable/reader.c: int reftable_reader_seek_log_at(struct reftable_reader *r,\n + \t\t.refname = (char *)name,\n + \t\t.update_index = update_index,\n + \t};\n +-\tstruct reftable_record rec = { 0 };\n ++\tstruct reftable_record rec = { NULL };\n + \treftable_record_from_log(&rec, &log);\n + \treturn reader_seek(r, it, &rec);\n + }\n +@@ reftable/reader.c: static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n + \t\t.hash_prefix = oid,\n + \t\t.hash_prefix_len = r->object_id_len,\n + \t};\n +-\tstruct reftable_record want_rec = { 0 };\n +-\tstruct reftable_iterator oit = { 0 };\n +-\tstruct reftable_obj_record got = { 0 };\n +-\tstruct reftable_record got_rec = { 0 };\n ++\tstruct reftable_record want_rec = { NULL };\n ++\tstruct reftable_iterator oit = { NULL };\n ++\tstruct reftable_obj_record got = { NULL };\n ++\tstruct reftable_record got_rec = { NULL };\n + \tint err = 0;\n + \tstruct indexed_table_ref_iter *itr = NULL;\n + \n +\n ## t/helper/test-reftable.c ##\n @@ t/helper/test-reftable.c: int cmd__reftable(int argc, const char **argv)\n \ttree_test_main(argc, argv);\n\n-- \ngitgitgadget\n"},{"id":"406772","messageId":"01a669a731dfce02c714a40af999b100c75815e4.1601568663.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v2.git.git.1601568663.gitgitgadget@gmail.com","subject":"[PATCH v2 03/13] vcxproj: adjust for the reftable changes","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-01T16:10:53Z","receivedAt":"2020-10-01T16:11:20Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis allows Git to be compiled via Visual Studio again after integrating\nthe `hn/reftable` branch.\n\nSigned-off-by: Johannes Schindelin \n---\n config.mak.uname | 2 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 ++++++++++-\n 2 files changed, 11 insertions(+), 2 deletions(-)\n\ndiff --git a/config.mak.uname b/config.mak.uname\nindex c7eba69e54..ae4e25a1a4 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -709,7 +709,7 @@ vcxproj:\n \t# Make .vcxproj files and add them\n \tunset QUIET_GEN QUIET_BUILT_IN; \\\n \tperl contrib/buildsystems/generate -g Vcxproj\n-\tgit add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj\n+\tgit add -f git.sln {*,*/lib,*/libreftable,t/helper/*}/*.vcxproj\n \n \t# Generate the LinkOrCopyBuiltins.targets and LinkOrCopyRemoteHttp.targets file\n \t(echo '' && \\\ndiff --git a/contrib/buildsystems/Generators/Vcxproj.pm b/contrib/buildsystems/Generators/Vcxproj.pm\nindex d2584450ba..1a25789d28 100644\n--- a/contrib/buildsystems/Generators/Vcxproj.pm\n+++ b/contrib/buildsystems/Generators/Vcxproj.pm\n@@ -77,7 +77,7 @@ sub createProject {\n my $libs_release = \"\\n \";\n my $libs_debug = \"\\n \";\n if (!$static_library) {\n- $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n+ $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib|reftable\\/libreftable\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n $libs_debug = $libs_release;\n $libs_debug =~ s/zlib\\.lib/zlibd\\.lib/g;\n $libs_debug =~ s/libexpat\\.lib/libexpatd\\.lib/g;\n@@ -232,6 +232,7 @@ sub createProject {\n EOM\n if (!$static_library || $target =~ 'vcs-svn' || $target =~ 'xdiff') {\n my $uuid_libgit = $$build_structure{\"LIBS_libgit_GUID\"};\n+ my $uuid_libreftable = $$build_structure{\"LIBS_reftable/libreftable_GUID\"};\n my $uuid_xdiff_lib = $$build_structure{\"LIBS_xdiff/lib_GUID\"};\n \n print F << \"EOM\";\n@@ -241,6 +242,14 @@ sub createProject {\n false\n \n EOM\n+ if (!($name =~ /xdiff|libreftable/)) {\n+ print F << \"EOM\";\n+ \n+ $uuid_libreftable\n+ false\n+ \n+EOM\n+ }\n if (!($name =~ 'xdiff')) {\n print F << \"EOM\";\n \n-- \ngitgitgadget\n\n"},{"id":"406773","messageId":"e30a7e02815121eeba34d7f27b352f675a7d9f60.1601568664.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v2.git.git.1601568663.gitgitgadget@gmail.com","subject":"[PATCH v2 08/13] reftable: a generic binary tree implementation","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-01T16:10:58Z","receivedAt":"2020-10-01T16:11:22Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis is necessary for building a OID => ref map on write\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 ++\n reftable/tree.c | 63 ++++++++++++++++++++++++++++++++++++++++\n reftable/tree.h | 34 ++++++++++++++++++++++\n reftable/tree_test.c | 62 +++++++++++++++++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 162 insertions(+)\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n\ndiff --git a/Makefile b/Makefile\nindex eda92b00ef..870e540f22 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2379,12 +2379,14 @@ REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/compat.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/strbuf.o\n+REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/strbuf_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n+REFTABLE_TEST_OBJS += reftable/tree_test.o\n \n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \\\ndiff --git a/reftable/tree.c b/reftable/tree.c\nnew file mode 100644\nindex 0000000000..0061d14e30\n--- /dev/null\n+++ b/reftable/tree.c\n@@ -0,0 +1,63 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert)\n+{\n+\tint res;\n+\tif (*rootp == NULL) {\n+\t\tif (!insert) {\n+\t\t\treturn NULL;\n+\t\t} else {\n+\t\t\tstruct tree_node *n =\n+\t\t\t\treftable_calloc(sizeof(struct tree_node));\n+\t\t\tn->key = key;\n+\t\t\t*rootp = n;\n+\t\t\treturn *rootp;\n+\t\t}\n+\t}\n+\n+\tres = compare(key, (*rootp)->key);\n+\tif (res < 0)\n+\t\treturn tree_search(key, &(*rootp)->left, compare, insert);\n+\telse if (res > 0)\n+\t\treturn tree_search(key, &(*rootp)->right, compare, insert);\n+\treturn *rootp;\n+}\n+\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg)\n+{\n+\tif (t->left != NULL) {\n+\t\tinfix_walk(t->left, action, arg);\n+\t}\n+\taction(arg, t->key);\n+\tif (t->right != NULL) {\n+\t\tinfix_walk(t->right, action, arg);\n+\t}\n+}\n+\n+void tree_free(struct tree_node *t)\n+{\n+\tif (t == NULL) {\n+\t\treturn;\n+\t}\n+\tif (t->left != NULL) {\n+\t\ttree_free(t->left);\n+\t}\n+\tif (t->right != NULL) {\n+\t\ttree_free(t->right);\n+\t}\n+\treftable_free(t);\n+}\ndiff --git a/reftable/tree.h b/reftable/tree.h\nnew file mode 100644\nindex 0000000000..954512e9a3\n--- /dev/null\n+++ b/reftable/tree.h\n@@ -0,0 +1,34 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TREE_H\n+#define TREE_H\n+\n+/* tree_node is a generic binary search tree. */\n+struct tree_node {\n+\tvoid *key;\n+\tstruct tree_node *left, *right;\n+};\n+\n+/* looks for `key` in `rootp` using `compare` as comparison function. If insert\n+ is set, insert the key if it's not found. Else, return NULL.\n+*/\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert);\n+\n+/* performs an infix walk of the tree. */\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg);\n+\n+/*\n+ deallocates the tree nodes recursively. Keys should be deallocated separately\n+ by walking over the tree. */\n+void tree_free(struct tree_node *t);\n+\n+#endif\ndiff --git a/reftable/tree_test.c b/reftable/tree_test.c\nnew file mode 100644\nindex 0000000000..320fd02a0f\n--- /dev/null\n+++ b/reftable/tree_test.c\n@@ -0,0 +1,62 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static int test_compare(const void *a, const void *b)\n+{\n+\treturn (char *)a - (char *)b;\n+}\n+\n+struct curry {\n+\tvoid *last;\n+};\n+\n+static void check_increasing(void *arg, void *key)\n+{\n+\tstruct curry *c = (struct curry *)arg;\n+\tif (c->last != NULL) {\n+\t\tassert(test_compare(c->last, key) < 0);\n+\t}\n+\tc->last = key;\n+}\n+\n+static void test_tree(void)\n+{\n+\tstruct tree_node *root = NULL;\n+\n+\tvoid *values[11] = { NULL };\n+\tstruct tree_node *nodes[11] = { NULL };\n+\tint i = 1;\n+\tstruct curry c = { NULL };\n+\tdo {\n+\t\tnodes[i] = tree_search(values + i, &root, &test_compare, 1);\n+\t\ti = (i * 7) % 11;\n+\t} while (i != 1);\n+\n+\tfor (i = 1; i < ARRAY_SIZE(nodes); i++) {\n+\t\tassert(values + i == nodes[i]->key);\n+\t\tassert(nodes[i] ==\n+\t\t tree_search(values + i, &root, &test_compare, 0));\n+\t}\n+\n+\tinfix_walk(root, check_increasing, &c);\n+\ttree_free(root);\n+}\n+\n+int tree_test_main(int argc, const char *argv[])\n+{\n+\tadd_test_case(\"test_tree\", &test_tree);\n+\treturn test_main(argc, argv);\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 81a9bd5667..9c4e0f42dc 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,5 +6,6 @@ int cmd__reftable(int argc, const char **argv)\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \tstrbuf_test_main(argc, argv);\n+\ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"406774","messageId":"757dd30fe2cdd19affc0208d8e8091d12810af80.1601568664.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v2.git.git.1601568663.gitgitgadget@gmail.com","subject":"[PATCH v2 07/13] reftable: reading/writing blocks","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-01T16:10:57Z","receivedAt":"2020-10-01T16:11:22Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nIncludes a code snippet copied from zlib\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 3 +\n reftable/.gitattributes | 1 +\n reftable/block.c | 441 +++++++++++++++++++++++++++++++++++++++\n reftable/block.h | 129 ++++++++++++\n reftable/block_test.c | 158 ++++++++++++++\n reftable/zlib-compat.c | 92 ++++++++\n t/helper/test-reftable.c | 1 +\n 7 files changed, 825 insertions(+)\n create mode 100644 reftable/.gitattributes\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n create mode 100644 reftable/zlib-compat.c\n\ndiff --git a/Makefile b/Makefile\nindex b5e43d79ed..eda92b00ef 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2373,12 +2373,15 @@ XDIFF_OBJS += xdiff/xprepare.o\n XDIFF_OBJS += xdiff/xutils.o\n \n REFTABLE_OBJS += reftable/basics.o\n+REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/compat.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/strbuf.o\n+REFTABLE_OBJS += reftable/zlib-compat.o\n \n+REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/strbuf_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\ndiff --git a/reftable/.gitattributes b/reftable/.gitattributes\nnew file mode 100644\nindex 0000000000..f44451a379\n--- /dev/null\n+++ b/reftable/.gitattributes\n@@ -0,0 +1 @@\n+/zlib-compat.c\twhitespace=-indent-with-non-tab,-trailing-space\ndiff --git a/reftable/block.c b/reftable/block.c\nnew file mode 100644\nindex 0000000000..d49e80df7c\n--- /dev/null\n+++ b/reftable/block.c\n@@ -0,0 +1,441 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+#include \"system.h\"\n+#include \"zlib.h\"\n+\n+int header_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 24;\n+\tcase 2:\n+\t\treturn 28;\n+\t}\n+\tabort();\n+}\n+\n+int footer_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 68;\n+\tcase 2:\n+\t\treturn 72;\n+\t}\n+\tabort();\n+}\n+\n+static int block_writer_register_restart(struct block_writer *w, int n,\n+\t\t\t\t\t int is_restart, struct strbuf *key)\n+{\n+\tint rlen = w->restart_len;\n+\tif (rlen >= MAX_RESTARTS) {\n+\t\tis_restart = 0;\n+\t}\n+\n+\tif (is_restart) {\n+\t\trlen++;\n+\t}\n+\tif (2 + 3 * rlen + n > w->block_size - w->next)\n+\t\treturn -1;\n+\tif (is_restart) {\n+\t\tif (w->restart_len == w->restart_cap) {\n+\t\t\tw->restart_cap = w->restart_cap * 2 + 1;\n+\t\t\tw->restarts = reftable_realloc(\n+\t\t\t\tw->restarts, sizeof(uint32_t) * w->restart_cap);\n+\t\t}\n+\n+\t\tw->restarts[w->restart_len++] = w->next;\n+\t}\n+\n+\tw->next += n;\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, key);\n+\tw->entries++;\n+\treturn 0;\n+}\n+\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n+{\n+\tbw->buf = buf;\n+\tbw->hash_size = hash_size;\n+\tbw->block_size = block_size;\n+\tbw->header_off = header_off;\n+\tbw->buf[header_off] = typ;\n+\tbw->next = header_off + 4;\n+\tbw->restart_interval = 16;\n+\tbw->entries = 0;\n+\tbw->restart_len = 0;\n+\tbw->last_key.len = 0;\n+}\n+\n+uint8_t block_writer_type(struct block_writer *bw)\n+{\n+\treturn bw->buf[bw->header_off];\n+}\n+\n+/* adds the reftable_record to the block. Returns -1 if it does not fit, 0 on\n+ success */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tstruct strbuf last =\n+\t\tw->entries % w->restart_interval == 0 ? empty : w->last_key;\n+\tstruct string_view out = {\n+\t\t.buf = w->buf + w->next,\n+\t\t.len = w->block_size - w->next,\n+\t};\n+\n+\tstruct string_view start = out;\n+\n+\tint is_restart = 0;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint n = 0;\n+\n+\treftable_record_key(rec, &key);\n+\tn = reftable_encode_key(&is_restart, out, last, key,\n+\t\t\t\treftable_record_val_type(rec));\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tn = reftable_record_encode(rec, out, w->hash_size);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tif (block_writer_register_restart(w, start.len - out.len, is_restart,\n+\t\t\t\t\t &key) < 0)\n+\t\tgoto done;\n+\n+\tstrbuf_release(&key);\n+\treturn 0;\n+\n+done:\n+\tstrbuf_release(&key);\n+\treturn -1;\n+}\n+\n+\n+int block_writer_finish(struct block_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->restart_len; i++) {\n+\t\tput_be24(w->buf + w->next, w->restarts[i]);\n+\t\tw->next += 3;\n+\t}\n+\n+\tput_be16(w->buf + w->next, w->restart_len);\n+\tw->next += 2;\n+\tput_be24(w->buf + 1 + w->header_off, w->next);\n+\n+\tif (block_writer_type(w) == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + w->header_off;\n+\t\tuint8_t *compressed = NULL;\n+\t\tint zresult = 0;\n+\t\tuLongf src_len = w->next - block_header_skip;\n+\t\tsize_t dest_cap = src_len;\n+\n+\t\tcompressed = reftable_malloc(dest_cap);\n+\t\twhile (1) {\n+\t\t\tuLongf out_dest_len = dest_cap;\n+\n+\t\t\tzresult = compress2(compressed, &out_dest_len,\n+\t\t\t\t\t w->buf + block_header_skip, src_len,\n+\t\t\t\t\t 9);\n+\t\t\tif (zresult == Z_BUF_ERROR) {\n+\t\t\t\tdest_cap *= 2;\n+\t\t\t\tcompressed =\n+\t\t\t\t\treftable_realloc(compressed, dest_cap);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (Z_OK != zresult) {\n+\t\t\t\treftable_free(compressed);\n+\t\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t\t}\n+\n+\t\t\tmemcpy(w->buf + block_header_skip, compressed,\n+\t\t\t out_dest_len);\n+\t\t\tw->next = out_dest_len + block_header_skip;\n+\t\t\treftable_free(compressed);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\treturn w->next;\n+}\n+\n+uint8_t block_reader_type(struct block_reader *r)\n+{\n+\treturn r->block.data[r->header_off];\n+}\n+\n+int block_reader_init(struct block_reader *br, struct reftable_block *block,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size)\n+{\n+\tuint32_t full_block_size = table_block_size;\n+\tuint8_t typ = block->data[header_off];\n+\tuint32_t sz = get_be24(block->data + header_off + 1);\n+\n+\tuint16_t restart_count = 0;\n+\tuint32_t restart_start = 0;\n+\tuint8_t *restart_bytes = NULL;\n+\n+\tif (!reftable_is_block_type(typ))\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tif (typ == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + header_off;\n+\t\tuLongf dst_len = sz - block_header_skip; /* total size of dest\n+\t\t\t\t\t\t\t buffer. */\n+\t\tuLongf src_len = block->len - block_header_skip;\n+\t\t/* Log blocks specify the *uncompressed* size in their header.\n+\t\t */\n+\t\tuint8_t *uncompressed = reftable_malloc(sz);\n+\n+\t\t/* Copy over the block header verbatim. It's not compressed. */\n+\t\tmemcpy(uncompressed, block->data, block_header_skip);\n+\n+\t\t/* Uncompress */\n+\t\tif (Z_OK != uncompress_return_consumed(\n+\t\t\t\t uncompressed + block_header_skip, &dst_len,\n+\t\t\t\t block->data + block_header_skip,\n+\t\t\t\t &src_len)) {\n+\t\t\treftable_free(uncompressed);\n+\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t}\n+\n+\t\tif (dst_len + block_header_skip != sz)\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\t\t/* We're done with the input data. */\n+\t\treftable_block_done(block);\n+\t\tblock->data = uncompressed;\n+\t\tblock->len = sz;\n+\t\tblock->source = malloc_block_source();\n+\t\tfull_block_size = src_len + block_header_skip;\n+\t} else if (full_block_size == 0) {\n+\t\tfull_block_size = sz;\n+\t} else if (sz < full_block_size && sz < block->len &&\n+\t\t block->data[sz] != 0) {\n+\t\t/* If the block is smaller than the full block size, it is\n+\t\t padded (data followed by '\\0') or the next block is\n+\t\t unaligned. */\n+\t\tfull_block_size = sz;\n+\t}\n+\n+\trestart_count = get_be16(block->data + sz - 2);\n+\trestart_start = sz - 2 - 3 * restart_count;\n+\trestart_bytes = block->data + restart_start;\n+\n+\t/* transfer ownership. */\n+\tbr->block = *block;\n+\tblock->data = NULL;\n+\tblock->len = 0;\n+\n+\tbr->hash_size = hash_size;\n+\tbr->block_len = restart_start;\n+\tbr->full_block_size = full_block_size;\n+\tbr->header_off = header_off;\n+\tbr->restart_count = restart_count;\n+\tbr->restart_bytes = restart_bytes;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t block_reader_restart_offset(struct block_reader *br, int i)\n+{\n+\treturn get_be24(br->restart_bytes + 3 * i);\n+}\n+\n+void block_reader_start(struct block_reader *br, struct block_iter *it)\n+{\n+\tit->br = br;\n+\tstrbuf_reset(&it->last_key);\n+\tit->next_off = br->header_off + 4;\n+}\n+\n+struct restart_find_args {\n+\tint error;\n+\tstruct strbuf key;\n+\tstruct block_reader *r;\n+};\n+\n+static int restart_key_less(size_t idx, void *args)\n+{\n+\tstruct restart_find_args *a = (struct restart_find_args *)args;\n+\tuint32_t off = block_reader_restart_offset(a->r, idx);\n+\tstruct string_view in = {\n+\t\t.buf = a->r->block.data + off,\n+\t\t.len = a->r->block_len - off,\n+\t};\n+\n+\t/* the restart key is verbatim in the block, so this could avoid the\n+\t alloc for decoding the key */\n+\tstruct strbuf rkey = STRBUF_INIT;\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tuint8_t unused_extra;\n+\tint n = reftable_decode_key(&rkey, &unused_extra, last_key, in);\n+\tint result;\n+\tif (n < 0) {\n+\t\ta->error = 1;\n+\t\treturn -1;\n+\t}\n+\n+\tresult = strbuf_cmp(&a->key, &rkey);\n+\tstrbuf_release(&rkey);\n+\treturn result;\n+}\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src)\n+{\n+\tdest->br = src->br;\n+\tdest->next_off = src->next_off;\n+\tstrbuf_reset(&dest->last_key);\n+\tstrbuf_addbuf(&dest->last_key, &src->last_key);\n+}\n+\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec)\n+{\n+\tstruct string_view in = {\n+\t\t.buf = it->br->block.data + it->next_off,\n+\t\t.len = it->br->block_len - it->next_off,\n+\t};\n+\tstruct string_view start = in;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tuint8_t extra = 0;\n+\tint n = 0;\n+\n+\tif (it->next_off >= it->br->block_len)\n+\t\treturn 1;\n+\n+\tn = reftable_decode_key(&key, &extra, it->last_key, in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\n+\tstring_view_consume(&in, n);\n+\tn = reftable_record_decode(rec, key, extra, in, it->br->hash_size);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tstrbuf_reset(&it->last_key);\n+\tstrbuf_addbuf(&it->last_key, &key);\n+\tit->next_off += start.len - in.len;\n+\tstrbuf_release(&key);\n+\treturn 0;\n+}\n+\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tint off = br->header_off + 4;\n+\tstruct string_view in = {\n+\t\t.buf = br->block.data + off,\n+\t\t.len = br->block_len - off,\n+\t};\n+\n+\tuint8_t extra = 0;\n+\tint n = reftable_decode_key(key, &extra, empty, in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\treturn 0;\n+}\n+\n+int block_iter_seek(struct block_iter *it, struct strbuf *want)\n+{\n+\treturn block_reader_seek(it->br, it, want);\n+}\n+\n+void block_iter_close(struct block_iter *it)\n+{\n+\tstrbuf_release(&it->last_key);\n+}\n+\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want)\n+{\n+\tstruct restart_find_args args = {\n+\t\t.key = *want,\n+\t\t.r = br,\n+\t};\n+\tstruct reftable_record rec = reftable_new_record(block_reader_type(br));\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = 0;\n+\tstruct block_iter next = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\n+\tint i = binsearch(br->restart_count, &restart_key_less, &args);\n+\tif (args.error) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tit->br = br;\n+\tif (i > 0) {\n+\t\ti--;\n+\t\tit->next_off = block_reader_restart_offset(br, i);\n+\t} else {\n+\t\tit->next_off = br->header_off + 4;\n+\t}\n+\n+\t/* We're looking for the last entry less/equal than the wanted key, so\n+\t we have to go one entry too far and then back up.\n+\t*/\n+\twhile (1) {\n+\t\tblock_iter_copy_from(&next, it);\n+\t\terr = block_iter_next(&next, &rec);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tif (err > 0 || strbuf_cmp(&key, want) >= 0) {\n+\t\t\terr = 0;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tblock_iter_copy_from(it, &next);\n+\t}\n+\n+done:\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&next.last_key);\n+\treftable_record_destroy(&rec);\n+\n+\treturn err;\n+}\n+\n+void block_writer_clear(struct block_writer *bw)\n+{\n+\tFREE_AND_NULL(bw->restarts);\n+\tstrbuf_release(&bw->last_key);\n+\t/* the block is not owned. */\n+}\n+\n+void reftable_block_done(struct reftable_block *blockp)\n+{\n+\tstruct reftable_block_source source = blockp->source;\n+\tif (blockp != NULL && source.ops != NULL)\n+\t\tsource.ops->return_block(source.arg, blockp);\n+\tblockp->data = NULL;\n+\tblockp->len = 0;\n+\tblockp->source.ops = NULL;\n+\tblockp->source.arg = NULL;\n+}\ndiff --git a/reftable/block.h b/reftable/block.h\nnew file mode 100644\nindex 0000000000..14a6f835f4\n--- /dev/null\n+++ b/reftable/block.h\n@@ -0,0 +1,129 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCK_H\n+#define BLOCK_H\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+\n+/*\n+ Writes reftable blocks. The block_writer is reused across blocks to minimize\n+ allocation overhead.\n+*/\n+struct block_writer {\n+\tuint8_t *buf;\n+\tuint32_t block_size;\n+\n+\t/* Offset ofof the global header. Nonzero in the first block only. */\n+\tuint32_t header_off;\n+\n+\t/* How often to restart keys. */\n+\tint restart_interval;\n+\tint hash_size;\n+\n+\t/* Offset of next uint8_t to write. */\n+\tuint32_t next;\n+\tuint32_t *restarts;\n+\tuint32_t restart_len;\n+\tuint32_t restart_cap;\n+\n+\tstruct strbuf last_key;\n+\tint entries;\n+};\n+\n+/*\n+ initializes the blockwriter to write `typ` entries, using `buf` as temporary\n+ storage. `buf` is not owned by the block_writer. */\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n+\n+/*\n+ returns the block type (eg. 'r' for ref records.\n+*/\n+uint8_t block_writer_type(struct block_writer *bw);\n+\n+/* appends the record, or -1 if it doesn't fit. */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec);\n+\n+/* appends the key restarts, and compress the block if necessary. */\n+int block_writer_finish(struct block_writer *w);\n+\n+/* clears out internally allocated block_writer members. */\n+void block_writer_clear(struct block_writer *bw);\n+\n+/* Read a block. */\n+struct block_reader {\n+\t/* offset of the block header; nonzero for the first block in a\n+\t * reftable. */\n+\tuint32_t header_off;\n+\n+\t/* the memory block */\n+\tstruct reftable_block block;\n+\tint hash_size;\n+\n+\t/* size of the data, excluding restart data. */\n+\tuint32_t block_len;\n+\tuint8_t *restart_bytes;\n+\tuint16_t restart_count;\n+\n+\t/* size of the data in the file. For log blocks, this is the compressed\n+\t * size. */\n+\tuint32_t full_block_size;\n+};\n+\n+/* Iterate over entries in a block */\n+struct block_iter {\n+\t/* offset within the block of the next entry to read. */\n+\tuint32_t next_off;\n+\tstruct block_reader *br;\n+\n+\t/* key for last entry we read. */\n+\tstruct strbuf last_key;\n+};\n+\n+/* initializes a block reader. */\n+int block_reader_init(struct block_reader *br, struct reftable_block *bl,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size);\n+\n+/* Position `it` at start of the block */\n+void block_reader_start(struct block_reader *br, struct block_iter *it);\n+\n+/* Position `it` to the `want` key in the block */\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want);\n+\n+/* Returns the block type (eg. 'r' for refs) */\n+uint8_t block_reader_type(struct block_reader *r);\n+\n+/* Decodes the first key in the block */\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key);\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src);\n+\n+/* return < 0 for error, 0 for OK, > 0 for EOF. */\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec);\n+\n+/* Seek to `want` with in the block pointed to by `it` */\n+int block_iter_seek(struct block_iter *it, struct strbuf *want);\n+\n+/* deallocate memory for `it`. The block reader and its block is left intact. */\n+void block_iter_close(struct block_iter *it);\n+\n+/* size of file header, depending on format version */\n+int header_size(int version);\n+\n+/* size of file footer, depending on format version */\n+int footer_size(int version);\n+\n+/* returns a block to its source. */\n+void reftable_block_done(struct reftable_block *ret);\n+\n+#endif\ndiff --git a/reftable/block_test.c b/reftable/block_test.c\nnew file mode 100644\nindex 0000000000..2fa8ca9831\n--- /dev/null\n+++ b/reftable/block_test.c\n@@ -0,0 +1,158 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"system.h\"\n+\n+#include \"blocksource.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct binsearch_args {\n+\tint key;\n+\tint *arr;\n+};\n+\n+static int binsearch_func(size_t i, void *void_args)\n+{\n+\tstruct binsearch_args *args = (struct binsearch_args *)void_args;\n+\n+\treturn args->key < args->arr[i];\n+}\n+\n+static void test_binsearch(void)\n+{\n+\tint arr[] = { 2, 4, 6, 8, 10 };\n+\tsize_t sz = ARRAY_SIZE(arr);\n+\tstruct binsearch_args args = {\n+\t\t.arr = arr,\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 1; i < 11; i++) {\n+\t\tint res;\n+\t\targs.key = i;\n+\t\tres = binsearch(sz, &binsearch_func, &args);\n+\n+\t\tif (res < sz) {\n+\t\t\tassert(args.key < arr[res]);\n+\t\t\tif (res > 0) {\n+\t\t\t\tassert(args.key >= arr[res - 1]);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tassert(args.key == 10 || args.key == 11);\n+\t\t}\n+\t}\n+}\n+\n+static void test_block_read_write(void)\n+{\n+\tconst int header_off = 21; /* random */\n+\tchar *names[30];\n+\tconst int N = ARRAY_SIZE(names);\n+\tconst int block_size = 1024;\n+\tstruct reftable_block block = { NULL };\n+\tstruct block_writer bw = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_record rec = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tstruct block_reader br = { 0 };\n+\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\tint j = 0;\n+\tstruct strbuf want = STRBUF_INIT;\n+\n+\tblock.data = reftable_calloc(block_size);\n+\tblock.len = block_size;\n+\tblock.source = malloc_block_source();\n+\tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n+\t\t\t header_off, hash_size(SHA1_ID));\n+\treftable_record_from_ref(&rec, &ref);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tuint8_t hash[SHA1_SIZE];\n+\t\tsnprintf(name, sizeof(name), \"branch%02d\", i);\n+\t\tmemset(hash, i, sizeof(hash));\n+\n+\t\tref.refname = name;\n+\t\tref.value = hash;\n+\t\tnames[i] = xstrdup(name);\n+\t\tn = block_writer_add(&bw, &rec);\n+\t\tref.refname = NULL;\n+\t\tref.value = NULL;\n+\t\tassert(n == 0);\n+\t}\n+\n+\tn = block_writer_finish(&bw);\n+\tassert(n > 0);\n+\n+\tblock_writer_clear(&bw);\n+\n+\tblock_reader_init(&br, &block, header_off, block_size, SHA1_SIZE);\n+\n+\tblock_reader_start(&br, &it);\n+\n+\twhile (1) {\n+\t\tint r = block_iter_next(&it, &rec);\n+\t\tassert(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tassert_streq(names[j], ref.refname);\n+\t\tj++;\n+\t}\n+\n+\treftable_record_clear(&rec);\n+\tblock_iter_close(&it);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\t\tstrbuf_reset(&want);\n+\t\tstrbuf_addstr(&want, names[i]);\n+\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tassert(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tassert(n == 0);\n+\n+\t\tassert_streq(names[i], ref.refname);\n+\n+\t\twant.len--;\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tassert(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tassert(n == 0);\n+\t\tassert_streq(names[10 * (i / 10)], ref.refname);\n+\n+\t\tblock_iter_close(&it);\n+\t}\n+\n+\treftable_record_clear(&rec);\n+\treftable_block_done(&br.block);\n+\tstrbuf_release(&want);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+}\n+\n+int block_test_main(int argc, const char *argv[])\n+{\n+\tadd_test_case(\"binsearch\", &test_binsearch);\n+\tadd_test_case(\"block_read_write\", &test_block_read_write);\n+\treturn test_main(argc, argv);\n+}\ndiff --git a/reftable/zlib-compat.c b/reftable/zlib-compat.c\nnew file mode 100644\nindex 0000000000..3e0b0f24f1\n--- /dev/null\n+++ b/reftable/zlib-compat.c\n@@ -0,0 +1,92 @@\n+/* taken from zlib's uncompr.c\n+\n+ commit cacf7f1d4e3d44d871b605da3b647f07d718623f\n+ Author: Mark Adler \n+ Date: Sun Jan 15 09:18:46 2017 -0800\n+\n+ zlib 1.2.11\n+\n+*/\n+\n+/*\n+ * Copyright (C) 1995-2003, 2010, 2014, 2016 Jean-loup Gailly, Mark Adler\n+ * For conditions of distribution and use, see copyright notice in zlib.h\n+ */\n+\n+#include \"system.h\"\n+\n+/* clang-format off */\n+\n+/* ===========================================================================\n+ Decompresses the source buffer into the destination buffer. *sourceLen is\n+ the byte length of the source buffer. Upon entry, *destLen is the total size\n+ of the destination buffer, which must be large enough to hold the entire\n+ uncompressed data. (The size of the uncompressed data must have been saved\n+ previously by the compressor and transmitted to the decompressor by some\n+ mechanism outside the scope of this compression library.) Upon exit,\n+ *destLen is the size of the decompressed data and *sourceLen is the number\n+ of source bytes consumed. Upon return, source + *sourceLen points to the\n+ first unused input byte.\n+\n+ uncompress returns Z_OK if success, Z_MEM_ERROR if there was not enough\n+ memory, Z_BUF_ERROR if there was not enough room in the output buffer, or\n+ Z_DATA_ERROR if the input data was corrupted, including if the input data is\n+ an incomplete zlib stream.\n+*/\n+int ZEXPORT uncompress_return_consumed (\n+ Bytef *dest,\n+ uLongf *destLen,\n+ const Bytef *source,\n+ uLong *sourceLen) {\n+ z_stream stream;\n+ int err;\n+ const uInt max = (uInt)-1;\n+ uLong len, left;\n+ Byte buf[1]; /* for detection of incomplete stream when *destLen == 0 */\n+\n+ len = *sourceLen;\n+ if (*destLen) {\n+ left = *destLen;\n+ *destLen = 0;\n+ }\n+ else {\n+ left = 1;\n+ dest = buf;\n+ }\n+\n+ stream.next_in = (z_const Bytef *)source;\n+ stream.avail_in = 0;\n+ stream.zalloc = (alloc_func)0;\n+ stream.zfree = (free_func)0;\n+ stream.opaque = (voidpf)0;\n+\n+ err = inflateInit(&stream);\n+ if (err != Z_OK) return err;\n+\n+ stream.next_out = dest;\n+ stream.avail_out = 0;\n+\n+ do {\n+ if (stream.avail_out == 0) {\n+ stream.avail_out = left > (uLong)max ? max : (uInt)left;\n+ left -= stream.avail_out;\n+ }\n+ if (stream.avail_in == 0) {\n+ stream.avail_in = len > (uLong)max ? max : (uInt)len;\n+ len -= stream.avail_in;\n+ }\n+ err = inflate(&stream, Z_NO_FLUSH);\n+ } while (err == Z_OK);\n+\n+ *sourceLen -= len + stream.avail_in;\n+ if (dest != buf)\n+ *destLen = stream.total_out;\n+ else if (stream.total_out && err == Z_BUF_ERROR)\n+ left = 1;\n+\n+ inflateEnd(&stream);\n+ return err == Z_STREAM_END ? Z_OK :\n+ err == Z_NEED_DICT ? Z_DATA_ERROR :\n+ err == Z_BUF_ERROR && left + stream.avail_out ? Z_DATA_ERROR :\n+ err;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 9341272089..81a9bd5667 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -3,6 +3,7 @@\n \n int cmd__reftable(int argc, const char **argv)\n {\n+\tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \tstrbuf_test_main(argc, argv);\n \treturn 0;\n-- \ngitgitgadget\n\n"},{"id":"406775","messageId":"4190da597e65bce072fa3c37c9410a56def4b489.1601568663.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v2.git.git.1601568663.gitgitgadget@gmail.com","subject":"[PATCH v2 05/13] reftable: utility functions","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-01T16:10:55Z","receivedAt":"2020-10-01T16:11:27Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis commit provides basic utility classes for the reftable library.\n\nSince the reftable library must compile standalone, there may be some overlap\nwith git-core utility functions.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 26 ++++++-\n reftable/basics.c | 131 +++++++++++++++++++++++++++++++++\n reftable/basics.h | 48 +++++++++++++\n reftable/blocksource.c | 148 ++++++++++++++++++++++++++++++++++++++\n reftable/blocksource.h | 22 ++++++\n reftable/compat.c | 110 ++++++++++++++++++++++++++++\n reftable/compat.h | 48 +++++++++++++\n reftable/publicbasics.c | 100 ++++++++++++++++++++++++++\n reftable/reftable-tests.h | 22 ++++++\n reftable/strbuf.c | 142 ++++++++++++++++++++++++++++++++++++\n reftable/strbuf.h | 80 +++++++++++++++++++++\n reftable/strbuf_test.c | 37 ++++++++++\n reftable/system.h | 51 +++++++++++++\n t/helper/test-reftable.c | 8 +++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 16 files changed, 972 insertions(+), 3 deletions(-)\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/compat.c\n create mode 100644 reftable/compat.h\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/strbuf.c\n create mode 100644 reftable/strbuf.h\n create mode 100644 reftable/strbuf_test.c\n create mode 100644 reftable/system.h\n create mode 100644 t/helper/test-reftable.c\n\ndiff --git a/Makefile b/Makefile\nindex de53954590..e40d55cd87 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -727,6 +727,7 @@ TEST_BUILTINS_OBJS += test-read-cache.o\n TEST_BUILTINS_OBJS += test-read-graph.o\n TEST_BUILTINS_OBJS += test-read-midx.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n+TEST_BUILTINS_OBJS += test-reftable.o\n TEST_BUILTINS_OBJS += test-regex.o\n TEST_BUILTINS_OBJS += test-repository.o\n TEST_BUILTINS_OBJS += test-revision-walking.o\n@@ -806,6 +807,8 @@ TEST_SHELL_PATH = $(SHELL_PATH)\n \n LIB_FILE = libgit.a\n XDIFF_LIB = xdiff/lib.a\n+REFTABLE_LIB = reftable/libreftable.a\n+REFTABLE_TEST_LIB = reftable/libreftable_test.a\n \n GENERATED_H += config-list.h\n GENERATED_H += command-list.h\n@@ -1167,7 +1170,7 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n-GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB)\n+GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB)\n EXTLIBS =\n \n GIT_USER_AGENT = git/$(GIT_VERSION)\n@@ -2369,10 +2372,21 @@ XDIFF_OBJS += xdiff/xpatience.o\n XDIFF_OBJS += xdiff/xprepare.o\n XDIFF_OBJS += xdiff/xutils.o\n \n+REFTABLE_OBJS += reftable/basics.o\n+REFTABLE_OBJS += reftable/blocksource.o\n+REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/compat.o\n+REFTABLE_OBJS += reftable/strbuf.o\n+\n+REFTABLE_TEST_OBJS += reftable/strbuf_test.o\n+REFTABLE_TEST_OBJS += reftable/test_framework.o\n+\n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \\\n \t$(XDIFF_OBJS) \\\n \t$(FUZZ_OBJS) \\\n+\t$(REFTABLE_OBJS) \\\n+\t$(REFTABLE_TEST_OBJS) \\\n \tcommon-main.o \\\n \tgit.o\n ifndef NO_CURL\n@@ -2524,6 +2538,12 @@ $(LIB_FILE): $(LIB_OBJS)\n $(XDIFF_LIB): $(XDIFF_OBJS)\n \t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n \n+$(REFTABLE_LIB): $(REFTABLE_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n+$(REFTABLE_TEST_LIB): $(REFTABLE_TEST_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n export DEFAULT_EDITOR DEFAULT_PAGER\n \n Documentation/GIT-EXCLUDED-PROGRAMS: FORCE\n@@ -2802,7 +2822,7 @@ perf: all\n \n t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n \n-t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS)\n+t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) $(REFTABLE_TEST_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n \n check-sha1:: t/helper/test-tool$X\n@@ -3133,7 +3153,7 @@ cocciclean:\n clean: profile-clean coverage-clean cocciclean\n \t$(RM) *.res\n \t$(RM) $(OBJECTS)\n-\t$(RM) $(LIB_FILE) $(XDIFF_LIB)\n+\t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) git$X\n \t$(RM) $(TEST_PROGRAMS)\n \t$(RM) $(FUZZ_PROGRAMS)\ndiff --git a/reftable/basics.c b/reftable/basics.c\nnew file mode 100644\nindex 0000000000..c429055d15\n--- /dev/null\n+++ b/reftable/basics.c\n@@ -0,0 +1,131 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+\n+void put_be24(uint8_t *out, uint32_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 16) & 0xff);\n+\tout[1] = (uint8_t)((i >> 8) & 0xff);\n+\tout[2] = (uint8_t)(i & 0xff);\n+}\n+\n+uint32_t get_be24(uint8_t *in)\n+{\n+\treturn (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 |\n+\t (uint32_t)(in[2]);\n+}\n+\n+void put_be16(uint8_t *out, uint16_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 8) & 0xff);\n+\tout[1] = (uint8_t)(i & 0xff);\n+}\n+\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n+{\n+\tsize_t lo = 0;\n+\tsize_t hi = sz;\n+\n+\t/* invariant: (hi == sz) || f(hi) == true\n+\t (lo == 0 && f(0) == true) || fi(lo) == false\n+\t */\n+\twhile (hi - lo > 1) {\n+\t\tsize_t mid = lo + (hi - lo) / 2;\n+\n+\t\tint val = f(mid, args);\n+\t\tif (val) {\n+\t\t\thi = mid;\n+\t\t} else {\n+\t\t\tlo = mid;\n+\t\t}\n+\t}\n+\n+\tif (lo == 0) {\n+\t\tif (f(0, args)) {\n+\t\t\treturn 0;\n+\t\t} else {\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\treturn hi;\n+}\n+\n+void free_names(char **a)\n+{\n+\tchar **p = a;\n+\tif (p == NULL) {\n+\t\treturn;\n+\t}\n+\twhile (*p) {\n+\t\treftable_free(*p);\n+\t\tp++;\n+\t}\n+\treftable_free(a);\n+}\n+\n+int names_length(char **names)\n+{\n+\tint len = 0;\n+\tchar **p = names;\n+\twhile (*p) {\n+\t\tp++;\n+\t\tlen++;\n+\t}\n+\treturn len;\n+}\n+\n+void parse_names(char *buf, int size, char ***namesp)\n+{\n+\tchar **names = NULL;\n+\tsize_t names_cap = 0;\n+\tsize_t names_len = 0;\n+\n+\tchar *p = buf;\n+\tchar *end = buf + size;\n+\twhile (p < end) {\n+\t\tchar *next = strchr(p, '\\n');\n+\t\tif (next != NULL) {\n+\t\t\t*next = 0;\n+\t\t} else {\n+\t\t\tnext = end;\n+\t\t}\n+\t\tif (p < next) {\n+\t\t\tif (names_len == names_cap) {\n+\t\t\t\tnames_cap = 2 * names_cap + 1;\n+\t\t\t\tnames = reftable_realloc(\n+\t\t\t\t\tnames, names_cap * sizeof(char *));\n+\t\t\t}\n+\t\t\tnames[names_len++] = xstrdup(p);\n+\t\t}\n+\t\tp = next + 1;\n+\t}\n+\n+\tif (names_len == names_cap) {\n+\t\tnames_cap = 2 * names_cap + 1;\n+\t\tnames = reftable_realloc(names, names_cap * sizeof(char *));\n+\t}\n+\n+\tnames[names_len] = NULL;\n+\t*namesp = names;\n+}\n+\n+int names_equal(char **a, char **b)\n+{\n+\twhile (*a && *b) {\n+\t\tif (strcmp(*a, *b)) {\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\ta++;\n+\t\tb++;\n+\t}\n+\n+\treturn *a == *b;\n+}\ndiff --git a/reftable/basics.h b/reftable/basics.h\nnew file mode 100644\nindex 0000000000..90639865a7\n--- /dev/null\n+++ b/reftable/basics.h\n@@ -0,0 +1,48 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BASICS_H\n+#define BASICS_H\n+\n+#include \"system.h\"\n+\n+/* Bigendian en/decoding of integers */\n+\n+void put_be24(uint8_t *out, uint32_t i);\n+uint32_t get_be24(uint8_t *in);\n+void put_be16(uint8_t *out, uint16_t i);\n+\n+/*\n+ find smallest index i in [0, sz) at which f(i) is true, assuming\n+ that f is ascending. Return sz if f(i) is false for all indices.\n+*/\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args);\n+\n+/*\n+ Frees a NULL terminated array of malloced strings. The array itself is also\n+ freed.\n+ */\n+void free_names(char **a);\n+\n+/* parse a newline separated list of names. Empty names are discarded. */\n+void parse_names(char *buf, int size, char ***namesp);\n+\n+/* compares two NULL-terminated arrays of strings. */\n+int names_equal(char **a, char **b);\n+\n+/* returns the array size of a NULL-terminated array of strings. */\n+int names_length(char **names);\n+\n+/* Allocation routines; they invoke the functions set through\n+ * reftable_set_alloc() */\n+void *reftable_malloc(size_t sz);\n+void *reftable_realloc(void *p, size_t sz);\n+void reftable_free(void *p);\n+void *reftable_calloc(size_t sz);\n+\n+#endif\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nnew file mode 100644\nindex 0000000000..7f29b864f9\n--- /dev/null\n+++ b/reftable/blocksource.c\n@@ -0,0 +1,148 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"strbuf.h\"\n+#include \"reftable.h\"\n+\n+static void strbuf_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void strbuf_close(void *b)\n+{\n+}\n+\n+static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct strbuf *b = (struct strbuf *)v;\n+\tassert(off + size <= b->len);\n+\tdest->data = reftable_calloc(size);\n+\tmemcpy(dest->data, b->buf + off, size);\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static uint64_t strbuf_size(void *b)\n+{\n+\treturn ((struct strbuf *)b)->len;\n+}\n+\n+static struct reftable_block_source_vtable strbuf_vtable = {\n+\t.size = &strbuf_size,\n+\t.read_block = &strbuf_read_block,\n+\t.return_block = &strbuf_return_block,\n+\t.close = &strbuf_close,\n+};\n+\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf)\n+{\n+\tassert(bs->ops == NULL);\n+\tbs->ops = &strbuf_vtable;\n+\tbs->arg = buf;\n+}\n+\n+static void malloc_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static struct reftable_block_source_vtable malloc_vtable = {\n+\t.return_block = &malloc_return_block,\n+};\n+\n+static struct reftable_block_source malloc_block_source_instance = {\n+\t.ops = &malloc_vtable,\n+};\n+\n+struct reftable_block_source malloc_block_source(void)\n+{\n+\treturn malloc_block_source_instance;\n+}\n+\n+struct file_block_source {\n+\tint fd;\n+\tuint64_t size;\n+};\n+\n+static uint64_t file_size(void *b)\n+{\n+\treturn ((struct file_block_source *)b)->size;\n+}\n+\n+static void file_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void file_close(void *b)\n+{\n+\tint fd = ((struct file_block_source *)b)->fd;\n+\tif (fd > 0) {\n+\t\tclose(fd);\n+\t\t((struct file_block_source *)b)->fd = 0;\n+\t}\n+\n+\treftable_free(b);\n+}\n+\n+static int file_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct file_block_source *b = (struct file_block_source *)v;\n+\tassert(off + size <= b->size);\n+\tdest->data = reftable_malloc(size);\n+\tif (pread(b->fd, dest->data, size, off) != size)\n+\t\treturn -1;\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static struct reftable_block_source_vtable file_vtable = {\n+\t.size = &file_size,\n+\t.read_block = &file_read_block,\n+\t.return_block = &file_return_block,\n+\t.close = &file_close,\n+};\n+\n+int reftable_block_source_from_file(struct reftable_block_source *bs,\n+\t\t\t\t const char *name)\n+{\n+\tstruct stat st = { 0 };\n+\tint err = 0;\n+\tint fd = open(name, O_RDONLY);\n+\tstruct file_block_source *p = NULL;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\treturn REFTABLE_NOT_EXIST_ERROR;\n+\t\t}\n+\t\treturn -1;\n+\t}\n+\n+\terr = fstat(fd, &st);\n+\tif (err < 0)\n+\t\treturn -1;\n+\n+\tp = reftable_calloc(sizeof(struct file_block_source));\n+\tp->size = st.st_size;\n+\tp->fd = fd;\n+\n+\tassert(bs->ops == NULL);\n+\tbs->ops = &file_vtable;\n+\tbs->arg = p;\n+\treturn 0;\n+}\ndiff --git a/reftable/blocksource.h b/reftable/blocksource.h\nnew file mode 100644\nindex 0000000000..3faf83fa9d\n--- /dev/null\n+++ b/reftable/blocksource.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCKSOURCE_H\n+#define BLOCKSOURCE_H\n+\n+#include \"strbuf.h\"\n+\n+struct reftable_block_source;\n+\n+/* Create an in-memory block source for reading reftables */\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf);\n+\n+struct reftable_block_source malloc_block_source(void);\n+\n+#endif\ndiff --git a/reftable/compat.c b/reftable/compat.c\nnew file mode 100644\nindex 0000000000..a48c5aa5e3\n--- /dev/null\n+++ b/reftable/compat.c\n@@ -0,0 +1,110 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+\n+*/\n+\n+/* compat.c - compatibility functions for standalone compilation */\n+\n+#include \"system.h\"\n+#include \"basics.h\"\n+\n+#ifdef REFTABLE_STANDALONE\n+\n+#include \n+\n+void put_be32(void *p, uint32_t i)\n+{\n+\tuint8_t *out = (uint8_t *)p;\n+\n+\tout[0] = (uint8_t)((i >> 24) & 0xff);\n+\tout[1] = (uint8_t)((i >> 16) & 0xff);\n+\tout[2] = (uint8_t)((i >> 8) & 0xff);\n+\tout[3] = (uint8_t)((i)&0xff);\n+}\n+\n+uint32_t get_be32(uint8_t *in)\n+{\n+\treturn (uint32_t)(in[0]) << 24 | (uint32_t)(in[1]) << 16 |\n+\t (uint32_t)(in[2]) << 8 | (uint32_t)(in[3]);\n+}\n+\n+void put_be64(void *p, uint64_t v)\n+{\n+\tuint8_t *out = (uint8_t *)p;\n+\tint i = sizeof(uint64_t);\n+\twhile (i--) {\n+\t\tout[i] = (uint8_t)(v & 0xff);\n+\t\tv >>= 8;\n+\t}\n+}\n+\n+uint64_t get_be64(void *out)\n+{\n+\tuint8_t *bytes = (uint8_t *)out;\n+\tuint64_t v = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < sizeof(uint64_t); i++) {\n+\t\tv = (v << 8) | (uint8_t)(bytes[i] & 0xff);\n+\t}\n+\treturn v;\n+}\n+\n+uint16_t get_be16(uint8_t *in)\n+{\n+\treturn (uint32_t)(in[0]) << 8 | (uint32_t)(in[1]);\n+}\n+\n+char *xstrdup(const char *s)\n+{\n+\tint l = strlen(s);\n+\tchar *dest = (char *)reftable_malloc(l + 1);\n+\tstrncpy(dest, s, l + 1);\n+\treturn dest;\n+}\n+\n+void sleep_millisec(int millisecs)\n+{\n+\tusleep(millisecs * 1000);\n+}\n+\n+void reftable_clear_dir(const char *dirname)\n+{\n+\tDIR *dir = opendir(dirname);\n+\tstruct dirent *ent = NULL;\n+\tassert(dir);\n+\twhile ((ent = readdir(dir)) != NULL) {\n+\t\tunlinkat(dirfd(dir), ent->d_name, 0);\n+\t}\n+\tclosedir(dir);\n+\trmdir(dirname);\n+}\n+\n+#else\n+\n+#include \"../dir.h\"\n+\n+void reftable_clear_dir(const char *dirname)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstrbuf_addstr(&path, dirname);\n+\tremove_dir_recursively(&path, 0);\n+\tstrbuf_release(&path);\n+}\n+\n+#endif\n+\n+int hash_size(uint32_t id)\n+{\n+\tswitch (id) {\n+\tcase 0:\n+\tcase SHA1_ID:\n+\t\treturn SHA1_SIZE;\n+\tcase SHA256_ID:\n+\t\treturn SHA256_SIZE;\n+\t}\n+\tabort();\n+}\ndiff --git a/reftable/compat.h b/reftable/compat.h\nnew file mode 100644\nindex 0000000000..a765c57e96\n--- /dev/null\n+++ b/reftable/compat.h\n@@ -0,0 +1,48 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef COMPAT_H\n+#define COMPAT_H\n+\n+#include \"system.h\"\n+\n+#ifdef REFTABLE_STANDALONE\n+\n+/* functions that git-core provides, for standalone compilation */\n+#include \n+\n+uint64_t get_be64(void *in);\n+void put_be64(void *out, uint64_t i);\n+\n+void put_be32(void *out, uint32_t i);\n+uint32_t get_be32(uint8_t *in);\n+\n+uint16_t get_be16(uint8_t *in);\n+\n+#define ARRAY_SIZE(a) sizeof((a)) / sizeof((a)[0])\n+#define FREE_AND_NULL(x) \\\n+\tdo { \\\n+\t\treftable_free(x); \\\n+\t\t(x) = NULL; \\\n+\t} while (0)\n+#define QSORT(arr, n, cmp) qsort(arr, n, sizeof(arr[0]), cmp)\n+#define SWAP(a, b) \\\n+\t{ \\\n+\t\tchar tmp[sizeof(a)]; \\\n+\t\tassert(sizeof(a) == sizeof(b)); \\\n+\t\tmemcpy(&tmp[0], &a, sizeof(a)); \\\n+\t\tmemcpy(&a, &b, sizeof(a)); \\\n+\t\tmemcpy(&b, &tmp[0], sizeof(a)); \\\n+\t}\n+\n+char *xstrdup(const char *s);\n+\n+void sleep_millisec(int millisecs);\n+\n+#endif\n+#endif\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\nnew file mode 100644\nindex 0000000000..12d547d70d\n--- /dev/null\n+++ b/reftable/publicbasics.c\n@@ -0,0 +1,100 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+const char *reftable_error_str(int err)\n+{\n+\tstatic char buf[250];\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn \"I/O error\";\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn \"corrupt reftable file\";\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn \"file does not exist\";\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn \"data is outdated\";\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn \"misuse of the reftable API\";\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn \"zlib failure\";\n+\tcase REFTABLE_NAME_CONFLICT:\n+\t\treturn \"file/directory conflict\";\n+\tcase REFTABLE_REFNAME_ERROR:\n+\t\treturn \"invalid refname\";\n+\tcase -1:\n+\t\treturn \"general error\";\n+\tdefault:\n+\t\tsnprintf(buf, sizeof(buf), \"unknown error code %d\", err);\n+\t\treturn buf;\n+\t}\n+}\n+\n+int reftable_error_to_errno(int err)\n+{\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn EIO;\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn EFAULT;\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn ENOENT;\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn EBUSY;\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn EINVAL;\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn EDOM;\n+\tdefault:\n+\t\treturn ERANGE;\n+\t}\n+}\n+\n+static void *(*reftable_malloc_ptr)(size_t sz) = &malloc;\n+static void *(*reftable_realloc_ptr)(void *, size_t) = &realloc;\n+static void (*reftable_free_ptr)(void *) = &free;\n+\n+void *reftable_malloc(size_t sz)\n+{\n+\treturn (*reftable_malloc_ptr)(sz);\n+}\n+\n+void *reftable_realloc(void *p, size_t sz)\n+{\n+\treturn (*reftable_realloc_ptr)(p, sz);\n+}\n+\n+void reftable_free(void *p)\n+{\n+\treftable_free_ptr(p);\n+}\n+\n+void *reftable_calloc(size_t sz)\n+{\n+\tvoid *p = reftable_malloc(sz);\n+\tmemset(p, 0, sz);\n+\treturn p;\n+}\n+\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n+{\n+\treftable_malloc_ptr = malloc;\n+\treftable_realloc_ptr = realloc;\n+\treftable_free_ptr = free;\n+}\n+\n+int reftable_fd_write(void *arg, const void *data, size_t sz)\n+{\n+\tint *fdp = (int *)arg;\n+\treturn write(*fdp, data, sz);\n+}\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nnew file mode 100644\nindex 0000000000..e38471888f\n--- /dev/null\n+++ b/reftable/reftable-tests.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_TESTS_H\n+#define REFTABLE_TESTS_H\n+\n+int block_test_main(int argc, const char **argv);\n+int merged_test_main(int argc, const char **argv);\n+int record_test_main(int argc, const char **argv);\n+int refname_test_main(int argc, const char **argv);\n+int reftable_test_main(int argc, const char **argv);\n+int strbuf_test_main(int argc, const char **argv);\n+int stack_test_main(int argc, const char **argv);\n+int tree_test_main(int argc, const char **argv);\n+int reftable_dump_main(int argc, char *const *argv);\n+\n+#endif\ndiff --git a/reftable/strbuf.c b/reftable/strbuf.c\nnew file mode 100644\nindex 0000000000..136bf65591\n--- /dev/null\n+++ b/reftable/strbuf.c\n@@ -0,0 +1,142 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"strbuf.h\"\n+\n+#ifdef REFTABLE_STANDALONE\n+\n+void strbuf_init(struct strbuf *s, size_t alloc)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\t*s = empty;\n+}\n+\n+void strbuf_grow(struct strbuf *s, size_t extra)\n+{\n+\tsize_t newcap = s->len + extra + 1;\n+\tif (newcap > s->cap) {\n+\t\ts->buf = reftable_realloc(s->buf, newcap);\n+\t\ts->cap = newcap;\n+\t}\n+}\n+\n+static void strbuf_resize(struct strbuf *s, int l)\n+{\n+\tint zl = l + 1; /* one uint8_t for 0 termination. */\n+\tassert(s->canary == STRBUF_CANARY);\n+\tif (s->cap < zl) {\n+\t\tint c = s->cap * 2;\n+\t\tif (c < zl) {\n+\t\t\tc = zl;\n+\t\t}\n+\t\ts->cap = c;\n+\t\ts->buf = reftable_realloc(s->buf, s->cap);\n+\t}\n+\ts->len = l;\n+\ts->buf[l] = 0;\n+}\n+\n+void strbuf_setlen(struct strbuf *s, size_t l)\n+{\n+\tassert(s->cap >= l + 1);\n+\ts->len = l;\n+\ts->buf[l] = 0;\n+}\n+\n+void strbuf_reset(struct strbuf *s)\n+{\n+\tstrbuf_resize(s, 0);\n+}\n+\n+void strbuf_addstr(struct strbuf *d, const char *s)\n+{\n+\tint l1 = d->len;\n+\tint l2 = strlen(s);\n+\tassert(d->canary == STRBUF_CANARY);\n+\n+\tstrbuf_resize(d, l2 + l1);\n+\tmemcpy(d->buf + l1, s, l2);\n+}\n+\n+void strbuf_addbuf(struct strbuf *s, struct strbuf *a)\n+{\n+\tint end = s->len;\n+\tassert(s->canary == STRBUF_CANARY);\n+\tstrbuf_resize(s, s->len + a->len);\n+\tmemcpy(s->buf + end, a->buf, a->len);\n+}\n+\n+char *strbuf_detach(struct strbuf *s, size_t *sz)\n+{\n+\tchar *p = NULL;\n+\tp = (char *)s->buf;\n+\tif (sz)\n+\t\t*sz = s->len;\n+\ts->buf = NULL;\n+\ts->cap = 0;\n+\ts->len = 0;\n+\treturn p;\n+}\n+\n+void strbuf_release(struct strbuf *s)\n+{\n+\tassert(s->canary == STRBUF_CANARY);\n+\ts->cap = 0;\n+\ts->len = 0;\n+\treftable_free(s->buf);\n+\ts->buf = NULL;\n+}\n+\n+int strbuf_cmp(const struct strbuf *a, const struct strbuf *b)\n+{\n+\tint min = a->len < b->len ? a->len : b->len;\n+\tint res = memcmp(a->buf, b->buf, min);\n+\tassert(a->canary == STRBUF_CANARY);\n+\tassert(b->canary == STRBUF_CANARY);\n+\tif (res != 0)\n+\t\treturn res;\n+\tif (a->len < b->len)\n+\t\treturn -1;\n+\telse if (a->len > b->len)\n+\t\treturn 1;\n+\telse\n+\t\treturn 0;\n+}\n+\n+int strbuf_add(struct strbuf *b, const void *data, size_t sz)\n+{\n+\tassert(b->canary == STRBUF_CANARY);\n+\tstrbuf_grow(b, sz);\n+\tmemcpy(b->buf + b->len, data, sz);\n+\tb->len += sz;\n+\tb->buf[b->len] = 0;\n+\treturn sz;\n+}\n+\n+#endif\n+\n+int strbuf_add_void(void *b, const void *data, size_t sz)\n+{\n+\tstrbuf_add((struct strbuf *)b, data, sz);\n+\treturn sz;\n+}\n+\n+int common_prefix_size(struct strbuf *a, struct strbuf *b)\n+{\n+\tint p = 0;\n+\twhile (p < a->len && p < b->len) {\n+\t\tif (a->buf[p] != b->buf[p]) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tp++;\n+\t}\n+\n+\treturn p;\n+}\n+\n+struct strbuf reftable_empty_strbuf = STRBUF_INIT;\ndiff --git a/reftable/strbuf.h b/reftable/strbuf.h\nnew file mode 100644\nindex 0000000000..c2d7aca8dd\n--- /dev/null\n+++ b/reftable/strbuf.h\n@@ -0,0 +1,80 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef SLICE_H\n+#define SLICE_H\n+\n+#ifdef REFTABLE_STANDALONE\n+\n+#include \"basics.h\"\n+\n+/*\n+ Provides a bounds-checked, growable byte ranges. To use, initialize as \"strbuf\n+ x = STRBUF_INIT;\"\n+ */\n+struct strbuf {\n+\tsize_t len;\n+\tsize_t cap;\n+\tchar *buf;\n+\n+\t/* Used to enforce initialization with STRBUF_INIT */\n+\tuint8_t canary;\n+};\n+\n+#define STRBUF_CANARY 0x42\n+#define STRBUF_INIT \\\n+\t{ \\\n+\t\t0, 0, NULL, STRBUF_CANARY \\\n+\t}\n+\n+void strbuf_addstr(struct strbuf *dest, const char *src);\n+\n+/* Deallocate and clear strbuf */\n+void strbuf_release(struct strbuf *strbuf);\n+\n+/* Set strbuf to 0 length, but retain buffer. */\n+void strbuf_reset(struct strbuf *strbuf);\n+\n+/* Initializes a strbuf. Accepts a strbuf with random garbage. */\n+void strbuf_init(struct strbuf *strbuf, size_t alloc);\n+\n+/* Return `buf`, clearing out `s`. Optionally return len (not cap) in `sz`. */\n+char *strbuf_detach(struct strbuf *s, size_t *sz);\n+\n+/* Set length of the slace to `l`, but don't reallocated. */\n+void strbuf_setlen(struct strbuf *s, size_t l);\n+\n+/* Ensure `l` bytes beyond current length are available */\n+void strbuf_grow(struct strbuf *s, size_t l);\n+\n+/* Signed comparison */\n+int strbuf_cmp(const struct strbuf *a, const struct strbuf *b);\n+\n+/* Append `data` to the `dest` strbuf. */\n+int strbuf_add(struct strbuf *dest, const void *data, size_t sz);\n+\n+/* Append `add` to `dest. */\n+void strbuf_addbuf(struct strbuf *dest, struct strbuf *add);\n+\n+#else\n+\n+#include \"../git-compat-util.h\"\n+#include \"../strbuf.h\"\n+\n+#endif\n+\n+extern struct strbuf reftable_empty_strbuf;\n+\n+/* Like strbuf_add, but suitable for passing to reftable_new_writer\n+ */\n+int strbuf_add_void(void *b, const void *data, size_t sz);\n+\n+/* Find the longest shared prefix size of `a` and `b` */\n+int common_prefix_size(struct strbuf *a, struct strbuf *b);\n+\n+#endif\ndiff --git a/reftable/strbuf_test.c b/reftable/strbuf_test.c\nnew file mode 100644\nindex 0000000000..39f561c81a\n--- /dev/null\n+++ b/reftable/strbuf_test.c\n@@ -0,0 +1,37 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"strbuf.h\"\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_strbuf(void)\n+{\n+\tstruct strbuf s = STRBUF_INIT;\n+\tstruct strbuf t = STRBUF_INIT;\n+\n+\tstrbuf_addstr(&s, \"abc\");\n+\tassert(0 == strcmp(\"abc\", s.buf));\n+\n+\tstrbuf_addstr(&t, \"pqr\");\n+\tstrbuf_addbuf(&s, &t);\n+\tassert(0 == strcmp(\"abcpqr\", s.buf));\n+\n+\tstrbuf_release(&s);\n+\tstrbuf_release(&t);\n+}\n+\n+int strbuf_test_main(int argc, const char *argv[])\n+{\n+\tadd_test_case(\"test_strbuf\", &test_strbuf);\n+\treturn test_main(argc, argv);\n+}\ndiff --git a/reftable/system.h b/reftable/system.h\nnew file mode 100644\nindex 0000000000..567eb8a87d\n--- /dev/null\n+++ b/reftable/system.h\n@@ -0,0 +1,51 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef SYSTEM_H\n+#define SYSTEM_H\n+\n+#ifndef REFTABLE_STANDALONE\n+\n+#include \"git-compat-util.h\"\n+#include \"cache.h\"\n+#include \n+\n+#else\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+\n+#include \"compat.h\"\n+\n+#endif /* REFTABLE_STANDALONE */\n+\n+void reftable_clear_dir(const char *dirname);\n+\n+#define SHA1_ID 0x73686131\n+#define SHA256_ID 0x73323536\n+#define SHA1_SIZE 20\n+#define SHA256_SIZE 32\n+\n+/* This is uncompress2, which is only available in zlib as of 2017.\n+ */\n+int uncompress_return_consumed(Bytef *dest, uLongf *destLen,\n+\t\t\t const Bytef *source, uLong *sourceLen);\n+int hash_size(uint32_t id);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nnew file mode 100644\nindex 0000000000..7d50aa6bcc\n--- /dev/null\n+++ b/t/helper/test-reftable.c\n@@ -0,0 +1,8 @@\n+#include \"reftable/reftable-tests.h\"\n+#include \"test-tool.h\"\n+\n+int cmd__reftable(int argc, const char **argv)\n+{\n+\tstrbuf_test_main(argc, argv);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex a0d3966b29..c7ab5bc4a6 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -53,6 +53,7 @@ static struct test_cmd cmds[] = {\n \t{ \"read-graph\", cmd__read_graph },\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n+\t{ \"reftable\", cmd__reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 07034d3f38..fa2b11ace6 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -42,6 +42,7 @@ int cmd__read_cache(int argc, const char **argv);\n int cmd__read_graph(int argc, const char **argv);\n int cmd__read_midx(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n+int cmd__reftable(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__repository(int argc, const char **argv);\n int cmd__revision_walking(int argc, const char **argv);\n-- \ngitgitgadget\n\n"},{"id":"406776","messageId":"bf6b929b860d68cc043d5eb0740a26b0f020cd42.1601568664.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v2.git.git.1601568663.gitgitgadget@gmail.com","subject":"[PATCH v2 11/13] reftable: file level tests","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-01T16:11:01Z","receivedAt":"2020-10-01T16:11:28Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable_test.c | 585 +++++++++++++++++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 3 files changed, 587 insertions(+)\n create mode 100644 reftable/reftable_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 8b09a56d91..17d5fba0a0 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2388,6 +2388,7 @@ REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n+REFTABLE_TEST_OBJS += reftable/reftable_test.o\n REFTABLE_TEST_OBJS += reftable/strbuf_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\ndiff --git a/reftable/reftable_test.c b/reftable/reftable_test.c\nnew file mode 100644\nindex 0000000000..9fea692905\n--- /dev/null\n+++ b/reftable/reftable_test.c\n@@ -0,0 +1,585 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable.h\"\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static const int update_index = 5;\n+\n+static void test_buffer(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_block out = { NULL };\n+\tint n;\n+\tuint8_t in[] = \"hello\";\n+\tstrbuf_add(&buf, in, sizeof(in));\n+\tblock_source_from_strbuf(&source, &buf);\n+\tassert(block_source_size(&source) == 6);\n+\tn = block_source_read_block(&source, &out, 0, sizeof(in));\n+\tassert(n == sizeof(in));\n+\tassert(!memcmp(in, out.data, n));\n+\treftable_block_done(&out);\n+\n+\tn = block_source_read_block(&source, &out, 1, 2);\n+\tassert(n == 2);\n+\tassert(!memcmp(out.data, \"el\", 2));\n+\n+\treftable_block_done(&out);\n+\tblock_source_close(&source);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void write_table(char ***names, struct strbuf *buf, int N,\n+\t\t\tint block_size, uint32_t hash_id)\n+{\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = block_size,\n+\t\t.hash_id = hash_id,\n+\t};\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, buf, &opts);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0, n;\n+\tstruct reftable_log_record log = { NULL };\n+\tconst struct reftable_stats *stats = NULL;\n+\t*names = reftable_calloc(sizeof(char *) * (N + 1));\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[SHA256_SIZE] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tref.refname = name;\n+\t\tref.value = hash;\n+\t\tref.update_index = update_index;\n+\t\t(*names)[i] = xstrdup(name);\n+\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tassert(n == 0);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[SHA256_SIZE] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tlog.refname = name;\n+\t\tlog.new_hash = hash;\n+\t\tlog.update_index = update_index;\n+\t\tlog.message = \"message\";\n+\n+\t\tn = reftable_writer_add_log(w, &log);\n+\t\tassert(n == 0);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tassert(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tfor (i = 0; i < stats->ref_stats.blocks; i++) {\n+\t\tint off = i * opts.block_size;\n+\t\tif (off == 0) {\n+\t\t\toff = header_size((hash_id == SHA256_ID) ? 2 : 1);\n+\t\t}\n+\t\tassert(buf->buf[off] == 'r');\n+\t}\n+\n+\tassert(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+}\n+\n+static void test_log_buffer_size(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 4096,\n+\t};\n+\tint err;\n+\tstruct reftable_log_record log = {\n+\t\t.refname = \"refs/heads/master\",\n+\t\t.name = \"Han-Wen Nienhuys\",\n+\t\t.email = \"hanwen@google.com\",\n+\t\t.tz_offset = 100,\n+\t\t.time = 0x5e430672,\n+\t\t.update_index = 0xa,\n+\t\t.message = \"commit: 9\\n\",\n+\t};\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\t/* This tests buffer extension for log compression. Must use a random\n+\t hash, to ensure that the compressed part is larger than the original.\n+\t*/\n+\tuint8_t hash1[SHA1_SIZE], hash2[SHA1_SIZE];\n+\tfor (int i = 0; i < SHA1_SIZE; i++) {\n+\t\thash1[i] = (uint8_t)(rand() % 256);\n+\t\thash2[i] = (uint8_t)(rand() % 256);\n+\t}\n+\tlog.old_hash = hash1;\n+\tlog.new_hash = hash2;\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\terr = reftable_writer_add_log(w, &log);\n+\tassert_err(err);\n+\terr = reftable_writer_close(w);\n+\tassert_err(err);\n+\treftable_writer_free(w);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_log_write_read(void)\n+{\n+\tint N = 2;\n+\tchar **names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint err;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\tint n;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tconst struct reftable_stats *stats = NULL;\n+\treftable_writer_set_limits(w, 0, N);\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[256];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tsnprintf(name, sizeof(name), \"b%02d%0*d\", i, 130, 7);\n+\t\tnames[i] = xstrdup(name);\n+\t\tref.refname = name;\n+\t\tref.update_index = i;\n+\n+\t\terr = reftable_writer_add_ref(w, &ref);\n+\t\tassert_err(err);\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash1[SHA1_SIZE], hash2[SHA1_SIZE];\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tset_test_hash(hash1, i);\n+\t\tset_test_hash(hash2, i + 1);\n+\n+\t\tlog.refname = names[i];\n+\t\tlog.update_index = i;\n+\t\tlog.old_hash = hash1;\n+\t\tlog.new_hash = hash2;\n+\n+\t\terr = reftable_writer_add_log(w, &log);\n+\t\tassert_err(err);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tassert(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tassert(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tassert_err(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[N - 1]);\n+\tassert_err(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tassert_err(err);\n+\n+\t/* end of iteration. */\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tassert(0 < err);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_clear(&ref);\n+\n+\terr = reftable_reader_seek_log(&rd, &it, \"\");\n+\tassert_err(err);\n+\n+\ti = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tassert_err(err);\n+\t\tassert_streq(names[i], log.refname);\n+\t\tassert(i == log.update_index);\n+\t\ti++;\n+\t\treftable_log_record_clear(&log);\n+\t}\n+\n+\tassert(i == N);\n+\treftable_iterator_destroy(&it);\n+\n+\t/* cleanup. */\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_sequential(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tint err = 0;\n+\tint j = 0;\n+\n+\twrite_table(&names, &buf, N, 256, SHA1_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tassert_err(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tassert_err(err);\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint r = reftable_iterator_next_ref(&it, &ref);\n+\t\tassert(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tassert(0 == strcmp(names[j], ref.refname));\n+\t\tassert(update_index == ref.update_index);\n+\n+\t\tj++;\n+\t\treftable_ref_record_clear(&ref);\n+\t}\n+\tassert(j == N);\n+\treftable_iterator_destroy(&it);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\n+\treader_close(&rd);\n+}\n+\n+static void test_table_write_small_table(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 1;\n+\twrite_table(&names, &buf, N, 4096, SHA1_ID);\n+\tassert(buf.len < 200);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+}\n+\n+static void test_table_read_api(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i;\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, SHA1_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tassert_err(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[0]);\n+\tassert_err(err);\n+\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tassert(err == REFTABLE_API_ERROR);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_free(names);\n+\treader_close(&rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_table_read_write_seek(int index, int hash_id)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i = 0;\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct strbuf pastLast = STRBUF_INIT;\n+\tstruct reftable_ref_record ref = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, hash_id);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tassert_err(err);\n+\tassert(hash_id == reftable_reader_hash_id(&rd));\n+\n+\tif (!index) {\n+\t\trd.ref_offsets.index_offset = 0;\n+\t} else {\n+\t\tassert(rd.ref_offsets.index_offset > 0);\n+\t}\n+\n+\tfor (i = 1; i < N; i++) {\n+\t\tint err = reftable_reader_seek_ref(&rd, &it, names[i]);\n+\t\tassert_err(err);\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tassert_err(err);\n+\t\tassert(0 == strcmp(names[i], ref.refname));\n+\t\tassert(i == ref.value[0]);\n+\n+\t\treftable_ref_record_clear(&ref);\n+\t\treftable_iterator_destroy(&it);\n+\t}\n+\n+\tstrbuf_addstr(&pastLast, names[N - 1]);\n+\tstrbuf_addstr(&pastLast, \"/\");\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, pastLast.buf);\n+\tif (err == 0) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tassert(err > 0);\n+\t} else {\n+\t\tassert(err > 0);\n+\t}\n+\n+\tstrbuf_release(&pastLast);\n+\treftable_iterator_destroy(&it);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_free(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_seek_linear(void)\n+{\n+\ttest_table_read_write_seek(0, SHA1_ID);\n+}\n+\n+static void test_table_read_write_seek_linear_sha256(void)\n+{\n+\ttest_table_read_write_seek(0, SHA256_ID);\n+}\n+\n+static void test_table_read_write_seek_index(void)\n+{\n+\ttest_table_read_write_seek(1, SHA1_ID);\n+}\n+\n+static void test_table_refs_for(int indexed)\n+{\n+\tint N = 50;\n+\tchar **want_names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint want_names_len = 0;\n+\tuint8_t want_hash[SHA1_SIZE];\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tint err;\n+\tstruct reftable_reader rd;\n+\tstruct reftable_block_source source = { NULL };\n+\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint j;\n+\n+\tset_test_hash(want_hash, 4);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[SHA1_SIZE];\n+\t\tchar fill[51] = { 0 };\n+\t\tchar name[100];\n+\t\tuint8_t hash1[SHA1_SIZE];\n+\t\tuint8_t hash2[SHA1_SIZE];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\n+\t\tmemset(hash, i, sizeof(hash));\n+\t\tmemset(fill, 'x', 50);\n+\t\t/* Put the variable part in the start */\n+\t\tsnprintf(name, sizeof(name), \"br%02d%s\", i, fill);\n+\t\tname[40] = 0;\n+\t\tref.refname = name;\n+\n+\t\tset_test_hash(hash1, i / 4);\n+\t\tset_test_hash(hash2, 3 + i / 4);\n+\t\tref.value = hash1;\n+\t\tref.target_value = hash2;\n+\n+\t\t/* 80 bytes / entry, so 3 entries per block. Yields 17\n+\t\t */\n+\t\t/* blocks. */\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tassert(n == 0);\n+\n+\t\tif (!memcmp(hash1, want_hash, SHA1_SIZE) ||\n+\t\t !memcmp(hash2, want_hash, SHA1_SIZE)) {\n+\t\t\twant_names[want_names_len++] = xstrdup(name);\n+\t\t}\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tassert(n == 0);\n+\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tassert_err(err);\n+\tif (!indexed) {\n+\t\trd.obj_offsets.is_present = 0;\n+\t}\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tassert_err(err);\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_reader_refs_for(&rd, &it, want_hash);\n+\tassert_err(err);\n+\n+\tj = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tassert(err >= 0);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tassert(j < want_names_len);\n+\t\tassert(0 == strcmp(ref.refname, want_names[j]));\n+\t\tj++;\n+\t\treftable_ref_record_clear(&ref);\n+\t}\n+\tassert(j == want_names_len);\n+\n+\tstrbuf_release(&buf);\n+\tfree_names(want_names);\n+\treftable_iterator_destroy(&it);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_refs_for_no_index(void)\n+{\n+\ttest_table_refs_for(0);\n+}\n+\n+static void test_table_refs_for_obj_index(void)\n+{\n+\ttest_table_refs_for(1);\n+}\n+\n+static void test_table_empty(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_ref_record rec = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_close(w);\n+\tassert(err == REFTABLE_EMPTY_TABLE_ERROR);\n+\treftable_writer_free(w);\n+\n+\tassert(buf.len == header_size(1) + footer_size(1));\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tassert_err(err);\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tassert_err(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tassert(err > 0);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+int reftable_test_main(int argc, const char *argv[])\n+{\n+\tadd_test_case(\"test_log_write_read\", test_log_write_read);\n+\tadd_test_case(\"test_table_read_write_seek_linear_sha256\",\n+\t\t &test_table_read_write_seek_linear_sha256);\n+\tadd_test_case(\"test_log_buffer_size\", test_log_buffer_size);\n+\tadd_test_case(\"test_table_write_small_table\",\n+\t\t &test_table_write_small_table);\n+\tadd_test_case(\"test_buffer\", &test_buffer);\n+\tadd_test_case(\"test_table_read_api\", &test_table_read_api);\n+\tadd_test_case(\"test_table_read_write_sequential\",\n+\t\t &test_table_read_write_sequential);\n+\tadd_test_case(\"test_table_read_write_seek_linear\",\n+\t\t &test_table_read_write_seek_linear);\n+\tadd_test_case(\"test_table_read_write_seek_index\",\n+\t\t &test_table_read_write_seek_index);\n+\tadd_test_case(\"test_table_read_write_refs_for_no_index\",\n+\t\t &test_table_refs_for_no_index);\n+\tadd_test_case(\"test_table_read_write_refs_for_obj_index\",\n+\t\t &test_table_refs_for_obj_index);\n+\tadd_test_case(\"test_table_empty\", &test_table_empty);\n+\treturn test_main(argc, argv);\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 9c4e0f42dc..2273fdfe39 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -5,6 +5,7 @@ int cmd__reftable(int argc, const char **argv)\n {\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\treftable_test_main(argc, argv);\n \tstrbuf_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n-- \ngitgitgadget\n\n"},{"id":"406777","messageId":"c196de7f0639f42133cf520c0e04db8a23c7e8ce.1601568664.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v2.git.git.1601568663.gitgitgadget@gmail.com","subject":"[PATCH v2 10/13] reftable: read reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-01T16:11:00Z","receivedAt":"2020-10-01T16:11:29Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 3 +\n reftable/iter.c | 242 +++++++++++++++\n reftable/iter.h | 72 +++++\n reftable/reader.c | 733 ++++++++++++++++++++++++++++++++++++++++++++\n reftable/reader.h | 78 +++++\n reftable/reftable.c | 104 +++++++\n 6 files changed, 1232 insertions(+)\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/reftable.c\n\ndiff --git a/Makefile b/Makefile\nindex c1dd046155..8b09a56d91 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2377,7 +2377,10 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/compat.o\n+REFTABLE_OBJS += reftable/iter.o\n+REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/reftable.o\n REFTABLE_OBJS += reftable/strbuf.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\ndiff --git a/reftable/iter.c b/reftable/iter.c\nnew file mode 100644\nindex 0000000000..2cff447323\n--- /dev/null\n+++ b/reftable/iter.c\n@@ -0,0 +1,242 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"iter.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"reftable.h\"\n+\n+int iterator_is_null(struct reftable_iterator *it)\n+{\n+\treturn it->ops == NULL;\n+}\n+\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n+{\n+\treturn 1;\n+}\n+\n+static void empty_iterator_close(void *arg)\n+{\n+}\n+\n+static struct reftable_iterator_vtable empty_vtable = {\n+\t.next = &empty_iterator_next,\n+\t.close = &empty_iterator_close,\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = NULL;\n+\tit->ops = &empty_vtable;\n+}\n+\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n+{\n+\treturn it->ops->next(it->iter_arg, rec);\n+}\n+\n+void reftable_iterator_destroy(struct reftable_iterator *it)\n+{\n+\tif (it->ops == NULL) {\n+\t\treturn;\n+\t}\n+\tit->ops->close(it->iter_arg);\n+\tit->ops = NULL;\n+\tFREE_AND_NULL(it->iter_arg);\n+}\n+\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { 0 };\n+\treftable_record_from_ref(&rec, ref);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { 0 };\n+\treftable_record_from_log(&rec, log);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+static void filtering_ref_iterator_close(void *iter_arg)\n+{\n+\tstruct filtering_ref_iterator *fri =\n+\t\t(struct filtering_ref_iterator *)iter_arg;\n+\tstrbuf_release(&fri->oid);\n+\treftable_iterator_destroy(&fri->it);\n+}\n+\n+static int filtering_ref_iterator_next(void *iter_arg,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct filtering_ref_iterator *fri =\n+\t\t(struct filtering_ref_iterator *)iter_arg;\n+\tstruct reftable_ref_record *ref =\n+\t\t(struct reftable_ref_record *)rec->data;\n+\tint err = 0;\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&fri->it, ref);\n+\t\tif (err != 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (fri->double_check) {\n+\t\t\tstruct reftable_iterator it = { 0 };\n+\n+\t\t\terr = reftable_table_seek_ref(&fri->tab, &it,\n+\t\t\t\t\t\t ref->refname);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = reftable_iterator_next_ref(&it, ref);\n+\t\t\t}\n+\n+\t\t\treftable_iterator_destroy(&it);\n+\n+\t\t\tif (err < 0) {\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (err > 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif ((ref->target_value != NULL &&\n+\t\t !memcmp(fri->oid.buf, ref->target_value, fri->oid.len)) ||\n+\t\t (ref->value != NULL &&\n+\t\t !memcmp(fri->oid.buf, ref->value, fri->oid.len))) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treftable_ref_record_clear(ref);\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable filtering_ref_iterator_vtable = {\n+\t.next = &filtering_ref_iterator_next,\n+\t.close = &filtering_ref_iterator_close,\n+};\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *it,\n+\t\t\t\t\t struct filtering_ref_iterator *fri)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = fri;\n+\tit->ops = &filtering_ref_iterator_vtable;\n+}\n+\n+static void indexed_table_ref_iter_close(void *p)\n+{\n+\tstruct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p;\n+\tblock_iter_close(&it->cur);\n+\treftable_block_done(&it->block_reader.block);\n+\tstrbuf_release(&it->oid);\n+}\n+\n+static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n+{\n+\tuint64_t off;\n+\tint err = 0;\n+\tif (it->offset_idx == it->offset_len) {\n+\t\tit->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\n+\treftable_block_done(&it->block_reader.block);\n+\n+\toff = it->offsets[it->offset_idx++];\n+\terr = reader_init_block_reader(it->r, &it->block_reader, off,\n+\t\t\t\t BLOCK_TYPE_REF);\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\tif (err > 0) {\n+\t\t/* indexed block does not exist. */\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\t}\n+\tblock_reader_start(&it->block_reader, &it->cur);\n+\treturn 0;\n+}\n+\n+static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\n+{\n+\tstruct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p;\n+\tstruct reftable_ref_record *ref =\n+\t\t(struct reftable_ref_record *)rec->data;\n+\n+\twhile (1) {\n+\t\tint err = block_iter_next(&it->cur, rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\terr = indexed_table_ref_iter_next_block(it);\n+\t\t\tif (err < 0) {\n+\t\t\t\treturn err;\n+\t\t\t}\n+\n+\t\t\tif (it->is_finished) {\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!memcmp(it->oid.buf, ref->target_value, it->oid.len) ||\n+\t\t !memcmp(it->oid.buf, ref->value, it->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+}\n+\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len)\n+{\n+\tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n+\tstruct indexed_table_ref_iter *itr =\n+\t\treftable_calloc(sizeof(struct indexed_table_ref_iter));\n+\tint err = 0;\n+\n+\t*itr = empty;\n+\titr->r = r;\n+\tstrbuf_add(&itr->oid, oid, oid_len);\n+\n+\titr->offsets = offsets;\n+\titr->offset_len = offset_len;\n+\n+\terr = indexed_table_ref_iter_next_block(itr);\n+\tif (err < 0) {\n+\t\treftable_free(itr);\n+\t} else {\n+\t\t*dest = itr;\n+\t}\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable indexed_table_ref_iter_vtable = {\n+\t.next = &indexed_table_ref_iter_next,\n+\t.close = &indexed_table_ref_iter_close,\n+};\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = itr;\n+\tit->ops = &indexed_table_ref_iter_vtable;\n+}\ndiff --git a/reftable/iter.h b/reftable/iter.h\nnew file mode 100644\nindex 0000000000..c92a178773\n--- /dev/null\n+++ b/reftable/iter.h\n@@ -0,0 +1,72 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef ITER_H\n+#define ITER_H\n+\n+#include \"block.h\"\n+#include \"record.h\"\n+#include \"strbuf.h\"\n+\n+struct reftable_iterator_vtable {\n+\tint (*next)(void *iter_arg, struct reftable_record *rec);\n+\tvoid (*close)(void *iter_arg);\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it);\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec);\n+\n+/* Returns true for a zeroed out iterator, such as the one returned from\n+ iterator_destroy. */\n+int iterator_is_null(struct reftable_iterator *it);\n+\n+/* iterator that produces only ref records that point to `oid` */\n+struct filtering_ref_iterator {\n+\tint double_check;\n+\tstruct reftable_table tab;\n+\tstruct strbuf oid;\n+\tstruct reftable_iterator it;\n+};\n+#define FILTERING_REF_ITERATOR_INIT \\\n+\t{ \\\n+\t\t.oid = STRBUF_INIT \\\n+\t}\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *,\n+\t\t\t\t\t struct filtering_ref_iterator *);\n+\n+/* iterator that produces only ref records that point to `oid`,\n+ but using the object index.\n+ */\n+struct indexed_table_ref_iter {\n+\tstruct reftable_reader *r;\n+\tstruct strbuf oid;\n+\n+\t/* mutable */\n+\tuint64_t *offsets;\n+\n+\t/* Points to the next offset to read. */\n+\tint offset_idx;\n+\tint offset_len;\n+\tstruct block_reader block_reader;\n+\tstruct block_iter cur;\n+\tint is_finished;\n+};\n+\n+#define INDEXED_TABLE_REF_ITER_INIT \\\n+\t{ \\\n+\t\t.cur = { .last_key = STRBUF_INIT }, .oid = STRBUF_INIT, \\\n+\t}\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr);\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len);\n+\n+#endif\ndiff --git a/reftable/reader.c b/reftable/reader.c\nnew file mode 100644\nindex 0000000000..c7f56b5fdc\n--- /dev/null\n+++ b/reftable/reader.c\n@@ -0,0 +1,733 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reader.h\"\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"iter.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+#include \"tree.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source)\n+{\n+\treturn source->ops->size(source->arg);\n+}\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tint result = source->ops->read_block(source->arg, dest, off, size);\n+\tdest->source = *source;\n+\treturn result;\n+}\n+\n+void block_source_close(struct reftable_block_source *source)\n+{\n+\tif (source->ops == NULL) {\n+\t\treturn;\n+\t}\n+\n+\tsource->ops->close(source->arg);\n+\tsource->ops = NULL;\n+}\n+\n+static struct reftable_reader_offsets *\n+reader_offsets_for(struct reftable_reader *r, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\t\treturn &r->ref_offsets;\n+\tcase BLOCK_TYPE_LOG:\n+\t\treturn &r->log_offsets;\n+\tcase BLOCK_TYPE_OBJ:\n+\t\treturn &r->obj_offsets;\n+\t}\n+\tabort();\n+}\n+\n+static int reader_get_block(struct reftable_reader *r,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t sz)\n+{\n+\tif (off >= r->size)\n+\t\treturn 0;\n+\n+\tif (off + sz > r->size) {\n+\t\tsz = r->size - off;\n+\t}\n+\n+\treturn block_source_read_block(&r->source, dest, off, sz);\n+}\n+\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r)\n+{\n+\treturn r->hash_id;\n+}\n+\n+const char *reader_name(struct reftable_reader *r)\n+{\n+\treturn r->name;\n+}\n+\n+static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n+\t\t\tuint8_t *header)\n+{\n+\tuint8_t *f = footer;\n+\tuint8_t first_block_typ;\n+\tint err = 0;\n+\tuint32_t computed_crc;\n+\tuint32_t file_crc;\n+\n+\tif (memcmp(f, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tf += 4;\n+\n+\tif (memcmp(footer, header, header_size(r->version))) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tf++;\n+\tr->block_size = get_be24(f);\n+\n+\tf += 3;\n+\tr->min_update_index = get_be64(f);\n+\tf += 8;\n+\tr->max_update_index = get_be64(f);\n+\tf += 8;\n+\n+\tif (r->version == 1) {\n+\t\tr->hash_id = SHA1_ID;\n+\t} else {\n+\t\tr->hash_id = get_be32(f);\n+\t\tswitch (r->hash_id) {\n+\t\tcase SHA1_ID:\n+\t\t\tbreak;\n+\t\tcase SHA256_ID:\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tf += 4;\n+\t}\n+\n+\tr->ref_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->obj_offsets.offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1);\n+\tr->obj_offsets.offset >>= 5;\n+\n+\tr->obj_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tcomputed_crc = crc32(0, footer, f - footer);\n+\tfile_crc = get_be32(f);\n+\tf += 4;\n+\tif (computed_crc != file_crc) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tfirst_block_typ = header[header_size(r->version)];\n+\tr->ref_offsets.is_present = (first_block_typ == BLOCK_TYPE_REF);\n+\tr->ref_offsets.offset = 0;\n+\tr->log_offsets.is_present = (first_block_typ == BLOCK_TYPE_LOG ||\n+\t\t\t\t r->log_offsets.offset > 0);\n+\tr->obj_offsets.is_present = r->obj_offsets.offset > 0;\n+\terr = 0;\n+done:\n+\treturn err;\n+}\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name)\n+{\n+\tstruct reftable_block footer = { 0 };\n+\tstruct reftable_block header = { 0 };\n+\tint err = 0;\n+\n+\tmemset(r, 0, sizeof(struct reftable_reader));\n+\n+\t/* Need +1 to read type of first block. */\n+\terr = block_source_read_block(source, &header, 0, header_size(2) + 1);\n+\tif (err != header_size(2) + 1) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tif (memcmp(header.data, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tr->version = header.data[4];\n+\tif (r->version != 1 && r->version != 2) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tr->size = block_source_size(source) - footer_size(r->version);\n+\tr->source = *source;\n+\tr->name = xstrdup(name);\n+\tr->hash_id = 0;\n+\n+\terr = block_source_read_block(source, &footer, r->size,\n+\t\t\t\t footer_size(r->version));\n+\tif (err != footer_size(r->version)) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = parse_footer(r, footer.data, header.data);\n+done:\n+\treftable_block_done(&footer);\n+\treftable_block_done(&header);\n+\treturn err;\n+}\n+\n+struct table_iter {\n+\tstruct reftable_reader *r;\n+\tuint8_t typ;\n+\tuint64_t block_off;\n+\tstruct block_iter bi;\n+\tint is_finished;\n+};\n+#define TABLE_ITER_INIT \\\n+\t{ \\\n+\t\t.bi = {.last_key = STRBUF_INIT } \\\n+\t}\n+\n+static void table_iter_copy_from(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = src->block_off;\n+\tdest->is_finished = src->is_finished;\n+\tblock_iter_copy_from(&dest->bi, &src->bi);\n+}\n+\n+static int table_iter_next_in_block(struct table_iter *ti,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tint res = block_iter_next(&ti->bi, rec);\n+\tif (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {\n+\t\t((struct reftable_ref_record *)rec->data)->update_index +=\n+\t\t\tti->r->min_update_index;\n+\t}\n+\n+\treturn res;\n+}\n+\n+static void table_iter_block_done(struct table_iter *ti)\n+{\n+\tif (ti->bi.br == NULL) {\n+\t\treturn;\n+\t}\n+\treftable_block_done(&ti->bi.br->block);\n+\tFREE_AND_NULL(ti->bi.br);\n+\n+\tti->bi.last_key.len = 0;\n+\tti->bi.next_off = 0;\n+}\n+\n+static int32_t extract_block_size(uint8_t *data, uint8_t *typ, uint64_t off,\n+\t\t\t\t int version)\n+{\n+\tint32_t result = 0;\n+\n+\tif (off == 0) {\n+\t\tdata += header_size(version);\n+\t}\n+\n+\t*typ = data[0];\n+\tif (reftable_is_block_type(*typ)) {\n+\t\tresult = get_be24(data + 1);\n+\t}\n+\treturn result;\n+}\n+\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ)\n+{\n+\tint32_t guess_block_size = r->block_size ? r->block_size :\n+\t\t\t\t\t\t\t DEFAULT_BLOCK_SIZE;\n+\tstruct reftable_block block = { 0 };\n+\tuint8_t block_typ = 0;\n+\tint err = 0;\n+\tuint32_t header_off = next_off ? 0 : header_size(r->version);\n+\tint32_t block_size = 0;\n+\n+\tif (next_off >= r->size)\n+\t\treturn 1;\n+\n+\terr = reader_get_block(r, &block, next_off, guess_block_size);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tblock_size = extract_block_size(block.data, &block_typ, next_off,\n+\t\t\t\t\tr->version);\n+\tif (block_size < 0)\n+\t\treturn block_size;\n+\n+\tif (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {\n+\t\treftable_block_done(&block);\n+\t\treturn 1;\n+\t}\n+\n+\tif (block_size > guess_block_size) {\n+\t\treftable_block_done(&block);\n+\t\terr = reader_get_block(r, &block, next_off, block_size);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\treturn block_reader_init(br, &block, header_off, r->block_size,\n+\t\t\t\t hash_size(r->hash_id));\n+}\n+\n+static int table_iter_next_block(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tuint64_t next_block_off = src->block_off + src->bi.br->full_block_size;\n+\tstruct block_reader br = { 0 };\n+\tint err = 0;\n+\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = next_block_off;\n+\n+\terr = reader_init_block_reader(src->r, &br, next_block_off, src->typ);\n+\tif (err > 0) {\n+\t\tdest->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\tif (err != 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct block_reader *brp =\n+\t\t\treftable_malloc(sizeof(struct block_reader));\n+\t\t*brp = br;\n+\n+\t\tdest->is_finished = 0;\n+\t\tblock_reader_start(brp, &dest->bi);\n+\t}\n+\treturn 0;\n+}\n+\n+static int table_iter_next(struct table_iter *ti, struct reftable_record *rec)\n+{\n+\tif (reftable_record_type(rec) != ti->typ)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\twhile (1) {\n+\t\tstruct table_iter next = TABLE_ITER_INIT;\n+\t\tint err = 0;\n+\t\tif (ti->is_finished) {\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\terr = table_iter_next_in_block(ti, rec);\n+\t\tif (err <= 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err != 0) {\n+\t\t\tti->is_finished = 1;\n+\t\t}\n+\t\ttable_iter_block_done(ti);\n+\t\tif (err != 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\ttable_iter_copy_from(ti, &next);\n+\t\tblock_iter_close(&next.bi);\n+\t}\n+}\n+\n+static int table_iter_next_void(void *ti, struct reftable_record *rec)\n+{\n+\treturn table_iter_next((struct table_iter *)ti, rec);\n+}\n+\n+static void table_iter_close(void *p)\n+{\n+\tstruct table_iter *ti = (struct table_iter *)p;\n+\ttable_iter_block_done(ti);\n+\tblock_iter_close(&ti->bi);\n+}\n+\n+static struct reftable_iterator_vtable table_iter_vtable = {\n+\t.next = &table_iter_next_void,\n+\t.close = &table_iter_close,\n+};\n+\n+static void iterator_from_table_iter(struct reftable_iterator *it,\n+\t\t\t\t struct table_iter *ti)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = ti;\n+\tit->ops = &table_iter_vtable;\n+}\n+\n+static int reader_table_iter_at(struct reftable_reader *r,\n+\t\t\t\tstruct table_iter *ti, uint64_t off,\n+\t\t\t\tuint8_t typ)\n+{\n+\tstruct block_reader br = { 0 };\n+\tstruct block_reader *brp = NULL;\n+\n+\tint err = reader_init_block_reader(r, &br, off, typ);\n+\tif (err != 0)\n+\t\treturn err;\n+\n+\tbrp = reftable_malloc(sizeof(struct block_reader));\n+\t*brp = br;\n+\tti->r = r;\n+\tti->typ = block_reader_type(brp);\n+\tti->block_off = off;\n+\tblock_reader_start(brp, &ti->bi);\n+\treturn 0;\n+}\n+\n+static int reader_start(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\tuint8_t typ, int index)\n+{\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tuint64_t off = offs->offset;\n+\tif (index) {\n+\t\toff = offs->index_offset;\n+\t\tif (off == 0) {\n+\t\t\treturn 1;\n+\t\t}\n+\t\ttyp = BLOCK_TYPE_INDEX;\n+\t}\n+\n+\treturn reader_table_iter_at(r, ti, off, typ);\n+}\n+\n+static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\t struct reftable_record *want)\n+{\n+\tstruct reftable_record rec =\n+\t\treftable_new_record(reftable_record_type(want));\n+\tstruct strbuf want_key = STRBUF_INIT;\n+\tstruct strbuf got_key = STRBUF_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = -1;\n+\n+\treftable_record_key(want, &want_key);\n+\n+\twhile (1) {\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = block_reader_first_key(next.bi.br, &got_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (strbuf_cmp(&got_key, &want_key) > 0) {\n+\t\t\ttable_iter_block_done(&next);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_block_done(ti);\n+\t\ttable_iter_copy_from(ti, &next);\n+\t}\n+\n+\terr = block_iter_seek(&ti->bi, &want_key);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = 0;\n+\n+done:\n+\tblock_iter_close(&next.bi);\n+\treftable_record_destroy(&rec);\n+\tstrbuf_release(&want_key);\n+\tstrbuf_release(&got_key);\n+\treturn err;\n+}\n+\n+static int reader_seek_indexed(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_index_record want_index = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record want_index_rec = { 0 };\n+\tstruct reftable_index_record index_result = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record index_result_rec = { 0 };\n+\tstruct table_iter index_iter = TABLE_ITER_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = 0;\n+\n+\treftable_record_key(rec, &want_index.last_key);\n+\treftable_record_from_index(&want_index_rec, &want_index);\n+\treftable_record_from_index(&index_result_rec, &index_result);\n+\n+\terr = reader_start(r, &index_iter, reftable_record_type(rec), 1);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reader_seek_linear(r, &index_iter, &want_index_rec);\n+\twhile (1) {\n+\t\terr = table_iter_next(&index_iter, &index_result_rec);\n+\t\ttable_iter_block_done(&index_iter);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = reader_table_iter_at(r, &next, index_result.offset, 0);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = block_iter_seek(&next.bi, &want_index.last_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (next.typ == reftable_record_type(rec)) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (next.typ != BLOCK_TYPE_INDEX) {\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_copy_from(&index_iter, &next);\n+\t}\n+\n+\tif (err == 0) {\n+\t\tstruct table_iter empty = TABLE_ITER_INIT;\n+\t\tstruct table_iter *malloced =\n+\t\t\treftable_calloc(sizeof(struct table_iter));\n+\t\t*malloced = empty;\n+\t\ttable_iter_copy_from(malloced, &next);\n+\t\titerator_from_table_iter(it, malloced);\n+\t}\n+done:\n+\tblock_iter_close(&next.bi);\n+\ttable_iter_close(&index_iter);\n+\treftable_record_clear(&want_index_rec);\n+\treftable_record_clear(&index_result_rec);\n+\treturn err;\n+}\n+\n+static int reader_seek_internal(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it,\n+\t\t\t\tstruct reftable_record *rec)\n+{\n+\tstruct reftable_reader_offsets *offs =\n+\t\treader_offsets_for(r, reftable_record_type(rec));\n+\tuint64_t idx = offs->index_offset;\n+\tstruct table_iter ti = TABLE_ITER_INIT;\n+\tint err = 0;\n+\tif (idx > 0)\n+\t\treturn reader_seek_indexed(r, it, rec);\n+\n+\terr = reader_start(r, &ti, reftable_record_type(rec), 0);\n+\tif (err < 0)\n+\t\treturn err;\n+\terr = reader_seek_linear(r, &ti, rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct table_iter *p =\n+\t\t\treftable_malloc(sizeof(struct table_iter));\n+\t\t*p = ti;\n+\t\titerator_from_table_iter(it, p);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec)\n+{\n+\tuint8_t typ = reftable_record_type(rec);\n+\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tif (!offs->is_present) {\n+\t\titerator_set_empty(it);\n+\t\treturn 0;\n+\t}\n+\n+\treturn reader_seek_internal(r, it, rec);\n+}\n+\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { 0 };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { 0 };\n+\treftable_record_from_log(&rec, &log);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_reader_seek_log_at(r, it, name, max);\n+}\n+\n+void reader_close(struct reftable_reader *r)\n+{\n+\tblock_source_close(&r->source);\n+\tFREE_AND_NULL(r->name);\n+}\n+\n+int reftable_new_reader(struct reftable_reader **p,\n+\t\t\tstruct reftable_block_source *src, char const *name)\n+{\n+\tstruct reftable_reader *rd =\n+\t\treftable_calloc(sizeof(struct reftable_reader));\n+\tint err = init_reader(rd, src, name);\n+\tif (err == 0) {\n+\t\t*p = rd;\n+\t} else {\n+\t\tblock_source_close(src);\n+\t\treftable_free(rd);\n+\t}\n+\treturn err;\n+}\n+\n+void reftable_reader_free(struct reftable_reader *r)\n+{\n+\treader_close(r);\n+\treftable_free(r);\n+}\n+\n+static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct reftable_obj_record want = {\n+\t\t.hash_prefix = oid,\n+\t\t.hash_prefix_len = r->object_id_len,\n+\t};\n+\tstruct reftable_record want_rec = { 0 };\n+\tstruct reftable_iterator oit = { 0 };\n+\tstruct reftable_obj_record got = { 0 };\n+\tstruct reftable_record got_rec = { 0 };\n+\tint err = 0;\n+\tstruct indexed_table_ref_iter *itr = NULL;\n+\n+\t/* Look through the reverse index. */\n+\treftable_record_from_obj(&want_rec, &want);\n+\terr = reader_seek(r, &oit, &want_rec);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\t/* read out the reftable_obj_record */\n+\treftable_record_from_obj(&got_rec, &got);\n+\terr = iterator_next(&oit, &got_rec);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 0 ||\n+\t memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {\n+\t\t/* didn't find it; return empty iterator */\n+\t\titerator_set_empty(it);\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),\n+\t\t\t\t\t got.offsets, got.offset_len);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tgot.offsets = NULL;\n+\titerator_from_indexed_table_ref_iter(it, itr);\n+\n+done:\n+\treftable_iterator_destroy(&oit);\n+\treftable_record_clear(&got_rec);\n+\treturn err;\n+}\n+\n+static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct table_iter ti_empty = TABLE_ITER_INIT;\n+\tstruct table_iter *ti = reftable_calloc(sizeof(struct table_iter));\n+\tstruct filtering_ref_iterator *filter = NULL;\n+\tstruct filtering_ref_iterator empty = FILTERING_REF_ITERATOR_INIT;\n+\tint oid_len = hash_size(r->hash_id);\n+\tint err;\n+\n+\t*ti = ti_empty;\n+\terr = reader_start(r, ti, BLOCK_TYPE_REF, 0);\n+\tif (err < 0) {\n+\t\treftable_free(ti);\n+\t\treturn err;\n+\t}\n+\n+\tfilter = reftable_malloc(sizeof(struct filtering_ref_iterator));\n+\t*filter = empty;\n+\n+\tstrbuf_add(&filter->oid, oid, oid_len);\n+\treftable_table_from_reader(&filter->tab, r);\n+\tfilter->double_check = 0;\n+\titerator_from_table_iter(&filter->it, ti);\n+\n+\titerator_from_filtering_ref_iterator(it, filter);\n+\treturn 0;\n+}\n+\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid)\n+{\n+\tif (r->obj_offsets.is_present)\n+\t\treturn reftable_reader_refs_for_indexed(r, it, oid);\n+\treturn reftable_reader_refs_for_unindexed(r, it, oid);\n+}\n+\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r)\n+{\n+\treturn r->max_update_index;\n+}\n+\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r)\n+{\n+\treturn r->min_update_index;\n+}\ndiff --git a/reftable/reader.h b/reftable/reader.h\nnew file mode 100644\nindex 0000000000..8170258881\n--- /dev/null\n+++ b/reftable/reader.h\n@@ -0,0 +1,78 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef READER_H\n+#define READER_H\n+\n+#include \"block.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source);\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size);\n+void block_source_close(struct reftable_block_source *source);\n+\n+/* metadata for a block type */\n+struct reftable_reader_offsets {\n+\tint is_present;\n+\tuint64_t offset;\n+\tuint64_t index_offset;\n+};\n+\n+/* The state for reading a reftable file. */\n+struct reftable_reader {\n+\t/* for convience, associate a name with the instance. */\n+\tchar *name;\n+\tstruct reftable_block_source source;\n+\n+\t/* Size of the file, excluding the footer. */\n+\tuint64_t size;\n+\n+\t/* 'sha1' for SHA1, 's256' for SHA-256 */\n+\tuint32_t hash_id;\n+\n+\tuint32_t block_size;\n+\tuint64_t min_update_index;\n+\tuint64_t max_update_index;\n+\t/* Length of the OID keys in the 'o' section */\n+\tint object_id_len;\n+\tint version;\n+\n+\tstruct reftable_reader_offsets ref_offsets;\n+\tstruct reftable_reader_offsets obj_offsets;\n+\tstruct reftable_reader_offsets log_offsets;\n+};\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name);\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec);\n+void reader_close(struct reftable_reader *r);\n+const char *reader_name(struct reftable_reader *r);\n+\n+/* initialize a block reader to read from `r` */\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ);\n+\n+/* generic interface to reftables */\n+struct reftable_table_vtable {\n+\tint (*seek_record)(void *tab, struct reftable_iterator *it,\n+\t\t\t struct reftable_record *);\n+\tuint32_t (*hash_id)(void *tab);\n+\tuint64_t (*min_update_index)(void *tab);\n+\tuint64_t (*max_update_index)(void *tab);\n+};\n+\n+int reftable_table_seek_record(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t struct reftable_record *rec);\n+\n+#endif\ndiff --git a/reftable/reftable.c b/reftable/reftable.c\nnew file mode 100644\nindex 0000000000..1425aef6f2\n--- /dev/null\n+++ b/reftable/reftable.c\n@@ -0,0 +1,104 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable.h\"\n+#include \"record.h\"\n+#include \"reader.h\"\n+\n+static int reftable_reader_seek_void(void *tab, struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn reader_seek((struct reftable_reader *)tab, it, rec);\n+}\n+\n+static uint32_t reftable_reader_hash_id_void(void *tab)\n+{\n+\treturn reftable_reader_hash_id((struct reftable_reader *)tab);\n+}\n+\n+static uint64_t reftable_reader_min_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_min_update_index((struct reftable_reader *)tab);\n+}\n+\n+static uint64_t reftable_reader_max_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_max_update_index((struct reftable_reader *)tab);\n+}\n+\n+static struct reftable_table_vtable reader_vtable = {\n+\t.seek_record = reftable_reader_seek_void,\n+\t.hash_id = reftable_reader_hash_id_void,\n+\t.min_update_index = reftable_reader_min_update_index_void,\n+\t.max_update_index = reftable_reader_max_update_index_void,\n+};\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { 0 };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader)\n+{\n+\tassert(tab->ops == NULL);\n+\ttab->ops = &reader_vtable;\n+\ttab->table_arg = reader;\n+}\n+\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_iterator it = { 0 };\n+\tint err = reftable_table_seek_ref(tab, &it, name);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_ref(&it, ref);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref->refname, name) ||\n+\t reftable_ref_record_is_deletion(ref)) {\n+\t\treftable_ref_record_clear(ref);\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+int reftable_table_seek_record(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t struct reftable_record *rec)\n+{\n+\treturn tab->ops->seek_record(tab->table_arg, it, rec);\n+}\n+\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->max_update_index(tab->table_arg);\n+}\n+\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->min_update_index(tab->table_arg);\n+}\n+\n+uint32_t reftable_table_hash_id(struct reftable_table *tab)\n+{\n+\treturn tab->ops->hash_id(tab->table_arg);\n+}\n-- \ngitgitgadget\n\n"},{"id":"406778","messageId":"68aee16e603ed66d7f7fe296df77bfa3d21db36f.1601568664.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v2.git.git.1601568663.gitgitgadget@gmail.com","subject":"[PATCH v2 09/13] reftable: write reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-01T16:10:59Z","receivedAt":"2020-10-01T16:11:31Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/writer.c | 673 ++++++++++++++++++++++++++++++++++++++++++++++\n reftable/writer.h | 51 ++++\n 3 files changed, 725 insertions(+)\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n\ndiff --git a/Makefile b/Makefile\nindex 870e540f22..c1dd046155 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2380,6 +2380,7 @@ REFTABLE_OBJS += reftable/compat.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/strbuf.o\n REFTABLE_OBJS += reftable/tree.o\n+REFTABLE_OBJS += reftable/writer.o\n REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/block_test.o\ndiff --git a/reftable/writer.c b/reftable/writer.c\nnew file mode 100644\nindex 0000000000..436756ad24\n--- /dev/null\n+++ b/reftable/writer.c\n@@ -0,0 +1,673 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"writer.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+#include \"tree.h\"\n+\n+/* finishes a block, and writes it to storage */\n+static int writer_flush_block(struct reftable_writer *w);\n+\n+/* deallocates memory related to the index */\n+static void writer_clear_index(struct reftable_writer *w);\n+\n+/* finishes writing a 'r' (refs) or 'g' (reflogs) section */\n+static int writer_finish_public_section(struct reftable_writer *w);\n+\n+static struct reftable_block_stats *\n+writer_reftable_block_stats(struct reftable_writer *w, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase 'r':\n+\t\treturn &w->stats.ref_stats;\n+\tcase 'o':\n+\t\treturn &w->stats.obj_stats;\n+\tcase 'i':\n+\t\treturn &w->stats.idx_stats;\n+\tcase 'g':\n+\t\treturn &w->stats.log_stats;\n+\t}\n+\tabort();\n+\treturn NULL;\n+}\n+\n+/* write data, queuing the padding for the next write. Returns negative for\n+ * error. */\n+static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n+\t\t\tint padding)\n+{\n+\tint n = 0;\n+\tif (w->pending_padding > 0) {\n+\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding);\n+\t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n+\t\tif (n < 0)\n+\t\t\treturn n;\n+\n+\t\tw->pending_padding = 0;\n+\t\treftable_free(zeroed);\n+\t}\n+\n+\tw->pending_padding = padding;\n+\tn = w->write(w->write_arg, data, len);\n+\tif (n < 0)\n+\t\treturn n;\n+\tn += padding;\n+\treturn 0;\n+}\n+\n+static void options_set_defaults(struct reftable_write_options *opts)\n+{\n+\tif (opts->restart_interval == 0) {\n+\t\topts->restart_interval = 16;\n+\t}\n+\n+\tif (opts->hash_id == 0) {\n+\t\topts->hash_id = SHA1_ID;\n+\t}\n+\tif (opts->block_size == 0) {\n+\t\topts->block_size = DEFAULT_BLOCK_SIZE;\n+\t}\n+}\n+\n+static int writer_version(struct reftable_writer *w)\n+{\n+\treturn (w->opts.hash_id == 0 || w->opts.hash_id == SHA1_ID) ? 1 : 2;\n+}\n+\n+static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n+{\n+\tmemcpy((char *)dest, \"REFT\", 4);\n+\n+\tdest[4] = writer_version(w);\n+\n+\tput_be24(dest + 5, w->opts.block_size);\n+\tput_be64(dest + 8, w->min_update_index);\n+\tput_be64(dest + 16, w->max_update_index);\n+\tif (writer_version(w) == 2) {\n+\t\tput_be32(dest + 24, w->opts.hash_id);\n+\t}\n+\treturn header_size(writer_version(w));\n+}\n+\n+static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n+{\n+\tint block_start = 0;\n+\tif (w->next == 0) {\n+\t\tblock_start = header_size(writer_version(w));\n+\t}\n+\n+\tstrbuf_release(&w->last_key);\n+\tblock_writer_init(&w->block_writer_data, typ, w->block,\n+\t\t\t w->opts.block_size, block_start,\n+\t\t\t hash_size(w->opts.hash_id));\n+\tw->block_writer = &w->block_writer_data;\n+\tw->block_writer->restart_interval = w->opts.restart_interval;\n+}\n+\n+struct reftable_writer *\n+reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts)\n+{\n+\tstruct reftable_writer *wp =\n+\t\treftable_calloc(sizeof(struct reftable_writer));\n+\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n+\toptions_set_defaults(opts);\n+\tif (opts->block_size >= (1 << 24)) {\n+\t\t/* TODO - error return? */\n+\t\tabort();\n+\t}\n+\twp->last_key = reftable_empty_strbuf;\n+\twp->block = reftable_calloc(opts->block_size);\n+\twp->write = writer_func;\n+\twp->write_arg = writer_arg;\n+\twp->opts = *opts;\n+\twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n+\n+\treturn wp;\n+}\n+\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max)\n+{\n+\tw->min_update_index = min;\n+\tw->max_update_index = max;\n+}\n+\n+void reftable_writer_free(struct reftable_writer *w)\n+{\n+\treftable_free(w->block);\n+\treftable_free(w);\n+}\n+\n+struct obj_index_tree_node {\n+\tstruct strbuf hash;\n+\tuint64_t *offsets;\n+\tsize_t offset_len;\n+\tsize_t offset_cap;\n+};\n+\n+#define OBJ_INDEX_TREE_NODE_INIT \\\n+\t{ \\\n+\t\t.hash = STRBUF_INIT \\\n+\t}\n+\n+static int obj_index_tree_node_compare(const void *a, const void *b)\n+{\n+\treturn strbuf_cmp(&((const struct obj_index_tree_node *)a)->hash,\n+\t\t\t &((const struct obj_index_tree_node *)b)->hash);\n+}\n+\n+static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n+{\n+\tuint64_t off = w->next;\n+\n+\tstruct obj_index_tree_node want = { .hash = *hash };\n+\n+\tstruct tree_node *node = tree_search(&want, &w->obj_index_tree,\n+\t\t\t\t\t &obj_index_tree_node_compare, 0);\n+\tstruct obj_index_tree_node *key = NULL;\n+\tif (node == NULL) {\n+\t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n+\t\tkey = reftable_malloc(sizeof(struct obj_index_tree_node));\n+\t\t*key = empty;\n+\n+\t\tstrbuf_reset(&key->hash);\n+\t\tstrbuf_addbuf(&key->hash, hash);\n+\t\ttree_search((void *)key, &w->obj_index_tree,\n+\t\t\t &obj_index_tree_node_compare, 1);\n+\t} else {\n+\t\tkey = node->key;\n+\t}\n+\n+\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off) {\n+\t\treturn;\n+\t}\n+\n+\tif (key->offset_len == key->offset_cap) {\n+\t\tkey->offset_cap = 2 * key->offset_cap + 1;\n+\t\tkey->offsets = reftable_realloc(\n+\t\t\tkey->offsets, sizeof(uint64_t) * key->offset_cap);\n+\t}\n+\n+\tkey->offsets[key->offset_len++] = off;\n+}\n+\n+static int writer_add_record(struct reftable_writer *w,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tint result = -1;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = 0;\n+\treftable_record_key(rec, &key);\n+\tif (strbuf_cmp(&w->last_key, &key) >= 0)\n+\t\tgoto done;\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, &key);\n+\tif (w->block_writer == NULL) {\n+\t\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\t}\n+\n+\tassert(block_writer_type(w->block_writer) == reftable_record_type(rec));\n+\n+\tif (block_writer_add(w->block_writer, rec) == 0) {\n+\t\tresult = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0) {\n+\t\tresult = err;\n+\t\tgoto done;\n+\t}\n+\n+\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\terr = block_writer_add(w->block_writer, rec);\n+\tif (err < 0) {\n+\t\tresult = err;\n+\t\tgoto done;\n+\t}\n+\n+\tresult = 0;\n+done:\n+\tstrbuf_release(&key);\n+\treturn result;\n+}\n+\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_ref_record copy = *ref;\n+\tint err = 0;\n+\n+\tif (ref->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\tif (ref->update_index < w->min_update_index ||\n+\t ref->update_index > w->max_update_index)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\treftable_record_from_ref(&rec, ©);\n+\tcopy.update_index -= w->min_update_index;\n+\terr = writer_add_record(w, &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (!w->opts.skip_index_objects && ref->value != NULL) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, (char *)ref->value, hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\n+\tif (!w->opts.skip_index_objects && ref->target_value != NULL) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, ref->target_value, hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(refs, n, reftable_ref_record_compare_name);\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_ref(w, &refs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tchar *input_log_message = log->message;\n+\tstruct strbuf cleaned_message = STRBUF_INIT;\n+\tint err;\n+\tif (log->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\tif (w->block_writer != NULL &&\n+\t block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {\n+\t\tint err = writer_finish_public_section(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tif (!w->opts.exact_log_message && log->message != NULL) {\n+\t\tstrbuf_addstr(&cleaned_message, log->message);\n+\t\twhile (cleaned_message.len &&\n+\t\t cleaned_message.buf[cleaned_message.len - 1] == '\\n')\n+\t\t\tstrbuf_setlen(&cleaned_message,\n+\t\t\t\t cleaned_message.len - 1);\n+\t\tif (strchr(cleaned_message.buf, '\\n')) {\n+\t\t\t// multiple lines not allowed.\n+\t\t\terr = REFTABLE_API_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tstrbuf_addstr(&cleaned_message, \"\\n\");\n+\t\tlog->message = cleaned_message.buf;\n+\t}\n+\n+\tw->next -= w->pending_padding;\n+\tw->pending_padding = 0;\n+\n+\treftable_record_from_log(&rec, log);\n+\terr = writer_add_record(w, &rec);\n+\n+done:\n+\tlog->message = input_log_message;\n+\tstrbuf_release(&cleaned_message);\n+\treturn err;\n+}\n+\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(logs, n, reftable_log_record_compare_key);\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_log(w, &logs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+static int writer_finish_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tuint64_t index_start = 0;\n+\tint max_level = 0;\n+\tint threshold = w->opts.unpadded ? 1 : 3;\n+\tint before_blocks = w->stats.idx_stats.blocks;\n+\tint err = writer_flush_block(w);\n+\tint i = 0;\n+\tstruct reftable_block_stats *bstats = NULL;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\twhile (w->index_len > threshold) {\n+\t\tstruct reftable_index_record *idx = NULL;\n+\t\tint idx_len = 0;\n+\n+\t\tmax_level++;\n+\t\tindex_start = w->next;\n+\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\tidx = w->index;\n+\t\tidx_len = w->index_len;\n+\n+\t\tw->index = NULL;\n+\t\tw->index_len = 0;\n+\t\tw->index_cap = 0;\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstruct reftable_record rec = { NULL };\n+\t\t\treftable_record_from_index(&rec, idx + i);\n+\t\t\tif (block_writer_add(w->block_writer, &rec) == 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\terr = writer_flush_block(w);\n+\t\t\tif (err < 0)\n+\t\t\t\treturn err;\n+\n+\t\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\t\terr = block_writer_add(w->block_writer, &rec);\n+\t\t\tif (err != 0) {\n+\t\t\t\t/* write into fresh block should always succeed\n+\t\t\t\t */\n+\t\t\t\tabort();\n+\t\t\t}\n+\t\t}\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstrbuf_release(&idx[i].last_key);\n+\t\t}\n+\t\treftable_free(idx);\n+\t}\n+\n+\twriter_clear_index(w);\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tbstats = writer_reftable_block_stats(w, typ);\n+\tbstats->index_blocks = w->stats.idx_stats.blocks - before_blocks;\n+\tbstats->index_offset = index_start;\n+\tbstats->max_index_level = max_level;\n+\n+\t/* Reinit lastKey, as the next section can start with any key. */\n+\tw->last_key.len = 0;\n+\n+\treturn 0;\n+}\n+\n+struct common_prefix_arg {\n+\tstruct strbuf *last;\n+\tint max;\n+};\n+\n+static void update_common(void *void_arg, void *key)\n+{\n+\tstruct common_prefix_arg *arg = (struct common_prefix_arg *)void_arg;\n+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n+\tif (arg->last != NULL) {\n+\t\tint n = common_prefix_size(&entry->hash, arg->last);\n+\t\tif (n > arg->max) {\n+\t\t\targ->max = n;\n+\t\t}\n+\t}\n+\targ->last = &entry->hash;\n+}\n+\n+struct write_record_arg {\n+\tstruct reftable_writer *w;\n+\tint err;\n+};\n+\n+static void write_object_record(void *void_arg, void *key)\n+{\n+\tstruct write_record_arg *arg = (struct write_record_arg *)void_arg;\n+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n+\tstruct reftable_obj_record obj_rec = {\n+\t\t.hash_prefix = (uint8_t *)entry->hash.buf,\n+\t\t.hash_prefix_len = arg->w->stats.object_id_len,\n+\t\t.offsets = entry->offsets,\n+\t\t.offset_len = entry->offset_len,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\treftable_record_from_obj(&rec, &obj_rec);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\n+\targ->err = writer_flush_block(arg->w);\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\twriter_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\tobj_rec.offset_len = 0;\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\n+\t/* Should be able to write into a fresh block. */\n+\tassert(arg->err == 0);\n+\n+done:;\n+}\n+\n+static void object_record_free(void *void_arg, void *key)\n+{\n+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n+\n+\tFREE_AND_NULL(entry->offsets);\n+\tstrbuf_release(&entry->hash);\n+\treftable_free(entry);\n+}\n+\n+static int writer_dump_object_index(struct reftable_writer *w)\n+{\n+\tstruct write_record_arg closure = { .w = w };\n+\tstruct common_prefix_arg common = { NULL };\n+\tif (w->obj_index_tree != NULL) {\n+\t\tinfix_walk(w->obj_index_tree, &update_common, &common);\n+\t}\n+\tw->stats.object_id_len = common.max + 1;\n+\n+\twriter_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\n+\tif (w->obj_index_tree != NULL) {\n+\t\tinfix_walk(w->obj_index_tree, &write_object_record, &closure);\n+\t}\n+\n+\tif (closure.err < 0)\n+\t\treturn closure.err;\n+\treturn writer_finish_section(w);\n+}\n+\n+static int writer_finish_public_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = 0;\n+\tint err = 0;\n+\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\n+\ttyp = block_writer_type(w->block_writer);\n+\terr = writer_finish_section(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\tif (typ == BLOCK_TYPE_REF && !w->opts.skip_index_objects &&\n+\t w->stats.ref_stats.index_blocks > 0) {\n+\t\terr = writer_dump_object_index(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tif (w->obj_index_tree != NULL) {\n+\t\tinfix_walk(w->obj_index_tree, &object_record_free, NULL);\n+\t\ttree_free(w->obj_index_tree);\n+\t\tw->obj_index_tree = NULL;\n+\t}\n+\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+int reftable_writer_close(struct reftable_writer *w)\n+{\n+\tuint8_t footer[72];\n+\tuint8_t *p = footer;\n+\tint err = writer_finish_public_section(w);\n+\tint empty_table = w->next == 0;\n+\tif (err != 0)\n+\t\tgoto done;\n+\tw->pending_padding = 0;\n+\tif (empty_table) {\n+\t\t/* Empty tables need a header anyway. */\n+\t\tuint8_t header[28];\n+\t\tint n = writer_write_header(w, header);\n+\t\terr = padded_write(w, header, n, 0);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\tp += writer_write_header(w, footer);\n+\tput_be64(p, w->stats.ref_stats.index_offset);\n+\tp += 8;\n+\tput_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n+\tp += 8;\n+\tput_be64(p, w->stats.obj_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be64(p, w->stats.log_stats.offset);\n+\tp += 8;\n+\tput_be64(p, w->stats.log_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be32(p, crc32(0, footer, p - footer));\n+\tp += 4;\n+\n+\terr = padded_write(w, footer, footer_size(writer_version(w)), 0);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (empty_table) {\n+\t\terr = REFTABLE_EMPTY_TABLE_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\t/* free up memory. */\n+\tblock_writer_clear(&w->block_writer_data);\n+\twriter_clear_index(w);\n+\tstrbuf_release(&w->last_key);\n+\treturn err;\n+}\n+\n+static void writer_clear_index(struct reftable_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->index_len; i++) {\n+\t\tstrbuf_release(&w->index[i].last_key);\n+\t}\n+\n+\tFREE_AND_NULL(w->index);\n+\tw->index_len = 0;\n+\tw->index_cap = 0;\n+}\n+\n+static const int debug = 0;\n+\n+static int writer_flush_nonempty_block(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tstruct reftable_block_stats *bstats =\n+\t\twriter_reftable_block_stats(w, typ);\n+\tuint64_t block_typ_off = (bstats->blocks == 0) ? w->next : 0;\n+\tint raw_bytes = block_writer_finish(w->block_writer);\n+\tint padding = 0;\n+\tint err = 0;\n+\tstruct reftable_index_record ir = { .last_key = STRBUF_INIT };\n+\tif (raw_bytes < 0)\n+\t\treturn raw_bytes;\n+\n+\tif (!w->opts.unpadded && typ != BLOCK_TYPE_LOG) {\n+\t\tpadding = w->opts.block_size - raw_bytes;\n+\t}\n+\n+\tif (block_typ_off > 0) {\n+\t\tbstats->offset = block_typ_off;\n+\t}\n+\n+\tbstats->entries += w->block_writer->entries;\n+\tbstats->restarts += w->block_writer->restart_len;\n+\tbstats->blocks++;\n+\tw->stats.blocks++;\n+\n+\tif (debug) {\n+\t\tfprintf(stderr, \"block %c off %\" PRIu64 \" sz %d (%d)\\n\", typ,\n+\t\t\tw->next, raw_bytes,\n+\t\t\tget_be24(w->block + w->block_writer->header_off + 1));\n+\t}\n+\n+\tif (w->next == 0) {\n+\t\twriter_write_header(w, w->block);\n+\t}\n+\n+\terr = padded_write(w, w->block, raw_bytes, padding);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (w->index_cap == w->index_len) {\n+\t\tw->index_cap = 2 * w->index_cap + 1;\n+\t\tw->index = reftable_realloc(\n+\t\t\tw->index,\n+\t\t\tsizeof(struct reftable_index_record) * w->index_cap);\n+\t}\n+\n+\tir.offset = w->next;\n+\tstrbuf_reset(&ir.last_key);\n+\tstrbuf_addbuf(&ir.last_key, &w->block_writer->last_key);\n+\tw->index[w->index_len] = ir;\n+\n+\tw->index_len++;\n+\tw->next += padding + raw_bytes;\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+static int writer_flush_block(struct reftable_writer *w)\n+{\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\tif (w->block_writer->entries == 0)\n+\t\treturn 0;\n+\treturn writer_flush_nonempty_block(w);\n+}\n+\n+const struct reftable_stats *writer_stats(struct reftable_writer *w)\n+{\n+\treturn &w->stats;\n+}\ndiff --git a/reftable/writer.h b/reftable/writer.h\nnew file mode 100644\nindex 0000000000..06e0705a79\n--- /dev/null\n+++ b/reftable/writer.h\n@@ -0,0 +1,51 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef WRITER_H\n+#define WRITER_H\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"reftable.h\"\n+#include \"strbuf.h\"\n+#include \"tree.h\"\n+\n+struct reftable_writer {\n+\tint (*write)(void *, const void *, size_t);\n+\tvoid *write_arg;\n+\tint pending_padding;\n+\tstruct strbuf last_key;\n+\n+\t/* offset of next block to write. */\n+\tuint64_t next;\n+\tuint64_t min_update_index, max_update_index;\n+\tstruct reftable_write_options opts;\n+\n+\t/* memory buffer for writing */\n+\tuint8_t *block;\n+\n+\t/* writer for the current section. NULL or points to\n+\t * block_writer_data */\n+\tstruct block_writer *block_writer;\n+\n+\tstruct block_writer block_writer_data;\n+\n+\t/* pending index records for the current section */\n+\tstruct reftable_index_record *index;\n+\tsize_t index_len;\n+\tsize_t index_cap;\n+\n+\t/*\n+\t tree for use with tsearch; used to populate the 'o' inverse OID\n+\t map */\n+\tstruct tree_node *obj_index_tree;\n+\n+\tstruct reftable_stats stats;\n+};\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"406779","messageId":"8eb944ea9b1c0bdc4a3ebb4e79fd12187f6c988c.1601568663.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v2.git.git.1601568663.gitgitgadget@gmail.com","subject":"[PATCH v2 06/13] reftable: (de)serialization for the polymorphic record type.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-01T16:10:56Z","receivedAt":"2020-10-01T16:11:31Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/constants.h | 21 +\n reftable/record.c | 1116 ++++++++++++++++++++++++++++++++++++++\n reftable/record.h | 137 +++++\n reftable/record_test.c | 410 ++++++++++++++\n t/helper/test-reftable.c | 1 +\n 6 files changed, 1687 insertions(+)\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n\ndiff --git a/Makefile b/Makefile\nindex e40d55cd87..b5e43d79ed 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2376,8 +2376,10 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/compat.o\n+REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/strbuf.o\n \n+REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/strbuf_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n \ndiff --git a/reftable/constants.h b/reftable/constants.h\nnew file mode 100644\nindex 0000000000..5eee72c4c1\n--- /dev/null\n+++ b/reftable/constants.h\n@@ -0,0 +1,21 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef CONSTANTS_H\n+#define CONSTANTS_H\n+\n+#define BLOCK_TYPE_LOG 'g'\n+#define BLOCK_TYPE_INDEX 'i'\n+#define BLOCK_TYPE_REF 'r'\n+#define BLOCK_TYPE_OBJ 'o'\n+#define BLOCK_TYPE_ANY 0\n+\n+#define MAX_RESTARTS ((1 << 16) - 1)\n+#define DEFAULT_BLOCK_SIZE 4096\n+\n+#endif\ndiff --git a/reftable/record.c b/reftable/record.c\nnew file mode 100644\nindex 0000000000..f050667ea4\n--- /dev/null\n+++ b/reftable/record.c\n@@ -0,0 +1,1116 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+/* record.c - methods for different types of records. */\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"constants.h\"\n+#include \"reftable.h\"\n+#include \"basics.h\"\n+\n+int get_var_int(uint64_t *dest, struct string_view *in)\n+{\n+\tint ptr = 0;\n+\tuint64_t val;\n+\n+\tif (in->len == 0)\n+\t\treturn -1;\n+\tval = in->buf[ptr] & 0x7f;\n+\n+\twhile (in->buf[ptr] & 0x80) {\n+\t\tptr++;\n+\t\tif (ptr > in->len) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tval = (val + 1) << 7 | (uint64_t)(in->buf[ptr] & 0x7f);\n+\t}\n+\n+\t*dest = val;\n+\treturn ptr + 1;\n+}\n+\n+int put_var_int(struct string_view *dest, uint64_t val)\n+{\n+\tuint8_t buf[10] = { 0 };\n+\tint i = 9;\n+\tint n = 0;\n+\tbuf[i] = (uint8_t)(val & 0x7f);\n+\ti--;\n+\twhile (1) {\n+\t\tval >>= 7;\n+\t\tif (!val) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tval--;\n+\t\tbuf[i] = 0x80 | (uint8_t)(val & 0x7f);\n+\t\ti--;\n+\t}\n+\n+\tn = sizeof(buf) - i - 1;\n+\tif (dest->len < n)\n+\t\treturn -1;\n+\tmemcpy(dest->buf, &buf[i + 1], n);\n+\treturn n;\n+}\n+\n+int reftable_is_block_type(uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\tcase BLOCK_TYPE_LOG:\n+\tcase BLOCK_TYPE_OBJ:\n+\tcase BLOCK_TYPE_INDEX:\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+static int decode_string(struct strbuf *dest, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t tsize = 0;\n+\tint n = get_var_int(&tsize, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\tif (in.len < tsize)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, in.buf, tsize);\n+\tstring_view_consume(&in, tsize);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static int encode_string(char *str, struct string_view s)\n+{\n+\tstruct string_view start = s;\n+\tint l = strlen(str);\n+\tint n = put_var_int(&s, l);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\tif (s.len < l)\n+\t\treturn -1;\n+\tmemcpy(s.buf, str, l);\n+\tstring_view_consume(&s, l);\n+\n+\treturn start.len - s.len;\n+}\n+\n+int reftable_encode_key(int *restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra)\n+{\n+\tstruct string_view start = dest;\n+\tint prefix_len = common_prefix_size(&prev_key, &key);\n+\tuint64_t suffix_len = key.len - prefix_len;\n+\tint n = put_var_int(&dest, (uint64_t)prefix_len);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\t*restart = (prefix_len == 0);\n+\n+\tn = put_var_int(&dest, suffix_len << 3 | (uint64_t)extra);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\tif (dest.len < suffix_len)\n+\t\treturn -1;\n+\tmemcpy(dest.buf, key.buf + prefix_len, suffix_len);\n+\tstring_view_consume(&dest, suffix_len);\n+\n+\treturn start.len - dest.len;\n+}\n+\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t prefix_len = 0;\n+\tuint64_t suffix_len = 0;\n+\tint n = get_var_int(&prefix_len, &in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tif (prefix_len > last_key.len)\n+\t\treturn -1;\n+\n+\tn = get_var_int(&suffix_len, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\t*extra = (uint8_t)(suffix_len & 0x7);\n+\tsuffix_len >>= 3;\n+\n+\tif (in.len < suffix_len)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(key);\n+\tstrbuf_add(key, last_key.buf, prefix_len);\n+\tstrbuf_add(key, in.buf, suffix_len);\n+\tstring_view_consume(&in, suffix_len);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static void reftable_ref_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_ref_record *rec =\n+\t\t(const struct reftable_ref_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, rec->refname);\n+}\n+\n+static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *ref = (struct reftable_ref_record *)rec;\n+\tstruct reftable_ref_record *src = (struct reftable_ref_record *)src_rec;\n+\tassert(hash_size > 0);\n+\n+\t/* This is simple and correct, but we could probably reuse the hash\n+\t fields. */\n+\treftable_ref_record_clear(ref);\n+\tif (src->refname != NULL) {\n+\t\tref->refname = xstrdup(src->refname);\n+\t}\n+\n+\tif (src->target != NULL) {\n+\t\tref->target = xstrdup(src->target);\n+\t}\n+\n+\tif (src->target_value != NULL) {\n+\t\tref->target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->target_value, src->target_value, hash_size);\n+\t}\n+\n+\tif (src->value != NULL) {\n+\t\tref->value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value, src->value, hash_size);\n+\t}\n+\tref->update_index = src->update_index;\n+}\n+\n+static char hexdigit(int c)\n+{\n+\tif (c <= 9)\n+\t\treturn '0' + c;\n+\treturn 'a' + (c - 10);\n+}\n+\n+static void hex_format(char *dest, uint8_t *src, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (src != NULL) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < hash_size; i++) {\n+\t\t\tdest[2 * i] = hexdigit(src[i] >> 4);\n+\t\t\tdest[2 * i + 1] = hexdigit(src[i] & 0xf);\n+\t\t}\n+\t\tdest[2 * hash_size] = 0;\n+\t}\n+}\n+\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[SHA256_SIZE + 1] = { 0 };\n+\tprintf(\"ref{%s(%\" PRIu64 \") \", ref->refname, ref->update_index);\n+\tif (ref->value != NULL) {\n+\t\thex_format(hex, ref->value, hash_size(hash_id));\n+\t\tprintf(\"%s\", hex);\n+\t}\n+\tif (ref->target_value != NULL) {\n+\t\thex_format(hex, ref->target_value, hash_size(hash_id));\n+\t\tprintf(\" (T %s)\", hex);\n+\t}\n+\tif (ref->target != NULL) {\n+\t\tprintf(\"=> %s\", ref->target);\n+\t}\n+\tprintf(\"}\\n\");\n+}\n+\n+static void reftable_ref_record_clear_void(void *rec)\n+{\n+\treftable_ref_record_clear((struct reftable_ref_record *)rec);\n+}\n+\n+void reftable_ref_record_clear(struct reftable_ref_record *ref)\n+{\n+\treftable_free(ref->refname);\n+\treftable_free(ref->target);\n+\treftable_free(ref->target_value);\n+\treftable_free(ref->value);\n+\tmemset(ref, 0, sizeof(struct reftable_ref_record));\n+}\n+\n+static uint8_t reftable_ref_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\tif (r->value != NULL) {\n+\t\tif (r->target_value != NULL) {\n+\t\t\treturn 2;\n+\t\t} else {\n+\t\t\treturn 1;\n+\t\t}\n+\t} else if (r->target != NULL)\n+\t\treturn 3;\n+\treturn 0;\n+}\n+\n+static int reftable_ref_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\tstruct string_view start = s;\n+\tint n = put_var_int(&s, r->update_index);\n+\tassert(hash_size > 0);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tif (r->value != NULL) {\n+\t\tif (s.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t}\n+\n+\tif (r->target_value != NULL) {\n+\t\tif (s.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->target_value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t}\n+\n+\tif (r->target != NULL) {\n+\t\tint n = encode_string(r->target, s);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t}\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_ref_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *r = (struct reftable_ref_record *)rec;\n+\tstruct string_view start = in;\n+\tint seen_value = 0;\n+\tint seen_target_value = 0;\n+\tint seen_target = 0;\n+\n+\tint n = get_var_int(&r->update_index, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tassert(hash_size > 0);\n+\n+\tstring_view_consume(&in, n);\n+\n+\tr->refname = reftable_realloc(r->refname, key.len + 1);\n+\tmemcpy(r->refname, key.buf, key.len);\n+\tr->refname[key.len] = 0;\n+\n+\tswitch (val_type) {\n+\tcase 1:\n+\tcase 2:\n+\t\tif (in.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (r->value == NULL) {\n+\t\t\tr->value = reftable_malloc(hash_size);\n+\t\t}\n+\t\tseen_value = 1;\n+\t\tmemcpy(r->value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tif (val_type == 1) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (r->target_value == NULL) {\n+\t\t\tr->target_value = reftable_malloc(hash_size);\n+\t\t}\n+\t\tseen_target_value = 1;\n+\t\tmemcpy(r->target_value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\tcase 3: {\n+\t\tstruct strbuf dest = STRBUF_INIT;\n+\t\tint n = decode_string(&dest, in);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\t\tseen_target = 1;\n+\t\tif (r->target != NULL) {\n+\t\t\treftable_free(r->target);\n+\t\t}\n+\t\tr->target = dest.buf;\n+\t} break;\n+\n+\tcase 0:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t\tbreak;\n+\t}\n+\n+\tif (!seen_target && r->target != NULL) {\n+\t\tFREE_AND_NULL(r->target);\n+\t}\n+\tif (!seen_target_value && r->target_value != NULL) {\n+\t\tFREE_AND_NULL(r->target_value);\n+\t}\n+\tif (!seen_value && r->value != NULL) {\n+\t\tFREE_AND_NULL(r->value);\n+\t}\n+\n+\treturn start.len - in.len;\n+}\n+\n+static int reftable_ref_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_ref_record_is_deletion(\n+\t\t(const struct reftable_ref_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_ref_record_vtable = {\n+\t.key = &reftable_ref_record_key,\n+\t.type = BLOCK_TYPE_REF,\n+\t.copy_from = &reftable_ref_record_copy_from,\n+\t.val_type = &reftable_ref_record_val_type,\n+\t.encode = &reftable_ref_record_encode,\n+\t.decode = &reftable_ref_record_decode,\n+\t.clear = &reftable_ref_record_clear_void,\n+\t.is_deletion = &reftable_ref_record_is_deletion_void,\n+};\n+\n+static void reftable_obj_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_obj_record *rec =\n+\t\t(const struct reftable_obj_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, rec->hash_prefix, rec->hash_prefix_len);\n+}\n+\n+static void reftable_obj_record_clear(void *rec)\n+{\n+\tstruct reftable_obj_record *obj = (struct reftable_obj_record *)rec;\n+\tFREE_AND_NULL(obj->hash_prefix);\n+\tFREE_AND_NULL(obj->offsets);\n+\tmemset(obj, 0, sizeof(struct reftable_obj_record));\n+}\n+\n+static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_obj_record *obj = (struct reftable_obj_record *)rec;\n+\tconst struct reftable_obj_record *src =\n+\t\t(const struct reftable_obj_record *)src_rec;\n+\tint olen;\n+\n+\treftable_obj_record_clear(obj);\n+\t*obj = *src;\n+\tobj->hash_prefix = reftable_malloc(obj->hash_prefix_len);\n+\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n+\n+\tolen = obj->offset_len * sizeof(uint64_t);\n+\tobj->offsets = reftable_malloc(olen);\n+\tmemcpy(obj->offsets, src->offsets, olen);\n+}\n+\n+static uint8_t reftable_obj_record_val_type(const void *rec)\n+{\n+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n+\tif (r->offset_len > 0 && r->offset_len < 8)\n+\t\treturn r->offset_len;\n+\treturn 0;\n+}\n+\n+static int reftable_obj_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n+\tstruct string_view start = s;\n+\tint i = 0;\n+\tint n = 0;\n+\tuint64_t last = 0;\n+\tif (r->offset_len == 0 || r->offset_len >= 8) {\n+\t\tn = put_var_int(&s, r->offset_len);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t}\n+\tif (r->offset_len == 0)\n+\t\treturn start.len - s.len;\n+\tn = put_var_int(&s, r->offsets[0]);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tlast = r->offsets[0];\n+\tfor (i = 1; i < r->offset_len; i++) {\n+\t\tint n = put_var_int(&s, r->offsets[i] - last);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tlast = r->offsets[i];\n+\t}\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_obj_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n+\tuint64_t count = val_type;\n+\tint n = 0;\n+\tuint64_t last;\n+\tint j;\n+\tr->hash_prefix = reftable_malloc(key.len);\n+\tmemcpy(r->hash_prefix, key.buf, key.len);\n+\tr->hash_prefix_len = key.len;\n+\n+\tif (val_type == 0) {\n+\t\tn = get_var_int(&count, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\n+\t\tstring_view_consume(&in, n);\n+\t}\n+\n+\tr->offsets = NULL;\n+\tr->offset_len = 0;\n+\tif (count == 0)\n+\t\treturn start.len - in.len;\n+\n+\tr->offsets = reftable_malloc(count * sizeof(uint64_t));\n+\tr->offset_len = count;\n+\n+\tn = get_var_int(&r->offsets[0], &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\tlast = r->offsets[0];\n+\tj = 1;\n+\twhile (j < count) {\n+\t\tuint64_t delta = 0;\n+\t\tint n = get_var_int(&delta, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\n+\t\tlast = r->offsets[j] = (delta + last);\n+\t\tj++;\n+\t}\n+\treturn start.len - in.len;\n+}\n+\n+static int not_a_deletion(const void *p)\n+{\n+\treturn 0;\n+}\n+\n+static struct reftable_record_vtable reftable_obj_record_vtable = {\n+\t.key = &reftable_obj_record_key,\n+\t.type = BLOCK_TYPE_OBJ,\n+\t.copy_from = &reftable_obj_record_copy_from,\n+\t.val_type = &reftable_obj_record_val_type,\n+\t.encode = &reftable_obj_record_encode,\n+\t.decode = &reftable_obj_record_decode,\n+\t.clear = &reftable_obj_record_clear,\n+\t.is_deletion = not_a_deletion,\n+};\n+\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[SHA256_SIZE + 1] = { 0 };\n+\n+\tprintf(\"log{%s(%\" PRIu64 \") %s <%s> %\" PRIu64 \" %04d\\n\", log->refname,\n+\t log->update_index, log->name, log->email, log->time,\n+\t log->tz_offset);\n+\thex_format(hex, log->old_hash, hash_size(hash_id));\n+\tprintf(\"%s => \", hex);\n+\thex_format(hex, log->new_hash, hash_size(hash_id));\n+\tprintf(\"%s\\n\\n%s\\n}\\n\", hex, log->message);\n+}\n+\n+static void reftable_log_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_log_record *rec =\n+\t\t(const struct reftable_log_record *)r;\n+\tint len = strlen(rec->refname);\n+\tuint8_t i64[8];\n+\tuint64_t ts = 0;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, (uint8_t *)rec->refname, len + 1);\n+\n+\tts = (~ts) - rec->update_index;\n+\tput_be64(&i64[0], ts);\n+\tstrbuf_add(dest, i64, sizeof(i64));\n+}\n+\n+static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_log_record *dst = (struct reftable_log_record *)rec;\n+\tconst struct reftable_log_record *src =\n+\t\t(const struct reftable_log_record *)src_rec;\n+\n+\treftable_log_record_clear(dst);\n+\t*dst = *src;\n+\tif (dst->refname != NULL) {\n+\t\tdst->refname = xstrdup(dst->refname);\n+\t}\n+\tif (dst->email != NULL) {\n+\t\tdst->email = xstrdup(dst->email);\n+\t}\n+\tif (dst->name != NULL) {\n+\t\tdst->name = xstrdup(dst->name);\n+\t}\n+\tif (dst->message != NULL) {\n+\t\tdst->message = xstrdup(dst->message);\n+\t}\n+\n+\tif (dst->new_hash != NULL) {\n+\t\tdst->new_hash = reftable_malloc(hash_size);\n+\t\tmemcpy(dst->new_hash, src->new_hash, hash_size);\n+\t}\n+\tif (dst->old_hash != NULL) {\n+\t\tdst->old_hash = reftable_malloc(hash_size);\n+\t\tmemcpy(dst->old_hash, src->old_hash, hash_size);\n+\t}\n+}\n+\n+static void reftable_log_record_clear_void(void *rec)\n+{\n+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n+\treftable_log_record_clear(r);\n+}\n+\n+void reftable_log_record_clear(struct reftable_log_record *r)\n+{\n+\treftable_free(r->refname);\n+\treftable_free(r->new_hash);\n+\treftable_free(r->old_hash);\n+\treftable_free(r->name);\n+\treftable_free(r->email);\n+\treftable_free(r->message);\n+\tmemset(r, 0, sizeof(struct reftable_log_record));\n+}\n+\n+static uint8_t reftable_log_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_log_record *log =\n+\t\t(const struct reftable_log_record *)rec;\n+\n+\treturn reftable_log_record_is_deletion(log) ? 0 : 1;\n+}\n+\n+static uint8_t zero[SHA256_SIZE] = { 0 };\n+\n+static int reftable_log_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n+\tstruct string_view start = s;\n+\tint n = 0;\n+\tuint8_t *oldh = r->old_hash;\n+\tuint8_t *newh = r->new_hash;\n+\tif (reftable_log_record_is_deletion(r))\n+\t\treturn 0;\n+\n+\tif (oldh == NULL) {\n+\t\toldh = zero;\n+\t}\n+\tif (newh == NULL) {\n+\t\tnewh = zero;\n+\t}\n+\n+\tif (s.len < 2 * hash_size)\n+\t\treturn -1;\n+\n+\tmemcpy(s.buf, oldh, hash_size);\n+\tmemcpy(s.buf + hash_size, newh, hash_size);\n+\tstring_view_consume(&s, 2 * hash_size);\n+\n+\tn = encode_string(r->name ? r->name : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = encode_string(r->email ? r->email : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = put_var_int(&s, r->time);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tif (s.len < 2)\n+\t\treturn -1;\n+\n+\tput_be16(s.buf, r->tz_offset);\n+\tstring_view_consume(&s, 2);\n+\n+\tn = encode_string(r->message ? r->message : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_log_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n+\tuint64_t max = 0;\n+\tuint64_t ts = 0;\n+\tstruct strbuf dest = STRBUF_INIT;\n+\tint n;\n+\n+\tif (key.len <= 9 || key.buf[key.len - 9] != 0)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->refname = reftable_realloc(r->refname, key.len - 8);\n+\tmemcpy(r->refname, key.buf, key.len - 8);\n+\tts = get_be64(key.buf + key.len - 8);\n+\n+\tr->update_index = (~max) - ts;\n+\n+\tif (val_type == 0) {\n+\t\tFREE_AND_NULL(r->old_hash);\n+\t\tFREE_AND_NULL(r->new_hash);\n+\t\tFREE_AND_NULL(r->message);\n+\t\tFREE_AND_NULL(r->email);\n+\t\tFREE_AND_NULL(r->name);\n+\t\treturn 0;\n+\t}\n+\n+\tif (in.len < 2 * hash_size)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->old_hash = reftable_realloc(r->old_hash, hash_size);\n+\tr->new_hash = reftable_realloc(r->new_hash, hash_size);\n+\n+\tmemcpy(r->old_hash, in.buf, hash_size);\n+\tmemcpy(r->new_hash, in.buf + hash_size, hash_size);\n+\n+\tstring_view_consume(&in, 2 * hash_size);\n+\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->name = reftable_realloc(r->name, dest.len + 1);\n+\tmemcpy(r->name, dest.buf, dest.len);\n+\tr->name[dest.len] = 0;\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->email = reftable_realloc(r->email, dest.len + 1);\n+\tmemcpy(r->email, dest.buf, dest.len);\n+\tr->email[dest.len] = 0;\n+\n+\tts = 0;\n+\tn = get_var_int(&ts, &in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\tr->time = ts;\n+\tif (in.len < 2)\n+\t\tgoto done;\n+\n+\tr->tz_offset = get_be16(in.buf);\n+\tstring_view_consume(&in, 2);\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->message = reftable_realloc(r->message, dest.len + 1);\n+\tmemcpy(r->message, dest.buf, dest.len);\n+\tr->message[dest.len] = 0;\n+\n+\tstrbuf_release(&dest);\n+\treturn start.len - in.len;\n+\n+done:\n+\tstrbuf_release(&dest);\n+\treturn REFTABLE_FORMAT_ERROR;\n+}\n+\n+static int null_streq(char *a, char *b)\n+{\n+\tchar *empty = \"\";\n+\tif (a == NULL)\n+\t\ta = empty;\n+\n+\tif (b == NULL)\n+\t\tb = empty;\n+\n+\treturn 0 == strcmp(a, b);\n+}\n+\n+static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)\n+{\n+\tif (a == NULL)\n+\t\ta = zero;\n+\n+\tif (b == NULL)\n+\t\tb = zero;\n+\n+\treturn !memcmp(a, b, sz);\n+}\n+\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size)\n+{\n+\treturn null_streq(a->name, b->name) && null_streq(a->email, b->email) &&\n+\t null_streq(a->message, b->message) &&\n+\t zero_hash_eq(a->old_hash, b->old_hash, hash_size) &&\n+\t zero_hash_eq(a->new_hash, b->new_hash, hash_size) &&\n+\t a->time == b->time && a->tz_offset == b->tz_offset &&\n+\t a->update_index == b->update_index;\n+}\n+\n+static int reftable_log_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_log_record_is_deletion(\n+\t\t(const struct reftable_log_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_log_record_vtable = {\n+\t.key = &reftable_log_record_key,\n+\t.type = BLOCK_TYPE_LOG,\n+\t.copy_from = &reftable_log_record_copy_from,\n+\t.val_type = &reftable_log_record_val_type,\n+\t.encode = &reftable_log_record_encode,\n+\t.decode = &reftable_log_record_decode,\n+\t.clear = &reftable_log_record_clear_void,\n+\t.is_deletion = &reftable_log_record_is_deletion_void,\n+};\n+\n+struct reftable_record reftable_new_record(uint8_t typ)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF: {\n+\t\tstruct reftable_ref_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_ref_record));\n+\t\treftable_record_from_ref(&rec, r);\n+\t\treturn rec;\n+\t}\n+\n+\tcase BLOCK_TYPE_OBJ: {\n+\t\tstruct reftable_obj_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_obj_record));\n+\t\treftable_record_from_obj(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_LOG: {\n+\t\tstruct reftable_log_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_log_record));\n+\t\treftable_record_from_log(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_INDEX: {\n+\t\tstruct reftable_index_record empty = { .last_key =\n+\t\t\t\t\t\t\t STRBUF_INIT };\n+\t\tstruct reftable_index_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_index_record));\n+\t\t*r = empty;\n+\t\treftable_record_from_index(&rec, r);\n+\t\treturn rec;\n+\t}\n+\t}\n+\tabort();\n+\treturn rec;\n+}\n+\n+/* clear out the record, yielding the reftable_record data that was\n+ * encapsulated. */\n+static void *reftable_record_yield(struct reftable_record *rec)\n+{\n+\tvoid *p = rec->data;\n+\trec->data = NULL;\n+\treturn p;\n+}\n+\n+void reftable_record_destroy(struct reftable_record *rec)\n+{\n+\treftable_record_clear(rec);\n+\treftable_free(reftable_record_yield(rec));\n+}\n+\n+static void reftable_index_record_key(const void *r, struct strbuf *dest)\n+{\n+\tstruct reftable_index_record *rec = (struct reftable_index_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addbuf(dest, &rec->last_key);\n+}\n+\n+static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_index_record *dst = (struct reftable_index_record *)rec;\n+\tstruct reftable_index_record *src =\n+\t\t(struct reftable_index_record *)src_rec;\n+\n+\tstrbuf_reset(&dst->last_key);\n+\tstrbuf_addbuf(&dst->last_key, &src->last_key);\n+\tdst->offset = src->offset;\n+}\n+\n+static void reftable_index_record_clear(void *rec)\n+{\n+\tstruct reftable_index_record *idx = (struct reftable_index_record *)rec;\n+\tstrbuf_release(&idx->last_key);\n+}\n+\n+static uint8_t reftable_index_record_val_type(const void *rec)\n+{\n+\treturn 0;\n+}\n+\n+static int reftable_index_record_encode(const void *rec, struct string_view out,\n+\t\t\t\t\tint hash_size)\n+{\n+\tconst struct reftable_index_record *r =\n+\t\t(const struct reftable_index_record *)rec;\n+\tstruct string_view start = out;\n+\n+\tint n = put_var_int(&out, r->offset);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&out, n);\n+\n+\treturn start.len - out.len;\n+}\n+\n+static int reftable_index_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t\tuint8_t val_type, struct string_view in,\n+\t\t\t\t\tint hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_index_record *r = (struct reftable_index_record *)rec;\n+\tint n = 0;\n+\n+\tstrbuf_reset(&r->last_key);\n+\tstrbuf_addbuf(&r->last_key, &key);\n+\n+\tn = get_var_int(&r->offset, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&in, n);\n+\treturn start.len - in.len;\n+}\n+\n+static struct reftable_record_vtable reftable_index_record_vtable = {\n+\t.key = &reftable_index_record_key,\n+\t.type = BLOCK_TYPE_INDEX,\n+\t.copy_from = &reftable_index_record_copy_from,\n+\t.val_type = &reftable_index_record_val_type,\n+\t.encode = &reftable_index_record_encode,\n+\t.decode = &reftable_index_record_decode,\n+\t.clear = &reftable_index_record_clear,\n+\t.is_deletion = ¬_a_deletion,\n+};\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)\n+{\n+\trec->ops->key(rec->data, dest);\n+}\n+\n+uint8_t reftable_record_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->type;\n+}\n+\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size)\n+{\n+\treturn rec->ops->encode(rec->data, dest, hash_size);\n+}\n+\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size)\n+{\n+\tassert(src->ops->type == rec->ops->type);\n+\n+\trec->ops->copy_from(rec->data, src->data, hash_size);\n+}\n+\n+uint8_t reftable_record_val_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->val_type(rec->data);\n+}\n+\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src, int hash_size)\n+{\n+\treturn rec->ops->decode(rec->data, key, extra, src, hash_size);\n+}\n+\n+void reftable_record_clear(struct reftable_record *rec)\n+{\n+\trec->ops->clear(rec->data);\n+}\n+\n+int reftable_record_is_deletion(struct reftable_record *rec)\n+{\n+\treturn rec->ops->is_deletion(rec->data);\n+}\n+\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *ref_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = ref_rec;\n+\trec->ops = &reftable_ref_record_vtable;\n+}\n+\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *obj_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = obj_rec;\n+\trec->ops = &reftable_obj_record_vtable;\n+}\n+\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *index_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = index_rec;\n+\trec->ops = &reftable_index_record_vtable;\n+}\n+\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *log_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = log_rec;\n+\trec->ops = &reftable_log_record_vtable;\n+}\n+\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_REF);\n+\treturn (struct reftable_ref_record *)rec->data;\n+}\n+\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_LOG);\n+\treturn (struct reftable_log_record *)rec->data;\n+}\n+\n+static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)\n+{\n+\tif (a != NULL && b != NULL)\n+\t\treturn !memcmp(a, b, hash_size);\n+\n+\treturn a == b;\n+}\n+\n+static int str_equal(char *a, char *b)\n+{\n+\tif (a != NULL && b != NULL)\n+\t\treturn 0 == strcmp(a, b);\n+\n+\treturn a == b;\n+}\n+\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\treturn 0 == strcmp(a->refname, b->refname) &&\n+\t a->update_index == b->update_index &&\n+\t hash_equal(a->value, b->value, hash_size) &&\n+\t hash_equal(a->target_value, b->target_value, hash_size) &&\n+\t str_equal(a->target, b->target);\n+}\n+\n+int reftable_ref_record_compare_name(const void *a, const void *b)\n+{\n+\treturn strcmp(((struct reftable_ref_record *)a)->refname,\n+\t\t ((struct reftable_ref_record *)b)->refname);\n+}\n+\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref)\n+{\n+\treturn ref->value == NULL && ref->target == NULL &&\n+\t ref->target_value == NULL;\n+}\n+\n+int reftable_log_record_compare_key(const void *a, const void *b)\n+{\n+\tstruct reftable_log_record *la = (struct reftable_log_record *)a;\n+\tstruct reftable_log_record *lb = (struct reftable_log_record *)b;\n+\n+\tint cmp = strcmp(la->refname, lb->refname);\n+\tif (cmp)\n+\t\treturn cmp;\n+\tif (la->update_index > lb->update_index)\n+\t\treturn -1;\n+\treturn (la->update_index < lb->update_index) ? 1 : 0;\n+}\n+\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log)\n+{\n+\treturn (log->new_hash == NULL && log->old_hash == NULL &&\n+\t\tlog->name == NULL && log->email == NULL &&\n+\t\tlog->message == NULL && log->time == 0 && log->tz_offset == 0 &&\n+\t\tlog->message == NULL);\n+}\n+\n+void string_view_consume(struct string_view *s, int n)\n+{\n+\ts->buf += n;\n+\ts->len -= n;\n+}\ndiff --git a/reftable/record.h b/reftable/record.h\nnew file mode 100644\nindex 0000000000..dd5881ed75\n--- /dev/null\n+++ b/reftable/record.h\n@@ -0,0 +1,137 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef RECORD_H\n+#define RECORD_H\n+\n+#include \"reftable.h\"\n+#include \"strbuf.h\"\n+#include \"system.h\"\n+\n+/*\n+ A substring of existing string data. This structure takes no responsibility\n+ for the lifetime of the data it points to.\n+*/\n+struct string_view {\n+\tuint8_t *buf;\n+\tsize_t len;\n+};\n+\n+/* Advance `s.buf` by `n`, and decrease length. */\n+void string_view_consume(struct string_view *s, int n);\n+\n+/* utilities for de/encoding varints */\n+\n+int get_var_int(uint64_t *dest, struct string_view *in);\n+int put_var_int(struct string_view *dest, uint64_t val);\n+\n+/* Methods for records. */\n+struct reftable_record_vtable {\n+\t/* encode the key of to a uint8_t strbuf. */\n+\tvoid (*key)(const void *rec, struct strbuf *dest);\n+\n+\t/* The record type of ('r' for ref). */\n+\tuint8_t type;\n+\n+\tvoid (*copy_from)(void *dest, const void *src, int hash_size);\n+\n+\t/* a value of [0..7], indicating record subvariants (eg. ref vs. symref\n+\t * vs ref deletion) */\n+\tuint8_t (*val_type)(const void *rec);\n+\n+\t/* encodes rec into dest, returning how much space was used. */\n+\tint (*encode)(const void *rec, struct string_view dest, int hash_size);\n+\n+\t/* decode data from `src` into the record. */\n+\tint (*decode)(void *rec, struct strbuf key, uint8_t extra,\n+\t\t struct string_view src, int hash_size);\n+\n+\t/* deallocate and null the record. */\n+\tvoid (*clear)(void *rec);\n+\n+\t/* is this a tombstone? */\n+\tint (*is_deletion)(const void *rec);\n+};\n+\n+/* record is a generic wrapper for different types of records. */\n+struct reftable_record {\n+\tvoid *data;\n+\tstruct reftable_record_vtable *ops;\n+};\n+\n+/* returns true for recognized block types. Block start with the block type. */\n+int reftable_is_block_type(uint8_t typ);\n+\n+/* creates a malloced record of the given type. Dispose with record_destroy */\n+struct reftable_record reftable_new_record(uint8_t typ);\n+\n+/* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns\n+ number of bytes written. */\n+int reftable_encode_key(int *is_restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra);\n+\n+/* Decode into `key` and `extra` from `in` */\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in);\n+\n+/* reftable_index_record are used internally to speed up lookups. */\n+struct reftable_index_record {\n+\tuint64_t offset; /* Offset of block */\n+\tstruct strbuf last_key; /* Last key of the block. */\n+};\n+\n+/* reftable_obj_record stores an object ID => ref mapping. */\n+struct reftable_obj_record {\n+\tuint8_t *hash_prefix; /* leading bytes of the object ID */\n+\tint hash_prefix_len; /* number of leading bytes. Constant\n+\t\t\t * across a single table. */\n+\tuint64_t *offsets; /* a vector of file offsets. */\n+\tint offset_len;\n+};\n+\n+/* see struct record_vtable */\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);\n+uint8_t reftable_record_type(struct reftable_record *rec);\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size);\n+uint8_t reftable_record_val_type(struct reftable_record *rec);\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size);\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src,\n+\t\t\t int hash_size);\n+int reftable_record_is_deletion(struct reftable_record *rec);\n+\n+/* zeroes out the embedded record */\n+void reftable_record_clear(struct reftable_record *rec);\n+\n+/* clear and deallocate embedded record, and zero `rec`. */\n+void reftable_record_destroy(struct reftable_record *rec);\n+\n+/* initialize generic records from concrete records. The generic record should\n+ * be zeroed out. */\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *objrec);\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *idxrec);\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *refrec);\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *logrec);\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *ref);\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *ref);\n+\n+/* for qsort. */\n+int reftable_ref_record_compare_name(const void *a, const void *b);\n+\n+/* for qsort. */\n+int reftable_log_record_compare_key(const void *a, const void *b);\n+\n+#endif\ndiff --git a/reftable/record_test.c b/reftable/record_test.c\nnew file mode 100644\nindex 0000000000..dcf1c74cb7\n--- /dev/null\n+++ b/reftable/record_test.c\n@@ -0,0 +1,410 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"reftable.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_copy(struct reftable_record *rec)\n+{\n+\tstruct reftable_record copy =\n+\t\treftable_new_record(reftable_record_type(rec));\n+\treftable_record_copy_from(©, rec, SHA1_SIZE);\n+\t/* do it twice to catch memory leaks */\n+\treftable_record_copy_from(©, rec, SHA1_SIZE);\n+\tswitch (reftable_record_type(©)) {\n+\tcase BLOCK_TYPE_REF:\n+\t\tassert(reftable_ref_record_equal(reftable_record_as_ref(©),\n+\t\t\t\t\t\t reftable_record_as_ref(rec),\n+\t\t\t\t\t\t SHA1_SIZE));\n+\t\tbreak;\n+\tcase BLOCK_TYPE_LOG:\n+\t\tassert(reftable_log_record_equal(reftable_record_as_log(©),\n+\t\t\t\t\t\t reftable_record_as_log(rec),\n+\t\t\t\t\t\t SHA1_SIZE));\n+\t\tbreak;\n+\t}\n+\treftable_record_destroy(©);\n+}\n+\n+static void test_varint_roundtrip(void)\n+{\n+\tuint64_t inputs[] = { 0,\n+\t\t\t 1,\n+\t\t\t 27,\n+\t\t\t 127,\n+\t\t\t 128,\n+\t\t\t 257,\n+\t\t\t 4096,\n+\t\t\t ((uint64_t)1 << 63),\n+\t\t\t ((uint64_t)1 << 63) + ((uint64_t)1 << 63) - 1 };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(inputs); i++) {\n+\t\tuint8_t dest[10];\n+\n+\t\tstruct string_view out = {\n+\t\t\t.buf = dest,\n+\t\t\t.len = sizeof(dest),\n+\t\t};\n+\t\tuint64_t in = inputs[i];\n+\t\tint n = put_var_int(&out, in);\n+\t\tuint64_t got = 0;\n+\n+\t\tassert(n > 0);\n+\t\tout.len = n;\n+\t\tn = get_var_int(&got, &out);\n+\t\tassert(n > 0);\n+\n+\t\tassert(got == in);\n+\t}\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct {\n+\t\tconst char *a, *b;\n+\t\tint want;\n+\t} cases[] = {\n+\t\t{ \"abc\", \"ab\", 2 },\n+\t\t{ \"\", \"abc\", 0 },\n+\t\t{ \"abc\", \"abd\", 2 },\n+\t\t{ \"abc\", \"pqr\", 0 },\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct strbuf a = STRBUF_INIT;\n+\t\tstruct strbuf b = STRBUF_INIT;\n+\t\tstrbuf_addstr(&a, cases[i].a);\n+\t\tstrbuf_addstr(&b, cases[i].b);\n+\t\tassert(common_prefix_size(&a, &b) == cases[i].want);\n+\n+\t\tstrbuf_release(&a);\n+\t\tstrbuf_release(&b);\n+\t}\n+}\n+\n+static void set_hash(uint8_t *h, int j)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < hash_size(SHA1_ID); i++) {\n+\t\th[i] = (j >> i) & 0xff;\n+\t}\n+}\n+\n+static void test_reftable_ref_record_roundtrip(void)\n+{\n+\tint i = 0;\n+\n+\tfor (i = 0; i <= 3; i++) {\n+\t\tstruct reftable_ref_record in = { NULL };\n+\t\tstruct reftable_ref_record out = {\n+\t\t\t.refname = xstrdup(\"old name\"),\n+\t\t\t.value = reftable_calloc(SHA1_SIZE),\n+\t\t\t.target_value = reftable_calloc(SHA1_SIZE),\n+\t\t\t.target = xstrdup(\"old value\"),\n+\t\t};\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\n+\t\tint n, m;\n+\n+\t\tswitch (i) {\n+\t\tcase 0:\n+\t\t\tbreak;\n+\t\tcase 1:\n+\t\t\tin.value = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_hash(in.value, 1);\n+\t\t\tbreak;\n+\t\tcase 2:\n+\t\t\tin.value = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_hash(in.value, 1);\n+\t\t\tin.target_value = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_hash(in.target_value, 2);\n+\t\t\tbreak;\n+\t\tcase 3:\n+\t\t\tin.target = xstrdup(\"target\");\n+\t\t\tbreak;\n+\t\t}\n+\t\tin.refname = xstrdup(\"refs/heads/master\");\n+\n+\t\treftable_record_from_ref(&rec, &in);\n+\t\ttest_copy(&rec);\n+\n+\t\tassert(reftable_record_val_type(&rec) == i);\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\t\tassert(n > 0);\n+\n+\t\t/* decode into a non-zero reftable_record to test for leaks. */\n+\n+\t\treftable_record_from_ref(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, i, dest, SHA1_SIZE);\n+\t\tassert(n == m);\n+\n+\t\tassert((out.value != NULL) == (in.value != NULL));\n+\t\tassert((out.target_value != NULL) == (in.target_value != NULL));\n+\t\tassert((out.target != NULL) == (in.target != NULL));\n+\t\treftable_record_clear(&rec_out);\n+\n+\t\tstrbuf_release(&key);\n+\t\treftable_ref_record_clear(&in);\n+\t}\n+}\n+\n+static void test_reftable_log_record_equal(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 42,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t}\n+\t};\n+\n+\tassert(!reftable_log_record_equal(&in[0], &in[1], SHA1_SIZE));\n+\tin[1].update_index = in[0].update_index;\n+\tassert(reftable_log_record_equal(&in[0], &in[1], SHA1_SIZE));\n+\treftable_log_record_clear(&in[0]);\n+\treftable_log_record_clear(&in[1]);\n+}\n+\n+static void test_reftable_log_record_roundtrip(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.old_hash = reftable_malloc(SHA1_SIZE),\n+\t\t\t.new_hash = reftable_malloc(SHA1_SIZE),\n+\t\t\t.name = xstrdup(\"han-wen\"),\n+\t\t\t.email = xstrdup(\"hanwen@google.com\"),\n+\t\t\t.message = xstrdup(\"test\"),\n+\t\t\t.update_index = 42,\n+\t\t\t.time = 1577123507,\n+\t\t\t.tz_offset = 100,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t}\n+\t};\n+\tset_test_hash(in[0].new_hash, 1);\n+\tset_test_hash(in[0].old_hash, 2);\n+\tfor (int i = 0; i < ARRAY_SIZE(in); i++) {\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\t/* populate out, to check for leaks. */\n+\t\tstruct reftable_log_record out = {\n+\t\t\t.refname = xstrdup(\"old name\"),\n+\t\t\t.new_hash = reftable_calloc(SHA1_SIZE),\n+\t\t\t.old_hash = reftable_calloc(SHA1_SIZE),\n+\t\t\t.name = xstrdup(\"old name\"),\n+\t\t\t.email = xstrdup(\"old@email\"),\n+\t\t\t.message = xstrdup(\"old message\"),\n+\t\t};\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m, valtype;\n+\n+\t\treftable_record_from_log(&rec, &in[i]);\n+\n+\t\ttest_copy(&rec);\n+\n+\t\treftable_record_key(&rec, &key);\n+\n+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\t\tassert(n >= 0);\n+\t\treftable_record_from_log(&rec_out, &out);\n+\t\tvaltype = reftable_record_val_type(&rec);\n+\t\tm = reftable_record_decode(&rec_out, key, valtype, dest,\n+\t\t\t\t\t SHA1_SIZE);\n+\t\tassert(n == m);\n+\n+\t\tassert(reftable_log_record_equal(&in[i], &out, SHA1_SIZE));\n+\t\treftable_log_record_clear(&in[i]);\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_clear(&rec_out);\n+\t}\n+}\n+\n+static void test_u24_roundtrip(void)\n+{\n+\tuint32_t in = 0x112233;\n+\tuint8_t dest[3];\n+\tuint32_t out;\n+\tput_be24(dest, in);\n+\tout = get_be24(dest);\n+\tassert(in == out);\n+}\n+\n+static void test_key_roundtrip(void)\n+{\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct strbuf roundtrip = STRBUF_INIT;\n+\tint restart;\n+\tuint8_t extra;\n+\tint n, m;\n+\tuint8_t rt_extra;\n+\n+\tstrbuf_addstr(&last_key, \"refs/heads/master\");\n+\tstrbuf_addstr(&key, \"refs/tags/bla\");\n+\textra = 6;\n+\tn = reftable_encode_key(&restart, dest, last_key, key, extra);\n+\tassert(!restart);\n+\tassert(n > 0);\n+\n+\tm = reftable_decode_key(&roundtrip, &rt_extra, last_key, dest);\n+\tassert(n == m);\n+\tassert(0 == strbuf_cmp(&key, &roundtrip));\n+\tassert(rt_extra == extra);\n+\n+\tstrbuf_release(&last_key);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&roundtrip);\n+}\n+\n+static void test_reftable_obj_record_roundtrip(void)\n+{\n+\tuint8_t testHash1[SHA1_SIZE] = { 1, 2, 3, 4, 0 };\n+\tuint64_t till9[] = { 1, 2, 3, 4, 500, 600, 700, 800, 9000 };\n+\tstruct reftable_obj_record recs[3] = { {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 3,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 9,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t } };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(recs); i++) {\n+\t\tstruct reftable_obj_record in = recs[i];\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_obj_record out = { NULL };\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m;\n+\t\tuint8_t extra;\n+\n+\t\treftable_record_from_obj(&rec, &in);\n+\t\ttest_copy(&rec);\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\t\tassert(n > 0);\n+\t\textra = reftable_record_val_type(&rec);\n+\t\treftable_record_from_obj(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, extra, dest,\n+\t\t\t\t\t SHA1_SIZE);\n+\t\tassert(n == m);\n+\n+\t\tassert(in.hash_prefix_len == out.hash_prefix_len);\n+\t\tassert(in.offset_len == out.offset_len);\n+\n+\t\tassert(!memcmp(in.hash_prefix, out.hash_prefix,\n+\t\t\t in.hash_prefix_len));\n+\t\tassert(0 == memcmp(in.offsets, out.offsets,\n+\t\t\t\t sizeof(uint64_t) * in.offset_len));\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_clear(&rec_out);\n+\t}\n+}\n+\n+static void test_reftable_index_record_roundtrip(void)\n+{\n+\tstruct reftable_index_record in = {\n+\t\t.offset = 42,\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_index_record out = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record out_rec = { NULL };\n+\tint n, m;\n+\tuint8_t extra;\n+\n+\tstrbuf_addstr(&in.last_key, \"refs/heads/master\");\n+\treftable_record_from_index(&rec, &in);\n+\treftable_record_key(&rec, &key);\n+\ttest_copy(&rec);\n+\n+\tassert(0 == strbuf_cmp(&key, &in.last_key));\n+\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\tassert(n > 0);\n+\n+\textra = reftable_record_val_type(&rec);\n+\treftable_record_from_index(&out_rec, &out);\n+\tm = reftable_record_decode(&out_rec, key, extra, dest, SHA1_SIZE);\n+\tassert(m == n);\n+\n+\tassert(in.offset == out.offset);\n+\n+\treftable_record_clear(&out_rec);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&in.last_key);\n+}\n+\n+int record_test_main(int argc, const char *argv[])\n+{\n+\tadd_test_case(\"test_reftable_log_record_equal\",\n+\t\t &test_reftable_log_record_equal);\n+\tadd_test_case(\"test_reftable_log_record_roundtrip\",\n+\t\t &test_reftable_log_record_roundtrip);\n+\tadd_test_case(\"test_reftable_ref_record_roundtrip\",\n+\t\t &test_reftable_ref_record_roundtrip);\n+\tadd_test_case(\"test_varint_roundtrip\", &test_varint_roundtrip);\n+\tadd_test_case(\"test_key_roundtrip\", &test_key_roundtrip);\n+\tadd_test_case(\"test_common_prefix\", &test_common_prefix);\n+\tadd_test_case(\"test_reftable_obj_record_roundtrip\",\n+\t\t &test_reftable_obj_record_roundtrip);\n+\tadd_test_case(\"test_reftable_index_record_roundtrip\",\n+\t\t &test_reftable_index_record_roundtrip);\n+\tadd_test_case(\"test_u24_roundtrip\", &test_u24_roundtrip);\n+\treturn test_main(argc, argv);\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 7d50aa6bcc..9341272089 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -3,6 +3,7 @@\n \n int cmd__reftable(int argc, const char **argv)\n {\n+\trecord_test_main(argc, argv);\n \tstrbuf_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"406780","messageId":"c535f838d62b90b4f24919323fdfaecfdadb2e2b.1601568664.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v2.git.git.1601568663.gitgitgadget@gmail.com","subject":"[PATCH v2 13/13] reftable: \"test-tool dump-reftable\" command.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-01T16:11:03Z","receivedAt":"2020-10-01T16:11:42Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis command dumps individual tables or a stack of of tables.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 ++\n reftable/iter.c | 6 +++---\n reftable/reader.c | 22 +++++++++++-----------\n t/helper/test-reftable.c | 5 +++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 6 files changed, 23 insertions(+), 14 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 05021f79a1..5f92e5cfc2 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2391,6 +2391,8 @@ REFTABLE_OBJS += reftable/writer.o\n REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/dump.o\n+REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/refname_test.o\n REFTABLE_TEST_OBJS += reftable/reftable_test.o\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex 2cff447323..80bc65e07d 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -59,7 +59,7 @@ void reftable_iterator_destroy(struct reftable_iterator *it)\n int reftable_iterator_next_ref(struct reftable_iterator *it,\n \t\t\t struct reftable_ref_record *ref)\n {\n-\tstruct reftable_record rec = { 0 };\n+\tstruct reftable_record rec = { NULL };\n \treftable_record_from_ref(&rec, ref);\n \treturn iterator_next(it, &rec);\n }\n@@ -67,7 +67,7 @@ int reftable_iterator_next_ref(struct reftable_iterator *it,\n int reftable_iterator_next_log(struct reftable_iterator *it,\n \t\t\t struct reftable_log_record *log)\n {\n-\tstruct reftable_record rec = { 0 };\n+\tstruct reftable_record rec = { NULL };\n \treftable_record_from_log(&rec, log);\n \treturn iterator_next(it, &rec);\n }\n@@ -95,7 +95,7 @@ static int filtering_ref_iterator_next(void *iter_arg,\n \t\t}\n \n \t\tif (fri->double_check) {\n-\t\t\tstruct reftable_iterator it = { 0 };\n+\t\t\tstruct reftable_iterator it = { NULL };\n \n \t\t\terr = reftable_table_seek_ref(&fri->tab, &it,\n \t\t\t\t\t\t ref->refname);\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex c7f56b5fdc..8d3f63b0d8 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -161,8 +161,8 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n \t\tconst char *name)\n {\n-\tstruct reftable_block footer = { 0 };\n-\tstruct reftable_block header = { 0 };\n+\tstruct reftable_block footer = { NULL };\n+\tstruct reftable_block header = { NULL };\n \tint err = 0;\n \n \tmemset(r, 0, sizeof(struct reftable_reader));\n@@ -270,7 +270,7 @@ int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n {\n \tint32_t guess_block_size = r->block_size ? r->block_size :\n \t\t\t\t\t\t\t DEFAULT_BLOCK_SIZE;\n-\tstruct reftable_block block = { 0 };\n+\tstruct reftable_block block = { NULL };\n \tuint8_t block_typ = 0;\n \tint err = 0;\n \tuint32_t header_off = next_off ? 0 : header_size(r->version);\n@@ -477,9 +477,9 @@ static int reader_seek_indexed(struct reftable_reader *r,\n \t\t\t struct reftable_record *rec)\n {\n \tstruct reftable_index_record want_index = { .last_key = STRBUF_INIT };\n-\tstruct reftable_record want_index_rec = { 0 };\n+\tstruct reftable_record want_index_rec = { NULL };\n \tstruct reftable_index_record index_result = { .last_key = STRBUF_INIT };\n-\tstruct reftable_record index_result_rec = { 0 };\n+\tstruct reftable_record index_result_rec = { NULL };\n \tstruct table_iter index_iter = TABLE_ITER_INIT;\n \tstruct table_iter next = TABLE_ITER_INIT;\n \tint err = 0;\n@@ -584,7 +584,7 @@ int reftable_reader_seek_ref(struct reftable_reader *r,\n \tstruct reftable_ref_record ref = {\n \t\t.refname = (char *)name,\n \t};\n-\tstruct reftable_record rec = { 0 };\n+\tstruct reftable_record rec = { NULL };\n \treftable_record_from_ref(&rec, &ref);\n \treturn reader_seek(r, it, &rec);\n }\n@@ -597,7 +597,7 @@ int reftable_reader_seek_log_at(struct reftable_reader *r,\n \t\t.refname = (char *)name,\n \t\t.update_index = update_index,\n \t};\n-\tstruct reftable_record rec = { 0 };\n+\tstruct reftable_record rec = { NULL };\n \treftable_record_from_log(&rec, &log);\n \treturn reader_seek(r, it, &rec);\n }\n@@ -644,10 +644,10 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n \t\t.hash_prefix = oid,\n \t\t.hash_prefix_len = r->object_id_len,\n \t};\n-\tstruct reftable_record want_rec = { 0 };\n-\tstruct reftable_iterator oit = { 0 };\n-\tstruct reftable_obj_record got = { 0 };\n-\tstruct reftable_record got_rec = { 0 };\n+\tstruct reftable_record want_rec = { NULL };\n+\tstruct reftable_iterator oit = { NULL };\n+\tstruct reftable_obj_record got = { NULL };\n+\tstruct reftable_record got_rec = { NULL };\n \tint err = 0;\n \tstruct indexed_table_ref_iter *itr = NULL;\n \ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex def8883439..aff4fbccda 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -13,3 +13,8 @@ int cmd__reftable(int argc, const char **argv)\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n+\n+int cmd__dump_reftable(int argc, const char **argv)\n+{\n+\treturn reftable_dump_main(argc, (char *const *)argv);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex c7ab5bc4a6..ce27041ab2 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -54,6 +54,7 @@ static struct test_cmd cmds[] = {\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"reftable\", cmd__reftable },\n+\t{ \"dump-reftable\", cmd__dump_reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex fa2b11ace6..8b5790d93f 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -17,6 +17,7 @@ int cmd__dump_cache_tree(int argc, const char **argv);\n int cmd__dump_fsmonitor(int argc, const char **argv);\n int cmd__dump_split_index(int argc, const char **argv);\n int cmd__dump_untracked_cache(int argc, const char **argv);\n+int cmd__dump_reftable(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__genzeros(int argc, const char **argv);\n-- \ngitgitgadget\n"},{"id":"406781","messageId":"4e38db7f48295c4e6d729880f5f1a63b0f90274e.1601568664.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v2.git.git.1601568663.gitgitgadget@gmail.com","subject":"[PATCH v2 12/13] reftable: rest of library","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-01T16:11:02Z","receivedAt":"2020-10-01T16:11:43Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis will be further split up once preceding commits have passed review.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 8 +-\n reftable/VERSION | 1 +\n reftable/dump.c | 212 +++++++\n reftable/merged.c | 358 +++++++++++\n reftable/merged.h | 36 ++\n reftable/merged_test.c | 331 ++++++++++\n reftable/pq.c | 115 ++++\n reftable/pq.h | 32 +\n reftable/refname.c | 209 +++++++\n reftable/refname.h | 28 +\n reftable/refname_test.c | 100 +++\n reftable/reftable.c | 4 +-\n reftable/stack.c | 1240 ++++++++++++++++++++++++++++++++++++++\n reftable/stack.h | 40 ++\n reftable/stack_test.c | 788 ++++++++++++++++++++++++\n reftable/update.sh | 22 +\n t/helper/test-reftable.c | 3 +\n 17 files changed, 3524 insertions(+), 3 deletions(-)\n create mode 100644 reftable/VERSION\n create mode 100644 reftable/dump.c\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n create mode 100755 reftable/update.sh\n\ndiff --git a/Makefile b/Makefile\nindex 17d5fba0a0..05021f79a1 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2375,12 +2375,16 @@ XDIFF_OBJS += xdiff/xutils.o\n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n-REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/compat.o\n REFTABLE_OBJS += reftable/iter.o\n+REFTABLE_OBJS += reftable/merged.o\n+REFTABLE_OBJS += reftable/pq.o\n+REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/refname.o\n REFTABLE_OBJS += reftable/reftable.o\n+REFTABLE_OBJS += reftable/stack.o\n REFTABLE_OBJS += reftable/strbuf.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n@@ -2388,7 +2392,9 @@ REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n+REFTABLE_TEST_OBJS += reftable/refname_test.o\n REFTABLE_TEST_OBJS += reftable/reftable_test.o\n+REFTABLE_TEST_OBJS += reftable/stack_test.o\n REFTABLE_TEST_OBJS += reftable/strbuf_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\ndiff --git a/reftable/VERSION b/reftable/VERSION\nnew file mode 100644\nindex 0000000000..cf8ae4958f\n--- /dev/null\n+++ b/reftable/VERSION\n@@ -0,0 +1 @@\n+7134eb9f8171a9759800f4187f9e6dde997335e7 C: NULL iso 0 for init\ndiff --git a/reftable/dump.c b/reftable/dump.c\nnew file mode 100644\nindex 0000000000..ed09f2e2f9\n--- /dev/null\n+++ b/reftable/dump.c\n@@ -0,0 +1,212 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+\n+#include \"reftable.h\"\n+#include \"reftable-tests.h\"\n+\n+static uint32_t hash_id;\n+\n+static int dump_table(const char *tablename)\n+{\n+\tstruct reftable_block_source src = { NULL };\n+\tint err = reftable_block_source_from_file(&src, tablename);\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_reader *r = NULL;\n+\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\terr = reftable_new_reader(&r, &src, tablename);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\terr = reftable_reader_seek_ref(r, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_ref_record_print(&ref, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_clear(&ref);\n+\n+\terr = reftable_reader_seek_log(r, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_log_record_print(&log, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_clear(&log);\n+\n+\treftable_reader_free(r);\n+\treturn 0;\n+}\n+\n+static int compact_stack(const char *stackdir)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = {};\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_stack_compact_all(stack, NULL);\n+\tif (err < 0)\n+\t\tgoto done;\n+done:\n+\tif (stack != NULL) {\n+\t\treftable_stack_destroy(stack);\n+\t}\n+\treturn err;\n+}\n+\n+static int dump_stack(const char *stackdir)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = {};\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_merged_table *merged = NULL;\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tmerged = reftable_stack_merged_table(stack);\n+\n+\terr = reftable_merged_table_seek_ref(merged, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_ref_record_print(&ref, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_clear(&ref);\n+\n+\terr = reftable_merged_table_seek_log(merged, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_log_record_print(&log, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_clear(&log);\n+\n+\treftable_stack_destroy(stack);\n+\treturn 0;\n+}\n+\n+static void print_help(void)\n+{\n+\tprintf(\"usage: dump [-cst] arg\\n\\n\"\n+\t \"options: \\n\"\n+\t \" -c compact\\n\"\n+\t \" -t dump table\\n\"\n+\t \" -s dump stack\\n\"\n+\t \" -h this help\\n\"\n+\t \" -2 use SHA256\\n\"\n+\t \"\\n\");\n+}\n+\n+int reftable_dump_main(int argc, char *const *argv)\n+{\n+\tint err = 0;\n+\tint opt;\n+\tint opt_dump_table = 0;\n+\tint opt_dump_stack = 0;\n+\tint opt_compact = 0;\n+\tconst char *arg = NULL;\n+\twhile ((opt = getopt(argc, argv, \"2chts\")) != -1) {\n+\t\tswitch (opt) {\n+\t\tcase '2':\n+\t\t\thash_id = 0x73323536;\n+\t\t\tbreak;\n+\t\tcase 't':\n+\t\t\topt_dump_table = 1;\n+\t\t\tbreak;\n+\t\tcase 's':\n+\t\t\topt_dump_stack = 1;\n+\t\t\tbreak;\n+\t\tcase 'c':\n+\t\t\topt_compact = 1;\n+\t\t\tbreak;\n+\t\tcase '?':\n+\t\tcase 'h':\n+\t\t\tprint_help();\n+\t\t\treturn 2;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (argv[optind] == NULL) {\n+\t\tfprintf(stderr, \"need argument\\n\");\n+\t\tprint_help();\n+\t\treturn 2;\n+\t}\n+\n+\targ = argv[optind];\n+\n+\tif (opt_dump_table) {\n+\t\terr = dump_table(arg);\n+\t} else if (opt_dump_stack) {\n+\t\terr = dump_stack(arg);\n+\t} else if (opt_compact) {\n+\t\terr = compact_stack(arg);\n+\t}\n+\n+\tif (err < 0) {\n+\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv[0], arg,\n+\t\t\treftable_error_str(err));\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\ndiff --git a/reftable/merged.c b/reftable/merged.c\nnew file mode 100644\nindex 0000000000..bfa89cfaee\n--- /dev/null\n+++ b/reftable/merged.c\n@@ -0,0 +1,358 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"constants.h\"\n+#include \"iter.h\"\n+#include \"pq.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+#include \"system.h\"\n+\n+static int merged_iter_init(struct merged_iter *mi)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\t\tint err = iterator_next(&mi->stack[i], &rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t\t\treftable_record_destroy(&rec);\n+\t\t} else {\n+\t\t\tstruct pq_entry e = {\n+\t\t\t\t.rec = rec,\n+\t\t\t\t.index = i,\n+\t\t\t};\n+\t\t\tmerged_iter_pqueue_add(&mi->pq, e);\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void merged_iter_close(void *p)\n+{\n+\tstruct merged_iter *mi = (struct merged_iter *)p;\n+\tint i = 0;\n+\tmerged_iter_pqueue_clear(&mi->pq);\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t}\n+\treftable_free(mi->stack);\n+}\n+\n+static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,\n+\t\t\t\t\t size_t idx)\n+{\n+\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\tstruct pq_entry e = {\n+\t\t.rec = rec,\n+\t\t.index = idx,\n+\t};\n+\tint err = iterator_next(&mi->stack[idx], &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (err > 0) {\n+\t\treftable_iterator_destroy(&mi->stack[idx]);\n+\t\treftable_record_destroy(&rec);\n+\t\treturn 0;\n+\t}\n+\n+\tmerged_iter_pqueue_add(&mi->pq, e);\n+\treturn 0;\n+}\n+\n+static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)\n+{\n+\tif (iterator_is_null(&mi->stack[idx]))\n+\t\treturn 0;\n+\treturn merged_iter_advance_nonnull_subiter(mi, idx);\n+}\n+\n+static int merged_iter_next_entry(struct merged_iter *mi,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct strbuf entry_key = STRBUF_INIT;\n+\tstruct pq_entry entry = { 0 };\n+\tint err = 0;\n+\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\tentry = merged_iter_pqueue_remove(&mi->pq);\n+\terr = merged_iter_advance_subiter(mi, entry.index);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\t/*\n+\t One can also use reftable as datacenter-local storage, where the ref\n+\t database is maintained in globally consistent database (eg.\n+\t CockroachDB or Spanner). In this scenario, replication delays together\n+\t with compaction may cause newer tables to contain older entries. In\n+\t such a deployment, the loop below must be changed to collect all\n+\t entries for the same key, and return new the newest one.\n+\t*/\n+\treftable_record_key(&entry.rec, &entry_key);\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq)) {\n+\t\tstruct pq_entry top = merged_iter_pqueue_top(mi->pq);\n+\t\tstruct strbuf k = STRBUF_INIT;\n+\t\tint err = 0, cmp = 0;\n+\n+\t\treftable_record_key(&top.rec, &k);\n+\n+\t\tcmp = strbuf_cmp(&k, &entry_key);\n+\t\tstrbuf_release(&k);\n+\n+\t\tif (cmp > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\t\terr = merged_iter_advance_subiter(mi, top.index);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_record_destroy(&top.rec);\n+\t}\n+\n+\treftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));\n+\treftable_record_destroy(&entry.rec);\n+\tstrbuf_release(&entry_key);\n+\treturn 0;\n+}\n+\n+static int merged_iter_next(struct merged_iter *mi, struct reftable_record *rec)\n+{\n+\twhile (1) {\n+\t\tint err = merged_iter_next_entry(mi, rec);\n+\t\tif (err == 0 && mi->suppress_deletions &&\n+\t\t reftable_record_is_deletion(rec)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\treturn err;\n+\t}\n+}\n+\n+static int merged_iter_next_void(void *p, struct reftable_record *rec)\n+{\n+\tstruct merged_iter *mi = (struct merged_iter *)p;\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\treturn merged_iter_next(mi, rec);\n+}\n+\n+static struct reftable_iterator_vtable merged_iter_vtable = {\n+\t.next = &merged_iter_next_void,\n+\t.close = &merged_iter_close,\n+};\n+\n+static void iterator_from_merged_iter(struct reftable_iterator *it,\n+\t\t\t\t struct merged_iter *mi)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = mi;\n+\tit->ops = &merged_iter_vtable;\n+}\n+\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id)\n+{\n+\tstruct reftable_merged_table *m = NULL;\n+\tuint64_t last_max = 0;\n+\tuint64_t first_min = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t min = reftable_table_min_update_index(&stack[i]);\n+\t\tuint64_t max = reftable_table_max_update_index(&stack[i]);\n+\n+\t\tif (reftable_table_hash_id(&stack[i]) != hash_id) {\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\t\t}\n+\t\tif (i == 0 || min < first_min) {\n+\t\t\tfirst_min = min;\n+\t\t}\n+\t\tif (i == 0 || max > last_max) {\n+\t\t\tlast_max = max;\n+\t\t}\n+\t}\n+\n+\tm = (struct reftable_merged_table *)reftable_calloc(\n+\t\tsizeof(struct reftable_merged_table));\n+\tm->stack = stack;\n+\tm->stack_len = n;\n+\tm->min = first_min;\n+\tm->max = last_max;\n+\tm->hash_id = hash_id;\n+\t*dest = m;\n+\treturn 0;\n+}\n+\n+/* clears the list of subtable, without affecting the readers themselves. */\n+void merged_table_clear(struct reftable_merged_table *mt)\n+{\n+\tFREE_AND_NULL(mt->stack);\n+\tmt->stack_len = 0;\n+}\n+\n+void reftable_merged_table_free(struct reftable_merged_table *mt)\n+{\n+\tif (mt == NULL) {\n+\t\treturn;\n+\t}\n+\tmerged_table_clear(mt);\n+\treftable_free(mt);\n+}\n+\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->max;\n+}\n+\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->min;\n+}\n+\n+static int merged_table_seek_record(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_iterator *iters = reftable_calloc(\n+\t\tsizeof(struct reftable_iterator) * mt->stack_len);\n+\tstruct merged_iter merged = {\n+\t\t.stack = iters,\n+\t\t.typ = reftable_record_type(rec),\n+\t\t.hash_id = mt->hash_id,\n+\t\t.suppress_deletions = mt->suppress_deletions,\n+\t};\n+\tint n = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < mt->stack_len && err == 0; i++) {\n+\t\tint e = reftable_table_seek_record(&mt->stack[i], &iters[n],\n+\t\t\t\t\t\t rec);\n+\t\tif (e < 0) {\n+\t\t\terr = e;\n+\t\t}\n+\t\tif (e == 0) {\n+\t\t\tn++;\n+\t\t}\n+\t}\n+\tif (err < 0) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < n; i++) {\n+\t\t\treftable_iterator_destroy(&iters[i]);\n+\t\t}\n+\t\treftable_free(iters);\n+\t\treturn err;\n+\t}\n+\n+\tmerged.stack_len = n;\n+\terr = merged_iter_init(&merged);\n+\tif (err < 0) {\n+\t\tmerged_iter_close(&merged);\n+\t\treturn err;\n+\t} else {\n+\t\tstruct merged_iter *p =\n+\t\t\treftable_malloc(sizeof(struct merged_iter));\n+\t\t*p = merged;\n+\t\titerator_from_merged_iter(it, p);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_merged_table_seek_log_at(mt, it, name, max);\n+}\n+\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\n+{\n+\treturn mt->hash_id;\n+}\n+\n+static int reftable_merged_table_seek_void(void *tab,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn merged_table_seek_record((struct reftable_merged_table *)tab, it,\n+\t\t\t\t\trec);\n+}\n+\n+static uint32_t reftable_merged_table_hash_id_void(void *tab)\n+{\n+\treturn reftable_merged_table_hash_id(\n+\t\t(struct reftable_merged_table *)tab);\n+}\n+\n+static uint64_t reftable_merged_table_min_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_min_update_index(\n+\t\t(struct reftable_merged_table *)tab);\n+}\n+\n+static uint64_t reftable_merged_table_max_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_max_update_index(\n+\t\t(struct reftable_merged_table *)tab);\n+}\n+\n+static struct reftable_table_vtable merged_table_vtable = {\n+\t.seek_record = reftable_merged_table_seek_void,\n+\t.hash_id = reftable_merged_table_hash_id_void,\n+\t.min_update_index = reftable_merged_table_min_update_index_void,\n+\t.max_update_index = reftable_merged_table_max_update_index_void,\n+};\n+\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *merged)\n+{\n+\tassert(tab->ops == NULL);\n+\ttab->ops = &merged_table_vtable;\n+\ttab->table_arg = merged;\n+}\ndiff --git a/reftable/merged.h b/reftable/merged.h\nnew file mode 100644\nindex 0000000000..76c29928c3\n--- /dev/null\n+++ b/reftable/merged.h\n@@ -0,0 +1,36 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef MERGED_H\n+#define MERGED_H\n+\n+#include \"pq.h\"\n+#include \"reftable.h\"\n+\n+struct reftable_merged_table {\n+\tstruct reftable_table *stack;\n+\tsize_t stack_len;\n+\tuint32_t hash_id;\n+\tint suppress_deletions;\n+\n+\tuint64_t min;\n+\tuint64_t max;\n+};\n+\n+struct merged_iter {\n+\tstruct reftable_iterator *stack;\n+\tuint32_t hash_id;\n+\tsize_t stack_len;\n+\tuint8_t typ;\n+\tint suppress_deletions;\n+\tstruct merged_iter_pqueue pq;\n+};\n+\n+void merged_table_clear(struct reftable_merged_table *mt);\n+\n+#endif\ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nnew file mode 100644\nindex 0000000000..66c65a8ea4\n--- /dev/null\n+++ b/reftable/merged_test.c\n@@ -0,0 +1,331 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"pq.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_pq(void)\n+{\n+\tchar *names[54] = { NULL };\n+\tint N = ARRAY_SIZE(names) - 1;\n+\n+\tstruct merged_iter_pqueue pq = { NULL };\n+\tconst char *last = NULL;\n+\n+\tint i = 0;\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tsnprintf(name, sizeof(name), \"%02d\", i);\n+\t\tnames[i] = xstrdup(name);\n+\t}\n+\n+\ti = 1;\n+\tdo {\n+\t\tstruct reftable_record rec =\n+\t\t\treftable_new_record(BLOCK_TYPE_REF);\n+\t\tstruct pq_entry e = { 0 };\n+\n+\t\treftable_record_as_ref(&rec)->refname = names[i];\n+\t\te.rec = rec;\n+\t\tmerged_iter_pqueue_add(&pq, e);\n+\t\tmerged_iter_pqueue_check(pq);\n+\t\ti = (i * 7) % N;\n+\t} while (i != 1);\n+\n+\twhile (!merged_iter_pqueue_is_empty(pq)) {\n+\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct reftable_ref_record *ref =\n+\t\t\treftable_record_as_ref(&e.rec);\n+\n+\t\tmerged_iter_pqueue_check(pq);\n+\n+\t\tif (last != NULL) {\n+\t\t\tassert(strcmp(last, ref->refname) < 0);\n+\t\t}\n+\t\tlast = ref->refname;\n+\t\tref->refname = NULL;\n+\t\treftable_free(ref);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\n+\tmerged_iter_pqueue_clear(&pq);\n+}\n+\n+static void write_test_table(struct strbuf *buf,\n+\t\t\t struct reftable_ref_record refs[], int n)\n+{\n+\tint min = 0xffffffff;\n+\tint max = 0;\n+\tint i = 0;\n+\tint err;\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_writer *w = NULL;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t ui = refs[i].update_index;\n+\t\tif (ui > max) {\n+\t\t\tmax = ui;\n+\t\t}\n+\t\tif (ui < min) {\n+\t\t\tmin = ui;\n+\t\t}\n+\t}\n+\n+\tw = reftable_new_writer(&strbuf_add_void, buf, &opts);\n+\treftable_writer_set_limits(w, min, max);\n+\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t before = refs[i].update_index;\n+\t\tint n = reftable_writer_add_ref(w, &refs[i]);\n+\t\tassert(n == 0);\n+\t\tassert(before == refs[i].update_index);\n+\t}\n+\n+\terr = reftable_writer_close(w);\n+\tassert_err(err);\n+\n+\treftable_writer_free(w);\n+}\n+\n+static struct reftable_merged_table *\n+merged_table_from_records(struct reftable_ref_record **refs,\n+\t\t\t struct reftable_block_source **source,\n+\t\t\t struct reftable_reader ***readers, int *sizes,\n+\t\t\t struct strbuf *buf, int n)\n+{\n+\tint i = 0;\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err;\n+\tstruct reftable_table *tabs =\n+\t\treftable_calloc(n * sizeof(struct reftable_table));\n+\t*readers = reftable_calloc(n * sizeof(struct reftable_reader *));\n+\t*source = reftable_calloc(n * sizeof(**source));\n+\tfor (i = 0; i < n; i++) {\n+\t\twrite_test_table(&buf[i], refs[i], sizes[i]);\n+\t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n+\n+\t\terr = reftable_new_reader(&(*readers)[i], &(*source)[i],\n+\t\t\t\t\t \"name\");\n+\t\tassert_err(err);\n+\t\treftable_table_from_reader(&tabs[i], (*readers)[i]);\n+\t}\n+\n+\terr = reftable_new_merged_table(&mt, tabs, n, SHA1_ID);\n+\tassert_err(err);\n+\treturn mt;\n+}\n+\n+static void readers_destroy(struct reftable_reader **readers, size_t n)\n+{\n+\tint i = 0;\n+\tfor (; i < n; i++)\n+\t\treftable_reader_free(readers[i]);\n+\treftable_free(readers);\n+}\n+\n+static void test_merged_between(void)\n+{\n+\tuint8_t hash1[SHA1_SIZE] = { 1, 2, 3, 0 };\n+\n+\tstruct reftable_ref_record r1[] = { {\n+\t\t.refname = \"b\",\n+\t\t.update_index = 1,\n+\t\t.value = hash1,\n+\t} };\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t} };\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2 };\n+\tint sizes[] = { 1, 1 };\n+\tstruct strbuf bufs[2] = { STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 2);\n+\tint i;\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tassert_err(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tassert_err(err);\n+\tassert(ref.update_index == 2);\n+\treftable_ref_record_clear(&ref);\n+\treftable_iterator_destroy(&it);\n+\treaders_destroy(readers, 2);\n+\treftable_merged_table_free(mt);\n+\tfor (i = 0; i < ARRAY_SIZE(bufs); i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treftable_free(bs);\n+}\n+\n+static void test_merged(void)\n+{\n+\tuint8_t hash1[SHA1_SIZE] = { 1 };\n+\tuint8_t hash2[SHA1_SIZE] = { 2 };\n+\tstruct reftable_ref_record r1[] = { {\n+\t\t\t\t\t\t .refname = \"a\",\n+\t\t\t\t\t\t .update_index = 1,\n+\t\t\t\t\t\t .value = hash1,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .refname = \"b\",\n+\t\t\t\t\t\t .update_index = 1,\n+\t\t\t\t\t\t .value = hash1,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .refname = \"c\",\n+\t\t\t\t\t\t .update_index = 1,\n+\t\t\t\t\t\t .value = hash1,\n+\t\t\t\t\t } };\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t} };\n+\tstruct reftable_ref_record r3[] = {\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value = hash2,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"d\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value = hash1,\n+\t\t},\n+\t};\n+\n+\tstruct reftable_ref_record want[] = {\n+\t\tr2[0],\n+\t\tr1[1],\n+\t\tr3[0],\n+\t\tr3[1],\n+\t};\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2, r3 };\n+\tint sizes[3] = { 3, 1, 2 };\n+\tstruct strbuf bufs[3] = { STRBUF_INIT, STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 3);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tstruct reftable_ref_record *out = NULL;\n+\tsize_t len = 0;\n+\tsize_t cap = 0;\n+\tint i = 0;\n+\n+\tassert_err(err);\n+\twhile (len < 100) { /* cap loops/recursion. */\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tout = reftable_realloc(\n+\t\t\t\tout, sizeof(struct reftable_ref_record) * cap);\n+\t\t}\n+\t\tout[len++] = ref;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\tassert(ARRAY_SIZE(want) == len);\n+\tfor (i = 0; i < len; i++) {\n+\t\tassert(reftable_ref_record_equal(&want[i], &out[i], SHA1_SIZE));\n+\t}\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_clear(&out[i]);\n+\t}\n+\treftable_free(out);\n+\n+\tfor (i = 0; i < 3; i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treaders_destroy(readers, 3);\n+\treftable_merged_table_free(mt);\n+\treftable_free(bs);\n+}\n+\n+static void test_default_write_opts(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"master\",\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_table *tab = reftable_calloc(sizeof(*tab) * 1);\n+\tuint32_t hash_id;\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_merged_table *merged = NULL;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tassert_err(err);\n+\n+\terr = reftable_writer_close(w);\n+\tassert_err(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tassert_err(err);\n+\n+\thash_id = reftable_reader_hash_id(rd);\n+\tassert(hash_id == SHA1_ID);\n+\n+\treftable_table_from_reader(&tab[0], rd);\n+\terr = reftable_new_merged_table(&merged, tab, 1, SHA1_ID);\n+\tassert_err(err);\n+\n+\treftable_reader_free(rd);\n+\treftable_merged_table_free(merged);\n+\tstrbuf_release(&buf);\n+}\n+\n+/* XXX test refs_for(oid) */\n+\n+int merged_test_main(int argc, const char *argv[])\n+{\n+\tadd_test_case(\"test_merged_between\", &test_merged_between);\n+\tadd_test_case(\"test_pq\", &test_pq);\n+\tadd_test_case(\"test_merged\", &test_merged);\n+\tadd_test_case(\"test_default_write_opts\", &test_default_write_opts);\n+\treturn test_main(argc, argv);\n+}\ndiff --git a/reftable/pq.c b/reftable/pq.c\nnew file mode 100644\nindex 0000000000..81f7745f36\n--- /dev/null\n+++ b/reftable/pq.c\n@@ -0,0 +1,115 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"pq.h\"\n+\n+#include \"reftable.h\"\n+#include \"system.h\"\n+#include \"basics.h\"\n+\n+static int pq_less(struct pq_entry a, struct pq_entry b)\n+{\n+\tstruct strbuf ak = STRBUF_INIT;\n+\tstruct strbuf bk = STRBUF_INIT;\n+\tint cmp = 0;\n+\treftable_record_key(&a.rec, &ak);\n+\treftable_record_key(&b.rec, &bk);\n+\n+\tcmp = strbuf_cmp(&ak, &bk);\n+\n+\tstrbuf_release(&ak);\n+\tstrbuf_release(&bk);\n+\n+\tif (cmp == 0)\n+\t\treturn a.index > b.index;\n+\n+\treturn cmp < 0;\n+}\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.heap[0];\n+}\n+\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.len == 0;\n+}\n+\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq)\n+{\n+\tint i = 0;\n+\tfor (i = 1; i < pq.len; i++) {\n+\t\tint parent = (i - 1) / 2;\n+\n+\t\tassert(pq_less(pq.heap[parent], pq.heap[i]));\n+\t}\n+}\n+\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tstruct pq_entry e = pq->heap[0];\n+\tpq->heap[0] = pq->heap[pq->len - 1];\n+\tpq->len--;\n+\n+\ti = 0;\n+\twhile (i < pq->len) {\n+\t\tint min = i;\n+\t\tint j = 2 * i + 1;\n+\t\tint k = 2 * i + 2;\n+\t\tif (j < pq->len && pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tmin = j;\n+\t\t}\n+\t\tif (k < pq->len && pq_less(pq->heap[k], pq->heap[min])) {\n+\t\t\tmin = k;\n+\t\t}\n+\n+\t\tif (min == i) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[i], pq->heap[min]);\n+\t\ti = min;\n+\t}\n+\n+\treturn e;\n+}\n+\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)\n+{\n+\tint i = 0;\n+\tif (pq->len == pq->cap) {\n+\t\tpq->cap = 2 * pq->cap + 1;\n+\t\tpq->heap = reftable_realloc(pq->heap,\n+\t\t\t\t\t pq->cap * sizeof(struct pq_entry));\n+\t}\n+\n+\tpq->heap[pq->len++] = e;\n+\ti = pq->len - 1;\n+\twhile (i > 0) {\n+\t\tint j = (i - 1) / 2;\n+\t\tif (pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[j], pq->heap[i]);\n+\n+\t\ti = j;\n+\t}\n+}\n+\n+void merged_iter_pqueue_clear(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < pq->len; i++) {\n+\t\treftable_record_destroy(&pq->heap[i].rec);\n+\t}\n+\tFREE_AND_NULL(pq->heap);\n+\tpq->len = pq->cap = 0;\n+}\ndiff --git a/reftable/pq.h b/reftable/pq.h\nnew file mode 100644\nindex 0000000000..9c2c0e0bfe\n--- /dev/null\n+++ b/reftable/pq.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef PQ_H\n+#define PQ_H\n+\n+#include \"record.h\"\n+\n+struct pq_entry {\n+\tint index;\n+\tstruct reftable_record rec;\n+};\n+\n+struct merged_iter_pqueue {\n+\tstruct pq_entry *heap;\n+\tsize_t len;\n+\tsize_t cap;\n+};\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq);\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq);\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq);\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e);\n+void merged_iter_pqueue_clear(struct merged_iter_pqueue *pq);\n+\n+#endif\ndiff --git a/reftable/refname.c b/reftable/refname.c\nnew file mode 100644\nindex 0000000000..9051c1f574\n--- /dev/null\n+++ b/reftable/refname.c\n@@ -0,0 +1,209 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"reftable.h\"\n+#include \"basics.h\"\n+#include \"refname.h\"\n+#include \"strbuf.h\"\n+\n+struct find_arg {\n+\tchar **names;\n+\tconst char *want;\n+};\n+\n+static int find_name(size_t k, void *arg)\n+{\n+\tstruct find_arg *f_arg = (struct find_arg *)arg;\n+\treturn strcmp(f_arg->names[k], f_arg->want) >= 0;\n+}\n+\n+static int modification_has_ref(struct modification *mod, const char *name)\n+{\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len && !strcmp(mod->add[idx], name)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tif (mod->del_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->del,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\tif (idx < mod->del_len && !strcmp(mod->del[idx], name)) {\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\terr = reftable_table_read_ref(&mod->tab, name, &ref);\n+\treftable_ref_record_clear(&ref);\n+\treturn err;\n+}\n+\n+static void modification_clear(struct modification *mod)\n+{\n+\t/* don't delete the strings themselves; they're owned by ref records.\n+\t */\n+\tFREE_AND_NULL(mod->add);\n+\tFREE_AND_NULL(mod->del);\n+\tmod->add_len = 0;\n+\tmod->del_len = 0;\n+}\n+\n+static int modification_has_ref_with_prefix(struct modification *mod,\n+\t\t\t\t\t const char *prefix)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = prefix,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len &&\n+\t\t !strncmp(prefix, mod->add[idx], strlen(prefix)))\n+\t\t\tgoto done;\n+\t}\n+\terr = reftable_table_seek_ref(&mod->tab, &it, prefix);\n+\tif (err)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\n+\t\tif (mod->del_len > 0) {\n+\t\t\tstruct find_arg arg = {\n+\t\t\t\t.names = mod->del,\n+\t\t\t\t.want = ref.refname,\n+\t\t\t};\n+\t\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\t\tif (idx < mod->del_len &&\n+\t\t\t !strcmp(ref.refname, mod->del[idx])) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (strncmp(ref.refname, prefix, strlen(prefix))) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_ref_record_clear(&ref);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int validate_refname(const char *name)\n+{\n+\twhile (1) {\n+\t\tchar *next = strchr(name, '/');\n+\t\tif (!*name) {\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\t}\n+\t\tif (!next) {\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (next - name == 0 || (next - name == 1 && *name == '.') ||\n+\t\t (next - name == 2 && name[0] == '.' && name[1] == '.'))\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\tname = next + 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz)\n+{\n+\tstruct modification mod = {\n+\t\t.tab = tab,\n+\t\t.add = reftable_calloc(sizeof(char *) * sz),\n+\t\t.del = reftable_calloc(sizeof(char *) * sz),\n+\t};\n+\tint i = 0;\n+\tint err = 0;\n+\tfor (; i < sz; i++) {\n+\t\tif (reftable_ref_record_is_deletion(&recs[i])) {\n+\t\t\tmod.del[mod.del_len++] = recs[i].refname;\n+\t\t} else {\n+\t\t\tmod.add[mod.add_len++] = recs[i].refname;\n+\t\t}\n+\t}\n+\n+\terr = modification_validate(&mod);\n+\tmodification_clear(&mod);\n+\treturn err;\n+}\n+\n+static void strbuf_trim_component(struct strbuf *sl)\n+{\n+\twhile (sl->len > 0) {\n+\t\tint is_slash = (sl->buf[sl->len - 1] == '/');\n+\t\tstrbuf_setlen(sl, sl->len - 1);\n+\t\tif (is_slash)\n+\t\t\tbreak;\n+\t}\n+}\n+\n+int modification_validate(struct modification *mod)\n+{\n+\tstruct strbuf slashed = STRBUF_INIT;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (; i < mod->add_len; i++) {\n+\t\terr = validate_refname(mod->add[i]);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\tstrbuf_addstr(&slashed, \"/\");\n+\n+\t\terr = modification_has_ref_with_prefix(mod, slashed.buf);\n+\t\tif (err == 0) {\n+\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\twhile (slashed.len) {\n+\t\t\tstrbuf_trim_component(&slashed);\n+\t\t\terr = modification_has_ref(mod, slashed.buf);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\terr = 0;\n+done:\n+\tstrbuf_release(&slashed);\n+\treturn err;\n+}\ndiff --git a/reftable/refname.h b/reftable/refname.h\nnew file mode 100644\nindex 0000000000..fcd645a99f\n--- /dev/null\n+++ b/reftable/refname.h\n@@ -0,0 +1,28 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+#ifndef REFNAME_H\n+#define REFNAME_H\n+\n+#include \"reftable.h\"\n+\n+struct modification {\n+\tstruct reftable_table tab;\n+\n+\tchar **add;\n+\tsize_t add_len;\n+\n+\tchar **del;\n+\tsize_t del_len;\n+};\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz);\n+\n+int modification_validate(struct modification *mod);\n+\n+#endif\ndiff --git a/reftable/refname_test.c b/reftable/refname_test.c\nnew file mode 100644\nindex 0000000000..6e9a15da38\n--- /dev/null\n+++ b/reftable/refname_test.c\n@@ -0,0 +1,100 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable.h\"\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"refname.h\"\n+#include \"system.h\"\n+\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct testcase {\n+\tchar *add;\n+\tchar *del;\n+\tint error_code;\n+};\n+\n+static void test_conflict(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"a/b\",\n+\t\t.target = \"destination\", /* make sure it's not a symref. */\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tint i;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct testcase cases[] = {\n+\t\t{ \"a/b/c\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"b\", NULL, 0 },\n+\t\t{ \"a\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"a\", \"a/b\", 0 },\n+\n+\t\t{ \"p/\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p//q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/./q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/../q\", NULL, REFTABLE_REFNAME_ERROR },\n+\n+\t\t{ \"a/b/c\", \"a/b\", 0 },\n+\t\t{ NULL, \"a//b\", 0 },\n+\t};\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tassert_err(err);\n+\n+\terr = reftable_writer_close(w);\n+\tassert_err(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tassert_err(err);\n+\n+\treftable_table_from_reader(&tab, rd);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct modification mod = {\n+\t\t\t.tab = tab,\n+\t\t};\n+\n+\t\tif (cases[i].add != NULL) {\n+\t\t\tmod.add = &cases[i].add;\n+\t\t\tmod.add_len = 1;\n+\t\t}\n+\t\tif (cases[i].del != NULL) {\n+\t\t\tmod.del = &cases[i].del;\n+\t\t\tmod.del_len = 1;\n+\t\t}\n+\n+\t\terr = modification_validate(&mod);\n+\t\tassert(err == cases[i].error_code);\n+\t}\n+\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+int refname_test_main(int argc, const char *argv[])\n+{\n+\tadd_test_case(\"test_conflict\", &test_conflict);\n+\treturn test_main(argc, argv);\n+}\ndiff --git a/reftable/reftable.c b/reftable/reftable.c\nindex 1425aef6f2..0f3bf0291e 100644\n--- a/reftable/reftable.c\n+++ b/reftable/reftable.c\n@@ -44,7 +44,7 @@ int reftable_table_seek_ref(struct reftable_table *tab,\n \tstruct reftable_ref_record ref = {\n \t\t.refname = (char *)name,\n \t};\n-\tstruct reftable_record rec = { 0 };\n+\tstruct reftable_record rec = { NULL };\n \treftable_record_from_ref(&rec, &ref);\n \treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n }\n@@ -60,7 +60,7 @@ void reftable_table_from_reader(struct reftable_table *tab,\n int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n \t\t\t struct reftable_ref_record *ref)\n {\n-\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_iterator it = { NULL };\n \tint err = reftable_table_seek_ref(tab, &it, name);\n \tif (err)\n \t\tgoto done;\ndiff --git a/reftable/stack.c b/reftable/stack.c\nnew file mode 100644\nindex 0000000000..b20bee407b\n--- /dev/null\n+++ b/reftable/stack.c\n@@ -0,0 +1,1240 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+#include \"merged.h\"\n+#include \"reader.h\"\n+#include \"refname.h\"\n+#include \"reftable.h\"\n+#include \"writer.h\"\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config);\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name);\n+static void reftable_addition_close(struct reftable_addition *add);\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open);\n+\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config)\n+{\n+\tstruct reftable_stack *p =\n+\t\treftable_calloc(sizeof(struct reftable_stack));\n+\tstruct strbuf list_file_name = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tif (config.hash_id == 0) {\n+\t\tconfig.hash_id = SHA1_ID;\n+\t}\n+\n+\t*dest = NULL;\n+\n+\tstrbuf_reset(&list_file_name);\n+\tstrbuf_addstr(&list_file_name, dir);\n+\tstrbuf_addstr(&list_file_name, \"/tables.list\");\n+\n+\tp->list_file = strbuf_detach(&list_file_name, NULL);\n+\tp->reftable_dir = xstrdup(dir);\n+\tp->config = config;\n+\n+\terr = reftable_stack_reload_maybe_reuse(p, 1);\n+\tif (err < 0) {\n+\t\treftable_stack_destroy(p);\n+\t} else {\n+\t\t*dest = p;\n+\t}\n+\treturn err;\n+}\n+\n+static int fd_read_lines(int fd, char ***namesp)\n+{\n+\toff_t size = lseek(fd, 0, SEEK_END);\n+\tchar *buf = NULL;\n+\tint err = 0;\n+\tif (size < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\terr = lseek(fd, 0, SEEK_SET);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tbuf = reftable_malloc(size + 1);\n+\tif (read(fd, buf, size) != size) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\tbuf[size] = 0;\n+\n+\tparse_names(buf, size, namesp);\n+\n+done:\n+\treftable_free(buf);\n+\treturn err;\n+}\n+\n+int read_lines(const char *filename, char ***namesp)\n+{\n+\tint fd = open(filename, O_RDONLY, 0644);\n+\tint err = 0;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\t*namesp = reftable_calloc(sizeof(char *));\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\treturn REFTABLE_IO_ERROR;\n+\t}\n+\terr = fd_read_lines(fd, namesp);\n+\tclose(fd);\n+\treturn err;\n+}\n+\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st)\n+{\n+\treturn st->merged;\n+}\n+\n+/* Close and free the stack */\n+void reftable_stack_destroy(struct reftable_stack *st)\n+{\n+\tif (st->merged != NULL) {\n+\t\treftable_merged_table_free(st->merged);\n+\t\tst->merged = NULL;\n+\t}\n+\n+\tif (st->readers != NULL) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < st->readers_len; i++) {\n+\t\t\treftable_reader_free(st->readers[i]);\n+\t\t}\n+\t\tst->readers_len = 0;\n+\t\tFREE_AND_NULL(st->readers);\n+\t}\n+\tFREE_AND_NULL(st->list_file);\n+\tFREE_AND_NULL(st->reftable_dir);\n+\treftable_free(st);\n+}\n+\n+static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n+\t\t\t\t\t\t int cur_len)\n+{\n+\tstruct reftable_reader **cur =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * cur_len);\n+\tint i = 0;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tcur[i] = st->readers[i];\n+\t}\n+\treturn cur;\n+}\n+\n+static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n+\t\t\t\t int reuse_open)\n+{\n+\tint cur_len = st->merged == NULL ? 0 : st->merged->stack_len;\n+\tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n+\tint err = 0;\n+\tint names_len = names_length(names);\n+\tstruct reftable_reader **new_readers =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * names_len);\n+\tstruct reftable_table *new_tables =\n+\t\treftable_calloc(sizeof(struct reftable_table) * names_len);\n+\tint new_readers_len = 0;\n+\tstruct reftable_merged_table *new_merged = NULL;\n+\tint i;\n+\n+\twhile (*names) {\n+\t\tstruct reftable_reader *rd = NULL;\n+\t\tchar *name = *names++;\n+\n+\t\t/* this is linear; we assume compaction keeps the number of\n+\t\t tables under control so this is not quadratic. */\n+\t\tint j = 0;\n+\t\tfor (j = 0; reuse_open && j < cur_len; j++) {\n+\t\t\tif (cur[j] != NULL && 0 == strcmp(cur[j]->name, name)) {\n+\t\t\t\trd = cur[j];\n+\t\t\t\tcur[j] = NULL;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (rd == NULL) {\n+\t\t\tstruct reftable_block_source src = { NULL };\n+\t\t\tstruct strbuf table_path = STRBUF_INIT;\n+\t\t\tstrbuf_addstr(&table_path, st->reftable_dir);\n+\t\t\tstrbuf_addstr(&table_path, \"/\");\n+\t\t\tstrbuf_addstr(&table_path, name);\n+\n+\t\t\terr = reftable_block_source_from_file(&src,\n+\t\t\t\t\t\t\t table_path.buf);\n+\t\t\tstrbuf_release(&table_path);\n+\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\n+\t\t\terr = reftable_new_reader(&rd, &src, name);\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\n+\t\tnew_readers[new_readers_len] = rd;\n+\t\treftable_table_from_reader(&new_tables[new_readers_len], rd);\n+\t\tnew_readers_len++;\n+\t}\n+\n+\t/* success! */\n+\terr = reftable_new_merged_table(&new_merged, new_tables,\n+\t\t\t\t\tnew_readers_len, st->config.hash_id);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tnew_tables = NULL;\n+\tst->readers_len = new_readers_len;\n+\tif (st->merged != NULL) {\n+\t\tmerged_table_clear(st->merged);\n+\t\treftable_merged_table_free(st->merged);\n+\t}\n+\tif (st->readers != NULL) {\n+\t\treftable_free(st->readers);\n+\t}\n+\tst->readers = new_readers;\n+\tnew_readers = NULL;\n+\tnew_readers_len = 0;\n+\n+\tnew_merged->suppress_deletions = 1;\n+\tst->merged = new_merged;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tif (cur[i] != NULL) {\n+\t\t\treader_close(cur[i]);\n+\t\t\treftable_reader_free(cur[i]);\n+\t\t}\n+\t}\n+\n+done:\n+\tfor (i = 0; i < new_readers_len; i++) {\n+\t\treader_close(new_readers[i]);\n+\t\treftable_reader_free(new_readers[i]);\n+\t}\n+\treftable_free(new_readers);\n+\treftable_free(new_tables);\n+\treftable_free(cur);\n+\treturn err;\n+}\n+\n+/* return negative if a before b. */\n+static int tv_cmp(struct timeval *a, struct timeval *b)\n+{\n+\ttime_t diff = a->tv_sec - b->tv_sec;\n+\tint udiff = a->tv_usec - b->tv_usec;\n+\n+\tif (diff != 0)\n+\t\treturn diff;\n+\n+\treturn udiff;\n+}\n+\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open)\n+{\n+\tstruct timeval deadline = { 0 };\n+\tint err = gettimeofday(&deadline, NULL);\n+\tint64_t delay = 0;\n+\tint tries = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tdeadline.tv_sec += 3;\n+\twhile (1) {\n+\t\tchar **names = NULL;\n+\t\tchar **names_after = NULL;\n+\t\tstruct timeval now = { 0 };\n+\t\tint err = gettimeofday(&now, NULL);\n+\t\tint err2 = 0;\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* Only look at deadlines after the first few times. This\n+\t\t simplifies debugging in GDB */\n+\t\ttries++;\n+\t\tif (tries > 3 && tv_cmp(&now, &deadline) >= 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = read_lines(st->list_file, &names);\n+\t\tif (err < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\t\terr = reftable_stack_reload_once(st, names, reuse_open);\n+\t\tif (err == 0) {\n+\t\t\tfree_names(names);\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err != REFTABLE_NOT_EXIST_ERROR) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* err == REFTABLE_NOT_EXIST_ERROR can be caused by a concurrent\n+\t\t writer. Check if there was one by checking if the name list\n+\t\t changed.\n+\t\t*/\n+\t\terr2 = read_lines(st->list_file, &names_after);\n+\t\tif (err2 < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err2;\n+\t\t}\n+\n+\t\tif (names_equal(names_after, names)) {\n+\t\t\tfree_names(names);\n+\t\t\tfree_names(names_after);\n+\t\t\treturn err;\n+\t\t}\n+\t\tfree_names(names);\n+\t\tfree_names(names_after);\n+\n+\t\tdelay = delay + (delay * rand()) / RAND_MAX + 1;\n+\t\tsleep_millisec(delay);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/* -1 = error\n+ 0 = up to date\n+ 1 = changed. */\n+static int stack_uptodate(struct reftable_stack *st)\n+{\n+\tchar **names = NULL;\n+\tint err = read_lines(st->list_file, &names);\n+\tint i = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tfor (i = 0; i < st->readers_len; i++) {\n+\t\tif (names[i] == NULL) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (strcmp(st->readers[i]->name, names[i])) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (names[st->merged->stack_len] != NULL) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tfree_names(names);\n+\treturn err;\n+}\n+\n+int reftable_stack_reload(struct reftable_stack *st)\n+{\n+\tint err = stack_uptodate(st);\n+\tif (err > 0)\n+\t\treturn reftable_stack_reload_maybe_reuse(st, 1);\n+\treturn err;\n+}\n+\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write)(struct reftable_writer *wr, void *arg),\n+\t\t void *arg)\n+{\n+\tint err = stack_try_add(st, write, arg);\n+\tif (err < 0) {\n+\t\tif (err == REFTABLE_LOCK_ERROR) {\n+\t\t\t/* Ignore error return, we want to propagate\n+\t\t\t REFTABLE_LOCK_ERROR.\n+\t\t\t*/\n+\t\t\treftable_stack_reload(st);\n+\t\t}\n+\t\treturn err;\n+\t}\n+\n+\tif (!st->disable_auto_compact)\n+\t\treturn reftable_stack_auto_compact(st);\n+\n+\treturn 0;\n+}\n+\n+static void format_name(struct strbuf *dest, uint64_t min, uint64_t max)\n+{\n+\tchar buf[100];\n+\tsnprintf(buf, sizeof(buf), \"0x%012\" PRIx64 \"-0x%012\" PRIx64, min, max);\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, buf);\n+}\n+\n+struct reftable_addition {\n+\tint lock_file_fd;\n+\tstruct strbuf lock_file_name;\n+\tstruct reftable_stack *stack;\n+\tchar **names;\n+\tchar **new_tables;\n+\tint new_tables_len;\n+\tuint64_t next_update_index;\n+};\n+\n+#define REFTABLE_ADDITION_INIT \\\n+\t{ \\\n+\t\t.lock_file_name = STRBUF_INIT \\\n+\t}\n+\n+static int reftable_stack_init_addition(struct reftable_addition *add,\n+\t\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tadd->stack = st;\n+\n+\tstrbuf_reset(&add->lock_file_name);\n+\tstrbuf_addstr(&add->lock_file_name, st->list_file);\n+\tstrbuf_addstr(&add->lock_file_name, \".lock\");\n+\n+\tadd->lock_file_fd = open(add->lock_file_name.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (add->lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = REFTABLE_LOCK_ERROR;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\terr = stack_uptodate(st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 1) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->next_update_index = reftable_stack_next_update_index(st);\n+done:\n+\tif (err) {\n+\t\treftable_addition_close(add);\n+\t}\n+\treturn err;\n+}\n+\n+static void reftable_addition_close(struct reftable_addition *add)\n+{\n+\tint i = 0;\n+\tstruct strbuf nm = STRBUF_INIT;\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstrbuf_reset(&nm);\n+\t\tstrbuf_addstr(&nm, add->stack->list_file);\n+\t\tstrbuf_addstr(&nm, \"/\");\n+\t\tstrbuf_addstr(&nm, add->new_tables[i]);\n+\t\tunlink(nm.buf);\n+\t\treftable_free(add->new_tables[i]);\n+\t\tadd->new_tables[i] = NULL;\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\tif (add->lock_file_fd > 0) {\n+\t\tclose(add->lock_file_fd);\n+\t\tadd->lock_file_fd = 0;\n+\t}\n+\tif (add->lock_file_name.len > 0) {\n+\t\tunlink(add->lock_file_name.buf);\n+\t\tstrbuf_release(&add->lock_file_name);\n+\t}\n+\n+\tfree_names(add->names);\n+\tadd->names = NULL;\n+\tstrbuf_release(&nm);\n+}\n+\n+void reftable_addition_destroy(struct reftable_addition *add)\n+{\n+\tif (add == NULL) {\n+\t\treturn;\n+\t}\n+\treftable_addition_close(add);\n+\treftable_free(add);\n+}\n+\n+int reftable_addition_commit(struct reftable_addition *add)\n+{\n+\tstruct strbuf table_list = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\tif (add->new_tables_len == 0)\n+\t\tgoto done;\n+\n+\tfor (i = 0; i < add->stack->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->stack->readers[i]->name);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->new_tables[i]);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\n+\terr = write(add->lock_file_fd, table_list.buf, table_list.len);\n+\tstrbuf_release(&table_list);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = close(add->lock_file_fd);\n+\tadd->lock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(add->lock_file_name.buf, add->stack->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(add->stack);\n+\n+done:\n+\treftable_addition_close(add);\n+\treturn err;\n+}\n+\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n+\t*dest = reftable_calloc(sizeof(**dest));\n+\t**dest = empty;\n+\terr = reftable_stack_init_addition(*dest, st);\n+\tif (err) {\n+\t\treftable_free(*dest);\n+\t\t*dest = NULL;\n+\t}\n+\treturn err;\n+}\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct reftable_addition add = REFTABLE_ADDITION_INIT;\n+\tint err = reftable_stack_init_addition(&add, st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tif (err > 0) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_addition_add(&add, write_table, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_addition_commit(&add);\n+done:\n+\treftable_addition_close(&add);\n+\treturn err;\n+}\n+\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf tab_file_name = STRBUF_INIT;\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\tint tab_fd = 0;\n+\n+\tstrbuf_reset(&next_name);\n+\tformat_name(&next_name, add->next_update_index, add->next_update_index);\n+\n+\tstrbuf_addstr(&temp_tab_file_name, add->stack->reftable_dir);\n+\tstrbuf_addstr(&temp_tab_file_name, \"/\");\n+\tstrbuf_addbuf(&temp_tab_file_name, &next_name);\n+\tstrbuf_addstr(&temp_tab_file_name, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab_file_name.buf);\n+\tif (tab_fd < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd,\n+\t\t\t\t &add->stack->config);\n+\terr = write_table(wr, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_writer_close(wr);\n+\tif (err == REFTABLE_EMPTY_TABLE_ERROR) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = stack_check_addition(add->stack, temp_tab_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (wr->min_update_index < add->next_update_index) {\n+\t\terr = REFTABLE_API_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tformat_name(&next_name, wr->min_update_index, wr->max_update_index);\n+\tstrbuf_addstr(&next_name, \".ref\");\n+\n+\tstrbuf_addstr(&tab_file_name, add->stack->reftable_dir);\n+\tstrbuf_addstr(&tab_file_name, \"/\");\n+\tstrbuf_addbuf(&tab_file_name, &next_name);\n+\n+\t/* TODO: should check destination out of paranoia */\n+\terr = rename(temp_tab_file_name.buf, tab_file_name.buf);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->new_tables = reftable_realloc(add->new_tables,\n+\t\t\t\t\t sizeof(*add->new_tables) *\n+\t\t\t\t\t\t (add->new_tables_len + 1));\n+\tadd->new_tables[add->new_tables_len] = strbuf_detach(&next_name, NULL);\n+\tadd->new_tables_len++;\n+done:\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (temp_tab_file_name.len > 0) {\n+\t\tunlink(temp_tab_file_name.buf);\n+\t}\n+\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&tab_file_name);\n+\tstrbuf_release(&next_name);\n+\treftable_writer_free(wr);\n+\treturn err;\n+}\n+\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st)\n+{\n+\tint sz = st->merged->stack_len;\n+\tif (sz > 0)\n+\t\treturn reftable_reader_max_update_index(st->readers[sz - 1]) +\n+\t\t 1;\n+\treturn 1;\n+}\n+\n+static int stack_compact_locked(struct reftable_stack *st, int first, int last,\n+\t\t\t\tstruct strbuf *temp_tab,\n+\t\t\t\tstruct reftable_log_expiry_config *config)\n+{\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tint tab_fd = -1;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\n+\tformat_name(&next_name,\n+\t\t reftable_reader_min_update_index(st->readers[first]),\n+\t\t reftable_reader_max_update_index(st->readers[first]));\n+\n+\tstrbuf_reset(temp_tab);\n+\tstrbuf_addstr(temp_tab, st->reftable_dir);\n+\tstrbuf_addstr(temp_tab, \"/\");\n+\tstrbuf_addbuf(temp_tab, &next_name);\n+\tstrbuf_addstr(temp_tab, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab->buf);\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd, &st->config);\n+\n+\terr = stack_write_compact(st, wr, first, last, config);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = reftable_writer_close(wr);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\n+done:\n+\treftable_writer_free(wr);\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (err != 0 && temp_tab->len > 0) {\n+\t\tunlink(temp_tab->buf);\n+\t\tstrbuf_release(temp_tab);\n+\t}\n+\tstrbuf_release(&next_name);\n+\treturn err;\n+}\n+\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint subtabs_len = last - first + 1;\n+\tstruct reftable_table *subtabs = reftable_calloc(\n+\t\tsizeof(struct reftable_table) * (last - first + 1));\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tuint64_t entries = 0;\n+\n+\tint i = 0, j = 0;\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct reftable_reader *t = st->readers[i];\n+\t\treftable_table_from_reader(&subtabs[j++], t);\n+\t\tst->stats.bytes += t->size;\n+\t}\n+\treftable_writer_set_limits(wr, st->readers[first]->min_update_index,\n+\t\t\t\t st->readers[last]->max_update_index);\n+\n+\terr = reftable_new_merged_table(&mt, subtabs, subtabs_len,\n+\t\t\t\t\tst->config.hash_id);\n+\tif (err < 0) {\n+\t\treftable_free(subtabs);\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_merged_table_seek_ref(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (first == 0 && reftable_ref_record_is_deletion(&ref)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_ref(wr, &ref);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_merged_table_seek_log(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (first == 0 && reftable_log_record_is_deletion(&log)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config != NULL && config->time > 0 &&\n+\t\t log.time < config->time) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config != NULL && config->min_update_index > 0 &&\n+\t\t log.update_index < config->min_update_index) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_log(wr, &log);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\tif (mt != NULL) {\n+\t\tmerged_table_clear(mt);\n+\t\treftable_merged_table_free(mt);\n+\t}\n+\treftable_ref_record_clear(&ref);\n+\treftable_log_record_clear(&log);\n+\tst->stats.entries_written += entries;\n+\treturn err;\n+}\n+\n+/* < 0: error. 0 == OK, > 0 attempt failed; could retry. */\n+static int stack_compact_range(struct reftable_stack *st, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *expiry)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf new_table_name = STRBUF_INIT;\n+\tstruct strbuf lock_file_name = STRBUF_INIT;\n+\tstruct strbuf ref_list_contents = STRBUF_INIT;\n+\tstruct strbuf new_table_path = STRBUF_INIT;\n+\tint err = 0;\n+\tint have_lock = 0;\n+\tint lock_file_fd = 0;\n+\tint compact_count = last - first + 1;\n+\tchar **listp = NULL;\n+\tchar **delete_on_success =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tchar **subtable_locks =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tint i = 0;\n+\tint j = 0;\n+\tint is_empty_table = 0;\n+\n+\tif (first > last || (expiry == NULL && first == last)) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\tst->stats.attempts++;\n+\n+\tstrbuf_reset(&lock_file_name);\n+\tstrbuf_addstr(&lock_file_name, st->list_file);\n+\tstrbuf_addstr(&lock_file_name, \".lock\");\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\t/* Don't want to write to the lock for now. */\n+\tclose(lock_file_fd);\n+\tlock_file_fd = 0;\n+\n+\thave_lock = 1;\n+\terr = stack_uptodate(st);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct strbuf subtab_file_name = STRBUF_INIT;\n+\t\tstruct strbuf subtab_lock = STRBUF_INIT;\n+\t\tint sublock_file_fd = -1;\n+\n+\t\tstrbuf_addstr(&subtab_file_name, st->reftable_dir);\n+\t\tstrbuf_addstr(&subtab_file_name, \"/\");\n+\t\tstrbuf_addstr(&subtab_file_name, reader_name(st->readers[i]));\n+\n+\t\tstrbuf_reset(&subtab_lock);\n+\t\tstrbuf_addbuf(&subtab_lock, &subtab_file_name);\n+\t\tstrbuf_addstr(&subtab_lock, \".lock\");\n+\n+\t\tsublock_file_fd = open(subtab_lock.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\t\tif (sublock_file_fd > 0) {\n+\t\t\tclose(sublock_file_fd);\n+\t\t} else if (sublock_file_fd < 0) {\n+\t\t\tif (errno == EEXIST) {\n+\t\t\t\terr = 1;\n+\t\t\t} else {\n+\t\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\t}\n+\t\t}\n+\n+\t\tsubtable_locks[j] = subtab_lock.buf;\n+\t\tdelete_on_success[j] = subtab_file_name.buf;\n+\t\tj++;\n+\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\terr = unlink(lock_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\thave_lock = 0;\n+\n+\terr = stack_compact_locked(st, first, last, &temp_tab_file_name,\n+\t\t\t\t expiry);\n+\t/* Compaction + tombstones can create an empty table out of non-empty\n+\t * tables. */\n+\tis_empty_table = (err == REFTABLE_EMPTY_TABLE_ERROR);\n+\tif (is_empty_table) {\n+\t\terr = 0;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\thave_lock = 1;\n+\n+\tformat_name(&new_table_name, st->readers[first]->min_update_index,\n+\t\t st->readers[last]->max_update_index);\n+\tstrbuf_addstr(&new_table_name, \".ref\");\n+\n+\tstrbuf_reset(&new_table_path);\n+\tstrbuf_addstr(&new_table_path, st->reftable_dir);\n+\tstrbuf_addstr(&new_table_path, \"/\");\n+\tstrbuf_addbuf(&new_table_path, &new_table_name);\n+\n+\tif (!is_empty_table) {\n+\t\terr = rename(temp_tab_file_name.buf, new_table_path.buf);\n+\t\tif (err < 0) {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < first; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tif (!is_empty_table) {\n+\t\tstrbuf_addbuf(&ref_list_contents, &new_table_name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tfor (i = last + 1; i < st->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\n+\terr = write(lock_file_fd, ref_list_contents.buf, ref_list_contents.len);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\terr = close(lock_file_fd);\n+\tlock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(lock_file_name.buf, st->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\thave_lock = 0;\n+\n+\t/* Reload the stack before deleting. On windows, we can only delete the\n+\t files after we closed them.\n+\t*/\n+\terr = reftable_stack_reload_maybe_reuse(st, first < last);\n+\n+\tlistp = delete_on_success;\n+\twhile (*listp) {\n+\t\tif (strcmp(*listp, new_table_path.buf)) {\n+\t\t\tunlink(*listp);\n+\t\t}\n+\t\tlistp++;\n+\t}\n+\n+done:\n+\tfree_names(delete_on_success);\n+\n+\tlistp = subtable_locks;\n+\twhile (*listp) {\n+\t\tunlink(*listp);\n+\t\tlistp++;\n+\t}\n+\tfree_names(subtable_locks);\n+\tif (lock_file_fd > 0) {\n+\t\tclose(lock_file_fd);\n+\t\tlock_file_fd = 0;\n+\t}\n+\tif (have_lock) {\n+\t\tunlink(lock_file_name.buf);\n+\t}\n+\tstrbuf_release(&new_table_name);\n+\tstrbuf_release(&new_table_path);\n+\tstrbuf_release(&ref_list_contents);\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&lock_file_name);\n+\treturn err;\n+}\n+\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\treturn stack_compact_range(st, 0, st->merged->stack_len - 1, config);\n+}\n+\n+static int stack_compact_range_stats(struct reftable_stack *st, int first,\n+\t\t\t\t int last,\n+\t\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint err = stack_compact_range(st, first, last, config);\n+\tif (err > 0) {\n+\t\tst->stats.failures++;\n+\t}\n+\treturn err;\n+}\n+\n+static int segment_size(struct segment *s)\n+{\n+\treturn s->end - s->start;\n+}\n+\n+int fastlog2(uint64_t sz)\n+{\n+\tint l = 0;\n+\tif (sz == 0)\n+\t\treturn 0;\n+\tfor (; sz; sz /= 2) {\n+\t\tl++;\n+\t}\n+\treturn l - 1;\n+}\n+\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n+{\n+\tstruct segment *segs = reftable_calloc(sizeof(struct segment) * n);\n+\tint next = 0;\n+\tstruct segment cur = { 0 };\n+\tint i = 0;\n+\n+\tif (n == 0) {\n+\t\t*seglen = 0;\n+\t\treturn segs;\n+\t}\n+\tfor (i = 0; i < n; i++) {\n+\t\tint log = fastlog2(sizes[i]);\n+\t\tif (cur.log != log && cur.bytes > 0) {\n+\t\t\tstruct segment fresh = {\n+\t\t\t\t.start = i,\n+\t\t\t};\n+\n+\t\t\tsegs[next++] = cur;\n+\t\t\tcur = fresh;\n+\t\t}\n+\n+\t\tcur.log = log;\n+\t\tcur.end = i + 1;\n+\t\tcur.bytes += sizes[i];\n+\t}\n+\tsegs[next++] = cur;\n+\t*seglen = next;\n+\treturn segs;\n+}\n+\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, sizes, n);\n+\tstruct segment min_seg = {\n+\t\t.log = 64,\n+\t};\n+\tint i = 0;\n+\tfor (i = 0; i < seglen; i++) {\n+\t\tif (segment_size(&segs[i]) == 1) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (segs[i].log < min_seg.log) {\n+\t\t\tmin_seg = segs[i];\n+\t\t}\n+\t}\n+\n+\twhile (min_seg.start > 0) {\n+\t\tint prev = min_seg.start - 1;\n+\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmin_seg.start = prev;\n+\t\tmin_seg.bytes += sizes[prev];\n+\t}\n+\n+\treftable_free(segs);\n+\treturn min_seg;\n+}\n+\n+static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes =\n+\t\treftable_calloc(sizeof(uint64_t) * st->merged->stack_len);\n+\tint version = (st->config.hash_id == SHA1_ID) ? 1 : 2;\n+\tint overhead = header_size(version) - 1;\n+\tint i = 0;\n+\tfor (i = 0; i < st->merged->stack_len; i++) {\n+\t\tsizes[i] = st->readers[i]->size - overhead;\n+\t}\n+\treturn sizes;\n+}\n+\n+int reftable_stack_auto_compact(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n+\tstruct segment seg =\n+\t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n+\treftable_free(sizes);\n+\tif (segment_size(&seg) > 0)\n+\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n+\t\t\t\t\t\t NULL);\n+\n+\treturn 0;\n+}\n+\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st)\n+{\n+\treturn &st->stats;\n+}\n+\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_table tab = { NULL };\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\treturn reftable_table_read_ref(&tab, refname, ref);\n+}\n+\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(st);\n+\tint err = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_log(&it, log);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(log->refname, refname) ||\n+\t reftable_log_record_is_deletion(log)) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tif (err) {\n+\t\treftable_log_record_clear(log);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name)\n+{\n+\tint err = 0;\n+\tstruct reftable_block_source src = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct reftable_ref_record *refs = NULL;\n+\tstruct reftable_iterator it = { NULL };\n+\tint cap = 0;\n+\tint len = 0;\n+\tint i = 0;\n+\n+\tif (st->config.skip_name_check)\n+\t\treturn 0;\n+\n+\terr = reftable_block_source_from_file(&src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&rd, &src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tif (err > 0) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (len >= cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\trefs = reftable_realloc(refs, cap * sizeof(refs[0]));\n+\t\t}\n+\n+\t\trefs[len++] = ref;\n+\t}\n+\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\n+\terr = validate_ref_record_addition(tab, refs, len);\n+\n+done:\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_clear(&refs[i]);\n+\t}\n+\n+\tfree(refs);\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\treturn err;\n+}\ndiff --git a/reftable/stack.h b/reftable/stack.h\nnew file mode 100644\nindex 0000000000..4e9557f08d\n--- /dev/null\n+++ b/reftable/stack.h\n@@ -0,0 +1,40 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef STACK_H\n+#define STACK_H\n+\n+#include \"reftable.h\"\n+#include \"system.h\"\n+\n+struct reftable_stack {\n+\tchar *list_file;\n+\tchar *reftable_dir;\n+\tint disable_auto_compact;\n+\n+\tstruct reftable_write_options config;\n+\n+\tstruct reftable_reader **readers;\n+\tsize_t readers_len;\n+\tstruct reftable_merged_table *merged;\n+\tstruct reftable_compaction_stats stats;\n+};\n+\n+int read_lines(const char *filename, char ***lines);\n+\n+struct segment {\n+\tint start, end;\n+\tint log;\n+\tuint64_t bytes;\n+};\n+\n+int fastlog2(uint64_t sz);\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n);\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n);\n+\n+#endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nnew file mode 100644\nindex 0000000000..534463829c\n--- /dev/null\n+++ b/reftable/stack_test.c\n@@ -0,0 +1,788 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+\n+#include \"merged.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"reftable.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+#include \n+#include \n+\n+static void test_read_file(void)\n+{\n+\tchar fn[256] = \"/tmp/stack.test_read_file.XXXXXX\";\n+\tint fd = mkstemp(fn);\n+\tchar out[1024] = \"line1\\n\\nline2\\nline3\";\n+\tint n, err;\n+\tchar **names = NULL;\n+\tchar *want[] = { \"line1\", \"line2\", \"line3\" };\n+\tint i = 0;\n+\n+\tassert(fd > 0);\n+\tn = write(fd, out, strlen(out));\n+\tassert(n == strlen(out));\n+\terr = close(fd);\n+\tassert(err >= 0);\n+\n+\terr = read_lines(fn, &names);\n+\tassert_err(err);\n+\n+\tfor (i = 0; names[i] != NULL; i++) {\n+\t\tassert(0 == strcmp(want[i], names[i]));\n+\t}\n+\tfree_names(names);\n+\tremove(fn);\n+}\n+\n+static void test_parse_names(void)\n+{\n+\tchar buf[] = \"line\\n\";\n+\tchar **names = NULL;\n+\tparse_names(buf, strlen(buf), &names);\n+\n+\tassert(NULL != names[0]);\n+\tassert(0 == strcmp(names[0], \"line\"));\n+\tassert(NULL == names[1]);\n+\tfree_names(names);\n+}\n+\n+static void test_names_equal(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", \"c\", NULL };\n+\tchar *b[] = { \"a\", \"b\", \"d\", NULL };\n+\tchar *c[] = { \"a\", \"b\", NULL };\n+\n+\tassert(names_equal(a, a));\n+\tassert(!names_equal(a, b));\n+\tassert(!names_equal(a, c));\n+}\n+\n+static int write_test_ref(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct reftable_ref_record *ref = arg;\n+\treftable_writer_set_limits(wr, ref->update_index, ref->update_index);\n+\treturn reftable_writer_add_ref(wr, ref);\n+}\n+\n+struct write_log_arg {\n+\tstruct reftable_log_record *log;\n+\tuint64_t update_index;\n+};\n+\n+static int write_test_log(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct write_log_arg *wla = arg;\n+\n+\treftable_writer_set_limits(wr, wla->update_index, wla->update_index);\n+\treturn reftable_writer_add_log(wr, wla->log);\n+}\n+\n+static void test_reftable_stack_add_one(void)\n+{\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.target = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tassert_err(err);\n+\tassert(0 == strcmp(\"master\", dest.target));\n+\n+\treftable_ref_record_clear(&dest);\n+\treftable_stack_destroy(st);\n+\treftable_clear_dir(dir);\n+}\n+\n+static void test_reftable_stack_uptodate(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL;\n+\tstruct reftable_stack *st2 = NULL;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.target = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"branch2\",\n+\t\t.update_index = 2,\n+\t\t.target = \"master\",\n+\t};\n+\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tassert_err(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_add(st1, &write_test_ref, &ref1);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tassert(err == REFTABLE_LOCK_ERROR);\n+\n+\terr = reftable_stack_reload(st2);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tassert_err(err);\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\treftable_clear_dir(dir);\n+}\n+\n+static void test_reftable_stack_transaction_api(void)\n+{\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_addition *add = NULL;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.target = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_new_addition(&add, st);\n+\tassert_err(err);\n+\n+\terr = reftable_addition_add(add, &write_test_ref, &ref);\n+\tassert_err(err);\n+\n+\terr = reftable_addition_commit(add);\n+\tassert_err(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tassert_err(err);\n+\tassert(0 == strcmp(\"master\", dest.target));\n+\n+\treftable_ref_record_clear(&dest);\n+\treftable_stack_destroy(st);\n+\treftable_clear_dir(dir);\n+}\n+\n+static void test_reftable_stack_validate_refname(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tint i;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"a/b\",\n+\t\t.update_index = 1,\n+\t\t.target = \"master\",\n+\t};\n+\tchar *additions[] = { \"a\", \"a/b/c\" };\n+\n+\tassert(mkdtemp(dir));\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tassert_err(err);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(additions); i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = additions[i],\n+\t\t\t.update_index = 1,\n+\t\t\t.target = \"master\",\n+\t\t};\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tassert(err == REFTABLE_NAME_CONFLICT);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\treftable_clear_dir(dir);\n+}\n+\n+static int write_error(struct reftable_writer *wr, void *arg)\n+{\n+\treturn *((int *)arg);\n+}\n+\n+static void test_reftable_stack_update_index_check(void)\n+{\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"name1\",\n+\t\t.update_index = 1,\n+\t\t.target = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"name2\",\n+\t\t.update_index = 1,\n+\t\t.target = \"master\",\n+\t};\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref1);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref2);\n+\tassert(err == REFTABLE_API_ERROR);\n+\treftable_stack_destroy(st);\n+\treftable_clear_dir(dir);\n+}\n+\n+static void test_reftable_stack_lock_failure(void)\n+{\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err, i;\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\tfor (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) {\n+\t\terr = reftable_stack_add(st, &write_error, &i);\n+\t\tassert(err == i);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\treftable_clear_dir(dir);\n+}\n+\n+static void test_reftable_stack_add(void)\n+{\n+\tint i = 0;\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t.exact_log_message = 1,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\tst->disable_auto_compact = 1;\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].value = reftable_malloc(SHA1_SIZE);\n+\t\trefs[i].update_index = i + 1;\n+\t\tset_test_hash(refs[i].value, i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = N + i + 1;\n+\t\tlogs[i].new_hash = reftable_malloc(SHA1_SIZE);\n+\t\tlogs[i].email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].new_hash, i);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tassert_err(err);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tassert_err(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tassert_err(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_ref_record dest = { NULL };\n+\n+\t\tint err = reftable_stack_read_ref(st, refs[i].refname, &dest);\n+\t\tassert_err(err);\n+\t\tassert(reftable_ref_record_equal(&dest, refs + i, SHA1_SIZE));\n+\t\treftable_ref_record_clear(&dest);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_log_record dest = { NULL };\n+\t\tint err = reftable_stack_read_log(st, refs[i].refname, &dest);\n+\t\tassert_err(err);\n+\t\tassert(reftable_log_record_equal(&dest, logs + i, SHA1_SIZE));\n+\t\treftable_log_record_clear(&dest);\n+\t}\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_clear(&refs[i]);\n+\t\treftable_log_record_clear(&logs[i]);\n+\t}\n+\treftable_clear_dir(dir);\n+}\n+\n+static void test_reftable_stack_log_normalize(void)\n+{\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t0,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\n+\tuint8_t h1[SHA1_SIZE] = { 0x01 }, h2[SHA1_SIZE] = { 0x02 };\n+\n+\tstruct reftable_log_record input = {\n+\t\t.refname = \"branch\",\n+\t\t.update_index = 1,\n+\t\t.new_hash = h1,\n+\t\t.old_hash = h2,\n+\t};\n+\tstruct reftable_log_record dest = {\n+\t\t.update_index = 0,\n+\t};\n+\tstruct write_log_arg arg = {\n+\t\t.log = &input,\n+\t\t.update_index = 1,\n+\t};\n+\n+\tassert(mkdtemp(dir));\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\tinput.message = \"one\\ntwo\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tassert(err == REFTABLE_API_ERROR);\n+\n+\tinput.message = \"one\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tassert_err(err);\n+\tassert(0 == strcmp(dest.message, \"one\\n\"));\n+\n+\tinput.message = \"two\\n\";\n+\targ.update_index = 2;\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tassert_err(err);\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tassert_err(err);\n+\tassert(0 == strcmp(dest.message, \"two\\n\"));\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\treftable_log_record_clear(&dest);\n+\treftable_clear_dir(dir);\n+}\n+\n+static void test_reftable_stack_tombstone(void)\n+{\n+\tint i = 0;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\tstruct reftable_ref_record dest = { NULL };\n+\tstruct reftable_log_record log_dest = { NULL };\n+\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tconst char *buf = \"branch\";\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\tif (i % 2 == 0) {\n+\t\t\trefs[i].value = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_test_hash(refs[i].value, i);\n+\t\t}\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\t/* update_index is part of the key. */\n+\t\tlogs[i].update_index = 42;\n+\t\tif (i % 2 == 0) {\n+\t\t\tlogs[i].new_hash = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_test_hash(logs[i].new_hash, i);\n+\t\t\tlogs[i].email = xstrdup(\"identity@invalid\");\n+\t\t}\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tassert_err(err);\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tassert_err(err);\n+\t}\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tassert(err == 1);\n+\treftable_ref_record_clear(&dest);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tassert(err == 1);\n+\treftable_log_record_clear(&log_dest);\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tassert(err == 1);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tassert(err == 1);\n+\treftable_ref_record_clear(&dest);\n+\treftable_log_record_clear(&log_dest);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_clear(&refs[i]);\n+\t\treftable_log_record_clear(&logs[i]);\n+\t}\n+\treftable_clear_dir(dir);\n+}\n+\n+static void test_reftable_stack_hash_id(void)\n+{\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"master\",\n+\t\t.target = \"target\",\n+\t\t.update_index = 1,\n+\t};\n+\tstruct reftable_write_options cfg32 = { .hash_id = SHA256_ID };\n+\tstruct reftable_stack *st32 = NULL;\n+\tstruct reftable_write_options cfg_default = { 0 };\n+\tstruct reftable_stack *st_default = NULL;\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\tassert(mkdtemp(dir));\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tassert_err(err);\n+\n+\t/* can't read it with the wrong hash ID. */\n+\terr = reftable_new_stack(&st32, dir, cfg32);\n+\tassert(err == REFTABLE_FORMAT_ERROR);\n+\n+\t/* check that we can read it back with default config too. */\n+\terr = reftable_new_stack(&st_default, dir, cfg_default);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n+\tassert_err(err);\n+\n+\tassert(!strcmp(dest.target, ref.target));\n+\treftable_ref_record_clear(&dest);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st_default);\n+\treftable_clear_dir(dir);\n+}\n+\n+static void test_log2(void)\n+{\n+\tassert(1 == fastlog2(3));\n+\tassert(2 == fastlog2(4));\n+\tassert(2 == fastlog2(5));\n+}\n+\n+static void test_sizes_to_segments(void)\n+{\n+\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n+\t/* .................0 1 2 3 4 5 */\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tassert(segs[2].log == 3);\n+\tassert(segs[2].start == 5);\n+\tassert(segs[2].end == 6);\n+\n+\tassert(segs[1].log == 2);\n+\tassert(segs[1].start == 2);\n+\tassert(segs[1].end == 5);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_empty(void)\n+{\n+\tuint64_t sizes[0];\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tassert(seglen == 0);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_all_equal(void)\n+{\n+\tuint64_t sizes[] = { 5, 5 };\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tassert(seglen == 1);\n+\tassert(segs[0].start == 0);\n+\tassert(segs[0].end == 2);\n+\treftable_free(segs);\n+}\n+\n+static void test_suggest_compaction_segment(void)\n+{\n+\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n+\t/* .................0 1 2 3 4 5 6 */\n+\tstruct segment min =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tassert(min.start == 2);\n+\tassert(min.end == 7);\n+}\n+\n+static void test_suggest_compaction_segment_nothing(void)\n+{\n+\tuint64_t sizes[] = { 64, 32, 16, 8, 4, 2 };\n+\tstruct segment result =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tassert(result.start == result.end);\n+}\n+\n+static void test_reflog_expire(void)\n+{\n+\tchar dir[256] = \"/tmp/stack.test_reflog_expire.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tstruct reftable_log_record logs[20] = { { NULL } };\n+\tint N = ARRAY_SIZE(logs) - 1;\n+\tint i = 0;\n+\tint err;\n+\tstruct reftable_log_expiry_config expiry = {\n+\t\t.time = 10,\n+\t};\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = i;\n+\t\tlogs[i].time = i;\n+\t\tlogs[i].new_hash = reftable_malloc(SHA1_SIZE);\n+\t\tlogs[i].email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].new_hash, i);\n+\t}\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tassert_err(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_read_log(st, logs[9].refname, &log);\n+\tassert(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[11].refname, &log);\n+\tassert_err(err);\n+\n+\texpiry.min_update_index = 15;\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_read_log(st, logs[14].refname, &log);\n+\tassert(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[16].refname, &log);\n+\tassert_err(err);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i <= N; i++) {\n+\t\treftable_log_record_clear(&logs[i]);\n+\t}\n+\treftable_clear_dir(dir);\n+\treftable_log_record_clear(&log);\n+}\n+\n+static int write_nothing(struct reftable_writer *wr, void *arg)\n+{\n+\treftable_writer_set_limits(wr, 1, 1);\n+\treturn 0;\n+}\n+\n+static void test_empty_add(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_stack *st2 = NULL;\n+\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\terr = reftable_stack_add(st, &write_nothing, NULL);\n+\tassert_err(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tassert_err(err);\n+\treftable_clear_dir(dir);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st2);\n+}\n+\n+static void test_reftable_stack_auto_compaction(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tint err, i;\n+\tint N = 100;\n+\tassert(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tassert_err(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t\t.target = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tassert_err(err);\n+\n+\t\tassert(i < 3 || st->merged->stack_len < 2 * fastlog2(i));\n+\t}\n+\n+\tassert(reftable_stack_compaction_stats(st)->entries_written <\n+\t (uint64_t)(N * fastlog2(N)));\n+\n+\treftable_stack_destroy(st);\n+\treftable_clear_dir(dir);\n+}\n+\n+int stack_test_main(int argc, const char *argv[])\n+{\n+\tadd_test_case(\"test_reftable_stack_uptodate\",\n+\t\t &test_reftable_stack_uptodate);\n+\tadd_test_case(\"test_reftable_stack_transaction_api\",\n+\t\t &test_reftable_stack_transaction_api);\n+\tadd_test_case(\"test_reftable_stack_hash_id\",\n+\t\t &test_reftable_stack_hash_id);\n+\tadd_test_case(\"test_sizes_to_segments_all_equal\",\n+\t\t &test_sizes_to_segments_all_equal);\n+\tadd_test_case(\"test_reftable_stack_auto_compaction\",\n+\t\t &test_reftable_stack_auto_compaction);\n+\tadd_test_case(\"test_reftable_stack_validate_refname\",\n+\t\t &test_reftable_stack_validate_refname);\n+\tadd_test_case(\"test_reftable_stack_update_index_check\",\n+\t\t &test_reftable_stack_update_index_check);\n+\tadd_test_case(\"test_reftable_stack_lock_failure\",\n+\t\t &test_reftable_stack_lock_failure);\n+\tadd_test_case(\"test_reftable_stack_log_normalize\",\n+\t\t &test_reftable_stack_log_normalize);\n+\tadd_test_case(\"test_reftable_stack_tombstone\",\n+\t\t &test_reftable_stack_tombstone);\n+\tadd_test_case(\"test_reftable_stack_add_one\",\n+\t\t &test_reftable_stack_add_one);\n+\tadd_test_case(\"test_empty_add\", test_empty_add);\n+\tadd_test_case(\"test_reflog_expire\", test_reflog_expire);\n+\tadd_test_case(\"test_suggest_compaction_segment\",\n+\t\t &test_suggest_compaction_segment);\n+\tadd_test_case(\"test_suggest_compaction_segment_nothing\",\n+\t\t &test_suggest_compaction_segment_nothing);\n+\tadd_test_case(\"test_sizes_to_segments\", &test_sizes_to_segments);\n+\tadd_test_case(\"test_sizes_to_segments_empty\",\n+\t\t &test_sizes_to_segments_empty);\n+\tadd_test_case(\"test_log2\", &test_log2);\n+\tadd_test_case(\"test_parse_names\", &test_parse_names);\n+\tadd_test_case(\"test_read_file\", &test_read_file);\n+\tadd_test_case(\"test_names_equal\", &test_names_equal);\n+\tadd_test_case(\"test_reftable_stack_add\", &test_reftable_stack_add);\n+\treturn test_main(argc, argv);\n+}\ndiff --git a/reftable/update.sh b/reftable/update.sh\nnew file mode 100755\nindex 0000000000..4dadd875f4\n--- /dev/null\n+++ b/reftable/update.sh\n@@ -0,0 +1,22 @@\n+#!/bin/sh\n+\n+set -eu\n+\n+# Override this to import from somewhere else, say \"../reftable\".\n+SRC=${SRC:-origin}\n+BRANCH=${BRANCH:-master}\n+\n+((git --git-dir reftable-repo/.git fetch -f ${SRC} ${BRANCH}:import && cd reftable-repo && git checkout -f $(git rev-parse import) ) ||\n+ git clone https://github.com/google/reftable reftable-repo)\n+\n+cp reftable-repo/c/*.[ch] reftable/\n+cp reftable-repo/c/include/*.[ch] reftable/\n+cp reftable-repo/LICENSE reftable/\n+\n+git --git-dir reftable-repo/.git show --no-patch --format=oneline HEAD \\\n+ > reftable/VERSION\n+\n+mv reftable/system.h reftable/system.h~\n+sed 's|if REFTABLE_IN_GITCORE|if 1 /* REFTABLE_IN_GITCORE */|' < reftable/system.h~ > reftable/system.h\n+\n+git add reftable/*.[ch] reftable/LICENSE reftable/VERSION\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 2273fdfe39..def8883439 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,9 +4,12 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tblock_test_main(argc, argv);\n+\tmerged_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\trefname_test_main(argc, argv);\n \treftable_test_main(argc, argv);\n \tstrbuf_test_main(argc, argv);\n+\tstack_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"406788","messageId":"xmqqk0w9bvir.fsf@gitster.c.googlers.com","threadId":"54245","inReplyTo":"8eb944ea9b1c0bdc4a3ebb4e79fd12187f6c988c.1601568663.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 06/13] reftable: (de)serialization for the polymorphic record type.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-01T19:23:24Z","receivedAt":"2020-10-01T19:23:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Han-Wen Nienhuys via GitGitGadget\" writes:\n\n> +struct reftable_record reftable_new_record(uint8_t typ)\n> +{\n> +\tstruct reftable_record rec = { NULL };\n\nI can see a lot of \"sparse\" inspired work went into this, but we\nwould want to take the previous discussion into account, as nothing\nhas changed since then.\n\nhttps://lore.kernel.org/git/41a60e60-d2c0-7d54-5456-e44d106548a4@kdbg.org/\n\nhttps://lore.kernel.org/git/1df91aa4-dda5-64da-6ae3-5d65e50a55c5@ramsayjones.plus.com/\n\nUnfortunately we didn't reach a clear consensus back then; I thought\nwe saw a fix to make sparse silent when we use the \"= { 0 }\" idiom\nin recent versions of sparse, but the above threads do not have any\nmention of it, so either I am misremembering or there were other\ndiscussions on the same topic where it was also mentioned.\n\n> +\tswitch (typ) {\n\nNeed blank before this line.\n\nThanks.\n"},{"id":"406789","messageId":"18bd6127-be72-b7b7-8e2a-17bbe7214a2e@ramsayjones.plus.com","threadId":"54245","inReplyTo":"xmqqk0w9bvir.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 06/13] reftable: (de)serialization for the polymorphic record type.","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2020-10-01T19:59:23Z","receivedAt":"2020-10-01T20:07:11Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\n\nOn 01/10/2020 20:23, Junio C Hamano wrote:\n> \"Han-Wen Nienhuys via GitGitGadget\" writes:\n> \n>> +struct reftable_record reftable_new_record(uint8_t typ)\n>> +{\n>> +\tstruct reftable_record rec = { NULL };\n> \n> I can see a lot of \"sparse\" inspired work went into this, but we\n> would want to take the previous discussion into account, as nothing\n> has changed since then.\n> \n> https://lore.kernel.org/git/41a60e60-d2c0-7d54-5456-e44d106548a4@kdbg.org/\n> \n> https://lore.kernel.org/git/1df91aa4-dda5-64da-6ae3-5d65e50a55c5@ramsayjones.plus.com/\n\nyep, sparse is equally happy with = { 0 }; here - at least any sparse\nversion after 'v0.6.1-246-g41f651b4'. ;-) So, as I said yesterday, the\nlast official release v0.6.2 should be fine, and I _think_ Debian Testing\nhas a suitable version. (or you could build from source).\n\n> Unfortunately we didn't reach a clear consensus back then; I thought\n> we saw a fix to make sparse silent when we use the \"= { 0 }\" idiom\n> in recent versions of sparse, but the above threads do not have any\n> mention of it, so either I am misremembering or there were other\n> discussions on the same topic where it was also mentioned.\n\nSee commit 1c96642326 (sparse: allow '{ 0 }' to be used without warnings,\n22-05-2020). We could actually remove that now, since '-Wno-universal-initializer'\nis now the default (but it doesn't hurt).\n\nATB,\nRamsay Jones\n\n\n\n"},{"id":"406797","messageId":"20201002031845.GA3252492@google.com","threadId":"54245","inReplyTo":"6228103b4a6dab23ce3a4498eada8ff649909ff0.1601568663.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 01/13] reftable: add LICENSE","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-10-02T03:18:45Z","receivedAt":"2020-10-02T03:18:50Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nHan-Wen Nienhuys wrote:\n\n> TODO: relicense?\n\nThis line presumably doesn't want to be part of permanent history. :)\n\nInstead, it can say a little about rationale. For example, does this\nmake the code easier for other projects like libgit2 to share?\n\n> Signed-off-by: Han-Wen Nienhuys \n> ---\n> reftable/LICENSE | 31 +++++++++++++++++++++++++++++++\n> 1 file changed, 31 insertions(+)\n> create mode 100644 reftable/LICENSE\n\nShould there be a reftable/README as well?\n\n[...]\n> --- /dev/null\n> +++ b/reftable/LICENSE\n> @@ -0,0 +1,31 @@\n> +BSD License\n> +\n> +Copyright (c) 2020, Google LLC\n> +All rights reserved.\n\nnit: Google seems to prefer a slightly more spartan form of this\nlicense text:\nhttps://opensource.google/docs/releasing/licenses/#bsd-license-file\n\nNits aside, except for the TODO line, this patch looks reasonable to\nme.\n\nThanks,\nJonathan\n"},{"id":"406798","messageId":"20201002035851.GB3252492@google.com","threadId":"54245","inReplyTo":"5d1b946ab5473504e9599d90c0feda407a179a05.1601568663.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 02/13] reftable: define the public API","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-10-02T03:58:51Z","receivedAt":"2020-10-02T04:00:03Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nHan-Wen Nienhuys wrote:\n\n> reftable/reftable.h | 585 ++++++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 585 insertions(+)\n> create mode 100644 reftable/reftable.h\n\nAdding a header in a separate patch from the implementation doesn't\nmatch the usual practice. Can we add declarations in the same patch\nas the functions being declared instead?\n\nWe could still introduce the header early in its own patch if we want,\nbut it would be a skeleton of a header that gets filled out by later\npatches.\n\n[...]\n> --- /dev/null\n> +++ b/reftable/reftable.h\n> @@ -0,0 +1,585 @@\n[...]\n> +#ifndef REFTABLE_H\n> +#define REFTABLE_H\n> +\n> +#include \n> +#include \n\nGit tends to prefer to rely on the caller to have #include-ed\ngit-compat-util.h instead. That helps avoid having to worry about\nportability gotchas around what feature test macros need to have been\nset before the first #include of a standard header.\n\n> +\n> +void reftable_set_alloc(void *(*malloc)(size_t),\n> +\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n> +\n> +/****************************************************************\n> + Basic data types\n> +\n> + Reftables store the state of each ref in struct reftable_ref_record, and they\n> + store a sequence of reflog updates in struct reftable_log_record.\n> + ****************************************************************/\n\nnit: Git style for comments would be\n\n/*\n * Basic data types\n *\n * Reftables store the state of each [etc]\n */\n\nIn other words:\n\n- no lines of stars\n- \"/*\" with no text after it as first line\n- \" *\" leading subsequent lines\n- \" */\" as last line\n\n\n> +\n> +/* reftable_ref_record holds a ref database entry target_value */\n> +struct reftable_ref_record {\n> +\tchar *refname; /* Name of the ref, malloced. */\n> +\tuint64_t update_index; /* Logical timestamp at which this value is\n> +\t\t\t\t written */\n\nLikewise: Git doesn't have this style of multi-line comment, but this\ncould be e.g. a single-line comment before\n\n\t/* logical timestamp at which this value was written */\n\tuint64_t update_index;\n\n> +\tuint8_t *value; /* SHA1, or NULL. malloced. */\n\nWhat does \"NULL. malloced\" mean? Is the point that the\nreftable_ref_record owns the memory? (We don't do as good a job of\ndocumenting that as we should. Often a non-const pointer is owned,\nbut in this example it's probably better to be explicit that this\nstruct owns the memory pointed to by all its fields in the comment\ndescribing the struct as a whole.)\n\nIt would be more idiomatic for this to be a 'struct object_id' inline\nin the reftable_ref_record.\n\n> +\tuint8_t *target_value; /* peeled annotated tag, or NULL. malloced. */\n> +\tchar *target; /* symref, or NULL. malloced. */\n> +};\n> +\n> +/* returns whether 'ref' represents a deletion */\n> +int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);\n\nNice, this looks pleasant to use.\n\n> +\n> +/* prints a reftable_ref_record onto stdout */\n> +void reftable_ref_record_print(struct reftable_ref_record *ref,\n> +\t\t\t uint32_t hash_id);\n\nProbably worth being explicit about the purpose: is this for\ndebugging?\n\nWhat is a hash_id?\n\n> +\n> +/* frees and nulls all pointer values. */\n> +void reftable_ref_record_clear(struct reftable_ref_record *ref);\n> +\n> +/* returns whether two reftable_ref_records are the same */\n> +int reftable_ref_record_equal(struct reftable_ref_record *a,\n> +\t\t\t struct reftable_ref_record *b, int hash_size);\n> +\n> +/* reftable_log_record holds a reflog entry */\n> +struct reftable_log_record {\n\nShould some of this documentation mention the reftable spec? E.g. an\nopening comment at the beginning of the header. That might help avoid\nhaving to duplicate too much explanation when terminology matches.\n\n> +\tchar *refname;\n> +\tuint64_t update_index; /* logical timestamp of a transactional update.\n> +\t\t\t\t*/\n\nmultiline comment (I won't mention the rest).\n\n> +\tuint8_t *new_hash;\n> +\tuint8_t *old_hash;\n> +\tchar *name;\n> +\tchar *email;\n> +\tuint64_t time;\n> +\tint16_t tz_offset;\n> +\tchar *message;\n> +};\n> +\n> +/* returns whether 'ref' represents the deletion of a log record. */\n\nThese are a bit subtle --- should the comment include a little more\nexplanation (e.g. about how this is mostly for tools like \"git\nstash\")?\n\n> +int reftable_log_record_is_deletion(const struct reftable_log_record *log);\n> +\n> +/* frees and nulls all pointer values. */\n> +void reftable_log_record_clear(struct reftable_log_record *log);\n> +\n> +/* returns whether two records are equal. */\n> +int reftable_log_record_equal(struct reftable_log_record *a,\n> +\t\t\t struct reftable_log_record *b, int hash_size);\n> +\n> +/* dumps a reftable_log_record on stdout, for debugging/testing. */\n> +void reftable_log_record_print(struct reftable_log_record *log,\n> +\t\t\t uint32_t hash_id);\n> +\n> +/****************************************************************\n> + Error handling\n> +\n> + Error are signaled with negative integer return values. 0 means success.\n> + ****************************************************************/\n> +\n> +/* different types of errors */\n> +enum reftable_error {\n\nReminds me of zlib and liblzma. I like it.\n\n> +\t/* Unexpected file system behavior */\n> +\tREFTABLE_IO_ERROR = -2,\n> +\n> +\t/* Format inconsistency on reading data\n> +\t */\n> +\tREFTABLE_FORMAT_ERROR = -3,\n> +\n> +\t/* File does not exist. Returned from block_source_from_file(), because\n> +\t it needs special handling in stack.\n> +\t*/\n> +\tREFTABLE_NOT_EXIST_ERROR = -4,\n> +\n> +\t/* Trying to write out-of-date data. */\n> +\tREFTABLE_LOCK_ERROR = -5,\n> +\n> +\t/* Misuse of the API:\n> +\t - on writing a record with NULL refname.\n> +\t - on writing a reftable_ref_record outside the table limits\n> +\t - on writing a ref or log record before the stack's next_update_index\n> +\t - on writing a log record with multiline message with\n> +\t exact_log_message unset\n> +\t - on reading a reftable_ref_record from log iterator, or vice versa.\n> +\t*/\n> +\tREFTABLE_API_ERROR = -6,\n\nShould these call BUG()? Or is it useful for some callers to be able\nto recover from these errors?\n\n> +\n> +\t/* Decompression error */\n> +\tREFTABLE_ZLIB_ERROR = -7,\n> +\n> +\t/* Wrote a table without blocks. */\n> +\tREFTABLE_EMPTY_TABLE_ERROR = -8,\n> +\n> +\t/* Dir/file conflict. */\n\nThis is when adding a ref?\n\n> +\tREFTABLE_NAME_CONFLICT = -9,\n> +\n> +\t/* Illegal ref name. */\n> +\tREFTABLE_REFNAME_ERROR = -10,\n> +};\n> +\n> +/* convert the numeric error code to a string. The string should not be\n> + * deallocated. */\n> +const char *reftable_error_str(int err);\n\nIn the ZLIB_ERROR case this would produce a frustratingly vague\nmessage. I think I don't mind, but it might be worth a NEEDSWORK\ncomment to revisit in the future.\n\n> +\n> +/*\n> + * Convert the numeric error code to an equivalent errno code.\n> + */\n> +int reftable_error_to_errno(int err);\n\nWhat is the intended use of this function?\n\n[...]\n> +/* reftable_block_stats holds statistics for a single block type */\n> +struct reftable_block_stats {\n\nAre these for use in tracing, debugging, or some other purpose?\n\n[...]\n> +/* reftable_new_writer creates a new writer */\n> +struct reftable_writer *\n> +reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n> +\t\t void *writer_arg, struct reftable_write_options *opts);\n\nWhat is a writer? What is a writer_func? Is this a handle that\nallows me to determine where I write my reftable? (Sounds reasonable\nbut the comment could be more explicit.)\n\n[...]\n> +\n> +/* write to a file descriptor. fdp should be an int* pointing to the fd. */\n> +int reftable_fd_write(void *fdp, const void *data, size_t size);\n\nWhat does this do for interrupted writes?\n\n> +\n> +/* Set the range of update indices for the records we will add. When\n> + writing a table into a stack, the min should be at least\n> + reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned.\n> +\n> + For transactional updates, typically min==max. When converting an existing\n> + ref database into a single reftable, this would be a range of update-index\n> + timestamps.\n> + */\n> +void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n> +\t\t\t\tuint64_t max);\n\nWhat happens if I write to a reftable_writer without setting limits\nfirst?\n\n> +\n> +/* adds a reftable_ref_record. Must be called in ascending\n> + order. The update_index must be within the limits set by\n> + reftable_writer_set_limits(), or REFTABLE_API_ERROR is returned.\n> +\n> + It is an error to write a ref record after a log record.\n> + */\n> +int reftable_writer_add_ref(struct reftable_writer *w,\n> +\t\t\t struct reftable_ref_record *ref);\n\noptional: since it's called a writer, _write_ref() feels like a\nslightly more natural verb (likewise for the others).\n\n[...]\n> +/* adds a reftable_log_record. Must be called in ascending order (with more\n> + recent log entries first.)\n> + */\n> +int reftable_writer_add_log(struct reftable_writer *w,\n> +\t\t\t struct reftable_log_record *log);\n\nIf I want to write multiple log records with the same update_index to\nrepresent a multi-ref transaction, do I have to do anything careful?\n\n> +\n> +/* Convenience function to add multiple logs. Will sort the records by\n> + key before adding. */\n> +int reftable_writer_add_logs(struct reftable_writer *w,\n> +\t\t\t struct reftable_log_record *logs, int n);\n> +\n> +/* reftable_writer_close finalizes the reftable. The writer is retained so\n> + * statistics can be inspected. */\n> +int reftable_writer_close(struct reftable_writer *w);\n\nNice.\n\n> +\n> +/* writer_stats returns the statistics on the reftable being written.\n> +\n> + This struct becomes invalid when the writer is freed.\n> + */\n> +const struct reftable_stats *writer_stats(struct reftable_writer *w);\n\nAlso nice.\n\n[...]\n> +/* iterator is the generic interface for walking over data stored in a\n> + reftable. It is generally passed around by value.\n> +*/\n> +struct reftable_iterator {\n> +\tstruct reftable_iterator_vtable *ops;\n> +\tvoid *iter_arg;\n> +};\n\nEven small structs like this one tend not to be passed by value in the\nGit codebase. The only counterexample I found with a quick search\n(builtin/stash.c::save_untracked_files) appears to be a typo.\n\n[...]\n> +/* a contiguous segment of bytes. It keeps track of its generating block_source\n> + so it can return itself into the pool.\n> +*/\n> +struct reftable_block {\n> +\tuint8_t *data;\n> +\tint len;\n> +\tstruct reftable_block_source source;\n> +};\n\nIs this something that could use mem-pool.h?\n\n> +/* block_source_vtable are the operations that make up block_source */\n> +struct reftable_block_source_vtable {\n> +\t/* returns the size of a block source */\n> +\tuint64_t (*size)(void *source);\n> +\n> +\t/* reads a segment from the block source. It is an error to read\n> +\t beyond the end of the block */\n> +\tint (*read_block)(void *source, struct reftable_block *dest,\n> +\t\t\t uint64_t off, uint32_t size);\n> +\t/* mark the block as read; may return the data back to malloc */\n> +\tvoid (*return_block)(void *source, struct reftable_block *blockp);\n> +\n> +\t/* release all resources associated with the block source */\n> +\tvoid (*close)(void *source);\n> +};\n\nThis is abstracting file input? I wonder if some name emphasizing the\n'read' operation like block_reader might be clearer.\n\nDo I pass in the 'struct block_source *' as the source arg? If so, why\nare these declared as void *?\n\nIs the reason this manages the buffer instead of requiring a\ncaller-supplied buffer to support zero-copy?\n\n[...]\n> +\n> +/* opens a file on the file system as a block_source */\n> +int reftable_block_source_from_file(struct reftable_block_source *block_src,\n> +\t\t\t\t const char *name);\n> +\n> +/* The reader struct is a handle to an open reftable file. */\n> +struct reftable_reader;\n\n... ah, this is why the underlying file handle isn't called a reader.\nHmm.\n\n[...]\n> +/* reftable_reader_seek_ref returns an iterator where 'name' would be inserted\n> + in the table. To seek to the start of the table, use name = \"\".\n\nnit: s/iterator where/iterator pointing where/\n\n> +\n> + example:\n> +\n> + struct reftable_reader *r = NULL;\n> + int err = reftable_new_reader(&r, &src, \"filename\");\n> + if (err < 0) { ... }\n> + struct reftable_iterator it = {0};\n> + err = reftable_reader_seek_ref(r, &it, \"refs/heads/master\");\n> + if (err < 0) { ... }\n> + struct reftable_ref_record ref = {0};\n> + while (1) {\n> + err = reftable_iterator_next_ref(&it, &ref);\n> + if (err > 0) {\n> + break;\n> + }\n> + if (err < 0) {\n> + ..error handling..\n> + }\n> + ..found..\n> + }\n> + reftable_iterator_destroy(&it);\n> + reftable_ref_record_clear(&ref);\n\nNice.\n\n> + */\n> +int reftable_reader_seek_ref(struct reftable_reader *r,\n> +\t\t\t struct reftable_iterator *it, const char *name);\n> +\n> +/* returns the hash ID used in this table. */\n> +uint32_t reftable_reader_hash_id(struct reftable_reader *r);\n\nWhat is a hash id?\n\n[...]\n> + Merged tables\n> +\n> + A ref database kept in a sequence of table files. The merged_table presents a\n> + unified view to reading (seeking, iterating) a sequence of immutable tables.\n\nNice as well.\n\n[...]\n> +/* returns the max update_index covered by this merged table. */\n> +uint64_t\n> +reftable_merged_table_max_update_index(struct reftable_merged_table *mt);\n\nnit: return type should be on the same line as the function name\n\n\tuint64_t reftable_merged_table_max_update_index(\n\t\t\t\t\tstruct reftable_merged_table *mt);\n\n[...]\n> + Generic tables\n> +\n> + A unified API for reading tables, either merged tables, or single readers.\n\nAre there callers/helpers that don't know whether they want one or the\nother?\n\n[...]\n> + * a stack is a stack of reftables, which can be mutated by pushing a table to\n> + * the top of the stack\n\nNice.\n\n[...]\n> +/* holds a transaction to add tables at the top of a stack. */\n> +struct reftable_addition;\n\nWhen would I add multiple tables at once to the top? Is this used\nduring compacting, for example?\n\n[...]\n> +/* heuristically compact unbalanced table stack. */\n> +int reftable_stack_auto_compact(struct reftable_stack *st);\n\nWhen do I call this? What are the semantics? (By this I don't mean\nwhat heuristic does it use, but what expectation should I have as a\ncaller?)\n\n> +/* convenience function to read a single log. Returns < 0 for error, 0\n> + for success, and 1 if ref not found. */\n> +int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n> +\t\t\t struct reftable_log_record *log);\n\nnit: single log *record*?\n\n> +/* statistics on past compactions. */\n> +struct reftable_compaction_stats {\n> +\tuint64_t bytes; /* total number of bytes written */\n> +\tuint64_t entries_written; /* total number of entries written, including\n> +\t\t\t\t failures. */\n> +\tint attempts; /* how often we tried to compact */\n> +\tint failures; /* failures happen on concurrent updates */\n> +};\n> +\n> +/* return statistics for compaction up till now. */\n> +struct reftable_compaction_stats *\n> +reftable_stack_compaction_stats(struct reftable_stack *st);\n\nAll in all, I like this API. Thanks for putting it together\nthoughtfully.\n\nThe API builds up in layers (blocks, reftables, merged reftables, etc),\nsuggesting a natural division for the patch series. I think the rest of\nthe series follows that --- let's see.\n\nThanks,\nJonathan\n"},{"id":"406799","messageId":"20201002040200.GC3252492@google.com","threadId":"54245","inReplyTo":"01a669a731dfce02c714a40af999b100c75815e4.1601568663.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 03/13] vcxproj: adjust for the reftable changes","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-10-02T04:02:00Z","receivedAt":"2020-10-02T04:02:05Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Han-Wen Nienhuys wrote:\n\n> From: Johannes Schindelin \n>\n> This allows Git to be compiled via Visual Studio again after integrating\n> the `hn/reftable` branch.\n\nnit: This branch name is no longer meaningful to the primary audience\nfor the commit message (people discovering this commit in git history\nlater).\n\n> Signed-off-by: Johannes Schindelin \n> ---\n> config.mak.uname | 2 +-\n> contrib/buildsystems/Generators/Vcxproj.pm | 11 ++++++++++-\n> 2 files changed, 11 insertions(+), 2 deletions(-)\n\nCan this be squashed into or put immediately after patch 5 which\nintroduces the Makefile?\n\nThanks,\nJonathan\n"},{"id":"406800","messageId":"20201002040559.GD3252492@google.com","threadId":"54245","inReplyTo":"b94c5f5c6120347fd97bcc2dcc187fc86a802dff.1601568663.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 04/13] reftable: add a barebones unittest framework","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-10-02T04:05:59Z","receivedAt":"2020-10-02T04:06:04Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nHan-Wen Nienhuys wrote:\n\n> Signed-off-by: Han-Wen Nienhuys \n> ---\n> reftable/test_framework.c | 68 +++++++++++++++++++++++++++++++++++++++\n> reftable/test_framework.h | 59 +++++++++++++++++++++++++++++++++\n> 2 files changed, 127 insertions(+)\n> create mode 100644 reftable/test_framework.c\n> create mode 100644 reftable/test_framework.h\n\nHm, can the commit message say a little about the motivation for this\ntest framework and what it allows?\n\nFor example, does it allow unit tests to write output in TAP format?\nHow does it compare to existing frameworks like\nhttps://www.eyrie.org/~eagle/software/c-tap-harness/? How does this\napproach compare to what Git's existing unit-style tests with helpers\nin t/helper/ driven by scripts in t/ do?\n\nThanks,\nJonathan\n"},{"id":"406801","messageId":"20201002041214.GE3252492@google.com","threadId":"54245","inReplyTo":"4190da597e65bce072fa3c37c9410a56def4b489.1601568663.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-10-02T04:12:14Z","receivedAt":"2020-10-02T04:12:19Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Han-Wen Nienhuys wrote:\n\n> --- /dev/null\n> +++ b/reftable/basics.c\n> @@ -0,0 +1,131 @@\n> +/*\n> +Copyright 2020 Google LLC\n> +\n> +Use of this source code is governed by a BSD-style\n> +license that can be found in the LICENSE file or at\n> +https://developers.google.com/open-source/licenses/bsd\n> +*/\n> +\n> +#include \"basics.h\"\n\nGit's source files start with #include-ing \"git-compat-util.h\"; how do\nwe approach that here?\n\nE.g. would we pass -include on the command line, or could we #include\nsome kind of compat-util.h that #include-s \"git-compat-util.h\" in Git\nand does the appropriate steps for enabling feature test macros and\nincluding system headers when used outside?\n\n[...]\n> +int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n\nHow does this compare to stdlib's bsearch?\n\n[...]\n> +void free_names(char **a)\n> +{\n> +\tchar **p = a;\n> +\tif (p == NULL) {\n> +\t\treturn;\n> +\t}\n> +\twhile (*p) {\n> +\t\treftable_free(*p);\n> +\t\tp++;\n> +\t}\n> +\treftable_free(a);\n> +}\n\nAre there other callers that need custom free?\n\n[...]\n> +int names_length(char **names)\n> +{\n> +\tint len = 0;\n> +\tchar **p = names;\n> +\twhile (*p) {\n> +\t\tp++;\n> +\t\tlen++;\n> +\t}\n> +\treturn len;\n> +}\n\nThe rest are probably easier to evaluate when I look at the callers, and\nit's time for me to go to sleep now. I'll try to find some time soon to\npick the review back up.\n\nThanks for a thoughtfully put together library so far.\n\nJonathan\n"},{"id":"406813","messageId":"nycvar.QRO.7.76.6.2010021342200.50@tvgsbejvaqbjf.bet","threadId":"54245","inReplyTo":"20201002040200.GC3252492@google.com","subject":"Re: [PATCH v2 03/13] vcxproj: adjust for the reftable changes","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-02T11:43:05Z","receivedAt":"2020-10-02T11:43:16Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jonathan,\n\nOn Thu, 1 Oct 2020, Jonathan Nieder wrote:\n\n> Han-Wen Nienhuys wrote:\n>\n> > From: Johannes Schindelin \n> >\n> > This allows Git to be compiled via Visual Studio again after integrating\n> > the `hn/reftable` branch.\n>\n> nit: This branch name is no longer meaningful to the primary audience\n> for the commit message (people discovering this commit in git history\n> later).\n>\n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > config.mak.uname | 2 +-\n> > contrib/buildsystems/Generators/Vcxproj.pm | 11 ++++++++++-\n> > 2 files changed, 11 insertions(+), 2 deletions(-)\n>\n> Can this be squashed into or put immediately after patch 5 which\n> introduces the Makefile?\n\nI never intended this to be a stand-alone patch. So yes, I would be very\nmuch in favor of squashing these changes into the appropriate commit.\n\nCiao,\nDscho\n"},{"id":"406818","messageId":"nycvar.QRO.7.76.6.2010021555290.50@tvgsbejvaqbjf.bet","threadId":"54245","inReplyTo":"4e38db7f48295c4e6d729880f5f1a63b0f90274e.1601568664.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 12/13] reftable: rest of library","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-02T13:57:45Z","receivedAt":"2020-10-02T13:57:54Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Han-Wen,\n\nOn Thu, 1 Oct 2020, Han-Wen Nienhuys via GitGitGadget wrote:\n\n> From: Han-Wen Nienhuys \n>\n> This will be further split up once preceding commits have passed review.\n\nBefore you further split it up, I encourage you to include these patches\nwithout which the CI builds will continue to fail (Junio, could I ask you\nto either cherry-pick them from https://github.com/git-for-windows/git's\nshears/seen branch, or apply them from the mbox?):\n\n-- snipsnap --\nFrom e485e006f34922439f2e971a1c5c38b8ca56c011 Mon Sep 17 00:00:00 2001\nFrom: Johannes Schindelin \nDate: Wed, 30 Sep 2020 14:46:59 +0200\nSubject: [PATCH 1/3] fixup??? reftable: rest of library\n\n\tstruct abc x = {}\n\nis a GNUism.\n\nSigned-off-by: Johannes Schindelin \n---\n reftable/dump.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/reftable/dump.c b/reftable/dump.c\nindex ed09f2e2f94..233f0434e39 100644\n--- a/reftable/dump.c\n+++ b/reftable/dump.c\n@@ -75,7 +75,7 @@ static int dump_table(const char *tablename)\n static int compact_stack(const char *stackdir)\n {\n \tstruct reftable_stack *stack = NULL;\n-\tstruct reftable_write_options cfg = {};\n+\tstruct reftable_write_options cfg = { 0 };\n\n \tint err = reftable_new_stack(&stack, stackdir, cfg);\n \tif (err < 0)\n@@ -94,7 +94,7 @@ static int compact_stack(const char *stackdir)\n static int dump_stack(const char *stackdir)\n {\n \tstruct reftable_stack *stack = NULL;\n-\tstruct reftable_write_options cfg = {};\n+\tstruct reftable_write_options cfg = { 0 };\n \tstruct reftable_iterator it = { NULL };\n \tstruct reftable_ref_record ref = { NULL };\n \tstruct reftable_log_record log = { NULL };\n--\n2.28.0.windows.1.18.g5300e52e185\n\n\nFrom d5faa818a1bc00016e310d27602551127db620fb Mon Sep 17 00:00:00 2001\nFrom: Johannes Schindelin \nDate: Wed, 30 Sep 2020 14:55:28 +0200\nSubject: [PATCH 2/3] fixup??? reftable: rest of library\n\n0-sized arrays are actually not portable.\n\nSigned-off-by: Johannes Schindelin \n---\n reftable/stack_test.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 534463829cb..e863cc3c0a2 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -572,11 +572,11 @@ static void test_sizes_to_segments(void)\n\n static void test_sizes_to_segments_empty(void)\n {\n-\tuint64_t sizes[0];\n+\tuint64_t sizes[1];\n\n \tint seglen = 0;\n \tstruct segment *segs =\n-\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\t\tsizes_to_segments(&seglen, sizes, 0);\n \tassert(seglen == 0);\n \treftable_free(segs);\n }\n--\n2.28.0.windows.1.18.g5300e52e185\n\n\nFrom d446e1a7354c676d60114b50ba96a6ea083441ae Mon Sep 17 00:00:00 2001\nFrom: Johannes Schindelin \nDate: Wed, 30 Sep 2020 15:19:28 +0200\nSubject: [PATCH 3/3] fixup??? reftable: rest of library\n\nAvoid using `getopt()`: it might be POSIX, but Git's audience is much\nlarger than POSIX. MSVC, for example, does not support `getopt()`.\n\nSigned-off-by: Johannes Schindelin \n---\n reftable/dump.c | 32 +++++++++++++-------------------\n 1 file changed, 13 insertions(+), 19 deletions(-)\n\ndiff --git a/reftable/dump.c b/reftable/dump.c\nindex 233f0434e39..b63c9fe9e81 100644\n--- a/reftable/dump.c\n+++ b/reftable/dump.c\n@@ -160,40 +160,34 @@ static void print_help(void)\n int reftable_dump_main(int argc, char *const *argv)\n {\n \tint err = 0;\n-\tint opt;\n \tint opt_dump_table = 0;\n \tint opt_dump_stack = 0;\n \tint opt_compact = 0;\n-\tconst char *arg = NULL;\n-\twhile ((opt = getopt(argc, argv, \"2chts\")) != -1) {\n-\t\tswitch (opt) {\n-\t\tcase '2':\n-\t\t\thash_id = 0x73323536;\n+\tconst char *arg = NULL, *argv0 = argv[0];\n+\n+\tfor (; argc > 1; argv++, argc--)\n+\t\tif (*argv[1] != '-')\n \t\t\tbreak;\n-\t\tcase 't':\n+\t\telse if (!strcmp(\"-2\", argv[1]))\n+\t\t\thash_id = 0x73323536;\n+\t\telse if (!strcmp(\"-t\", argv[1]))\n \t\t\topt_dump_table = 1;\n-\t\t\tbreak;\n-\t\tcase 's':\n+\t\telse if (!strcmp(\"-s\", argv[1]))\n \t\t\topt_dump_stack = 1;\n-\t\t\tbreak;\n-\t\tcase 'c':\n+\t\telse if (!strcmp(\"-c\", argv[1]))\n \t\t\topt_compact = 1;\n-\t\t\tbreak;\n-\t\tcase '?':\n-\t\tcase 'h':\n+\t\telse if (!strcmp(\"-?\", argv[1]) || !strcmp(\"-h\", argv[1])) {\n \t\t\tprint_help();\n \t\t\treturn 2;\n-\t\t\tbreak;\n \t\t}\n-\t}\n\n-\tif (argv[optind] == NULL) {\n+\tif (argc != 2) {\n \t\tfprintf(stderr, \"need argument\\n\");\n \t\tprint_help();\n \t\treturn 2;\n \t}\n\n-\targ = argv[optind];\n+\targ = argv[1];\n\n \tif (opt_dump_table) {\n \t\terr = dump_table(arg);\n@@ -204,7 +198,7 @@ int reftable_dump_main(int argc, char *const *argv)\n \t}\n\n \tif (err < 0) {\n-\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv[0], arg,\n+\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv0, arg,\n \t\t\treftable_error_str(err));\n \t\treturn 1;\n \t}\n--\n2.28.0.windows.1.18.g5300e52e185\n\n"},{"id":"406819","messageId":"nycvar.QRO.7.76.6.2010021557570.50@tvgsbejvaqbjf.bet","threadId":"54245","inReplyTo":"4190da597e65bce072fa3c37c9410a56def4b489.1601568663.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-02T14:01:50Z","receivedAt":"2020-10-02T14:02:05Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Han-Wen,\n\nOn Thu, 1 Oct 2020, Han-Wen Nienhuys via GitGitGadget wrote:\n\n> From: Han-Wen Nienhuys \n>\n> This commit provides basic utility classes for the reftable library.\n>\n> Since the reftable library must compile standalone, there may be some overlap\n> with git-core utility functions.\n\nMy position on this idea to duplicate functionality in order to somehow\npretend that the reftable code is independent of Git's source code has not\nchanged.\n\nBe that as it may, the CI build failures impacted my notifications'\nsignal/noise ratio too much, so here goes (Junio, I would be delighted if\nyou could apply this on top of your branch):\n\n-- snipsnap --\nFrom 1698c3450b5e47927c2e8fe35cad2634963bad89 Mon Sep 17 00:00:00 2001\nFrom: Johannes Schindelin \nDate: Wed, 30 Sep 2020 16:53:53 +0200\nSubject: [PATCH] fixup??? reftable: utility functions\n\nLet's not forget our CMake configuration.\n\nSigned-off-by: Johannes Schindelin \n---\n contrib/buildsystems/CMakeLists.txt | 14 ++++++++++++--\n 1 file changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 1bd53c4ad51..284bf4402d7 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -591,6 +591,12 @@ parse_makefile_for_sources(libxdiff_SOURCES \"XDIFF_OBJS\")\n list(TRANSFORM libxdiff_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n add_library(xdiff STATIC ${libxdiff_SOURCES})\n\n+#reftable\n+parse_makefile_for_sources(reftable_SOURCES \"REFTABLE_OBJS\")\n+\n+list(TRANSFORM reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n+add_library(reftable STATIC ${reftable_SOURCES})\n+\n if(WIN32)\n \tif(NOT MSVC)#use windres when compiling with gcc and clang\n \t\tadd_custom_command(OUTPUT ${CMAKE_BINARY_DIR}/git.res\n@@ -613,7 +619,7 @@ endif()\n #link all required libraries to common-main\n add_library(common-main OBJECT ${CMAKE_SOURCE_DIR}/common-main.c)\n\n-target_link_libraries(common-main libgit xdiff ${ZLIB_LIBRARIES})\n+target_link_libraries(common-main libgit xdiff reftable ${ZLIB_LIBRARIES})\n if(Intl_FOUND)\n \ttarget_link_libraries(common-main ${Intl_LIBRARIES})\n endif()\n@@ -848,11 +854,15 @@ if(BUILD_TESTING)\n add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)\n target_link_libraries(test-fake-ssh common-main)\n\n+#reftable-tests\n+parse_makefile_for_sources(test-reftable_SOURCES \"REFTABLE_TEST_OBJS\")\n+list(TRANSFORM test-reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n+\n #test-tool\n parse_makefile_for_sources(test-tool_SOURCES \"TEST_BUILTINS_OBJS\")\n\n list(TRANSFORM test-tool_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/t/helper/\")\n-add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES})\n+add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES} ${test-reftable_SOURCES})\n target_link_libraries(test-tool common-main)\n\n set_target_properties(test-fake-ssh test-tool\n--\n2.28.0.windows.1.18.g5300e52e185\n\n"},{"id":"406831","messageId":"xmqqzh548sid.fsf@gitster.c.googlers.com","threadId":"54245","inReplyTo":"nycvar.QRO.7.76.6.2010021555290.50@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 12/13] reftable: rest of library","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-02T17:08:58Z","receivedAt":"2020-10-02T17:09:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> Subject: [PATCH 1/3] fixup??? reftable: rest of library\n\nThis is unambiguously good.\n\n> Subject: [PATCH 2/3] fixup??? reftable: rest of library\n>\n> 0-sized arrays are actually not portable.\n> ...\n> static void test_sizes_to_segments_empty(void)\n> {\n> -\tuint64_t sizes[0];\n> +\tuint64_t sizes[1];\n>\n> \tint seglen = 0;\n> \tstruct segment *segs =\n> -\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n> +\t\tsizes_to_segments(&seglen, sizes, 0);\n> \tassert(seglen == 0);\n> \treftable_free(segs);\n\nQuestion to Han-Wen.\n\nIt is unclear what this test wants to test. Do we even need sizes[]\narray if we know we are passing a hardcoded 0? IOW, I would\nunderstand if the test were\n\n\tsizes_to_segments(&seglen, NULL, 0);\n\nto ensure that sizes_to_segments do not even attempt to look at sizes[]\narray when the number of elements is 0.\n\n> Subject: [PATCH 3/3] fixup??? reftable: rest of library\n>\n> Avoid using `getopt()`: it might be POSIX, but Git's audience is much\n> larger than POSIX. MSVC, for example, does not support `getopt()`.\n\nEither that, or we could use parse-options(). I do not care either\nway, as this seems to be purely for debugging?\n\n"},{"id":"406841","messageId":"xmqqh7rc8idd.fsf@gitster.c.googlers.com","threadId":"54245","inReplyTo":"nycvar.QRO.7.76.6.2010021557570.50@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-02T20:47:58Z","receivedAt":"2020-10-02T20:48:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n>> This commit provides basic utility classes for the reftable library.\n>>\n>> Since the reftable library must compile standalone, there may be some overlap\n>> with git-core utility functions.\n>\n> My position on this idea to duplicate functionality in order to somehow\n> pretend that the reftable code is independent of Git's source code has not\n> changed.\n\nThe above may be sufficient between you and Han-Wen, and a few\nselected others who still remember, but please do not forget that we\nare collaborating in the open. It would help those who are learning\nopen source interactions by watching from sidelines, if you spelled\nout what your position is or at least left a pointer to a previous\ndiscussion.\n\nFWIW, I think it is a mistake to try to make this directory so that\nit can be lifted out of our code base and used independently, as it\nhas to create unnecessary friction to the code when used here. It\nis not like other code that we are not their primary intended\naudience and we simply \"borrow\" from them (e.g. xdiff/ & sha1dc/).\n\nThe previous paragraph agrees with my guess of your position, but\nperhaps you have something else in mind. I dunno.\n\n> Be that as it may, the CI build failures impacted my notifications'\n> signal/noise ratio too much, so here goes (Junio, I would be delighted if\n> you could apply this on top of your branch):\n\nQuite honestly, this close to the first preview release for 2.29,\nI'd rather drop this round from 'seen' and expect an updated topic,\nthan piling fixup on top of fixups.\n\nThanks.\n"},{"id":"406858","messageId":"nycvar.QRO.7.76.6.2010030939020.50@tvgsbejvaqbjf.bet","threadId":"54245","inReplyTo":"xmqqh7rc8idd.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-03T08:07:18Z","receivedAt":"2020-10-03T08:07:34Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Fri, 2 Oct 2020, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n>\n> >> This commit provides basic utility classes for the reftable library.\n> >>\n> >> Since the reftable library must compile standalone, there may be some overlap\n> >> with git-core utility functions.\n> >\n> > My position on this idea to duplicate functionality in order to somehow\n> > pretend that the reftable code is independent of Git's source code has not\n> > changed.\n>\n> The above may be sufficient between you and Han-Wen, and a few\n> selected others who still remember, but please do not forget that we\n> are collaborating in the open. It would help those who are learning\n> open source interactions by watching from sidelines, if you spelled\n> out what your position is or at least left a pointer to a previous\n> discussion.\n\nYou're right, sorry.\n\n> FWIW, I think it is a mistake to try to make this directory so that\n> it can be lifted out of our code base and used independently, as it\n> has to create unnecessary friction to the code when used here. It\n> is not like other code that we are not their primary intended\n> audience and we simply \"borrow\" from them (e.g. xdiff/ & sha1dc/).\n>\n> The previous paragraph agrees with my guess of your position, but\n> perhaps you have something else in mind. I dunno.\n\nYes, you were spot on.\n\nI am rather strongly opposed to introducing duplicated functionality.\nRather than spending time on finding half a dozen links to conversations\nin the past, let me try to make the case afresh:\n\nI have a very concrete example of the undesirable consequences: the\ncommit-graph breakage Stolee and I debugged yesterday, where a bug hid in\n`in_merge_bases_many()` for two years, hidden by the fact that there was\n_duplicate logic_ in `get_reachable_subset()`.\n\nPretty much _all_ of the code in `reftable/` is _very_ new, has not seen\nany real-world testing, and as such cannot really be trusted, in\nparticular the code that duplicates functionality already present in\nlibgit.a. Note that I do not claim that Han-Wen cannot be trusted: to the\ncontrary, I trust him very much (we have a history going all the way back\nto Lilypond, the musical typesetter). The code, however, cannot be\ntrusted, and needs to be reviewed and tested in practice.\n\nThe simple fact is that even the thorough review that the commit-graph\npatches received did not prevent the `min_generation`/`max_generation` bug\nin `in_merge_bases_many()` from entering the code base, and if we had not\nhad _another_ function doing very, very similar things, that bug would\nmost likely not have lingered for _this_ long.\n\nLikewise for the reimplementation of the convenient functionality like\n`strbuf`. This kind of duplication (in the case of `struct strbuf`, quite\n_literally_: there are now confusingly _two_ `strbuf.h` files that even\ntry to implement the _exact same_ API) is _prone_ to lead to stale, or\nlong undiscovered, bugs in one of the duplicated implementations.\n\nWhich means that we're likely to introduce bugs with those new functions\nintroduced in `reftable/`. For the reftable functionality, that has to be\naccepted. For functions that do the same as equivalents in libgit.a, we do\nnot have to accept it.\n\nIn any case, duplicated functionality is a maintenance burden, and\nespecially in this case, an unnecessary one at that: unlike xdiff or\ncompat/regex/, the reftable functionality is co-dependent with `libgit.a`.\nYes, you can implement libreftable as a stand-alone library (including\nduplicated functionality from libgit.a, as I pointed out), but you _cannot\nuse it independently of Git_.\n\nThe reftable concerns _Git_ refs.\n\nThis is not a compressor that could come in handy in a messenger app, or a\ndiff engine that could potentially help with a semantic merging tool.\n\nIt is code whose entire purpose is to manage boatloads of Git refs\nefficiently. Therefore, I see no convincing reason to insist on\nmaintaining this code outside of git/git.\n\nIt would make more sense to maintain parse-options or strbuf or\nt/test-lib.sh outside of git/git (because at least that functionality is\nindependent enough of Git to be potentially useful to other projects) and\nwe don't. We maintain them inside git/git, and for good reason: it makes\ndevelopment and maintenance a hell of a lot easier.\n\nAnd once we agree that reftable is intended to be so integral a part of\nGit that it should absolutely have its home inside git/git, all of that\nduplicated functionality (with almost totally untested implementations, at\nleast when it comes to \"battle testing in production\") can totally\nevaporate and does not need to worry us any more.\n\nAnd then we can start to spend reviewers' time on the actual code\nrevolving around the actual reftables rather than having to bother with\nreviewing, say, a close, but non-identical `strbuf` that might even have\n\"borrowed\" parts of the `strbuf` code and put it under a more permissive\nlicense (I don't know, and I don't want to know, that's part of the reason\nwhy I don't want to review the reftable patches in their current form).\n\nIn other words, we can then finally start to actually review the reftable\ncode on its merits of teaching Git a valuable new feature.\n\n> > Be that as it may, the CI build failures impacted my notifications'\n> > signal/noise ratio too much, so here goes (Junio, I would be delighted if\n> > you could apply this on top of your branch):\n>\n> Quite honestly, this close to the first preview release for 2.29,\n> I'd rather drop this round from 'seen' and expect an updated topic,\n> than piling fixup on top of fixups.\n\nSounds good to me.\n\nCiao,\nDscho\n"},{"id":"406950","messageId":"nycvar.QRO.7.76.6.2010042036221.50@tvgsbejvaqbjf.bet","threadId":"54245","inReplyTo":"xmqqzh548sid.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 12/13] reftable: rest of library","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-04T18:39:14Z","receivedAt":"2020-10-05T12:29:41Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Fri, 2 Oct 2020, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n>\n> > Subject: [PATCH 3/3] fixup??? reftable: rest of library\n> >\n> > Avoid using `getopt()`: it might be POSIX, but Git's audience is much\n> > larger than POSIX. MSVC, for example, does not support `getopt()`.\n>\n> Either that, or we could use parse-options(). I do not care either\n> way, as this seems to be purely for debugging?\n\nI actually wanted to use `parse_options()`, but due to the way `reftable/`\ninsists on being completely separate from `libgit.a`, it is not really\ntrivial, as `#include \"parse-options.h\"` fails (because\n`git-compat-util.h` was not included, and IIRC including _that_ resulted\nin more compile errors, at which stage I opted for a more minimal\nsolution).\n\nSo yeah, I would strongly expect `reftable/` to become considerably\nsimpler once it starts using `libgit.a`'s goodies.\n\nCiao,\nDscho\n"},{"id":"407114","messageId":"20201008014124.1410535-1-jonathantanmy@google.com","threadId":"54245","inReplyTo":"5d1b946ab5473504e9599d90c0feda407a179a05.1601568663.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 02/13] reftable: define the public API","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-10-08T01:41:24Z","receivedAt":"2020-10-08T01:42:32Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"My initial impression is that this public API is much larger than I\nwould expect from reading the reftable spec. Let me look at it in\ndetail...\n\n> +/****************************************************************\n> + Basic data types\n> +\n> + Reftables store the state of each ref in struct reftable_ref_record, and they\n> + store a sequence of reflog updates in struct reftable_log_record.\n> + ****************************************************************/\n> +\n> +/* reftable_ref_record holds a ref database entry target_value */\n> +struct reftable_ref_record {\n> +\tchar *refname; /* Name of the ref, malloced. */\n> +\tuint64_t update_index; /* Logical timestamp at which this value is\n> +\t\t\t\t written */\n> +\tuint8_t *value; /* SHA1, or NULL. malloced. */\n> +\tuint8_t *target_value; /* peeled annotated tag, or NULL. malloced. */\n> +\tchar *target; /* symref, or NULL. malloced. */\n> +};\n\nA typical user would just want to do refname->hash, refname->log, or possibly\nhash->refname lookups, so a ref record would be meant for low-level\nusers. I can't think of a typical use case, but even if such existed, I\nwouldn't expect this API - in particular, the refname is not written as\nsuch on disk (it is composed from a prefix and a suffix, if I remember\ncorrectly), so I would expect a function that one can call to write the\nrefname, but not for it to appear in a data structure.\n\n> +/* returns whether 'ref' represents a deletion */\n> +int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);\n\nThis looks unorthogonal - looking at the spec, the \"value type\" can\nrepresent a deletion, one object name (value of the ref), two object\nnames (value of the ref + peeled target), or symbolic reference. If\nwe're willing to allocate memory for a ref record, I think we can afford\nthe extra 4 bytes (maybe 8) and use a tagged union instead.\n\n> +/* prints a reftable_ref_record onto stdout */\n> +void reftable_ref_record_print(struct reftable_ref_record *ref,\n> +\t\t\t uint32_t hash_id);\n\nNot sure if this belongs here.\n\n> +/* frees and nulls all pointer values. */\n> +void reftable_ref_record_clear(struct reftable_ref_record *ref);\n\nUsually we call this \"release\" in Git.\n\n> +/* returns whether two reftable_ref_records are the same */\n> +int reftable_ref_record_equal(struct reftable_ref_record *a,\n> +\t\t\t struct reftable_ref_record *b, int hash_size);\n\nNot sure what this will be used for.\n\n> +/* reftable_log_record holds a reflog entry */\n> +struct reftable_log_record {\n> +\tchar *refname;\n> +\tuint64_t update_index; /* logical timestamp of a transactional update.\n> +\t\t\t\t*/\n> +\tuint8_t *new_hash;\n> +\tuint8_t *old_hash;\n> +\tchar *name;\n> +\tchar *email;\n> +\tuint64_t time;\n> +\tint16_t tz_offset;\n> +\tchar *message;\n> +};\n\nMostly same comments as above.\n\n[snip]\n\n> +/****************************************************************\n> + Error handling\n> +\n> + Error are signaled with negative integer return values. 0 means success.\n> + ****************************************************************/\n\nI didn't look at this section deeply, but this looks OK at first glance.\n\n> +/****************************************************************\n> + Writing\n> +\n> + Writing single reftables\n> + ****************************************************************/\n\nOK - writing a single reftable makes sense.\n\n> +/* reftable_write_options sets options for writing a single reftable. */\n> +struct reftable_write_options {\n> +\t/* boolean: do not pad out blocks to block size. */\n> +\tint unpadded;\n> +\n> +\t/* the blocksize. Should be less than 2^24. */\n> +\tuint32_t block_size;\n> +\n> +\t/* boolean: do not generate a SHA1 => ref index. */\n> +\tint skip_index_objects;\n> +\n> +\t/* how often to write complete keys in each block. */\n> +\tint restart_interval;\n> +\n> +\t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n> +\t * Defaults to SHA1 if unset\n> +\t */\n> +\tuint32_t hash_id;\n> +\n> +\t/* boolean: do not check ref names for validity or dir/file conflicts.\n> +\t */\n> +\tint skip_name_check;\n> +\n> +\t/* boolean: copy log messages exactly. If unset, check that the message\n> +\t * is a single line, and add '\\n' if missing.\n> +\t */\n> +\tint exact_log_message;\n> +};\n\nI'm not sure we need so many options, but nothing jumps out to me\noff-hand. Some minor things - \"hash_id\" should probably be an enum and\nthe booleans should be \"unsigned variable_name : 1\".\n\n[snip statistics]\n\nStatistics look useful.\n\n> +/* reftable_new_writer creates a new writer */\n> +struct reftable_writer *\n> +reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n> +\t\t void *writer_arg, struct reftable_write_options *opts);\n> +\n> +/* write to a file descriptor. fdp should be an int* pointing to the fd. */\n> +int reftable_fd_write(void *fdp, const void *data, size_t size);\n\nNot sure if we'll need to write to things other than an fd, but if we\ndo, this makes sense.\n\n> +/* Set the range of update indices for the records we will add. When\n> + writing a table into a stack, the min should be at least\n> + reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned.\n> +\n> + For transactional updates, typically min==max. When converting an existing\n> + ref database into a single reftable, this would be a range of update-index\n> + timestamps.\n> + */\n> +void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n> +\t\t\t\tuint64_t max);\n\nThis seems to be here because we want to write the file in a single\npass, and the update index maximum and minimum appear in the header. If\nwe were allowed to seek while writing, could the update index maximum\nand minimum be deduced instead? That might be a compelling reason to\nonly support FDs, and require that the FDs be seekable.\n\n> +/* adds a reftable_ref_record. Must be called in ascending\n> + order. The update_index must be within the limits set by\n> + reftable_writer_set_limits(), or REFTABLE_API_ERROR is returned.\n> +\n> + It is an error to write a ref record after a log record.\n> + */\n> +int reftable_writer_add_ref(struct reftable_writer *w,\n> +\t\t\t struct reftable_ref_record *ref);\n> +\n> +/* Convenience function to add multiple refs. Will sort the refs by\n> + name before adding. */\n> +int reftable_writer_add_refs(struct reftable_writer *w,\n> +\t\t\t struct reftable_ref_record *refs, int n);\n\nSince refs is an array of objects and not an array of pointers, these\ntwo functions could be combined - the first is the same as calling the\nsecond with n=1.\n\nAlso, ascending order of what?\n\nThe user will also need to know where to get the update index from - I\npresume that this will be the maximum update index of any record with\nthe given refname + 1.\n\n[snip similar functions for log]\n\n> +/* reftable_writer_close finalizes the reftable. The writer is retained so\n> + * statistics can be inspected. */\n> +int reftable_writer_close(struct reftable_writer *w);\n> +\n> +/* writer_stats returns the statistics on the reftable being written.\n> +\n> + This struct becomes invalid when the writer is freed.\n> + */\n> +const struct reftable_stats *writer_stats(struct reftable_writer *w);\n> +\n> +/* reftable_writer_free deallocates memory for the writer */\n> +void reftable_writer_free(struct reftable_writer *w);\n\nOK.\n\n> +/****************************************************************\n> + * ITERATING\n> + ****************************************************************/\n\n[snip]\n\n> +/* reads the next reftable_ref_record. Returns < 0 for error, 0 for OK and > 0:\n> + end of iteration.\n> +*/\n> +int reftable_iterator_next_ref(struct reftable_iterator *it,\n> +\t\t\t struct reftable_ref_record *ref);\n> +\n> +/* reads the next reftable_log_record. Returns < 0 for error, 0 for OK and > 0:\n> + end of iteration.\n> +*/\n> +int reftable_iterator_next_log(struct reftable_iterator *it,\n> +\t\t\t struct reftable_log_record *log);\n\nFrom my recollection, in Git, we typically use foreach functions with a\ncallback that is invoked once for each result. I think that's preferable\nto this approach.\n\n> +/****************************************************************\n> + Reading single tables\n> +\n> + The follow routines are for reading single files. For an application-level\n> + interface, skip ahead to struct reftable_merged_table and struct\n> + reftable_stack.\n> + ****************************************************************/\n\nI think that this whole section could be skipped - if we really wanted\nto read a single table, we could just use the merged-table interface\nwith one table.\n\n[snip]\n\n> +/****************************************************************\n> + Merged tables\n> +\n> + A ref database kept in a sequence of table files. The merged_table presents a\n> + unified view to reading (seeking, iterating) a sequence of immutable tables.\n> + ****************************************************************/\n> +\n> +/* A merged table is implements seeking/iterating over a stack of tables. */\n> +struct reftable_merged_table;\n> +\n> +/* A generic reftable; see below. */\n> +struct reftable_table;\n\nWhy would we need to see an individual reftable? Could we just represent\na merged table as a virtual concatenation of blocks (as if all the\nblocks were in the same file)?\n\n> +/* reftable_new_merged_table creates a new merged table. It takes ownership of\n> + the stack array.\n> +*/\n> +int reftable_new_merged_table(struct reftable_merged_table **dest,\n> +\t\t\t struct reftable_table *stack, int n,\n> +\t\t\t uint32_t hash_id);\n\nI presume this would be used for things like compacting a few reftables\ntogether? In which case, I would expect this function to just take a\nlist of filenames.\n\n> +/* returns an iterator positioned just before 'name' */\n> +int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n> +\t\t\t\t struct reftable_iterator *it,\n> +\t\t\t\t const char *name);\n> +\n> +/* returns an iterator for log entry, at given update_index */\n> +int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n> +\t\t\t\t struct reftable_iterator *it,\n> +\t\t\t\t const char *name, uint64_t update_index);\n> +\n> +/* like reftable_merged_table_seek_log_at but look for the newest entry. */\n> +int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n> +\t\t\t\t struct reftable_iterator *it,\n> +\t\t\t\t const char *name);\n\nWhy do iterators need to be precisely positioned if this merged table is\nimmutable?\n\n> +/* returns the max update_index covered by this merged table. */\n> +uint64_t\n> +reftable_merged_table_max_update_index(struct reftable_merged_table *mt);\n> +\n> +/* returns the min update_index covered by this merged table. */\n> +uint64_t\n> +reftable_merged_table_min_update_index(struct reftable_merged_table *mt);\n> +\n> +/* releases memory for the merged_table */\n> +void reftable_merged_table_free(struct reftable_merged_table *m);\n> +\n> +/* return the hash ID of the merged table. */\n> +uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m);\n\nOK - makes sense.\n\n> +/****************************************************************\n> + Generic tables\n> +\n> + A unified API for reading tables, either merged tables, or single readers.\n> + ****************************************************************/\n\nAgain, we could just use the merged table API.\n\n[snip]\n\n> +/* convenience function to read a single ref. Returns < 0 for error, 0\n> + for success, and 1 if ref not found. */\n> +int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n> +\t\t\t struct reftable_ref_record *ref);\n\nI presume this returns the most up-to-date record in the (possibly\nmerged) table? So we can just read the hash off the record to know what\nthis ref points to.\n\n> +/****************************************************************\n> + Mutable ref database\n> +\n> + The stack presents an interface to a mutable sequence of reftables.\n> + ****************************************************************/\n\nSo I would expect ref mutations to be done by opening the existing\nreftables as a merged table, open a reftable writer, write all the\nchanges that need to be written, and update the tables.list file...\n\n> +/* a stack is a stack of reftables, which can be mutated by pushing a table to\n> + * the top of the stack */\n> +struct reftable_stack;\n> +\n> +/* open a new reftable stack. The tables along with the table list will be\n> + stored in 'dir'. Typically, this should be .git/reftables.\n> +*/\n> +int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n> +\t\t struct reftable_write_options config);\n> +\n> +/* returns the update_index at which a next table should be written. */\n> +uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\n\n...so I don't see why we need a stack.\n\n> +/* holds a transaction to add tables at the top of a stack. */\n> +struct reftable_addition;\n> +\n> +/*\n> + returns a new transaction to add reftables to the given stack. As a side\n> + effect, the ref database is locked.\n> +*/\n> +int reftable_stack_new_addition(struct reftable_addition **dest,\n> +\t\t\t\tstruct reftable_stack *st);\n> +\n> +/* Adds a reftable to transaction. */\n> +int reftable_addition_add(struct reftable_addition *add,\n> +\t\t\t int (*write_table)(struct reftable_writer *wr,\n> +\t\t\t\t\t void *arg),\n> +\t\t\t void *arg);\n> +\n> +/* Commits the transaction, releasing the lock. */\n> +int reftable_addition_commit(struct reftable_addition *add);\n> +\n> +/* Release all non-committed data from the transaction, and deallocate the\n> + transaction. Releases the lock if held. */\n> +void reftable_addition_destroy(struct reftable_addition *add);\n\nWe do need a transaction to write the new tables.list and then\natomically update the repository with it, but I don't think we need one\njust to add a reftable to the stack.\n\n[snip rest of stack functions]\n\n> +/* Policy for expiring reflog entries. */\n> +struct reftable_log_expiry_config {\n> +\t/* Drop entries older than this timestamp */\n> +\tuint64_t time;\n> +\n> +\t/* Drop older entries */\n> +\tuint64_t min_update_index;\n> +};\n> +\n> +/* compacts all reftables into a giant table. Expire reflog entries if config is\n> + * non-NULL */\n> +int reftable_stack_compact_all(struct reftable_stack *st,\n> +\t\t\t struct reftable_log_expiry_config *config);\n\nAh, the stack is used for compacting as well. I don't think compacting\nbelongs here though - it should be its own thing that can make use of\nthe merged-table reader and the single-table writer.\n\n[snip]\n\n> +/* convenience function to read a single ref. Returns < 0 for error, 0\n> + for success, and 1 if ref not found. */\n> +int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n> +\t\t\t struct reftable_ref_record *ref);\n> +\n> +/* convenience function to read a single log. Returns < 0 for error, 0\n> + for success, and 1 if ref not found. */\n> +int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n> +\t\t\t struct reftable_log_record *log);\n\nOther things that belong to the merged table reader instead of the\nstack.\n\nSome things that I think are missing:\n\n - foreach all latest record of all refs (in any order) (I see some\n functions above that can set the position of the iterator, but I\n don't think that the iterator skips over irrelevant refs? So we can't\n use it for iterating over all refs if we don't care about the history\n of refs)\n - foreach all log entries of one ref (or all refs)\n\nIn summary, I would have expected a mechanism to read multiple (possibly\none) reftable and perform queries on it, a mechanism to write a single\nreftable, and some functions to update tables.list.\n"},{"id":"407115","messageId":"20201008014512.1413826-1-jonathantanmy@google.com","threadId":"54245","inReplyTo":"b94c5f5c6120347fd97bcc2dcc187fc86a802dff.1601568663.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 04/13] reftable: add a barebones unittest framework","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-10-08T01:45:12Z","receivedAt":"2020-10-08T01:45:20Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> From: Han-Wen Nienhuys \n> \n> Signed-off-by: Han-Wen Nienhuys \n> ---\n> reftable/test_framework.c | 68 +++++++++++++++++++++++++++++++++++++++\n> reftable/test_framework.h | 59 +++++++++++++++++++++++++++++++++\n\nEven if reftable will be a project maintained by the Git project but\nindependent enough to be used by other projects, I don't think it's\nworth creating a separate test framework. E.g. as far as I can tell,\nwhen we import sha1dc, we don't import any tests, so I don't think they\nneed to import tests either - they can run the tests in the Git repo\nthemselves if they need to.\n"},{"id":"407116","messageId":"20201008014855.1416580-1-jonathantanmy@google.com","threadId":"54245","inReplyTo":"4190da597e65bce072fa3c37c9410a56def4b489.1601568663.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-10-08T01:48:55Z","receivedAt":"2020-10-08T01:48:59Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> From: Han-Wen Nienhuys \n> \n> This commit provides basic utility classes for the reftable library.\n> \n> Since the reftable library must compile standalone, there may be some overlap\n> with git-core utility functions.\n> \n> Signed-off-by: Han-Wen Nienhuys \n> ---\n> Makefile | 26 ++++++-\n> reftable/basics.c | 131 +++++++++++++++++++++++++++++++++\n> reftable/basics.h | 48 +++++++++++++\n> reftable/blocksource.c | 148 ++++++++++++++++++++++++++++++++++++++\n> reftable/blocksource.h | 22 ++++++\n> reftable/compat.c | 110 ++++++++++++++++++++++++++++\n> reftable/compat.h | 48 +++++++++++++\n> reftable/publicbasics.c | 100 ++++++++++++++++++++++++++\n> reftable/reftable-tests.h | 22 ++++++\n> reftable/strbuf.c | 142 ++++++++++++++++++++++++++++++++++++\n> reftable/strbuf.h | 80 +++++++++++++++++++++\n> reftable/strbuf_test.c | 37 ++++++++++\n> reftable/system.h | 51 +++++++++++++\n> t/helper/test-reftable.c | 8 +++\n> t/helper/test-tool.c | 1 +\n> t/helper/test-tool.h | 1 +\n\nI think duplicating things like strbuf is an unnecessary burden if Git\nis to maintain this library. Something like \"reftable will only import\ngit-compat-util.h and strbuf.h, and any project that wants to use\nreftable must make sure that these functions and data structures are\navailable\" would be more plausible.\n"},{"id":"407188","messageId":"20201008223123.GC4095761@google.com","threadId":"54245","inReplyTo":"20201008014512.1413826-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 04/13] reftable: add a barebones unittest framework","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2020-10-08T22:31:23Z","receivedAt":"2020-10-08T22:31:33Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2020.10.07 18:45, Jonathan Tan wrote:\n> > From: Han-Wen Nienhuys \n> > \n> > Signed-off-by: Han-Wen Nienhuys \n> > ---\n> > reftable/test_framework.c | 68 +++++++++++++++++++++++++++++++++++++++\n> > reftable/test_framework.h | 59 +++++++++++++++++++++++++++++++++\n> \n> Even if reftable will be a project maintained by the Git project but\n> independent enough to be used by other projects, I don't think it's\n> worth creating a separate test framework. E.g. as far as I can tell,\n> when we import sha1dc, we don't import any tests, so I don't think they\n> need to import tests either - they can run the tests in the Git repo\n> themselves if they need to.\n\nI agree with Jonathan here, I think we should stick with the existing\ntest framework.\n"},{"id":"407251","messageId":"20201009211300.GA2774782@google.com","threadId":"54245","inReplyTo":"20201002035851.GB3252492@google.com","subject":"Re: [PATCH v2 02/13] reftable: define the public API","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-10-09T21:13:00Z","receivedAt":"2020-10-09T21:13:08Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Oct 01, 2020 at 08:58:51PM -0700, Jonathan Nieder wrote:\n> \n> Hi,\n> \n> Han-Wen Nienhuys wrote:\n> \n> > reftable/reftable.h | 585 ++++++++++++++++++++++++++++++++++++++++++++\n> > 1 file changed, 585 insertions(+)\n> > create mode 100644 reftable/reftable.h\n> \n> Adding a header in a separate patch from the implementation doesn't\n> match the usual practice. Can we add declarations in the same patch\n> as the functions being declared instead?\n> \n> We could still introduce the header early in its own patch if we want,\n> but it would be a skeleton of a header that gets filled out by later\n> patches.\n\nTo poke a little deeper into what I think Jonathan is trying to say:\n\nWhen we reviewed this for review club, the general feeling was that A)\nthis commit structure is still pretty difficult to review, and B) we\nwant to make sure you aren't spending a ton of time reorganizing the\ncode only to find that people still don't like the structure (as may\nhave happened with the current iteration).\n\nOne point I think Jonathan is making is that each commit should stand\nalone: compile, be tested, and do something succinct. Another point (or\nmaybe the same point from a different angle) is that a series of commits\nshould tell a progressive story. That is, something like\n\n add infrastructure to support reftable library\n reftable: read/write blocks\n reftable: write files\n reftable: read files\n reftable: generate binary tree from file\n ...\n\nis much more compelling (and easier to review) than\n\n reftable: LICENSE\n reftable: headers\n reftable: tests\n ...\n\nTowards the latter end of your series it seems like you started to take\nthat approach; but some commits, like this one (reftable: define the\npublic API) are not quite so. That is, this commit is hard to review\nwithout the context of the rest of the series: I read it, I say, \"well\nWHY do we need these functions?\", and then I cannot continue my review\nof this patch until I've completed my review of the rest of the series.\n\nOf course, taking a completed project - like your initial reftable\nsubmission - and then chopping it up into a cute story of commits is a\npain in the ***. Doing it twice - or more - is just aggravating. So I\nwonder whether we can bikeshed what story would look nice before you\neven pick up your 'git rebase -i'? Doing that bikeshedding here on list\nmeans that we also have a chance for someone to interrupt and say, \"no,\nthat organization doesn't make sense\" - or even for someone to say\n\"Emily, there is no need to reorganize these commits, go sit down\" ;)\n\n> All in all, I like this API. Thanks for putting it together\n> thoughtfully.\n> \n> The API builds up in layers (blocks, reftables, merged reftables, etc),\n> suggesting a natural division for the patch series. I think the rest of\n> the series follows that --- let's see.\n\nOr maybe what I said runs contrary to what Jonathan was actually saying.\nPlease, correct me :)\n\n - Emily\n"},{"id":"407266","messageId":"CAFQ2z_NL1UrmonMH3qLKrEkjsPjm9qTbtoeY0OHQZzkVW2t3-w@mail.gmail.com","threadId":"54245","inReplyTo":"20201002041214.GE3252492@google.com","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2020-10-10T17:32:07Z","receivedAt":"2020-10-10T22:56:19Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Fri, Oct 2, 2020 at 6:12 AM Jonathan Nieder wrote:\n> [...]\n> > +int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n>\n> How does this compare to stdlib's bsearch?\n\nbsearch gives you back NULL if it doesn't find an exact match.\n\n> > + reftable_free(a);\n> > +}\n>\n> Are there other callers that need custom free?\n\nThe libgit2 folks requested the ability to set memory allocation\nroutines, hence reftable_free().\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"407270","messageId":"CAFQ2z_NU_hBF2Gz4yFK+Q-KreBQONeMF+pJYESp7hVnKCWvmDg@mail.gmail.com","threadId":"54245","inReplyTo":"20201002035851.GB3252492@google.com","subject":"Re: [PATCH v2 02/13] reftable: define the public API","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2020-10-10T13:43:25Z","receivedAt":"2020-10-10T22:56:40Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Fri, Oct 2, 2020 at 5:58 AM Jonathan Nieder wrote:\n> Adding a header in a separate patch from the implementation doesn't\n> match the usual practice. Can we add declarations in the same patch\n> as the functions being declared instead?\n\nWe could, but it would considerably complicate work on this patch\nseries, as the commit boundary then doesn't fall on file boundaries\nanymore. Would you be open to having multiple headers for the public\ninterface? eg. reftable-record.h, reftable-reader.h etc. ?\n\n> > + /* Misuse of the API:\n> > + - on writing a record with NULL refname.\n> > + - on writing a reftable_ref_record outside the table limits\n> > + - on writing a ref or log record before the stack's next_update_index\n> > + - on writing a log record with multiline message with\n> > + exact_log_message unset\n> > + - on reading a reftable_ref_record from log iterator, or vice versa.\n> > + */\n> > + REFTABLE_API_ERROR = -6,\n>\n> Should these call BUG()? Or is it useful for some callers to be able\n> to recover from these errors?\n\nSince this was written as a standalone library, it's up to the caller\nto decide what should be done. It also simplifies testing: you can\nverify that incorrect API usage returns a certain error code, but a\nBUG() that prints an error or crashes the program is much harder to\nverify.\n\n> > + /* Dir/file conflict. */\n>\n> This is when adding a ref?\n\nyes.\n\n> > +\n> > +/*\n> > + * Convert the numeric error code to an equivalent errno code.\n> > + */\n> > +int reftable_error_to_errno(int err);\n>\n> What is the intended use of this function?\n\nThe read_raw_ref method in the ref backend API uses errno values as\nout-of-band communication mechanism.\n\n> > +/* Set the range of update indices for the records we will add. When\n> > + writing a table into a stack, the min should be at least\n> > + reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned.\n> > +\n> > + For transactional updates, typically min==max. When converting an existing\n> > + ref database into a single reftable, this would be a range of update-index\n> > + timestamps.\n> > + */\n> > +void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n> > + uint64_t max);\n>\n> What happens if I write to a reftable_writer without setting limits\n> first?\n\nyou get an REFTABLE_API_ERROR.\n\n> > +/* block_source_vtable are the operations that make up block_source */\n> > +struct reftable_block_source_vtable {\n> > + /* returns the size of a block source */\n> > + uint64_t (*size)(void *source);\n> > +\n> > + /* reads a segment from the block source. It is an error to read\n> > + beyond the end of the block */\n> > + int (*read_block)(void *source, struct reftable_block *dest,\n> > + uint64_t off, uint32_t size);\n> > + /* mark the block as read; may return the data back to malloc */\n> > + void (*return_block)(void *source, struct reftable_block *blockp);\n> > +\n> > + /* release all resources associated with the block source */\n> > + void (*close)(void *source);\n> > +};\n>\n> This is abstracting file input? I wonder if some name emphasizing the\n> 'read' operation like block_reader might be clearer.\n>\n> Do I pass in the 'struct block_source *' as the source arg? If so, why\n> are these declared as void *?\n\nyou pass in block_source->arg. Should struct implementations of\npolymorphic types carry a pointer to their own vtable instead?\n\n> Is the reason this manages the buffer instead of requiring a\n> caller-supplied buffer to support zero-copy?\n\nLog blocks are compressed, so the caller doesn't know the correct size\nto supply. By letting the block source handle the management, we can\nswap out the block read from the file for a block managed by malloc on\ndecompressing a log block.\n\n> > +/* returns the hash ID used in this table. */\n> > +uint32_t reftable_reader_hash_id(struct reftable_reader *r);\n>\n> What is a hash id?\n\nThe hash identifier, eg. \"sha1\" for SHA1 and \"s256\" for SHA-256.\n\n> [...]\n> > + Generic tables\n> > +\n> > + A unified API for reading tables, either merged tables, or single readers.\n>\n> Are there callers/helpers that don't know whether they want one or the\n> other?\n\nThe setup with per-worktree refs means that there are two reftable\nstacks in a .git repo. In order to iterate over the entire ref space,\nyou have to merge two merged tables, but the stack itslef merges a set\nof simple reftables.\n\n> [...]\n> > +/* holds a transaction to add tables at the top of a stack. */\n> > +struct reftable_addition;\n>\n> When would I add multiple tables at once to the top? Is this used\n> during compacting, for example?\n\nNo. You'd usually add only one table, but at the same time, it seemed\nkind of arbitrary to allow adding only one table. FWIW, I added this\nbecause libgit2 wants to be able to lock a ref database: creating the\nreftable_addition institutes the lock.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"407278","messageId":"CAFQ2z_PGyOyjDwPsbwUHZZayNUZYVZteHSWRGKdtLBw4_J26vA@mail.gmail.com","threadId":"54245","inReplyTo":"20201008014124.1410535-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 02/13] reftable: define the public API","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2020-10-10T16:57:56Z","receivedAt":"2020-10-10T23:10:30Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"Hi Jonathan,\n\nthanks for a detailed look at the API. I'm replying to some queries\nhere, but I won't have time for another 2 weeks to revisit the code.\n\nOn Thu, Oct 8, 2020 at 3:41 AM Jonathan Tan wrote:\n> A typical user would just want to do refname->hash, refname->log, or possibly\n> hash->refname lookups, so a ref record would be meant for low-level\n\nYou are forgetting refname -> symref target and refname -> peeled tag.\n\n> users. I can't think of a typical use case, but even if such existed, I\n> wouldn't expect this API - in particular, the refname is not written as\n> such on disk (it is composed from a prefix and a suffix, if I remember\n> correctly), so I would expect a function that one can call to write the\n> refname, but not for it to appear in a data structure.\n\nIt works with records, because it is more practical for the low level\ncode, but also because it obviates defining many functions for the\ndifferent types of lookups and seeks.\n\n> > +/* returns whether 'ref' represents a deletion */\n> > +int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);\n>\n> This looks unorthogonal - looking at the spec, the \"value type\" can\n> represent a deletion, one object name (value of the ref), two object\n> names (value of the ref + peeled target), or symbolic reference. If\n> we're willing to allocate memory for a ref record, I think we can afford\n> the extra 4 bytes (maybe 8) and use a tagged union instead.\n\nI guess you want something like\n\n struct {\n enum TYPE { REF_DEL, REF_1VAL, REF_2VAL, REF_SYM } record_type;\n char *ref_name;\n uint64_t update_index;\n union {\n uint_8 *val1;\n struct { uint8_t *val1, *val2; } val2;\n char *sym;\n };\n };\n\nI suspect it makes initialization more verbose, but yes, I could do that.\n\n> > +/* returns whether two reftable_ref_records are the same */\n> > +int reftable_ref_record_equal(struct reftable_ref_record *a,\n> > + struct reftable_ref_record *b, int hash_size);\n>\n> Not sure what this will be used for.\n\nThis is useful for testing.\n\n> > +/* Set the range of update indices for the records we will add. When\n> > + writing a table into a stack, the min should be at least\n> > + reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned.\n> > +\n> > + For transactional updates, typically min==max. When converting an existing\n> > + ref database into a single reftable, this would be a range of update-index\n> > + timestamps.\n> > + */\n> > +void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n> > + uint64_t max);\n>\n> This seems to be here because we want to write the file in a single\n> pass, and the update index maximum and minimum appear in the header. If\n> we were allowed to seek while writing, could the update index maximum\n> and minimum be deduced instead?\n\nNo. The ref record update_index is delta encoded against the min\nupdate_index, so you have to know it upfront.\n\nIf I could redesign the format, I'd leave out the max update_index\nfrom the header (because it can be determined afterwards.), but it's\ntoo late now.\n\n>\n> > +/* adds a reftable_ref_record. Must be called in ascending\n> > + order. The update_index must be within the limits set by\n> > + reftable_writer_set_limits(), or REFTABLE_API_ERROR is returned.\n> > +\n> > + It is an error to write a ref record after a log record.\n> > + */\n> > +int reftable_writer_add_ref(struct reftable_writer *w,\n> > + struct reftable_ref_record *ref);\n> > +\n> > +/* Convenience function to add multiple refs. Will sort the refs by\n> > + name before adding. */\n> > +int reftable_writer_add_refs(struct reftable_writer *w,\n> > + struct reftable_ref_record *refs, int n);\n>\n> Since refs is an array of objects and not an array of pointers, these\n> two functions could be combined - the first is the same as calling the\n> second with n=1.\n>\n> Also, ascending order of what?\n\nOf record keys, ie. refname in case of ref records.\n\n> The user will also need to know where to get the update index from - I\n> presume that this will be the maximum update index of any record with\n> the given refname + 1.\n\nthere is a reftable_stack_next_update_index() for that.\n\n> > +/* reads the next reftable_ref_record. Returns < 0 for error, 0 for OK and > 0:\n> > + end of iteration.\n> > +*/\n> > +int reftable_iterator_next_ref(struct reftable_iterator *it,\n> > + struct reftable_ref_record *ref);\n> > +\n> > +/* reads the next reftable_log_record. Returns < 0 for error, 0 for OK and > 0:\n> > + end of iteration.\n> > +*/\n> > +int reftable_iterator_next_log(struct reftable_iterator *it,\n> > + struct reftable_log_record *log);\n>\n> From my recollection, in Git, we typically use foreach functions with a\n> callback that is invoked once for each result. I think that's preferable\n> to this approach.\n\nThe iterator interface has the advantage that it captures reading both\nindividual entries (read_raw_ref) and iteration. It's also a natural\nstructure, because internally the iteration state has to be stored\n(iterating merged tables stores the iterators in a priority queue.)\n\n\n> [snip]\n>\n> > +/****************************************************************\n> > + Merged tables\n> > +\n> > + A ref database kept in a sequence of table files. The merged_table presents a\n> > + unified view to reading (seeking, iterating) a sequence of immutable tables.\n> > + ****************************************************************/\n> > +\n> > +/* A merged table is implements seeking/iterating over a stack of tables. */\n> > +struct reftable_merged_table;\n> > +\n> > +/* A generic reftable; see below. */\n> > +struct reftable_table;\n>\n> Why would we need to see an individual reftable?\n\nIt could be useful diagnostics and troubleshooting. For example, if\nyou are debugging or trying to troubleshoot/fsck a repo, it might be\ninteresting to read a single file from .git/reftable/\n\nPhilosophically, since we have to expose the functionality to write a\nsingle table, it seems reasonable to read single tables for symmetry\nas well.\n\n> Could we just represent\n> a merged table as a virtual concatenation of blocks (as if all the\n> blocks were in the same file)?\n\nNo. The format doesn't work like that.\n\n> > +/* reftable_new_merged_table creates a new merged table. It takes ownership of\n> > + the stack array.\n> > +*/\n> > +int reftable_new_merged_table(struct reftable_merged_table **dest,\n> > + struct reftable_table *stack, int n,\n> > + uint32_t hash_id);\n>\n> I presume this would be used for things like compacting a few reftables\n> together? In which case, I would expect this function to just take a\n> list of filenames.\n\nThe merging of tables has nothing to do with the filesystem, and could\nbe done with reftables represented as in memory structure. This is\nalso how the unittests work.\n\n> > +/* returns an iterator positioned just before 'name' */\n> > +int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n> > + struct reftable_iterator *it,\n> > + const char *name);\n> > +\n> > +/* returns an iterator for log entry, at given update_index */\n> > +int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n> > + struct reftable_iterator *it,\n> > + const char *name, uint64_t update_index);\n> > +\n> > +/* like reftable_merged_table_seek_log_at but look for the newest entry. */\n> > +int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n> > + struct reftable_iterator *it,\n> > + const char *name);\n>\n> Why do iterators need to be precisely positioned if this merged table is\n> immutable?\n\nI don't understand the question. How would you read a single ref from\na merged table without a seek function?\n\n> > +/* convenience function to read a single ref. Returns < 0 for error, 0\n> > + for success, and 1 if ref not found. */\n> > +int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n> > + struct reftable_ref_record *ref);\n>\n> I presume this returns the most up-to-date record in the (possibly\n> merged) table? So we can just read the hash off the record to know what\n> this ref points to.\n\ncorrect.\n\n> > +/****************************************************************\n> > + Mutable ref database\n> > +\n> > + The stack presents an interface to a mutable sequence of reftables.\n> > + ****************************************************************/\n>\n> So I would expect ref mutations to be done by opening the existing\n> reftables as a merged table, open a reftable writer, write all the\n> changes that need to be written, and update the tables.list file...\n\nRoughly, yes, but there are other considerations: you have to check\nfor concurrent updates (by the time you finish reading tables.list, a\ncompaction may have deleted some of the tables referenced, so you have\nto retry). If there are any failures, the transaction fails and you\nhave to clean up already written tables. If the write succeeds, but\nthe stack becomes unbalanced, you have to do a compaction. If you have\nto reload data, you want to avoid closing and opening the same file.\n\nIf you are curious what the stack does, have a look at stack.c.\n\nAlso, this is for separation of logic. The merged table is composed of\nreftables that have no particular type of storage, but support the\nread interface. The stack is what ties the merged table to a\nposix-like file system, and provides mutations.\n\n> > +/* holds a transaction to add tables at the top of a stack. */\n> > +struct reftable_addition;\n> > +\n> > +/*\n> > + returns a new transaction to add reftables to the given stack. As a side\n> > + effect, the ref database is locked.\n> > +*/\n> > +int reftable_stack_new_addition(struct reftable_addition **dest,\n> > + struct reftable_stack *st);\n> > +\n> > +/* Adds a reftable to transaction. */\n> > +int reftable_addition_add(struct reftable_addition *add,\n> > + int (*write_table)(struct reftable_writer *wr,\n> > + void *arg),\n> > + void *arg);\n> > +\n> > +/* Commits the transaction, releasing the lock. */\n> > +int reftable_addition_commit(struct reftable_addition *add);\n> > +\n> > +/* Release all non-committed data from the transaction, and deallocate the\n> > + transaction. Releases the lock if held. */\n> > +void reftable_addition_destroy(struct reftable_addition *add);\n>\n> We do need a transaction to write the new tables.list and then\n> atomically update the repository with it, but I don't think we need one\n> just to add a reftable to the stack.\n\nsee above.\n\n> > +/* compacts all reftables into a giant table. Expire reflog entries if config is\n> > + * non-NULL */\n> > +int reftable_stack_compact_all(struct reftable_stack *st,\n> > + struct reftable_log_expiry_config *config);\n>\n> Ah, the stack is used for compacting as well. I don't think compacting\n> belongs here though - it should be its own thing that can make use of\n> the merged-table reader and the single-table writer.\n\nWhy should it be like that, and where should compaction live?\n\n> Some things that I think are missing:\n>\n> - foreach all latest record of all refs (in any order) (I see some\n\nRef records are keyed by name, so the iteration produces each ref only once.\n\n> functions above that can set the position of the iterator, but I\n> don't think that the iterator skips over irrelevant refs?\n\nWhat makes you think that?\n\n> So we can't\n> use it for iterating over all refs if we don't care about the history\n> of refs)\n\nI think you might be confusing the refs with reflogs here. The history\nof a ref is kept in the reflog. This is why functions that want to\nseek to a log record need an update_index timestamp.\n\n> - foreach all log entries of one ref (or all refs)\n\nBefore reviewing the API header here, I think it might be useful to\nalso look at the commit in the other series that uses the API to\nimplement the git ref backend. It shows how the foreach functions can\nbe implemented using the iterator interface.\n\n> In summary, I would have expected a mechanism to read multiple (possibly\n> one) reftable and perform queries on it, a mechanism to write a single\n> reftable, and some functions to update tables.list.\n\nI understand your expectation, but I think that expectation glosses\nover important details of how it actually works.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\nRegistergericht und -nummer: Hamburg, HRB 86891\nSitz der Gesellschaft: Hamburg\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"407279","messageId":"CAFQ2z_MRzz41x0Osvf6unvQ4Bk-RsA9NxbWZWpfwwJ2D=4Pv7A@mail.gmail.com","threadId":"54245","inReplyTo":"20201008014855.1416580-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2020-10-10T17:28:05Z","receivedAt":"2020-10-10T23:10:37Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Thu, Oct 8, 2020 at 3:48 AM Jonathan Tan wrote:\n>\n> > From: Han-Wen Nienhuys \n> >\n> > This commit provides basic utility classes for the reftable library.\n> >\n> > Since the reftable library must compile standalone, there may be some overlap\n> > with git-core utility functions.\n\n> I think duplicating things like strbuf is an unnecessary burden if Git\n> is to maintain this library. Something like \"reftable will only import\n> git-compat-util.h and strbuf.h, and any project that wants to use\n> reftable must make sure that these functions and data structures are\n> available\" would be more plausible.\n\nSure, but how do we ensure that the directory won't take on\ndependencies beyond these headers? I am worried that I will be\ninvolved in a tedious back & forth process to keep updates going into\nlibgit2 and/or also have to keep maintaining\ngithub.com/google/reftable.\n\nFWIW, the duplication is really tiny: according to\n\n $ wc $(grep -l REFTABLE_STANDALONE *[ch])\n\nit's just 431 lines of code.\n\n--\nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\nRegistergericht und -nummer: Hamburg, HRB 86891\nSitz der Gesellschaft: Hamburg\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"407285","messageId":"CAFQ2z_PVVGHi_N1uw7A4xUczm9WPkpXkQ9Yte9gCNO9ExAupyw@mail.gmail.com","threadId":"54245","inReplyTo":"20201009211300.GA2774782@google.com","subject":"Re: [PATCH v2 02/13] reftable: define the public API","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2020-10-10T17:03:45Z","receivedAt":"2020-10-10T23:10:59Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Fri, Oct 9, 2020 at 11:13 PM Emily Shaffer wrote:\n> Of course, taking a completed project - like your initial reftable\n> submission - and then chopping it up into a cute story of commits is a\n> pain in the ***. Doing it twice - or more - is just aggravating. So I\n> wonder whether we can bikeshed what story would look nice before you\n> even pick up your 'git rebase -i'? Doing that bikeshedding here on list\n> means that we also have a chance for someone to interrupt and say, \"no,\n> that organization doesn't make sense\" - or even for someone to say\n\nIt would be great if you could bikeshed about this while I'm on holiday :)\n\nThat said, I think it's worthwhile to first look at the commit that\nuses the library to implement a ref backend, because that will show\nbetter how the reftable library fits with the way that Git likes to\nconsume its data.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"407319","messageId":"nycvar.QRO.7.76.6.2010111247450.50@tvgsbejvaqbjf.bet","threadId":"54245","inReplyTo":"CAFQ2z_MRzz41x0Osvf6unvQ4Bk-RsA9NxbWZWpfwwJ2D=4Pv7A@mail.gmail.com","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-11T10:52:01Z","receivedAt":"2020-10-12T10:02:06Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Han-Wen,\n\nOn Sat, 10 Oct 2020, Han-Wen Nienhuys wrote:\n\n> On Thu, Oct 8, 2020 at 3:48 AM Jonathan Tan wrote:\n> >\n> > > From: Han-Wen Nienhuys \n> > >\n> > > This commit provides basic utility classes for the reftable library.\n> > >\n> > > Since the reftable library must compile standalone, there may be some overlap\n> > > with git-core utility functions.\n>\n> > I think duplicating things like strbuf is an unnecessary burden if Git\n> > is to maintain this library. Something like \"reftable will only import\n> > git-compat-util.h and strbuf.h, and any project that wants to use\n> > reftable must make sure that these functions and data structures are\n> > available\" would be more plausible.\n>\n> Sure, but how do we ensure that the directory won't take on\n> dependencies beyond these headers? I am worried that I will be\n> involved in a tedious back & forth process to keep updates going into\n> libgit2 and/or also have to keep maintaining\n> github.com/google/reftable.\n>\n> FWIW, the duplication is really tiny: according to\n>\n> $ wc $(grep -l REFTABLE_STANDALONE *[ch])\n>\n> it's just 431 lines of code.\n\nThe `merge_bases_many()` function has only 33 lines of code, partially\nduplicating `get_reachable_subset()`. Yet, it had a bug in it for two\nyears that was not found.\n\nHow much worse will the situation be with your 431 lines of code.\n\nEven more so when you consider the fact that you intend to shove the same\nduplication down libgit2's throat. It's \"triplicating\" code.\n\nSo I find the argument you made above quite unconvincing.\n\nCiao,\nDscho\n"},{"id":"407339","messageId":"20201012151919.GA3740546@google.com","threadId":"54245","inReplyTo":"nycvar.QRO.7.76.6.2010111247450.50@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-10-12T15:19:19Z","receivedAt":"2020-10-12T15:19:25Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJohannes Schindelin wrote:\n\n> The `merge_bases_many()` function has only 33 lines of code, partially\n> duplicating `get_reachable_subset()`. Yet, it had a bug in it for two\n> years that was not found.\n>\n> How much worse will the situation be with your 431 lines of code.\n>\n> Even more so when you consider the fact that you intend to shove the same\n> duplication down libgit2's throat. It's \"triplicating\" code.\n\nCareful: you seem to be making a bunch of assumptions here (for\nexample, around Han-Wen having some intent around shoving things down\nlibgit2's throat), and you seem to be focusing on the person instead\nof the contribution.\n\nWould you mind restating your point in a way that is a little easier\nto process, for example by focusing on the effect that this patch\nwould have on you as a Git developer?\n\nThanks,\nJonathan\n\n> So I find the argument you made above quite unconvincing.\n"},{"id":"407340","messageId":"20201012152505.GB3740546@google.com","threadId":"54245","inReplyTo":"CAFQ2z_NL1UrmonMH3qLKrEkjsPjm9qTbtoeY0OHQZzkVW2t3-w@mail.gmail.com","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-10-12T15:25:05Z","receivedAt":"2020-10-12T15:25:09Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"(+cc: Patrick Steinhardt from libgit2)\nHi,\n\nHan-Wen Nienhuys wrote[1]:\n> On Fri, Oct 2, 2020 at 6:12 AM Jonathan Nieder wrote:\n>> Han-Wen Nienhuys wrote:\n\n>>> + reftable_free(a);\n>>> +}\n>>\n>> Are there other callers that need custom free?\n>\n> The libgit2 folks requested the ability to set memory allocation\n> routines, hence reftable_free().\n\nThanks. Patrick or Han-Wen, can you say a little more about this use\ncase? That would help with making sure we are making an API that\nmeets its needs.\n\nFor example, is a custom allocator something that would be set\nglobally or something attached to a handle? If the former, would code\nthat uses xmalloc and free and gets #define-d away when used in\nlibgit2 work? If the latter, what does the handle look like?\n\nThanks,\nJonathan\n\n[1] context: https://lore.kernel.org/git/pull.847.git.git.1600283416.gitgitgadget@gmail.com/T/#r2337ddc80848c80f6c9699d904ab64d4297d2c54\n"},{"id":"407346","messageId":"xmqqeem3qttu.fsf@gitster.c.googlers.com","threadId":"54245","inReplyTo":"nycvar.QRO.7.76.6.2010111247450.50@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-12T16:42:37Z","receivedAt":"2020-10-12T16:42:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n>> FWIW, the duplication is really tiny: according to\n>>\n>> $ wc $(grep -l REFTABLE_STANDALONE *[ch])\n>>\n>> it's just 431 lines of code.\n>\n> The `merge_bases_many()` function has only 33 lines of code, partially\n> duplicating `get_reachable_subset()`. Yet, it had a bug in it for two\n> years that was not found.\n\nIt does not affect the current discussion, but what you are giving\nis a revisionist view of the world. The latter function came MUCH\nlater to do a bit more than the former. The bug was caused by the\nfact that those that added the latter neglected the responsibility\nof maintaining the former to the same degree when new feature like\ncommit-graph were added to the latter.\n\nThe root cause was that the latter one did not share the code with\nthe former one when it was introduced. That does make it appear\nsimilar to the situation we have at hand with duplicated utility\nfunctions.\n\n> How much worse will the situation be with your 431 lines of code.\n>\n> Even more so when you consider the fact that you intend to shove the same\n> duplication down libgit2's throat. It's \"triplicating\" code.\n>\n> So I find the argument you made above quite unconvincing.\n>\n> Ciao,\n> Dscho\n"},{"id":"407349","messageId":"20201012165730.GC3740546@google.com","threadId":"54245","inReplyTo":"CAFQ2z_NU_hBF2Gz4yFK+Q-KreBQONeMF+pJYESp7hVnKCWvmDg@mail.gmail.com","subject":"Re: [PATCH v2 02/13] reftable: define the public API","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-10-12T16:57:30Z","receivedAt":"2020-10-12T16:57:36Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Han-Wen Nienhuys wrote:\n> On Fri, Oct 2, 2020 at 5:58 AM Jonathan Nieder wrote:\n\n>> Adding a header in a separate patch from the implementation doesn't\n>> match the usual practice. Can we add declarations in the same patch\n>> as the functions being declared instead?\n>\n> We could, but it would considerably complicate work on this patch\n> series, as the commit boundary then doesn't fall on file boundaries\n> anymore. Would you be open to having multiple headers for the public\n> interface? eg. reftable-record.h, reftable-reader.h etc. ?\n\nI'd be more than open to it: I think that would make for a clearer API.\n\nFor example, a caller like refs.c wouldn't have any need to use\nreftable-block.h because that's a lower-level detail that operates\nbehind the scenes when operating on a stack of reftables. So I think\nthis would be a good step toward addressing the feedback Jonathan gave\nabout the set of functions exposed by the API being overwhelming.\n\n>>> + /* Misuse of the API:\n>>> + - on writing a record with NULL refname.\n>>> + - on writing a reftable_ref_record outside the table limits\n>>> + - on writing a ref or log record before the stack's next_update_index\n>>> + - on writing a log record with multiline message with\n>>> + exact_log_message unset\n>>> + - on reading a reftable_ref_record from log iterator, or vice versa.\n>>> + */\n>>> + REFTABLE_API_ERROR = -6,\n>>\n>> Should these call BUG()? Or is it useful for some callers to be able\n>> to recover from these errors?\n>\n> Since this was written as a standalone library, it's up to the caller\n> to decide what should be done.\n\nI'm not strongly opinionated about this, but just a quick note: this\nimplies that the library would need to make sure it is producing a\nvalid state in error cases.\n\nDoes the API documentation describe what state a handle is in after\nan error --- e.g., what operations are permitted after that?\n\n>>> +\n>>> +/*\n>>> + * Convert the numeric error code to an equivalent errno code.\n>>> + */\n>>> +int reftable_error_to_errno(int err);\n>>\n>> What is the intended use of this function?\n>\n> The read_raw_ref method in the ref backend API uses errno values as\n> out-of-band communication mechanism.\n\nCould we change that? It sounds error-prone.\n\n[...]\n>>> +/* Set the range of update indices for the records we will add. When\n>>> + writing a table into a stack, the min should be at least\n>>> + reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned.\n>>> +\n>>> + For transactional updates, typically min==max. When converting an existing\n>>> + ref database into a single reftable, this would be a range of update-index\n>>> + timestamps.\n>>> + */\n>>> +void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n>>> + uint64_t max);\n>>\n>> What happens if I write to a reftable_writer without setting limits\n>> first?\n>\n> you get an REFTABLE_API_ERROR.\n\n(To be clear, with all of these questions, I am asking not only for my\nown curiosity but so that the documentation, naming, etc can be\ncleared up to help the next person who wonders the same thing.)\n\n[...]\n>> Do I pass in the 'struct block_source *' as the source arg? If so, why\n>> are these declared as void *?\n>\n> you pass in block_source->arg. Should struct implementations of\n> polymorphic types carry a pointer to their own vtable instead?\n\nI think a pointer to block_source would make it more self-explanatory,\nyes.\n\n(Aside: this is a difference between how Go and e.g. Java and C++\nimplement polymorphism. In Go, an interface object contains a vtable\nand a pointer to the object. In Java and C++, the object contains a\nvptr.)\n\n>> Is the reason this manages the buffer instead of requiring a\n>> caller-supplied buffer to support zero-copy?\n>\n> Log blocks are compressed, so the caller doesn't know the correct size\n> to supply. By letting the block source handle the management, we can\n> swap out the block read from the file for a block managed by malloc on\n> decompressing a log block.\n\nHm, my naive assumption would have been that we'd use different\nbuffers for the compressed and uncompressed data.\n\n>>> + Generic tables\n>>> +\n>>> + A unified API for reading tables, either merged tables, or single readers.\n>>\n>> Are there callers/helpers that don't know whether they want one or the\n>> other?\n>\n> The setup with per-worktree refs means that there are two reftable\n> stacks in a .git repo. In order to iterate over the entire ref space,\n> you have to merge two merged tables, but the stack itslef merges a set\n> of simple reftables.\n\nI see. That's subtle, so it seems worth documenting for the next\nperson reading this documentation and wondering why.\n\nThanks,\nJonathan\n"},{"id":"407350","messageId":"20201012170527.GA21606@xps","threadId":"54245","inReplyTo":"20201012152505.GB3740546@google.com","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2020-10-12T17:05:27Z","receivedAt":"2020-10-12T17:04:10Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Oct 12, 2020 at 08:25:05AM -0700, Jonathan Nieder wrote:\n> (+cc: Patrick Steinhardt from libgit2)\n> Hi,\n> \n> Han-Wen Nienhuys wrote[1]:\n> > On Fri, Oct 2, 2020 at 6:12 AM Jonathan Nieder wrote:\n> >> Han-Wen Nienhuys wrote:\n> \n> >>> + reftable_free(a);\n> >>> +}\n> >>\n> >> Are there other callers that need custom free?\n> >\n> > The libgit2 folks requested the ability to set memory allocation\n> > routines, hence reftable_free().\n> \n> Thanks. Patrick or Han-Wen, can you say a little more about this use\n> case? That would help with making sure we are making an API that\n> meets its needs.\n> \n> For example, is a custom allocator something that would be set\n> globally or something attached to a handle? If the former, would code\n> that uses xmalloc and free and gets #define-d away when used in\n> libgit2 work? If the latter, what does the handle look like?\n\nWe have global pluggable allocators in libgit2 which can be set up\nbefore calling `git_libgit2_init()`. The user of libgit2 can fill a\n`git_allocator` structure with a set of funtcion pointers, most\nimportantly with implementations of `free` and `malloc`. Those then get\nused across all of libgit2 for all subsequent allocations.\n\nIn order to be as thorough as possible, we thus also need to replace\nthese function pointers for libgit2's dependencies. As registration of\nthe allocator happens at runtime, we need to also be able to replace\nfunction pointers of dependencies at runtime. E.g. for OpenSSL, it\nprovides an API `CRYPTO_set_mem_functions(malloc, realloc, free)` which\nwe call on global initialization of the libgit2 library.\n\nSo, to answer your questions:\n\n - The allocator is global and cannot be changed after initialization\n of libgit2.\n\n - It is pluggable, users can set up their own allocators by filling\n a structure with function pointers for `free`, `malloc`, `realloc`\n etc.\n\n - Due to the pluggable nature, we need to be able to set up those\n pointers at runtime. We can provide a set of static wrappers\n though which then call into the pluggable functions, so defines\n would probably work for us, too.\n\nI hope that answers all of your questions.\n\nPatrick\n"},{"id":"407353","messageId":"20201012174507.GD3740546@google.com","threadId":"54245","inReplyTo":"20201012170527.GA21606@xps","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-10-12T17:45:07Z","receivedAt":"2020-10-12T17:45:12Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Patrick Steinhardt wrote:\n\n> In order to be as thorough as possible, we thus also need to replace\n> these function pointers for libgit2's dependencies. As registration of\n> the allocator happens at runtime, we need to also be able to replace\n> function pointers of dependencies at runtime. E.g. for OpenSSL, it\n> provides an API `CRYPTO_set_mem_functions(malloc, realloc, free)` which\n> we call on global initialization of the libgit2 library.\n>\n> So, to answer your questions:\n>\n> - The allocator is global and cannot be changed after initialization\n> of libgit2.\n>\n> - It is pluggable, users can set up their own allocators by filling\n> a structure with function pointers for `free`, `malloc`, `realloc`\n> etc.\n>\n> - Due to the pluggable nature, we need to be able to set up those\n> pointers at runtime. We can provide a set of static wrappers\n> though which then call into the pluggable functions, so defines\n> would probably work for us, too.\n>\n> I hope that answers all of your questions.\n\nThanks! Yes, that's very helpful.\n\nJonathan\n"},{"id":"407362","messageId":"nycvar.QRO.7.76.6.2010122035330.50@tvgsbejvaqbjf.bet","threadId":"54245","inReplyTo":"20201012151919.GA3740546@google.com","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-12T18:44:11Z","receivedAt":"2020-10-12T18:44:25Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jonathan,\n\nOn Mon, 12 Oct 2020, Jonathan Nieder wrote:\n\n> Johannes Schindelin wrote:\n>\n> > The `merge_bases_many()` function has only 33 lines of code, partially\n> > duplicating `get_reachable_subset()`. Yet, it had a bug in it for two\n> > years that was not found.\n> >\n> > How much worse will the situation be with your 431 lines of code.\n> >\n> > Even more so when you consider the fact that you intend to shove the same\n> > duplication down libgit2's throat. It's \"triplicating\" code.\n>\n> Careful: you seem to be making a bunch of assumptions here (for\n> example, around Han-Wen having some intent around shoving things down\n> libgit2's throat), and you seem to be focusing on the person instead\n> of the contribution.\n>\n> Would you mind restating your point in a way that is a little easier\n> to process, for example by focusing on the effect that this patch\n> would have on you as a Git developer?\n\nI really hope that Han-Wen did not take this as a personal attack.\n\nIn case he did, I'll gladly try to re-phrase my point: It does not matter\nhow much code is duplicated. The fact that we run into bugs even in\nunintentionally duplicated code is reason enough to not lightly accept\nthis design decision as being set in stone.\n\nWhich is why I, you, Emily and Josh pointed that out, and it is the same\nreason why Peff and Junio had made the same point in the past.\n\nI was quite unpleasantly surprised to see that all of those objections\nseemed to be insufficient, hence my rather forceful reply.\n\nAs to the libgit2 angle: libgit2 also has `strbuf` (which it calls\n`git_buf` IIRC), and I am certain that it also has those other helper\nfunctions à la `put_be32()`. So yes, the triplication I mentioned is a\nvery real, undesirable feature of this patch series.\n\nTo be honest, I would much rather spend my time reviewing patches that\nadded `reftable` support using as much of libgit.a's functionality as\npossible rather than repeating the same point over and over again: that it\nmakes very little sense _not_ to use that functionality. Of course, at\nthis stage I understand that my feedback is not very welcome, so I will\ntry to refrain from commenting on this patch series further (I do reserve\nthe right to send patches that fix CI failures, just like I've done quite\na few times up to this point).\n\nCiao,\nDscho\n"},{"id":"407366","messageId":"nycvar.QRO.7.76.6.2010122058100.50@tvgsbejvaqbjf.bet","threadId":"54245","inReplyTo":"xmqqeem3qttu.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-12T19:01:30Z","receivedAt":"2020-10-12T19:01:43Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Mon, 12 Oct 2020, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n>\n> >> FWIW, the duplication is really tiny: according to\n> >>\n> >> $ wc $(grep -l REFTABLE_STANDALONE *[ch])\n> >>\n> >> it's just 431 lines of code.\n> >\n> > The `merge_bases_many()` function has only 33 lines of code, partially\n> > duplicating `get_reachable_subset()`. Yet, it had a bug in it for two\n> > years that was not found.\n>\n> It does not affect the current discussion, but what you are giving\n> is a revisionist view of the world. The latter function came MUCH\n> later to do a bit more than the former. The bug was caused by the\n> fact that those that added the latter neglected the responsibility\n> of maintaining the former to the same degree when new feature like\n> commit-graph were added to the latter.\n\nYou're right, I mixed up the direction: by introducing new code, the old\ncode became under-tested and a bug creeped in that was not detected for\ntwo years.\n\n> The root cause was that the latter one did not share the code with\n> the former one when it was introduced. That does make it appear\n> similar to the situation we have at hand with duplicated utility\n> functions.\n\nIndeed, and even if this is just one concrete example, experience (and\nteachers and instructors and mentors) repeat the lesson that code\nduplication should be avoided because it _does_ cause problems.\n\nThanks for setting the record straight,\nDscho\n\n>\n> > How much worse will the situation be with your 431 lines of code.\n> >\n> > Even more so when you consider the fact that you intend to shove the same\n> > duplication down libgit2's throat. It's \"triplicating\" code.\n> >\n> > So I find the argument you made above quite unconvincing.\n> >\n> > Ciao,\n> > Dscho\n>\n"},{"id":"407374","messageId":"20201012194121.GA3814681@google.com","threadId":"54245","inReplyTo":"nycvar.QRO.7.76.6.2010122035330.50@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-10-12T19:41:21Z","receivedAt":"2020-10-12T19:41:26Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Johannes Schindelin wrote:\n\n> Of course, at\n> this stage I understand that my feedback is not very welcome,\n\nDo you mean because you don't feel that your feedback has been\nsufficiently acted upon, or is there something else you are alluding\nto?\n\nPuzzled,\nJonathan\n"},{"id":"407386","messageId":"nycvar.QRO.7.76.6.2010122222250.50@tvgsbejvaqbjf.bet","threadId":"54245","inReplyTo":"20201012194121.GA3814681@google.com","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-12T20:27:46Z","receivedAt":"2020-10-12T20:27:59Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jonathan,\n\nOn Mon, 12 Oct 2020, Jonathan Nieder wrote:\n\n> Johannes Schindelin wrote:\n>\n> > Of course, at this stage I understand that my feedback is not very\n> > welcome,\n>\n> Do you mean because you don't feel that your feedback has been\n> sufficiently acted upon, or is there something else you are alluding to?\n\nMy feedback has not been acted upon, and my arguments have been dismissed\n(or there were attempts to dismiss them, I should say, that did not\nconvince me). At least I saw that when gitster repeated some of my\nconcerns in #git-devel, there were attempts to address them. While I feel\nbad about leaving the burden on gitster's shoulders, at least I am glad\nthat those concerns are being addressed.\n\nCiao,\nDscho\n\n"},{"id":"407445","messageId":"nycvar.QRO.7.76.6.2010131405380.50@tvgsbejvaqbjf.bet","threadId":"54245","inReplyTo":"20201012170527.GA21606@xps","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-13T12:12:05Z","receivedAt":"2020-10-13T14:19:55Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Patrick,\n\nOn Mon, 12 Oct 2020, Patrick Steinhardt wrote:\n\n> On Mon, Oct 12, 2020 at 08:25:05AM -0700, Jonathan Nieder wrote:\n> > (+cc: Patrick Steinhardt from libgit2)\n> > Hi,\n> >\n> > Han-Wen Nienhuys wrote[1]:\n> > > On Fri, Oct 2, 2020 at 6:12 AM Jonathan Nieder wrote:\n> > >> Han-Wen Nienhuys wrote:\n> >\n> > >>> + reftable_free(a);\n> > >>> +}\n> > >>\n> > >> Are there other callers that need custom free?\n> > >\n> > > The libgit2 folks requested the ability to set memory allocation\n> > > routines, hence reftable_free().\n> >\n> > Thanks. Patrick or Han-Wen, can you say a little more about this use\n> > case? That would help with making sure we are making an API that\n> > meets its needs.\n> >\n> > For example, is a custom allocator something that would be set\n> > globally or something attached to a handle? If the former, would code\n> > that uses xmalloc and free and gets #define-d away when used in\n> > libgit2 work? If the latter, what does the handle look like?\n>\n> We have global pluggable allocators in libgit2 which can be set up\n> before calling `git_libgit2_init()`. The user of libgit2 can fill a\n> `git_allocator` structure with a set of funtcion pointers, most\n> importantly with implementations of `free` and `malloc`. Those then get\n> used across all of libgit2 for all subsequent allocations.\n\nI did not find out how those are used in the `deps/` part of libgit2's\nsource code. For example, I see a couple of instances where `malloc()` is\nused in `ntlmclient` and in `pcre`.\n\nThe thing I was looking for would have been something like\n\n\t#define malloc(size) git__malloc(size)\n\t...\n\nThis would also have been what I imagined to be the best strategy to\nintegrate the reftable code once it is properly embedded in libgit.a (and\nof course using libgit.a's API as much as it can).\n\nSomewhat related: I was wondering whether it would make sense for git.git\nto rename `strbuf` to `git_buf`? Would that make it easier to exchange\ncode between the two projects? Or would it just be unnecessary churn?\n\nCiao,\nDscho\n"},{"id":"407456","messageId":"xmqqo8l6ku0y.fsf@gitster.c.googlers.com","threadId":"54245","inReplyTo":"nycvar.QRO.7.76.6.2010131405380.50@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-13T15:47:09Z","receivedAt":"2020-10-13T15:47:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> Somewhat related: I was wondering whether it would make sense for git.git\n> to rename `strbuf` to `git_buf`? Would that make it easier to exchange\n> code between the two projects? Or would it just be unnecessary churn?\n\nTo us, \"git_buf\" is just as descriptive as \"buf\" and does not say\nanything about the nature of 'buf' (other than apparently it was\ninvented and widely used here). \"git_strbuf\" I can understand, but\nwhy should we?\n\n"},{"id":"407597","messageId":"nycvar.QRO.7.76.6.2010151346060.56@tvgsbejvaqbjf.bet","threadId":"54245","inReplyTo":"xmqqo8l6ku0y.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-15T11:46:50Z","receivedAt":"2020-10-15T12:01:58Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Tue, 13 Oct 2020, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n>\n> > Somewhat related: I was wondering whether it would make sense for git.git\n> > to rename `strbuf` to `git_buf`? Would that make it easier to exchange\n> > code between the two projects? Or would it just be unnecessary churn?\n>\n> To us, \"git_buf\" is just as descriptive as \"buf\" and does not say\n> anything about the nature of 'buf' (other than apparently it was\n> invented and widely used here). \"git_strbuf\" I can understand, but\n> why should we?\n\nIf it makes code sharing between git.git and libgit2 easier, why shouldn't\nwe ;-)\n\nObviously, if it doesn't make life easier, we shouldn't bother.\n\nCiao,\nDscho\n"},{"id":"407621","messageId":"xmqqwnzrh30f.fsf@gitster.c.googlers.com","threadId":"54245","inReplyTo":"nycvar.QRO.7.76.6.2010151346060.56@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-15T16:23:28Z","receivedAt":"2020-10-15T16:23:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> Hi Junio,\n>\n> On Tue, 13 Oct 2020, Junio C Hamano wrote:\n>\n>> Johannes Schindelin writes:\n>>\n>> > Somewhat related: I was wondering whether it would make sense for git.git\n>> > to rename `strbuf` to `git_buf`? Would that make it easier to exchange\n>> > code between the two projects? Or would it just be unnecessary churn?\n>>\n>> To us, \"git_buf\" is just as descriptive as \"buf\" and does not say\n>> anything about the nature of 'buf' (other than apparently it was\n>> invented and widely used here). \"git_strbuf\" I can understand, but\n>> why should we?\n>\n> If it makes code sharing between git.git and libgit2 easier, why shouldn't\n> we ;-)\n\nI see no reasonably explanation why libgit2 is not the one who uses\n\"#define strbuf git_buf\" to make \"sharing easier\", though.\n\n"},{"id":"407668","messageId":"nycvar.QRO.7.76.6.2010152139390.56@tvgsbejvaqbjf.bet","threadId":"54245","inReplyTo":"xmqqwnzrh30f.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-15T19:39:57Z","receivedAt":"2020-10-15T19:40:10Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Thu, 15 Oct 2020, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n>\n> > Hi Junio,\n> >\n> > On Tue, 13 Oct 2020, Junio C Hamano wrote:\n> >\n> >> Johannes Schindelin writes:\n> >>\n> >> > Somewhat related: I was wondering whether it would make sense for git.git\n> >> > to rename `strbuf` to `git_buf`? Would that make it easier to exchange\n> >> > code between the two projects? Or would it just be unnecessary churn?\n> >>\n> >> To us, \"git_buf\" is just as descriptive as \"buf\" and does not say\n> >> anything about the nature of 'buf' (other than apparently it was\n> >> invented and widely used here). \"git_strbuf\" I can understand, but\n> >> why should we?\n> >\n> > If it makes code sharing between git.git and libgit2 easier, why shouldn't\n> > we ;-)\n>\n> I see no reasonably explanation why libgit2 is not the one who uses\n> \"#define strbuf git_buf\" to make \"sharing easier\", though.\n\nThat's a fair point, thanks for bringing it up!\n\nCiao,\nDscho\n"},{"id":"407710","messageId":"20201016091526.GA70821@tanuki","threadId":"54245","inReplyTo":"xmqqwnzrh30f.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2020-10-16T09:15:26Z","receivedAt":"2020-10-16T09:16:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Oct 15, 2020 at 09:23:28AM -0700, Junio C Hamano wrote:\n> Johannes Schindelin writes:\n> \n> > Hi Junio,\n> >\n> > On Tue, 13 Oct 2020, Junio C Hamano wrote:\n> >\n> >> Johannes Schindelin writes:\n> >>\n> >> > Somewhat related: I was wondering whether it would make sense for git.git\n> >> > to rename `strbuf` to `git_buf`? Would that make it easier to exchange\n> >> > code between the two projects? Or would it just be unnecessary churn?\n> >>\n> >> To us, \"git_buf\" is just as descriptive as \"buf\" and does not say\n> >> anything about the nature of 'buf' (other than apparently it was\n> >> invented and widely used here). \"git_strbuf\" I can understand, but\n> >> why should we?\n> >\n> > If it makes code sharing between git.git and libgit2 easier, why shouldn't\n> > we ;-)\n> \n> I see no reasonably explanation why libgit2 is not the one who uses\n> \"#define strbuf git_buf\" to make \"sharing easier\", though.\n\nIt probably wouldn't help much anyway. We already have our own buffer\ntype which we can't easily replace with yours as it's part of the public\ninterface. If the need arises, providing a compatibility interface for\nit shouldn't be too hard.\n\nPatrick\n"},{"id":"408217","messageId":"873625i9tc.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"CAFQ2z_MRzz41x0Osvf6unvQ4Bk-RsA9NxbWZWpfwwJ2D=4Pv7A@mail.gmail.com","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2020-10-23T09:13:51Z","receivedAt":"2020-10-23T09:13:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Oct 10 2020, Han-Wen Nienhuys wrote:\n\n> On Thu, Oct 8, 2020 at 3:48 AM Jonathan Tan wrote:\n>>\n>> > From: Han-Wen Nienhuys \n>> >\n>> > This commit provides basic utility classes for the reftable library.\n>> >\n>> > Since the reftable library must compile standalone, there may be some overlap\n>> > with git-core utility functions.\n>\n>> I think duplicating things like strbuf is an unnecessary burden if Git\n>> is to maintain this library. Something like \"reftable will only import\n>> git-compat-util.h and strbuf.h, and any project that wants to use\n>> reftable must make sure that these functions and data structures are\n>> available\" would be more plausible.\n>\n> Sure, but how do we ensure that the directory won't take on\n> dependencies beyond these headers? I am worried that I will be\n> involved in a tedious back & forth process to keep updates going into\n> libgit2 and/or also have to keep maintaining\n> github.com/google/reftable.\n>\n> FWIW, the duplication is really tiny: according to\n>\n> $ wc $(grep -l REFTABLE_STANDALONE *[ch])\n>\n> it's just 431 lines of code.\n\nWhy import the dead code at all? If I add this to your update script:\n\n # Not a general solution, but works for the specific code here.\n perl -0777 -pi -e 's[(#ifndef REFTABLE_STANDALONE\\n.*?\\n#else\\n).*?(?=\\n#endif)][$1/* Removed upstream code for git.git import */]gs' reftable/system.h\n perl -0777 -pi -e 's[(?<=#ifdef REFTABLE_STANDALONE\\n).*?(?=\\n#endif)][/* Removed upstream code for git.git import */]gs' reftable/strbuf.c reftable/compat.h\n perl -0777 -pi -e 's[(?<=#ifdef REFTABLE_STANDALONE\\n).*?(?=\\n#else)][/* Removed upstream code for git.git import */]gs' reftable/compat.c reftable/strbuf.h\n\nIt's now 157 lines instead of 431.\n\nI think doing that with a tiny bit more complexity in the update.sh\nscript is a much lower maintenance burden.\n\nIt's not just about the number of lines, but things coming up in grep,\nand now unique code really stands out (e.g. strbuf_add_void, should\nprobably be just added to the main strbuf.h if it's needed...), and of\ncourse the cost of attention of eyeballing an already big series on the\nML & the churn every time we have updates.\n\nOverall I'm all for having this carved out in its own directory at a\ncost of a bit more maintenance burden if it can be shared with libgit2 &\nbe a standalone library.\n\nI am concerned that it seems this code can't be maintained in git.git by\nanyone not willing to sign a contract with Google. I sent a tiny PR for\na typo fix at [1] and got directed to sign [2] before someone at Google\ncould look at it. I see brian raised this before in [3] but there wasn't\na reply to that point.\n\nIs there some summary of how this part of integrating it is supposed to\nwork going forward?\n\nAt first glance that seems like a recipe for perma-forking this pretty\nmuch from day one, i.e.:\n\n A. Upstream changes happen\n B. We get a patch to the bundled library to this ML\n ==> Google employees maintaining A can't even look at B\n ==> Patch B can't be integrated into A\n\n1. https://github.com/google/reftable/pull/2\n2. https://cla.developers.google.com/about/google-individual\n3. https://lore.kernel.org/git/20200512233741.GB6605@camp.crustytoothpaste.net/\n"},{"id":"408264","messageId":"xmqqlffwhmjp.fsf@gitster.c.googlers.com","threadId":"54245","inReplyTo":"873625i9tc.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 05/13] reftable: utility functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-23T17:36:26Z","receivedAt":"2020-10-23T17:36:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> It's not just about the number of lines, but things coming up in grep,\n> and now unique code really stands out (e.g. strbuf_add_void, should\n> probably be just added to the main strbuf.h if it's needed...), and of\n> course the cost of attention of eyeballing an already big series on the\n> ML & the churn every time we have updates.\n\nThese are all valid concerns.\n\n> Overall I'm all for having this carved out in its own directory at a\n> cost of a bit more maintenance burden if it can be shared with libgit2 &\n> be a standalone library.\n\nDo you mean by \"this\" the reftable stuff as a whole, or only the\nduplicated support library part? If the latter is split out of what\nwe import as our reftable/ directory, and stored in a separate\ndirectory that we do not have to import (because we have strbuf and\nother stuff) but other people may choose to use (because they may\nnot have strbuf and other stuff), that may work. Also, if the\ncontents of reftable/ directory wants a pluggable allocator, the\nmain code can be written to call reftable_malloc (or whatever), with\na thin shim to interface with us (i.e. reftable_malloc() would be\nimplemented as a thin wrapper around xmalloc() for us) which is\nstored in a separate directory just for us to interface with the\nmain reftable library. For libgit2, there will be a separate\ndirectory that uses a different implementation of reftable_malloc()\nthat would let them plug their preferred allocator. An arrangement\nlike that might work.\n\nI do not offhand know if that kind of overhead is worth the trouble\nor if there are better ways, though.\n\n> I am concerned that it seems this code can't be maintained in git.git by\n> anyone not willing to sign a contract with Google.\n\nIt can be maintained in git.git; the trouble comes when they want to\nupdate us.\n\nI however suspect that, as the primary intended audience, it is hard\nto imagine that the reftable library as a standalone project will be\nsuccessful without going through the usual reviews and testing in\ngit.git. So even though there exists github.com/google/reftable\nrepository, its contents may not matter very much in practice,\nunless they come here and beg for the change we make ourselves\nanyway. Perhaps I am being naive. I dunno.\n\n> I sent a tiny PR for\n> a typo fix at [1] and got directed to sign [2] before someone at Google\n> could look at it. I see brian raised this before in [3] but there wasn't\n> a reply to that point.\n>\n> Is there some summary of how this part of integrating it is supposed to\n> work going forward?\n\nYeah, thanks for raising a good point. We definitely need to figure\nthis part out.\n\n> At first glance that seems like a recipe for perma-forking this pretty\n> much from day one, i.e.:\n>\n> A. Upstream changes happen\n> B. We get a patch to the bundled library to this ML\n> ==> Google employees maintaining A can't even look at B\n> ==> Patch B can't be integrated into A\n>\n> 1. https://github.com/google/reftable/pull/2\n> 2. https://cla.developers.google.com/about/google-individual\n> 3. https://lore.kernel.org/git/20200512233741.GB6605@camp.crustytoothpaste.net/\n"},{"id":"410883","messageId":"030998a732ac54f41473b07c7376ca46b817bf4e.1606419752.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v3.git.git.1606419752.gitgitgadget@gmail.com","subject":"[PATCH v3 01/16] move sleep_millisec to git-compat-util.h","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-11-26T19:42:16Z","receivedAt":"2020-11-26T19:42:54Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe sleep function is defined in wrapper.c, so it makes more sense to be a in\nsystem compatibility header.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n cache.h | 1 -\n git-compat-util.h | 2 ++\n 2 files changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex c0072d43b1..e986cf4ea9 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1960,7 +1960,6 @@ int stat_validity_check(struct stat_validity *sv, const char *path);\n void stat_validity_update(struct stat_validity *sv, int fd);\n \n int versioncmp(const char *s1, const char *s2);\n-void sleep_millisec(int millisec);\n \n /*\n * Create a directory and (if share is nonzero) adjust its permissions\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex adfea06897..7d509c5022 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -1354,4 +1354,6 @@ static inline void *container_of_or_null_offset(void *ptr, size_t offset)\n \t((uintptr_t)&(ptr)->member - (uintptr_t)(ptr))\n #endif /* !__GNUC__ */\n \n+void sleep_millisec(int millisec);\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"410884","messageId":"91c3ac2449edc45db643b0ba6cba66ef31176600.1606419752.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v3.git.git.1606419752.gitgitgadget@gmail.com","subject":"[PATCH v3 03/16] reftable: add LICENSE","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-11-26T19:42:18Z","receivedAt":"2020-11-26T19:42:54Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nTODO: relicense?\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/LICENSE | 31 +++++++++++++++++++++++++++++++\n 1 file changed, 31 insertions(+)\n create mode 100644 reftable/LICENSE\n\ndiff --git a/reftable/LICENSE b/reftable/LICENSE\nnew file mode 100644\nindex 0000000000..402e0f9356\n--- /dev/null\n+++ b/reftable/LICENSE\n@@ -0,0 +1,31 @@\n+BSD License\n+\n+Copyright (c) 2020, Google LLC\n+All rights reserved.\n+\n+Redistribution and use in source and binary forms, with or without\n+modification, are permitted provided that the following conditions are\n+met:\n+\n+* Redistributions of source code must retain the above copyright notice,\n+this list of conditions and the following disclaimer.\n+\n+* Redistributions in binary form must reproduce the above copyright\n+notice, this list of conditions and the following disclaimer in the\n+documentation and/or other materials provided with the distribution.\n+\n+* Neither the name of Google LLC nor the names of its contributors may\n+be used to endorse or promote products derived from this software\n+without specific prior written permission.\n+\n+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n+\"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n-- \ngitgitgadget\n\n"},{"id":"410885","messageId":"2aa30f536fb7ce5501d1ecf0315cbcb1c1c5ce38.1606419752.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v3.git.git.1606419752.gitgitgadget@gmail.com","subject":"[PATCH v3 04/16] reftable: add error related functionality","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-11-26T19:42:19Z","receivedAt":"2020-11-26T19:42:54Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable/ directory is structured as a library, so it cannot\ncrash on misuse. Instead, it returns an error codes.\n\nIn addition, the error code can be used to signal conditions from lower levels\nof the library to be handled by higher levels of the library. For example, a\ntransaction might legitimately write an empty reftable file, but in that case,\nwe'd want to shortcut the transaction overhead.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/error.c | 39 ++++++++++++++++++++++++\n reftable/reftable-error.h | 62 +++++++++++++++++++++++++++++++++++++++\n 2 files changed, 101 insertions(+)\n create mode 100644 reftable/error.c\n create mode 100644 reftable/reftable-error.h\n\ndiff --git a/reftable/error.c b/reftable/error.c\nnew file mode 100644\nindex 0000000000..1d60137ea7\n--- /dev/null\n+++ b/reftable/error.c\n@@ -0,0 +1,39 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-error.h\"\n+\n+#include \n+\n+const char *reftable_error_str(int err)\n+{\n+\tstatic char buf[250];\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn \"I/O error\";\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn \"corrupt reftable file\";\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn \"file does not exist\";\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn \"data is outdated\";\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn \"misuse of the reftable API\";\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn \"zlib failure\";\n+\tcase REFTABLE_NAME_CONFLICT:\n+\t\treturn \"file/directory conflict\";\n+\tcase REFTABLE_REFNAME_ERROR:\n+\t\treturn \"invalid refname\";\n+\tcase -1:\n+\t\treturn \"general error\";\n+\tdefault:\n+\t\tsnprintf(buf, sizeof(buf), \"unknown error code %d\", err);\n+\t\treturn buf;\n+\t}\n+}\ndiff --git a/reftable/reftable-error.h b/reftable/reftable-error.h\nnew file mode 100644\nindex 0000000000..7e55a16aac\n--- /dev/null\n+++ b/reftable/reftable-error.h\n@@ -0,0 +1,62 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_ERROR_H\n+#define REFTABLE_ERROR_H\n+\n+/*\n+ Errors in reftable calls are signaled with negative integer return values. 0\n+ means success.\n+*/\n+enum reftable_error {\n+\t/* Unexpected file system behavior */\n+\tREFTABLE_IO_ERROR = -2,\n+\n+\t/* Format inconsistency on reading data\n+\t */\n+\tREFTABLE_FORMAT_ERROR = -3,\n+\n+\t/* File does not exist. Returned from block_source_from_file(), because\n+\t it needs special handling in stack.\n+\t*/\n+\tREFTABLE_NOT_EXIST_ERROR = -4,\n+\n+\t/* Trying to write out-of-date data. */\n+\tREFTABLE_LOCK_ERROR = -5,\n+\n+\t/* Misuse of the API:\n+\t - on writing a record with NULL refname.\n+\t - on writing a reftable_ref_record outside the table limits\n+\t - on writing a ref or log record before the stack's next_update_index\n+\t - on writing a log record with multiline message with\n+\t exact_log_message unset\n+\t - on reading a reftable_ref_record from log iterator, or vice versa.\n+\n+\t When a call misuses the API, the internal state of the library is kept\n+\t unchanged.\n+\t*/\n+\tREFTABLE_API_ERROR = -6,\n+\n+\t/* Decompression error */\n+\tREFTABLE_ZLIB_ERROR = -7,\n+\n+\t/* Wrote a table without blocks. */\n+\tREFTABLE_EMPTY_TABLE_ERROR = -8,\n+\n+\t/* Dir/file conflict. */\n+\tREFTABLE_NAME_CONFLICT = -9,\n+\n+\t/* Illegal ref name. */\n+\tREFTABLE_REFNAME_ERROR = -10,\n+};\n+\n+/* convert the numeric error code to a string. The string should not be\n+ * deallocated. */\n+const char *reftable_error_str(int err);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"410889","messageId":"cf667653dff437cee4c0d04e8c061a8b9ccc46a7.1606419752.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v3.git.git.1606419752.gitgitgadget@gmail.com","subject":"[PATCH v3 02/16] init-db: set the_repository->hash_algo early on","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-11-26T19:42:17Z","receivedAt":"2020-11-26T19:42:54Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable backend needs to know the hash algorithm for writing the\ninitialization hash table.\n\nThe initial reftable contains a symref HEAD => \"main\" (or \"master\"), which is\nagnostic to the size of hash value, but this is an exceptional circumstance, and\nthe reftable library does not cater for this exceptional. It insists that all\ntables in the stack have a consistent format ID for the hash algorithm.\n\nCall set_repo_hash_algo directly after reading out GIT_DEFAULT_HASH.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n builtin/init-db.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 01bc648d41..5c8c67fec6 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -391,6 +391,7 @@ static void validate_hash_algorithm(struct repository_format *repo_fmt, int hash\n \t\t\tdie(_(\"unknown hash algorithm '%s'\"), env);\n \t\trepo_fmt->hash_algo = env_algo;\n \t}\n+\trepo_set_hash_algo(the_repository, repo_fmt->hash_algo);\n }\n \n int init_db(const char *git_dir, const char *real_git_dir,\n-- \ngitgitgadget\n\n"},{"id":"410895","messageId":"pull.847.v3.git.git.1606419752.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v2.git.git.1601568663.gitgitgadget@gmail.com","subject":"[PATCH v3 00/16] reftable library","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-11-26T19:42:15Z","receivedAt":"2020-11-26T19:42:54Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"This splits the giant commit from \nhttps://github.com/gitgitgadget/git/pull/539 into a series of smaller\ncommits, which build and have unittests.\n\nThe final commit should also be split up, but I want to wait until we have\nconsensus that the bottom commits look good.\n\nversion 26 Nov 2020:\n\n * split public API header into multiple headers\n * minor adaptions in response to review feedback\n * filter out code for standalone compile\n * get rid of test framework\n\nHan-Wen Nienhuys (15):\n move sleep_millisec to git-compat-util.h\n init-db: set the_repository->hash_algo early on\n reftable: add LICENSE\n reftable: add error related functionality\n reftable: utility functions\n reftable: add blocksource, an abstraction for random access reads\n reftable: (de)serialization for the polymorphic record type.\n reftable: reading/writing blocks\n reftable: a generic binary tree implementation\n reftable: write reftable files\n reftable: read reftable files\n reftable: reftable file level tests\n reftable: rest of library\n Reftable support for git-core\n Add \"test-tool dump-reftable\" command.\n\nSZEDER Gábor (1):\n git-prompt: prepare for reftable refs backend\n\n .../technical/repository-version.txt | 7 +\n Makefile | 46 +-\n builtin/clone.c | 5 +-\n builtin/init-db.c | 56 +-\n builtin/worktree.c | 27 +-\n cache.h | 9 +-\n config.mak.uname | 2 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n contrib/completion/git-prompt.sh | 7 +-\n git-compat-util.h | 2 +\n refs.c | 27 +-\n refs.h | 3 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1418 +++++++++++++++++\n reftable/.gitattributes | 1 +\n reftable/LICENSE | 31 +\n reftable/VERSION | 1 +\n reftable/basics.c | 144 ++\n reftable/basics.h | 56 +\n reftable/basics_test.c | 56 +\n reftable/block.c | 440 +++++\n reftable/block.h | 129 ++\n reftable/block_test.c | 119 ++\n reftable/blocksource.c | 148 ++\n reftable/blocksource.h | 22 +\n reftable/compat.h | 48 +\n reftable/constants.h | 21 +\n reftable/dump.c | 219 +++\n reftable/error.c | 39 +\n reftable/iter.c | 242 +++\n reftable/iter.h | 75 +\n reftable/merged.c | 366 +++++\n reftable/merged.h | 35 +\n reftable/merged_test.c | 333 ++++\n reftable/pq.c | 115 ++\n reftable/pq.h | 32 +\n reftable/publicbasics.c | 58 +\n reftable/reader.c | 733 +++++++++\n reftable/reader.h | 75 +\n reftable/record.c | 1116 +++++++++++++\n reftable/record.h | 139 ++\n reftable/record_test.c | 404 +++++\n reftable/refname.c | 209 +++\n reftable/refname.h | 29 +\n reftable/refname_test.c | 100 ++\n reftable/reftable-blocksource.h | 50 +\n reftable/reftable-error.h | 62 +\n reftable/reftable-generic.h | 49 +\n reftable/reftable-iterator.h | 37 +\n reftable/reftable-malloc.h | 20 +\n reftable/reftable-merged.h | 68 +\n reftable/reftable-reader.h | 89 ++\n reftable/reftable-record.h | 73 +\n reftable/reftable-stack.h | 116 ++\n reftable/reftable-tests.h | 22 +\n reftable/reftable-writer.h | 149 ++\n reftable/reftable.c | 98 ++\n reftable/reftable_test.c | 577 +++++++\n reftable/stack.c | 1247 +++++++++++++++\n reftable/stack.h | 41 +\n reftable/stack_test.c | 781 +++++++++\n reftable/system.h | 32 +\n reftable/test_framework.c | 23 +\n reftable/test_framework.h | 48 +\n reftable/tree.c | 63 +\n reftable/tree.h | 34 +\n reftable/tree_test.c | 61 +\n reftable/writer.c | 676 ++++++++\n reftable/writer.h | 50 +\n reftable/zlib-compat.c | 92 ++\n repository.c | 2 +\n repository.h | 3 +\n setup.c | 9 +-\n t/helper/test-reftable.c | 20 +\n t/helper/test-tool.c | 4 +-\n t/helper/test-tool.h | 2 +\n t/t0031-reftable.sh | 199 +++\n t/t0032-reftable-unittest.sh | 14 +\n t/t1409-avoid-packing-refs.sh | 6 +\n t/t1450-fsck.sh | 6 +\n t/t3210-pack-refs.sh | 6 +\n t/test-lib.sh | 5 +\n 82 files changed, 11951 insertions(+), 39 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100644 reftable/.gitattributes\n create mode 100644 reftable/LICENSE\n create mode 100644 reftable/VERSION\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/basics_test.c\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/compat.h\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/dump.c\n create mode 100644 reftable/error.c\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n create mode 100644 reftable/reftable-blocksource.h\n create mode 100644 reftable/reftable-error.h\n create mode 100644 reftable/reftable-generic.h\n create mode 100644 reftable/reftable-iterator.h\n create mode 100644 reftable/reftable-malloc.h\n create mode 100644 reftable/reftable-merged.h\n create mode 100644 reftable/reftable-reader.h\n create mode 100644 reftable/reftable-record.h\n create mode 100644 reftable/reftable-stack.h\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/reftable-writer.h\n create mode 100644 reftable/reftable.c\n create mode 100644 reftable/reftable_test.c\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n create mode 100644 reftable/zlib-compat.c\n create mode 100644 t/helper/test-reftable.c\n create mode 100755 t/t0031-reftable.sh\n create mode 100755 t/t0032-reftable-unittest.sh\n\n\nbase-commit: faefdd61ec7c7f6f3c8c9907891465ac9a2a1475\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-847%2Fhanwen%2Flibreftable-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-847/hanwen/libreftable-v3\nPull-Request: https://github.com/git/git/pull/847\n\nRange-diff vs v2:\n\n -: ---------- > 1: 030998a732 move sleep_millisec to git-compat-util.h\n -: ---------- > 2: cf667653df init-db: set the_repository->hash_algo early on\n 1: 6228103b4a = 3: 91c3ac2449 reftable: add LICENSE\n 2: 5d1b946ab5 ! 4: 2aa30f536f reftable: define the public API\n @@ Metadata\n Author: Han-Wen Nienhuys \n \n ## Commit message ##\n - reftable: define the public API\n + reftable: add error related functionality\n +\n + The reftable/ directory is structured as a library, so it cannot\n + crash on misuse. Instead, it returns an error codes.\n +\n + In addition, the error code can be used to signal conditions from lower levels\n + of the library to be handled by higher levels of the library. For example, a\n + transaction might legitimately write an empty reftable file, but in that case,\n + we'd want to shortcut the transaction overhead.\n \n Signed-off-by: Han-Wen Nienhuys \n \n - ## reftable/reftable.h (new) ##\n + ## reftable/error.c (new) ##\n @@\n +/*\n +Copyright 2020 Google LLC\n @@ reftable/reftable.h (new)\n +https://developers.google.com/open-source/licenses/bsd\n +*/\n +\n -+#ifndef REFTABLE_H\n -+#define REFTABLE_H\n -+\n -+#include \n -+#include \n -+\n -+void reftable_set_alloc(void *(*malloc)(size_t),\n -+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n -+\n -+/****************************************************************\n -+ Basic data types\n -+\n -+ Reftables store the state of each ref in struct reftable_ref_record, and they\n -+ store a sequence of reflog updates in struct reftable_log_record.\n -+ ****************************************************************/\n -+\n -+/* reftable_ref_record holds a ref database entry target_value */\n -+struct reftable_ref_record {\n -+\tchar *refname; /* Name of the ref, malloced. */\n -+\tuint64_t update_index; /* Logical timestamp at which this value is\n -+\t\t\t\t written */\n -+\tuint8_t *value; /* SHA1, or NULL. malloced. */\n -+\tuint8_t *target_value; /* peeled annotated tag, or NULL. malloced. */\n -+\tchar *target; /* symref, or NULL. malloced. */\n -+};\n -+\n -+/* returns whether 'ref' represents a deletion */\n -+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);\n -+\n -+/* prints a reftable_ref_record onto stdout */\n -+void reftable_ref_record_print(struct reftable_ref_record *ref,\n -+\t\t\t uint32_t hash_id);\n -+\n -+/* frees and nulls all pointer values. */\n -+void reftable_ref_record_clear(struct reftable_ref_record *ref);\n -+\n -+/* returns whether two reftable_ref_records are the same */\n -+int reftable_ref_record_equal(struct reftable_ref_record *a,\n -+\t\t\t struct reftable_ref_record *b, int hash_size);\n -+\n -+/* reftable_log_record holds a reflog entry */\n -+struct reftable_log_record {\n -+\tchar *refname;\n -+\tuint64_t update_index; /* logical timestamp of a transactional update.\n -+\t\t\t\t*/\n -+\tuint8_t *new_hash;\n -+\tuint8_t *old_hash;\n -+\tchar *name;\n -+\tchar *email;\n -+\tuint64_t time;\n -+\tint16_t tz_offset;\n -+\tchar *message;\n -+};\n -+\n -+/* returns whether 'ref' represents the deletion of a log record. */\n -+int reftable_log_record_is_deletion(const struct reftable_log_record *log);\n -+\n -+/* frees and nulls all pointer values. */\n -+void reftable_log_record_clear(struct reftable_log_record *log);\n -+\n -+/* returns whether two records are equal. */\n -+int reftable_log_record_equal(struct reftable_log_record *a,\n -+\t\t\t struct reftable_log_record *b, int hash_size);\n -+\n -+/* dumps a reftable_log_record on stdout, for debugging/testing. */\n -+void reftable_log_record_print(struct reftable_log_record *log,\n -+\t\t\t uint32_t hash_id);\n ++#include \"reftable-error.h\"\n ++\n ++#include \n ++\n ++const char *reftable_error_str(int err)\n ++{\n ++\tstatic char buf[250];\n ++\tswitch (err) {\n ++\tcase REFTABLE_IO_ERROR:\n ++\t\treturn \"I/O error\";\n ++\tcase REFTABLE_FORMAT_ERROR:\n ++\t\treturn \"corrupt reftable file\";\n ++\tcase REFTABLE_NOT_EXIST_ERROR:\n ++\t\treturn \"file does not exist\";\n ++\tcase REFTABLE_LOCK_ERROR:\n ++\t\treturn \"data is outdated\";\n ++\tcase REFTABLE_API_ERROR:\n ++\t\treturn \"misuse of the reftable API\";\n ++\tcase REFTABLE_ZLIB_ERROR:\n ++\t\treturn \"zlib failure\";\n ++\tcase REFTABLE_NAME_CONFLICT:\n ++\t\treturn \"file/directory conflict\";\n ++\tcase REFTABLE_REFNAME_ERROR:\n ++\t\treturn \"invalid refname\";\n ++\tcase -1:\n ++\t\treturn \"general error\";\n ++\tdefault:\n ++\t\tsnprintf(buf, sizeof(buf), \"unknown error code %d\", err);\n ++\t\treturn buf;\n ++\t}\n ++}\n +\n + ## reftable/reftable-error.h (new) ##\n +@@\n ++/*\n ++Copyright 2020 Google LLC\n +\n -+/****************************************************************\n -+ Error handling\n ++Use of this source code is governed by a BSD-style\n ++license that can be found in the LICENSE file or at\n ++https://developers.google.com/open-source/licenses/bsd\n ++*/\n +\n -+ Error are signaled with negative integer return values. 0 means success.\n -+ ****************************************************************/\n ++#ifndef REFTABLE_ERROR_H\n ++#define REFTABLE_ERROR_H\n +\n -+/* different types of errors */\n ++/*\n ++ Errors in reftable calls are signaled with negative integer return values. 0\n ++ means success.\n ++*/\n +enum reftable_error {\n +\t/* Unexpected file system behavior */\n +\tREFTABLE_IO_ERROR = -2,\n @@ reftable/reftable.h (new)\n +\t - on writing a log record with multiline message with\n +\t exact_log_message unset\n +\t - on reading a reftable_ref_record from log iterator, or vice versa.\n ++\n ++\t When a call misuses the API, the internal state of the library is kept\n ++\t unchanged.\n +\t*/\n +\tREFTABLE_API_ERROR = -6,\n +\n @@ reftable/reftable.h (new)\n + * deallocated. */\n +const char *reftable_error_str(int err);\n +\n -+/*\n -+ * Convert the numeric error code to an equivalent errno code.\n -+ */\n -+int reftable_error_to_errno(int err);\n -+\n -+/****************************************************************\n -+ Writing\n -+\n -+ Writing single reftables\n -+ ****************************************************************/\n -+\n -+/* reftable_write_options sets options for writing a single reftable. */\n -+struct reftable_write_options {\n -+\t/* boolean: do not pad out blocks to block size. */\n -+\tint unpadded;\n -+\n -+\t/* the blocksize. Should be less than 2^24. */\n -+\tuint32_t block_size;\n -+\n -+\t/* boolean: do not generate a SHA1 => ref index. */\n -+\tint skip_index_objects;\n -+\n -+\t/* how often to write complete keys in each block. */\n -+\tint restart_interval;\n -+\n -+\t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n -+\t * Defaults to SHA1 if unset\n -+\t */\n -+\tuint32_t hash_id;\n -+\n -+\t/* boolean: do not check ref names for validity or dir/file conflicts.\n -+\t */\n -+\tint skip_name_check;\n -+\n -+\t/* boolean: copy log messages exactly. If unset, check that the message\n -+\t * is a single line, and add '\\n' if missing.\n -+\t */\n -+\tint exact_log_message;\n -+};\n -+\n -+/* reftable_block_stats holds statistics for a single block type */\n -+struct reftable_block_stats {\n -+\t/* total number of entries written */\n -+\tint entries;\n -+\t/* total number of key restarts */\n -+\tint restarts;\n -+\t/* total number of blocks */\n -+\tint blocks;\n -+\t/* total number of index blocks */\n -+\tint index_blocks;\n -+\t/* depth of the index */\n -+\tint max_index_level;\n -+\n -+\t/* offset of the first block for this type */\n -+\tuint64_t offset;\n -+\t/* offset of the top level index block for this type, or 0 if not\n -+\t * present */\n -+\tuint64_t index_offset;\n -+};\n -+\n -+/* stats holds overall statistics for a single reftable */\n -+struct reftable_stats {\n -+\t/* total number of blocks written. */\n -+\tint blocks;\n -+\t/* stats for ref data */\n -+\tstruct reftable_block_stats ref_stats;\n -+\t/* stats for the SHA1 to ref map. */\n -+\tstruct reftable_block_stats obj_stats;\n -+\t/* stats for index blocks */\n -+\tstruct reftable_block_stats idx_stats;\n -+\t/* stats for log blocks */\n -+\tstruct reftable_block_stats log_stats;\n -+\n -+\t/* disambiguation length of shortened object IDs. */\n -+\tint object_id_len;\n -+};\n -+\n -+/* reftable_new_writer creates a new writer */\n -+struct reftable_writer *\n -+reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n -+\t\t void *writer_arg, struct reftable_write_options *opts);\n -+\n -+/* write to a file descriptor. fdp should be an int* pointing to the fd. */\n -+int reftable_fd_write(void *fdp, const void *data, size_t size);\n -+\n -+/* Set the range of update indices for the records we will add. When\n -+ writing a table into a stack, the min should be at least\n -+ reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned.\n -+\n -+ For transactional updates, typically min==max. When converting an existing\n -+ ref database into a single reftable, this would be a range of update-index\n -+ timestamps.\n -+ */\n -+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n -+\t\t\t\tuint64_t max);\n -+\n -+/* adds a reftable_ref_record. Must be called in ascending\n -+ order. The update_index must be within the limits set by\n -+ reftable_writer_set_limits(), or REFTABLE_API_ERROR is returned.\n -+\n -+ It is an error to write a ref record after a log record.\n -+ */\n -+int reftable_writer_add_ref(struct reftable_writer *w,\n -+\t\t\t struct reftable_ref_record *ref);\n -+\n -+/* Convenience function to add multiple refs. Will sort the refs by\n -+ name before adding. */\n -+int reftable_writer_add_refs(struct reftable_writer *w,\n -+\t\t\t struct reftable_ref_record *refs, int n);\n -+\n -+/* adds a reftable_log_record. Must be called in ascending order (with more\n -+ recent log entries first.)\n -+ */\n -+int reftable_writer_add_log(struct reftable_writer *w,\n -+\t\t\t struct reftable_log_record *log);\n -+\n -+/* Convenience function to add multiple logs. Will sort the records by\n -+ key before adding. */\n -+int reftable_writer_add_logs(struct reftable_writer *w,\n -+\t\t\t struct reftable_log_record *logs, int n);\n -+\n -+/* reftable_writer_close finalizes the reftable. The writer is retained so\n -+ * statistics can be inspected. */\n -+int reftable_writer_close(struct reftable_writer *w);\n -+\n -+/* writer_stats returns the statistics on the reftable being written.\n -+\n -+ This struct becomes invalid when the writer is freed.\n -+ */\n -+const struct reftable_stats *writer_stats(struct reftable_writer *w);\n -+\n -+/* reftable_writer_free deallocates memory for the writer */\n -+void reftable_writer_free(struct reftable_writer *w);\n -+\n -+/****************************************************************\n -+ * ITERATING\n -+ ****************************************************************/\n -+\n -+/* iterator is the generic interface for walking over data stored in a\n -+ reftable. It is generally passed around by value.\n -+*/\n -+struct reftable_iterator {\n -+\tstruct reftable_iterator_vtable *ops;\n -+\tvoid *iter_arg;\n -+};\n -+\n -+/* reads the next reftable_ref_record. Returns < 0 for error, 0 for OK and > 0:\n -+ end of iteration.\n -+*/\n -+int reftable_iterator_next_ref(struct reftable_iterator *it,\n -+\t\t\t struct reftable_ref_record *ref);\n -+\n -+/* reads the next reftable_log_record. Returns < 0 for error, 0 for OK and > 0:\n -+ end of iteration.\n -+*/\n -+int reftable_iterator_next_log(struct reftable_iterator *it,\n -+\t\t\t struct reftable_log_record *log);\n -+\n -+/* releases resources associated with an iterator. */\n -+void reftable_iterator_destroy(struct reftable_iterator *it);\n -+\n -+/****************************************************************\n -+ Reading single tables\n -+\n -+ The follow routines are for reading single files. For an application-level\n -+ interface, skip ahead to struct reftable_merged_table and struct\n -+ reftable_stack.\n -+ ****************************************************************/\n -+\n -+/* block_source is a generic wrapper for a seekable readable file.\n -+ It is generally passed around by value.\n -+ */\n -+struct reftable_block_source {\n -+\tstruct reftable_block_source_vtable *ops;\n -+\tvoid *arg;\n -+};\n -+\n -+/* a contiguous segment of bytes. It keeps track of its generating block_source\n -+ so it can return itself into the pool.\n -+*/\n -+struct reftable_block {\n -+\tuint8_t *data;\n -+\tint len;\n -+\tstruct reftable_block_source source;\n -+};\n -+\n -+/* block_source_vtable are the operations that make up block_source */\n -+struct reftable_block_source_vtable {\n -+\t/* returns the size of a block source */\n -+\tuint64_t (*size)(void *source);\n -+\n -+\t/* reads a segment from the block source. It is an error to read\n -+\t beyond the end of the block */\n -+\tint (*read_block)(void *source, struct reftable_block *dest,\n -+\t\t\t uint64_t off, uint32_t size);\n -+\t/* mark the block as read; may return the data back to malloc */\n -+\tvoid (*return_block)(void *source, struct reftable_block *blockp);\n -+\n -+\t/* release all resources associated with the block source */\n -+\tvoid (*close)(void *source);\n -+};\n -+\n -+/* opens a file on the file system as a block_source */\n -+int reftable_block_source_from_file(struct reftable_block_source *block_src,\n -+\t\t\t\t const char *name);\n -+\n -+/* The reader struct is a handle to an open reftable file. */\n -+struct reftable_reader;\n -+\n -+/* reftable_new_reader opens a reftable for reading. If successful, returns 0\n -+ * code and sets pp. The name is used for creating a stack. Typically, it is the\n -+ * basename of the file. The block source `src` is owned by the reader, and is\n -+ * closed on calling reftable_reader_destroy().\n -+ */\n -+int reftable_new_reader(struct reftable_reader **pp,\n -+\t\t\tstruct reftable_block_source *src, const char *name);\n -+\n -+/* reftable_reader_seek_ref returns an iterator where 'name' would be inserted\n -+ in the table. To seek to the start of the table, use name = \"\".\n -+\n -+ example:\n -+\n -+ struct reftable_reader *r = NULL;\n -+ int err = reftable_new_reader(&r, &src, \"filename\");\n -+ if (err < 0) { ... }\n -+ struct reftable_iterator it = {0};\n -+ err = reftable_reader_seek_ref(r, &it, \"refs/heads/master\");\n -+ if (err < 0) { ... }\n -+ struct reftable_ref_record ref = {0};\n -+ while (1) {\n -+ err = reftable_iterator_next_ref(&it, &ref);\n -+ if (err > 0) {\n -+ break;\n -+ }\n -+ if (err < 0) {\n -+ ..error handling..\n -+ }\n -+ ..found..\n -+ }\n -+ reftable_iterator_destroy(&it);\n -+ reftable_ref_record_clear(&ref);\n -+ */\n -+int reftable_reader_seek_ref(struct reftable_reader *r,\n -+\t\t\t struct reftable_iterator *it, const char *name);\n -+\n -+/* returns the hash ID used in this table. */\n -+uint32_t reftable_reader_hash_id(struct reftable_reader *r);\n -+\n -+/* seek to logs for the given name, older than update_index. To seek to the\n -+ start of the table, use name = \"\".\n -+ */\n -+int reftable_reader_seek_log_at(struct reftable_reader *r,\n -+\t\t\t\tstruct reftable_iterator *it, const char *name,\n -+\t\t\t\tuint64_t update_index);\n -+\n -+/* seek to newest log entry for given name. */\n -+int reftable_reader_seek_log(struct reftable_reader *r,\n -+\t\t\t struct reftable_iterator *it, const char *name);\n -+\n -+/* closes and deallocates a reader. */\n -+void reftable_reader_free(struct reftable_reader *);\n -+\n -+/* return an iterator for the refs pointing to `oid`. */\n -+int reftable_reader_refs_for(struct reftable_reader *r,\n -+\t\t\t struct reftable_iterator *it, uint8_t *oid);\n -+\n -+/* return the max_update_index for a table */\n -+uint64_t reftable_reader_max_update_index(struct reftable_reader *r);\n -+\n -+/* return the min_update_index for a table */\n -+uint64_t reftable_reader_min_update_index(struct reftable_reader *r);\n -+\n -+/****************************************************************\n -+ Merged tables\n -+\n -+ A ref database kept in a sequence of table files. The merged_table presents a\n -+ unified view to reading (seeking, iterating) a sequence of immutable tables.\n -+ ****************************************************************/\n -+\n -+/* A merged table is implements seeking/iterating over a stack of tables. */\n -+struct reftable_merged_table;\n -+\n -+/* A generic reftable; see below. */\n -+struct reftable_table;\n -+\n -+/* reftable_new_merged_table creates a new merged table. It takes ownership of\n -+ the stack array.\n -+*/\n -+int reftable_new_merged_table(struct reftable_merged_table **dest,\n -+\t\t\t struct reftable_table *stack, int n,\n -+\t\t\t uint32_t hash_id);\n -+\n -+/* returns an iterator positioned just before 'name' */\n -+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n -+\t\t\t\t struct reftable_iterator *it,\n -+\t\t\t\t const char *name);\n -+\n -+/* returns an iterator for log entry, at given update_index */\n -+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n -+\t\t\t\t struct reftable_iterator *it,\n -+\t\t\t\t const char *name, uint64_t update_index);\n -+\n -+/* like reftable_merged_table_seek_log_at but look for the newest entry. */\n -+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n -+\t\t\t\t struct reftable_iterator *it,\n -+\t\t\t\t const char *name);\n -+\n -+/* returns the max update_index covered by this merged table. */\n -+uint64_t\n -+reftable_merged_table_max_update_index(struct reftable_merged_table *mt);\n -+\n -+/* returns the min update_index covered by this merged table. */\n -+uint64_t\n -+reftable_merged_table_min_update_index(struct reftable_merged_table *mt);\n -+\n -+/* releases memory for the merged_table */\n -+void reftable_merged_table_free(struct reftable_merged_table *m);\n -+\n -+/* return the hash ID of the merged table. */\n -+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m);\n -+\n -+/****************************************************************\n -+ Generic tables\n -+\n -+ A unified API for reading tables, either merged tables, or single readers.\n -+ ****************************************************************/\n -+\n -+struct reftable_table {\n -+\tstruct reftable_table_vtable *ops;\n -+\tvoid *table_arg;\n -+};\n -+\n -+int reftable_table_seek_ref(struct reftable_table *tab,\n -+\t\t\t struct reftable_iterator *it, const char *name);\n -+\n -+void reftable_table_from_reader(struct reftable_table *tab,\n -+\t\t\t\tstruct reftable_reader *reader);\n -+\n -+/* returns the hash ID from a generic reftable_table */\n -+uint32_t reftable_table_hash_id(struct reftable_table *tab);\n -+\n -+/* create a generic table from reftable_merged_table */\n -+void reftable_table_from_merged_table(struct reftable_table *tab,\n -+\t\t\t\t struct reftable_merged_table *table);\n -+\n -+/* returns the max update_index covered by this table. */\n -+uint64_t reftable_table_max_update_index(struct reftable_table *tab);\n -+\n -+/* returns the min update_index covered by this table. */\n -+uint64_t reftable_table_min_update_index(struct reftable_table *tab);\n -+\n -+/* convenience function to read a single ref. Returns < 0 for error, 0\n -+ for success, and 1 if ref not found. */\n -+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n -+\t\t\t struct reftable_ref_record *ref);\n -+\n -+/****************************************************************\n -+ Mutable ref database\n -+\n -+ The stack presents an interface to a mutable sequence of reftables.\n -+ ****************************************************************/\n -+\n -+/* a stack is a stack of reftables, which can be mutated by pushing a table to\n -+ * the top of the stack */\n -+struct reftable_stack;\n -+\n -+/* open a new reftable stack. The tables along with the table list will be\n -+ stored in 'dir'. Typically, this should be .git/reftables.\n -+*/\n -+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n -+\t\t struct reftable_write_options config);\n -+\n -+/* returns the update_index at which a next table should be written. */\n -+uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\n -+\n -+/* holds a transaction to add tables at the top of a stack. */\n -+struct reftable_addition;\n -+\n -+/*\n -+ returns a new transaction to add reftables to the given stack. As a side\n -+ effect, the ref database is locked.\n -+*/\n -+int reftable_stack_new_addition(struct reftable_addition **dest,\n -+\t\t\t\tstruct reftable_stack *st);\n -+\n -+/* Adds a reftable to transaction. */\n -+int reftable_addition_add(struct reftable_addition *add,\n -+\t\t\t int (*write_table)(struct reftable_writer *wr,\n -+\t\t\t\t\t void *arg),\n -+\t\t\t void *arg);\n -+\n -+/* Commits the transaction, releasing the lock. */\n -+int reftable_addition_commit(struct reftable_addition *add);\n -+\n -+/* Release all non-committed data from the transaction, and deallocate the\n -+ transaction. Releases the lock if held. */\n -+void reftable_addition_destroy(struct reftable_addition *add);\n -+\n -+/* add a new table to the stack. The write_table function must call\n -+ reftable_writer_set_limits, add refs and return an error value. */\n -+int reftable_stack_add(struct reftable_stack *st,\n -+\t\t int (*write_table)(struct reftable_writer *wr,\n -+\t\t\t\t\t void *write_arg),\n -+\t\t void *write_arg);\n -+\n -+/* returns the merged_table for seeking. This table is valid until the\n -+ next write or reload, and should not be closed or deleted.\n -+*/\n -+struct reftable_merged_table *\n -+reftable_stack_merged_table(struct reftable_stack *st);\n -+\n -+/* frees all resources associated with the stack. */\n -+void reftable_stack_destroy(struct reftable_stack *st);\n -+\n -+/* Reloads the stack if necessary. This is very cheap to run if the stack was up\n -+ * to date */\n -+int reftable_stack_reload(struct reftable_stack *st);\n -+\n -+/* Policy for expiring reflog entries. */\n -+struct reftable_log_expiry_config {\n -+\t/* Drop entries older than this timestamp */\n -+\tuint64_t time;\n -+\n -+\t/* Drop older entries */\n -+\tuint64_t min_update_index;\n -+};\n -+\n -+/* compacts all reftables into a giant table. Expire reflog entries if config is\n -+ * non-NULL */\n -+int reftable_stack_compact_all(struct reftable_stack *st,\n -+\t\t\t struct reftable_log_expiry_config *config);\n -+\n -+/* heuristically compact unbalanced table stack. */\n -+int reftable_stack_auto_compact(struct reftable_stack *st);\n -+\n -+/* convenience function to read a single ref. Returns < 0 for error, 0\n -+ for success, and 1 if ref not found. */\n -+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n -+\t\t\t struct reftable_ref_record *ref);\n -+\n -+/* convenience function to read a single log. Returns < 0 for error, 0\n -+ for success, and 1 if ref not found. */\n -+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n -+\t\t\t struct reftable_log_record *log);\n -+\n -+/* statistics on past compactions. */\n -+struct reftable_compaction_stats {\n -+\tuint64_t bytes; /* total number of bytes written */\n -+\tuint64_t entries_written; /* total number of entries written, including\n -+\t\t\t\t failures. */\n -+\tint attempts; /* how often we tried to compact */\n -+\tint failures; /* failures happen on concurrent updates */\n -+};\n -+\n -+/* return statistics for compaction up till now. */\n -+struct reftable_compaction_stats *\n -+reftable_stack_compaction_stats(struct reftable_stack *st);\n -+\n +#endif\n 3: 01a669a731 < -: ---------- vcxproj: adjust for the reftable changes\n 5: 4190da597e ! 5: 987d1d186f reftable: utility functions\n @@ Commit message\n \n This commit provides basic utility classes for the reftable library.\n \n - Since the reftable library must compile standalone, there may be some overlap\n - with git-core utility functions.\n -\n Signed-off-by: Han-Wen Nienhuys \n \n ## Makefile ##\n @@ Makefile: TEST_SHELL_PATH = $(SHELL_PATH)\n +REFTABLE_LIB = reftable/libreftable.a\n +REFTABLE_TEST_LIB = reftable/libreftable_test.a\n \n - GENERATED_H += config-list.h\n GENERATED_H += command-list.h\n + GENERATED_H += config-list.h\n @@ Makefile: THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n @@ Makefile: XDIFF_OBJS += xdiff/xpatience.o\n XDIFF_OBJS += xdiff/xutils.o\n \n +REFTABLE_OBJS += reftable/basics.o\n -+REFTABLE_OBJS += reftable/blocksource.o\n ++REFTABLE_OBJS += reftable/error.o\n +REFTABLE_OBJS += reftable/publicbasics.o\n -+REFTABLE_OBJS += reftable/compat.o\n -+REFTABLE_OBJS += reftable/strbuf.o\n +\n -+REFTABLE_TEST_OBJS += reftable/strbuf_test.o\n +REFTABLE_TEST_OBJS += reftable/test_framework.o\n ++REFTABLE_TEST_OBJS += reftable/basics_test.o\n +\n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \\\n @@ reftable/basics.c (new)\n +\t}\n +\n +\treturn *a == *b;\n ++}\n ++\n ++int common_prefix_size(struct strbuf *a, struct strbuf *b)\n ++{\n ++\tint p = 0;\n ++\twhile (p < a->len && p < b->len) {\n ++\t\tif (a->buf[p] != b->buf[p]) {\n ++\t\t\tbreak;\n ++\t\t}\n ++\t\tp++;\n ++\t}\n ++\n ++\treturn p;\n +}\n \n ## reftable/basics.h (new) ##\n @@ reftable/basics.h (new)\n +#ifndef BASICS_H\n +#define BASICS_H\n +\n ++/*\n ++ * miscellaneous utilities that are not provided by Git.\n ++ */\n ++\n +#include \"system.h\"\n +\n +/* Bigendian en/decoding of integers */\n @@ reftable/basics.h (new)\n +void reftable_free(void *p);\n +void *reftable_calloc(size_t sz);\n +\n ++/* Find the longest shared prefix size of `a` and `b` */\n ++struct strbuf;\n ++int common_prefix_size(struct strbuf *a, struct strbuf *b);\n ++\n +#endif\n \n - ## reftable/blocksource.c (new) ##\n + ## reftable/basics_test.c (new) ##\n @@\n +/*\n +Copyright 2020 Google LLC\n @@ reftable/blocksource.c (new)\n +#include \"system.h\"\n +\n +#include \"basics.h\"\n -+#include \"blocksource.h\"\n -+#include \"strbuf.h\"\n -+#include \"reftable.h\"\n -+\n -+static void strbuf_return_block(void *b, struct reftable_block *dest)\n -+{\n -+\tmemset(dest->data, 0xff, dest->len);\n -+\treftable_free(dest->data);\n -+}\n -+\n -+static void strbuf_close(void *b)\n -+{\n -+}\n -+\n -+static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n -+\t\t\t uint32_t size)\n -+{\n -+\tstruct strbuf *b = (struct strbuf *)v;\n -+\tassert(off + size <= b->len);\n -+\tdest->data = reftable_calloc(size);\n -+\tmemcpy(dest->data, b->buf + off, size);\n -+\tdest->len = size;\n -+\treturn size;\n -+}\n -+\n -+static uint64_t strbuf_size(void *b)\n -+{\n -+\treturn ((struct strbuf *)b)->len;\n -+}\n -+\n -+static struct reftable_block_source_vtable strbuf_vtable = {\n -+\t.size = &strbuf_size,\n -+\t.read_block = &strbuf_read_block,\n -+\t.return_block = &strbuf_return_block,\n -+\t.close = &strbuf_close,\n -+};\n -+\n -+void block_source_from_strbuf(struct reftable_block_source *bs,\n -+\t\t\t struct strbuf *buf)\n -+{\n -+\tassert(bs->ops == NULL);\n -+\tbs->ops = &strbuf_vtable;\n -+\tbs->arg = buf;\n -+}\n -+\n -+static void malloc_return_block(void *b, struct reftable_block *dest)\n -+{\n -+\tmemset(dest->data, 0xff, dest->len);\n -+\treftable_free(dest->data);\n -+}\n -+\n -+static struct reftable_block_source_vtable malloc_vtable = {\n -+\t.return_block = &malloc_return_block,\n -+};\n -+\n -+static struct reftable_block_source malloc_block_source_instance = {\n -+\t.ops = &malloc_vtable,\n -+};\n -+\n -+struct reftable_block_source malloc_block_source(void)\n -+{\n -+\treturn malloc_block_source_instance;\n -+}\n ++#include \"test_framework.h\"\n ++#include \"reftable-tests.h\"\n +\n -+struct file_block_source {\n -+\tint fd;\n -+\tuint64_t size;\n ++struct binsearch_args {\n ++\tint key;\n ++\tint *arr;\n +};\n +\n -+static uint64_t file_size(void *b)\n ++static int binsearch_func(size_t i, void *void_args)\n +{\n -+\treturn ((struct file_block_source *)b)->size;\n -+}\n ++\tstruct binsearch_args *args = (struct binsearch_args *)void_args;\n +\n -+static void file_return_block(void *b, struct reftable_block *dest)\n -+{\n -+\tmemset(dest->data, 0xff, dest->len);\n -+\treftable_free(dest->data);\n ++\treturn args->key < args->arr[i];\n +}\n +\n -+static void file_close(void *b)\n -+{\n -+\tint fd = ((struct file_block_source *)b)->fd;\n -+\tif (fd > 0) {\n -+\t\tclose(fd);\n -+\t\t((struct file_block_source *)b)->fd = 0;\n -+\t}\n -+\n -+\treftable_free(b);\n -+}\n -+\n -+static int file_read_block(void *v, struct reftable_block *dest, uint64_t off,\n -+\t\t\t uint32_t size)\n -+{\n -+\tstruct file_block_source *b = (struct file_block_source *)v;\n -+\tassert(off + size <= b->size);\n -+\tdest->data = reftable_malloc(size);\n -+\tif (pread(b->fd, dest->data, size, off) != size)\n -+\t\treturn -1;\n -+\tdest->len = size;\n -+\treturn size;\n -+}\n -+\n -+static struct reftable_block_source_vtable file_vtable = {\n -+\t.size = &file_size,\n -+\t.read_block = &file_read_block,\n -+\t.return_block = &file_return_block,\n -+\t.close = &file_close,\n -+};\n -+\n -+int reftable_block_source_from_file(struct reftable_block_source *bs,\n -+\t\t\t\t const char *name)\n ++static void test_binsearch(void)\n +{\n -+\tstruct stat st = { 0 };\n -+\tint err = 0;\n -+\tint fd = open(name, O_RDONLY);\n -+\tstruct file_block_source *p = NULL;\n -+\tif (fd < 0) {\n -+\t\tif (errno == ENOENT) {\n -+\t\t\treturn REFTABLE_NOT_EXIST_ERROR;\n -+\t\t}\n -+\t\treturn -1;\n -+\t}\n -+\n -+\terr = fstat(fd, &st);\n -+\tif (err < 0)\n -+\t\treturn -1;\n -+\n -+\tp = reftable_calloc(sizeof(struct file_block_source));\n -+\tp->size = st.st_size;\n -+\tp->fd = fd;\n -+\n -+\tassert(bs->ops == NULL);\n -+\tbs->ops = &file_vtable;\n -+\tbs->arg = p;\n -+\treturn 0;\n -+}\n -\n - ## reftable/blocksource.h (new) ##\n -@@\n -+/*\n -+Copyright 2020 Google LLC\n -+\n -+Use of this source code is governed by a BSD-style\n -+license that can be found in the LICENSE file or at\n -+https://developers.google.com/open-source/licenses/bsd\n -+*/\n -+\n -+#ifndef BLOCKSOURCE_H\n -+#define BLOCKSOURCE_H\n -+\n -+#include \"strbuf.h\"\n -+\n -+struct reftable_block_source;\n -+\n -+/* Create an in-memory block source for reading reftables */\n -+void block_source_from_strbuf(struct reftable_block_source *bs,\n -+\t\t\t struct strbuf *buf);\n ++\tint arr[] = { 2, 4, 6, 8, 10 };\n ++\tsize_t sz = ARRAY_SIZE(arr);\n ++\tstruct binsearch_args args = {\n ++\t\t.arr = arr,\n ++\t};\n +\n -+struct reftable_block_source malloc_block_source(void);\n -+\n -+#endif\n -\n - ## reftable/compat.c (new) ##\n -@@\n -+/*\n -+Copyright 2020 Google LLC\n -+\n -+Use of this source code is governed by a BSD-style\n -+license that can be found in the LICENSE file or at\n -+https://developers.google.com/open-source/licenses/bsd\n -+\n -+*/\n -+\n -+/* compat.c - compatibility functions for standalone compilation */\n -+\n -+#include \"system.h\"\n -+#include \"basics.h\"\n -+\n -+#ifdef REFTABLE_STANDALONE\n -+\n -+#include \n -+\n -+void put_be32(void *p, uint32_t i)\n -+{\n -+\tuint8_t *out = (uint8_t *)p;\n -+\n -+\tout[0] = (uint8_t)((i >> 24) & 0xff);\n -+\tout[1] = (uint8_t)((i >> 16) & 0xff);\n -+\tout[2] = (uint8_t)((i >> 8) & 0xff);\n -+\tout[3] = (uint8_t)((i)&0xff);\n -+}\n -+\n -+uint32_t get_be32(uint8_t *in)\n -+{\n -+\treturn (uint32_t)(in[0]) << 24 | (uint32_t)(in[1]) << 16 |\n -+\t (uint32_t)(in[2]) << 8 | (uint32_t)(in[3]);\n -+}\n -+\n -+void put_be64(void *p, uint64_t v)\n -+{\n -+\tuint8_t *out = (uint8_t *)p;\n -+\tint i = sizeof(uint64_t);\n -+\twhile (i--) {\n -+\t\tout[i] = (uint8_t)(v & 0xff);\n -+\t\tv >>= 8;\n -+\t}\n -+}\n -+\n -+uint64_t get_be64(void *out)\n -+{\n -+\tuint8_t *bytes = (uint8_t *)out;\n -+\tuint64_t v = 0;\n +\tint i = 0;\n -+\tfor (i = 0; i < sizeof(uint64_t); i++) {\n -+\t\tv = (v << 8) | (uint8_t)(bytes[i] & 0xff);\n -+\t}\n -+\treturn v;\n -+}\n -+\n -+uint16_t get_be16(uint8_t *in)\n -+{\n -+\treturn (uint32_t)(in[0]) << 8 | (uint32_t)(in[1]);\n -+}\n -+\n -+char *xstrdup(const char *s)\n -+{\n -+\tint l = strlen(s);\n -+\tchar *dest = (char *)reftable_malloc(l + 1);\n -+\tstrncpy(dest, s, l + 1);\n -+\treturn dest;\n -+}\n -+\n -+void sleep_millisec(int millisecs)\n -+{\n -+\tusleep(millisecs * 1000);\n -+}\n -+\n -+void reftable_clear_dir(const char *dirname)\n -+{\n -+\tDIR *dir = opendir(dirname);\n -+\tstruct dirent *ent = NULL;\n -+\tassert(dir);\n -+\twhile ((ent = readdir(dir)) != NULL) {\n -+\t\tunlinkat(dirfd(dir), ent->d_name, 0);\n ++\tfor (i = 1; i < 11; i++) {\n ++\t\tint res;\n ++\t\targs.key = i;\n ++\t\tres = binsearch(sz, &binsearch_func, &args);\n ++\n ++\t\tif (res < sz) {\n ++\t\t\tEXPECT(args.key < arr[res]);\n ++\t\t\tif (res > 0) {\n ++\t\t\t\tEXPECT(args.key >= arr[res - 1]);\n ++\t\t\t}\n ++\t\t} else {\n ++\t\t\tEXPECT(args.key == 10 || args.key == 11);\n ++\t\t}\n +\t}\n -+\tclosedir(dir);\n -+\trmdir(dirname);\n +}\n +\n -+#else\n -+\n -+#include \"../dir.h\"\n -+\n -+void reftable_clear_dir(const char *dirname)\n ++int basics_test_main(int argc, const char *argv[])\n +{\n -+\tstruct strbuf path = STRBUF_INIT;\n -+\tstrbuf_addstr(&path, dirname);\n -+\tremove_dir_recursively(&path, 0);\n -+\tstrbuf_release(&path);\n -+}\n -+\n -+#endif\n -+\n -+int hash_size(uint32_t id)\n -+{\n -+\tswitch (id) {\n -+\tcase 0:\n -+\tcase SHA1_ID:\n -+\t\treturn SHA1_SIZE;\n -+\tcase SHA256_ID:\n -+\t\treturn SHA256_SIZE;\n -+\t}\n -+\tabort();\n ++\ttest_binsearch();\n ++\treturn 0;\n +}\n \n ## reftable/compat.h (new) ##\n @@ reftable/publicbasics.c (new)\n +https://developers.google.com/open-source/licenses/bsd\n +*/\n +\n -+#include \"reftable.h\"\n ++#include \"reftable-malloc.h\"\n +\n +#include \"basics.h\"\n +#include \"system.h\"\n +\n -+const char *reftable_error_str(int err)\n -+{\n -+\tstatic char buf[250];\n -+\tswitch (err) {\n -+\tcase REFTABLE_IO_ERROR:\n -+\t\treturn \"I/O error\";\n -+\tcase REFTABLE_FORMAT_ERROR:\n -+\t\treturn \"corrupt reftable file\";\n -+\tcase REFTABLE_NOT_EXIST_ERROR:\n -+\t\treturn \"file does not exist\";\n -+\tcase REFTABLE_LOCK_ERROR:\n -+\t\treturn \"data is outdated\";\n -+\tcase REFTABLE_API_ERROR:\n -+\t\treturn \"misuse of the reftable API\";\n -+\tcase REFTABLE_ZLIB_ERROR:\n -+\t\treturn \"zlib failure\";\n -+\tcase REFTABLE_NAME_CONFLICT:\n -+\t\treturn \"file/directory conflict\";\n -+\tcase REFTABLE_REFNAME_ERROR:\n -+\t\treturn \"invalid refname\";\n -+\tcase -1:\n -+\t\treturn \"general error\";\n -+\tdefault:\n -+\t\tsnprintf(buf, sizeof(buf), \"unknown error code %d\", err);\n -+\t\treturn buf;\n -+\t}\n -+}\n -+\n -+int reftable_error_to_errno(int err)\n -+{\n -+\tswitch (err) {\n -+\tcase REFTABLE_IO_ERROR:\n -+\t\treturn EIO;\n -+\tcase REFTABLE_FORMAT_ERROR:\n -+\t\treturn EFAULT;\n -+\tcase REFTABLE_NOT_EXIST_ERROR:\n -+\t\treturn ENOENT;\n -+\tcase REFTABLE_LOCK_ERROR:\n -+\t\treturn EBUSY;\n -+\tcase REFTABLE_API_ERROR:\n -+\t\treturn EINVAL;\n -+\tcase REFTABLE_ZLIB_ERROR:\n -+\t\treturn EDOM;\n -+\tdefault:\n -+\t\treturn ERANGE;\n -+\t}\n -+}\n -+\n +static void *(*reftable_malloc_ptr)(size_t sz) = &malloc;\n +static void *(*reftable_realloc_ptr)(void *, size_t) = &realloc;\n +static void (*reftable_free_ptr)(void *) = &free;\n @@ reftable/publicbasics.c (new)\n +\treftable_free_ptr = free;\n +}\n +\n -+int reftable_fd_write(void *arg, const void *data, size_t sz)\n ++int hash_size(uint32_t id)\n +{\n -+\tint *fdp = (int *)arg;\n -+\treturn write(*fdp, data, sz);\n ++\tswitch (id) {\n ++\tcase 0:\n ++\tcase SHA1_ID:\n ++\t\treturn SHA1_SIZE;\n ++\tcase SHA256_ID:\n ++\t\treturn SHA256_SIZE;\n ++\t}\n ++\tabort();\n +}\n \n - ## reftable/reftable-tests.h (new) ##\n + ## reftable/reftable-malloc.h (new) ##\n @@\n +/*\n +Copyright 2020 Google LLC\n @@ reftable/reftable-tests.h (new)\n +https://developers.google.com/open-source/licenses/bsd\n +*/\n +\n -+#ifndef REFTABLE_TESTS_H\n -+#define REFTABLE_TESTS_H\n ++#ifndef REFTABLE_H\n ++#define REFTABLE_H\n +\n -+int block_test_main(int argc, const char **argv);\n -+int merged_test_main(int argc, const char **argv);\n -+int record_test_main(int argc, const char **argv);\n -+int refname_test_main(int argc, const char **argv);\n -+int reftable_test_main(int argc, const char **argv);\n -+int strbuf_test_main(int argc, const char **argv);\n -+int stack_test_main(int argc, const char **argv);\n -+int tree_test_main(int argc, const char **argv);\n -+int reftable_dump_main(int argc, char *const *argv);\n ++#include \n ++\n ++/*\n ++ Overrides the functions to use for memory management.\n ++ */\n ++void reftable_set_alloc(void *(*malloc)(size_t),\n ++\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n +\n +#endif\n \n - ## reftable/strbuf.c (new) ##\n + ## reftable/reftable-tests.h (new) ##\n @@\n +/*\n +Copyright 2020 Google LLC\n @@ reftable/strbuf.c (new)\n +https://developers.google.com/open-source/licenses/bsd\n +*/\n +\n -+#include \"strbuf.h\"\n -+\n -+#ifdef REFTABLE_STANDALONE\n -+\n -+void strbuf_init(struct strbuf *s, size_t alloc)\n -+{\n -+\tstruct strbuf empty = STRBUF_INIT;\n -+\t*s = empty;\n -+}\n -+\n -+void strbuf_grow(struct strbuf *s, size_t extra)\n -+{\n -+\tsize_t newcap = s->len + extra + 1;\n -+\tif (newcap > s->cap) {\n -+\t\ts->buf = reftable_realloc(s->buf, newcap);\n -+\t\ts->cap = newcap;\n -+\t}\n -+}\n -+\n -+static void strbuf_resize(struct strbuf *s, int l)\n -+{\n -+\tint zl = l + 1; /* one uint8_t for 0 termination. */\n -+\tassert(s->canary == STRBUF_CANARY);\n -+\tif (s->cap < zl) {\n -+\t\tint c = s->cap * 2;\n -+\t\tif (c < zl) {\n -+\t\t\tc = zl;\n -+\t\t}\n -+\t\ts->cap = c;\n -+\t\ts->buf = reftable_realloc(s->buf, s->cap);\n -+\t}\n -+\ts->len = l;\n -+\ts->buf[l] = 0;\n -+}\n -+\n -+void strbuf_setlen(struct strbuf *s, size_t l)\n -+{\n -+\tassert(s->cap >= l + 1);\n -+\ts->len = l;\n -+\ts->buf[l] = 0;\n -+}\n -+\n -+void strbuf_reset(struct strbuf *s)\n -+{\n -+\tstrbuf_resize(s, 0);\n -+}\n -+\n -+void strbuf_addstr(struct strbuf *d, const char *s)\n -+{\n -+\tint l1 = d->len;\n -+\tint l2 = strlen(s);\n -+\tassert(d->canary == STRBUF_CANARY);\n -+\n -+\tstrbuf_resize(d, l2 + l1);\n -+\tmemcpy(d->buf + l1, s, l2);\n -+}\n -+\n -+void strbuf_addbuf(struct strbuf *s, struct strbuf *a)\n -+{\n -+\tint end = s->len;\n -+\tassert(s->canary == STRBUF_CANARY);\n -+\tstrbuf_resize(s, s->len + a->len);\n -+\tmemcpy(s->buf + end, a->buf, a->len);\n -+}\n -+\n -+char *strbuf_detach(struct strbuf *s, size_t *sz)\n -+{\n -+\tchar *p = NULL;\n -+\tp = (char *)s->buf;\n -+\tif (sz)\n -+\t\t*sz = s->len;\n -+\ts->buf = NULL;\n -+\ts->cap = 0;\n -+\ts->len = 0;\n -+\treturn p;\n -+}\n -+\n -+void strbuf_release(struct strbuf *s)\n -+{\n -+\tassert(s->canary == STRBUF_CANARY);\n -+\ts->cap = 0;\n -+\ts->len = 0;\n -+\treftable_free(s->buf);\n -+\ts->buf = NULL;\n -+}\n -+\n -+int strbuf_cmp(const struct strbuf *a, const struct strbuf *b)\n -+{\n -+\tint min = a->len < b->len ? a->len : b->len;\n -+\tint res = memcmp(a->buf, b->buf, min);\n -+\tassert(a->canary == STRBUF_CANARY);\n -+\tassert(b->canary == STRBUF_CANARY);\n -+\tif (res != 0)\n -+\t\treturn res;\n -+\tif (a->len < b->len)\n -+\t\treturn -1;\n -+\telse if (a->len > b->len)\n -+\t\treturn 1;\n -+\telse\n -+\t\treturn 0;\n -+}\n ++#ifndef REFTABLE_TESTS_H\n ++#define REFTABLE_TESTS_H\n +\n -+int strbuf_add(struct strbuf *b, const void *data, size_t sz)\n -+{\n -+\tassert(b->canary == STRBUF_CANARY);\n -+\tstrbuf_grow(b, sz);\n -+\tmemcpy(b->buf + b->len, data, sz);\n -+\tb->len += sz;\n -+\tb->buf[b->len] = 0;\n -+\treturn sz;\n -+}\n ++int basics_test_main(int argc, const char **argv);\n ++int block_test_main(int argc, const char **argv);\n ++int merged_test_main(int argc, const char **argv);\n ++int record_test_main(int argc, const char **argv);\n ++int refname_test_main(int argc, const char **argv);\n ++int reftable_test_main(int argc, const char **argv);\n ++int stack_test_main(int argc, const char **argv);\n ++int tree_test_main(int argc, const char **argv);\n ++int reftable_dump_main(int argc, char *const *argv);\n +\n +#endif\n -+\n -+int strbuf_add_void(void *b, const void *data, size_t sz)\n -+{\n -+\tstrbuf_add((struct strbuf *)b, data, sz);\n -+\treturn sz;\n -+}\n -+\n -+int common_prefix_size(struct strbuf *a, struct strbuf *b)\n -+{\n -+\tint p = 0;\n -+\twhile (p < a->len && p < b->len) {\n -+\t\tif (a->buf[p] != b->buf[p]) {\n -+\t\t\tbreak;\n -+\t\t}\n -+\t\tp++;\n -+\t}\n -+\n -+\treturn p;\n -+}\n -+\n -+struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n \n - ## reftable/strbuf.h (new) ##\n + ## reftable/system.h (new) ##\n @@\n +/*\n +Copyright 2020 Google LLC\n @@ reftable/strbuf.h (new)\n +https://developers.google.com/open-source/licenses/bsd\n +*/\n +\n -+#ifndef SLICE_H\n -+#define SLICE_H\n -+\n -+#ifdef REFTABLE_STANDALONE\n -+\n -+#include \"basics.h\"\n -+\n -+/*\n -+ Provides a bounds-checked, growable byte ranges. To use, initialize as \"strbuf\n -+ x = STRBUF_INIT;\"\n -+ */\n -+struct strbuf {\n -+\tsize_t len;\n -+\tsize_t cap;\n -+\tchar *buf;\n -+\n -+\t/* Used to enforce initialization with STRBUF_INIT */\n -+\tuint8_t canary;\n -+};\n -+\n -+#define STRBUF_CANARY 0x42\n -+#define STRBUF_INIT \\\n -+\t{ \\\n -+\t\t0, 0, NULL, STRBUF_CANARY \\\n -+\t}\n -+\n -+void strbuf_addstr(struct strbuf *dest, const char *src);\n -+\n -+/* Deallocate and clear strbuf */\n -+void strbuf_release(struct strbuf *strbuf);\n -+\n -+/* Set strbuf to 0 length, but retain buffer. */\n -+void strbuf_reset(struct strbuf *strbuf);\n -+\n -+/* Initializes a strbuf. Accepts a strbuf with random garbage. */\n -+void strbuf_init(struct strbuf *strbuf, size_t alloc);\n -+\n -+/* Return `buf`, clearing out `s`. Optionally return len (not cap) in `sz`. */\n -+char *strbuf_detach(struct strbuf *s, size_t *sz);\n -+\n -+/* Set length of the slace to `l`, but don't reallocated. */\n -+void strbuf_setlen(struct strbuf *s, size_t l);\n -+\n -+/* Ensure `l` bytes beyond current length are available */\n -+void strbuf_grow(struct strbuf *s, size_t l);\n -+\n -+/* Signed comparison */\n -+int strbuf_cmp(const struct strbuf *a, const struct strbuf *b);\n -+\n -+/* Append `data` to the `dest` strbuf. */\n -+int strbuf_add(struct strbuf *dest, const void *data, size_t sz);\n ++#ifndef SYSTEM_H\n ++#define SYSTEM_H\n +\n -+/* Append `add` to `dest. */\n -+void strbuf_addbuf(struct strbuf *dest, struct strbuf *add);\n ++#include \"git-compat-util.h\"\n ++#include \"strbuf.h\"\n +\n -+#else\n ++#include \n +\n -+#include \"../git-compat-util.h\"\n -+#include \"../strbuf.h\"\n ++struct strbuf;\n ++/* In git, this is declared in dir.h */\n ++int remove_dir_recursively(struct strbuf *path, int flags);\n +\n -+#endif\n -+\n -+extern struct strbuf reftable_empty_strbuf;\n ++#define SHA1_ID 0x73686131\n ++#define SHA256_ID 0x73323536\n ++#define SHA1_SIZE 20\n ++#define SHA256_SIZE 32\n +\n -+/* Like strbuf_add, but suitable for passing to reftable_new_writer\n ++/* This is uncompress2, which is only available in zlib as of 2017.\n + */\n -+int strbuf_add_void(void *b, const void *data, size_t sz);\n -+\n -+/* Find the longest shared prefix size of `a` and `b` */\n -+int common_prefix_size(struct strbuf *a, struct strbuf *b);\n ++int uncompress_return_consumed(Bytef *dest, uLongf *destLen,\n ++\t\t\t const Bytef *source, uLong *sourceLen);\n ++int hash_size(uint32_t id);\n +\n +#endif\n \n - ## reftable/strbuf_test.c (new) ##\n + ## reftable/test_framework.c (new) ##\n @@\n +/*\n +Copyright 2020 Google LLC\n @@ reftable/strbuf_test.c (new)\n +https://developers.google.com/open-source/licenses/bsd\n +*/\n +\n -+#include \"strbuf.h\"\n -+\n +#include \"system.h\"\n ++#include \"test_framework.h\"\n +\n +#include \"basics.h\"\n -+#include \"test_framework.h\"\n -+#include \"reftable-tests.h\"\n +\n -+static void test_strbuf(void)\n ++void set_test_hash(uint8_t *p, int i)\n +{\n -+\tstruct strbuf s = STRBUF_INIT;\n -+\tstruct strbuf t = STRBUF_INIT;\n -+\n -+\tstrbuf_addstr(&s, \"abc\");\n -+\tassert(0 == strcmp(\"abc\", s.buf));\n -+\n -+\tstrbuf_addstr(&t, \"pqr\");\n -+\tstrbuf_addbuf(&s, &t);\n -+\tassert(0 == strcmp(\"abcpqr\", s.buf));\n -+\n -+\tstrbuf_release(&s);\n -+\tstrbuf_release(&t);\n ++\tmemset(p, (uint8_t)i, hash_size(SHA1_ID));\n +}\n +\n -+int strbuf_test_main(int argc, const char *argv[])\n ++int strbuf_add_void(void *b, const void *data, size_t sz)\n +{\n -+\tadd_test_case(\"test_strbuf\", &test_strbuf);\n -+\treturn test_main(argc, argv);\n ++\tstrbuf_add((struct strbuf *)b, data, sz);\n ++\treturn sz;\n +}\n \n - ## reftable/system.h (new) ##\n + ## reftable/test_framework.h (new) ##\n @@\n +/*\n +Copyright 2020 Google LLC\n @@ reftable/system.h (new)\n +https://developers.google.com/open-source/licenses/bsd\n +*/\n +\n -+#ifndef SYSTEM_H\n -+#define SYSTEM_H\n -+\n -+#ifndef REFTABLE_STANDALONE\n -+\n -+#include \"git-compat-util.h\"\n -+#include \"cache.h\"\n -+#include \n -+\n -+#else\n -+\n -+#include \n -+#include \n -+#include \n -+#include \n -+#include \n -+#include \n -+#include \n -+#include \n -+#include \n -+#include \n -+#include \n -+#include \n -+#include \n ++#ifndef TEST_FRAMEWORK_H\n ++#define TEST_FRAMEWORK_H\n +\n -+#include \"compat.h\"\n ++#include \"system.h\"\n ++#include \"reftable-error.h\"\n ++\n ++#define EXPECT_ERR(c) \\\n ++\tif (c != 0) { \\\n ++\t\tfflush(stderr); \\\n ++\t\tfflush(stdout); \\\n ++\t\tfprintf(stderr, \"%s: %d: error == %d (%s), want 0\\n\", \\\n ++\t\t\t__FILE__, __LINE__, c, reftable_error_str(c)); \\\n ++\t\tabort(); \\\n ++\t}\n +\n -+#endif /* REFTABLE_STANDALONE */\n ++#define EXPECT_STREQ(a, b) \\\n ++\tif (strcmp(a, b)) { \\\n ++\t\tfflush(stderr); \\\n ++\t\tfflush(stdout); \\\n ++\t\tfprintf(stderr, \"%s:%d: %s (%s) != %s (%s)\\n\", __FILE__, \\\n ++\t\t\t__LINE__, #a, a, #b, b); \\\n ++\t\tabort(); \\\n ++\t}\n +\n -+void reftable_clear_dir(const char *dirname);\n ++#define EXPECT(c) \\\n ++\tif (!(c)) { \\\n ++\t\tfflush(stderr); \\\n ++\t\tfflush(stdout); \\\n ++\t\tfprintf(stderr, \"%s: %d: failed assertion %s\\n\", __FILE__, \\\n ++\t\t\t__LINE__, #c); \\\n ++\t\tabort(); \\\n ++\t}\n +\n -+#define SHA1_ID 0x73686131\n -+#define SHA256_ID 0x73323536\n -+#define SHA1_SIZE 20\n -+#define SHA256_SIZE 32\n ++void set_test_hash(uint8_t *p, int i);\n +\n -+/* This is uncompress2, which is only available in zlib as of 2017.\n ++/* Like strbuf_add, but suitable for passing to reftable_new_writer\n + */\n -+int uncompress_return_consumed(Bytef *dest, uLongf *destLen,\n -+\t\t\t const Bytef *source, uLong *sourceLen);\n -+int hash_size(uint32_t id);\n ++int strbuf_add_void(void *b, const void *data, size_t sz);\n +\n +#endif\n \n @@ t/helper/test-reftable.c (new)\n +\n +int cmd__reftable(int argc, const char **argv)\n +{\n -+\tstrbuf_test_main(argc, argv);\n ++\tbasics_test_main(argc, argv);\n ++\n +\treturn 0;\n +}\n \n ## t/helper/test-tool.c ##\n @@ t/helper/test-tool.c: static struct test_cmd cmds[] = {\n + \t{ \"path-utils\", cmd__path_utils },\n + \t{ \"pkt-line\", cmd__pkt_line },\n + \t{ \"prio-queue\", cmd__prio_queue },\n +-\t{ \"proc-receive\", cmd__proc_receive},\n ++\t{ \"proc-receive\", cmd__proc_receive },\n + \t{ \"progress\", cmd__progress },\n + \t{ \"reach\", cmd__reach },\n + \t{ \"read-cache\", cmd__read_cache },\n \t{ \"read-graph\", cmd__read_graph },\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n @@ t/helper/test-tool.h: int cmd__read_cache(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__repository(int argc, const char **argv);\n int cmd__revision_walking(int argc, const char **argv);\n +\n + ## t/t0032-reftable-unittest.sh (new) ##\n +@@\n ++#!/bin/sh\n ++#\n ++# Copyright (c) 2020 Google LLC\n ++#\n ++\n ++test_description='reftable unittests'\n ++\n ++. ./test-lib.sh\n ++\n ++test_expect_success 'unittests' '\n ++ test-tool reftable\n ++'\n ++\n ++test_done\n 4: b94c5f5c61 ! 6: f71e82b995 reftable: add a barebones unittest framework\n @@ Metadata\n Author: Han-Wen Nienhuys \n \n ## Commit message ##\n - reftable: add a barebones unittest framework\n + reftable: add blocksource, an abstraction for random access reads\n +\n + The reftable format is usually used with files for storage. However, we abstract\n + away this using the blocksource data structure. This has two advantages:\n +\n + * log blocks are zlib compressed, and handling them is simplified if we can\n + discard byte segments from within the block layer.\n +\n + * for unittests, it is useful to read and write in-memory. The blocksource\n + allows us to abstract the data away from on-disk files.\n \n Signed-off-by: Han-Wen Nienhuys \n \n - ## reftable/test_framework.c (new) ##\n + ## Makefile ##\n +@@ Makefile: XDIFF_OBJS += xdiff/xutils.o\n + \n + REFTABLE_OBJS += reftable/basics.o\n + REFTABLE_OBJS += reftable/error.o\n ++REFTABLE_OBJS += reftable/blocksource.o\n + REFTABLE_OBJS += reftable/publicbasics.o\n + \n + REFTABLE_TEST_OBJS += reftable/test_framework.o\n +\n + ## reftable/blocksource.c (new) ##\n @@\n +/*\n +Copyright 2020 Google LLC\n @@ reftable/test_framework.c (new)\n +https://developers.google.com/open-source/licenses/bsd\n +*/\n +\n -+#include \"test_framework.h\"\n -+\n +#include \"system.h\"\n ++\n +#include \"basics.h\"\n ++#include \"blocksource.h\"\n ++#include \"reftable-blocksource.h\"\n ++#include \"reftable-error.h\"\n +\n -+static struct test_case **test_cases;\n -+static int test_case_len;\n -+static int test_case_cap;\n ++static void strbuf_return_block(void *b, struct reftable_block *dest)\n ++{\n ++\tmemset(dest->data, 0xff, dest->len);\n ++\treftable_free(dest->data);\n ++}\n +\n -+static struct test_case *new_test_case(const char *name, void (*testfunc)(void))\n ++static void strbuf_close(void *b)\n +{\n -+\tstruct test_case *tc = reftable_malloc(sizeof(struct test_case));\n -+\ttc->name = name;\n -+\ttc->testfunc = testfunc;\n -+\treturn tc;\n +}\n +\n -+struct test_case *add_test_case(const char *name, void (*testfunc)(void))\n ++static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n ++\t\t\t uint32_t size)\n +{\n -+\tstruct test_case *tc = new_test_case(name, testfunc);\n -+\tif (test_case_len == test_case_cap) {\n -+\t\ttest_case_cap = 2 * test_case_cap + 1;\n -+\t\ttest_cases = reftable_realloc(\n -+\t\t\ttest_cases, sizeof(struct test_case) * test_case_cap);\n -+\t}\n ++\tstruct strbuf *b = (struct strbuf *)v;\n ++\tassert(off + size <= b->len);\n ++\tdest->data = reftable_calloc(size);\n ++\tmemcpy(dest->data, b->buf + off, size);\n ++\tdest->len = size;\n ++\treturn size;\n ++}\n +\n -+\ttest_cases[test_case_len++] = tc;\n -+\treturn tc;\n ++static uint64_t strbuf_size(void *b)\n ++{\n ++\treturn ((struct strbuf *)b)->len;\n +}\n +\n -+int test_main(int argc, const char *argv[])\n ++static struct reftable_block_source_vtable strbuf_vtable = {\n ++\t.size = &strbuf_size,\n ++\t.read_block = &strbuf_read_block,\n ++\t.return_block = &strbuf_return_block,\n ++\t.close = &strbuf_close,\n ++};\n ++\n ++void block_source_from_strbuf(struct reftable_block_source *bs,\n ++\t\t\t struct strbuf *buf)\n +{\n -+\tconst char *filter = NULL;\n -+\tint i = 0;\n -+\tif (argc > 1) {\n -+\t\tfilter = argv[1];\n -+\t}\n ++\tassert(bs->ops == NULL);\n ++\tbs->ops = &strbuf_vtable;\n ++\tbs->arg = buf;\n ++}\n +\n -+\tfor (i = 0; i < test_case_len; i++) {\n -+\t\tconst char *name = test_cases[i]->name;\n -+\t\tif (filter == NULL || strstr(name, filter) != NULL) {\n -+\t\t\tprintf(\"case %s\\n\", name);\n -+\t\t\ttest_cases[i]->testfunc();\n -+\t\t} else {\n -+\t\t\tprintf(\"skip %s\\n\", name);\n -+\t\t}\n ++static void malloc_return_block(void *b, struct reftable_block *dest)\n ++{\n ++\tmemset(dest->data, 0xff, dest->len);\n ++\treftable_free(dest->data);\n ++}\n ++\n ++static struct reftable_block_source_vtable malloc_vtable = {\n ++\t.return_block = &malloc_return_block,\n ++};\n ++\n ++static struct reftable_block_source malloc_block_source_instance = {\n ++\t.ops = &malloc_vtable,\n ++};\n +\n -+\t\treftable_free(test_cases[i]);\n ++struct reftable_block_source malloc_block_source(void)\n ++{\n ++\treturn malloc_block_source_instance;\n ++}\n ++\n ++struct file_block_source {\n ++\tint fd;\n ++\tuint64_t size;\n ++};\n ++\n ++static uint64_t file_size(void *b)\n ++{\n ++\treturn ((struct file_block_source *)b)->size;\n ++}\n ++\n ++static void file_return_block(void *b, struct reftable_block *dest)\n ++{\n ++\tmemset(dest->data, 0xff, dest->len);\n ++\treftable_free(dest->data);\n ++}\n ++\n ++static void file_close(void *b)\n ++{\n ++\tint fd = ((struct file_block_source *)b)->fd;\n ++\tif (fd > 0) {\n ++\t\tclose(fd);\n ++\t\t((struct file_block_source *)b)->fd = 0;\n +\t}\n -+\treftable_free(test_cases);\n -+\ttest_cases = NULL;\n -+\ttest_case_len = 0;\n -+\ttest_case_cap = 0;\n -+\treturn 0;\n ++\n ++\treftable_free(b);\n +}\n +\n -+void set_test_hash(uint8_t *p, int i)\n ++static int file_read_block(void *v, struct reftable_block *dest, uint64_t off,\n ++\t\t\t uint32_t size)\n +{\n -+\tmemset(p, (uint8_t)i, hash_size(SHA1_ID));\n ++\tstruct file_block_source *b = (struct file_block_source *)v;\n ++\tassert(off + size <= b->size);\n ++\tdest->data = reftable_malloc(size);\n ++\tif (pread(b->fd, dest->data, size, off) != size)\n ++\t\treturn -1;\n ++\tdest->len = size;\n ++\treturn size;\n ++}\n ++\n ++static struct reftable_block_source_vtable file_vtable = {\n ++\t.size = &file_size,\n ++\t.read_block = &file_read_block,\n ++\t.return_block = &file_return_block,\n ++\t.close = &file_close,\n ++};\n ++\n ++int reftable_block_source_from_file(struct reftable_block_source *bs,\n ++\t\t\t\t const char *name)\n ++{\n ++\tstruct stat st = { 0 };\n ++\tint err = 0;\n ++\tint fd = open(name, O_RDONLY);\n ++\tstruct file_block_source *p = NULL;\n ++\tif (fd < 0) {\n ++\t\tif (errno == ENOENT) {\n ++\t\t\treturn REFTABLE_NOT_EXIST_ERROR;\n ++\t\t}\n ++\t\treturn -1;\n ++\t}\n ++\n ++\terr = fstat(fd, &st);\n ++\tif (err < 0)\n ++\t\treturn -1;\n ++\n ++\tp = reftable_calloc(sizeof(struct file_block_source));\n ++\tp->size = st.st_size;\n ++\tp->fd = fd;\n ++\n ++\tassert(bs->ops == NULL);\n ++\tbs->ops = &file_vtable;\n ++\tbs->arg = p;\n ++\treturn 0;\n +}\n \n - ## reftable/test_framework.h (new) ##\n + ## reftable/blocksource.h (new) ##\n @@\n +/*\n +Copyright 2020 Google LLC\n @@ reftable/test_framework.h (new)\n +https://developers.google.com/open-source/licenses/bsd\n +*/\n +\n -+#ifndef TEST_FRAMEWORK_H\n -+#define TEST_FRAMEWORK_H\n ++#ifndef BLOCKSOURCE_H\n ++#define BLOCKSOURCE_H\n +\n +#include \"system.h\"\n +\n -+#ifdef NDEBUG\n -+#undef NDEBUG\n -+#endif\n ++struct reftable_block_source;\n ++\n ++/* Create an in-memory block source for reading reftables */\n ++void block_source_from_strbuf(struct reftable_block_source *bs,\n ++\t\t\t struct strbuf *buf);\n ++\n ++struct reftable_block_source malloc_block_source(void);\n +\n -+#ifdef assert\n -+#undef assert\n +#endif\n +\n + ## reftable/reftable-blocksource.h (new) ##\n +@@\n ++/*\n ++Copyright 2020 Google LLC\n +\n -+#define assert_err(c) \\\n -+\tif (c != 0) { \\\n -+\t\tfflush(stderr); \\\n -+\t\tfflush(stdout); \\\n -+\t\tfprintf(stderr, \"%s: %d: error == %d (%s), want 0\\n\", \\\n -+\t\t\t__FILE__, __LINE__, c, reftable_error_str(c)); \\\n -+\t\tabort(); \\\n -+\t}\n ++Use of this source code is governed by a BSD-style\n ++license that can be found in the LICENSE file or at\n ++https://developers.google.com/open-source/licenses/bsd\n ++*/\n +\n -+#define assert_streq(a, b) \\\n -+\tif (strcmp(a, b)) { \\\n -+\t\tfflush(stderr); \\\n -+\t\tfflush(stdout); \\\n -+\t\tfprintf(stderr, \"%s:%d: %s (%s) != %s (%s)\\n\", __FILE__, \\\n -+\t\t\t__LINE__, #a, a, #b, b); \\\n -+\t\tabort(); \\\n -+\t}\n ++#ifndef REFTABLE_BLOCKSOURCE_H\n ++#define REFTABLE_BLOCKSOURCE_H\n +\n -+#define assert(c) \\\n -+\tif (!(c)) { \\\n -+\t\tfflush(stderr); \\\n -+\t\tfflush(stdout); \\\n -+\t\tfprintf(stderr, \"%s: %d: failed assertion %s\\n\", __FILE__, \\\n -+\t\t\t__LINE__, #c); \\\n -+\t\tabort(); \\\n -+\t}\n ++#include \n ++\n ++/* block_source is a generic wrapper for a seekable readable file.\n ++ */\n ++struct reftable_block_source {\n ++\tstruct reftable_block_source_vtable *ops;\n ++\tvoid *arg;\n ++};\n +\n -+struct test_case {\n -+\tconst char *name;\n -+\tvoid (*testfunc)(void);\n ++/* a contiguous segment of bytes. It keeps track of its generating block_source\n ++ so it can return itself into the pool.\n ++*/\n ++struct reftable_block {\n ++\tuint8_t *data;\n ++\tint len;\n ++\tstruct reftable_block_source source;\n +};\n +\n -+struct test_case *add_test_case(const char *name, void (*testfunc)(void));\n -+int test_main(int argc, const char *argv[]);\n ++/* block_source_vtable are the operations that make up block_source */\n ++struct reftable_block_source_vtable {\n ++\t/* returns the size of a block source */\n ++\tuint64_t (*size)(void *source);\n ++\n ++\t/* reads a segment from the block source. It is an error to read\n ++\t beyond the end of the block */\n ++\tint (*read_block)(void *source, struct reftable_block *dest,\n ++\t\t\t uint64_t off, uint32_t size);\n ++\t/* mark the block as read; may return the data back to malloc */\n ++\tvoid (*return_block)(void *source, struct reftable_block *blockp);\n ++\n ++\t/* release all resources associated with the block source */\n ++\tvoid (*close)(void *source);\n ++};\n +\n -+void set_test_hash(uint8_t *p, int i);\n ++/* opens a file on the file system as a block_source */\n ++int reftable_block_source_from_file(struct reftable_block_source *block_src,\n ++\t\t\t\t const char *name);\n +\n +#endif\n 6: 8eb944ea9b ! 7: bbe2ed7173 reftable: (de)serialization for the polymorphic record type.\n @@ Metadata\n ## Commit message ##\n reftable: (de)serialization for the polymorphic record type.\n \n + The reftable format is structured as a sequence of blocks, and each block\n + contains a sequence of prefix-compressed key-value records. There are 4 types of\n + records, and they have similarities in how they must be handled. This is\n + achieved by introducing a polymorphic 'record' type that encapsulates ref, log,\n + index and object records.\n +\n Signed-off-by: Han-Wen Nienhuys \n \n ## Makefile ##\n @@ Makefile: REFTABLE_OBJS += reftable/basics.o\n + REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n - REFTABLE_OBJS += reftable/compat.o\n +REFTABLE_OBJS += reftable/record.o\n - REFTABLE_OBJS += reftable/strbuf.o\n \n +REFTABLE_TEST_OBJS += reftable/record_test.o\n - REFTABLE_TEST_OBJS += reftable/strbuf_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n + REFTABLE_TEST_OBJS += reftable/basics_test.o\n \n \n ## reftable/constants.h (new) ##\n @@ reftable/record.c (new)\n +\n +#include \"system.h\"\n +#include \"constants.h\"\n -+#include \"reftable.h\"\n ++#include \"reftable-error.h\"\n +#include \"basics.h\"\n +\n +int get_var_int(uint64_t *dest, struct string_view *in)\n @@ reftable/record.c (new)\n +\n +\t/* This is simple and correct, but we could probably reuse the hash\n +\t fields. */\n -+\treftable_ref_record_clear(ref);\n ++\treftable_ref_record_release(ref);\n +\tif (src->refname != NULL) {\n +\t\tref->refname = xstrdup(src->refname);\n +\t}\n @@ reftable/record.c (new)\n +\tprintf(\"}\\n\");\n +}\n +\n -+static void reftable_ref_record_clear_void(void *rec)\n ++static void reftable_ref_record_release_void(void *rec)\n +{\n -+\treftable_ref_record_clear((struct reftable_ref_record *)rec);\n ++\treftable_ref_record_release((struct reftable_ref_record *)rec);\n +}\n +\n -+void reftable_ref_record_clear(struct reftable_ref_record *ref)\n ++void reftable_ref_record_release(struct reftable_ref_record *ref)\n +{\n +\treftable_free(ref->refname);\n +\treftable_free(ref->target);\n @@ reftable/record.c (new)\n +\t.val_type = &reftable_ref_record_val_type,\n +\t.encode = &reftable_ref_record_encode,\n +\t.decode = &reftable_ref_record_decode,\n -+\t.clear = &reftable_ref_record_clear_void,\n ++\t.release = &reftable_ref_record_release_void,\n +\t.is_deletion = &reftable_ref_record_is_deletion_void,\n +};\n +\n @@ reftable/record.c (new)\n +\tstrbuf_add(dest, rec->hash_prefix, rec->hash_prefix_len);\n +}\n +\n -+static void reftable_obj_record_clear(void *rec)\n ++static void reftable_obj_record_release(void *rec)\n +{\n +\tstruct reftable_obj_record *obj = (struct reftable_obj_record *)rec;\n +\tFREE_AND_NULL(obj->hash_prefix);\n @@ reftable/record.c (new)\n +\t\t(const struct reftable_obj_record *)src_rec;\n +\tint olen;\n +\n -+\treftable_obj_record_clear(obj);\n ++\treftable_obj_record_release(obj);\n +\t*obj = *src;\n +\tobj->hash_prefix = reftable_malloc(obj->hash_prefix_len);\n +\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n @@ reftable/record.c (new)\n +\t.val_type = &reftable_obj_record_val_type,\n +\t.encode = &reftable_obj_record_encode,\n +\t.decode = &reftable_obj_record_decode,\n -+\t.clear = &reftable_obj_record_clear,\n ++\t.release = &reftable_obj_record_release,\n +\t.is_deletion = not_a_deletion,\n +};\n +\n @@ reftable/record.c (new)\n +\tconst struct reftable_log_record *src =\n +\t\t(const struct reftable_log_record *)src_rec;\n +\n -+\treftable_log_record_clear(dst);\n ++\treftable_log_record_release(dst);\n +\t*dst = *src;\n +\tif (dst->refname != NULL) {\n +\t\tdst->refname = xstrdup(dst->refname);\n @@ reftable/record.c (new)\n +\t}\n +}\n +\n -+static void reftable_log_record_clear_void(void *rec)\n ++static void reftable_log_record_release_void(void *rec)\n +{\n +\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n -+\treftable_log_record_clear(r);\n ++\treftable_log_record_release(r);\n +}\n +\n -+void reftable_log_record_clear(struct reftable_log_record *r)\n ++void reftable_log_record_release(struct reftable_log_record *r)\n +{\n +\treftable_free(r->refname);\n +\treftable_free(r->new_hash);\n @@ reftable/record.c (new)\n +\t.val_type = &reftable_log_record_val_type,\n +\t.encode = &reftable_log_record_encode,\n +\t.decode = &reftable_log_record_decode,\n -+\t.clear = &reftable_log_record_clear_void,\n ++\t.release = &reftable_log_record_release_void,\n +\t.is_deletion = &reftable_log_record_is_deletion_void,\n +};\n +\n @@ reftable/record.c (new)\n +\n +void reftable_record_destroy(struct reftable_record *rec)\n +{\n -+\treftable_record_clear(rec);\n ++\treftable_record_release(rec);\n +\treftable_free(reftable_record_yield(rec));\n +}\n +\n @@ reftable/record.c (new)\n +\tdst->offset = src->offset;\n +}\n +\n -+static void reftable_index_record_clear(void *rec)\n ++static void reftable_index_record_release(void *rec)\n +{\n +\tstruct reftable_index_record *idx = (struct reftable_index_record *)rec;\n +\tstrbuf_release(&idx->last_key);\n @@ reftable/record.c (new)\n +\t.val_type = &reftable_index_record_val_type,\n +\t.encode = &reftable_index_record_encode,\n +\t.decode = &reftable_index_record_decode,\n -+\t.clear = &reftable_index_record_clear,\n ++\t.release = &reftable_index_record_release,\n +\t.is_deletion = ¬_a_deletion,\n +};\n +\n @@ reftable/record.c (new)\n +\treturn rec->ops->decode(rec->data, key, extra, src, hash_size);\n +}\n +\n -+void reftable_record_clear(struct reftable_record *rec)\n ++void reftable_record_release(struct reftable_record *rec)\n +{\n -+\trec->ops->clear(rec->data);\n ++\trec->ops->release(rec->data);\n +}\n +\n +int reftable_record_is_deletion(struct reftable_record *rec)\n @@ reftable/record.h (new)\n +#ifndef RECORD_H\n +#define RECORD_H\n +\n -+#include \"reftable.h\"\n -+#include \"strbuf.h\"\n +#include \"system.h\"\n +\n ++#include \n ++\n ++#include \"reftable-record.h\"\n ++\n +/*\n + A substring of existing string data. This structure takes no responsibility\n + for the lifetime of the data it points to.\n @@ reftable/record.h (new)\n +\t\t struct string_view src, int hash_size);\n +\n +\t/* deallocate and null the record. */\n -+\tvoid (*clear)(void *rec);\n ++\tvoid (*release)(void *rec);\n +\n +\t/* is this a tombstone? */\n +\tint (*is_deletion)(const void *rec);\n @@ reftable/record.h (new)\n +int reftable_record_is_deletion(struct reftable_record *rec);\n +\n +/* zeroes out the embedded record */\n -+void reftable_record_clear(struct reftable_record *rec);\n ++void reftable_record_release(struct reftable_record *rec);\n +\n +/* clear and deallocate embedded record, and zero `rec`. */\n +void reftable_record_destroy(struct reftable_record *rec);\n @@ reftable/record_test.c (new)\n +#include \"system.h\"\n +#include \"basics.h\"\n +#include \"constants.h\"\n -+#include \"reftable.h\"\n +#include \"test_framework.h\"\n +#include \"reftable-tests.h\"\n +\n @@ reftable/record_test.c (new)\n +\treftable_record_copy_from(©, rec, SHA1_SIZE);\n +\tswitch (reftable_record_type(©)) {\n +\tcase BLOCK_TYPE_REF:\n -+\t\tassert(reftable_ref_record_equal(reftable_record_as_ref(©),\n ++\t\tEXPECT(reftable_ref_record_equal(reftable_record_as_ref(©),\n +\t\t\t\t\t\t reftable_record_as_ref(rec),\n +\t\t\t\t\t\t SHA1_SIZE));\n +\t\tbreak;\n +\tcase BLOCK_TYPE_LOG:\n -+\t\tassert(reftable_log_record_equal(reftable_record_as_log(©),\n ++\t\tEXPECT(reftable_log_record_equal(reftable_record_as_log(©),\n +\t\t\t\t\t\t reftable_record_as_log(rec),\n +\t\t\t\t\t\t SHA1_SIZE));\n +\t\tbreak;\n @@ reftable/record_test.c (new)\n +\t\tint n = put_var_int(&out, in);\n +\t\tuint64_t got = 0;\n +\n -+\t\tassert(n > 0);\n ++\t\tEXPECT(n > 0);\n +\t\tout.len = n;\n +\t\tn = get_var_int(&got, &out);\n -+\t\tassert(n > 0);\n ++\t\tEXPECT(n > 0);\n +\n -+\t\tassert(got == in);\n ++\t\tEXPECT(got == in);\n +\t}\n +}\n +\n @@ reftable/record_test.c (new)\n +\t\tstruct strbuf b = STRBUF_INIT;\n +\t\tstrbuf_addstr(&a, cases[i].a);\n +\t\tstrbuf_addstr(&b, cases[i].b);\n -+\t\tassert(common_prefix_size(&a, &b) == cases[i].want);\n ++\t\tEXPECT(common_prefix_size(&a, &b) == cases[i].want);\n +\n +\t\tstrbuf_release(&a);\n +\t\tstrbuf_release(&b);\n @@ reftable/record_test.c (new)\n +\t\treftable_record_from_ref(&rec, &in);\n +\t\ttest_copy(&rec);\n +\n -+\t\tassert(reftable_record_val_type(&rec) == i);\n ++\t\tEXPECT(reftable_record_val_type(&rec) == i);\n +\n +\t\treftable_record_key(&rec, &key);\n +\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n -+\t\tassert(n > 0);\n ++\t\tEXPECT(n > 0);\n +\n +\t\t/* decode into a non-zero reftable_record to test for leaks. */\n +\n +\t\treftable_record_from_ref(&rec_out, &out);\n +\t\tm = reftable_record_decode(&rec_out, key, i, dest, SHA1_SIZE);\n -+\t\tassert(n == m);\n ++\t\tEXPECT(n == m);\n +\n -+\t\tassert((out.value != NULL) == (in.value != NULL));\n -+\t\tassert((out.target_value != NULL) == (in.target_value != NULL));\n -+\t\tassert((out.target != NULL) == (in.target != NULL));\n -+\t\treftable_record_clear(&rec_out);\n ++\t\tEXPECT((out.value != NULL) == (in.value != NULL));\n ++\t\tEXPECT((out.target_value != NULL) == (in.target_value != NULL));\n ++\t\tEXPECT((out.target != NULL) == (in.target != NULL));\n ++\t\treftable_record_release(&rec_out);\n +\n +\t\tstrbuf_release(&key);\n -+\t\treftable_ref_record_clear(&in);\n ++\t\treftable_ref_record_release(&in);\n +\t}\n +}\n +\n @@ reftable/record_test.c (new)\n +\t\t}\n +\t};\n +\n -+\tassert(!reftable_log_record_equal(&in[0], &in[1], SHA1_SIZE));\n ++\tEXPECT(!reftable_log_record_equal(&in[0], &in[1], SHA1_SIZE));\n +\tin[1].update_index = in[0].update_index;\n -+\tassert(reftable_log_record_equal(&in[0], &in[1], SHA1_SIZE));\n -+\treftable_log_record_clear(&in[0]);\n -+\treftable_log_record_clear(&in[1]);\n ++\tEXPECT(reftable_log_record_equal(&in[0], &in[1], SHA1_SIZE));\n ++\treftable_log_record_release(&in[0]);\n ++\treftable_log_record_release(&in[1]);\n +}\n +\n +static void test_reftable_log_record_roundtrip(void)\n @@ reftable/record_test.c (new)\n +\t\treftable_record_key(&rec, &key);\n +\n +\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n -+\t\tassert(n >= 0);\n ++\t\tEXPECT(n >= 0);\n +\t\treftable_record_from_log(&rec_out, &out);\n +\t\tvaltype = reftable_record_val_type(&rec);\n +\t\tm = reftable_record_decode(&rec_out, key, valtype, dest,\n +\t\t\t\t\t SHA1_SIZE);\n -+\t\tassert(n == m);\n ++\t\tEXPECT(n == m);\n +\n -+\t\tassert(reftable_log_record_equal(&in[i], &out, SHA1_SIZE));\n -+\t\treftable_log_record_clear(&in[i]);\n ++\t\tEXPECT(reftable_log_record_equal(&in[i], &out, SHA1_SIZE));\n ++\t\treftable_log_record_release(&in[i]);\n +\t\tstrbuf_release(&key);\n -+\t\treftable_record_clear(&rec_out);\n ++\t\treftable_record_release(&rec_out);\n +\t}\n +}\n +\n @@ reftable/record_test.c (new)\n +\tuint32_t out;\n +\tput_be24(dest, in);\n +\tout = get_be24(dest);\n -+\tassert(in == out);\n ++\tEXPECT(in == out);\n +}\n +\n +static void test_key_roundtrip(void)\n @@ reftable/record_test.c (new)\n +\tstrbuf_addstr(&key, \"refs/tags/bla\");\n +\textra = 6;\n +\tn = reftable_encode_key(&restart, dest, last_key, key, extra);\n -+\tassert(!restart);\n -+\tassert(n > 0);\n ++\tEXPECT(!restart);\n ++\tEXPECT(n > 0);\n +\n +\tm = reftable_decode_key(&roundtrip, &rt_extra, last_key, dest);\n -+\tassert(n == m);\n -+\tassert(0 == strbuf_cmp(&key, &roundtrip));\n -+\tassert(rt_extra == extra);\n ++\tEXPECT(n == m);\n ++\tEXPECT(0 == strbuf_cmp(&key, &roundtrip));\n ++\tEXPECT(rt_extra == extra);\n +\n +\tstrbuf_release(&last_key);\n +\tstrbuf_release(&key);\n @@ reftable/record_test.c (new)\n +\t\ttest_copy(&rec);\n +\t\treftable_record_key(&rec, &key);\n +\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n -+\t\tassert(n > 0);\n ++\t\tEXPECT(n > 0);\n +\t\textra = reftable_record_val_type(&rec);\n +\t\treftable_record_from_obj(&rec_out, &out);\n +\t\tm = reftable_record_decode(&rec_out, key, extra, dest,\n +\t\t\t\t\t SHA1_SIZE);\n -+\t\tassert(n == m);\n ++\t\tEXPECT(n == m);\n +\n -+\t\tassert(in.hash_prefix_len == out.hash_prefix_len);\n -+\t\tassert(in.offset_len == out.offset_len);\n ++\t\tEXPECT(in.hash_prefix_len == out.hash_prefix_len);\n ++\t\tEXPECT(in.offset_len == out.offset_len);\n +\n -+\t\tassert(!memcmp(in.hash_prefix, out.hash_prefix,\n ++\t\tEXPECT(!memcmp(in.hash_prefix, out.hash_prefix,\n +\t\t\t in.hash_prefix_len));\n -+\t\tassert(0 == memcmp(in.offsets, out.offsets,\n ++\t\tEXPECT(0 == memcmp(in.offsets, out.offsets,\n +\t\t\t\t sizeof(uint64_t) * in.offset_len));\n +\t\tstrbuf_release(&key);\n -+\t\treftable_record_clear(&rec_out);\n ++\t\treftable_record_release(&rec_out);\n +\t}\n +}\n +\n @@ reftable/record_test.c (new)\n +\treftable_record_key(&rec, &key);\n +\ttest_copy(&rec);\n +\n -+\tassert(0 == strbuf_cmp(&key, &in.last_key));\n ++\tEXPECT(0 == strbuf_cmp(&key, &in.last_key));\n +\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n -+\tassert(n > 0);\n ++\tEXPECT(n > 0);\n +\n +\textra = reftable_record_val_type(&rec);\n +\treftable_record_from_index(&out_rec, &out);\n +\tm = reftable_record_decode(&out_rec, key, extra, dest, SHA1_SIZE);\n -+\tassert(m == n);\n ++\tEXPECT(m == n);\n +\n -+\tassert(in.offset == out.offset);\n ++\tEXPECT(in.offset == out.offset);\n +\n -+\treftable_record_clear(&out_rec);\n ++\treftable_record_release(&out_rec);\n +\tstrbuf_release(&key);\n +\tstrbuf_release(&in.last_key);\n +}\n +\n +int record_test_main(int argc, const char *argv[])\n +{\n -+\tadd_test_case(\"test_reftable_log_record_equal\",\n -+\t\t &test_reftable_log_record_equal);\n -+\tadd_test_case(\"test_reftable_log_record_roundtrip\",\n -+\t\t &test_reftable_log_record_roundtrip);\n -+\tadd_test_case(\"test_reftable_ref_record_roundtrip\",\n -+\t\t &test_reftable_ref_record_roundtrip);\n -+\tadd_test_case(\"test_varint_roundtrip\", &test_varint_roundtrip);\n -+\tadd_test_case(\"test_key_roundtrip\", &test_key_roundtrip);\n -+\tadd_test_case(\"test_common_prefix\", &test_common_prefix);\n -+\tadd_test_case(\"test_reftable_obj_record_roundtrip\",\n -+\t\t &test_reftable_obj_record_roundtrip);\n -+\tadd_test_case(\"test_reftable_index_record_roundtrip\",\n -+\t\t &test_reftable_index_record_roundtrip);\n -+\tadd_test_case(\"test_u24_roundtrip\", &test_u24_roundtrip);\n -+\treturn test_main(argc, argv);\n ++\ttest_reftable_log_record_equal();\n ++\ttest_reftable_log_record_roundtrip();\n ++\ttest_reftable_ref_record_roundtrip();\n ++\ttest_varint_roundtrip();\n ++\ttest_key_roundtrip();\n ++\ttest_common_prefix();\n ++\ttest_reftable_obj_record_roundtrip();\n ++\ttest_reftable_index_record_roundtrip();\n ++\ttest_u24_roundtrip();\n ++\treturn 0;\n +}\n \n + ## reftable/reftable-record.h (new) ##\n +@@\n ++/*\n ++Copyright 2020 Google LLC\n ++\n ++Use of this source code is governed by a BSD-style\n ++license that can be found in the LICENSE file or at\n ++https://developers.google.com/open-source/licenses/bsd\n ++*/\n ++\n ++#ifndef REFTABLE_RECORD_H\n ++#define REFTABLE_RECORD_H\n ++\n ++#include \n ++\n ++/*\n ++ Basic data types\n ++\n ++ Reftables store the state of each ref in struct reftable_ref_record, and they\n ++ store a sequence of reflog updates in struct reftable_log_record.\n ++*/\n ++\n ++/* reftable_ref_record holds a ref database entry target_value */\n ++struct reftable_ref_record {\n ++\tchar *refname; /* Name of the ref, malloced. */\n ++\tuint64_t update_index; /* Logical timestamp at which this value is\n ++\t\t\t\t written */\n ++\tuint8_t *value; /* SHA1, or NULL. malloced. */\n ++\tuint8_t *target_value; /* peeled annotated tag, or NULL. malloced. */\n ++\tchar *target; /* symref, or NULL. malloced. */\n ++};\n ++\n ++/* returns whether 'ref' represents a deletion */\n ++int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);\n ++\n ++/* prints a reftable_ref_record onto stdout */\n ++void reftable_ref_record_print(struct reftable_ref_record *ref,\n ++\t\t\t uint32_t hash_id);\n ++\n ++/* frees and nulls all pointer values. */\n ++void reftable_ref_record_release(struct reftable_ref_record *ref);\n ++\n ++/* returns whether two reftable_ref_records are the same */\n ++int reftable_ref_record_equal(struct reftable_ref_record *a,\n ++\t\t\t struct reftable_ref_record *b, int hash_size);\n ++\n ++/* reftable_log_record holds a reflog entry */\n ++struct reftable_log_record {\n ++\tchar *refname;\n ++\tuint64_t update_index; /* logical timestamp of a transactional update.\n ++\t\t\t\t*/\n ++\tuint8_t *new_hash;\n ++\tuint8_t *old_hash;\n ++\tchar *name;\n ++\tchar *email;\n ++\tuint64_t time;\n ++\tint16_t tz_offset;\n ++\tchar *message;\n ++};\n ++\n ++/* returns whether 'ref' represents the deletion of a log record. */\n ++int reftable_log_record_is_deletion(const struct reftable_log_record *log);\n ++\n ++/* frees and nulls all pointer values. */\n ++void reftable_log_record_release(struct reftable_log_record *log);\n ++\n ++/* returns whether two records are equal. Useful for testing. */\n ++int reftable_log_record_equal(struct reftable_log_record *a,\n ++\t\t\t struct reftable_log_record *b, int hash_size);\n ++\n ++/* dumps a reftable_log_record on stdout, for debugging/testing. */\n ++void reftable_log_record_print(struct reftable_log_record *log,\n ++\t\t\t uint32_t hash_id);\n ++\n ++#endif\n +\n ## t/helper/test-reftable.c ##\n @@\n - \n int cmd__reftable(int argc, const char **argv)\n {\n + \tbasics_test_main(argc, argv);\n +-\n +\trecord_test_main(argc, argv);\n - \tstrbuf_test_main(argc, argv);\n \treturn 0;\n }\n 7: 757dd30fe2 ! 8: a358b052d5 reftable: reading/writing blocks\n @@ Metadata\n ## Commit message ##\n reftable: reading/writing blocks\n \n + The reftable format is structured as a sequence of block. Within a block,\n + records are prefix compressed, with an index of offsets for fully expand keys to\n + enable binary search within blocks.\n +\n + This commit provides the logic to read and write these blocks.\n +\n Includes a code snippet copied from zlib\n \n Signed-off-by: Han-Wen Nienhuys \n \n ## Makefile ##\n -@@ Makefile: XDIFF_OBJS += xdiff/xprepare.o\n - XDIFF_OBJS += xdiff/xutils.o\n +@@ Makefile: XDIFF_OBJS += xdiff/xutils.o\n \n REFTABLE_OBJS += reftable/basics.o\n + REFTABLE_OBJS += reftable/error.o\n +REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n - REFTABLE_OBJS += reftable/compat.o\n REFTABLE_OBJS += reftable/record.o\n - REFTABLE_OBJS += reftable/strbuf.o\n +REFTABLE_OBJS += reftable/zlib-compat.o\n \n +REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n - REFTABLE_TEST_OBJS += reftable/strbuf_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n + REFTABLE_TEST_OBJS += reftable/basics_test.o\n \n ## reftable/.gitattributes (new) ##\n @@\n @@ reftable/block.c (new)\n +#include \"blocksource.h\"\n +#include \"constants.h\"\n +#include \"record.h\"\n -+#include \"reftable.h\"\n ++#include \"reftable-error.h\"\n +#include \"system.h\"\n +#include \"zlib.h\"\n +\n @@ reftable/block.c (new)\n +\treturn -1;\n +}\n +\n -+\n +int block_writer_finish(struct block_writer *w)\n +{\n +\tint i = 0;\n @@ reftable/block.c (new)\n +\treturn err;\n +}\n +\n -+void block_writer_clear(struct block_writer *bw)\n ++void block_writer_release(struct block_writer *bw)\n +{\n +\tFREE_AND_NULL(bw->restarts);\n +\tstrbuf_release(&bw->last_key);\n @@ reftable/block.h (new)\n +\n +#include \"basics.h\"\n +#include \"record.h\"\n -+#include \"reftable.h\"\n ++#include \"reftable-blocksource.h\"\n +\n +/*\n + Writes reftable blocks. The block_writer is reused across blocks to minimize\n @@ reftable/block.h (new)\n +int block_writer_finish(struct block_writer *w);\n +\n +/* clears out internally allocated block_writer members. */\n -+void block_writer_clear(struct block_writer *bw);\n ++void block_writer_release(struct block_writer *bw);\n +\n +/* Read a block. */\n +struct block_reader {\n @@ reftable/block_test.c (new)\n +#include \"basics.h\"\n +#include \"constants.h\"\n +#include \"record.h\"\n -+#include \"reftable.h\"\n +#include \"test_framework.h\"\n +#include \"reftable-tests.h\"\n +\n -+struct binsearch_args {\n -+\tint key;\n -+\tint *arr;\n -+};\n -+\n -+static int binsearch_func(size_t i, void *void_args)\n -+{\n -+\tstruct binsearch_args *args = (struct binsearch_args *)void_args;\n -+\n -+\treturn args->key < args->arr[i];\n -+}\n -+\n -+static void test_binsearch(void)\n -+{\n -+\tint arr[] = { 2, 4, 6, 8, 10 };\n -+\tsize_t sz = ARRAY_SIZE(arr);\n -+\tstruct binsearch_args args = {\n -+\t\t.arr = arr,\n -+\t};\n -+\n -+\tint i = 0;\n -+\tfor (i = 1; i < 11; i++) {\n -+\t\tint res;\n -+\t\targs.key = i;\n -+\t\tres = binsearch(sz, &binsearch_func, &args);\n -+\n -+\t\tif (res < sz) {\n -+\t\t\tassert(args.key < arr[res]);\n -+\t\t\tif (res > 0) {\n -+\t\t\t\tassert(args.key >= arr[res - 1]);\n -+\t\t\t}\n -+\t\t} else {\n -+\t\t\tassert(args.key == 10 || args.key == 11);\n -+\t\t}\n -+\t}\n -+}\n -+\n +static void test_block_read_write(void)\n +{\n +\tconst int header_off = 21; /* random */\n @@ reftable/block_test.c (new)\n +\t\tn = block_writer_add(&bw, &rec);\n +\t\tref.refname = NULL;\n +\t\tref.value = NULL;\n -+\t\tassert(n == 0);\n ++\t\tEXPECT(n == 0);\n +\t}\n +\n +\tn = block_writer_finish(&bw);\n -+\tassert(n > 0);\n ++\tEXPECT(n > 0);\n +\n -+\tblock_writer_clear(&bw);\n ++\tblock_writer_release(&bw);\n +\n +\tblock_reader_init(&br, &block, header_off, block_size, SHA1_SIZE);\n +\n @@ reftable/block_test.c (new)\n +\n +\twhile (1) {\n +\t\tint r = block_iter_next(&it, &rec);\n -+\t\tassert(r >= 0);\n ++\t\tEXPECT(r >= 0);\n +\t\tif (r > 0) {\n +\t\t\tbreak;\n +\t\t}\n -+\t\tassert_streq(names[j], ref.refname);\n ++\t\tEXPECT_STREQ(names[j], ref.refname);\n +\t\tj++;\n +\t}\n +\n -+\treftable_record_clear(&rec);\n ++\treftable_record_release(&rec);\n +\tblock_iter_close(&it);\n +\n +\tfor (i = 0; i < N; i++) {\n @@ reftable/block_test.c (new)\n +\t\tstrbuf_addstr(&want, names[i]);\n +\n +\t\tn = block_reader_seek(&br, &it, &want);\n -+\t\tassert(n == 0);\n ++\t\tEXPECT(n == 0);\n +\n +\t\tn = block_iter_next(&it, &rec);\n -+\t\tassert(n == 0);\n ++\t\tEXPECT(n == 0);\n +\n -+\t\tassert_streq(names[i], ref.refname);\n ++\t\tEXPECT_STREQ(names[i], ref.refname);\n +\n +\t\twant.len--;\n +\t\tn = block_reader_seek(&br, &it, &want);\n -+\t\tassert(n == 0);\n ++\t\tEXPECT(n == 0);\n +\n +\t\tn = block_iter_next(&it, &rec);\n -+\t\tassert(n == 0);\n -+\t\tassert_streq(names[10 * (i / 10)], ref.refname);\n ++\t\tEXPECT(n == 0);\n ++\t\tEXPECT_STREQ(names[10 * (i / 10)], ref.refname);\n +\n +\t\tblock_iter_close(&it);\n +\t}\n +\n -+\treftable_record_clear(&rec);\n ++\treftable_record_release(&rec);\n +\treftable_block_done(&br.block);\n +\tstrbuf_release(&want);\n +\tfor (i = 0; i < N; i++) {\n @@ reftable/block_test.c (new)\n +\n +int block_test_main(int argc, const char *argv[])\n +{\n -+\tadd_test_case(\"binsearch\", &test_binsearch);\n -+\tadd_test_case(\"block_read_write\", &test_block_read_write);\n -+\treturn test_main(argc, argv);\n ++\ttest_block_read_write();\n ++\treturn 0;\n +}\n \n ## reftable/zlib-compat.c (new) ##\n @@ reftable/zlib-compat.c (new)\n \n ## t/helper/test-reftable.c ##\n @@\n - \n int cmd__reftable(int argc, const char **argv)\n {\n + \tbasics_test_main(argc, argv);\n +\tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n - \tstrbuf_test_main(argc, argv);\n \treturn 0;\n + }\n 8: e30a7e0281 ! 9: 24afac9c91 reftable: a generic binary tree implementation\n @@ Metadata\n ## Commit message ##\n reftable: a generic binary tree implementation\n \n - This is necessary for building a OID => ref map on write\n + The reftable format includes support for an (OID => ref) map. This map can speed\n + up visibility and reachability checks. In particular, various operations along\n + the fetch/push path within Gerrit have ben sped up by using this structure.\n +\n + The map is constructed with help of a binary tree. Object IDs are hashes, so\n + they are uniformly distributed. Hence, the tree does not attempt forced\n + rebalancing.\n \n Signed-off-by: Han-Wen Nienhuys \n \n ## Makefile ##\n -@@ Makefile: REFTABLE_OBJS += reftable/publicbasics.o\n - REFTABLE_OBJS += reftable/compat.o\n +@@ Makefile: REFTABLE_OBJS += reftable/block.o\n + REFTABLE_OBJS += reftable/blocksource.o\n + REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n - REFTABLE_OBJS += reftable/strbuf.o\n +REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/zlib-compat.o\n \n ++REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n - REFTABLE_TEST_OBJS += reftable/strbuf_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n +-REFTABLE_TEST_OBJS += reftable/basics_test.o\n +REFTABLE_TEST_OBJS += reftable/tree_test.o\n \n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n @@ reftable/tree_test.c (new)\n +\n +#include \"basics.h\"\n +#include \"record.h\"\n -+#include \"reftable.h\"\n +#include \"test_framework.h\"\n +#include \"reftable-tests.h\"\n +\n @@ reftable/tree_test.c (new)\n +\n +int tree_test_main(int argc, const char *argv[])\n +{\n -+\tadd_test_case(\"test_tree\", &test_tree);\n -+\treturn test_main(argc, argv);\n ++\ttest_tree();\n ++\treturn 0;\n +}\n \n ## t/helper/test-reftable.c ##\n @@ t/helper/test-reftable.c: int cmd__reftable(int argc, const char **argv)\n + \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n - \tstrbuf_test_main(argc, argv);\n +\ttree_test_main(argc, argv);\n \treturn 0;\n }\n 9: 68aee16e60 ! 10: 5f0b7c5592 reftable: write reftable files\n @@ Commit message\n Signed-off-by: Han-Wen Nienhuys \n \n ## Makefile ##\n -@@ Makefile: REFTABLE_OBJS += reftable/compat.o\n +@@ Makefile: REFTABLE_OBJS += reftable/blocksource.o\n + REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n - REFTABLE_OBJS += reftable/strbuf.o\n REFTABLE_OBJS += reftable/tree.o\n +REFTABLE_OBJS += reftable/writer.o\n REFTABLE_OBJS += reftable/zlib-compat.o\n \n - REFTABLE_TEST_OBJS += reftable/block_test.o\n + REFTABLE_TEST_OBJS += reftable/basics_test.o\n +\n + ## reftable/reftable-writer.h (new) ##\n +@@\n ++/*\n ++Copyright 2020 Google LLC\n ++\n ++Use of this source code is governed by a BSD-style\n ++license that can be found in the LICENSE file or at\n ++https://developers.google.com/open-source/licenses/bsd\n ++*/\n ++\n ++#ifndef REFTABLE_WRITER_H\n ++#define REFTABLE_WRITER_H\n ++\n ++#include \n ++\n ++#include \"reftable-record.h\"\n ++\n ++/*\n ++ Writing single reftables\n ++*/\n ++\n ++/* reftable_write_options sets options for writing a single reftable. */\n ++struct reftable_write_options {\n ++\t/* boolean: do not pad out blocks to block size. */\n ++\tunsigned unpadded : 1;\n ++\n ++\t/* the blocksize. Should be less than 2^24. */\n ++\tuint32_t block_size;\n ++\n ++\t/* boolean: do not generate a SHA1 => ref index. */\n ++\tunsigned skip_index_objects : 1;\n ++\n ++\t/* how often to write complete keys in each block. */\n ++\tint restart_interval;\n ++\n ++\t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n ++\t * Defaults to SHA1 if unset\n ++\t */\n ++\tuint32_t hash_id;\n ++\n ++\t/* boolean: do not check ref names for validity or dir/file conflicts.\n ++\t */\n ++\tunsigned skip_name_check : 1;\n ++\n ++\t/* boolean: copy log messages exactly. If unset, check that the message\n ++\t * is a single line, and add '\\n' if missing.\n ++\t */\n ++\tunsigned exact_log_message : 1;\n ++};\n ++\n ++/* reftable_block_stats holds statistics for a single block type */\n ++struct reftable_block_stats {\n ++\t/* total number of entries written */\n ++\tint entries;\n ++\t/* total number of key restarts */\n ++\tint restarts;\n ++\t/* total number of blocks */\n ++\tint blocks;\n ++\t/* total number of index blocks */\n ++\tint index_blocks;\n ++\t/* depth of the index */\n ++\tint max_index_level;\n ++\n ++\t/* offset of the first block for this type */\n ++\tuint64_t offset;\n ++\t/* offset of the top level index block for this type, or 0 if not\n ++\t * present */\n ++\tuint64_t index_offset;\n ++};\n ++\n ++/* stats holds overall statistics for a single reftable */\n ++struct reftable_stats {\n ++\t/* total number of blocks written. */\n ++\tint blocks;\n ++\t/* stats for ref data */\n ++\tstruct reftable_block_stats ref_stats;\n ++\t/* stats for the SHA1 to ref map. */\n ++\tstruct reftable_block_stats obj_stats;\n ++\t/* stats for index blocks */\n ++\tstruct reftable_block_stats idx_stats;\n ++\t/* stats for log blocks */\n ++\tstruct reftable_block_stats log_stats;\n ++\n ++\t/* disambiguation length of shortened object IDs. */\n ++\tint object_id_len;\n ++};\n ++\n ++/* reftable_new_writer creates a new writer */\n ++struct reftable_writer *\n ++reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n ++\t\t void *writer_arg, struct reftable_write_options *opts);\n ++\n ++/* Set the range of update indices for the records we will add. When writing a\n ++ table into a stack, the min should be at least\n ++ reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned.\n ++\n ++ For transactional updates to a stack, typically min==max, and the\n ++ update_index can be obtained by inspeciting the stack. When converting an\n ++ existing ref database into a single reftable, this would be a range of\n ++ update-index timestamps.\n ++ */\n ++void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n ++\t\t\t\tuint64_t max);\n ++\n ++/*\n ++ Add a reftable_ref_record. The record should have names that come after\n ++ already added records.\n ++\n ++ The update_index must be within the limits set by\n ++ reftable_writer_set_limits(), or REFTABLE_API_ERROR is returned. It is an\n ++ REFTABLE_API_ERROR error to write a ref record after a log record.\n ++*/\n ++int reftable_writer_add_ref(struct reftable_writer *w,\n ++\t\t\t struct reftable_ref_record *ref);\n ++\n ++/*\n ++ Convenience function to add multiple reftable_ref_records; the function sorts\n ++ the records before adding them, reordering the records array passed in.\n ++*/\n ++int reftable_writer_add_refs(struct reftable_writer *w,\n ++\t\t\t struct reftable_ref_record *refs, int n);\n ++\n ++/*\n ++ adds reftable_log_records. Log records are keyed by (refname, decreasing\n ++ update_index). The key for the record added must come after the already added\n ++ log records.\n ++*/\n ++int reftable_writer_add_log(struct reftable_writer *w,\n ++\t\t\t struct reftable_log_record *log);\n ++\n ++/*\n ++ Convenience function to add multiple reftable_log_records; the function sorts\n ++ the records before adding them, reordering records array passed in.\n ++*/\n ++int reftable_writer_add_logs(struct reftable_writer *w,\n ++\t\t\t struct reftable_log_record *logs, int n);\n ++\n ++/* reftable_writer_close finalizes the reftable. The writer is retained so\n ++ * statistics can be inspected. */\n ++int reftable_writer_close(struct reftable_writer *w);\n ++\n ++/* writer_stats returns the statistics on the reftable being written.\n ++\n ++ This struct becomes invalid when the writer is freed.\n ++ */\n ++const struct reftable_stats *writer_stats(struct reftable_writer *w);\n ++\n ++/* reftable_writer_free deallocates memory for the writer */\n ++void reftable_writer_free(struct reftable_writer *w);\n ++\n ++#endif\n \n ## reftable/writer.c (new) ##\n @@\n @@ reftable/writer.c (new)\n +#include \"block.h\"\n +#include \"constants.h\"\n +#include \"record.h\"\n -+#include \"reftable.h\"\n +#include \"tree.h\"\n ++#include \"reftable-error.h\"\n +\n +/* finishes a block, and writes it to storage */\n +static int writer_flush_block(struct reftable_writer *w);\n @@ reftable/writer.c (new)\n +\tw->block_writer->restart_interval = w->opts.restart_interval;\n +}\n +\n ++static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n ++\n +struct reftable_writer *\n +reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n +\t\t void *writer_arg, struct reftable_write_options *opts)\n @@ reftable/writer.c (new)\n +\tint err = 0;\n +\tint i = 0;\n +\tQSORT(logs, n, reftable_log_record_compare_key);\n ++\n +\tfor (i = 0; err == 0 && i < n; i++) {\n +\t\terr = reftable_writer_add_log(w, &logs[i]);\n +\t}\n @@ reftable/writer.c (new)\n +\n +done:\n +\t/* free up memory. */\n -+\tblock_writer_clear(&w->block_writer_data);\n ++\tblock_writer_release(&w->block_writer_data);\n +\twriter_clear_index(w);\n +\tstrbuf_release(&w->last_key);\n +\treturn err;\n @@ reftable/writer.h (new)\n +\n +#include \"basics.h\"\n +#include \"block.h\"\n -+#include \"reftable.h\"\n -+#include \"strbuf.h\"\n +#include \"tree.h\"\n ++#include \"reftable-writer.h\"\n +\n +struct reftable_writer {\n +\tint (*write)(void *, const void *, size_t);\n 10: c196de7f06 ! 11: 9aa2caade8 reftable: read reftable files\n @@ Metadata\n ## Commit message ##\n reftable: read reftable files\n \n + This supports reading a single reftable file.\n +\n + The commit introduces an abstract iterator type, which captures the usecases\n + both of reading individual refs, and iterating over a segment of the ref\n + namespace.\n +\n Signed-off-by: Han-Wen Nienhuys \n \n ## Makefile ##\n -@@ Makefile: REFTABLE_OBJS += reftable/block.o\n +@@ Makefile: REFTABLE_OBJS += reftable/basics.o\n + REFTABLE_OBJS += reftable/error.o\n + REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n - REFTABLE_OBJS += reftable/publicbasics.o\n - REFTABLE_OBJS += reftable/compat.o\n +REFTABLE_OBJS += reftable/iter.o\n + REFTABLE_OBJS += reftable/publicbasics.o\n +REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n +REFTABLE_OBJS += reftable/reftable.o\n - REFTABLE_OBJS += reftable/strbuf.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n + REFTABLE_OBJS += reftable/zlib-compat.o\n \n ## reftable/iter.c (new) ##\n @@\n @@ reftable/iter.c (new)\n +#include \"block.h\"\n +#include \"constants.h\"\n +#include \"reader.h\"\n -+#include \"reftable.h\"\n ++#include \"reftable-error.h\"\n +\n +int iterator_is_null(struct reftable_iterator *it)\n +{\n @@ reftable/iter.c (new)\n +int reftable_iterator_next_ref(struct reftable_iterator *it,\n +\t\t\t struct reftable_ref_record *ref)\n +{\n -+\tstruct reftable_record rec = { 0 };\n ++\tstruct reftable_record rec = { NULL };\n +\treftable_record_from_ref(&rec, ref);\n +\treturn iterator_next(it, &rec);\n +}\n @@ reftable/iter.c (new)\n +int reftable_iterator_next_log(struct reftable_iterator *it,\n +\t\t\t struct reftable_log_record *log)\n +{\n -+\tstruct reftable_record rec = { 0 };\n ++\tstruct reftable_record rec = { NULL };\n +\treftable_record_from_log(&rec, log);\n +\treturn iterator_next(it, &rec);\n +}\n @@ reftable/iter.c (new)\n +\t\t}\n +\n +\t\tif (fri->double_check) {\n -+\t\t\tstruct reftable_iterator it = { 0 };\n ++\t\t\tstruct reftable_iterator it = { NULL };\n +\n +\t\t\terr = reftable_table_seek_ref(&fri->tab, &it,\n +\t\t\t\t\t\t ref->refname);\n @@ reftable/iter.c (new)\n +\t\t}\n +\t}\n +\n -+\treftable_ref_record_clear(ref);\n ++\treftable_ref_record_release(ref);\n +\treturn err;\n +}\n +\n @@ reftable/iter.h (new)\n +#ifndef ITER_H\n +#define ITER_H\n +\n ++#include \"system.h\"\n +#include \"block.h\"\n +#include \"record.h\"\n -+#include \"strbuf.h\"\n ++\n ++#include \"reftable-iterator.h\"\n ++#include \"reftable-generic.h\"\n +\n +struct reftable_iterator_vtable {\n +\tint (*next)(void *iter_arg, struct reftable_record *rec);\n @@ reftable/reader.c (new)\n +#include \"constants.h\"\n +#include \"iter.h\"\n +#include \"record.h\"\n -+#include \"reftable.h\"\n ++#include \"reftable-error.h\"\n +#include \"tree.h\"\n +\n +uint64_t block_source_size(struct reftable_block_source *source)\n @@ reftable/reader.c (new)\n +int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n +\t\tconst char *name)\n +{\n -+\tstruct reftable_block footer = { 0 };\n -+\tstruct reftable_block header = { 0 };\n ++\tstruct reftable_block footer = { NULL };\n ++\tstruct reftable_block header = { NULL };\n +\tint err = 0;\n +\n +\tmemset(r, 0, sizeof(struct reftable_reader));\n @@ reftable/reader.c (new)\n +{\n +\tint32_t guess_block_size = r->block_size ? r->block_size :\n +\t\t\t\t\t\t\t DEFAULT_BLOCK_SIZE;\n -+\tstruct reftable_block block = { 0 };\n ++\tstruct reftable_block block = { NULL };\n +\tuint8_t block_typ = 0;\n +\tint err = 0;\n +\tuint32_t header_off = next_off ? 0 : header_size(r->version);\n @@ reftable/reader.c (new)\n +\t\t\t struct reftable_record *rec)\n +{\n +\tstruct reftable_index_record want_index = { .last_key = STRBUF_INIT };\n -+\tstruct reftable_record want_index_rec = { 0 };\n ++\tstruct reftable_record want_index_rec = { NULL };\n +\tstruct reftable_index_record index_result = { .last_key = STRBUF_INIT };\n -+\tstruct reftable_record index_result_rec = { 0 };\n ++\tstruct reftable_record index_result_rec = { NULL };\n +\tstruct table_iter index_iter = TABLE_ITER_INIT;\n +\tstruct table_iter next = TABLE_ITER_INIT;\n +\tint err = 0;\n @@ reftable/reader.c (new)\n +done:\n +\tblock_iter_close(&next.bi);\n +\ttable_iter_close(&index_iter);\n -+\treftable_record_clear(&want_index_rec);\n -+\treftable_record_clear(&index_result_rec);\n ++\treftable_record_release(&want_index_rec);\n ++\treftable_record_release(&index_result_rec);\n +\treturn err;\n +}\n +\n @@ reftable/reader.c (new)\n +\tstruct reftable_ref_record ref = {\n +\t\t.refname = (char *)name,\n +\t};\n -+\tstruct reftable_record rec = { 0 };\n ++\tstruct reftable_record rec = { NULL };\n +\treftable_record_from_ref(&rec, &ref);\n +\treturn reader_seek(r, it, &rec);\n +}\n @@ reftable/reader.c (new)\n +\t\t.refname = (char *)name,\n +\t\t.update_index = update_index,\n +\t};\n -+\tstruct reftable_record rec = { 0 };\n ++\tstruct reftable_record rec = { NULL };\n +\treftable_record_from_log(&rec, &log);\n +\treturn reader_seek(r, it, &rec);\n +}\n @@ reftable/reader.c (new)\n +\t\t.hash_prefix = oid,\n +\t\t.hash_prefix_len = r->object_id_len,\n +\t};\n -+\tstruct reftable_record want_rec = { 0 };\n -+\tstruct reftable_iterator oit = { 0 };\n -+\tstruct reftable_obj_record got = { 0 };\n -+\tstruct reftable_record got_rec = { 0 };\n ++\tstruct reftable_record want_rec = { NULL };\n ++\tstruct reftable_iterator oit = { NULL };\n ++\tstruct reftable_obj_record got = { NULL };\n ++\tstruct reftable_record got_rec = { NULL };\n +\tint err = 0;\n +\tstruct indexed_table_ref_iter *itr = NULL;\n +\n @@ reftable/reader.c (new)\n +\n +done:\n +\treftable_iterator_destroy(&oit);\n -+\treftable_record_clear(&got_rec);\n ++\treftable_record_release(&got_rec);\n +\treturn err;\n +}\n +\n @@ reftable/reader.h (new)\n +\n +#include \"block.h\"\n +#include \"record.h\"\n -+#include \"reftable.h\"\n ++#include \"reftable-iterator.h\"\n ++#include \"reftable-reader.h\"\n +\n +uint64_t block_source_size(struct reftable_block_source *source);\n +\n @@ reftable/reader.h (new)\n +\tuint64_t (*max_update_index)(void *tab);\n +};\n +\n -+int reftable_table_seek_record(struct reftable_table *tab,\n -+\t\t\t struct reftable_iterator *it,\n -+\t\t\t struct reftable_record *rec);\n -+\n +#endif\n \n - ## reftable/reftable.c (new) ##\n + ## reftable/reftable-iterator.h (new) ##\n @@\n +/*\n +Copyright 2020 Google LLC\n @@ reftable/reftable.c (new)\n +https://developers.google.com/open-source/licenses/bsd\n +*/\n +\n -+#include \"reftable.h\"\n -+#include \"record.h\"\n -+#include \"reader.h\"\n ++#ifndef REFTABLE_ITERATOR_H\n ++#define REFTABLE_ITERATOR_H\n +\n -+static int reftable_reader_seek_void(void *tab, struct reftable_iterator *it,\n -+\t\t\t\t struct reftable_record *rec)\n -+{\n -+\treturn reader_seek((struct reftable_reader *)tab, it, rec);\n -+}\n ++#include \"reftable-record.h\"\n +\n -+static uint32_t reftable_reader_hash_id_void(void *tab)\n -+{\n -+\treturn reftable_reader_hash_id((struct reftable_reader *)tab);\n -+}\n ++/* iterator is the generic interface for walking over data stored in a\n ++ reftable.\n ++*/\n ++struct reftable_iterator {\n ++\tstruct reftable_iterator_vtable *ops;\n ++\tvoid *iter_arg;\n ++};\n +\n -+static uint64_t reftable_reader_min_update_index_void(void *tab)\n -+{\n -+\treturn reftable_reader_min_update_index((struct reftable_reader *)tab);\n -+}\n ++/* reads the next reftable_ref_record. Returns < 0 for error, 0 for OK and > 0:\n ++ end of iteration.\n ++*/\n ++int reftable_iterator_next_ref(struct reftable_iterator *it,\n ++\t\t\t struct reftable_ref_record *ref);\n +\n -+static uint64_t reftable_reader_max_update_index_void(void *tab)\n -+{\n -+\treturn reftable_reader_max_update_index((struct reftable_reader *)tab);\n -+}\n ++/* reads the next reftable_log_record. Returns < 0 for error, 0 for OK and > 0:\n ++ end of iteration.\n ++*/\n ++int reftable_iterator_next_log(struct reftable_iterator *it,\n ++\t\t\t struct reftable_log_record *log);\n +\n -+static struct reftable_table_vtable reader_vtable = {\n -+\t.seek_record = reftable_reader_seek_void,\n -+\t.hash_id = reftable_reader_hash_id_void,\n -+\t.min_update_index = reftable_reader_min_update_index_void,\n -+\t.max_update_index = reftable_reader_max_update_index_void,\n -+};\n ++/* releases resources associated with an iterator. */\n ++void reftable_iterator_destroy(struct reftable_iterator *it);\n +\n -+int reftable_table_seek_ref(struct reftable_table *tab,\n -+\t\t\t struct reftable_iterator *it, const char *name)\n -+{\n -+\tstruct reftable_ref_record ref = {\n -+\t\t.refname = (char *)name,\n -+\t};\n -+\tstruct reftable_record rec = { 0 };\n -+\treftable_record_from_ref(&rec, &ref);\n -+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n -+}\n ++#endif\n +\n + ## reftable/reftable-reader.h (new) ##\n +@@\n ++/*\n ++Copyright 2020 Google LLC\n +\n -+void reftable_table_from_reader(struct reftable_table *tab,\n -+\t\t\t\tstruct reftable_reader *reader)\n -+{\n -+\tassert(tab->ops == NULL);\n -+\ttab->ops = &reader_vtable;\n -+\ttab->table_arg = reader;\n -+}\n ++Use of this source code is governed by a BSD-style\n ++license that can be found in the LICENSE file or at\n ++https://developers.google.com/open-source/licenses/bsd\n ++*/\n +\n -+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n -+\t\t\t struct reftable_ref_record *ref)\n -+{\n -+\tstruct reftable_iterator it = { 0 };\n -+\tint err = reftable_table_seek_ref(tab, &it, name);\n -+\tif (err)\n -+\t\tgoto done;\n ++#ifndef REFTABLE_READER_H\n ++#define REFTABLE_READER_H\n +\n -+\terr = reftable_iterator_next_ref(&it, ref);\n -+\tif (err)\n -+\t\tgoto done;\n ++#include \"reftable-iterator.h\"\n ++#include \"reftable-blocksource.h\"\n +\n -+\tif (strcmp(ref->refname, name) ||\n -+\t reftable_ref_record_is_deletion(ref)) {\n -+\t\treftable_ref_record_clear(ref);\n -+\t\terr = 1;\n -+\t\tgoto done;\n -+\t}\n ++/*\n ++ Reading single tables\n +\n -+done:\n -+\treftable_iterator_destroy(&it);\n -+\treturn err;\n -+}\n ++ The follow routines are for reading single files. For an application-level\n ++ interface, skip ahead to struct reftable_merged_table and struct\n ++ reftable_stack.\n ++*/\n +\n -+int reftable_table_seek_record(struct reftable_table *tab,\n -+\t\t\t struct reftable_iterator *it,\n -+\t\t\t struct reftable_record *rec)\n -+{\n -+\treturn tab->ops->seek_record(tab->table_arg, it, rec);\n -+}\n ++/* The reader struct is a handle to an open reftable file. */\n ++struct reftable_reader;\n +\n -+uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n -+{\n -+\treturn tab->ops->max_update_index(tab->table_arg);\n -+}\n ++/* reftable_new_reader opens a reftable for reading. If successful, returns 0\n ++ * code and sets pp. The name is used for creating a stack. Typically, it is the\n ++ * basename of the file. The block source `src` is owned by the reader, and is\n ++ * closed on calling reftable_reader_destroy().\n ++ */\n ++int reftable_new_reader(struct reftable_reader **pp,\n ++\t\t\tstruct reftable_block_source *src, const char *name);\n ++\n ++/* reftable_reader_seek_ref returns an iterator where 'name' would be inserted\n ++ in the table. To seek to the start of the table, use name = \"\".\n ++\n ++ example:\n ++\n ++ struct reftable_reader *r = NULL;\n ++ int err = reftable_new_reader(&r, &src, \"filename\");\n ++ if (err < 0) { ... }\n ++ struct reftable_iterator it = {0};\n ++ err = reftable_reader_seek_ref(r, &it, \"refs/heads/master\");\n ++ if (err < 0) { ... }\n ++ struct reftable_ref_record ref = {0};\n ++ while (1) {\n ++ err = reftable_iterator_next_ref(&it, &ref);\n ++ if (err > 0) {\n ++ break;\n ++ }\n ++ if (err < 0) {\n ++ ..error handling..\n ++ }\n ++ ..found..\n ++ }\n ++ reftable_iterator_destroy(&it);\n ++ reftable_ref_record_release(&ref);\n ++ */\n ++int reftable_reader_seek_ref(struct reftable_reader *r,\n ++\t\t\t struct reftable_iterator *it, const char *name);\n +\n -+uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n -+{\n -+\treturn tab->ops->min_update_index(tab->table_arg);\n -+}\n ++/* returns the hash ID used in this table. */\n ++uint32_t reftable_reader_hash_id(struct reftable_reader *r);\n +\n -+uint32_t reftable_table_hash_id(struct reftable_table *tab)\n -+{\n -+\treturn tab->ops->hash_id(tab->table_arg);\n -+}\n ++/* seek to logs for the given name, older than update_index. To seek to the\n ++ start of the table, use name = \"\".\n ++ */\n ++int reftable_reader_seek_log_at(struct reftable_reader *r,\n ++\t\t\t\tstruct reftable_iterator *it, const char *name,\n ++\t\t\t\tuint64_t update_index);\n ++\n ++/* seek to newest log entry for given name. */\n ++int reftable_reader_seek_log(struct reftable_reader *r,\n ++\t\t\t struct reftable_iterator *it, const char *name);\n ++\n ++/* closes and deallocates a reader. */\n ++void reftable_reader_free(struct reftable_reader *);\n ++\n ++/* return an iterator for the refs pointing to `oid`. */\n ++int reftable_reader_refs_for(struct reftable_reader *r,\n ++\t\t\t struct reftable_iterator *it, uint8_t *oid);\n ++\n ++/* return the max_update_index for a table */\n ++uint64_t reftable_reader_max_update_index(struct reftable_reader *r);\n ++\n ++/* return the min_update_index for a table */\n ++uint64_t reftable_reader_min_update_index(struct reftable_reader *r);\n ++\n ++#endif\n 11: bf6b929b86 ! 12: 581a4200d6 reftable: file level tests\n @@ Metadata\n Author: Han-Wen Nienhuys \n \n ## Commit message ##\n - reftable: file level tests\n + reftable: reftable file level tests\n +\n + With support for reading and writing files in place, we can construct files (in\n + memory) and attempt to read them back.\n +\n + Because some sections of the format are optional (eg. indices, log entries), we\n + have to exercise this code using multiple sizes of input data\n \n Signed-off-by: Han-Wen Nienhuys \n \n ## Makefile ##\n @@ Makefile: REFTABLE_OBJS += reftable/zlib-compat.o\n - \n + REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n +REFTABLE_TEST_OBJS += reftable/reftable_test.o\n - REFTABLE_TEST_OBJS += reftable/strbuf_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n + \n \n ## reftable/reftable_test.c (new) ##\n @@\n @@ reftable/reftable_test.c (new)\n +https://developers.google.com/open-source/licenses/bsd\n +*/\n +\n -+#include \"reftable.h\"\n -+\n +#include \"system.h\"\n +\n +#include \"basics.h\"\n @@ reftable/reftable_test.c (new)\n +#include \"record.h\"\n +#include \"test_framework.h\"\n +#include \"reftable-tests.h\"\n ++#include \"reftable-stack.h\"\n +\n +static const int update_index = 5;\n +\n @@ reftable/reftable_test.c (new)\n +\tuint8_t in[] = \"hello\";\n +\tstrbuf_add(&buf, in, sizeof(in));\n +\tblock_source_from_strbuf(&source, &buf);\n -+\tassert(block_source_size(&source) == 6);\n ++\tEXPECT(block_source_size(&source) == 6);\n +\tn = block_source_read_block(&source, &out, 0, sizeof(in));\n -+\tassert(n == sizeof(in));\n -+\tassert(!memcmp(in, out.data, n));\n ++\tEXPECT(n == sizeof(in));\n ++\tEXPECT(!memcmp(in, out.data, n));\n +\treftable_block_done(&out);\n +\n +\tn = block_source_read_block(&source, &out, 1, 2);\n -+\tassert(n == 2);\n -+\tassert(!memcmp(out.data, \"el\", 2));\n ++\tEXPECT(n == 2);\n ++\tEXPECT(!memcmp(out.data, \"el\", 2));\n +\n +\treftable_block_done(&out);\n +\tblock_source_close(&source);\n @@ reftable/reftable_test.c (new)\n +\t\t(*names)[i] = xstrdup(name);\n +\n +\t\tn = reftable_writer_add_ref(w, &ref);\n -+\t\tassert(n == 0);\n ++\t\tEXPECT(n == 0);\n +\t}\n +\n +\tfor (i = 0; i < N; i++) {\n @@ reftable/reftable_test.c (new)\n +\t\tlog.message = \"message\";\n +\n +\t\tn = reftable_writer_add_log(w, &log);\n -+\t\tassert(n == 0);\n ++\t\tEXPECT(n == 0);\n +\t}\n +\n +\tn = reftable_writer_close(w);\n -+\tassert(n == 0);\n ++\tEXPECT(n == 0);\n +\n +\tstats = writer_stats(w);\n +\tfor (i = 0; i < stats->ref_stats.blocks; i++) {\n @@ reftable/reftable_test.c (new)\n +\t\tif (off == 0) {\n +\t\t\toff = header_size((hash_id == SHA256_ID) ? 2 : 1);\n +\t\t}\n -+\t\tassert(buf->buf[off] == 'r');\n ++\t\tEXPECT(buf->buf[off] == 'r');\n +\t}\n +\n -+\tassert(stats->log_stats.blocks > 0);\n ++\tEXPECT(stats->log_stats.blocks > 0);\n +\treftable_writer_free(w);\n +}\n +\n @@ reftable/reftable_test.c (new)\n +\tlog.new_hash = hash2;\n +\treftable_writer_set_limits(w, update_index, update_index);\n +\terr = reftable_writer_add_log(w, &log);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\terr = reftable_writer_close(w);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\treftable_writer_free(w);\n +\tstrbuf_release(&buf);\n +}\n @@ reftable/reftable_test.c (new)\n +\t\tref.update_index = i;\n +\n +\t\terr = reftable_writer_add_ref(w, &ref);\n -+\t\tassert_err(err);\n ++\t\tEXPECT_ERR(err);\n +\t}\n +\tfor (i = 0; i < N; i++) {\n +\t\tuint8_t hash1[SHA1_SIZE], hash2[SHA1_SIZE];\n @@ reftable/reftable_test.c (new)\n +\t\tlog.new_hash = hash2;\n +\n +\t\terr = reftable_writer_add_log(w, &log);\n -+\t\tassert_err(err);\n ++\t\tEXPECT_ERR(err);\n +\t}\n +\n +\tn = reftable_writer_close(w);\n -+\tassert(n == 0);\n ++\tEXPECT(n == 0);\n +\n +\tstats = writer_stats(w);\n -+\tassert(stats->log_stats.blocks > 0);\n ++\tEXPECT(stats->log_stats.blocks > 0);\n +\treftable_writer_free(w);\n +\tw = NULL;\n +\n +\tblock_source_from_strbuf(&source, &buf);\n +\n +\terr = init_reader(&rd, &source, \"file.log\");\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_reader_seek_ref(&rd, &it, names[N - 1]);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_iterator_next_ref(&it, &ref);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\t/* end of iteration. */\n +\terr = reftable_iterator_next_ref(&it, &ref);\n -+\tassert(0 < err);\n ++\tEXPECT(0 < err);\n +\n +\treftable_iterator_destroy(&it);\n -+\treftable_ref_record_clear(&ref);\n ++\treftable_ref_record_release(&ref);\n +\n +\terr = reftable_reader_seek_log(&rd, &it, \"\");\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\ti = 0;\n +\twhile (1) {\n @@ reftable/reftable_test.c (new)\n +\t\t\tbreak;\n +\t\t}\n +\n -+\t\tassert_err(err);\n -+\t\tassert_streq(names[i], log.refname);\n -+\t\tassert(i == log.update_index);\n ++\t\tEXPECT_ERR(err);\n ++\t\tEXPECT_STREQ(names[i], log.refname);\n ++\t\tEXPECT(i == log.update_index);\n +\t\ti++;\n -+\t\treftable_log_record_clear(&log);\n ++\t\treftable_log_record_release(&log);\n +\t}\n +\n -+\tassert(i == N);\n ++\tEXPECT(i == N);\n +\treftable_iterator_destroy(&it);\n +\n +\t/* cleanup. */\n @@ reftable/reftable_test.c (new)\n +\tblock_source_from_strbuf(&source, &buf);\n +\n +\terr = init_reader(&rd, &source, \"file.ref\");\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\twhile (1) {\n +\t\tstruct reftable_ref_record ref = { NULL };\n +\t\tint r = reftable_iterator_next_ref(&it, &ref);\n -+\t\tassert(r >= 0);\n ++\t\tEXPECT(r >= 0);\n +\t\tif (r > 0) {\n +\t\t\tbreak;\n +\t\t}\n -+\t\tassert(0 == strcmp(names[j], ref.refname));\n -+\t\tassert(update_index == ref.update_index);\n ++\t\tEXPECT(0 == strcmp(names[j], ref.refname));\n ++\t\tEXPECT(update_index == ref.update_index);\n +\n +\t\tj++;\n -+\t\treftable_ref_record_clear(&ref);\n ++\t\treftable_ref_record_release(&ref);\n +\t}\n -+\tassert(j == N);\n ++\tEXPECT(j == N);\n +\treftable_iterator_destroy(&it);\n +\tstrbuf_release(&buf);\n +\tfree_names(names);\n @@ reftable/reftable_test.c (new)\n +\tstruct strbuf buf = STRBUF_INIT;\n +\tint N = 1;\n +\twrite_table(&names, &buf, N, 4096, SHA1_ID);\n -+\tassert(buf.len < 200);\n ++\tEXPECT(buf.len < 200);\n +\tstrbuf_release(&buf);\n +\tfree_names(names);\n +}\n @@ reftable/reftable_test.c (new)\n +\tblock_source_from_strbuf(&source, &buf);\n +\n +\terr = init_reader(&rd, &source, \"file.ref\");\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_reader_seek_ref(&rd, &it, names[0]);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_iterator_next_log(&it, &log);\n -+\tassert(err == REFTABLE_API_ERROR);\n ++\tEXPECT(err == REFTABLE_API_ERROR);\n +\n +\tstrbuf_release(&buf);\n +\tfor (i = 0; i < N; i++) {\n @@ reftable/reftable_test.c (new)\n +\tblock_source_from_strbuf(&source, &buf);\n +\n +\terr = init_reader(&rd, &source, \"file.ref\");\n -+\tassert_err(err);\n -+\tassert(hash_id == reftable_reader_hash_id(&rd));\n ++\tEXPECT_ERR(err);\n ++\tEXPECT(hash_id == reftable_reader_hash_id(&rd));\n +\n +\tif (!index) {\n +\t\trd.ref_offsets.index_offset = 0;\n +\t} else {\n -+\t\tassert(rd.ref_offsets.index_offset > 0);\n ++\t\tEXPECT(rd.ref_offsets.index_offset > 0);\n +\t}\n +\n +\tfor (i = 1; i < N; i++) {\n +\t\tint err = reftable_reader_seek_ref(&rd, &it, names[i]);\n -+\t\tassert_err(err);\n ++\t\tEXPECT_ERR(err);\n +\t\terr = reftable_iterator_next_ref(&it, &ref);\n -+\t\tassert_err(err);\n -+\t\tassert(0 == strcmp(names[i], ref.refname));\n -+\t\tassert(i == ref.value[0]);\n ++\t\tEXPECT_ERR(err);\n ++\t\tEXPECT(0 == strcmp(names[i], ref.refname));\n ++\t\tEXPECT(i == ref.value[0]);\n +\n -+\t\treftable_ref_record_clear(&ref);\n ++\t\treftable_ref_record_release(&ref);\n +\t\treftable_iterator_destroy(&it);\n +\t}\n +\n @@ reftable/reftable_test.c (new)\n +\tif (err == 0) {\n +\t\tstruct reftable_ref_record ref = { NULL };\n +\t\tint err = reftable_iterator_next_ref(&it, &ref);\n -+\t\tassert(err > 0);\n ++\t\tEXPECT(err > 0);\n +\t} else {\n -+\t\tassert(err > 0);\n ++\t\tEXPECT(err > 0);\n +\t}\n +\n +\tstrbuf_release(&pastLast);\n @@ reftable/reftable_test.c (new)\n +\t\t */\n +\t\t/* blocks. */\n +\t\tn = reftable_writer_add_ref(w, &ref);\n -+\t\tassert(n == 0);\n ++\t\tEXPECT(n == 0);\n +\n +\t\tif (!memcmp(hash1, want_hash, SHA1_SIZE) ||\n +\t\t !memcmp(hash2, want_hash, SHA1_SIZE)) {\n @@ reftable/reftable_test.c (new)\n +\t}\n +\n +\tn = reftable_writer_close(w);\n -+\tassert(n == 0);\n ++\tEXPECT(n == 0);\n +\n +\treftable_writer_free(w);\n +\tw = NULL;\n @@ reftable/reftable_test.c (new)\n +\tblock_source_from_strbuf(&source, &buf);\n +\n +\terr = init_reader(&rd, &source, \"file.ref\");\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\tif (!indexed) {\n +\t\trd.obj_offsets.is_present = 0;\n +\t}\n +\n +\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\treftable_iterator_destroy(&it);\n +\n +\terr = reftable_reader_refs_for(&rd, &it, want_hash);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\tj = 0;\n +\twhile (1) {\n +\t\tint err = reftable_iterator_next_ref(&it, &ref);\n -+\t\tassert(err >= 0);\n ++\t\tEXPECT(err >= 0);\n +\t\tif (err > 0) {\n +\t\t\tbreak;\n +\t\t}\n +\n -+\t\tassert(j < want_names_len);\n -+\t\tassert(0 == strcmp(ref.refname, want_names[j]));\n ++\t\tEXPECT(j < want_names_len);\n ++\t\tEXPECT(0 == strcmp(ref.refname, want_names[j]));\n +\t\tj++;\n -+\t\treftable_ref_record_clear(&ref);\n ++\t\treftable_ref_record_release(&ref);\n +\t}\n -+\tassert(j == want_names_len);\n ++\tEXPECT(j == want_names_len);\n +\n +\tstrbuf_release(&buf);\n +\tfree_names(want_names);\n @@ reftable/reftable_test.c (new)\n +\treftable_writer_set_limits(w, 1, 1);\n +\n +\terr = reftable_writer_close(w);\n -+\tassert(err == REFTABLE_EMPTY_TABLE_ERROR);\n ++\tEXPECT(err == REFTABLE_EMPTY_TABLE_ERROR);\n +\treftable_writer_free(w);\n +\n -+\tassert(buf.len == header_size(1) + footer_size(1));\n ++\tEXPECT(buf.len == header_size(1) + footer_size(1));\n +\n +\tblock_source_from_strbuf(&source, &buf);\n +\n +\terr = reftable_new_reader(&rd, &source, \"filename\");\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_reader_seek_ref(rd, &it, \"\");\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_iterator_next_ref(&it, &rec);\n -+\tassert(err > 0);\n ++\tEXPECT(err > 0);\n +\n +\treftable_iterator_destroy(&it);\n +\treftable_reader_free(rd);\n @@ reftable/reftable_test.c (new)\n +\n +int reftable_test_main(int argc, const char *argv[])\n +{\n -+\tadd_test_case(\"test_log_write_read\", test_log_write_read);\n -+\tadd_test_case(\"test_table_read_write_seek_linear_sha256\",\n -+\t\t &test_table_read_write_seek_linear_sha256);\n -+\tadd_test_case(\"test_log_buffer_size\", test_log_buffer_size);\n -+\tadd_test_case(\"test_table_write_small_table\",\n -+\t\t &test_table_write_small_table);\n -+\tadd_test_case(\"test_buffer\", &test_buffer);\n -+\tadd_test_case(\"test_table_read_api\", &test_table_read_api);\n -+\tadd_test_case(\"test_table_read_write_sequential\",\n -+\t\t &test_table_read_write_sequential);\n -+\tadd_test_case(\"test_table_read_write_seek_linear\",\n -+\t\t &test_table_read_write_seek_linear);\n -+\tadd_test_case(\"test_table_read_write_seek_index\",\n -+\t\t &test_table_read_write_seek_index);\n -+\tadd_test_case(\"test_table_read_write_refs_for_no_index\",\n -+\t\t &test_table_refs_for_no_index);\n -+\tadd_test_case(\"test_table_read_write_refs_for_obj_index\",\n -+\t\t &test_table_refs_for_obj_index);\n -+\tadd_test_case(\"test_table_empty\", &test_table_empty);\n -+\treturn test_main(argc, argv);\n ++\ttest_log_write_read();\n ++\ttest_table_read_write_seek_linear_sha256();\n ++\ttest_log_buffer_size();\n ++\ttest_table_write_small_table();\n ++\ttest_buffer();\n ++\ttest_table_read_api();\n ++\ttest_table_read_write_sequential();\n ++\ttest_table_read_write_seek_linear();\n ++\ttest_table_read_write_seek_index();\n ++\ttest_table_refs_for_no_index();\n ++\ttest_table_refs_for_obj_index();\n ++\ttest_table_empty();\n ++\treturn 0;\n +}\n \n ## t/helper/test-reftable.c ##\n @@ t/helper/test-reftable.c: int cmd__reftable(int argc, const char **argv)\n - {\n + \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n +\treftable_test_main(argc, argv);\n - \tstrbuf_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n + }\n 12: 4e38db7f48 ! 13: a26fb180ad reftable: rest of library\n @@ Metadata\n ## Commit message ##\n reftable: rest of library\n \n - This will be further split up once preceding commits have passed review.\n -\n Signed-off-by: Han-Wen Nienhuys \n \n ## Makefile ##\n -@@ Makefile: XDIFF_OBJS += xdiff/xutils.o\n - REFTABLE_OBJS += reftable/basics.o\n +@@ Makefile: REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n --REFTABLE_OBJS += reftable/publicbasics.o\n - REFTABLE_OBJS += reftable/compat.o\n REFTABLE_OBJS += reftable/iter.o\n +REFTABLE_OBJS += reftable/merged.o\n +REFTABLE_OBJS += reftable/pq.o\n -+REFTABLE_OBJS += reftable/publicbasics.o\n + REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n +REFTABLE_OBJS += reftable/refname.o\n REFTABLE_OBJS += reftable/reftable.o\n +REFTABLE_OBJS += reftable/stack.o\n - REFTABLE_OBJS += reftable/strbuf.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n -@@ Makefile: REFTABLE_OBJS += reftable/zlib-compat.o\n - \n - REFTABLE_TEST_OBJS += reftable/block_test.o\n - REFTABLE_TEST_OBJS += reftable/record_test.o\n -+REFTABLE_TEST_OBJS += reftable/refname_test.o\n - REFTABLE_TEST_OBJS += reftable/reftable_test.o\n -+REFTABLE_TEST_OBJS += reftable/stack_test.o\n - REFTABLE_TEST_OBJS += reftable/strbuf_test.o\n - REFTABLE_TEST_OBJS += reftable/test_framework.o\n - REFTABLE_TEST_OBJS += reftable/tree_test.o\n + REFTABLE_OBJS += reftable/zlib-compat.o\n \n ## reftable/VERSION (new) ##\n @@\n -+7134eb9f8171a9759800f4187f9e6dde997335e7 C: NULL iso 0 for init\n ++b8ec0f74c74cb6752eb2033ad8e755a9c19aad15 C: add missing header\n \n ## reftable/dump.c (new) ##\n @@\n @@ reftable/dump.c (new)\n +\n +static int dump_table(const char *tablename)\n +{\n -+\tstruct reftable_block_source src = { NULL };\n ++\tstruct reftable_block_source src = { 0 };\n +\tint err = reftable_block_source_from_file(&src, tablename);\n -+\tstruct reftable_iterator it = { NULL };\n -+\tstruct reftable_ref_record ref = { NULL };\n -+\tstruct reftable_log_record log = { NULL };\n ++\tstruct reftable_iterator it = { 0 };\n ++\tstruct reftable_ref_record ref = { 0 };\n ++\tstruct reftable_log_record log = { 0 };\n +\tstruct reftable_reader *r = NULL;\n +\n +\tif (err < 0)\n @@ reftable/dump.c (new)\n +{\n +\tstruct reftable_stack *stack = NULL;\n +\tstruct reftable_write_options cfg = {};\n -+\tstruct reftable_iterator it = { NULL };\n -+\tstruct reftable_ref_record ref = { NULL };\n -+\tstruct reftable_log_record log = { NULL };\n ++\tstruct reftable_iterator it = { 0 };\n ++\tstruct reftable_ref_record ref = { 0 };\n ++\tstruct reftable_log_record log = { 0 };\n +\tstruct reftable_merged_table *merged = NULL;\n +\n +\tint err = reftable_new_stack(&stack, stackdir, cfg);\n @@ reftable/merged.c (new)\n +#include \"pq.h\"\n +#include \"reader.h\"\n +#include \"record.h\"\n -+#include \"reftable.h\"\n ++#include \"reftable-merged.h\"\n ++#include \"reftable-error.h\"\n +#include \"system.h\"\n +\n +static int merged_iter_init(struct merged_iter *mi)\n @@ reftable/merged.c (new)\n +{\n +\tstruct merged_iter *mi = (struct merged_iter *)p;\n +\tint i = 0;\n -+\tmerged_iter_pqueue_clear(&mi->pq);\n ++\tmerged_iter_pqueue_release(&mi->pq);\n +\tfor (i = 0; i < mi->stack_len; i++) {\n +\t\treftable_iterator_destroy(&mi->stack[i]);\n +\t}\n @@ reftable/merged.c (new)\n +}\n +\n +/* clears the list of subtable, without affecting the readers themselves. */\n -+void merged_table_clear(struct reftable_merged_table *mt)\n ++void merged_table_release(struct reftable_merged_table *mt)\n +{\n +\tFREE_AND_NULL(mt->stack);\n +\tmt->stack_len = 0;\n @@ reftable/merged.c (new)\n +\tif (mt == NULL) {\n +\t\treturn;\n +\t}\n -+\tmerged_table_clear(mt);\n ++\tmerged_table_release(mt);\n +\treftable_free(mt);\n +}\n +\n @@ reftable/merged.c (new)\n +\treturn mt->min;\n +}\n +\n ++static int reftable_table_seek_record(struct reftable_table *tab,\n ++\t\t\t\t struct reftable_iterator *it,\n ++\t\t\t\t struct reftable_record *rec)\n ++{\n ++\treturn tab->ops->seek_record(tab->table_arg, it, rec);\n ++}\n ++\n +static int merged_table_seek_record(struct reftable_merged_table *mt,\n +\t\t\t\t struct reftable_iterator *it,\n +\t\t\t\t struct reftable_record *rec)\n @@ reftable/merged.h (new)\n +#define MERGED_H\n +\n +#include \"pq.h\"\n -+#include \"reftable.h\"\n +\n +struct reftable_merged_table {\n +\tstruct reftable_table *stack;\n @@ reftable/merged.h (new)\n +\tstruct merged_iter_pqueue pq;\n +};\n +\n -+void merged_table_clear(struct reftable_merged_table *mt);\n ++void merged_table_release(struct reftable_merged_table *mt);\n +\n +#endif\n \n @@ reftable/merged_test.c (new)\n +#include \"pq.h\"\n +#include \"reader.h\"\n +#include \"record.h\"\n -+#include \"reftable.h\"\n +#include \"test_framework.h\"\n ++#include \"reftable-merged.h\"\n +#include \"reftable-tests.h\"\n ++#include \"reftable-generic.h\"\n ++#include \"reftable-stack.h\"\n +\n +static void test_pq(void)\n +{\n @@ reftable/merged_test.c (new)\n +\t\treftable_free(names[i]);\n +\t}\n +\n -+\tmerged_iter_pqueue_clear(&pq);\n ++\tmerged_iter_pqueue_release(&pq);\n +}\n +\n +static void write_test_table(struct strbuf *buf,\n @@ reftable/merged_test.c (new)\n +\t}\n +\n +\terr = reftable_writer_close(w);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\treftable_writer_free(w);\n +}\n @@ reftable/merged_test.c (new)\n +\n +\t\terr = reftable_new_reader(&(*readers)[i], &(*source)[i],\n +\t\t\t\t\t \"name\");\n -+\t\tassert_err(err);\n ++\t\tEXPECT_ERR(err);\n +\t\treftable_table_from_reader(&tabs[i], (*readers)[i]);\n +\t}\n +\n +\terr = reftable_new_merged_table(&mt, tabs, n, SHA1_ID);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\treturn mt;\n +}\n +\n @@ reftable/merged_test.c (new)\n +\tstruct reftable_ref_record ref = { NULL };\n +\tstruct reftable_iterator it = { NULL };\n +\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_iterator_next_ref(&it, &ref);\n -+\tassert_err(err);\n -+\tassert(ref.update_index == 2);\n -+\treftable_ref_record_clear(&ref);\n ++\tEXPECT_ERR(err);\n ++\tEXPECT(ref.update_index == 2);\n ++\treftable_ref_record_release(&ref);\n +\treftable_iterator_destroy(&it);\n +\treaders_destroy(readers, 2);\n +\treftable_merged_table_free(mt);\n @@ reftable/merged_test.c (new)\n +\tsize_t cap = 0;\n +\tint i = 0;\n +\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\twhile (len < 100) { /* cap loops/recursion. */\n +\t\tstruct reftable_ref_record ref = { NULL };\n +\t\tint err = reftable_iterator_next_ref(&it, &ref);\n @@ reftable/merged_test.c (new)\n +\t\tassert(reftable_ref_record_equal(&want[i], &out[i], SHA1_SIZE));\n +\t}\n +\tfor (i = 0; i < len; i++) {\n -+\t\treftable_ref_record_clear(&out[i]);\n ++\t\treftable_ref_record_release(&out[i]);\n +\t}\n +\treftable_free(out);\n +\n @@ reftable/merged_test.c (new)\n +\treftable_writer_set_limits(w, 1, 1);\n +\n +\terr = reftable_writer_add_ref(w, &rec);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_writer_close(w);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\treftable_writer_free(w);\n +\n +\tblock_source_from_strbuf(&source, &buf);\n +\n +\terr = reftable_new_reader(&rd, &source, \"filename\");\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\thash_id = reftable_reader_hash_id(rd);\n +\tassert(hash_id == SHA1_ID);\n +\n +\treftable_table_from_reader(&tab[0], rd);\n +\terr = reftable_new_merged_table(&merged, tab, 1, SHA1_ID);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\treftable_reader_free(rd);\n +\treftable_merged_table_free(merged);\n @@ reftable/merged_test.c (new)\n +\n +int merged_test_main(int argc, const char *argv[])\n +{\n -+\tadd_test_case(\"test_merged_between\", &test_merged_between);\n -+\tadd_test_case(\"test_pq\", &test_pq);\n -+\tadd_test_case(\"test_merged\", &test_merged);\n -+\tadd_test_case(\"test_default_write_opts\", &test_default_write_opts);\n -+\treturn test_main(argc, argv);\n ++\ttest_merged_between();\n ++\ttest_pq();\n ++\ttest_merged();\n ++\ttest_default_write_opts();\n ++\treturn 0;\n +}\n \n ## reftable/pq.c (new) ##\n @@ reftable/pq.c (new)\n +\n +#include \"pq.h\"\n +\n -+#include \"reftable.h\"\n ++#include \"reftable-record.h\"\n +#include \"system.h\"\n +#include \"basics.h\"\n +\n @@ reftable/pq.c (new)\n +\t}\n +}\n +\n -+void merged_iter_pqueue_clear(struct merged_iter_pqueue *pq)\n ++void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)\n +{\n +\tint i = 0;\n +\tfor (i = 0; i < pq->len; i++) {\n @@ reftable/pq.h (new)\n +void merged_iter_pqueue_check(struct merged_iter_pqueue pq);\n +struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n +void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e);\n -+void merged_iter_pqueue_clear(struct merged_iter_pqueue *pq);\n ++void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n +\n +#endif\n \n @@ reftable/refname.c (new)\n +*/\n +\n +#include \"system.h\"\n -+#include \"reftable.h\"\n ++#include \"reftable-error.h\"\n +#include \"basics.h\"\n +#include \"refname.h\"\n -+#include \"strbuf.h\"\n ++#include \"reftable-iterator.h\"\n +\n +struct find_arg {\n +\tchar **names;\n @@ reftable/refname.c (new)\n +\t}\n +\n +\terr = reftable_table_read_ref(&mod->tab, name, &ref);\n -+\treftable_ref_record_clear(&ref);\n ++\treftable_ref_record_release(&ref);\n +\treturn err;\n +}\n +\n -+static void modification_clear(struct modification *mod)\n ++static void modification_release(struct modification *mod)\n +{\n +\t/* don't delete the strings themselves; they're owned by ref records.\n +\t */\n @@ reftable/refname.c (new)\n +\t}\n +\n +done:\n -+\treftable_ref_record_clear(&ref);\n ++\treftable_ref_record_release(&ref);\n +\treftable_iterator_destroy(&it);\n +\treturn err;\n +}\n @@ reftable/refname.c (new)\n +\t}\n +\n +\terr = modification_validate(&mod);\n -+\tmodification_clear(&mod);\n ++\tmodification_release(&mod);\n +\treturn err;\n +}\n +\n @@ reftable/refname.h (new)\n +#ifndef REFNAME_H\n +#define REFNAME_H\n +\n -+#include \"reftable.h\"\n ++#include \"reftable-record.h\"\n ++#include \"reftable-generic.h\"\n +\n +struct modification {\n +\tstruct reftable_table tab;\n @@ reftable/refname_test.c (new)\n +https://developers.google.com/open-source/licenses/bsd\n +*/\n +\n -+#include \"reftable.h\"\n -+\n +#include \"basics.h\"\n +#include \"block.h\"\n +#include \"blocksource.h\"\n @@ reftable/refname_test.c (new)\n +#include \"reader.h\"\n +#include \"record.h\"\n +#include \"refname.h\"\n ++#include \"reftable-error.h\"\n ++#include \"reftable-writer.h\"\n +#include \"system.h\"\n +\n +#include \"test_framework.h\"\n @@ reftable/refname_test.c (new)\n +\treftable_writer_set_limits(w, 1, 1);\n +\n +\terr = reftable_writer_add_ref(w, &rec);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_writer_close(w);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\treftable_writer_free(w);\n +\n +\tblock_source_from_strbuf(&source, &buf);\n +\terr = reftable_new_reader(&rd, &source, \"filename\");\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\treftable_table_from_reader(&tab, rd);\n +\n @@ reftable/refname_test.c (new)\n +\t\t}\n +\n +\t\terr = modification_validate(&mod);\n -+\t\tassert(err == cases[i].error_code);\n ++\t\tEXPECT(err == cases[i].error_code);\n +\t}\n +\n +\treftable_reader_free(rd);\n @@ reftable/refname_test.c (new)\n +\n +int refname_test_main(int argc, const char *argv[])\n +{\n -+\tadd_test_case(\"test_conflict\", &test_conflict);\n -+\treturn test_main(argc, argv);\n ++\ttest_conflict();\n ++\treturn 0;\n +}\n \n - ## reftable/reftable.c ##\n -@@ reftable/reftable.c: int reftable_table_seek_ref(struct reftable_table *tab,\n - \tstruct reftable_ref_record ref = {\n - \t\t.refname = (char *)name,\n - \t};\n --\tstruct reftable_record rec = { 0 };\n + ## reftable/reftable-generic.h (new) ##\n +@@\n ++/*\n ++Copyright 2020 Google LLC\n ++\n ++Use of this source code is governed by a BSD-style\n ++license that can be found in the LICENSE file or at\n ++https://developers.google.com/open-source/licenses/bsd\n ++*/\n ++\n ++#ifndef REFTABLE_GENERIC_H\n ++#define REFTABLE_GENERIC_H\n ++\n ++#include \"reftable-iterator.h\"\n ++#include \"reftable-reader.h\"\n ++#include \"reftable-merged.h\"\n ++\n ++/*\n ++ Provides a unified API for reading tables, either merged tables, or single\n ++ readers.\n ++*/\n ++struct reftable_table {\n ++\tstruct reftable_table_vtable *ops;\n ++\tvoid *table_arg;\n ++};\n ++\n ++int reftable_table_seek_ref(struct reftable_table *tab,\n ++\t\t\t struct reftable_iterator *it, const char *name);\n ++\n ++void reftable_table_from_reader(struct reftable_table *tab,\n ++\t\t\t\tstruct reftable_reader *reader);\n ++\n ++/* returns the hash ID from a generic reftable_table */\n ++uint32_t reftable_table_hash_id(struct reftable_table *tab);\n ++\n ++/* create a generic table from reftable_merged_table */\n ++void reftable_table_from_merged_table(struct reftable_table *tab,\n ++\t\t\t\t struct reftable_merged_table *table);\n ++\n ++/* returns the max update_index covered by this table. */\n ++uint64_t reftable_table_max_update_index(struct reftable_table *tab);\n ++\n ++/* returns the min update_index covered by this table. */\n ++uint64_t reftable_table_min_update_index(struct reftable_table *tab);\n ++\n ++/* convenience function to read a single ref. Returns < 0 for error, 0\n ++ for success, and 1 if ref not found. */\n ++int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n ++\t\t\t struct reftable_ref_record *ref);\n ++\n ++#endif\n +\n + ## reftable/reftable-merged.h (new) ##\n +@@\n ++/*\n ++Copyright 2020 Google LLC\n ++\n ++Use of this source code is governed by a BSD-style\n ++license that can be found in the LICENSE file or at\n ++https://developers.google.com/open-source/licenses/bsd\n ++*/\n ++\n ++#ifndef REFTABLE_MERGED_H\n ++#define REFTABLE_MERGED_H\n ++\n ++#include \"reftable-iterator.h\"\n ++\n ++/*\n ++ Merged tables\n ++\n ++ A ref database kept in a sequence of table files. The merged_table presents a\n ++ unified view to reading (seeking, iterating) a sequence of immutable tables.\n ++\n ++ The merged tables are on purpose kept disconnected from their actual storage\n ++ (eg. files on disk), because it is useful to merge tables aren't files. For\n ++ example, the per-workspace and global ref namespace can be implemented as a\n ++ merged table of two stacks of file-backed reftables.\n ++*/\n ++\n ++/* A merged table is implements seeking/iterating over a stack of tables. */\n ++struct reftable_merged_table;\n ++\n ++/* A generic reftable; see below. */\n ++struct reftable_table;\n ++\n ++/* reftable_new_merged_table creates a new merged table. It takes ownership of\n ++ the stack array.\n ++*/\n ++int reftable_new_merged_table(struct reftable_merged_table **dest,\n ++\t\t\t struct reftable_table *stack, int n,\n ++\t\t\t uint32_t hash_id);\n ++\n ++/* returns an iterator positioned just before 'name' */\n ++int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n ++\t\t\t\t struct reftable_iterator *it,\n ++\t\t\t\t const char *name);\n ++\n ++/* returns an iterator for log entry, at given update_index */\n ++int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n ++\t\t\t\t struct reftable_iterator *it,\n ++\t\t\t\t const char *name, uint64_t update_index);\n ++\n ++/* like reftable_merged_table_seek_log_at but look for the newest entry. */\n ++int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n ++\t\t\t\t struct reftable_iterator *it,\n ++\t\t\t\t const char *name);\n ++\n ++/* returns the max update_index covered by this merged table. */\n ++uint64_t\n ++reftable_merged_table_max_update_index(struct reftable_merged_table *mt);\n ++\n ++/* returns the min update_index covered by this merged table. */\n ++uint64_t\n ++reftable_merged_table_min_update_index(struct reftable_merged_table *mt);\n ++\n ++/* releases memory for the merged_table */\n ++void reftable_merged_table_free(struct reftable_merged_table *m);\n ++\n ++/* return the hash ID of the merged table. */\n ++uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m);\n ++\n ++#endif\n +\n + ## reftable/reftable-stack.h (new) ##\n +@@\n ++/*\n ++Copyright 2020 Google LLC\n ++\n ++Use of this source code is governed by a BSD-style\n ++license that can be found in the LICENSE file or at\n ++https://developers.google.com/open-source/licenses/bsd\n ++*/\n ++\n ++#ifndef REFTABLE_STACK_H\n ++#define REFTABLE_STACK_H\n ++\n ++#include \"reftable-writer.h\"\n ++\n ++/*\n ++ The stack presents an interface to a mutable sequence of reftables.\n ++\n ++ A stack can be mutated by pushing a table to the top of the stack.\n ++\n ++ The reftable_stack automatically compacts files on disk to ensure good\n ++ amortized performance.\n ++*/\n ++struct reftable_stack;\n ++\n ++/* open a new reftable stack. The tables along with the table list will be\n ++ stored in 'dir'. Typically, this should be .git/reftables.\n ++*/\n ++int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n ++\t\t struct reftable_write_options config);\n ++\n ++/* returns the update_index at which a next table should be written. */\n ++uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\n ++\n ++/* holds a transaction to add tables at the top of a stack. */\n ++struct reftable_addition;\n ++\n ++/*\n ++ returns a new transaction to add reftables to the given stack. As a side\n ++ effect, the ref database is locked.\n ++*/\n ++int reftable_stack_new_addition(struct reftable_addition **dest,\n ++\t\t\t\tstruct reftable_stack *st);\n ++\n ++/* Adds a reftable to transaction. */\n ++int reftable_addition_add(struct reftable_addition *add,\n ++\t\t\t int (*write_table)(struct reftable_writer *wr,\n ++\t\t\t\t\t void *arg),\n ++\t\t\t void *arg);\n ++\n ++/* Commits the transaction, releasing the lock. */\n ++int reftable_addition_commit(struct reftable_addition *add);\n ++\n ++/* Release all non-committed data from the transaction, and deallocate the\n ++ transaction. Releases the lock if held. */\n ++void reftable_addition_destroy(struct reftable_addition *add);\n ++\n ++/* add a new table to the stack. The write_table function must call\n ++ reftable_writer_set_limits, add refs and return an error value. */\n ++int reftable_stack_add(struct reftable_stack *st,\n ++\t\t int (*write_table)(struct reftable_writer *wr,\n ++\t\t\t\t\t void *write_arg),\n ++\t\t void *write_arg);\n ++\n ++/* returns the merged_table for seeking. This table is valid until the\n ++ next write or reload, and should not be closed or deleted.\n ++*/\n ++struct reftable_merged_table *\n ++reftable_stack_merged_table(struct reftable_stack *st);\n ++\n ++/* frees all resources associated with the stack. */\n ++void reftable_stack_destroy(struct reftable_stack *st);\n ++\n ++/* Reloads the stack if necessary. This is very cheap to run if the stack was up\n ++ * to date */\n ++int reftable_stack_reload(struct reftable_stack *st);\n ++\n ++/* Policy for expiring reflog entries. */\n ++struct reftable_log_expiry_config {\n ++\t/* Drop entries older than this timestamp */\n ++\tuint64_t time;\n ++\n ++\t/* Drop older entries */\n ++\tuint64_t min_update_index;\n ++};\n ++\n ++/* compacts all reftables into a giant table. Expire reflog entries if config is\n ++ * non-NULL */\n ++int reftable_stack_compact_all(struct reftable_stack *st,\n ++\t\t\t struct reftable_log_expiry_config *config);\n ++\n ++/* heuristically compact unbalanced table stack. */\n ++int reftable_stack_auto_compact(struct reftable_stack *st);\n ++\n ++/* convenience function to read a single ref. Returns < 0 for error, 0\n ++ for success, and 1 if ref not found. */\n ++int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n ++\t\t\t struct reftable_ref_record *ref);\n ++\n ++/* convenience function to read a single log. Returns < 0 for error, 0\n ++ for success, and 1 if ref not found. */\n ++int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n ++\t\t\t struct reftable_log_record *log);\n ++\n ++/* statistics on past compactions. */\n ++struct reftable_compaction_stats {\n ++\tuint64_t bytes; /* total number of bytes written */\n ++\tuint64_t entries_written; /* total number of entries written, including\n ++\t\t\t\t failures. */\n ++\tint attempts; /* how often we tried to compact */\n ++\tint failures; /* failures happen on concurrent updates */\n ++};\n ++\n ++/* return statistics for compaction up till now. */\n ++struct reftable_compaction_stats *\n ++reftable_stack_compaction_stats(struct reftable_stack *st);\n ++\n ++#endif\n +\n + ## reftable/reftable.c (new) ##\n +@@\n ++/*\n ++Copyright 2020 Google LLC\n ++\n ++Use of this source code is governed by a BSD-style\n ++license that can be found in the LICENSE file or at\n ++https://developers.google.com/open-source/licenses/bsd\n ++*/\n ++\n ++#include \"record.h\"\n ++#include \"reader.h\"\n ++#include \"reftable-iterator.h\"\n ++#include \"reftable-generic.h\"\n ++\n ++static int reftable_reader_seek_void(void *tab, struct reftable_iterator *it,\n ++\t\t\t\t struct reftable_record *rec)\n ++{\n ++\treturn reader_seek((struct reftable_reader *)tab, it, rec);\n ++}\n ++\n ++static uint32_t reftable_reader_hash_id_void(void *tab)\n ++{\n ++\treturn reftable_reader_hash_id((struct reftable_reader *)tab);\n ++}\n ++\n ++static uint64_t reftable_reader_min_update_index_void(void *tab)\n ++{\n ++\treturn reftable_reader_min_update_index((struct reftable_reader *)tab);\n ++}\n ++\n ++static uint64_t reftable_reader_max_update_index_void(void *tab)\n ++{\n ++\treturn reftable_reader_max_update_index((struct reftable_reader *)tab);\n ++}\n ++\n ++static struct reftable_table_vtable reader_vtable = {\n ++\t.seek_record = reftable_reader_seek_void,\n ++\t.hash_id = reftable_reader_hash_id_void,\n ++\t.min_update_index = reftable_reader_min_update_index_void,\n ++\t.max_update_index = reftable_reader_max_update_index_void,\n ++};\n ++\n ++int reftable_table_seek_ref(struct reftable_table *tab,\n ++\t\t\t struct reftable_iterator *it, const char *name)\n ++{\n ++\tstruct reftable_ref_record ref = {\n ++\t\t.refname = (char *)name,\n ++\t};\n +\tstruct reftable_record rec = { NULL };\n - \treftable_record_from_ref(&rec, &ref);\n - \treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n - }\n -@@ reftable/reftable.c: void reftable_table_from_reader(struct reftable_table *tab,\n - int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n - \t\t\t struct reftable_ref_record *ref)\n - {\n --\tstruct reftable_iterator it = { 0 };\n ++\treftable_record_from_ref(&rec, &ref);\n ++\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n ++}\n ++\n ++void reftable_table_from_reader(struct reftable_table *tab,\n ++\t\t\t\tstruct reftable_reader *reader)\n ++{\n ++\tassert(tab->ops == NULL);\n ++\ttab->ops = &reader_vtable;\n ++\ttab->table_arg = reader;\n ++}\n ++\n ++int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n ++\t\t\t struct reftable_ref_record *ref)\n ++{\n +\tstruct reftable_iterator it = { NULL };\n - \tint err = reftable_table_seek_ref(tab, &it, name);\n - \tif (err)\n - \t\tgoto done;\n ++\tint err = reftable_table_seek_ref(tab, &it, name);\n ++\tif (err)\n ++\t\tgoto done;\n ++\n ++\terr = reftable_iterator_next_ref(&it, ref);\n ++\tif (err)\n ++\t\tgoto done;\n ++\n ++\tif (strcmp(ref->refname, name) ||\n ++\t reftable_ref_record_is_deletion(ref)) {\n ++\t\treftable_ref_record_release(ref);\n ++\t\terr = 1;\n ++\t\tgoto done;\n ++\t}\n ++\n ++done:\n ++\treftable_iterator_destroy(&it);\n ++\treturn err;\n ++}\n ++\n ++uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n ++{\n ++\treturn tab->ops->max_update_index(tab->table_arg);\n ++}\n ++\n ++uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n ++{\n ++\treturn tab->ops->min_update_index(tab->table_arg);\n ++}\n ++\n ++uint32_t reftable_table_hash_id(struct reftable_table *tab)\n ++{\n ++\treturn tab->ops->hash_id(tab->table_arg);\n ++}\n \n ## reftable/stack.c (new) ##\n @@\n @@ reftable/stack.c (new)\n +#include \"merged.h\"\n +#include \"reader.h\"\n +#include \"refname.h\"\n -+#include \"reftable.h\"\n ++#include \"reftable-error.h\"\n ++#include \"reftable-record.h\"\n +#include \"writer.h\"\n +\n +static int stack_try_add(struct reftable_stack *st,\n @@ reftable/stack.c (new)\n +static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n +\t\t\t\t\t int reuse_open);\n +\n ++static int reftable_fd_write(void *arg, const void *data, size_t sz)\n ++{\n ++\tint *fdp = (int *)arg;\n ++\treturn write(*fdp, data, sz);\n ++}\n ++\n +int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n +\t\t struct reftable_write_options config)\n +{\n @@ reftable/stack.c (new)\n +\tnew_tables = NULL;\n +\tst->readers_len = new_readers_len;\n +\tif (st->merged != NULL) {\n -+\t\tmerged_table_clear(st->merged);\n ++\t\tmerged_table_release(st->merged);\n +\t\treftable_merged_table_free(st->merged);\n +\t}\n +\tif (st->readers != NULL) {\n @@ reftable/stack.c (new)\n +done:\n +\treftable_iterator_destroy(&it);\n +\tif (mt != NULL) {\n -+\t\tmerged_table_clear(mt);\n ++\t\tmerged_table_release(mt);\n +\t\treftable_merged_table_free(mt);\n +\t}\n -+\treftable_ref_record_clear(&ref);\n -+\treftable_log_record_clear(&log);\n ++\treftable_ref_record_release(&ref);\n ++\treftable_log_record_release(&log);\n +\tst->stats.entries_written += entries;\n +\treturn err;\n +}\n @@ reftable/stack.c (new)\n +\n +done:\n +\tif (err) {\n -+\t\treftable_log_record_clear(log);\n ++\t\treftable_log_record_release(log);\n +\t}\n +\treftable_iterator_destroy(&it);\n +\treturn err;\n @@ reftable/stack.c (new)\n +\n +done:\n +\tfor (i = 0; i < len; i++) {\n -+\t\treftable_ref_record_clear(&refs[i]);\n ++\t\treftable_ref_record_release(&refs[i]);\n +\t}\n +\n +\tfree(refs);\n @@ reftable/stack.h (new)\n +#ifndef STACK_H\n +#define STACK_H\n +\n -+#include \"reftable.h\"\n +#include \"system.h\"\n ++#include \"reftable-writer.h\"\n ++#include \"reftable-stack.h\"\n +\n +struct reftable_stack {\n +\tchar *list_file;\n @@ reftable/stack_test.c (new)\n +#include \"basics.h\"\n +#include \"constants.h\"\n +#include \"record.h\"\n -+#include \"reftable.h\"\n +#include \"test_framework.h\"\n +#include \"reftable-tests.h\"\n +\n +#include \n +#include \n +\n ++static void clear_dir(const char *dirname)\n ++{\n ++\tstruct strbuf path = STRBUF_INIT;\n ++\tstrbuf_addstr(&path, dirname);\n ++\tremove_dir_recursively(&path, 0);\n ++\tstrbuf_release(&path);\n ++}\n ++\n +static void test_read_file(void)\n +{\n +\tchar fn[256] = \"/tmp/stack.test_read_file.XXXXXX\";\n @@ reftable/stack_test.c (new)\n +\tchar *want[] = { \"line1\", \"line2\", \"line3\" };\n +\tint i = 0;\n +\n -+\tassert(fd > 0);\n ++\tEXPECT(fd > 0);\n +\tn = write(fd, out, strlen(out));\n -+\tassert(n == strlen(out));\n ++\tEXPECT(n == strlen(out));\n +\terr = close(fd);\n -+\tassert(err >= 0);\n ++\tEXPECT(err >= 0);\n +\n +\terr = read_lines(fn, &names);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\tfor (i = 0; names[i] != NULL; i++) {\n -+\t\tassert(0 == strcmp(want[i], names[i]));\n ++\t\tEXPECT(0 == strcmp(want[i], names[i]));\n +\t}\n +\tfree_names(names);\n +\tremove(fn);\n @@ reftable/stack_test.c (new)\n +\tchar **names = NULL;\n +\tparse_names(buf, strlen(buf), &names);\n +\n -+\tassert(NULL != names[0]);\n -+\tassert(0 == strcmp(names[0], \"line\"));\n -+\tassert(NULL == names[1]);\n ++\tEXPECT(NULL != names[0]);\n ++\tEXPECT(0 == strcmp(names[0], \"line\"));\n ++\tEXPECT(NULL == names[1]);\n +\tfree_names(names);\n +}\n +\n @@ reftable/stack_test.c (new)\n +\tchar *b[] = { \"a\", \"b\", \"d\", NULL };\n +\tchar *c[] = { \"a\", \"b\", NULL };\n +\n -+\tassert(names_equal(a, a));\n -+\tassert(!names_equal(a, b));\n -+\tassert(!names_equal(a, c));\n ++\tEXPECT(names_equal(a, a));\n ++\tEXPECT(!names_equal(a, b));\n ++\tEXPECT(!names_equal(a, c));\n +}\n +\n +static int write_test_ref(struct reftable_writer *wr, void *arg)\n @@ reftable/stack_test.c (new)\n +\t};\n +\tstruct reftable_ref_record dest = { NULL };\n +\n -+\tassert(mkdtemp(dir));\n ++\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st, dir, cfg);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_stack_add(st, &write_test_ref, &ref);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n -+\tassert_err(err);\n -+\tassert(0 == strcmp(\"master\", dest.target));\n ++\tEXPECT_ERR(err);\n ++\tEXPECT(0 == strcmp(\"master\", dest.target));\n +\n -+\treftable_ref_record_clear(&dest);\n ++\treftable_ref_record_release(&dest);\n +\treftable_stack_destroy(st);\n -+\treftable_clear_dir(dir);\n ++\tclear_dir(dir);\n +}\n +\n +static void test_reftable_stack_uptodate(void)\n @@ reftable/stack_test.c (new)\n +\t\t.target = \"master\",\n +\t};\n +\n -+\tassert(mkdtemp(dir));\n ++\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st1, dir, cfg);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_new_stack(&st2, dir, cfg);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_stack_add(st1, &write_test_ref, &ref1);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n -+\tassert(err == REFTABLE_LOCK_ERROR);\n ++\tEXPECT(err == REFTABLE_LOCK_ERROR);\n +\n +\terr = reftable_stack_reload(st2);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\treftable_stack_destroy(st1);\n +\treftable_stack_destroy(st2);\n -+\treftable_clear_dir(dir);\n ++\tclear_dir(dir);\n +}\n +\n +static void test_reftable_stack_transaction_api(void)\n @@ reftable/stack_test.c (new)\n +\t};\n +\tstruct reftable_ref_record dest = { NULL };\n +\n -+\tassert(mkdtemp(dir));\n ++\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st, dir, cfg);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\treftable_addition_destroy(add);\n +\n +\terr = reftable_stack_new_addition(&add, st);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_addition_add(add, &write_test_ref, &ref);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_addition_commit(add);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\treftable_addition_destroy(add);\n +\n +\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n -+\tassert_err(err);\n -+\tassert(0 == strcmp(\"master\", dest.target));\n ++\tEXPECT_ERR(err);\n ++\tEXPECT(0 == strcmp(\"master\", dest.target));\n +\n -+\treftable_ref_record_clear(&dest);\n ++\treftable_ref_record_release(&dest);\n +\treftable_stack_destroy(st);\n -+\treftable_clear_dir(dir);\n ++\tclear_dir(dir);\n +}\n +\n +static void test_reftable_stack_validate_refname(void)\n @@ reftable/stack_test.c (new)\n +\t};\n +\tchar *additions[] = { \"a\", \"a/b/c\" };\n +\n -+\tassert(mkdtemp(dir));\n ++\tEXPECT(mkdtemp(dir));\n +\terr = reftable_new_stack(&st, dir, cfg);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_stack_add(st, &write_test_ref, &ref);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\tfor (i = 0; i < ARRAY_SIZE(additions); i++) {\n +\t\tstruct reftable_ref_record ref = {\n @@ reftable/stack_test.c (new)\n +\t\t};\n +\n +\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n -+\t\tassert(err == REFTABLE_NAME_CONFLICT);\n ++\t\tEXPECT(err == REFTABLE_NAME_CONFLICT);\n +\t}\n +\n +\treftable_stack_destroy(st);\n -+\treftable_clear_dir(dir);\n ++\tclear_dir(dir);\n +}\n +\n +static int write_error(struct reftable_writer *wr, void *arg)\n @@ reftable/stack_test.c (new)\n +\t\t.update_index = 1,\n +\t\t.target = \"master\",\n +\t};\n -+\tassert(mkdtemp(dir));\n ++\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st, dir, cfg);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_stack_add(st, &write_test_ref, &ref1);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_stack_add(st, &write_test_ref, &ref2);\n -+\tassert(err == REFTABLE_API_ERROR);\n ++\tEXPECT(err == REFTABLE_API_ERROR);\n +\treftable_stack_destroy(st);\n -+\treftable_clear_dir(dir);\n ++\tclear_dir(dir);\n +}\n +\n +static void test_reftable_stack_lock_failure(void)\n @@ reftable/stack_test.c (new)\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tint err, i;\n -+\tassert(mkdtemp(dir));\n ++\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st, dir, cfg);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\tfor (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) {\n +\t\terr = reftable_stack_add(st, &write_error, &i);\n -+\t\tassert(err == i);\n ++\t\tEXPECT(err == i);\n +\t}\n +\n +\treftable_stack_destroy(st);\n -+\treftable_clear_dir(dir);\n ++\tclear_dir(dir);\n +}\n +\n +static void test_reftable_stack_add(void)\n @@ reftable/stack_test.c (new)\n +\tstruct reftable_log_record logs[2] = { { NULL } };\n +\tint N = ARRAY_SIZE(refs);\n +\n -+\tassert(mkdtemp(dir));\n ++\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st, dir, cfg);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\tst->disable_auto_compact = 1;\n +\n +\tfor (i = 0; i < N; i++) {\n @@ reftable/stack_test.c (new)\n +\n +\tfor (i = 0; i < N; i++) {\n +\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n -+\t\tassert_err(err);\n ++\t\tEXPECT_ERR(err);\n +\t}\n +\n +\tfor (i = 0; i < N; i++) {\n @@ reftable/stack_test.c (new)\n +\t\t\t.update_index = reftable_stack_next_update_index(st),\n +\t\t};\n +\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n -+\t\tassert_err(err);\n ++\t\tEXPECT_ERR(err);\n +\t}\n +\n +\terr = reftable_stack_compact_all(st, NULL);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\tfor (i = 0; i < N; i++) {\n +\t\tstruct reftable_ref_record dest = { NULL };\n +\n +\t\tint err = reftable_stack_read_ref(st, refs[i].refname, &dest);\n -+\t\tassert_err(err);\n -+\t\tassert(reftable_ref_record_equal(&dest, refs + i, SHA1_SIZE));\n -+\t\treftable_ref_record_clear(&dest);\n ++\t\tEXPECT_ERR(err);\n ++\t\tEXPECT(reftable_ref_record_equal(&dest, refs + i, SHA1_SIZE));\n ++\t\treftable_ref_record_release(&dest);\n +\t}\n +\n +\tfor (i = 0; i < N; i++) {\n +\t\tstruct reftable_log_record dest = { NULL };\n +\t\tint err = reftable_stack_read_log(st, refs[i].refname, &dest);\n -+\t\tassert_err(err);\n -+\t\tassert(reftable_log_record_equal(&dest, logs + i, SHA1_SIZE));\n -+\t\treftable_log_record_clear(&dest);\n ++\t\tEXPECT_ERR(err);\n ++\t\tEXPECT(reftable_log_record_equal(&dest, logs + i, SHA1_SIZE));\n ++\t\treftable_log_record_release(&dest);\n +\t}\n +\n +\t/* cleanup */\n +\treftable_stack_destroy(st);\n +\tfor (i = 0; i < N; i++) {\n -+\t\treftable_ref_record_clear(&refs[i]);\n -+\t\treftable_log_record_clear(&logs[i]);\n ++\t\treftable_ref_record_release(&refs[i]);\n ++\t\treftable_log_record_release(&logs[i]);\n +\t}\n -+\treftable_clear_dir(dir);\n ++\tclear_dir(dir);\n +}\n +\n +static void test_reftable_stack_log_normalize(void)\n @@ reftable/stack_test.c (new)\n +\t\t.update_index = 1,\n +\t};\n +\n -+\tassert(mkdtemp(dir));\n ++\tEXPECT(mkdtemp(dir));\n +\terr = reftable_new_stack(&st, dir, cfg);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\tinput.message = \"one\\ntwo\";\n +\terr = reftable_stack_add(st, &write_test_log, &arg);\n -+\tassert(err == REFTABLE_API_ERROR);\n ++\tEXPECT(err == REFTABLE_API_ERROR);\n +\n +\tinput.message = \"one\";\n +\terr = reftable_stack_add(st, &write_test_log, &arg);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_stack_read_log(st, input.refname, &dest);\n -+\tassert_err(err);\n -+\tassert(0 == strcmp(dest.message, \"one\\n\"));\n ++\tEXPECT_ERR(err);\n ++\tEXPECT(0 == strcmp(dest.message, \"one\\n\"));\n +\n +\tinput.message = \"two\\n\";\n +\targ.update_index = 2;\n +\terr = reftable_stack_add(st, &write_test_log, &arg);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\terr = reftable_stack_read_log(st, input.refname, &dest);\n -+\tassert_err(err);\n -+\tassert(0 == strcmp(dest.message, \"two\\n\"));\n ++\tEXPECT_ERR(err);\n ++\tEXPECT(0 == strcmp(dest.message, \"two\\n\"));\n +\n +\t/* cleanup */\n +\treftable_stack_destroy(st);\n -+\treftable_log_record_clear(&dest);\n -+\treftable_clear_dir(dir);\n ++\treftable_log_record_release(&dest);\n ++\tclear_dir(dir);\n +}\n +\n +static void test_reftable_stack_tombstone(void)\n @@ reftable/stack_test.c (new)\n +\tstruct reftable_ref_record dest = { NULL };\n +\tstruct reftable_log_record log_dest = { NULL };\n +\n -+\tassert(mkdtemp(dir));\n ++\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st, dir, cfg);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\tfor (i = 0; i < N; i++) {\n +\t\tconst char *buf = \"branch\";\n @@ reftable/stack_test.c (new)\n +\t}\n +\tfor (i = 0; i < N; i++) {\n +\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n -+\t\tassert_err(err);\n ++\t\tEXPECT_ERR(err);\n +\t}\n +\tfor (i = 0; i < N; i++) {\n +\t\tstruct write_log_arg arg = {\n @@ reftable/stack_test.c (new)\n +\t\t\t.update_index = reftable_stack_next_update_index(st),\n +\t\t};\n +\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n -+\t\tassert_err(err);\n ++\t\tEXPECT_ERR(err);\n +\t}\n +\n +\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n -+\tassert(err == 1);\n -+\treftable_ref_record_clear(&dest);\n ++\tEXPECT(err == 1);\n ++\treftable_ref_record_release(&dest);\n +\n +\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n -+\tassert(err == 1);\n -+\treftable_log_record_clear(&log_dest);\n ++\tEXPECT(err == 1);\n ++\treftable_log_record_release(&log_dest);\n +\n +\terr = reftable_stack_compact_all(st, NULL);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n -+\tassert(err == 1);\n ++\tEXPECT(err == 1);\n +\n +\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n -+\tassert(err == 1);\n -+\treftable_ref_record_clear(&dest);\n -+\treftable_log_record_clear(&log_dest);\n ++\tEXPECT(err == 1);\n ++\treftable_ref_record_release(&dest);\n ++\treftable_log_record_release(&log_dest);\n +\n +\t/* cleanup */\n +\treftable_stack_destroy(st);\n +\tfor (i = 0; i < N; i++) {\n -+\t\treftable_ref_record_clear(&refs[i]);\n -+\t\treftable_log_record_clear(&logs[i]);\n ++\t\treftable_ref_record_release(&refs[i]);\n ++\t\treftable_log_record_release(&logs[i]);\n +\t}\n -+\treftable_clear_dir(dir);\n ++\tclear_dir(dir);\n +}\n +\n +static void test_reftable_stack_hash_id(void)\n @@ reftable/stack_test.c (new)\n +\tstruct reftable_stack *st_default = NULL;\n +\tstruct reftable_ref_record dest = { NULL };\n +\n -+\tassert(mkdtemp(dir));\n ++\tEXPECT(mkdtemp(dir));\n +\terr = reftable_new_stack(&st, dir, cfg);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_stack_add(st, &write_test_ref, &ref);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\t/* can't read it with the wrong hash ID. */\n +\terr = reftable_new_stack(&st32, dir, cfg32);\n -+\tassert(err == REFTABLE_FORMAT_ERROR);\n ++\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n +\n +\t/* check that we can read it back with default config too. */\n +\terr = reftable_new_stack(&st_default, dir, cfg_default);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n -+\tassert(!strcmp(dest.target, ref.target));\n -+\treftable_ref_record_clear(&dest);\n ++\tEXPECT(!strcmp(dest.target, ref.target));\n ++\treftable_ref_record_release(&dest);\n +\treftable_stack_destroy(st);\n +\treftable_stack_destroy(st_default);\n -+\treftable_clear_dir(dir);\n ++\tclear_dir(dir);\n +}\n +\n +static void test_log2(void)\n +{\n -+\tassert(1 == fastlog2(3));\n -+\tassert(2 == fastlog2(4));\n -+\tassert(2 == fastlog2(5));\n ++\tEXPECT(1 == fastlog2(3));\n ++\tEXPECT(2 == fastlog2(4));\n ++\tEXPECT(2 == fastlog2(5));\n +}\n +\n +static void test_sizes_to_segments(void)\n @@ reftable/stack_test.c (new)\n +\tint seglen = 0;\n +\tstruct segment *segs =\n +\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n -+\tassert(segs[2].log == 3);\n -+\tassert(segs[2].start == 5);\n -+\tassert(segs[2].end == 6);\n ++\tEXPECT(segs[2].log == 3);\n ++\tEXPECT(segs[2].start == 5);\n ++\tEXPECT(segs[2].end == 6);\n +\n -+\tassert(segs[1].log == 2);\n -+\tassert(segs[1].start == 2);\n -+\tassert(segs[1].end == 5);\n ++\tEXPECT(segs[1].log == 2);\n ++\tEXPECT(segs[1].start == 2);\n ++\tEXPECT(segs[1].end == 5);\n +\treftable_free(segs);\n +}\n +\n @@ reftable/stack_test.c (new)\n +\tint seglen = 0;\n +\tstruct segment *segs =\n +\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n -+\tassert(seglen == 0);\n ++\tEXPECT(seglen == 0);\n +\treftable_free(segs);\n +}\n +\n @@ reftable/stack_test.c (new)\n +\tint seglen = 0;\n +\tstruct segment *segs =\n +\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n -+\tassert(seglen == 1);\n -+\tassert(segs[0].start == 0);\n -+\tassert(segs[0].end == 2);\n ++\tEXPECT(seglen == 1);\n ++\tEXPECT(segs[0].start == 0);\n ++\tEXPECT(segs[0].end == 2);\n +\treftable_free(segs);\n +}\n +\n @@ reftable/stack_test.c (new)\n +\t/* .................0 1 2 3 4 5 6 */\n +\tstruct segment min =\n +\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n -+\tassert(min.start == 2);\n -+\tassert(min.end == 7);\n ++\tEXPECT(min.start == 2);\n ++\tEXPECT(min.end == 7);\n +}\n +\n +static void test_suggest_compaction_segment_nothing(void)\n @@ reftable/stack_test.c (new)\n +\tuint64_t sizes[] = { 64, 32, 16, 8, 4, 2 };\n +\tstruct segment result =\n +\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n -+\tassert(result.start == result.end);\n ++\tEXPECT(result.start == result.end);\n +}\n +\n +static void test_reflog_expire(void)\n @@ reftable/stack_test.c (new)\n +\t};\n +\tstruct reftable_log_record log = { NULL };\n +\n -+\tassert(mkdtemp(dir));\n ++\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st, dir, cfg);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\tfor (i = 1; i <= N; i++) {\n +\t\tchar buf[256];\n @@ reftable/stack_test.c (new)\n +\t\t\t.update_index = reftable_stack_next_update_index(st),\n +\t\t};\n +\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n -+\t\tassert_err(err);\n ++\t\tEXPECT_ERR(err);\n +\t}\n +\n +\terr = reftable_stack_compact_all(st, NULL);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_stack_compact_all(st, &expiry);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_stack_read_log(st, logs[9].refname, &log);\n -+\tassert(err == 1);\n ++\tEXPECT(err == 1);\n +\n +\terr = reftable_stack_read_log(st, logs[11].refname, &log);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\texpiry.min_update_index = 15;\n +\terr = reftable_stack_compact_all(st, &expiry);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_stack_read_log(st, logs[14].refname, &log);\n -+\tassert(err == 1);\n ++\tEXPECT(err == 1);\n +\n +\terr = reftable_stack_read_log(st, logs[16].refname, &log);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\t/* cleanup */\n +\treftable_stack_destroy(st);\n +\tfor (i = 0; i <= N; i++) {\n -+\t\treftable_log_record_clear(&logs[i]);\n ++\t\treftable_log_record_release(&logs[i]);\n +\t}\n -+\treftable_clear_dir(dir);\n -+\treftable_log_record_clear(&log);\n ++\tclear_dir(dir);\n ++\treftable_log_record_release(&log);\n +}\n +\n +static int write_nothing(struct reftable_writer *wr, void *arg)\n @@ reftable/stack_test.c (new)\n +\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n +\tstruct reftable_stack *st2 = NULL;\n +\n -+\tassert(mkdtemp(dir));\n ++\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st, dir, cfg);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_stack_add(st, &write_nothing, NULL);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\terr = reftable_new_stack(&st2, dir, cfg);\n -+\tassert_err(err);\n -+\treftable_clear_dir(dir);\n ++\tEXPECT_ERR(err);\n ++\tclear_dir(dir);\n +\treftable_stack_destroy(st);\n +\treftable_stack_destroy(st2);\n +}\n @@ reftable/stack_test.c (new)\n +\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n +\tint err, i;\n +\tint N = 100;\n -+\tassert(mkdtemp(dir));\n ++\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st, dir, cfg);\n -+\tassert_err(err);\n ++\tEXPECT_ERR(err);\n +\n +\tfor (i = 0; i < N; i++) {\n +\t\tchar name[100];\n @@ reftable/stack_test.c (new)\n +\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n +\n +\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n -+\t\tassert_err(err);\n ++\t\tEXPECT_ERR(err);\n +\n -+\t\tassert(i < 3 || st->merged->stack_len < 2 * fastlog2(i));\n ++\t\tEXPECT(i < 3 || st->merged->stack_len < 2 * fastlog2(i));\n +\t}\n +\n -+\tassert(reftable_stack_compaction_stats(st)->entries_written <\n ++\tEXPECT(reftable_stack_compaction_stats(st)->entries_written <\n +\t (uint64_t)(N * fastlog2(N)));\n +\n +\treftable_stack_destroy(st);\n -+\treftable_clear_dir(dir);\n ++\tclear_dir(dir);\n +}\n +\n +int stack_test_main(int argc, const char *argv[])\n +{\n -+\tadd_test_case(\"test_reftable_stack_uptodate\",\n -+\t\t &test_reftable_stack_uptodate);\n -+\tadd_test_case(\"test_reftable_stack_transaction_api\",\n -+\t\t &test_reftable_stack_transaction_api);\n -+\tadd_test_case(\"test_reftable_stack_hash_id\",\n -+\t\t &test_reftable_stack_hash_id);\n -+\tadd_test_case(\"test_sizes_to_segments_all_equal\",\n -+\t\t &test_sizes_to_segments_all_equal);\n -+\tadd_test_case(\"test_reftable_stack_auto_compaction\",\n -+\t\t &test_reftable_stack_auto_compaction);\n -+\tadd_test_case(\"test_reftable_stack_validate_refname\",\n -+\t\t &test_reftable_stack_validate_refname);\n -+\tadd_test_case(\"test_reftable_stack_update_index_check\",\n -+\t\t &test_reftable_stack_update_index_check);\n -+\tadd_test_case(\"test_reftable_stack_lock_failure\",\n -+\t\t &test_reftable_stack_lock_failure);\n -+\tadd_test_case(\"test_reftable_stack_log_normalize\",\n -+\t\t &test_reftable_stack_log_normalize);\n -+\tadd_test_case(\"test_reftable_stack_tombstone\",\n -+\t\t &test_reftable_stack_tombstone);\n -+\tadd_test_case(\"test_reftable_stack_add_one\",\n -+\t\t &test_reftable_stack_add_one);\n -+\tadd_test_case(\"test_empty_add\", test_empty_add);\n -+\tadd_test_case(\"test_reflog_expire\", test_reflog_expire);\n -+\tadd_test_case(\"test_suggest_compaction_segment\",\n -+\t\t &test_suggest_compaction_segment);\n -+\tadd_test_case(\"test_suggest_compaction_segment_nothing\",\n -+\t\t &test_suggest_compaction_segment_nothing);\n -+\tadd_test_case(\"test_sizes_to_segments\", &test_sizes_to_segments);\n -+\tadd_test_case(\"test_sizes_to_segments_empty\",\n -+\t\t &test_sizes_to_segments_empty);\n -+\tadd_test_case(\"test_log2\", &test_log2);\n -+\tadd_test_case(\"test_parse_names\", &test_parse_names);\n -+\tadd_test_case(\"test_read_file\", &test_read_file);\n -+\tadd_test_case(\"test_names_equal\", &test_names_equal);\n -+\tadd_test_case(\"test_reftable_stack_add\", &test_reftable_stack_add);\n -+\treturn test_main(argc, argv);\n ++\ttest_reftable_stack_uptodate();\n ++\ttest_reftable_stack_transaction_api();\n ++\ttest_reftable_stack_hash_id();\n ++\ttest_sizes_to_segments_all_equal();\n ++\ttest_reftable_stack_auto_compaction();\n ++\ttest_reftable_stack_validate_refname();\n ++\ttest_reftable_stack_update_index_check();\n ++\ttest_reftable_stack_lock_failure();\n ++\ttest_reftable_stack_log_normalize();\n ++\ttest_reftable_stack_tombstone();\n ++\ttest_reftable_stack_add_one();\n ++\ttest_empty_add();\n ++\ttest_reflog_expire();\n ++\ttest_suggest_compaction_segment();\n ++\ttest_suggest_compaction_segment_nothing();\n ++\ttest_sizes_to_segments();\n ++\ttest_sizes_to_segments_empty();\n ++\ttest_log2();\n ++\ttest_parse_names();\n ++\ttest_read_file();\n ++\ttest_names_equal();\n ++\ttest_reftable_stack_add();\n ++\treturn 0;\n +}\n \n - ## reftable/update.sh (new) ##\n -@@\n -+#!/bin/sh\n -+\n -+set -eu\n -+\n -+# Override this to import from somewhere else, say \"../reftable\".\n -+SRC=${SRC:-origin}\n -+BRANCH=${BRANCH:-master}\n -+\n -+((git --git-dir reftable-repo/.git fetch -f ${SRC} ${BRANCH}:import && cd reftable-repo && git checkout -f $(git rev-parse import) ) ||\n -+ git clone https://github.com/google/reftable reftable-repo)\n -+\n -+cp reftable-repo/c/*.[ch] reftable/\n -+cp reftable-repo/c/include/*.[ch] reftable/\n -+cp reftable-repo/LICENSE reftable/\n -+\n -+git --git-dir reftable-repo/.git show --no-patch --format=oneline HEAD \\\n -+ > reftable/VERSION\n -+\n -+mv reftable/system.h reftable/system.h~\n -+sed 's|if REFTABLE_IN_GITCORE|if 1 /* REFTABLE_IN_GITCORE */|' < reftable/system.h~ > reftable/system.h\n -+\n -+git add reftable/*.[ch] reftable/LICENSE reftable/VERSION\n -\n ## t/helper/test-reftable.c ##\n -@@\n - int cmd__reftable(int argc, const char **argv)\n +@@ t/helper/test-reftable.c: int cmd__reftable(int argc, const char **argv)\n {\n + \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n +\tmerged_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n +\trefname_test_main(argc, argv);\n \treftable_test_main(argc, argv);\n - \tstrbuf_test_main(argc, argv);\n +\tstack_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n -: ---------- > 14: a590865a70 Reftable support for git-core\n -: ---------- > 15: 57626bfe2d git-prompt: prepare for reftable refs backend\n 13: c535f838d6 ! 16: 6229da992e reftable: \"test-tool dump-reftable\" command.\n @@ Metadata\n Author: Han-Wen Nienhuys \n \n ## Commit message ##\n - reftable: \"test-tool dump-reftable\" command.\n + Add \"test-tool dump-reftable\" command.\n \n This command dumps individual tables or a stack of of tables.\n \n Signed-off-by: Han-Wen Nienhuys \n \n ## Makefile ##\n -@@ Makefile: REFTABLE_OBJS += reftable/writer.o\n - REFTABLE_OBJS += reftable/zlib-compat.o\n +@@ Makefile: REFTABLE_OBJS += reftable/zlib-compat.o\n \n + REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n +REFTABLE_TEST_OBJS += reftable/dump.o\n -+REFTABLE_TEST_OBJS += reftable/merged_test.o\n + REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/refname_test.o\n - REFTABLE_TEST_OBJS += reftable/reftable_test.o\n \n - ## reftable/iter.c ##\n -@@ reftable/iter.c: void reftable_iterator_destroy(struct reftable_iterator *it)\n - int reftable_iterator_next_ref(struct reftable_iterator *it,\n - \t\t\t struct reftable_ref_record *ref)\n - {\n --\tstruct reftable_record rec = { 0 };\n -+\tstruct reftable_record rec = { NULL };\n - \treftable_record_from_ref(&rec, ref);\n - \treturn iterator_next(it, &rec);\n - }\n -@@ reftable/iter.c: int reftable_iterator_next_ref(struct reftable_iterator *it,\n - int reftable_iterator_next_log(struct reftable_iterator *it,\n - \t\t\t struct reftable_log_record *log)\n - {\n --\tstruct reftable_record rec = { 0 };\n -+\tstruct reftable_record rec = { NULL };\n - \treftable_record_from_log(&rec, log);\n - \treturn iterator_next(it, &rec);\n - }\n -@@ reftable/iter.c: static int filtering_ref_iterator_next(void *iter_arg,\n - \t\t}\n + ## reftable/dump.c ##\n +@@ reftable/dump.c: license that can be found in the LICENSE file or at\n + #include \n + #include \n \n - \t\tif (fri->double_check) {\n --\t\t\tstruct reftable_iterator it = { 0 };\n -+\t\t\tstruct reftable_iterator it = { NULL };\n +-#include \"reftable.h\"\n ++#include \"reftable-blocksource.h\"\n ++#include \"reftable-error.h\"\n ++#include \"reftable-merged.h\"\n ++#include \"reftable-record.h\"\n + #include \"reftable-tests.h\"\n ++#include \"reftable-writer.h\"\n ++#include \"reftable-iterator.h\"\n ++#include \"reftable-reader.h\"\n ++#include \"reftable-stack.h\"\n \n - \t\t\terr = reftable_table_seek_ref(&fri->tab, &it,\n - \t\t\t\t\t\t ref->refname);\n -\n - ## reftable/reader.c ##\n -@@ reftable/reader.c: static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n - int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n - \t\tconst char *name)\n - {\n --\tstruct reftable_block footer = { 0 };\n --\tstruct reftable_block header = { 0 };\n -+\tstruct reftable_block footer = { NULL };\n -+\tstruct reftable_block header = { NULL };\n - \tint err = 0;\n + static uint32_t hash_id;\n \n - \tmemset(r, 0, sizeof(struct reftable_reader));\n -@@ reftable/reader.c: int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n + static int dump_table(const char *tablename)\n {\n - \tint32_t guess_block_size = r->block_size ? r->block_size :\n - \t\t\t\t\t\t\t DEFAULT_BLOCK_SIZE;\n --\tstruct reftable_block block = { 0 };\n -+\tstruct reftable_block block = { NULL };\n - \tuint8_t block_typ = 0;\n - \tint err = 0;\n - \tuint32_t header_off = next_off ? 0 : header_size(r->version);\n -@@ reftable/reader.c: static int reader_seek_indexed(struct reftable_reader *r,\n - \t\t\t struct reftable_record *rec)\n +-\tstruct reftable_block_source src = { 0 };\n ++\tstruct reftable_block_source src = { NULL };\n + \tint err = reftable_block_source_from_file(&src, tablename);\n +-\tstruct reftable_iterator it = { 0 };\n +-\tstruct reftable_ref_record ref = { 0 };\n +-\tstruct reftable_log_record log = { 0 };\n ++\tstruct reftable_iterator it = { NULL };\n ++\tstruct reftable_ref_record ref = { NULL };\n ++\tstruct reftable_log_record log = { NULL };\n + \tstruct reftable_reader *r = NULL;\n + \n + \tif (err < 0)\n +@@ reftable/dump.c: static int dump_table(const char *tablename)\n + \t\treftable_ref_record_print(&ref, hash_id);\n + \t}\n + \treftable_iterator_destroy(&it);\n +-\treftable_ref_record_clear(&ref);\n ++\treftable_ref_record_release(&ref);\n + \n + \terr = reftable_reader_seek_log(r, &it, \"\");\n + \tif (err < 0) {\n +@@ reftable/dump.c: static int dump_table(const char *tablename)\n + \t\treftable_log_record_print(&log, hash_id);\n + \t}\n + \treftable_iterator_destroy(&it);\n +-\treftable_log_record_clear(&log);\n ++\treftable_log_record_release(&log);\n + \n + \treftable_reader_free(r);\n + \treturn 0;\n +@@ reftable/dump.c: static int dump_stack(const char *stackdir)\n {\n - \tstruct reftable_index_record want_index = { .last_key = STRBUF_INIT };\n --\tstruct reftable_record want_index_rec = { 0 };\n -+\tstruct reftable_record want_index_rec = { NULL };\n - \tstruct reftable_index_record index_result = { .last_key = STRBUF_INIT };\n --\tstruct reftable_record index_result_rec = { 0 };\n -+\tstruct reftable_record index_result_rec = { NULL };\n - \tstruct table_iter index_iter = TABLE_ITER_INIT;\n - \tstruct table_iter next = TABLE_ITER_INIT;\n - \tint err = 0;\n -@@ reftable/reader.c: int reftable_reader_seek_ref(struct reftable_reader *r,\n - \tstruct reftable_ref_record ref = {\n - \t\t.refname = (char *)name,\n - \t};\n --\tstruct reftable_record rec = { 0 };\n -+\tstruct reftable_record rec = { NULL };\n - \treftable_record_from_ref(&rec, &ref);\n - \treturn reader_seek(r, it, &rec);\n - }\n -@@ reftable/reader.c: int reftable_reader_seek_log_at(struct reftable_reader *r,\n - \t\t.refname = (char *)name,\n - \t\t.update_index = update_index,\n - \t};\n --\tstruct reftable_record rec = { 0 };\n -+\tstruct reftable_record rec = { NULL };\n - \treftable_record_from_log(&rec, &log);\n - \treturn reader_seek(r, it, &rec);\n - }\n -@@ reftable/reader.c: static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n - \t\t.hash_prefix = oid,\n - \t\t.hash_prefix_len = r->object_id_len,\n - \t};\n --\tstruct reftable_record want_rec = { 0 };\n --\tstruct reftable_iterator oit = { 0 };\n --\tstruct reftable_obj_recor"},{"id":"410886","messageId":"f71e82b995dbecffb75767268cb1838c6126325a.1606419752.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v3.git.git.1606419752.gitgitgadget@gmail.com","subject":"[PATCH v3 06/16] reftable: add blocksource, an abstraction for random access reads","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-11-26T19:42:21Z","receivedAt":"2020-11-26T19:42:55Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is usually used with files for storage. However, we abstract\naway this using the blocksource data structure. This has two advantages:\n\n* log blocks are zlib compressed, and handling them is simplified if we can\n discard byte segments from within the block layer.\n\n* for unittests, it is useful to read and write in-memory. The blocksource\n allows us to abstract the data away from on-disk files.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/blocksource.c | 148 ++++++++++++++++++++++++++++++++\n reftable/blocksource.h | 22 +++++\n reftable/reftable-blocksource.h | 50 +++++++++++\n 4 files changed, 221 insertions(+)\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/reftable-blocksource.h\n\ndiff --git a/Makefile b/Makefile\nindex 5ce78772fd..030b3c0204 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2391,6 +2391,7 @@ XDIFF_OBJS += xdiff/xutils.o\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n \n REFTABLE_TEST_OBJS += reftable/test_framework.o\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nnew file mode 100644\nindex 0000000000..25d4d95b52\n--- /dev/null\n+++ b/reftable/blocksource.c\n@@ -0,0 +1,148 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"reftable-blocksource.h\"\n+#include \"reftable-error.h\"\n+\n+static void strbuf_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void strbuf_close(void *b)\n+{\n+}\n+\n+static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct strbuf *b = (struct strbuf *)v;\n+\tassert(off + size <= b->len);\n+\tdest->data = reftable_calloc(size);\n+\tmemcpy(dest->data, b->buf + off, size);\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static uint64_t strbuf_size(void *b)\n+{\n+\treturn ((struct strbuf *)b)->len;\n+}\n+\n+static struct reftable_block_source_vtable strbuf_vtable = {\n+\t.size = &strbuf_size,\n+\t.read_block = &strbuf_read_block,\n+\t.return_block = &strbuf_return_block,\n+\t.close = &strbuf_close,\n+};\n+\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf)\n+{\n+\tassert(bs->ops == NULL);\n+\tbs->ops = &strbuf_vtable;\n+\tbs->arg = buf;\n+}\n+\n+static void malloc_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static struct reftable_block_source_vtable malloc_vtable = {\n+\t.return_block = &malloc_return_block,\n+};\n+\n+static struct reftable_block_source malloc_block_source_instance = {\n+\t.ops = &malloc_vtable,\n+};\n+\n+struct reftable_block_source malloc_block_source(void)\n+{\n+\treturn malloc_block_source_instance;\n+}\n+\n+struct file_block_source {\n+\tint fd;\n+\tuint64_t size;\n+};\n+\n+static uint64_t file_size(void *b)\n+{\n+\treturn ((struct file_block_source *)b)->size;\n+}\n+\n+static void file_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void file_close(void *b)\n+{\n+\tint fd = ((struct file_block_source *)b)->fd;\n+\tif (fd > 0) {\n+\t\tclose(fd);\n+\t\t((struct file_block_source *)b)->fd = 0;\n+\t}\n+\n+\treftable_free(b);\n+}\n+\n+static int file_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct file_block_source *b = (struct file_block_source *)v;\n+\tassert(off + size <= b->size);\n+\tdest->data = reftable_malloc(size);\n+\tif (pread(b->fd, dest->data, size, off) != size)\n+\t\treturn -1;\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static struct reftable_block_source_vtable file_vtable = {\n+\t.size = &file_size,\n+\t.read_block = &file_read_block,\n+\t.return_block = &file_return_block,\n+\t.close = &file_close,\n+};\n+\n+int reftable_block_source_from_file(struct reftable_block_source *bs,\n+\t\t\t\t const char *name)\n+{\n+\tstruct stat st = { 0 };\n+\tint err = 0;\n+\tint fd = open(name, O_RDONLY);\n+\tstruct file_block_source *p = NULL;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\treturn REFTABLE_NOT_EXIST_ERROR;\n+\t\t}\n+\t\treturn -1;\n+\t}\n+\n+\terr = fstat(fd, &st);\n+\tif (err < 0)\n+\t\treturn -1;\n+\n+\tp = reftable_calloc(sizeof(struct file_block_source));\n+\tp->size = st.st_size;\n+\tp->fd = fd;\n+\n+\tassert(bs->ops == NULL);\n+\tbs->ops = &file_vtable;\n+\tbs->arg = p;\n+\treturn 0;\n+}\ndiff --git a/reftable/blocksource.h b/reftable/blocksource.h\nnew file mode 100644\nindex 0000000000..072e2727ad\n--- /dev/null\n+++ b/reftable/blocksource.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCKSOURCE_H\n+#define BLOCKSOURCE_H\n+\n+#include \"system.h\"\n+\n+struct reftable_block_source;\n+\n+/* Create an in-memory block source for reading reftables */\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf);\n+\n+struct reftable_block_source malloc_block_source(void);\n+\n+#endif\ndiff --git a/reftable/reftable-blocksource.h b/reftable/reftable-blocksource.h\nnew file mode 100644\nindex 0000000000..fffd19deb2\n--- /dev/null\n+++ b/reftable/reftable-blocksource.h\n@@ -0,0 +1,50 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_BLOCKSOURCE_H\n+#define REFTABLE_BLOCKSOURCE_H\n+\n+#include \n+\n+/* block_source is a generic wrapper for a seekable readable file.\n+ */\n+struct reftable_block_source {\n+\tstruct reftable_block_source_vtable *ops;\n+\tvoid *arg;\n+};\n+\n+/* a contiguous segment of bytes. It keeps track of its generating block_source\n+ so it can return itself into the pool.\n+*/\n+struct reftable_block {\n+\tuint8_t *data;\n+\tint len;\n+\tstruct reftable_block_source source;\n+};\n+\n+/* block_source_vtable are the operations that make up block_source */\n+struct reftable_block_source_vtable {\n+\t/* returns the size of a block source */\n+\tuint64_t (*size)(void *source);\n+\n+\t/* reads a segment from the block source. It is an error to read\n+\t beyond the end of the block */\n+\tint (*read_block)(void *source, struct reftable_block *dest,\n+\t\t\t uint64_t off, uint32_t size);\n+\t/* mark the block as read; may return the data back to malloc */\n+\tvoid (*return_block)(void *source, struct reftable_block *blockp);\n+\n+\t/* release all resources associated with the block source */\n+\tvoid (*close)(void *source);\n+};\n+\n+/* opens a file on the file system as a block_source */\n+int reftable_block_source_from_file(struct reftable_block_source *block_src,\n+\t\t\t\t const char *name);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"410887","messageId":"987d1d186fd68d604268f9a66118b7f276094daf.1606419752.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v3.git.git.1606419752.gitgitgadget@gmail.com","subject":"[PATCH v3 05/16] reftable: utility functions","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-11-26T19:42:20Z","receivedAt":"2020-11-26T19:42:55Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis commit provides basic utility classes for the reftable library.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 24 +++++-\n reftable/basics.c | 144 +++++++++++++++++++++++++++++++++++\n reftable/basics.h | 56 ++++++++++++++\n reftable/basics_test.c | 56 ++++++++++++++\n reftable/compat.h | 48 ++++++++++++\n reftable/publicbasics.c | 58 ++++++++++++++\n reftable/reftable-malloc.h | 20 +++++\n reftable/reftable-tests.h | 22 ++++++\n reftable/system.h | 32 ++++++++\n reftable/test_framework.c | 23 ++++++\n reftable/test_framework.h | 48 ++++++++++++\n t/helper/test-reftable.c | 9 +++\n t/helper/test-tool.c | 3 +-\n t/helper/test-tool.h | 1 +\n t/t0032-reftable-unittest.sh | 14 ++++\n 15 files changed, 554 insertions(+), 4 deletions(-)\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/basics_test.c\n create mode 100644 reftable/compat.h\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-malloc.h\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 t/helper/test-reftable.c\n create mode 100755 t/t0032-reftable-unittest.sh\n\ndiff --git a/Makefile b/Makefile\nindex dd1cf41e00..5ce78772fd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -732,6 +732,7 @@ TEST_BUILTINS_OBJS += test-read-cache.o\n TEST_BUILTINS_OBJS += test-read-graph.o\n TEST_BUILTINS_OBJS += test-read-midx.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n+TEST_BUILTINS_OBJS += test-reftable.o\n TEST_BUILTINS_OBJS += test-regex.o\n TEST_BUILTINS_OBJS += test-repository.o\n TEST_BUILTINS_OBJS += test-revision-walking.o\n@@ -821,6 +822,8 @@ TEST_SHELL_PATH = $(SHELL_PATH)\n \n LIB_FILE = libgit.a\n XDIFF_LIB = xdiff/lib.a\n+REFTABLE_LIB = reftable/libreftable.a\n+REFTABLE_TEST_LIB = reftable/libreftable_test.a\n \n GENERATED_H += command-list.h\n GENERATED_H += config-list.h\n@@ -1185,7 +1188,7 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n-GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB)\n+GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB)\n EXTLIBS =\n \n GIT_USER_AGENT = git/$(GIT_VERSION)\n@@ -2386,10 +2389,19 @@ XDIFF_OBJS += xdiff/xpatience.o\n XDIFF_OBJS += xdiff/xprepare.o\n XDIFF_OBJS += xdiff/xutils.o\n \n+REFTABLE_OBJS += reftable/basics.o\n+REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/publicbasics.o\n+\n+REFTABLE_TEST_OBJS += reftable/test_framework.o\n+REFTABLE_TEST_OBJS += reftable/basics_test.o\n+\n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \\\n \t$(XDIFF_OBJS) \\\n \t$(FUZZ_OBJS) \\\n+\t$(REFTABLE_OBJS) \\\n+\t$(REFTABLE_TEST_OBJS) \\\n \tcommon-main.o \\\n \tgit.o\n ifndef NO_CURL\n@@ -2541,6 +2553,12 @@ $(LIB_FILE): $(LIB_OBJS)\n $(XDIFF_LIB): $(XDIFF_OBJS)\n \t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n \n+$(REFTABLE_LIB): $(REFTABLE_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n+$(REFTABLE_TEST_LIB): $(REFTABLE_TEST_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n export DEFAULT_EDITOR DEFAULT_PAGER\n \n Documentation/GIT-EXCLUDED-PROGRAMS: FORCE\n@@ -2822,7 +2840,7 @@ perf: all\n \n t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n \n-t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS)\n+t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) $(REFTABLE_TEST_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n \n check-sha1:: t/helper/test-tool$X\n@@ -3151,7 +3169,7 @@ cocciclean:\n clean: profile-clean coverage-clean cocciclean\n \t$(RM) *.res\n \t$(RM) $(OBJECTS)\n-\t$(RM) $(LIB_FILE) $(XDIFF_LIB)\n+\t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) git$X\n \t$(RM) $(TEST_PROGRAMS)\n \t$(RM) $(FUZZ_PROGRAMS)\ndiff --git a/reftable/basics.c b/reftable/basics.c\nnew file mode 100644\nindex 0000000000..915c55e0ee\n--- /dev/null\n+++ b/reftable/basics.c\n@@ -0,0 +1,144 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+\n+void put_be24(uint8_t *out, uint32_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 16) & 0xff);\n+\tout[1] = (uint8_t)((i >> 8) & 0xff);\n+\tout[2] = (uint8_t)(i & 0xff);\n+}\n+\n+uint32_t get_be24(uint8_t *in)\n+{\n+\treturn (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 |\n+\t (uint32_t)(in[2]);\n+}\n+\n+void put_be16(uint8_t *out, uint16_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 8) & 0xff);\n+\tout[1] = (uint8_t)(i & 0xff);\n+}\n+\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n+{\n+\tsize_t lo = 0;\n+\tsize_t hi = sz;\n+\n+\t/* invariant: (hi == sz) || f(hi) == true\n+\t (lo == 0 && f(0) == true) || fi(lo) == false\n+\t */\n+\twhile (hi - lo > 1) {\n+\t\tsize_t mid = lo + (hi - lo) / 2;\n+\n+\t\tint val = f(mid, args);\n+\t\tif (val) {\n+\t\t\thi = mid;\n+\t\t} else {\n+\t\t\tlo = mid;\n+\t\t}\n+\t}\n+\n+\tif (lo == 0) {\n+\t\tif (f(0, args)) {\n+\t\t\treturn 0;\n+\t\t} else {\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\treturn hi;\n+}\n+\n+void free_names(char **a)\n+{\n+\tchar **p = a;\n+\tif (p == NULL) {\n+\t\treturn;\n+\t}\n+\twhile (*p) {\n+\t\treftable_free(*p);\n+\t\tp++;\n+\t}\n+\treftable_free(a);\n+}\n+\n+int names_length(char **names)\n+{\n+\tint len = 0;\n+\tchar **p = names;\n+\twhile (*p) {\n+\t\tp++;\n+\t\tlen++;\n+\t}\n+\treturn len;\n+}\n+\n+void parse_names(char *buf, int size, char ***namesp)\n+{\n+\tchar **names = NULL;\n+\tsize_t names_cap = 0;\n+\tsize_t names_len = 0;\n+\n+\tchar *p = buf;\n+\tchar *end = buf + size;\n+\twhile (p < end) {\n+\t\tchar *next = strchr(p, '\\n');\n+\t\tif (next != NULL) {\n+\t\t\t*next = 0;\n+\t\t} else {\n+\t\t\tnext = end;\n+\t\t}\n+\t\tif (p < next) {\n+\t\t\tif (names_len == names_cap) {\n+\t\t\t\tnames_cap = 2 * names_cap + 1;\n+\t\t\t\tnames = reftable_realloc(\n+\t\t\t\t\tnames, names_cap * sizeof(char *));\n+\t\t\t}\n+\t\t\tnames[names_len++] = xstrdup(p);\n+\t\t}\n+\t\tp = next + 1;\n+\t}\n+\n+\tif (names_len == names_cap) {\n+\t\tnames_cap = 2 * names_cap + 1;\n+\t\tnames = reftable_realloc(names, names_cap * sizeof(char *));\n+\t}\n+\n+\tnames[names_len] = NULL;\n+\t*namesp = names;\n+}\n+\n+int names_equal(char **a, char **b)\n+{\n+\twhile (*a && *b) {\n+\t\tif (strcmp(*a, *b)) {\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\ta++;\n+\t\tb++;\n+\t}\n+\n+\treturn *a == *b;\n+}\n+\n+int common_prefix_size(struct strbuf *a, struct strbuf *b)\n+{\n+\tint p = 0;\n+\twhile (p < a->len && p < b->len) {\n+\t\tif (a->buf[p] != b->buf[p]) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tp++;\n+\t}\n+\n+\treturn p;\n+}\ndiff --git a/reftable/basics.h b/reftable/basics.h\nnew file mode 100644\nindex 0000000000..0948926637\n--- /dev/null\n+++ b/reftable/basics.h\n@@ -0,0 +1,56 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BASICS_H\n+#define BASICS_H\n+\n+/*\n+ * miscellaneous utilities that are not provided by Git.\n+ */\n+\n+#include \"system.h\"\n+\n+/* Bigendian en/decoding of integers */\n+\n+void put_be24(uint8_t *out, uint32_t i);\n+uint32_t get_be24(uint8_t *in);\n+void put_be16(uint8_t *out, uint16_t i);\n+\n+/*\n+ find smallest index i in [0, sz) at which f(i) is true, assuming\n+ that f is ascending. Return sz if f(i) is false for all indices.\n+*/\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args);\n+\n+/*\n+ Frees a NULL terminated array of malloced strings. The array itself is also\n+ freed.\n+ */\n+void free_names(char **a);\n+\n+/* parse a newline separated list of names. Empty names are discarded. */\n+void parse_names(char *buf, int size, char ***namesp);\n+\n+/* compares two NULL-terminated arrays of strings. */\n+int names_equal(char **a, char **b);\n+\n+/* returns the array size of a NULL-terminated array of strings. */\n+int names_length(char **names);\n+\n+/* Allocation routines; they invoke the functions set through\n+ * reftable_set_alloc() */\n+void *reftable_malloc(size_t sz);\n+void *reftable_realloc(void *p, size_t sz);\n+void reftable_free(void *p);\n+void *reftable_calloc(size_t sz);\n+\n+/* Find the longest shared prefix size of `a` and `b` */\n+struct strbuf;\n+int common_prefix_size(struct strbuf *a, struct strbuf *b);\n+\n+#endif\ndiff --git a/reftable/basics_test.c b/reftable/basics_test.c\nnew file mode 100644\nindex 0000000000..593d3e608a\n--- /dev/null\n+++ b/reftable/basics_test.c\n@@ -0,0 +1,56 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct binsearch_args {\n+\tint key;\n+\tint *arr;\n+};\n+\n+static int binsearch_func(size_t i, void *void_args)\n+{\n+\tstruct binsearch_args *args = (struct binsearch_args *)void_args;\n+\n+\treturn args->key < args->arr[i];\n+}\n+\n+static void test_binsearch(void)\n+{\n+\tint arr[] = { 2, 4, 6, 8, 10 };\n+\tsize_t sz = ARRAY_SIZE(arr);\n+\tstruct binsearch_args args = {\n+\t\t.arr = arr,\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 1; i < 11; i++) {\n+\t\tint res;\n+\t\targs.key = i;\n+\t\tres = binsearch(sz, &binsearch_func, &args);\n+\n+\t\tif (res < sz) {\n+\t\t\tEXPECT(args.key < arr[res]);\n+\t\t\tif (res > 0) {\n+\t\t\t\tEXPECT(args.key >= arr[res - 1]);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tEXPECT(args.key == 10 || args.key == 11);\n+\t\t}\n+\t}\n+}\n+\n+int basics_test_main(int argc, const char *argv[])\n+{\n+\ttest_binsearch();\n+\treturn 0;\n+}\ndiff --git a/reftable/compat.h b/reftable/compat.h\nnew file mode 100644\nindex 0000000000..a765c57e96\n--- /dev/null\n+++ b/reftable/compat.h\n@@ -0,0 +1,48 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef COMPAT_H\n+#define COMPAT_H\n+\n+#include \"system.h\"\n+\n+#ifdef REFTABLE_STANDALONE\n+\n+/* functions that git-core provides, for standalone compilation */\n+#include \n+\n+uint64_t get_be64(void *in);\n+void put_be64(void *out, uint64_t i);\n+\n+void put_be32(void *out, uint32_t i);\n+uint32_t get_be32(uint8_t *in);\n+\n+uint16_t get_be16(uint8_t *in);\n+\n+#define ARRAY_SIZE(a) sizeof((a)) / sizeof((a)[0])\n+#define FREE_AND_NULL(x) \\\n+\tdo { \\\n+\t\treftable_free(x); \\\n+\t\t(x) = NULL; \\\n+\t} while (0)\n+#define QSORT(arr, n, cmp) qsort(arr, n, sizeof(arr[0]), cmp)\n+#define SWAP(a, b) \\\n+\t{ \\\n+\t\tchar tmp[sizeof(a)]; \\\n+\t\tassert(sizeof(a) == sizeof(b)); \\\n+\t\tmemcpy(&tmp[0], &a, sizeof(a)); \\\n+\t\tmemcpy(&a, &b, sizeof(a)); \\\n+\t\tmemcpy(&b, &tmp[0], sizeof(a)); \\\n+\t}\n+\n+char *xstrdup(const char *s);\n+\n+void sleep_millisec(int millisecs);\n+\n+#endif\n+#endif\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\nnew file mode 100644\nindex 0000000000..25639f61af\n--- /dev/null\n+++ b/reftable/publicbasics.c\n@@ -0,0 +1,58 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-malloc.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+static void *(*reftable_malloc_ptr)(size_t sz) = &malloc;\n+static void *(*reftable_realloc_ptr)(void *, size_t) = &realloc;\n+static void (*reftable_free_ptr)(void *) = &free;\n+\n+void *reftable_malloc(size_t sz)\n+{\n+\treturn (*reftable_malloc_ptr)(sz);\n+}\n+\n+void *reftable_realloc(void *p, size_t sz)\n+{\n+\treturn (*reftable_realloc_ptr)(p, sz);\n+}\n+\n+void reftable_free(void *p)\n+{\n+\treftable_free_ptr(p);\n+}\n+\n+void *reftable_calloc(size_t sz)\n+{\n+\tvoid *p = reftable_malloc(sz);\n+\tmemset(p, 0, sz);\n+\treturn p;\n+}\n+\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n+{\n+\treftable_malloc_ptr = malloc;\n+\treftable_realloc_ptr = realloc;\n+\treftable_free_ptr = free;\n+}\n+\n+int hash_size(uint32_t id)\n+{\n+\tswitch (id) {\n+\tcase 0:\n+\tcase SHA1_ID:\n+\t\treturn SHA1_SIZE;\n+\tcase SHA256_ID:\n+\t\treturn SHA256_SIZE;\n+\t}\n+\tabort();\n+}\ndiff --git a/reftable/reftable-malloc.h b/reftable/reftable-malloc.h\nnew file mode 100644\nindex 0000000000..f3571b5f4f\n--- /dev/null\n+++ b/reftable/reftable-malloc.h\n@@ -0,0 +1,20 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_H\n+#define REFTABLE_H\n+\n+#include \n+\n+/*\n+ Overrides the functions to use for memory management.\n+ */\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n+\n+#endif\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nnew file mode 100644\nindex 0000000000..5e7698ae65\n--- /dev/null\n+++ b/reftable/reftable-tests.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_TESTS_H\n+#define REFTABLE_TESTS_H\n+\n+int basics_test_main(int argc, const char **argv);\n+int block_test_main(int argc, const char **argv);\n+int merged_test_main(int argc, const char **argv);\n+int record_test_main(int argc, const char **argv);\n+int refname_test_main(int argc, const char **argv);\n+int reftable_test_main(int argc, const char **argv);\n+int stack_test_main(int argc, const char **argv);\n+int tree_test_main(int argc, const char **argv);\n+int reftable_dump_main(int argc, char *const *argv);\n+\n+#endif\ndiff --git a/reftable/system.h b/reftable/system.h\nnew file mode 100644\nindex 0000000000..07277ca062\n--- /dev/null\n+++ b/reftable/system.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef SYSTEM_H\n+#define SYSTEM_H\n+\n+#include \"git-compat-util.h\"\n+#include \"strbuf.h\"\n+\n+#include \n+\n+struct strbuf;\n+/* In git, this is declared in dir.h */\n+int remove_dir_recursively(struct strbuf *path, int flags);\n+\n+#define SHA1_ID 0x73686131\n+#define SHA256_ID 0x73323536\n+#define SHA1_SIZE 20\n+#define SHA256_SIZE 32\n+\n+/* This is uncompress2, which is only available in zlib as of 2017.\n+ */\n+int uncompress_return_consumed(Bytef *dest, uLongf *destLen,\n+\t\t\t const Bytef *source, uLong *sourceLen);\n+int hash_size(uint32_t id);\n+\n+#endif\ndiff --git a/reftable/test_framework.c b/reftable/test_framework.c\nnew file mode 100644\nindex 0000000000..a5ff4e2a2d\n--- /dev/null\n+++ b/reftable/test_framework.c\n@@ -0,0 +1,23 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"test_framework.h\"\n+\n+#include \"basics.h\"\n+\n+void set_test_hash(uint8_t *p, int i)\n+{\n+\tmemset(p, (uint8_t)i, hash_size(SHA1_ID));\n+}\n+\n+int strbuf_add_void(void *b, const void *data, size_t sz)\n+{\n+\tstrbuf_add((struct strbuf *)b, data, sz);\n+\treturn sz;\n+}\ndiff --git a/reftable/test_framework.h b/reftable/test_framework.h\nnew file mode 100644\nindex 0000000000..18eec29aba\n--- /dev/null\n+++ b/reftable/test_framework.h\n@@ -0,0 +1,48 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TEST_FRAMEWORK_H\n+#define TEST_FRAMEWORK_H\n+\n+#include \"system.h\"\n+#include \"reftable-error.h\"\n+\n+#define EXPECT_ERR(c) \\\n+\tif (c != 0) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: error == %d (%s), want 0\\n\", \\\n+\t\t\t__FILE__, __LINE__, c, reftable_error_str(c)); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define EXPECT_STREQ(a, b) \\\n+\tif (strcmp(a, b)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s:%d: %s (%s) != %s (%s)\\n\", __FILE__, \\\n+\t\t\t__LINE__, #a, a, #b, b); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define EXPECT(c) \\\n+\tif (!(c)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: failed assertion %s\\n\", __FILE__, \\\n+\t\t\t__LINE__, #c); \\\n+\t\tabort(); \\\n+\t}\n+\n+void set_test_hash(uint8_t *p, int i);\n+\n+/* Like strbuf_add, but suitable for passing to reftable_new_writer\n+ */\n+int strbuf_add_void(void *b, const void *data, size_t sz);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nnew file mode 100644\nindex 0000000000..3b58e423e7\n--- /dev/null\n+++ b/t/helper/test-reftable.c\n@@ -0,0 +1,9 @@\n+#include \"reftable/reftable-tests.h\"\n+#include \"test-tool.h\"\n+\n+int cmd__reftable(int argc, const char **argv)\n+{\n+\tbasics_test_main(argc, argv);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 9d6d14d929..0208a0a41c 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -48,13 +48,14 @@ static struct test_cmd cmds[] = {\n \t{ \"path-utils\", cmd__path_utils },\n \t{ \"pkt-line\", cmd__pkt_line },\n \t{ \"prio-queue\", cmd__prio_queue },\n-\t{ \"proc-receive\", cmd__proc_receive},\n+\t{ \"proc-receive\", cmd__proc_receive },\n \t{ \"progress\", cmd__progress },\n \t{ \"reach\", cmd__reach },\n \t{ \"read-cache\", cmd__read_cache },\n \t{ \"read-graph\", cmd__read_graph },\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n+\t{ \"reftable\", cmd__reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex a6470ff62c..1de39ce5b5 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -44,6 +44,7 @@ int cmd__read_cache(int argc, const char **argv);\n int cmd__read_graph(int argc, const char **argv);\n int cmd__read_midx(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n+int cmd__reftable(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__repository(int argc, const char **argv);\n int cmd__revision_walking(int argc, const char **argv);\ndiff --git a/t/t0032-reftable-unittest.sh b/t/t0032-reftable-unittest.sh\nnew file mode 100755\nindex 0000000000..10e21a0590\n--- /dev/null\n+++ b/t/t0032-reftable-unittest.sh\n@@ -0,0 +1,14 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable unittests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'unittests' '\n+ test-tool reftable\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"410888","messageId":"24afac9c91a2eaf1529433a136d0f43aeae71550.1606419752.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v3.git.git.1606419752.gitgitgadget@gmail.com","subject":"[PATCH v3 09/16] reftable: a generic binary tree implementation","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-11-26T19:42:24Z","receivedAt":"2020-11-26T19:42:55Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format includes support for an (OID => ref) map. This map can speed\nup visibility and reachability checks. In particular, various operations along\nthe fetch/push path within Gerrit have ben sped up by using this structure.\n\nThe map is constructed with help of a binary tree. Object IDs are hashes, so\nthey are uniformly distributed. Hence, the tree does not attempt forced\nrebalancing.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 4 ++-\n reftable/tree.c | 63 ++++++++++++++++++++++++++++++++++++++++\n reftable/tree.h | 34 ++++++++++++++++++++++\n reftable/tree_test.c | 61 ++++++++++++++++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 162 insertions(+), 1 deletion(-)\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n\ndiff --git a/Makefile b/Makefile\nindex fcfe4075c3..89564d4c06 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2395,12 +2395,14 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/zlib-compat.o\n \n+REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n-REFTABLE_TEST_OBJS += reftable/basics_test.o\n+REFTABLE_TEST_OBJS += reftable/tree_test.o\n \n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \\\ndiff --git a/reftable/tree.c b/reftable/tree.c\nnew file mode 100644\nindex 0000000000..0061d14e30\n--- /dev/null\n+++ b/reftable/tree.c\n@@ -0,0 +1,63 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert)\n+{\n+\tint res;\n+\tif (*rootp == NULL) {\n+\t\tif (!insert) {\n+\t\t\treturn NULL;\n+\t\t} else {\n+\t\t\tstruct tree_node *n =\n+\t\t\t\treftable_calloc(sizeof(struct tree_node));\n+\t\t\tn->key = key;\n+\t\t\t*rootp = n;\n+\t\t\treturn *rootp;\n+\t\t}\n+\t}\n+\n+\tres = compare(key, (*rootp)->key);\n+\tif (res < 0)\n+\t\treturn tree_search(key, &(*rootp)->left, compare, insert);\n+\telse if (res > 0)\n+\t\treturn tree_search(key, &(*rootp)->right, compare, insert);\n+\treturn *rootp;\n+}\n+\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg)\n+{\n+\tif (t->left != NULL) {\n+\t\tinfix_walk(t->left, action, arg);\n+\t}\n+\taction(arg, t->key);\n+\tif (t->right != NULL) {\n+\t\tinfix_walk(t->right, action, arg);\n+\t}\n+}\n+\n+void tree_free(struct tree_node *t)\n+{\n+\tif (t == NULL) {\n+\t\treturn;\n+\t}\n+\tif (t->left != NULL) {\n+\t\ttree_free(t->left);\n+\t}\n+\tif (t->right != NULL) {\n+\t\ttree_free(t->right);\n+\t}\n+\treftable_free(t);\n+}\ndiff --git a/reftable/tree.h b/reftable/tree.h\nnew file mode 100644\nindex 0000000000..954512e9a3\n--- /dev/null\n+++ b/reftable/tree.h\n@@ -0,0 +1,34 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TREE_H\n+#define TREE_H\n+\n+/* tree_node is a generic binary search tree. */\n+struct tree_node {\n+\tvoid *key;\n+\tstruct tree_node *left, *right;\n+};\n+\n+/* looks for `key` in `rootp` using `compare` as comparison function. If insert\n+ is set, insert the key if it's not found. Else, return NULL.\n+*/\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert);\n+\n+/* performs an infix walk of the tree. */\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg);\n+\n+/*\n+ deallocates the tree nodes recursively. Keys should be deallocated separately\n+ by walking over the tree. */\n+void tree_free(struct tree_node *t);\n+\n+#endif\ndiff --git a/reftable/tree_test.c b/reftable/tree_test.c\nnew file mode 100644\nindex 0000000000..6de9c9dbe5\n--- /dev/null\n+++ b/reftable/tree_test.c\n@@ -0,0 +1,61 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static int test_compare(const void *a, const void *b)\n+{\n+\treturn (char *)a - (char *)b;\n+}\n+\n+struct curry {\n+\tvoid *last;\n+};\n+\n+static void check_increasing(void *arg, void *key)\n+{\n+\tstruct curry *c = (struct curry *)arg;\n+\tif (c->last != NULL) {\n+\t\tassert(test_compare(c->last, key) < 0);\n+\t}\n+\tc->last = key;\n+}\n+\n+static void test_tree(void)\n+{\n+\tstruct tree_node *root = NULL;\n+\n+\tvoid *values[11] = { NULL };\n+\tstruct tree_node *nodes[11] = { NULL };\n+\tint i = 1;\n+\tstruct curry c = { NULL };\n+\tdo {\n+\t\tnodes[i] = tree_search(values + i, &root, &test_compare, 1);\n+\t\ti = (i * 7) % 11;\n+\t} while (i != 1);\n+\n+\tfor (i = 1; i < ARRAY_SIZE(nodes); i++) {\n+\t\tassert(values + i == nodes[i]->key);\n+\t\tassert(nodes[i] ==\n+\t\t tree_search(values + i, &root, &test_compare, 0));\n+\t}\n+\n+\tinfix_walk(root, check_increasing, &c);\n+\ttree_free(root);\n+}\n+\n+int tree_test_main(int argc, const char *argv[])\n+{\n+\ttest_tree();\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex c9deeaf08c..050551fa69 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,5 +6,6 @@ int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"410894","messageId":"a358b052d5610a5c34283b3419ad0317865eb6f6.1606419752.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v3.git.git.1606419752.gitgitgadget@gmail.com","subject":"[PATCH v3 08/16] reftable: reading/writing blocks","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-11-26T19:42:23Z","receivedAt":"2020-11-26T19:42:55Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is structured as a sequence of block. Within a block,\nrecords are prefix compressed, with an index of offsets for fully expand keys to\nenable binary search within blocks.\n\nThis commit provides the logic to read and write these blocks.\n\nIncludes a code snippet copied from zlib\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 3 +\n reftable/.gitattributes | 1 +\n reftable/block.c | 440 +++++++++++++++++++++++++++++++++++++++\n reftable/block.h | 129 ++++++++++++\n reftable/block_test.c | 119 +++++++++++\n reftable/zlib-compat.c | 92 ++++++++\n t/helper/test-reftable.c | 1 +\n 7 files changed, 785 insertions(+)\n create mode 100644 reftable/.gitattributes\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n create mode 100644 reftable/zlib-compat.c\n\ndiff --git a/Makefile b/Makefile\nindex 199aec9267..fcfe4075c3 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2391,10 +2391,13 @@ XDIFF_OBJS += xdiff/xutils.o\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/zlib-compat.o\n \n+REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\ndiff --git a/reftable/.gitattributes b/reftable/.gitattributes\nnew file mode 100644\nindex 0000000000..f44451a379\n--- /dev/null\n+++ b/reftable/.gitattributes\n@@ -0,0 +1 @@\n+/zlib-compat.c\twhitespace=-indent-with-non-tab,-trailing-space\ndiff --git a/reftable/block.c b/reftable/block.c\nnew file mode 100644\nindex 0000000000..5b4e49fb2b\n--- /dev/null\n+++ b/reftable/block.c\n@@ -0,0 +1,440 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"reftable-error.h\"\n+#include \"system.h\"\n+#include \"zlib.h\"\n+\n+int header_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 24;\n+\tcase 2:\n+\t\treturn 28;\n+\t}\n+\tabort();\n+}\n+\n+int footer_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 68;\n+\tcase 2:\n+\t\treturn 72;\n+\t}\n+\tabort();\n+}\n+\n+static int block_writer_register_restart(struct block_writer *w, int n,\n+\t\t\t\t\t int is_restart, struct strbuf *key)\n+{\n+\tint rlen = w->restart_len;\n+\tif (rlen >= MAX_RESTARTS) {\n+\t\tis_restart = 0;\n+\t}\n+\n+\tif (is_restart) {\n+\t\trlen++;\n+\t}\n+\tif (2 + 3 * rlen + n > w->block_size - w->next)\n+\t\treturn -1;\n+\tif (is_restart) {\n+\t\tif (w->restart_len == w->restart_cap) {\n+\t\t\tw->restart_cap = w->restart_cap * 2 + 1;\n+\t\t\tw->restarts = reftable_realloc(\n+\t\t\t\tw->restarts, sizeof(uint32_t) * w->restart_cap);\n+\t\t}\n+\n+\t\tw->restarts[w->restart_len++] = w->next;\n+\t}\n+\n+\tw->next += n;\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, key);\n+\tw->entries++;\n+\treturn 0;\n+}\n+\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n+{\n+\tbw->buf = buf;\n+\tbw->hash_size = hash_size;\n+\tbw->block_size = block_size;\n+\tbw->header_off = header_off;\n+\tbw->buf[header_off] = typ;\n+\tbw->next = header_off + 4;\n+\tbw->restart_interval = 16;\n+\tbw->entries = 0;\n+\tbw->restart_len = 0;\n+\tbw->last_key.len = 0;\n+}\n+\n+uint8_t block_writer_type(struct block_writer *bw)\n+{\n+\treturn bw->buf[bw->header_off];\n+}\n+\n+/* adds the reftable_record to the block. Returns -1 if it does not fit, 0 on\n+ success */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tstruct strbuf last =\n+\t\tw->entries % w->restart_interval == 0 ? empty : w->last_key;\n+\tstruct string_view out = {\n+\t\t.buf = w->buf + w->next,\n+\t\t.len = w->block_size - w->next,\n+\t};\n+\n+\tstruct string_view start = out;\n+\n+\tint is_restart = 0;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint n = 0;\n+\n+\treftable_record_key(rec, &key);\n+\tn = reftable_encode_key(&is_restart, out, last, key,\n+\t\t\t\treftable_record_val_type(rec));\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tn = reftable_record_encode(rec, out, w->hash_size);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tif (block_writer_register_restart(w, start.len - out.len, is_restart,\n+\t\t\t\t\t &key) < 0)\n+\t\tgoto done;\n+\n+\tstrbuf_release(&key);\n+\treturn 0;\n+\n+done:\n+\tstrbuf_release(&key);\n+\treturn -1;\n+}\n+\n+int block_writer_finish(struct block_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->restart_len; i++) {\n+\t\tput_be24(w->buf + w->next, w->restarts[i]);\n+\t\tw->next += 3;\n+\t}\n+\n+\tput_be16(w->buf + w->next, w->restart_len);\n+\tw->next += 2;\n+\tput_be24(w->buf + 1 + w->header_off, w->next);\n+\n+\tif (block_writer_type(w) == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + w->header_off;\n+\t\tuint8_t *compressed = NULL;\n+\t\tint zresult = 0;\n+\t\tuLongf src_len = w->next - block_header_skip;\n+\t\tsize_t dest_cap = src_len;\n+\n+\t\tcompressed = reftable_malloc(dest_cap);\n+\t\twhile (1) {\n+\t\t\tuLongf out_dest_len = dest_cap;\n+\n+\t\t\tzresult = compress2(compressed, &out_dest_len,\n+\t\t\t\t\t w->buf + block_header_skip, src_len,\n+\t\t\t\t\t 9);\n+\t\t\tif (zresult == Z_BUF_ERROR) {\n+\t\t\t\tdest_cap *= 2;\n+\t\t\t\tcompressed =\n+\t\t\t\t\treftable_realloc(compressed, dest_cap);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (Z_OK != zresult) {\n+\t\t\t\treftable_free(compressed);\n+\t\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t\t}\n+\n+\t\t\tmemcpy(w->buf + block_header_skip, compressed,\n+\t\t\t out_dest_len);\n+\t\t\tw->next = out_dest_len + block_header_skip;\n+\t\t\treftable_free(compressed);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\treturn w->next;\n+}\n+\n+uint8_t block_reader_type(struct block_reader *r)\n+{\n+\treturn r->block.data[r->header_off];\n+}\n+\n+int block_reader_init(struct block_reader *br, struct reftable_block *block,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size)\n+{\n+\tuint32_t full_block_size = table_block_size;\n+\tuint8_t typ = block->data[header_off];\n+\tuint32_t sz = get_be24(block->data + header_off + 1);\n+\n+\tuint16_t restart_count = 0;\n+\tuint32_t restart_start = 0;\n+\tuint8_t *restart_bytes = NULL;\n+\n+\tif (!reftable_is_block_type(typ))\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tif (typ == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + header_off;\n+\t\tuLongf dst_len = sz - block_header_skip; /* total size of dest\n+\t\t\t\t\t\t\t buffer. */\n+\t\tuLongf src_len = block->len - block_header_skip;\n+\t\t/* Log blocks specify the *uncompressed* size in their header.\n+\t\t */\n+\t\tuint8_t *uncompressed = reftable_malloc(sz);\n+\n+\t\t/* Copy over the block header verbatim. It's not compressed. */\n+\t\tmemcpy(uncompressed, block->data, block_header_skip);\n+\n+\t\t/* Uncompress */\n+\t\tif (Z_OK != uncompress_return_consumed(\n+\t\t\t\t uncompressed + block_header_skip, &dst_len,\n+\t\t\t\t block->data + block_header_skip,\n+\t\t\t\t &src_len)) {\n+\t\t\treftable_free(uncompressed);\n+\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t}\n+\n+\t\tif (dst_len + block_header_skip != sz)\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\t\t/* We're done with the input data. */\n+\t\treftable_block_done(block);\n+\t\tblock->data = uncompressed;\n+\t\tblock->len = sz;\n+\t\tblock->source = malloc_block_source();\n+\t\tfull_block_size = src_len + block_header_skip;\n+\t} else if (full_block_size == 0) {\n+\t\tfull_block_size = sz;\n+\t} else if (sz < full_block_size && sz < block->len &&\n+\t\t block->data[sz] != 0) {\n+\t\t/* If the block is smaller than the full block size, it is\n+\t\t padded (data followed by '\\0') or the next block is\n+\t\t unaligned. */\n+\t\tfull_block_size = sz;\n+\t}\n+\n+\trestart_count = get_be16(block->data + sz - 2);\n+\trestart_start = sz - 2 - 3 * restart_count;\n+\trestart_bytes = block->data + restart_start;\n+\n+\t/* transfer ownership. */\n+\tbr->block = *block;\n+\tblock->data = NULL;\n+\tblock->len = 0;\n+\n+\tbr->hash_size = hash_size;\n+\tbr->block_len = restart_start;\n+\tbr->full_block_size = full_block_size;\n+\tbr->header_off = header_off;\n+\tbr->restart_count = restart_count;\n+\tbr->restart_bytes = restart_bytes;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t block_reader_restart_offset(struct block_reader *br, int i)\n+{\n+\treturn get_be24(br->restart_bytes + 3 * i);\n+}\n+\n+void block_reader_start(struct block_reader *br, struct block_iter *it)\n+{\n+\tit->br = br;\n+\tstrbuf_reset(&it->last_key);\n+\tit->next_off = br->header_off + 4;\n+}\n+\n+struct restart_find_args {\n+\tint error;\n+\tstruct strbuf key;\n+\tstruct block_reader *r;\n+};\n+\n+static int restart_key_less(size_t idx, void *args)\n+{\n+\tstruct restart_find_args *a = (struct restart_find_args *)args;\n+\tuint32_t off = block_reader_restart_offset(a->r, idx);\n+\tstruct string_view in = {\n+\t\t.buf = a->r->block.data + off,\n+\t\t.len = a->r->block_len - off,\n+\t};\n+\n+\t/* the restart key is verbatim in the block, so this could avoid the\n+\t alloc for decoding the key */\n+\tstruct strbuf rkey = STRBUF_INIT;\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tuint8_t unused_extra;\n+\tint n = reftable_decode_key(&rkey, &unused_extra, last_key, in);\n+\tint result;\n+\tif (n < 0) {\n+\t\ta->error = 1;\n+\t\treturn -1;\n+\t}\n+\n+\tresult = strbuf_cmp(&a->key, &rkey);\n+\tstrbuf_release(&rkey);\n+\treturn result;\n+}\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src)\n+{\n+\tdest->br = src->br;\n+\tdest->next_off = src->next_off;\n+\tstrbuf_reset(&dest->last_key);\n+\tstrbuf_addbuf(&dest->last_key, &src->last_key);\n+}\n+\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec)\n+{\n+\tstruct string_view in = {\n+\t\t.buf = it->br->block.data + it->next_off,\n+\t\t.len = it->br->block_len - it->next_off,\n+\t};\n+\tstruct string_view start = in;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tuint8_t extra = 0;\n+\tint n = 0;\n+\n+\tif (it->next_off >= it->br->block_len)\n+\t\treturn 1;\n+\n+\tn = reftable_decode_key(&key, &extra, it->last_key, in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\n+\tstring_view_consume(&in, n);\n+\tn = reftable_record_decode(rec, key, extra, in, it->br->hash_size);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tstrbuf_reset(&it->last_key);\n+\tstrbuf_addbuf(&it->last_key, &key);\n+\tit->next_off += start.len - in.len;\n+\tstrbuf_release(&key);\n+\treturn 0;\n+}\n+\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tint off = br->header_off + 4;\n+\tstruct string_view in = {\n+\t\t.buf = br->block.data + off,\n+\t\t.len = br->block_len - off,\n+\t};\n+\n+\tuint8_t extra = 0;\n+\tint n = reftable_decode_key(key, &extra, empty, in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\treturn 0;\n+}\n+\n+int block_iter_seek(struct block_iter *it, struct strbuf *want)\n+{\n+\treturn block_reader_seek(it->br, it, want);\n+}\n+\n+void block_iter_close(struct block_iter *it)\n+{\n+\tstrbuf_release(&it->last_key);\n+}\n+\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want)\n+{\n+\tstruct restart_find_args args = {\n+\t\t.key = *want,\n+\t\t.r = br,\n+\t};\n+\tstruct reftable_record rec = reftable_new_record(block_reader_type(br));\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = 0;\n+\tstruct block_iter next = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\n+\tint i = binsearch(br->restart_count, &restart_key_less, &args);\n+\tif (args.error) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tit->br = br;\n+\tif (i > 0) {\n+\t\ti--;\n+\t\tit->next_off = block_reader_restart_offset(br, i);\n+\t} else {\n+\t\tit->next_off = br->header_off + 4;\n+\t}\n+\n+\t/* We're looking for the last entry less/equal than the wanted key, so\n+\t we have to go one entry too far and then back up.\n+\t*/\n+\twhile (1) {\n+\t\tblock_iter_copy_from(&next, it);\n+\t\terr = block_iter_next(&next, &rec);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tif (err > 0 || strbuf_cmp(&key, want) >= 0) {\n+\t\t\terr = 0;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tblock_iter_copy_from(it, &next);\n+\t}\n+\n+done:\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&next.last_key);\n+\treftable_record_destroy(&rec);\n+\n+\treturn err;\n+}\n+\n+void block_writer_release(struct block_writer *bw)\n+{\n+\tFREE_AND_NULL(bw->restarts);\n+\tstrbuf_release(&bw->last_key);\n+\t/* the block is not owned. */\n+}\n+\n+void reftable_block_done(struct reftable_block *blockp)\n+{\n+\tstruct reftable_block_source source = blockp->source;\n+\tif (blockp != NULL && source.ops != NULL)\n+\t\tsource.ops->return_block(source.arg, blockp);\n+\tblockp->data = NULL;\n+\tblockp->len = 0;\n+\tblockp->source.ops = NULL;\n+\tblockp->source.arg = NULL;\n+}\ndiff --git a/reftable/block.h b/reftable/block.h\nnew file mode 100644\nindex 0000000000..85dfd493d2\n--- /dev/null\n+++ b/reftable/block.h\n@@ -0,0 +1,129 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCK_H\n+#define BLOCK_H\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"reftable-blocksource.h\"\n+\n+/*\n+ Writes reftable blocks. The block_writer is reused across blocks to minimize\n+ allocation overhead.\n+*/\n+struct block_writer {\n+\tuint8_t *buf;\n+\tuint32_t block_size;\n+\n+\t/* Offset ofof the global header. Nonzero in the first block only. */\n+\tuint32_t header_off;\n+\n+\t/* How often to restart keys. */\n+\tint restart_interval;\n+\tint hash_size;\n+\n+\t/* Offset of next uint8_t to write. */\n+\tuint32_t next;\n+\tuint32_t *restarts;\n+\tuint32_t restart_len;\n+\tuint32_t restart_cap;\n+\n+\tstruct strbuf last_key;\n+\tint entries;\n+};\n+\n+/*\n+ initializes the blockwriter to write `typ` entries, using `buf` as temporary\n+ storage. `buf` is not owned by the block_writer. */\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n+\n+/*\n+ returns the block type (eg. 'r' for ref records.\n+*/\n+uint8_t block_writer_type(struct block_writer *bw);\n+\n+/* appends the record, or -1 if it doesn't fit. */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec);\n+\n+/* appends the key restarts, and compress the block if necessary. */\n+int block_writer_finish(struct block_writer *w);\n+\n+/* clears out internally allocated block_writer members. */\n+void block_writer_release(struct block_writer *bw);\n+\n+/* Read a block. */\n+struct block_reader {\n+\t/* offset of the block header; nonzero for the first block in a\n+\t * reftable. */\n+\tuint32_t header_off;\n+\n+\t/* the memory block */\n+\tstruct reftable_block block;\n+\tint hash_size;\n+\n+\t/* size of the data, excluding restart data. */\n+\tuint32_t block_len;\n+\tuint8_t *restart_bytes;\n+\tuint16_t restart_count;\n+\n+\t/* size of the data in the file. For log blocks, this is the compressed\n+\t * size. */\n+\tuint32_t full_block_size;\n+};\n+\n+/* Iterate over entries in a block */\n+struct block_iter {\n+\t/* offset within the block of the next entry to read. */\n+\tuint32_t next_off;\n+\tstruct block_reader *br;\n+\n+\t/* key for last entry we read. */\n+\tstruct strbuf last_key;\n+};\n+\n+/* initializes a block reader. */\n+int block_reader_init(struct block_reader *br, struct reftable_block *bl,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size);\n+\n+/* Position `it` at start of the block */\n+void block_reader_start(struct block_reader *br, struct block_iter *it);\n+\n+/* Position `it` to the `want` key in the block */\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want);\n+\n+/* Returns the block type (eg. 'r' for refs) */\n+uint8_t block_reader_type(struct block_reader *r);\n+\n+/* Decodes the first key in the block */\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key);\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src);\n+\n+/* return < 0 for error, 0 for OK, > 0 for EOF. */\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec);\n+\n+/* Seek to `want` with in the block pointed to by `it` */\n+int block_iter_seek(struct block_iter *it, struct strbuf *want);\n+\n+/* deallocate memory for `it`. The block reader and its block is left intact. */\n+void block_iter_close(struct block_iter *it);\n+\n+/* size of file header, depending on format version */\n+int header_size(int version);\n+\n+/* size of file footer, depending on format version */\n+int footer_size(int version);\n+\n+/* returns a block to its source. */\n+void reftable_block_done(struct reftable_block *ret);\n+\n+#endif\ndiff --git a/reftable/block_test.c b/reftable/block_test.c\nnew file mode 100644\nindex 0000000000..597aa10b34\n--- /dev/null\n+++ b/reftable/block_test.c\n@@ -0,0 +1,119 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"system.h\"\n+\n+#include \"blocksource.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_block_read_write(void)\n+{\n+\tconst int header_off = 21; /* random */\n+\tchar *names[30];\n+\tconst int N = ARRAY_SIZE(names);\n+\tconst int block_size = 1024;\n+\tstruct reftable_block block = { NULL };\n+\tstruct block_writer bw = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_record rec = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tstruct block_reader br = { 0 };\n+\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\tint j = 0;\n+\tstruct strbuf want = STRBUF_INIT;\n+\n+\tblock.data = reftable_calloc(block_size);\n+\tblock.len = block_size;\n+\tblock.source = malloc_block_source();\n+\tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n+\t\t\t header_off, hash_size(SHA1_ID));\n+\treftable_record_from_ref(&rec, &ref);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tuint8_t hash[SHA1_SIZE];\n+\t\tsnprintf(name, sizeof(name), \"branch%02d\", i);\n+\t\tmemset(hash, i, sizeof(hash));\n+\n+\t\tref.refname = name;\n+\t\tref.value = hash;\n+\t\tnames[i] = xstrdup(name);\n+\t\tn = block_writer_add(&bw, &rec);\n+\t\tref.refname = NULL;\n+\t\tref.value = NULL;\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tn = block_writer_finish(&bw);\n+\tEXPECT(n > 0);\n+\n+\tblock_writer_release(&bw);\n+\n+\tblock_reader_init(&br, &block, header_off, block_size, SHA1_SIZE);\n+\n+\tblock_reader_start(&br, &it);\n+\n+\twhile (1) {\n+\t\tint r = block_iter_next(&it, &rec);\n+\t\tEXPECT(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tEXPECT_STREQ(names[j], ref.refname);\n+\t\tj++;\n+\t}\n+\n+\treftable_record_release(&rec);\n+\tblock_iter_close(&it);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\t\tstrbuf_reset(&want);\n+\t\tstrbuf_addstr(&want, names[i]);\n+\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tEXPECT(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tEXPECT(n == 0);\n+\n+\t\tEXPECT_STREQ(names[i], ref.refname);\n+\n+\t\twant.len--;\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tEXPECT(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tEXPECT(n == 0);\n+\t\tEXPECT_STREQ(names[10 * (i / 10)], ref.refname);\n+\n+\t\tblock_iter_close(&it);\n+\t}\n+\n+\treftable_record_release(&rec);\n+\treftable_block_done(&br.block);\n+\tstrbuf_release(&want);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+}\n+\n+int block_test_main(int argc, const char *argv[])\n+{\n+\ttest_block_read_write();\n+\treturn 0;\n+}\ndiff --git a/reftable/zlib-compat.c b/reftable/zlib-compat.c\nnew file mode 100644\nindex 0000000000..3e0b0f24f1\n--- /dev/null\n+++ b/reftable/zlib-compat.c\n@@ -0,0 +1,92 @@\n+/* taken from zlib's uncompr.c\n+\n+ commit cacf7f1d4e3d44d871b605da3b647f07d718623f\n+ Author: Mark Adler \n+ Date: Sun Jan 15 09:18:46 2017 -0800\n+\n+ zlib 1.2.11\n+\n+*/\n+\n+/*\n+ * Copyright (C) 1995-2003, 2010, 2014, 2016 Jean-loup Gailly, Mark Adler\n+ * For conditions of distribution and use, see copyright notice in zlib.h\n+ */\n+\n+#include \"system.h\"\n+\n+/* clang-format off */\n+\n+/* ===========================================================================\n+ Decompresses the source buffer into the destination buffer. *sourceLen is\n+ the byte length of the source buffer. Upon entry, *destLen is the total size\n+ of the destination buffer, which must be large enough to hold the entire\n+ uncompressed data. (The size of the uncompressed data must have been saved\n+ previously by the compressor and transmitted to the decompressor by some\n+ mechanism outside the scope of this compression library.) Upon exit,\n+ *destLen is the size of the decompressed data and *sourceLen is the number\n+ of source bytes consumed. Upon return, source + *sourceLen points to the\n+ first unused input byte.\n+\n+ uncompress returns Z_OK if success, Z_MEM_ERROR if there was not enough\n+ memory, Z_BUF_ERROR if there was not enough room in the output buffer, or\n+ Z_DATA_ERROR if the input data was corrupted, including if the input data is\n+ an incomplete zlib stream.\n+*/\n+int ZEXPORT uncompress_return_consumed (\n+ Bytef *dest,\n+ uLongf *destLen,\n+ const Bytef *source,\n+ uLong *sourceLen) {\n+ z_stream stream;\n+ int err;\n+ const uInt max = (uInt)-1;\n+ uLong len, left;\n+ Byte buf[1]; /* for detection of incomplete stream when *destLen == 0 */\n+\n+ len = *sourceLen;\n+ if (*destLen) {\n+ left = *destLen;\n+ *destLen = 0;\n+ }\n+ else {\n+ left = 1;\n+ dest = buf;\n+ }\n+\n+ stream.next_in = (z_const Bytef *)source;\n+ stream.avail_in = 0;\n+ stream.zalloc = (alloc_func)0;\n+ stream.zfree = (free_func)0;\n+ stream.opaque = (voidpf)0;\n+\n+ err = inflateInit(&stream);\n+ if (err != Z_OK) return err;\n+\n+ stream.next_out = dest;\n+ stream.avail_out = 0;\n+\n+ do {\n+ if (stream.avail_out == 0) {\n+ stream.avail_out = left > (uLong)max ? max : (uInt)left;\n+ left -= stream.avail_out;\n+ }\n+ if (stream.avail_in == 0) {\n+ stream.avail_in = len > (uLong)max ? max : (uInt)len;\n+ len -= stream.avail_in;\n+ }\n+ err = inflate(&stream, Z_NO_FLUSH);\n+ } while (err == Z_OK);\n+\n+ *sourceLen -= len + stream.avail_in;\n+ if (dest != buf)\n+ *destLen = stream.total_out;\n+ else if (stream.total_out && err == Z_BUF_ERROR)\n+ left = 1;\n+\n+ inflateEnd(&stream);\n+ return err == Z_STREAM_END ? Z_OK :\n+ err == Z_NEED_DICT ? Z_DATA_ERROR :\n+ err == Z_BUF_ERROR && left + stream.avail_out ? Z_DATA_ERROR :\n+ err;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 09d4b83ef9..c9deeaf08c 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,6 +4,7 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n+\tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"410890","messageId":"bbe2ed71734455661515a5636be3cd0b00b40d25.1606419752.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v3.git.git.1606419752.gitgitgadget@gmail.com","subject":"[PATCH v3 07/16] reftable: (de)serialization for the polymorphic record type.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-11-26T19:42:22Z","receivedAt":"2020-11-26T19:42:56Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is structured as a sequence of blocks, and each block\ncontains a sequence of prefix-compressed key-value records. There are 4 types of\nrecords, and they have similarities in how they must be handled. This is\nachieved by introducing a polymorphic 'record' type that encapsulates ref, log,\nindex and object records.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/constants.h | 21 +\n reftable/record.c | 1116 ++++++++++++++++++++++++++++++++++++\n reftable/record.h | 139 +++++\n reftable/record_test.c | 404 +++++++++++++\n reftable/reftable-record.h | 73 +++\n t/helper/test-reftable.c | 2 +-\n 7 files changed, 1756 insertions(+), 1 deletion(-)\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/reftable-record.h\n\ndiff --git a/Makefile b/Makefile\nindex 030b3c0204..199aec9267 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2393,7 +2393,9 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/record.o\n \n+REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n \ndiff --git a/reftable/constants.h b/reftable/constants.h\nnew file mode 100644\nindex 0000000000..5eee72c4c1\n--- /dev/null\n+++ b/reftable/constants.h\n@@ -0,0 +1,21 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef CONSTANTS_H\n+#define CONSTANTS_H\n+\n+#define BLOCK_TYPE_LOG 'g'\n+#define BLOCK_TYPE_INDEX 'i'\n+#define BLOCK_TYPE_REF 'r'\n+#define BLOCK_TYPE_OBJ 'o'\n+#define BLOCK_TYPE_ANY 0\n+\n+#define MAX_RESTARTS ((1 << 16) - 1)\n+#define DEFAULT_BLOCK_SIZE 4096\n+\n+#endif\ndiff --git a/reftable/record.c b/reftable/record.c\nnew file mode 100644\nindex 0000000000..dbe7a5571e\n--- /dev/null\n+++ b/reftable/record.c\n@@ -0,0 +1,1116 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+/* record.c - methods for different types of records. */\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"constants.h\"\n+#include \"reftable-error.h\"\n+#include \"basics.h\"\n+\n+int get_var_int(uint64_t *dest, struct string_view *in)\n+{\n+\tint ptr = 0;\n+\tuint64_t val;\n+\n+\tif (in->len == 0)\n+\t\treturn -1;\n+\tval = in->buf[ptr] & 0x7f;\n+\n+\twhile (in->buf[ptr] & 0x80) {\n+\t\tptr++;\n+\t\tif (ptr > in->len) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tval = (val + 1) << 7 | (uint64_t)(in->buf[ptr] & 0x7f);\n+\t}\n+\n+\t*dest = val;\n+\treturn ptr + 1;\n+}\n+\n+int put_var_int(struct string_view *dest, uint64_t val)\n+{\n+\tuint8_t buf[10] = { 0 };\n+\tint i = 9;\n+\tint n = 0;\n+\tbuf[i] = (uint8_t)(val & 0x7f);\n+\ti--;\n+\twhile (1) {\n+\t\tval >>= 7;\n+\t\tif (!val) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tval--;\n+\t\tbuf[i] = 0x80 | (uint8_t)(val & 0x7f);\n+\t\ti--;\n+\t}\n+\n+\tn = sizeof(buf) - i - 1;\n+\tif (dest->len < n)\n+\t\treturn -1;\n+\tmemcpy(dest->buf, &buf[i + 1], n);\n+\treturn n;\n+}\n+\n+int reftable_is_block_type(uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\tcase BLOCK_TYPE_LOG:\n+\tcase BLOCK_TYPE_OBJ:\n+\tcase BLOCK_TYPE_INDEX:\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+static int decode_string(struct strbuf *dest, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t tsize = 0;\n+\tint n = get_var_int(&tsize, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\tif (in.len < tsize)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, in.buf, tsize);\n+\tstring_view_consume(&in, tsize);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static int encode_string(char *str, struct string_view s)\n+{\n+\tstruct string_view start = s;\n+\tint l = strlen(str);\n+\tint n = put_var_int(&s, l);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\tif (s.len < l)\n+\t\treturn -1;\n+\tmemcpy(s.buf, str, l);\n+\tstring_view_consume(&s, l);\n+\n+\treturn start.len - s.len;\n+}\n+\n+int reftable_encode_key(int *restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra)\n+{\n+\tstruct string_view start = dest;\n+\tint prefix_len = common_prefix_size(&prev_key, &key);\n+\tuint64_t suffix_len = key.len - prefix_len;\n+\tint n = put_var_int(&dest, (uint64_t)prefix_len);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\t*restart = (prefix_len == 0);\n+\n+\tn = put_var_int(&dest, suffix_len << 3 | (uint64_t)extra);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\tif (dest.len < suffix_len)\n+\t\treturn -1;\n+\tmemcpy(dest.buf, key.buf + prefix_len, suffix_len);\n+\tstring_view_consume(&dest, suffix_len);\n+\n+\treturn start.len - dest.len;\n+}\n+\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t prefix_len = 0;\n+\tuint64_t suffix_len = 0;\n+\tint n = get_var_int(&prefix_len, &in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tif (prefix_len > last_key.len)\n+\t\treturn -1;\n+\n+\tn = get_var_int(&suffix_len, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\t*extra = (uint8_t)(suffix_len & 0x7);\n+\tsuffix_len >>= 3;\n+\n+\tif (in.len < suffix_len)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(key);\n+\tstrbuf_add(key, last_key.buf, prefix_len);\n+\tstrbuf_add(key, in.buf, suffix_len);\n+\tstring_view_consume(&in, suffix_len);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static void reftable_ref_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_ref_record *rec =\n+\t\t(const struct reftable_ref_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, rec->refname);\n+}\n+\n+static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *ref = (struct reftable_ref_record *)rec;\n+\tstruct reftable_ref_record *src = (struct reftable_ref_record *)src_rec;\n+\tassert(hash_size > 0);\n+\n+\t/* This is simple and correct, but we could probably reuse the hash\n+\t fields. */\n+\treftable_ref_record_release(ref);\n+\tif (src->refname != NULL) {\n+\t\tref->refname = xstrdup(src->refname);\n+\t}\n+\n+\tif (src->target != NULL) {\n+\t\tref->target = xstrdup(src->target);\n+\t}\n+\n+\tif (src->target_value != NULL) {\n+\t\tref->target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->target_value, src->target_value, hash_size);\n+\t}\n+\n+\tif (src->value != NULL) {\n+\t\tref->value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value, src->value, hash_size);\n+\t}\n+\tref->update_index = src->update_index;\n+}\n+\n+static char hexdigit(int c)\n+{\n+\tif (c <= 9)\n+\t\treturn '0' + c;\n+\treturn 'a' + (c - 10);\n+}\n+\n+static void hex_format(char *dest, uint8_t *src, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (src != NULL) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < hash_size; i++) {\n+\t\t\tdest[2 * i] = hexdigit(src[i] >> 4);\n+\t\t\tdest[2 * i + 1] = hexdigit(src[i] & 0xf);\n+\t\t}\n+\t\tdest[2 * hash_size] = 0;\n+\t}\n+}\n+\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[SHA256_SIZE + 1] = { 0 };\n+\tprintf(\"ref{%s(%\" PRIu64 \") \", ref->refname, ref->update_index);\n+\tif (ref->value != NULL) {\n+\t\thex_format(hex, ref->value, hash_size(hash_id));\n+\t\tprintf(\"%s\", hex);\n+\t}\n+\tif (ref->target_value != NULL) {\n+\t\thex_format(hex, ref->target_value, hash_size(hash_id));\n+\t\tprintf(\" (T %s)\", hex);\n+\t}\n+\tif (ref->target != NULL) {\n+\t\tprintf(\"=> %s\", ref->target);\n+\t}\n+\tprintf(\"}\\n\");\n+}\n+\n+static void reftable_ref_record_release_void(void *rec)\n+{\n+\treftable_ref_record_release((struct reftable_ref_record *)rec);\n+}\n+\n+void reftable_ref_record_release(struct reftable_ref_record *ref)\n+{\n+\treftable_free(ref->refname);\n+\treftable_free(ref->target);\n+\treftable_free(ref->target_value);\n+\treftable_free(ref->value);\n+\tmemset(ref, 0, sizeof(struct reftable_ref_record));\n+}\n+\n+static uint8_t reftable_ref_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\tif (r->value != NULL) {\n+\t\tif (r->target_value != NULL) {\n+\t\t\treturn 2;\n+\t\t} else {\n+\t\t\treturn 1;\n+\t\t}\n+\t} else if (r->target != NULL)\n+\t\treturn 3;\n+\treturn 0;\n+}\n+\n+static int reftable_ref_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\tstruct string_view start = s;\n+\tint n = put_var_int(&s, r->update_index);\n+\tassert(hash_size > 0);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tif (r->value != NULL) {\n+\t\tif (s.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t}\n+\n+\tif (r->target_value != NULL) {\n+\t\tif (s.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->target_value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t}\n+\n+\tif (r->target != NULL) {\n+\t\tint n = encode_string(r->target, s);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t}\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_ref_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *r = (struct reftable_ref_record *)rec;\n+\tstruct string_view start = in;\n+\tint seen_value = 0;\n+\tint seen_target_value = 0;\n+\tint seen_target = 0;\n+\n+\tint n = get_var_int(&r->update_index, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tassert(hash_size > 0);\n+\n+\tstring_view_consume(&in, n);\n+\n+\tr->refname = reftable_realloc(r->refname, key.len + 1);\n+\tmemcpy(r->refname, key.buf, key.len);\n+\tr->refname[key.len] = 0;\n+\n+\tswitch (val_type) {\n+\tcase 1:\n+\tcase 2:\n+\t\tif (in.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (r->value == NULL) {\n+\t\t\tr->value = reftable_malloc(hash_size);\n+\t\t}\n+\t\tseen_value = 1;\n+\t\tmemcpy(r->value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tif (val_type == 1) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (r->target_value == NULL) {\n+\t\t\tr->target_value = reftable_malloc(hash_size);\n+\t\t}\n+\t\tseen_target_value = 1;\n+\t\tmemcpy(r->target_value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\tcase 3: {\n+\t\tstruct strbuf dest = STRBUF_INIT;\n+\t\tint n = decode_string(&dest, in);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\t\tseen_target = 1;\n+\t\tif (r->target != NULL) {\n+\t\t\treftable_free(r->target);\n+\t\t}\n+\t\tr->target = dest.buf;\n+\t} break;\n+\n+\tcase 0:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t\tbreak;\n+\t}\n+\n+\tif (!seen_target && r->target != NULL) {\n+\t\tFREE_AND_NULL(r->target);\n+\t}\n+\tif (!seen_target_value && r->target_value != NULL) {\n+\t\tFREE_AND_NULL(r->target_value);\n+\t}\n+\tif (!seen_value && r->value != NULL) {\n+\t\tFREE_AND_NULL(r->value);\n+\t}\n+\n+\treturn start.len - in.len;\n+}\n+\n+static int reftable_ref_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_ref_record_is_deletion(\n+\t\t(const struct reftable_ref_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_ref_record_vtable = {\n+\t.key = &reftable_ref_record_key,\n+\t.type = BLOCK_TYPE_REF,\n+\t.copy_from = &reftable_ref_record_copy_from,\n+\t.val_type = &reftable_ref_record_val_type,\n+\t.encode = &reftable_ref_record_encode,\n+\t.decode = &reftable_ref_record_decode,\n+\t.release = &reftable_ref_record_release_void,\n+\t.is_deletion = &reftable_ref_record_is_deletion_void,\n+};\n+\n+static void reftable_obj_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_obj_record *rec =\n+\t\t(const struct reftable_obj_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, rec->hash_prefix, rec->hash_prefix_len);\n+}\n+\n+static void reftable_obj_record_release(void *rec)\n+{\n+\tstruct reftable_obj_record *obj = (struct reftable_obj_record *)rec;\n+\tFREE_AND_NULL(obj->hash_prefix);\n+\tFREE_AND_NULL(obj->offsets);\n+\tmemset(obj, 0, sizeof(struct reftable_obj_record));\n+}\n+\n+static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_obj_record *obj = (struct reftable_obj_record *)rec;\n+\tconst struct reftable_obj_record *src =\n+\t\t(const struct reftable_obj_record *)src_rec;\n+\tint olen;\n+\n+\treftable_obj_record_release(obj);\n+\t*obj = *src;\n+\tobj->hash_prefix = reftable_malloc(obj->hash_prefix_len);\n+\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n+\n+\tolen = obj->offset_len * sizeof(uint64_t);\n+\tobj->offsets = reftable_malloc(olen);\n+\tmemcpy(obj->offsets, src->offsets, olen);\n+}\n+\n+static uint8_t reftable_obj_record_val_type(const void *rec)\n+{\n+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n+\tif (r->offset_len > 0 && r->offset_len < 8)\n+\t\treturn r->offset_len;\n+\treturn 0;\n+}\n+\n+static int reftable_obj_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n+\tstruct string_view start = s;\n+\tint i = 0;\n+\tint n = 0;\n+\tuint64_t last = 0;\n+\tif (r->offset_len == 0 || r->offset_len >= 8) {\n+\t\tn = put_var_int(&s, r->offset_len);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t}\n+\tif (r->offset_len == 0)\n+\t\treturn start.len - s.len;\n+\tn = put_var_int(&s, r->offsets[0]);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tlast = r->offsets[0];\n+\tfor (i = 1; i < r->offset_len; i++) {\n+\t\tint n = put_var_int(&s, r->offsets[i] - last);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tlast = r->offsets[i];\n+\t}\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_obj_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n+\tuint64_t count = val_type;\n+\tint n = 0;\n+\tuint64_t last;\n+\tint j;\n+\tr->hash_prefix = reftable_malloc(key.len);\n+\tmemcpy(r->hash_prefix, key.buf, key.len);\n+\tr->hash_prefix_len = key.len;\n+\n+\tif (val_type == 0) {\n+\t\tn = get_var_int(&count, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\n+\t\tstring_view_consume(&in, n);\n+\t}\n+\n+\tr->offsets = NULL;\n+\tr->offset_len = 0;\n+\tif (count == 0)\n+\t\treturn start.len - in.len;\n+\n+\tr->offsets = reftable_malloc(count * sizeof(uint64_t));\n+\tr->offset_len = count;\n+\n+\tn = get_var_int(&r->offsets[0], &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\tlast = r->offsets[0];\n+\tj = 1;\n+\twhile (j < count) {\n+\t\tuint64_t delta = 0;\n+\t\tint n = get_var_int(&delta, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\n+\t\tlast = r->offsets[j] = (delta + last);\n+\t\tj++;\n+\t}\n+\treturn start.len - in.len;\n+}\n+\n+static int not_a_deletion(const void *p)\n+{\n+\treturn 0;\n+}\n+\n+static struct reftable_record_vtable reftable_obj_record_vtable = {\n+\t.key = &reftable_obj_record_key,\n+\t.type = BLOCK_TYPE_OBJ,\n+\t.copy_from = &reftable_obj_record_copy_from,\n+\t.val_type = &reftable_obj_record_val_type,\n+\t.encode = &reftable_obj_record_encode,\n+\t.decode = &reftable_obj_record_decode,\n+\t.release = &reftable_obj_record_release,\n+\t.is_deletion = not_a_deletion,\n+};\n+\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[SHA256_SIZE + 1] = { 0 };\n+\n+\tprintf(\"log{%s(%\" PRIu64 \") %s <%s> %\" PRIu64 \" %04d\\n\", log->refname,\n+\t log->update_index, log->name, log->email, log->time,\n+\t log->tz_offset);\n+\thex_format(hex, log->old_hash, hash_size(hash_id));\n+\tprintf(\"%s => \", hex);\n+\thex_format(hex, log->new_hash, hash_size(hash_id));\n+\tprintf(\"%s\\n\\n%s\\n}\\n\", hex, log->message);\n+}\n+\n+static void reftable_log_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_log_record *rec =\n+\t\t(const struct reftable_log_record *)r;\n+\tint len = strlen(rec->refname);\n+\tuint8_t i64[8];\n+\tuint64_t ts = 0;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, (uint8_t *)rec->refname, len + 1);\n+\n+\tts = (~ts) - rec->update_index;\n+\tput_be64(&i64[0], ts);\n+\tstrbuf_add(dest, i64, sizeof(i64));\n+}\n+\n+static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_log_record *dst = (struct reftable_log_record *)rec;\n+\tconst struct reftable_log_record *src =\n+\t\t(const struct reftable_log_record *)src_rec;\n+\n+\treftable_log_record_release(dst);\n+\t*dst = *src;\n+\tif (dst->refname != NULL) {\n+\t\tdst->refname = xstrdup(dst->refname);\n+\t}\n+\tif (dst->email != NULL) {\n+\t\tdst->email = xstrdup(dst->email);\n+\t}\n+\tif (dst->name != NULL) {\n+\t\tdst->name = xstrdup(dst->name);\n+\t}\n+\tif (dst->message != NULL) {\n+\t\tdst->message = xstrdup(dst->message);\n+\t}\n+\n+\tif (dst->new_hash != NULL) {\n+\t\tdst->new_hash = reftable_malloc(hash_size);\n+\t\tmemcpy(dst->new_hash, src->new_hash, hash_size);\n+\t}\n+\tif (dst->old_hash != NULL) {\n+\t\tdst->old_hash = reftable_malloc(hash_size);\n+\t\tmemcpy(dst->old_hash, src->old_hash, hash_size);\n+\t}\n+}\n+\n+static void reftable_log_record_release_void(void *rec)\n+{\n+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n+\treftable_log_record_release(r);\n+}\n+\n+void reftable_log_record_release(struct reftable_log_record *r)\n+{\n+\treftable_free(r->refname);\n+\treftable_free(r->new_hash);\n+\treftable_free(r->old_hash);\n+\treftable_free(r->name);\n+\treftable_free(r->email);\n+\treftable_free(r->message);\n+\tmemset(r, 0, sizeof(struct reftable_log_record));\n+}\n+\n+static uint8_t reftable_log_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_log_record *log =\n+\t\t(const struct reftable_log_record *)rec;\n+\n+\treturn reftable_log_record_is_deletion(log) ? 0 : 1;\n+}\n+\n+static uint8_t zero[SHA256_SIZE] = { 0 };\n+\n+static int reftable_log_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n+\tstruct string_view start = s;\n+\tint n = 0;\n+\tuint8_t *oldh = r->old_hash;\n+\tuint8_t *newh = r->new_hash;\n+\tif (reftable_log_record_is_deletion(r))\n+\t\treturn 0;\n+\n+\tif (oldh == NULL) {\n+\t\toldh = zero;\n+\t}\n+\tif (newh == NULL) {\n+\t\tnewh = zero;\n+\t}\n+\n+\tif (s.len < 2 * hash_size)\n+\t\treturn -1;\n+\n+\tmemcpy(s.buf, oldh, hash_size);\n+\tmemcpy(s.buf + hash_size, newh, hash_size);\n+\tstring_view_consume(&s, 2 * hash_size);\n+\n+\tn = encode_string(r->name ? r->name : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = encode_string(r->email ? r->email : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = put_var_int(&s, r->time);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tif (s.len < 2)\n+\t\treturn -1;\n+\n+\tput_be16(s.buf, r->tz_offset);\n+\tstring_view_consume(&s, 2);\n+\n+\tn = encode_string(r->message ? r->message : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_log_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n+\tuint64_t max = 0;\n+\tuint64_t ts = 0;\n+\tstruct strbuf dest = STRBUF_INIT;\n+\tint n;\n+\n+\tif (key.len <= 9 || key.buf[key.len - 9] != 0)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->refname = reftable_realloc(r->refname, key.len - 8);\n+\tmemcpy(r->refname, key.buf, key.len - 8);\n+\tts = get_be64(key.buf + key.len - 8);\n+\n+\tr->update_index = (~max) - ts;\n+\n+\tif (val_type == 0) {\n+\t\tFREE_AND_NULL(r->old_hash);\n+\t\tFREE_AND_NULL(r->new_hash);\n+\t\tFREE_AND_NULL(r->message);\n+\t\tFREE_AND_NULL(r->email);\n+\t\tFREE_AND_NULL(r->name);\n+\t\treturn 0;\n+\t}\n+\n+\tif (in.len < 2 * hash_size)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->old_hash = reftable_realloc(r->old_hash, hash_size);\n+\tr->new_hash = reftable_realloc(r->new_hash, hash_size);\n+\n+\tmemcpy(r->old_hash, in.buf, hash_size);\n+\tmemcpy(r->new_hash, in.buf + hash_size, hash_size);\n+\n+\tstring_view_consume(&in, 2 * hash_size);\n+\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->name = reftable_realloc(r->name, dest.len + 1);\n+\tmemcpy(r->name, dest.buf, dest.len);\n+\tr->name[dest.len] = 0;\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->email = reftable_realloc(r->email, dest.len + 1);\n+\tmemcpy(r->email, dest.buf, dest.len);\n+\tr->email[dest.len] = 0;\n+\n+\tts = 0;\n+\tn = get_var_int(&ts, &in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\tr->time = ts;\n+\tif (in.len < 2)\n+\t\tgoto done;\n+\n+\tr->tz_offset = get_be16(in.buf);\n+\tstring_view_consume(&in, 2);\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->message = reftable_realloc(r->message, dest.len + 1);\n+\tmemcpy(r->message, dest.buf, dest.len);\n+\tr->message[dest.len] = 0;\n+\n+\tstrbuf_release(&dest);\n+\treturn start.len - in.len;\n+\n+done:\n+\tstrbuf_release(&dest);\n+\treturn REFTABLE_FORMAT_ERROR;\n+}\n+\n+static int null_streq(char *a, char *b)\n+{\n+\tchar *empty = \"\";\n+\tif (a == NULL)\n+\t\ta = empty;\n+\n+\tif (b == NULL)\n+\t\tb = empty;\n+\n+\treturn 0 == strcmp(a, b);\n+}\n+\n+static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)\n+{\n+\tif (a == NULL)\n+\t\ta = zero;\n+\n+\tif (b == NULL)\n+\t\tb = zero;\n+\n+\treturn !memcmp(a, b, sz);\n+}\n+\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size)\n+{\n+\treturn null_streq(a->name, b->name) && null_streq(a->email, b->email) &&\n+\t null_streq(a->message, b->message) &&\n+\t zero_hash_eq(a->old_hash, b->old_hash, hash_size) &&\n+\t zero_hash_eq(a->new_hash, b->new_hash, hash_size) &&\n+\t a->time == b->time && a->tz_offset == b->tz_offset &&\n+\t a->update_index == b->update_index;\n+}\n+\n+static int reftable_log_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_log_record_is_deletion(\n+\t\t(const struct reftable_log_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_log_record_vtable = {\n+\t.key = &reftable_log_record_key,\n+\t.type = BLOCK_TYPE_LOG,\n+\t.copy_from = &reftable_log_record_copy_from,\n+\t.val_type = &reftable_log_record_val_type,\n+\t.encode = &reftable_log_record_encode,\n+\t.decode = &reftable_log_record_decode,\n+\t.release = &reftable_log_record_release_void,\n+\t.is_deletion = &reftable_log_record_is_deletion_void,\n+};\n+\n+struct reftable_record reftable_new_record(uint8_t typ)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF: {\n+\t\tstruct reftable_ref_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_ref_record));\n+\t\treftable_record_from_ref(&rec, r);\n+\t\treturn rec;\n+\t}\n+\n+\tcase BLOCK_TYPE_OBJ: {\n+\t\tstruct reftable_obj_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_obj_record));\n+\t\treftable_record_from_obj(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_LOG: {\n+\t\tstruct reftable_log_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_log_record));\n+\t\treftable_record_from_log(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_INDEX: {\n+\t\tstruct reftable_index_record empty = { .last_key =\n+\t\t\t\t\t\t\t STRBUF_INIT };\n+\t\tstruct reftable_index_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_index_record));\n+\t\t*r = empty;\n+\t\treftable_record_from_index(&rec, r);\n+\t\treturn rec;\n+\t}\n+\t}\n+\tabort();\n+\treturn rec;\n+}\n+\n+/* clear out the record, yielding the reftable_record data that was\n+ * encapsulated. */\n+static void *reftable_record_yield(struct reftable_record *rec)\n+{\n+\tvoid *p = rec->data;\n+\trec->data = NULL;\n+\treturn p;\n+}\n+\n+void reftable_record_destroy(struct reftable_record *rec)\n+{\n+\treftable_record_release(rec);\n+\treftable_free(reftable_record_yield(rec));\n+}\n+\n+static void reftable_index_record_key(const void *r, struct strbuf *dest)\n+{\n+\tstruct reftable_index_record *rec = (struct reftable_index_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addbuf(dest, &rec->last_key);\n+}\n+\n+static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_index_record *dst = (struct reftable_index_record *)rec;\n+\tstruct reftable_index_record *src =\n+\t\t(struct reftable_index_record *)src_rec;\n+\n+\tstrbuf_reset(&dst->last_key);\n+\tstrbuf_addbuf(&dst->last_key, &src->last_key);\n+\tdst->offset = src->offset;\n+}\n+\n+static void reftable_index_record_release(void *rec)\n+{\n+\tstruct reftable_index_record *idx = (struct reftable_index_record *)rec;\n+\tstrbuf_release(&idx->last_key);\n+}\n+\n+static uint8_t reftable_index_record_val_type(const void *rec)\n+{\n+\treturn 0;\n+}\n+\n+static int reftable_index_record_encode(const void *rec, struct string_view out,\n+\t\t\t\t\tint hash_size)\n+{\n+\tconst struct reftable_index_record *r =\n+\t\t(const struct reftable_index_record *)rec;\n+\tstruct string_view start = out;\n+\n+\tint n = put_var_int(&out, r->offset);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&out, n);\n+\n+\treturn start.len - out.len;\n+}\n+\n+static int reftable_index_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t\tuint8_t val_type, struct string_view in,\n+\t\t\t\t\tint hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_index_record *r = (struct reftable_index_record *)rec;\n+\tint n = 0;\n+\n+\tstrbuf_reset(&r->last_key);\n+\tstrbuf_addbuf(&r->last_key, &key);\n+\n+\tn = get_var_int(&r->offset, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&in, n);\n+\treturn start.len - in.len;\n+}\n+\n+static struct reftable_record_vtable reftable_index_record_vtable = {\n+\t.key = &reftable_index_record_key,\n+\t.type = BLOCK_TYPE_INDEX,\n+\t.copy_from = &reftable_index_record_copy_from,\n+\t.val_type = &reftable_index_record_val_type,\n+\t.encode = &reftable_index_record_encode,\n+\t.decode = &reftable_index_record_decode,\n+\t.release = &reftable_index_record_release,\n+\t.is_deletion = ¬_a_deletion,\n+};\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)\n+{\n+\trec->ops->key(rec->data, dest);\n+}\n+\n+uint8_t reftable_record_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->type;\n+}\n+\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size)\n+{\n+\treturn rec->ops->encode(rec->data, dest, hash_size);\n+}\n+\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size)\n+{\n+\tassert(src->ops->type == rec->ops->type);\n+\n+\trec->ops->copy_from(rec->data, src->data, hash_size);\n+}\n+\n+uint8_t reftable_record_val_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->val_type(rec->data);\n+}\n+\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src, int hash_size)\n+{\n+\treturn rec->ops->decode(rec->data, key, extra, src, hash_size);\n+}\n+\n+void reftable_record_release(struct reftable_record *rec)\n+{\n+\trec->ops->release(rec->data);\n+}\n+\n+int reftable_record_is_deletion(struct reftable_record *rec)\n+{\n+\treturn rec->ops->is_deletion(rec->data);\n+}\n+\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *ref_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = ref_rec;\n+\trec->ops = &reftable_ref_record_vtable;\n+}\n+\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *obj_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = obj_rec;\n+\trec->ops = &reftable_obj_record_vtable;\n+}\n+\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *index_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = index_rec;\n+\trec->ops = &reftable_index_record_vtable;\n+}\n+\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *log_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = log_rec;\n+\trec->ops = &reftable_log_record_vtable;\n+}\n+\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_REF);\n+\treturn (struct reftable_ref_record *)rec->data;\n+}\n+\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_LOG);\n+\treturn (struct reftable_log_record *)rec->data;\n+}\n+\n+static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)\n+{\n+\tif (a != NULL && b != NULL)\n+\t\treturn !memcmp(a, b, hash_size);\n+\n+\treturn a == b;\n+}\n+\n+static int str_equal(char *a, char *b)\n+{\n+\tif (a != NULL && b != NULL)\n+\t\treturn 0 == strcmp(a, b);\n+\n+\treturn a == b;\n+}\n+\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\treturn 0 == strcmp(a->refname, b->refname) &&\n+\t a->update_index == b->update_index &&\n+\t hash_equal(a->value, b->value, hash_size) &&\n+\t hash_equal(a->target_value, b->target_value, hash_size) &&\n+\t str_equal(a->target, b->target);\n+}\n+\n+int reftable_ref_record_compare_name(const void *a, const void *b)\n+{\n+\treturn strcmp(((struct reftable_ref_record *)a)->refname,\n+\t\t ((struct reftable_ref_record *)b)->refname);\n+}\n+\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref)\n+{\n+\treturn ref->value == NULL && ref->target == NULL &&\n+\t ref->target_value == NULL;\n+}\n+\n+int reftable_log_record_compare_key(const void *a, const void *b)\n+{\n+\tstruct reftable_log_record *la = (struct reftable_log_record *)a;\n+\tstruct reftable_log_record *lb = (struct reftable_log_record *)b;\n+\n+\tint cmp = strcmp(la->refname, lb->refname);\n+\tif (cmp)\n+\t\treturn cmp;\n+\tif (la->update_index > lb->update_index)\n+\t\treturn -1;\n+\treturn (la->update_index < lb->update_index) ? 1 : 0;\n+}\n+\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log)\n+{\n+\treturn (log->new_hash == NULL && log->old_hash == NULL &&\n+\t\tlog->name == NULL && log->email == NULL &&\n+\t\tlog->message == NULL && log->time == 0 && log->tz_offset == 0 &&\n+\t\tlog->message == NULL);\n+}\n+\n+void string_view_consume(struct string_view *s, int n)\n+{\n+\ts->buf += n;\n+\ts->len -= n;\n+}\ndiff --git a/reftable/record.h b/reftable/record.h\nnew file mode 100644\nindex 0000000000..c78b62e564\n--- /dev/null\n+++ b/reftable/record.h\n@@ -0,0 +1,139 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef RECORD_H\n+#define RECORD_H\n+\n+#include \"system.h\"\n+\n+#include \n+\n+#include \"reftable-record.h\"\n+\n+/*\n+ A substring of existing string data. This structure takes no responsibility\n+ for the lifetime of the data it points to.\n+*/\n+struct string_view {\n+\tuint8_t *buf;\n+\tsize_t len;\n+};\n+\n+/* Advance `s.buf` by `n`, and decrease length. */\n+void string_view_consume(struct string_view *s, int n);\n+\n+/* utilities for de/encoding varints */\n+\n+int get_var_int(uint64_t *dest, struct string_view *in);\n+int put_var_int(struct string_view *dest, uint64_t val);\n+\n+/* Methods for records. */\n+struct reftable_record_vtable {\n+\t/* encode the key of to a uint8_t strbuf. */\n+\tvoid (*key)(const void *rec, struct strbuf *dest);\n+\n+\t/* The record type of ('r' for ref). */\n+\tuint8_t type;\n+\n+\tvoid (*copy_from)(void *dest, const void *src, int hash_size);\n+\n+\t/* a value of [0..7], indicating record subvariants (eg. ref vs. symref\n+\t * vs ref deletion) */\n+\tuint8_t (*val_type)(const void *rec);\n+\n+\t/* encodes rec into dest, returning how much space was used. */\n+\tint (*encode)(const void *rec, struct string_view dest, int hash_size);\n+\n+\t/* decode data from `src` into the record. */\n+\tint (*decode)(void *rec, struct strbuf key, uint8_t extra,\n+\t\t struct string_view src, int hash_size);\n+\n+\t/* deallocate and null the record. */\n+\tvoid (*release)(void *rec);\n+\n+\t/* is this a tombstone? */\n+\tint (*is_deletion)(const void *rec);\n+};\n+\n+/* record is a generic wrapper for different types of records. */\n+struct reftable_record {\n+\tvoid *data;\n+\tstruct reftable_record_vtable *ops;\n+};\n+\n+/* returns true for recognized block types. Block start with the block type. */\n+int reftable_is_block_type(uint8_t typ);\n+\n+/* creates a malloced record of the given type. Dispose with record_destroy */\n+struct reftable_record reftable_new_record(uint8_t typ);\n+\n+/* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns\n+ number of bytes written. */\n+int reftable_encode_key(int *is_restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra);\n+\n+/* Decode into `key` and `extra` from `in` */\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in);\n+\n+/* reftable_index_record are used internally to speed up lookups. */\n+struct reftable_index_record {\n+\tuint64_t offset; /* Offset of block */\n+\tstruct strbuf last_key; /* Last key of the block. */\n+};\n+\n+/* reftable_obj_record stores an object ID => ref mapping. */\n+struct reftable_obj_record {\n+\tuint8_t *hash_prefix; /* leading bytes of the object ID */\n+\tint hash_prefix_len; /* number of leading bytes. Constant\n+\t\t\t * across a single table. */\n+\tuint64_t *offsets; /* a vector of file offsets. */\n+\tint offset_len;\n+};\n+\n+/* see struct record_vtable */\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);\n+uint8_t reftable_record_type(struct reftable_record *rec);\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size);\n+uint8_t reftable_record_val_type(struct reftable_record *rec);\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size);\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src,\n+\t\t\t int hash_size);\n+int reftable_record_is_deletion(struct reftable_record *rec);\n+\n+/* zeroes out the embedded record */\n+void reftable_record_release(struct reftable_record *rec);\n+\n+/* clear and deallocate embedded record, and zero `rec`. */\n+void reftable_record_destroy(struct reftable_record *rec);\n+\n+/* initialize generic records from concrete records. The generic record should\n+ * be zeroed out. */\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *objrec);\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *idxrec);\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *refrec);\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *logrec);\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *ref);\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *ref);\n+\n+/* for qsort. */\n+int reftable_ref_record_compare_name(const void *a, const void *b);\n+\n+/* for qsort. */\n+int reftable_log_record_compare_key(const void *a, const void *b);\n+\n+#endif\ndiff --git a/reftable/record_test.c b/reftable/record_test.c\nnew file mode 100644\nindex 0000000000..6dad76b0ac\n--- /dev/null\n+++ b/reftable/record_test.c\n@@ -0,0 +1,404 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_copy(struct reftable_record *rec)\n+{\n+\tstruct reftable_record copy =\n+\t\treftable_new_record(reftable_record_type(rec));\n+\treftable_record_copy_from(©, rec, SHA1_SIZE);\n+\t/* do it twice to catch memory leaks */\n+\treftable_record_copy_from(©, rec, SHA1_SIZE);\n+\tswitch (reftable_record_type(©)) {\n+\tcase BLOCK_TYPE_REF:\n+\t\tEXPECT(reftable_ref_record_equal(reftable_record_as_ref(©),\n+\t\t\t\t\t\t reftable_record_as_ref(rec),\n+\t\t\t\t\t\t SHA1_SIZE));\n+\t\tbreak;\n+\tcase BLOCK_TYPE_LOG:\n+\t\tEXPECT(reftable_log_record_equal(reftable_record_as_log(©),\n+\t\t\t\t\t\t reftable_record_as_log(rec),\n+\t\t\t\t\t\t SHA1_SIZE));\n+\t\tbreak;\n+\t}\n+\treftable_record_destroy(©);\n+}\n+\n+static void test_varint_roundtrip(void)\n+{\n+\tuint64_t inputs[] = { 0,\n+\t\t\t 1,\n+\t\t\t 27,\n+\t\t\t 127,\n+\t\t\t 128,\n+\t\t\t 257,\n+\t\t\t 4096,\n+\t\t\t ((uint64_t)1 << 63),\n+\t\t\t ((uint64_t)1 << 63) + ((uint64_t)1 << 63) - 1 };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(inputs); i++) {\n+\t\tuint8_t dest[10];\n+\n+\t\tstruct string_view out = {\n+\t\t\t.buf = dest,\n+\t\t\t.len = sizeof(dest),\n+\t\t};\n+\t\tuint64_t in = inputs[i];\n+\t\tint n = put_var_int(&out, in);\n+\t\tuint64_t got = 0;\n+\n+\t\tEXPECT(n > 0);\n+\t\tout.len = n;\n+\t\tn = get_var_int(&got, &out);\n+\t\tEXPECT(n > 0);\n+\n+\t\tEXPECT(got == in);\n+\t}\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct {\n+\t\tconst char *a, *b;\n+\t\tint want;\n+\t} cases[] = {\n+\t\t{ \"abc\", \"ab\", 2 },\n+\t\t{ \"\", \"abc\", 0 },\n+\t\t{ \"abc\", \"abd\", 2 },\n+\t\t{ \"abc\", \"pqr\", 0 },\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct strbuf a = STRBUF_INIT;\n+\t\tstruct strbuf b = STRBUF_INIT;\n+\t\tstrbuf_addstr(&a, cases[i].a);\n+\t\tstrbuf_addstr(&b, cases[i].b);\n+\t\tEXPECT(common_prefix_size(&a, &b) == cases[i].want);\n+\n+\t\tstrbuf_release(&a);\n+\t\tstrbuf_release(&b);\n+\t}\n+}\n+\n+static void set_hash(uint8_t *h, int j)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < hash_size(SHA1_ID); i++) {\n+\t\th[i] = (j >> i) & 0xff;\n+\t}\n+}\n+\n+static void test_reftable_ref_record_roundtrip(void)\n+{\n+\tint i = 0;\n+\n+\tfor (i = 0; i <= 3; i++) {\n+\t\tstruct reftable_ref_record in = { NULL };\n+\t\tstruct reftable_ref_record out = {\n+\t\t\t.refname = xstrdup(\"old name\"),\n+\t\t\t.value = reftable_calloc(SHA1_SIZE),\n+\t\t\t.target_value = reftable_calloc(SHA1_SIZE),\n+\t\t\t.target = xstrdup(\"old value\"),\n+\t\t};\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\n+\t\tint n, m;\n+\n+\t\tswitch (i) {\n+\t\tcase 0:\n+\t\t\tbreak;\n+\t\tcase 1:\n+\t\t\tin.value = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_hash(in.value, 1);\n+\t\t\tbreak;\n+\t\tcase 2:\n+\t\t\tin.value = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_hash(in.value, 1);\n+\t\t\tin.target_value = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_hash(in.target_value, 2);\n+\t\t\tbreak;\n+\t\tcase 3:\n+\t\t\tin.target = xstrdup(\"target\");\n+\t\t\tbreak;\n+\t\t}\n+\t\tin.refname = xstrdup(\"refs/heads/master\");\n+\n+\t\treftable_record_from_ref(&rec, &in);\n+\t\ttest_copy(&rec);\n+\n+\t\tEXPECT(reftable_record_val_type(&rec) == i);\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\t\tEXPECT(n > 0);\n+\n+\t\t/* decode into a non-zero reftable_record to test for leaks. */\n+\n+\t\treftable_record_from_ref(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, i, dest, SHA1_SIZE);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT((out.value != NULL) == (in.value != NULL));\n+\t\tEXPECT((out.target_value != NULL) == (in.target_value != NULL));\n+\t\tEXPECT((out.target != NULL) == (in.target != NULL));\n+\t\treftable_record_release(&rec_out);\n+\n+\t\tstrbuf_release(&key);\n+\t\treftable_ref_record_release(&in);\n+\t}\n+}\n+\n+static void test_reftable_log_record_equal(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 42,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t}\n+\t};\n+\n+\tEXPECT(!reftable_log_record_equal(&in[0], &in[1], SHA1_SIZE));\n+\tin[1].update_index = in[0].update_index;\n+\tEXPECT(reftable_log_record_equal(&in[0], &in[1], SHA1_SIZE));\n+\treftable_log_record_release(&in[0]);\n+\treftable_log_record_release(&in[1]);\n+}\n+\n+static void test_reftable_log_record_roundtrip(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.old_hash = reftable_malloc(SHA1_SIZE),\n+\t\t\t.new_hash = reftable_malloc(SHA1_SIZE),\n+\t\t\t.name = xstrdup(\"han-wen\"),\n+\t\t\t.email = xstrdup(\"hanwen@google.com\"),\n+\t\t\t.message = xstrdup(\"test\"),\n+\t\t\t.update_index = 42,\n+\t\t\t.time = 1577123507,\n+\t\t\t.tz_offset = 100,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t}\n+\t};\n+\tset_test_hash(in[0].new_hash, 1);\n+\tset_test_hash(in[0].old_hash, 2);\n+\tfor (int i = 0; i < ARRAY_SIZE(in); i++) {\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\t/* populate out, to check for leaks. */\n+\t\tstruct reftable_log_record out = {\n+\t\t\t.refname = xstrdup(\"old name\"),\n+\t\t\t.new_hash = reftable_calloc(SHA1_SIZE),\n+\t\t\t.old_hash = reftable_calloc(SHA1_SIZE),\n+\t\t\t.name = xstrdup(\"old name\"),\n+\t\t\t.email = xstrdup(\"old@email\"),\n+\t\t\t.message = xstrdup(\"old message\"),\n+\t\t};\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m, valtype;\n+\n+\t\treftable_record_from_log(&rec, &in[i]);\n+\n+\t\ttest_copy(&rec);\n+\n+\t\treftable_record_key(&rec, &key);\n+\n+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\t\tEXPECT(n >= 0);\n+\t\treftable_record_from_log(&rec_out, &out);\n+\t\tvaltype = reftable_record_val_type(&rec);\n+\t\tm = reftable_record_decode(&rec_out, key, valtype, dest,\n+\t\t\t\t\t SHA1_SIZE);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(reftable_log_record_equal(&in[i], &out, SHA1_SIZE));\n+\t\treftable_log_record_release(&in[i]);\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_release(&rec_out);\n+\t}\n+}\n+\n+static void test_u24_roundtrip(void)\n+{\n+\tuint32_t in = 0x112233;\n+\tuint8_t dest[3];\n+\tuint32_t out;\n+\tput_be24(dest, in);\n+\tout = get_be24(dest);\n+\tEXPECT(in == out);\n+}\n+\n+static void test_key_roundtrip(void)\n+{\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct strbuf roundtrip = STRBUF_INIT;\n+\tint restart;\n+\tuint8_t extra;\n+\tint n, m;\n+\tuint8_t rt_extra;\n+\n+\tstrbuf_addstr(&last_key, \"refs/heads/master\");\n+\tstrbuf_addstr(&key, \"refs/tags/bla\");\n+\textra = 6;\n+\tn = reftable_encode_key(&restart, dest, last_key, key, extra);\n+\tEXPECT(!restart);\n+\tEXPECT(n > 0);\n+\n+\tm = reftable_decode_key(&roundtrip, &rt_extra, last_key, dest);\n+\tEXPECT(n == m);\n+\tEXPECT(0 == strbuf_cmp(&key, &roundtrip));\n+\tEXPECT(rt_extra == extra);\n+\n+\tstrbuf_release(&last_key);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&roundtrip);\n+}\n+\n+static void test_reftable_obj_record_roundtrip(void)\n+{\n+\tuint8_t testHash1[SHA1_SIZE] = { 1, 2, 3, 4, 0 };\n+\tuint64_t till9[] = { 1, 2, 3, 4, 500, 600, 700, 800, 9000 };\n+\tstruct reftable_obj_record recs[3] = { {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 3,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 9,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t } };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(recs); i++) {\n+\t\tstruct reftable_obj_record in = recs[i];\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_obj_record out = { NULL };\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m;\n+\t\tuint8_t extra;\n+\n+\t\treftable_record_from_obj(&rec, &in);\n+\t\ttest_copy(&rec);\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\t\tEXPECT(n > 0);\n+\t\textra = reftable_record_val_type(&rec);\n+\t\treftable_record_from_obj(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, extra, dest,\n+\t\t\t\t\t SHA1_SIZE);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(in.hash_prefix_len == out.hash_prefix_len);\n+\t\tEXPECT(in.offset_len == out.offset_len);\n+\n+\t\tEXPECT(!memcmp(in.hash_prefix, out.hash_prefix,\n+\t\t\t in.hash_prefix_len));\n+\t\tEXPECT(0 == memcmp(in.offsets, out.offsets,\n+\t\t\t\t sizeof(uint64_t) * in.offset_len));\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_release(&rec_out);\n+\t}\n+}\n+\n+static void test_reftable_index_record_roundtrip(void)\n+{\n+\tstruct reftable_index_record in = {\n+\t\t.offset = 42,\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_index_record out = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record out_rec = { NULL };\n+\tint n, m;\n+\tuint8_t extra;\n+\n+\tstrbuf_addstr(&in.last_key, \"refs/heads/master\");\n+\treftable_record_from_index(&rec, &in);\n+\treftable_record_key(&rec, &key);\n+\ttest_copy(&rec);\n+\n+\tEXPECT(0 == strbuf_cmp(&key, &in.last_key));\n+\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\tEXPECT(n > 0);\n+\n+\textra = reftable_record_val_type(&rec);\n+\treftable_record_from_index(&out_rec, &out);\n+\tm = reftable_record_decode(&out_rec, key, extra, dest, SHA1_SIZE);\n+\tEXPECT(m == n);\n+\n+\tEXPECT(in.offset == out.offset);\n+\n+\treftable_record_release(&out_rec);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&in.last_key);\n+}\n+\n+int record_test_main(int argc, const char *argv[])\n+{\n+\ttest_reftable_log_record_equal();\n+\ttest_reftable_log_record_roundtrip();\n+\ttest_reftable_ref_record_roundtrip();\n+\ttest_varint_roundtrip();\n+\ttest_key_roundtrip();\n+\ttest_common_prefix();\n+\ttest_reftable_obj_record_roundtrip();\n+\ttest_reftable_index_record_roundtrip();\n+\ttest_u24_roundtrip();\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-record.h b/reftable/reftable-record.h\nnew file mode 100644\nindex 0000000000..7057ceba35\n--- /dev/null\n+++ b/reftable/reftable-record.h\n@@ -0,0 +1,73 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_RECORD_H\n+#define REFTABLE_RECORD_H\n+\n+#include \n+\n+/*\n+ Basic data types\n+\n+ Reftables store the state of each ref in struct reftable_ref_record, and they\n+ store a sequence of reflog updates in struct reftable_log_record.\n+*/\n+\n+/* reftable_ref_record holds a ref database entry target_value */\n+struct reftable_ref_record {\n+\tchar *refname; /* Name of the ref, malloced. */\n+\tuint64_t update_index; /* Logical timestamp at which this value is\n+\t\t\t\t written */\n+\tuint8_t *value; /* SHA1, or NULL. malloced. */\n+\tuint8_t *target_value; /* peeled annotated tag, or NULL. malloced. */\n+\tchar *target; /* symref, or NULL. malloced. */\n+};\n+\n+/* returns whether 'ref' represents a deletion */\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);\n+\n+/* prints a reftable_ref_record onto stdout */\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id);\n+\n+/* frees and nulls all pointer values. */\n+void reftable_ref_record_release(struct reftable_ref_record *ref);\n+\n+/* returns whether two reftable_ref_records are the same */\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size);\n+\n+/* reftable_log_record holds a reflog entry */\n+struct reftable_log_record {\n+\tchar *refname;\n+\tuint64_t update_index; /* logical timestamp of a transactional update.\n+\t\t\t\t*/\n+\tuint8_t *new_hash;\n+\tuint8_t *old_hash;\n+\tchar *name;\n+\tchar *email;\n+\tuint64_t time;\n+\tint16_t tz_offset;\n+\tchar *message;\n+};\n+\n+/* returns whether 'ref' represents the deletion of a log record. */\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log);\n+\n+/* frees and nulls all pointer values. */\n+void reftable_log_record_release(struct reftable_log_record *log);\n+\n+/* returns whether two records are equal. Useful for testing. */\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size);\n+\n+/* dumps a reftable_log_record on stdout, for debugging/testing. */\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 3b58e423e7..09d4b83ef9 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,6 +4,6 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n-\n+\trecord_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"410891","messageId":"5f0b7c55925ffc6d5ae2d1bdba2186e04079a865.1606419752.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v3.git.git.1606419752.gitgitgadget@gmail.com","subject":"[PATCH v3 10/16] reftable: write reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-11-26T19:42:25Z","receivedAt":"2020-11-26T19:42:56Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable-writer.h | 149 ++++++++\n reftable/writer.c | 676 +++++++++++++++++++++++++++++++++++++\n reftable/writer.h | 50 +++\n 4 files changed, 876 insertions(+)\n create mode 100644 reftable/reftable-writer.h\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n\ndiff --git a/Makefile b/Makefile\nindex 89564d4c06..3e33a409f0 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2396,6 +2396,7 @@ REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/tree.o\n+REFTABLE_OBJS += reftable/writer.o\n REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nnew file mode 100644\nindex 0000000000..b736614684\n--- /dev/null\n+++ b/reftable/reftable-writer.h\n@@ -0,0 +1,149 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_WRITER_H\n+#define REFTABLE_WRITER_H\n+\n+#include \n+\n+#include \"reftable-record.h\"\n+\n+/*\n+ Writing single reftables\n+*/\n+\n+/* reftable_write_options sets options for writing a single reftable. */\n+struct reftable_write_options {\n+\t/* boolean: do not pad out blocks to block size. */\n+\tunsigned unpadded : 1;\n+\n+\t/* the blocksize. Should be less than 2^24. */\n+\tuint32_t block_size;\n+\n+\t/* boolean: do not generate a SHA1 => ref index. */\n+\tunsigned skip_index_objects : 1;\n+\n+\t/* how often to write complete keys in each block. */\n+\tint restart_interval;\n+\n+\t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n+\t * Defaults to SHA1 if unset\n+\t */\n+\tuint32_t hash_id;\n+\n+\t/* boolean: do not check ref names for validity or dir/file conflicts.\n+\t */\n+\tunsigned skip_name_check : 1;\n+\n+\t/* boolean: copy log messages exactly. If unset, check that the message\n+\t * is a single line, and add '\\n' if missing.\n+\t */\n+\tunsigned exact_log_message : 1;\n+};\n+\n+/* reftable_block_stats holds statistics for a single block type */\n+struct reftable_block_stats {\n+\t/* total number of entries written */\n+\tint entries;\n+\t/* total number of key restarts */\n+\tint restarts;\n+\t/* total number of blocks */\n+\tint blocks;\n+\t/* total number of index blocks */\n+\tint index_blocks;\n+\t/* depth of the index */\n+\tint max_index_level;\n+\n+\t/* offset of the first block for this type */\n+\tuint64_t offset;\n+\t/* offset of the top level index block for this type, or 0 if not\n+\t * present */\n+\tuint64_t index_offset;\n+};\n+\n+/* stats holds overall statistics for a single reftable */\n+struct reftable_stats {\n+\t/* total number of blocks written. */\n+\tint blocks;\n+\t/* stats for ref data */\n+\tstruct reftable_block_stats ref_stats;\n+\t/* stats for the SHA1 to ref map. */\n+\tstruct reftable_block_stats obj_stats;\n+\t/* stats for index blocks */\n+\tstruct reftable_block_stats idx_stats;\n+\t/* stats for log blocks */\n+\tstruct reftable_block_stats log_stats;\n+\n+\t/* disambiguation length of shortened object IDs. */\n+\tint object_id_len;\n+};\n+\n+/* reftable_new_writer creates a new writer */\n+struct reftable_writer *\n+reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts);\n+\n+/* Set the range of update indices for the records we will add. When writing a\n+ table into a stack, the min should be at least\n+ reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned.\n+\n+ For transactional updates to a stack, typically min==max, and the\n+ update_index can be obtained by inspeciting the stack. When converting an\n+ existing ref database into a single reftable, this would be a range of\n+ update-index timestamps.\n+ */\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max);\n+\n+/*\n+ Add a reftable_ref_record. The record should have names that come after\n+ already added records.\n+\n+ The update_index must be within the limits set by\n+ reftable_writer_set_limits(), or REFTABLE_API_ERROR is returned. It is an\n+ REFTABLE_API_ERROR error to write a ref record after a log record.\n+*/\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/*\n+ Convenience function to add multiple reftable_ref_records; the function sorts\n+ the records before adding them, reordering the records array passed in.\n+*/\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n);\n+\n+/*\n+ adds reftable_log_records. Log records are keyed by (refname, decreasing\n+ update_index). The key for the record added must come after the already added\n+ log records.\n+*/\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log);\n+\n+/*\n+ Convenience function to add multiple reftable_log_records; the function sorts\n+ the records before adding them, reordering records array passed in.\n+*/\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n);\n+\n+/* reftable_writer_close finalizes the reftable. The writer is retained so\n+ * statistics can be inspected. */\n+int reftable_writer_close(struct reftable_writer *w);\n+\n+/* writer_stats returns the statistics on the reftable being written.\n+\n+ This struct becomes invalid when the writer is freed.\n+ */\n+const struct reftable_stats *writer_stats(struct reftable_writer *w);\n+\n+/* reftable_writer_free deallocates memory for the writer */\n+void reftable_writer_free(struct reftable_writer *w);\n+\n+#endif\ndiff --git a/reftable/writer.c b/reftable/writer.c\nnew file mode 100644\nindex 0000000000..8f30e7a114\n--- /dev/null\n+++ b/reftable/writer.c\n@@ -0,0 +1,676 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"writer.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"tree.h\"\n+#include \"reftable-error.h\"\n+\n+/* finishes a block, and writes it to storage */\n+static int writer_flush_block(struct reftable_writer *w);\n+\n+/* deallocates memory related to the index */\n+static void writer_clear_index(struct reftable_writer *w);\n+\n+/* finishes writing a 'r' (refs) or 'g' (reflogs) section */\n+static int writer_finish_public_section(struct reftable_writer *w);\n+\n+static struct reftable_block_stats *\n+writer_reftable_block_stats(struct reftable_writer *w, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase 'r':\n+\t\treturn &w->stats.ref_stats;\n+\tcase 'o':\n+\t\treturn &w->stats.obj_stats;\n+\tcase 'i':\n+\t\treturn &w->stats.idx_stats;\n+\tcase 'g':\n+\t\treturn &w->stats.log_stats;\n+\t}\n+\tabort();\n+\treturn NULL;\n+}\n+\n+/* write data, queuing the padding for the next write. Returns negative for\n+ * error. */\n+static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n+\t\t\tint padding)\n+{\n+\tint n = 0;\n+\tif (w->pending_padding > 0) {\n+\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding);\n+\t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n+\t\tif (n < 0)\n+\t\t\treturn n;\n+\n+\t\tw->pending_padding = 0;\n+\t\treftable_free(zeroed);\n+\t}\n+\n+\tw->pending_padding = padding;\n+\tn = w->write(w->write_arg, data, len);\n+\tif (n < 0)\n+\t\treturn n;\n+\tn += padding;\n+\treturn 0;\n+}\n+\n+static void options_set_defaults(struct reftable_write_options *opts)\n+{\n+\tif (opts->restart_interval == 0) {\n+\t\topts->restart_interval = 16;\n+\t}\n+\n+\tif (opts->hash_id == 0) {\n+\t\topts->hash_id = SHA1_ID;\n+\t}\n+\tif (opts->block_size == 0) {\n+\t\topts->block_size = DEFAULT_BLOCK_SIZE;\n+\t}\n+}\n+\n+static int writer_version(struct reftable_writer *w)\n+{\n+\treturn (w->opts.hash_id == 0 || w->opts.hash_id == SHA1_ID) ? 1 : 2;\n+}\n+\n+static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n+{\n+\tmemcpy((char *)dest, \"REFT\", 4);\n+\n+\tdest[4] = writer_version(w);\n+\n+\tput_be24(dest + 5, w->opts.block_size);\n+\tput_be64(dest + 8, w->min_update_index);\n+\tput_be64(dest + 16, w->max_update_index);\n+\tif (writer_version(w) == 2) {\n+\t\tput_be32(dest + 24, w->opts.hash_id);\n+\t}\n+\treturn header_size(writer_version(w));\n+}\n+\n+static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n+{\n+\tint block_start = 0;\n+\tif (w->next == 0) {\n+\t\tblock_start = header_size(writer_version(w));\n+\t}\n+\n+\tstrbuf_release(&w->last_key);\n+\tblock_writer_init(&w->block_writer_data, typ, w->block,\n+\t\t\t w->opts.block_size, block_start,\n+\t\t\t hash_size(w->opts.hash_id));\n+\tw->block_writer = &w->block_writer_data;\n+\tw->block_writer->restart_interval = w->opts.restart_interval;\n+}\n+\n+static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n+\n+struct reftable_writer *\n+reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts)\n+{\n+\tstruct reftable_writer *wp =\n+\t\treftable_calloc(sizeof(struct reftable_writer));\n+\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n+\toptions_set_defaults(opts);\n+\tif (opts->block_size >= (1 << 24)) {\n+\t\t/* TODO - error return? */\n+\t\tabort();\n+\t}\n+\twp->last_key = reftable_empty_strbuf;\n+\twp->block = reftable_calloc(opts->block_size);\n+\twp->write = writer_func;\n+\twp->write_arg = writer_arg;\n+\twp->opts = *opts;\n+\twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n+\n+\treturn wp;\n+}\n+\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max)\n+{\n+\tw->min_update_index = min;\n+\tw->max_update_index = max;\n+}\n+\n+void reftable_writer_free(struct reftable_writer *w)\n+{\n+\treftable_free(w->block);\n+\treftable_free(w);\n+}\n+\n+struct obj_index_tree_node {\n+\tstruct strbuf hash;\n+\tuint64_t *offsets;\n+\tsize_t offset_len;\n+\tsize_t offset_cap;\n+};\n+\n+#define OBJ_INDEX_TREE_NODE_INIT \\\n+\t{ \\\n+\t\t.hash = STRBUF_INIT \\\n+\t}\n+\n+static int obj_index_tree_node_compare(const void *a, const void *b)\n+{\n+\treturn strbuf_cmp(&((const struct obj_index_tree_node *)a)->hash,\n+\t\t\t &((const struct obj_index_tree_node *)b)->hash);\n+}\n+\n+static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n+{\n+\tuint64_t off = w->next;\n+\n+\tstruct obj_index_tree_node want = { .hash = *hash };\n+\n+\tstruct tree_node *node = tree_search(&want, &w->obj_index_tree,\n+\t\t\t\t\t &obj_index_tree_node_compare, 0);\n+\tstruct obj_index_tree_node *key = NULL;\n+\tif (node == NULL) {\n+\t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n+\t\tkey = reftable_malloc(sizeof(struct obj_index_tree_node));\n+\t\t*key = empty;\n+\n+\t\tstrbuf_reset(&key->hash);\n+\t\tstrbuf_addbuf(&key->hash, hash);\n+\t\ttree_search((void *)key, &w->obj_index_tree,\n+\t\t\t &obj_index_tree_node_compare, 1);\n+\t} else {\n+\t\tkey = node->key;\n+\t}\n+\n+\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off) {\n+\t\treturn;\n+\t}\n+\n+\tif (key->offset_len == key->offset_cap) {\n+\t\tkey->offset_cap = 2 * key->offset_cap + 1;\n+\t\tkey->offsets = reftable_realloc(\n+\t\t\tkey->offsets, sizeof(uint64_t) * key->offset_cap);\n+\t}\n+\n+\tkey->offsets[key->offset_len++] = off;\n+}\n+\n+static int writer_add_record(struct reftable_writer *w,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tint result = -1;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = 0;\n+\treftable_record_key(rec, &key);\n+\tif (strbuf_cmp(&w->last_key, &key) >= 0)\n+\t\tgoto done;\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, &key);\n+\tif (w->block_writer == NULL) {\n+\t\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\t}\n+\n+\tassert(block_writer_type(w->block_writer) == reftable_record_type(rec));\n+\n+\tif (block_writer_add(w->block_writer, rec) == 0) {\n+\t\tresult = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0) {\n+\t\tresult = err;\n+\t\tgoto done;\n+\t}\n+\n+\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\terr = block_writer_add(w->block_writer, rec);\n+\tif (err < 0) {\n+\t\tresult = err;\n+\t\tgoto done;\n+\t}\n+\n+\tresult = 0;\n+done:\n+\tstrbuf_release(&key);\n+\treturn result;\n+}\n+\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_ref_record copy = *ref;\n+\tint err = 0;\n+\n+\tif (ref->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\tif (ref->update_index < w->min_update_index ||\n+\t ref->update_index > w->max_update_index)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\treftable_record_from_ref(&rec, ©);\n+\tcopy.update_index -= w->min_update_index;\n+\terr = writer_add_record(w, &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (!w->opts.skip_index_objects && ref->value != NULL) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, (char *)ref->value, hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\n+\tif (!w->opts.skip_index_objects && ref->target_value != NULL) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, ref->target_value, hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(refs, n, reftable_ref_record_compare_name);\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_ref(w, &refs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tchar *input_log_message = log->message;\n+\tstruct strbuf cleaned_message = STRBUF_INIT;\n+\tint err;\n+\tif (log->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\tif (w->block_writer != NULL &&\n+\t block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {\n+\t\tint err = writer_finish_public_section(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tif (!w->opts.exact_log_message && log->message != NULL) {\n+\t\tstrbuf_addstr(&cleaned_message, log->message);\n+\t\twhile (cleaned_message.len &&\n+\t\t cleaned_message.buf[cleaned_message.len - 1] == '\\n')\n+\t\t\tstrbuf_setlen(&cleaned_message,\n+\t\t\t\t cleaned_message.len - 1);\n+\t\tif (strchr(cleaned_message.buf, '\\n')) {\n+\t\t\t// multiple lines not allowed.\n+\t\t\terr = REFTABLE_API_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tstrbuf_addstr(&cleaned_message, \"\\n\");\n+\t\tlog->message = cleaned_message.buf;\n+\t}\n+\n+\tw->next -= w->pending_padding;\n+\tw->pending_padding = 0;\n+\n+\treftable_record_from_log(&rec, log);\n+\terr = writer_add_record(w, &rec);\n+\n+done:\n+\tlog->message = input_log_message;\n+\tstrbuf_release(&cleaned_message);\n+\treturn err;\n+}\n+\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(logs, n, reftable_log_record_compare_key);\n+\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_log(w, &logs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+static int writer_finish_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tuint64_t index_start = 0;\n+\tint max_level = 0;\n+\tint threshold = w->opts.unpadded ? 1 : 3;\n+\tint before_blocks = w->stats.idx_stats.blocks;\n+\tint err = writer_flush_block(w);\n+\tint i = 0;\n+\tstruct reftable_block_stats *bstats = NULL;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\twhile (w->index_len > threshold) {\n+\t\tstruct reftable_index_record *idx = NULL;\n+\t\tint idx_len = 0;\n+\n+\t\tmax_level++;\n+\t\tindex_start = w->next;\n+\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\tidx = w->index;\n+\t\tidx_len = w->index_len;\n+\n+\t\tw->index = NULL;\n+\t\tw->index_len = 0;\n+\t\tw->index_cap = 0;\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstruct reftable_record rec = { NULL };\n+\t\t\treftable_record_from_index(&rec, idx + i);\n+\t\t\tif (block_writer_add(w->block_writer, &rec) == 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\terr = writer_flush_block(w);\n+\t\t\tif (err < 0)\n+\t\t\t\treturn err;\n+\n+\t\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\t\terr = block_writer_add(w->block_writer, &rec);\n+\t\t\tif (err != 0) {\n+\t\t\t\t/* write into fresh block should always succeed\n+\t\t\t\t */\n+\t\t\t\tabort();\n+\t\t\t}\n+\t\t}\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstrbuf_release(&idx[i].last_key);\n+\t\t}\n+\t\treftable_free(idx);\n+\t}\n+\n+\twriter_clear_index(w);\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tbstats = writer_reftable_block_stats(w, typ);\n+\tbstats->index_blocks = w->stats.idx_stats.blocks - before_blocks;\n+\tbstats->index_offset = index_start;\n+\tbstats->max_index_level = max_level;\n+\n+\t/* Reinit lastKey, as the next section can start with any key. */\n+\tw->last_key.len = 0;\n+\n+\treturn 0;\n+}\n+\n+struct common_prefix_arg {\n+\tstruct strbuf *last;\n+\tint max;\n+};\n+\n+static void update_common(void *void_arg, void *key)\n+{\n+\tstruct common_prefix_arg *arg = (struct common_prefix_arg *)void_arg;\n+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n+\tif (arg->last != NULL) {\n+\t\tint n = common_prefix_size(&entry->hash, arg->last);\n+\t\tif (n > arg->max) {\n+\t\t\targ->max = n;\n+\t\t}\n+\t}\n+\targ->last = &entry->hash;\n+}\n+\n+struct write_record_arg {\n+\tstruct reftable_writer *w;\n+\tint err;\n+};\n+\n+static void write_object_record(void *void_arg, void *key)\n+{\n+\tstruct write_record_arg *arg = (struct write_record_arg *)void_arg;\n+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n+\tstruct reftable_obj_record obj_rec = {\n+\t\t.hash_prefix = (uint8_t *)entry->hash.buf,\n+\t\t.hash_prefix_len = arg->w->stats.object_id_len,\n+\t\t.offsets = entry->offsets,\n+\t\t.offset_len = entry->offset_len,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\treftable_record_from_obj(&rec, &obj_rec);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\n+\targ->err = writer_flush_block(arg->w);\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\twriter_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\tobj_rec.offset_len = 0;\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\n+\t/* Should be able to write into a fresh block. */\n+\tassert(arg->err == 0);\n+\n+done:;\n+}\n+\n+static void object_record_free(void *void_arg, void *key)\n+{\n+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n+\n+\tFREE_AND_NULL(entry->offsets);\n+\tstrbuf_release(&entry->hash);\n+\treftable_free(entry);\n+}\n+\n+static int writer_dump_object_index(struct reftable_writer *w)\n+{\n+\tstruct write_record_arg closure = { .w = w };\n+\tstruct common_prefix_arg common = { NULL };\n+\tif (w->obj_index_tree != NULL) {\n+\t\tinfix_walk(w->obj_index_tree, &update_common, &common);\n+\t}\n+\tw->stats.object_id_len = common.max + 1;\n+\n+\twriter_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\n+\tif (w->obj_index_tree != NULL) {\n+\t\tinfix_walk(w->obj_index_tree, &write_object_record, &closure);\n+\t}\n+\n+\tif (closure.err < 0)\n+\t\treturn closure.err;\n+\treturn writer_finish_section(w);\n+}\n+\n+static int writer_finish_public_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = 0;\n+\tint err = 0;\n+\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\n+\ttyp = block_writer_type(w->block_writer);\n+\terr = writer_finish_section(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\tif (typ == BLOCK_TYPE_REF && !w->opts.skip_index_objects &&\n+\t w->stats.ref_stats.index_blocks > 0) {\n+\t\terr = writer_dump_object_index(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tif (w->obj_index_tree != NULL) {\n+\t\tinfix_walk(w->obj_index_tree, &object_record_free, NULL);\n+\t\ttree_free(w->obj_index_tree);\n+\t\tw->obj_index_tree = NULL;\n+\t}\n+\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+int reftable_writer_close(struct reftable_writer *w)\n+{\n+\tuint8_t footer[72];\n+\tuint8_t *p = footer;\n+\tint err = writer_finish_public_section(w);\n+\tint empty_table = w->next == 0;\n+\tif (err != 0)\n+\t\tgoto done;\n+\tw->pending_padding = 0;\n+\tif (empty_table) {\n+\t\t/* Empty tables need a header anyway. */\n+\t\tuint8_t header[28];\n+\t\tint n = writer_write_header(w, header);\n+\t\terr = padded_write(w, header, n, 0);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\tp += writer_write_header(w, footer);\n+\tput_be64(p, w->stats.ref_stats.index_offset);\n+\tp += 8;\n+\tput_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n+\tp += 8;\n+\tput_be64(p, w->stats.obj_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be64(p, w->stats.log_stats.offset);\n+\tp += 8;\n+\tput_be64(p, w->stats.log_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be32(p, crc32(0, footer, p - footer));\n+\tp += 4;\n+\n+\terr = padded_write(w, footer, footer_size(writer_version(w)), 0);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (empty_table) {\n+\t\terr = REFTABLE_EMPTY_TABLE_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\t/* free up memory. */\n+\tblock_writer_release(&w->block_writer_data);\n+\twriter_clear_index(w);\n+\tstrbuf_release(&w->last_key);\n+\treturn err;\n+}\n+\n+static void writer_clear_index(struct reftable_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->index_len; i++) {\n+\t\tstrbuf_release(&w->index[i].last_key);\n+\t}\n+\n+\tFREE_AND_NULL(w->index);\n+\tw->index_len = 0;\n+\tw->index_cap = 0;\n+}\n+\n+static const int debug = 0;\n+\n+static int writer_flush_nonempty_block(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tstruct reftable_block_stats *bstats =\n+\t\twriter_reftable_block_stats(w, typ);\n+\tuint64_t block_typ_off = (bstats->blocks == 0) ? w->next : 0;\n+\tint raw_bytes = block_writer_finish(w->block_writer);\n+\tint padding = 0;\n+\tint err = 0;\n+\tstruct reftable_index_record ir = { .last_key = STRBUF_INIT };\n+\tif (raw_bytes < 0)\n+\t\treturn raw_bytes;\n+\n+\tif (!w->opts.unpadded && typ != BLOCK_TYPE_LOG) {\n+\t\tpadding = w->opts.block_size - raw_bytes;\n+\t}\n+\n+\tif (block_typ_off > 0) {\n+\t\tbstats->offset = block_typ_off;\n+\t}\n+\n+\tbstats->entries += w->block_writer->entries;\n+\tbstats->restarts += w->block_writer->restart_len;\n+\tbstats->blocks++;\n+\tw->stats.blocks++;\n+\n+\tif (debug) {\n+\t\tfprintf(stderr, \"block %c off %\" PRIu64 \" sz %d (%d)\\n\", typ,\n+\t\t\tw->next, raw_bytes,\n+\t\t\tget_be24(w->block + w->block_writer->header_off + 1));\n+\t}\n+\n+\tif (w->next == 0) {\n+\t\twriter_write_header(w, w->block);\n+\t}\n+\n+\terr = padded_write(w, w->block, raw_bytes, padding);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (w->index_cap == w->index_len) {\n+\t\tw->index_cap = 2 * w->index_cap + 1;\n+\t\tw->index = reftable_realloc(\n+\t\t\tw->index,\n+\t\t\tsizeof(struct reftable_index_record) * w->index_cap);\n+\t}\n+\n+\tir.offset = w->next;\n+\tstrbuf_reset(&ir.last_key);\n+\tstrbuf_addbuf(&ir.last_key, &w->block_writer->last_key);\n+\tw->index[w->index_len] = ir;\n+\n+\tw->index_len++;\n+\tw->next += padding + raw_bytes;\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+static int writer_flush_block(struct reftable_writer *w)\n+{\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\tif (w->block_writer->entries == 0)\n+\t\treturn 0;\n+\treturn writer_flush_nonempty_block(w);\n+}\n+\n+const struct reftable_stats *writer_stats(struct reftable_writer *w)\n+{\n+\treturn &w->stats;\n+}\ndiff --git a/reftable/writer.h b/reftable/writer.h\nnew file mode 100644\nindex 0000000000..f2ac732358\n--- /dev/null\n+++ b/reftable/writer.h\n@@ -0,0 +1,50 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef WRITER_H\n+#define WRITER_H\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"tree.h\"\n+#include \"reftable-writer.h\"\n+\n+struct reftable_writer {\n+\tint (*write)(void *, const void *, size_t);\n+\tvoid *write_arg;\n+\tint pending_padding;\n+\tstruct strbuf last_key;\n+\n+\t/* offset of next block to write. */\n+\tuint64_t next;\n+\tuint64_t min_update_index, max_update_index;\n+\tstruct reftable_write_options opts;\n+\n+\t/* memory buffer for writing */\n+\tuint8_t *block;\n+\n+\t/* writer for the current section. NULL or points to\n+\t * block_writer_data */\n+\tstruct block_writer *block_writer;\n+\n+\tstruct block_writer block_writer_data;\n+\n+\t/* pending index records for the current section */\n+\tstruct reftable_index_record *index;\n+\tsize_t index_len;\n+\tsize_t index_cap;\n+\n+\t/*\n+\t tree for use with tsearch; used to populate the 'o' inverse OID\n+\t map */\n+\tstruct tree_node *obj_index_tree;\n+\n+\tstruct reftable_stats stats;\n+};\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"410897","messageId":"581a4200d64559a46f6f27916c78b9b01af63893.1606419752.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v3.git.git.1606419752.gitgitgadget@gmail.com","subject":"[PATCH v3 12/16] reftable: reftable file level tests","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-11-26T19:42:27Z","receivedAt":"2020-11-26T19:42:56Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nWith support for reading and writing files in place, we can construct files (in\nmemory) and attempt to read them back.\n\nBecause some sections of the format are optional (eg. indices, log entries), we\nhave to exercise this code using multiple sizes of input data\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable_test.c | 577 +++++++++++++++++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 3 files changed, 579 insertions(+)\n create mode 100644 reftable/reftable_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 51dbcb1b9c..b20acdac0f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2405,6 +2405,7 @@ REFTABLE_OBJS += reftable/zlib-compat.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n+REFTABLE_TEST_OBJS += reftable/reftable_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/reftable_test.c b/reftable/reftable_test.c\nnew file mode 100644\nindex 0000000000..a6fe633b1d\n--- /dev/null\n+++ b/reftable/reftable_test.c\n@@ -0,0 +1,577 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-stack.h\"\n+\n+static const int update_index = 5;\n+\n+static void test_buffer(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_block out = { NULL };\n+\tint n;\n+\tuint8_t in[] = \"hello\";\n+\tstrbuf_add(&buf, in, sizeof(in));\n+\tblock_source_from_strbuf(&source, &buf);\n+\tEXPECT(block_source_size(&source) == 6);\n+\tn = block_source_read_block(&source, &out, 0, sizeof(in));\n+\tEXPECT(n == sizeof(in));\n+\tEXPECT(!memcmp(in, out.data, n));\n+\treftable_block_done(&out);\n+\n+\tn = block_source_read_block(&source, &out, 1, 2);\n+\tEXPECT(n == 2);\n+\tEXPECT(!memcmp(out.data, \"el\", 2));\n+\n+\treftable_block_done(&out);\n+\tblock_source_close(&source);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void write_table(char ***names, struct strbuf *buf, int N,\n+\t\t\tint block_size, uint32_t hash_id)\n+{\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = block_size,\n+\t\t.hash_id = hash_id,\n+\t};\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, buf, &opts);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0, n;\n+\tstruct reftable_log_record log = { NULL };\n+\tconst struct reftable_stats *stats = NULL;\n+\t*names = reftable_calloc(sizeof(char *) * (N + 1));\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[SHA256_SIZE] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tref.refname = name;\n+\t\tref.value = hash;\n+\t\tref.update_index = update_index;\n+\t\t(*names)[i] = xstrdup(name);\n+\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[SHA256_SIZE] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tlog.refname = name;\n+\t\tlog.new_hash = hash;\n+\t\tlog.update_index = update_index;\n+\t\tlog.message = \"message\";\n+\n+\t\tn = reftable_writer_add_log(w, &log);\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tfor (i = 0; i < stats->ref_stats.blocks; i++) {\n+\t\tint off = i * opts.block_size;\n+\t\tif (off == 0) {\n+\t\t\toff = header_size((hash_id == SHA256_ID) ? 2 : 1);\n+\t\t}\n+\t\tEXPECT(buf->buf[off] == 'r');\n+\t}\n+\n+\tEXPECT(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+}\n+\n+static void test_log_buffer_size(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 4096,\n+\t};\n+\tint err;\n+\tstruct reftable_log_record log = {\n+\t\t.refname = \"refs/heads/master\",\n+\t\t.name = \"Han-Wen Nienhuys\",\n+\t\t.email = \"hanwen@google.com\",\n+\t\t.tz_offset = 100,\n+\t\t.time = 0x5e430672,\n+\t\t.update_index = 0xa,\n+\t\t.message = \"commit: 9\\n\",\n+\t};\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\t/* This tests buffer extension for log compression. Must use a random\n+\t hash, to ensure that the compressed part is larger than the original.\n+\t*/\n+\tuint8_t hash1[SHA1_SIZE], hash2[SHA1_SIZE];\n+\tfor (int i = 0; i < SHA1_SIZE; i++) {\n+\t\thash1[i] = (uint8_t)(rand() % 256);\n+\t\thash2[i] = (uint8_t)(rand() % 256);\n+\t}\n+\tlog.old_hash = hash1;\n+\tlog.new_hash = hash2;\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\terr = reftable_writer_add_log(w, &log);\n+\tEXPECT_ERR(err);\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_log_write_read(void)\n+{\n+\tint N = 2;\n+\tchar **names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint err;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\tint n;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tconst struct reftable_stats *stats = NULL;\n+\treftable_writer_set_limits(w, 0, N);\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[256];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tsnprintf(name, sizeof(name), \"b%02d%0*d\", i, 130, 7);\n+\t\tnames[i] = xstrdup(name);\n+\t\tref.refname = name;\n+\t\tref.update_index = i;\n+\n+\t\terr = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash1[SHA1_SIZE], hash2[SHA1_SIZE];\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tset_test_hash(hash1, i);\n+\t\tset_test_hash(hash2, i + 1);\n+\n+\t\tlog.refname = names[i];\n+\t\tlog.update_index = i;\n+\t\tlog.old_hash = hash1;\n+\t\tlog.new_hash = hash2;\n+\n+\t\terr = reftable_writer_add_log(w, &log);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tEXPECT(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[N - 1]);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT_ERR(err);\n+\n+\t/* end of iteration. */\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT(0 < err);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_release(&ref);\n+\n+\terr = reftable_reader_seek_log(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\ti = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT_STREQ(names[i], log.refname);\n+\t\tEXPECT(i == log.update_index);\n+\t\ti++;\n+\t\treftable_log_record_release(&log);\n+\t}\n+\n+\tEXPECT(i == N);\n+\treftable_iterator_destroy(&it);\n+\n+\t/* cleanup. */\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_sequential(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tint err = 0;\n+\tint j = 0;\n+\n+\twrite_table(&names, &buf, N, 256, SHA1_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint r = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tEXPECT(0 == strcmp(names[j], ref.refname));\n+\t\tEXPECT(update_index == ref.update_index);\n+\n+\t\tj++;\n+\t\treftable_ref_record_release(&ref);\n+\t}\n+\tEXPECT(j == N);\n+\treftable_iterator_destroy(&it);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\n+\treader_close(&rd);\n+}\n+\n+static void test_table_write_small_table(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 1;\n+\twrite_table(&names, &buf, N, 4096, SHA1_ID);\n+\tEXPECT(buf.len < 200);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+}\n+\n+static void test_table_read_api(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i;\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, SHA1_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[0]);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_free(names);\n+\treader_close(&rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_table_read_write_seek(int index, int hash_id)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i = 0;\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct strbuf pastLast = STRBUF_INIT;\n+\tstruct reftable_ref_record ref = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, hash_id);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\tEXPECT(hash_id == reftable_reader_hash_id(&rd));\n+\n+\tif (!index) {\n+\t\trd.ref_offsets.index_offset = 0;\n+\t} else {\n+\t\tEXPECT(rd.ref_offsets.index_offset > 0);\n+\t}\n+\n+\tfor (i = 1; i < N; i++) {\n+\t\tint err = reftable_reader_seek_ref(&rd, &it, names[i]);\n+\t\tEXPECT_ERR(err);\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(0 == strcmp(names[i], ref.refname));\n+\t\tEXPECT(i == ref.value[0]);\n+\n+\t\treftable_ref_record_release(&ref);\n+\t\treftable_iterator_destroy(&it);\n+\t}\n+\n+\tstrbuf_addstr(&pastLast, names[N - 1]);\n+\tstrbuf_addstr(&pastLast, \"/\");\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, pastLast.buf);\n+\tif (err == 0) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(err > 0);\n+\t} else {\n+\t\tEXPECT(err > 0);\n+\t}\n+\n+\tstrbuf_release(&pastLast);\n+\treftable_iterator_destroy(&it);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_free(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_seek_linear(void)\n+{\n+\ttest_table_read_write_seek(0, SHA1_ID);\n+}\n+\n+static void test_table_read_write_seek_linear_sha256(void)\n+{\n+\ttest_table_read_write_seek(0, SHA256_ID);\n+}\n+\n+static void test_table_read_write_seek_index(void)\n+{\n+\ttest_table_read_write_seek(1, SHA1_ID);\n+}\n+\n+static void test_table_refs_for(int indexed)\n+{\n+\tint N = 50;\n+\tchar **want_names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint want_names_len = 0;\n+\tuint8_t want_hash[SHA1_SIZE];\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tint err;\n+\tstruct reftable_reader rd;\n+\tstruct reftable_block_source source = { NULL };\n+\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint j;\n+\n+\tset_test_hash(want_hash, 4);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[SHA1_SIZE];\n+\t\tchar fill[51] = { 0 };\n+\t\tchar name[100];\n+\t\tuint8_t hash1[SHA1_SIZE];\n+\t\tuint8_t hash2[SHA1_SIZE];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\n+\t\tmemset(hash, i, sizeof(hash));\n+\t\tmemset(fill, 'x', 50);\n+\t\t/* Put the variable part in the start */\n+\t\tsnprintf(name, sizeof(name), \"br%02d%s\", i, fill);\n+\t\tname[40] = 0;\n+\t\tref.refname = name;\n+\n+\t\tset_test_hash(hash1, i / 4);\n+\t\tset_test_hash(hash2, 3 + i / 4);\n+\t\tref.value = hash1;\n+\t\tref.target_value = hash2;\n+\n+\t\t/* 80 bytes / entry, so 3 entries per block. Yields 17\n+\t\t */\n+\t\t/* blocks. */\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT(n == 0);\n+\n+\t\tif (!memcmp(hash1, want_hash, SHA1_SIZE) ||\n+\t\t !memcmp(hash2, want_hash, SHA1_SIZE)) {\n+\t\t\twant_names[want_names_len++] = xstrdup(name);\n+\t\t}\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\tif (!indexed) {\n+\t\trd.obj_offsets.is_present = 0;\n+\t}\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_reader_refs_for(&rd, &it, want_hash);\n+\tEXPECT_ERR(err);\n+\n+\tj = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(err >= 0);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tEXPECT(j < want_names_len);\n+\t\tEXPECT(0 == strcmp(ref.refname, want_names[j]));\n+\t\tj++;\n+\t\treftable_ref_record_release(&ref);\n+\t}\n+\tEXPECT(j == want_names_len);\n+\n+\tstrbuf_release(&buf);\n+\tfree_names(want_names);\n+\treftable_iterator_destroy(&it);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_refs_for_no_index(void)\n+{\n+\ttest_table_refs_for(0);\n+}\n+\n+static void test_table_refs_for_obj_index(void)\n+{\n+\ttest_table_refs_for(1);\n+}\n+\n+static void test_table_empty(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_ref_record rec = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT(err == REFTABLE_EMPTY_TABLE_ERROR);\n+\treftable_writer_free(w);\n+\n+\tEXPECT(buf.len == header_size(1) + footer_size(1));\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tEXPECT(err > 0);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+int reftable_test_main(int argc, const char *argv[])\n+{\n+\ttest_log_write_read();\n+\ttest_table_read_write_seek_linear_sha256();\n+\ttest_log_buffer_size();\n+\ttest_table_write_small_table();\n+\ttest_buffer();\n+\ttest_table_read_api();\n+\ttest_table_read_write_sequential();\n+\ttest_table_read_write_seek_linear();\n+\ttest_table_read_write_seek_index();\n+\ttest_table_refs_for_no_index();\n+\ttest_table_refs_for_obj_index();\n+\ttest_table_empty();\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 050551fa69..fdf9258673 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,6 +6,7 @@ int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\treftable_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"410898","messageId":"9aa2caade8c50149a2a855606eda284d4cd9acff.1606419752.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v3.git.git.1606419752.gitgitgadget@gmail.com","subject":"[PATCH v3 11/16] reftable: read reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-11-26T19:42:26Z","receivedAt":"2020-11-26T19:42:56Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis supports reading a single reftable file.\n\nThe commit introduces an abstract iterator type, which captures the usecases\nboth of reading individual refs, and iterating over a segment of the ref\nnamespace.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 3 +\n reftable/iter.c | 242 ++++++++++++\n reftable/iter.h | 75 ++++\n reftable/reader.c | 733 +++++++++++++++++++++++++++++++++++\n reftable/reader.h | 75 ++++\n reftable/reftable-iterator.h | 37 ++\n reftable/reftable-reader.h | 89 +++++\n 7 files changed, 1254 insertions(+)\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/reftable-iterator.h\n create mode 100644 reftable/reftable-reader.h\n\ndiff --git a/Makefile b/Makefile\nindex 3e33a409f0..51dbcb1b9c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2393,8 +2393,11 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n+REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/reftable.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n REFTABLE_OBJS += reftable/zlib-compat.o\ndiff --git a/reftable/iter.c b/reftable/iter.c\nnew file mode 100644\nindex 0000000000..63d87b758c\n--- /dev/null\n+++ b/reftable/iter.c\n@@ -0,0 +1,242 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"iter.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"reftable-error.h\"\n+\n+int iterator_is_null(struct reftable_iterator *it)\n+{\n+\treturn it->ops == NULL;\n+}\n+\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n+{\n+\treturn 1;\n+}\n+\n+static void empty_iterator_close(void *arg)\n+{\n+}\n+\n+static struct reftable_iterator_vtable empty_vtable = {\n+\t.next = &empty_iterator_next,\n+\t.close = &empty_iterator_close,\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = NULL;\n+\tit->ops = &empty_vtable;\n+}\n+\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n+{\n+\treturn it->ops->next(it->iter_arg, rec);\n+}\n+\n+void reftable_iterator_destroy(struct reftable_iterator *it)\n+{\n+\tif (it->ops == NULL) {\n+\t\treturn;\n+\t}\n+\tit->ops->close(it->iter_arg);\n+\tit->ops = NULL;\n+\tFREE_AND_NULL(it->iter_arg);\n+}\n+\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, ref);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, log);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+static void filtering_ref_iterator_close(void *iter_arg)\n+{\n+\tstruct filtering_ref_iterator *fri =\n+\t\t(struct filtering_ref_iterator *)iter_arg;\n+\tstrbuf_release(&fri->oid);\n+\treftable_iterator_destroy(&fri->it);\n+}\n+\n+static int filtering_ref_iterator_next(void *iter_arg,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct filtering_ref_iterator *fri =\n+\t\t(struct filtering_ref_iterator *)iter_arg;\n+\tstruct reftable_ref_record *ref =\n+\t\t(struct reftable_ref_record *)rec->data;\n+\tint err = 0;\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&fri->it, ref);\n+\t\tif (err != 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (fri->double_check) {\n+\t\t\tstruct reftable_iterator it = { NULL };\n+\n+\t\t\terr = reftable_table_seek_ref(&fri->tab, &it,\n+\t\t\t\t\t\t ref->refname);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = reftable_iterator_next_ref(&it, ref);\n+\t\t\t}\n+\n+\t\t\treftable_iterator_destroy(&it);\n+\n+\t\t\tif (err < 0) {\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (err > 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif ((ref->target_value != NULL &&\n+\t\t !memcmp(fri->oid.buf, ref->target_value, fri->oid.len)) ||\n+\t\t (ref->value != NULL &&\n+\t\t !memcmp(fri->oid.buf, ref->value, fri->oid.len))) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treftable_ref_record_release(ref);\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable filtering_ref_iterator_vtable = {\n+\t.next = &filtering_ref_iterator_next,\n+\t.close = &filtering_ref_iterator_close,\n+};\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *it,\n+\t\t\t\t\t struct filtering_ref_iterator *fri)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = fri;\n+\tit->ops = &filtering_ref_iterator_vtable;\n+}\n+\n+static void indexed_table_ref_iter_close(void *p)\n+{\n+\tstruct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p;\n+\tblock_iter_close(&it->cur);\n+\treftable_block_done(&it->block_reader.block);\n+\tstrbuf_release(&it->oid);\n+}\n+\n+static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n+{\n+\tuint64_t off;\n+\tint err = 0;\n+\tif (it->offset_idx == it->offset_len) {\n+\t\tit->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\n+\treftable_block_done(&it->block_reader.block);\n+\n+\toff = it->offsets[it->offset_idx++];\n+\terr = reader_init_block_reader(it->r, &it->block_reader, off,\n+\t\t\t\t BLOCK_TYPE_REF);\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\tif (err > 0) {\n+\t\t/* indexed block does not exist. */\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\t}\n+\tblock_reader_start(&it->block_reader, &it->cur);\n+\treturn 0;\n+}\n+\n+static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\n+{\n+\tstruct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p;\n+\tstruct reftable_ref_record *ref =\n+\t\t(struct reftable_ref_record *)rec->data;\n+\n+\twhile (1) {\n+\t\tint err = block_iter_next(&it->cur, rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\terr = indexed_table_ref_iter_next_block(it);\n+\t\t\tif (err < 0) {\n+\t\t\t\treturn err;\n+\t\t\t}\n+\n+\t\t\tif (it->is_finished) {\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!memcmp(it->oid.buf, ref->target_value, it->oid.len) ||\n+\t\t !memcmp(it->oid.buf, ref->value, it->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+}\n+\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len)\n+{\n+\tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n+\tstruct indexed_table_ref_iter *itr =\n+\t\treftable_calloc(sizeof(struct indexed_table_ref_iter));\n+\tint err = 0;\n+\n+\t*itr = empty;\n+\titr->r = r;\n+\tstrbuf_add(&itr->oid, oid, oid_len);\n+\n+\titr->offsets = offsets;\n+\titr->offset_len = offset_len;\n+\n+\terr = indexed_table_ref_iter_next_block(itr);\n+\tif (err < 0) {\n+\t\treftable_free(itr);\n+\t} else {\n+\t\t*dest = itr;\n+\t}\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable indexed_table_ref_iter_vtable = {\n+\t.next = &indexed_table_ref_iter_next,\n+\t.close = &indexed_table_ref_iter_close,\n+};\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = itr;\n+\tit->ops = &indexed_table_ref_iter_vtable;\n+}\ndiff --git a/reftable/iter.h b/reftable/iter.h\nnew file mode 100644\nindex 0000000000..258bc107af\n--- /dev/null\n+++ b/reftable/iter.h\n@@ -0,0 +1,75 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef ITER_H\n+#define ITER_H\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"record.h\"\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+struct reftable_iterator_vtable {\n+\tint (*next)(void *iter_arg, struct reftable_record *rec);\n+\tvoid (*close)(void *iter_arg);\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it);\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec);\n+\n+/* Returns true for a zeroed out iterator, such as the one returned from\n+ iterator_destroy. */\n+int iterator_is_null(struct reftable_iterator *it);\n+\n+/* iterator that produces only ref records that point to `oid` */\n+struct filtering_ref_iterator {\n+\tint double_check;\n+\tstruct reftable_table tab;\n+\tstruct strbuf oid;\n+\tstruct reftable_iterator it;\n+};\n+#define FILTERING_REF_ITERATOR_INIT \\\n+\t{ \\\n+\t\t.oid = STRBUF_INIT \\\n+\t}\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *,\n+\t\t\t\t\t struct filtering_ref_iterator *);\n+\n+/* iterator that produces only ref records that point to `oid`,\n+ but using the object index.\n+ */\n+struct indexed_table_ref_iter {\n+\tstruct reftable_reader *r;\n+\tstruct strbuf oid;\n+\n+\t/* mutable */\n+\tuint64_t *offsets;\n+\n+\t/* Points to the next offset to read. */\n+\tint offset_idx;\n+\tint offset_len;\n+\tstruct block_reader block_reader;\n+\tstruct block_iter cur;\n+\tint is_finished;\n+};\n+\n+#define INDEXED_TABLE_REF_ITER_INIT \\\n+\t{ \\\n+\t\t.cur = { .last_key = STRBUF_INIT }, .oid = STRBUF_INIT, \\\n+\t}\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr);\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len);\n+\n+#endif\ndiff --git a/reftable/reader.c b/reftable/reader.c\nnew file mode 100644\nindex 0000000000..a5b7b1d4fe\n--- /dev/null\n+++ b/reftable/reader.c\n@@ -0,0 +1,733 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reader.h\"\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"iter.h\"\n+#include \"record.h\"\n+#include \"reftable-error.h\"\n+#include \"tree.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source)\n+{\n+\treturn source->ops->size(source->arg);\n+}\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tint result = source->ops->read_block(source->arg, dest, off, size);\n+\tdest->source = *source;\n+\treturn result;\n+}\n+\n+void block_source_close(struct reftable_block_source *source)\n+{\n+\tif (source->ops == NULL) {\n+\t\treturn;\n+\t}\n+\n+\tsource->ops->close(source->arg);\n+\tsource->ops = NULL;\n+}\n+\n+static struct reftable_reader_offsets *\n+reader_offsets_for(struct reftable_reader *r, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\t\treturn &r->ref_offsets;\n+\tcase BLOCK_TYPE_LOG:\n+\t\treturn &r->log_offsets;\n+\tcase BLOCK_TYPE_OBJ:\n+\t\treturn &r->obj_offsets;\n+\t}\n+\tabort();\n+}\n+\n+static int reader_get_block(struct reftable_reader *r,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t sz)\n+{\n+\tif (off >= r->size)\n+\t\treturn 0;\n+\n+\tif (off + sz > r->size) {\n+\t\tsz = r->size - off;\n+\t}\n+\n+\treturn block_source_read_block(&r->source, dest, off, sz);\n+}\n+\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r)\n+{\n+\treturn r->hash_id;\n+}\n+\n+const char *reader_name(struct reftable_reader *r)\n+{\n+\treturn r->name;\n+}\n+\n+static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n+\t\t\tuint8_t *header)\n+{\n+\tuint8_t *f = footer;\n+\tuint8_t first_block_typ;\n+\tint err = 0;\n+\tuint32_t computed_crc;\n+\tuint32_t file_crc;\n+\n+\tif (memcmp(f, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tf += 4;\n+\n+\tif (memcmp(footer, header, header_size(r->version))) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tf++;\n+\tr->block_size = get_be24(f);\n+\n+\tf += 3;\n+\tr->min_update_index = get_be64(f);\n+\tf += 8;\n+\tr->max_update_index = get_be64(f);\n+\tf += 8;\n+\n+\tif (r->version == 1) {\n+\t\tr->hash_id = SHA1_ID;\n+\t} else {\n+\t\tr->hash_id = get_be32(f);\n+\t\tswitch (r->hash_id) {\n+\t\tcase SHA1_ID:\n+\t\t\tbreak;\n+\t\tcase SHA256_ID:\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tf += 4;\n+\t}\n+\n+\tr->ref_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->obj_offsets.offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1);\n+\tr->obj_offsets.offset >>= 5;\n+\n+\tr->obj_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tcomputed_crc = crc32(0, footer, f - footer);\n+\tfile_crc = get_be32(f);\n+\tf += 4;\n+\tif (computed_crc != file_crc) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tfirst_block_typ = header[header_size(r->version)];\n+\tr->ref_offsets.is_present = (first_block_typ == BLOCK_TYPE_REF);\n+\tr->ref_offsets.offset = 0;\n+\tr->log_offsets.is_present = (first_block_typ == BLOCK_TYPE_LOG ||\n+\t\t\t\t r->log_offsets.offset > 0);\n+\tr->obj_offsets.is_present = r->obj_offsets.offset > 0;\n+\terr = 0;\n+done:\n+\treturn err;\n+}\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name)\n+{\n+\tstruct reftable_block footer = { NULL };\n+\tstruct reftable_block header = { NULL };\n+\tint err = 0;\n+\n+\tmemset(r, 0, sizeof(struct reftable_reader));\n+\n+\t/* Need +1 to read type of first block. */\n+\terr = block_source_read_block(source, &header, 0, header_size(2) + 1);\n+\tif (err != header_size(2) + 1) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tif (memcmp(header.data, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tr->version = header.data[4];\n+\tif (r->version != 1 && r->version != 2) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tr->size = block_source_size(source) - footer_size(r->version);\n+\tr->source = *source;\n+\tr->name = xstrdup(name);\n+\tr->hash_id = 0;\n+\n+\terr = block_source_read_block(source, &footer, r->size,\n+\t\t\t\t footer_size(r->version));\n+\tif (err != footer_size(r->version)) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = parse_footer(r, footer.data, header.data);\n+done:\n+\treftable_block_done(&footer);\n+\treftable_block_done(&header);\n+\treturn err;\n+}\n+\n+struct table_iter {\n+\tstruct reftable_reader *r;\n+\tuint8_t typ;\n+\tuint64_t block_off;\n+\tstruct block_iter bi;\n+\tint is_finished;\n+};\n+#define TABLE_ITER_INIT \\\n+\t{ \\\n+\t\t.bi = {.last_key = STRBUF_INIT } \\\n+\t}\n+\n+static void table_iter_copy_from(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = src->block_off;\n+\tdest->is_finished = src->is_finished;\n+\tblock_iter_copy_from(&dest->bi, &src->bi);\n+}\n+\n+static int table_iter_next_in_block(struct table_iter *ti,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tint res = block_iter_next(&ti->bi, rec);\n+\tif (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {\n+\t\t((struct reftable_ref_record *)rec->data)->update_index +=\n+\t\t\tti->r->min_update_index;\n+\t}\n+\n+\treturn res;\n+}\n+\n+static void table_iter_block_done(struct table_iter *ti)\n+{\n+\tif (ti->bi.br == NULL) {\n+\t\treturn;\n+\t}\n+\treftable_block_done(&ti->bi.br->block);\n+\tFREE_AND_NULL(ti->bi.br);\n+\n+\tti->bi.last_key.len = 0;\n+\tti->bi.next_off = 0;\n+}\n+\n+static int32_t extract_block_size(uint8_t *data, uint8_t *typ, uint64_t off,\n+\t\t\t\t int version)\n+{\n+\tint32_t result = 0;\n+\n+\tif (off == 0) {\n+\t\tdata += header_size(version);\n+\t}\n+\n+\t*typ = data[0];\n+\tif (reftable_is_block_type(*typ)) {\n+\t\tresult = get_be24(data + 1);\n+\t}\n+\treturn result;\n+}\n+\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ)\n+{\n+\tint32_t guess_block_size = r->block_size ? r->block_size :\n+\t\t\t\t\t\t\t DEFAULT_BLOCK_SIZE;\n+\tstruct reftable_block block = { NULL };\n+\tuint8_t block_typ = 0;\n+\tint err = 0;\n+\tuint32_t header_off = next_off ? 0 : header_size(r->version);\n+\tint32_t block_size = 0;\n+\n+\tif (next_off >= r->size)\n+\t\treturn 1;\n+\n+\terr = reader_get_block(r, &block, next_off, guess_block_size);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tblock_size = extract_block_size(block.data, &block_typ, next_off,\n+\t\t\t\t\tr->version);\n+\tif (block_size < 0)\n+\t\treturn block_size;\n+\n+\tif (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {\n+\t\treftable_block_done(&block);\n+\t\treturn 1;\n+\t}\n+\n+\tif (block_size > guess_block_size) {\n+\t\treftable_block_done(&block);\n+\t\terr = reader_get_block(r, &block, next_off, block_size);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\treturn block_reader_init(br, &block, header_off, r->block_size,\n+\t\t\t\t hash_size(r->hash_id));\n+}\n+\n+static int table_iter_next_block(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tuint64_t next_block_off = src->block_off + src->bi.br->full_block_size;\n+\tstruct block_reader br = { 0 };\n+\tint err = 0;\n+\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = next_block_off;\n+\n+\terr = reader_init_block_reader(src->r, &br, next_block_off, src->typ);\n+\tif (err > 0) {\n+\t\tdest->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\tif (err != 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct block_reader *brp =\n+\t\t\treftable_malloc(sizeof(struct block_reader));\n+\t\t*brp = br;\n+\n+\t\tdest->is_finished = 0;\n+\t\tblock_reader_start(brp, &dest->bi);\n+\t}\n+\treturn 0;\n+}\n+\n+static int table_iter_next(struct table_iter *ti, struct reftable_record *rec)\n+{\n+\tif (reftable_record_type(rec) != ti->typ)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\twhile (1) {\n+\t\tstruct table_iter next = TABLE_ITER_INIT;\n+\t\tint err = 0;\n+\t\tif (ti->is_finished) {\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\terr = table_iter_next_in_block(ti, rec);\n+\t\tif (err <= 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err != 0) {\n+\t\t\tti->is_finished = 1;\n+\t\t}\n+\t\ttable_iter_block_done(ti);\n+\t\tif (err != 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\ttable_iter_copy_from(ti, &next);\n+\t\tblock_iter_close(&next.bi);\n+\t}\n+}\n+\n+static int table_iter_next_void(void *ti, struct reftable_record *rec)\n+{\n+\treturn table_iter_next((struct table_iter *)ti, rec);\n+}\n+\n+static void table_iter_close(void *p)\n+{\n+\tstruct table_iter *ti = (struct table_iter *)p;\n+\ttable_iter_block_done(ti);\n+\tblock_iter_close(&ti->bi);\n+}\n+\n+static struct reftable_iterator_vtable table_iter_vtable = {\n+\t.next = &table_iter_next_void,\n+\t.close = &table_iter_close,\n+};\n+\n+static void iterator_from_table_iter(struct reftable_iterator *it,\n+\t\t\t\t struct table_iter *ti)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = ti;\n+\tit->ops = &table_iter_vtable;\n+}\n+\n+static int reader_table_iter_at(struct reftable_reader *r,\n+\t\t\t\tstruct table_iter *ti, uint64_t off,\n+\t\t\t\tuint8_t typ)\n+{\n+\tstruct block_reader br = { 0 };\n+\tstruct block_reader *brp = NULL;\n+\n+\tint err = reader_init_block_reader(r, &br, off, typ);\n+\tif (err != 0)\n+\t\treturn err;\n+\n+\tbrp = reftable_malloc(sizeof(struct block_reader));\n+\t*brp = br;\n+\tti->r = r;\n+\tti->typ = block_reader_type(brp);\n+\tti->block_off = off;\n+\tblock_reader_start(brp, &ti->bi);\n+\treturn 0;\n+}\n+\n+static int reader_start(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\tuint8_t typ, int index)\n+{\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tuint64_t off = offs->offset;\n+\tif (index) {\n+\t\toff = offs->index_offset;\n+\t\tif (off == 0) {\n+\t\t\treturn 1;\n+\t\t}\n+\t\ttyp = BLOCK_TYPE_INDEX;\n+\t}\n+\n+\treturn reader_table_iter_at(r, ti, off, typ);\n+}\n+\n+static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\t struct reftable_record *want)\n+{\n+\tstruct reftable_record rec =\n+\t\treftable_new_record(reftable_record_type(want));\n+\tstruct strbuf want_key = STRBUF_INIT;\n+\tstruct strbuf got_key = STRBUF_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = -1;\n+\n+\treftable_record_key(want, &want_key);\n+\n+\twhile (1) {\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = block_reader_first_key(next.bi.br, &got_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (strbuf_cmp(&got_key, &want_key) > 0) {\n+\t\t\ttable_iter_block_done(&next);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_block_done(ti);\n+\t\ttable_iter_copy_from(ti, &next);\n+\t}\n+\n+\terr = block_iter_seek(&ti->bi, &want_key);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = 0;\n+\n+done:\n+\tblock_iter_close(&next.bi);\n+\treftable_record_destroy(&rec);\n+\tstrbuf_release(&want_key);\n+\tstrbuf_release(&got_key);\n+\treturn err;\n+}\n+\n+static int reader_seek_indexed(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_index_record want_index = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record want_index_rec = { NULL };\n+\tstruct reftable_index_record index_result = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record index_result_rec = { NULL };\n+\tstruct table_iter index_iter = TABLE_ITER_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = 0;\n+\n+\treftable_record_key(rec, &want_index.last_key);\n+\treftable_record_from_index(&want_index_rec, &want_index);\n+\treftable_record_from_index(&index_result_rec, &index_result);\n+\n+\terr = reader_start(r, &index_iter, reftable_record_type(rec), 1);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reader_seek_linear(r, &index_iter, &want_index_rec);\n+\twhile (1) {\n+\t\terr = table_iter_next(&index_iter, &index_result_rec);\n+\t\ttable_iter_block_done(&index_iter);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = reader_table_iter_at(r, &next, index_result.offset, 0);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = block_iter_seek(&next.bi, &want_index.last_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (next.typ == reftable_record_type(rec)) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (next.typ != BLOCK_TYPE_INDEX) {\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_copy_from(&index_iter, &next);\n+\t}\n+\n+\tif (err == 0) {\n+\t\tstruct table_iter empty = TABLE_ITER_INIT;\n+\t\tstruct table_iter *malloced =\n+\t\t\treftable_calloc(sizeof(struct table_iter));\n+\t\t*malloced = empty;\n+\t\ttable_iter_copy_from(malloced, &next);\n+\t\titerator_from_table_iter(it, malloced);\n+\t}\n+done:\n+\tblock_iter_close(&next.bi);\n+\ttable_iter_close(&index_iter);\n+\treftable_record_release(&want_index_rec);\n+\treftable_record_release(&index_result_rec);\n+\treturn err;\n+}\n+\n+static int reader_seek_internal(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it,\n+\t\t\t\tstruct reftable_record *rec)\n+{\n+\tstruct reftable_reader_offsets *offs =\n+\t\treader_offsets_for(r, reftable_record_type(rec));\n+\tuint64_t idx = offs->index_offset;\n+\tstruct table_iter ti = TABLE_ITER_INIT;\n+\tint err = 0;\n+\tif (idx > 0)\n+\t\treturn reader_seek_indexed(r, it, rec);\n+\n+\terr = reader_start(r, &ti, reftable_record_type(rec), 0);\n+\tif (err < 0)\n+\t\treturn err;\n+\terr = reader_seek_linear(r, &ti, rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct table_iter *p =\n+\t\t\treftable_malloc(sizeof(struct table_iter));\n+\t\t*p = ti;\n+\t\titerator_from_table_iter(it, p);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec)\n+{\n+\tuint8_t typ = reftable_record_type(rec);\n+\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tif (!offs->is_present) {\n+\t\titerator_set_empty(it);\n+\t\treturn 0;\n+\t}\n+\n+\treturn reader_seek_internal(r, it, rec);\n+}\n+\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_reader_seek_log_at(r, it, name, max);\n+}\n+\n+void reader_close(struct reftable_reader *r)\n+{\n+\tblock_source_close(&r->source);\n+\tFREE_AND_NULL(r->name);\n+}\n+\n+int reftable_new_reader(struct reftable_reader **p,\n+\t\t\tstruct reftable_block_source *src, char const *name)\n+{\n+\tstruct reftable_reader *rd =\n+\t\treftable_calloc(sizeof(struct reftable_reader));\n+\tint err = init_reader(rd, src, name);\n+\tif (err == 0) {\n+\t\t*p = rd;\n+\t} else {\n+\t\tblock_source_close(src);\n+\t\treftable_free(rd);\n+\t}\n+\treturn err;\n+}\n+\n+void reftable_reader_free(struct reftable_reader *r)\n+{\n+\treader_close(r);\n+\treftable_free(r);\n+}\n+\n+static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct reftable_obj_record want = {\n+\t\t.hash_prefix = oid,\n+\t\t.hash_prefix_len = r->object_id_len,\n+\t};\n+\tstruct reftable_record want_rec = { NULL };\n+\tstruct reftable_iterator oit = { NULL };\n+\tstruct reftable_obj_record got = { NULL };\n+\tstruct reftable_record got_rec = { NULL };\n+\tint err = 0;\n+\tstruct indexed_table_ref_iter *itr = NULL;\n+\n+\t/* Look through the reverse index. */\n+\treftable_record_from_obj(&want_rec, &want);\n+\terr = reader_seek(r, &oit, &want_rec);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\t/* read out the reftable_obj_record */\n+\treftable_record_from_obj(&got_rec, &got);\n+\terr = iterator_next(&oit, &got_rec);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 0 ||\n+\t memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {\n+\t\t/* didn't find it; return empty iterator */\n+\t\titerator_set_empty(it);\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),\n+\t\t\t\t\t got.offsets, got.offset_len);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tgot.offsets = NULL;\n+\titerator_from_indexed_table_ref_iter(it, itr);\n+\n+done:\n+\treftable_iterator_destroy(&oit);\n+\treftable_record_release(&got_rec);\n+\treturn err;\n+}\n+\n+static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct table_iter ti_empty = TABLE_ITER_INIT;\n+\tstruct table_iter *ti = reftable_calloc(sizeof(struct table_iter));\n+\tstruct filtering_ref_iterator *filter = NULL;\n+\tstruct filtering_ref_iterator empty = FILTERING_REF_ITERATOR_INIT;\n+\tint oid_len = hash_size(r->hash_id);\n+\tint err;\n+\n+\t*ti = ti_empty;\n+\terr = reader_start(r, ti, BLOCK_TYPE_REF, 0);\n+\tif (err < 0) {\n+\t\treftable_free(ti);\n+\t\treturn err;\n+\t}\n+\n+\tfilter = reftable_malloc(sizeof(struct filtering_ref_iterator));\n+\t*filter = empty;\n+\n+\tstrbuf_add(&filter->oid, oid, oid_len);\n+\treftable_table_from_reader(&filter->tab, r);\n+\tfilter->double_check = 0;\n+\titerator_from_table_iter(&filter->it, ti);\n+\n+\titerator_from_filtering_ref_iterator(it, filter);\n+\treturn 0;\n+}\n+\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid)\n+{\n+\tif (r->obj_offsets.is_present)\n+\t\treturn reftable_reader_refs_for_indexed(r, it, oid);\n+\treturn reftable_reader_refs_for_unindexed(r, it, oid);\n+}\n+\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r)\n+{\n+\treturn r->max_update_index;\n+}\n+\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r)\n+{\n+\treturn r->min_update_index;\n+}\ndiff --git a/reftable/reader.h b/reftable/reader.h\nnew file mode 100644\nindex 0000000000..6d4927e1c5\n--- /dev/null\n+++ b/reftable/reader.h\n@@ -0,0 +1,75 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef READER_H\n+#define READER_H\n+\n+#include \"block.h\"\n+#include \"record.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source);\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size);\n+void block_source_close(struct reftable_block_source *source);\n+\n+/* metadata for a block type */\n+struct reftable_reader_offsets {\n+\tint is_present;\n+\tuint64_t offset;\n+\tuint64_t index_offset;\n+};\n+\n+/* The state for reading a reftable file. */\n+struct reftable_reader {\n+\t/* for convience, associate a name with the instance. */\n+\tchar *name;\n+\tstruct reftable_block_source source;\n+\n+\t/* Size of the file, excluding the footer. */\n+\tuint64_t size;\n+\n+\t/* 'sha1' for SHA1, 's256' for SHA-256 */\n+\tuint32_t hash_id;\n+\n+\tuint32_t block_size;\n+\tuint64_t min_update_index;\n+\tuint64_t max_update_index;\n+\t/* Length of the OID keys in the 'o' section */\n+\tint object_id_len;\n+\tint version;\n+\n+\tstruct reftable_reader_offsets ref_offsets;\n+\tstruct reftable_reader_offsets obj_offsets;\n+\tstruct reftable_reader_offsets log_offsets;\n+};\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name);\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec);\n+void reader_close(struct reftable_reader *r);\n+const char *reader_name(struct reftable_reader *r);\n+\n+/* initialize a block reader to read from `r` */\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ);\n+\n+/* generic interface to reftables */\n+struct reftable_table_vtable {\n+\tint (*seek_record)(void *tab, struct reftable_iterator *it,\n+\t\t\t struct reftable_record *);\n+\tuint32_t (*hash_id)(void *tab);\n+\tuint64_t (*min_update_index)(void *tab);\n+\tuint64_t (*max_update_index)(void *tab);\n+};\n+\n+#endif\ndiff --git a/reftable/reftable-iterator.h b/reftable/reftable-iterator.h\nnew file mode 100644\nindex 0000000000..a3d6be7ae0\n--- /dev/null\n+++ b/reftable/reftable-iterator.h\n@@ -0,0 +1,37 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_ITERATOR_H\n+#define REFTABLE_ITERATOR_H\n+\n+#include \"reftable-record.h\"\n+\n+/* iterator is the generic interface for walking over data stored in a\n+ reftable.\n+*/\n+struct reftable_iterator {\n+\tstruct reftable_iterator_vtable *ops;\n+\tvoid *iter_arg;\n+};\n+\n+/* reads the next reftable_ref_record. Returns < 0 for error, 0 for OK and > 0:\n+ end of iteration.\n+*/\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* reads the next reftable_log_record. Returns < 0 for error, 0 for OK and > 0:\n+ end of iteration.\n+*/\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* releases resources associated with an iterator. */\n+void reftable_iterator_destroy(struct reftable_iterator *it);\n+\n+#endif\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nnew file mode 100644\nindex 0000000000..827957a8fb\n--- /dev/null\n+++ b/reftable/reftable-reader.h\n@@ -0,0 +1,89 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_READER_H\n+#define REFTABLE_READER_H\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-blocksource.h\"\n+\n+/*\n+ Reading single tables\n+\n+ The follow routines are for reading single files. For an application-level\n+ interface, skip ahead to struct reftable_merged_table and struct\n+ reftable_stack.\n+*/\n+\n+/* The reader struct is a handle to an open reftable file. */\n+struct reftable_reader;\n+\n+/* reftable_new_reader opens a reftable for reading. If successful, returns 0\n+ * code and sets pp. The name is used for creating a stack. Typically, it is the\n+ * basename of the file. The block source `src` is owned by the reader, and is\n+ * closed on calling reftable_reader_destroy().\n+ */\n+int reftable_new_reader(struct reftable_reader **pp,\n+\t\t\tstruct reftable_block_source *src, const char *name);\n+\n+/* reftable_reader_seek_ref returns an iterator where 'name' would be inserted\n+ in the table. To seek to the start of the table, use name = \"\".\n+\n+ example:\n+\n+ struct reftable_reader *r = NULL;\n+ int err = reftable_new_reader(&r, &src, \"filename\");\n+ if (err < 0) { ... }\n+ struct reftable_iterator it = {0};\n+ err = reftable_reader_seek_ref(r, &it, \"refs/heads/master\");\n+ if (err < 0) { ... }\n+ struct reftable_ref_record ref = {0};\n+ while (1) {\n+ err = reftable_iterator_next_ref(&it, &ref);\n+ if (err > 0) {\n+ break;\n+ }\n+ if (err < 0) {\n+ ..error handling..\n+ }\n+ ..found..\n+ }\n+ reftable_iterator_destroy(&it);\n+ reftable_ref_record_release(&ref);\n+ */\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* returns the hash ID used in this table. */\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r);\n+\n+/* seek to logs for the given name, older than update_index. To seek to the\n+ start of the table, use name = \"\".\n+ */\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index);\n+\n+/* seek to newest log entry for given name. */\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* closes and deallocates a reader. */\n+void reftable_reader_free(struct reftable_reader *);\n+\n+/* return an iterator for the refs pointing to `oid`. */\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid);\n+\n+/* return the max_update_index for a table */\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r);\n+\n+/* return the min_update_index for a table */\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"410892","messageId":"57626bfe2d4ad48481a66f42efbe46f4b2daf549.1606419752.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v3.git.git.1606419752.gitgitgadget@gmail.com","subject":"[PATCH v3 15/16] git-prompt: prepare for reftable refs backend","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-11-26T19:42:30Z","receivedAt":"2020-11-26T19:42:57Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nIn our git-prompt script we strive to use Bash builtins wherever\npossible, because fork()-ing subshells for command substitutions and\nfork()+exec()-ing Git commands are expensive on some platforms. We\neven read and parse '.git/HEAD' using Bash builtins to get the name of\nthe current branch [1]. However, the upcoming reftable refs backend\nwon't use '.git/HEAD' at all, but will write an invalid refname as\nplaceholder for backwards compatibility instead, which will break our\ngit-prompt script.\n\nUpdate the git-prompt script to recognize the placeholder '.git/HEAD'\nwritten by the reftable backend (its content is specified in the\nreftable specs), and then fall back to use 'git symbolic-ref' to get\nthe name of the current branch.\n\n[1] 3a43c4b5bd (bash prompt: use bash builtins to find out current\n branch, 2011-03-31)\n\nSigned-off-by: SZEDER Gábor \n---\n contrib/completion/git-prompt.sh | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/completion/git-prompt.sh b/contrib/completion/git-prompt.sh\nindex 4640a1535d..2e5a5d8027 100644\n--- a/contrib/completion/git-prompt.sh\n+++ b/contrib/completion/git-prompt.sh\n@@ -478,10 +478,15 @@ __git_ps1 ()\n \t\t\tif ! __git_eread \"$g/HEAD\" head; then\n \t\t\t\treturn $exit\n \t\t\tfi\n-\t\t\t# is it a symbolic ref?\n \t\t\tb=\"${head#ref: }\"\n \t\t\tif [ \"$head\" = \"$b\" ]; then\n \t\t\t\tdetached=yes\n+\t\t\telif [ \"$b\" = \"refs/heads/.invalid\" ]; then\n+\t\t\t\t# Reftable\n+\t\t\t\tb=\"$(git symbolic-ref HEAD 2>/dev/null)\" ||\n+\t\t\t\tdetached=yes\n+\t\t\tfi\n+\t\t\tif [ \"$detached\" = yes ]; then\n \t\t\t\tb=\"$(\n \t\t\t\tcase \"${GIT_PS1_DESCRIBE_STYLE-}\" in\n \t\t\t\t(contains)\n-- \ngitgitgadget\n\n"},{"id":"410893","messageId":"a26fb180addb4753ceb1fe5fd64cc4844bbd896e.1606419752.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v3.git.git.1606419752.gitgitgadget@gmail.com","subject":"[PATCH v3 13/16] reftable: rest of library","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-11-26T19:42:28Z","receivedAt":"2020-11-26T19:42:57Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 4 +\n reftable/VERSION | 1 +\n reftable/dump.c | 212 ++++++\n reftable/merged.c | 366 ++++++++++\n reftable/merged.h | 35 +\n reftable/merged_test.c | 333 ++++++++++\n reftable/pq.c | 115 ++++\n reftable/pq.h | 32 +\n reftable/refname.c | 209 ++++++\n reftable/refname.h | 29 +\n reftable/refname_test.c | 100 +++\n reftable/reftable-generic.h | 49 ++\n reftable/reftable-merged.h | 68 ++\n reftable/reftable-stack.h | 116 ++++\n reftable/reftable.c | 98 +++\n reftable/stack.c | 1247 +++++++++++++++++++++++++++++++++++\n reftable/stack.h | 41 ++\n reftable/stack_test.c | 781 ++++++++++++++++++++++\n t/helper/test-reftable.c | 3 +\n 19 files changed, 3839 insertions(+)\n create mode 100644 reftable/VERSION\n create mode 100644 reftable/dump.c\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n create mode 100644 reftable/reftable-generic.h\n create mode 100644 reftable/reftable-merged.h\n create mode 100644 reftable/reftable-stack.h\n create mode 100644 reftable/reftable.c\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n\ndiff --git a/Makefile b/Makefile\nindex b20acdac0f..278c489da4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2394,10 +2394,14 @@ REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n+REFTABLE_OBJS += reftable/merged.o\n+REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/refname.o\n REFTABLE_OBJS += reftable/reftable.o\n+REFTABLE_OBJS += reftable/stack.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n REFTABLE_OBJS += reftable/zlib-compat.o\ndiff --git a/reftable/VERSION b/reftable/VERSION\nnew file mode 100644\nindex 0000000000..7553c395eb\n--- /dev/null\n+++ b/reftable/VERSION\n@@ -0,0 +1 @@\n+b8ec0f74c74cb6752eb2033ad8e755a9c19aad15 C: add missing header\ndiff --git a/reftable/dump.c b/reftable/dump.c\nnew file mode 100644\nindex 0000000000..00b444e8c9\n--- /dev/null\n+++ b/reftable/dump.c\n@@ -0,0 +1,212 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+\n+#include \"reftable.h\"\n+#include \"reftable-tests.h\"\n+\n+static uint32_t hash_id;\n+\n+static int dump_table(const char *tablename)\n+{\n+\tstruct reftable_block_source src = { 0 };\n+\tint err = reftable_block_source_from_file(&src, tablename);\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_ref_record ref = { 0 };\n+\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_reader *r = NULL;\n+\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\terr = reftable_new_reader(&r, &src, tablename);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\terr = reftable_reader_seek_ref(r, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_ref_record_print(&ref, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_clear(&ref);\n+\n+\terr = reftable_reader_seek_log(r, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_log_record_print(&log, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_clear(&log);\n+\n+\treftable_reader_free(r);\n+\treturn 0;\n+}\n+\n+static int compact_stack(const char *stackdir)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = {};\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_stack_compact_all(stack, NULL);\n+\tif (err < 0)\n+\t\tgoto done;\n+done:\n+\tif (stack != NULL) {\n+\t\treftable_stack_destroy(stack);\n+\t}\n+\treturn err;\n+}\n+\n+static int dump_stack(const char *stackdir)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = {};\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_ref_record ref = { 0 };\n+\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_merged_table *merged = NULL;\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tmerged = reftable_stack_merged_table(stack);\n+\n+\terr = reftable_merged_table_seek_ref(merged, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_ref_record_print(&ref, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_clear(&ref);\n+\n+\terr = reftable_merged_table_seek_log(merged, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_log_record_print(&log, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_clear(&log);\n+\n+\treftable_stack_destroy(stack);\n+\treturn 0;\n+}\n+\n+static void print_help(void)\n+{\n+\tprintf(\"usage: dump [-cst] arg\\n\\n\"\n+\t \"options: \\n\"\n+\t \" -c compact\\n\"\n+\t \" -t dump table\\n\"\n+\t \" -s dump stack\\n\"\n+\t \" -h this help\\n\"\n+\t \" -2 use SHA256\\n\"\n+\t \"\\n\");\n+}\n+\n+int reftable_dump_main(int argc, char *const *argv)\n+{\n+\tint err = 0;\n+\tint opt;\n+\tint opt_dump_table = 0;\n+\tint opt_dump_stack = 0;\n+\tint opt_compact = 0;\n+\tconst char *arg = NULL;\n+\twhile ((opt = getopt(argc, argv, \"2chts\")) != -1) {\n+\t\tswitch (opt) {\n+\t\tcase '2':\n+\t\t\thash_id = 0x73323536;\n+\t\t\tbreak;\n+\t\tcase 't':\n+\t\t\topt_dump_table = 1;\n+\t\t\tbreak;\n+\t\tcase 's':\n+\t\t\topt_dump_stack = 1;\n+\t\t\tbreak;\n+\t\tcase 'c':\n+\t\t\topt_compact = 1;\n+\t\t\tbreak;\n+\t\tcase '?':\n+\t\tcase 'h':\n+\t\t\tprint_help();\n+\t\t\treturn 2;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (argv[optind] == NULL) {\n+\t\tfprintf(stderr, \"need argument\\n\");\n+\t\tprint_help();\n+\t\treturn 2;\n+\t}\n+\n+\targ = argv[optind];\n+\n+\tif (opt_dump_table) {\n+\t\terr = dump_table(arg);\n+\t} else if (opt_dump_stack) {\n+\t\terr = dump_stack(arg);\n+\t} else if (opt_compact) {\n+\t\terr = compact_stack(arg);\n+\t}\n+\n+\tif (err < 0) {\n+\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv[0], arg,\n+\t\t\treftable_error_str(err));\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\ndiff --git a/reftable/merged.c b/reftable/merged.c\nnew file mode 100644\nindex 0000000000..ed02625bb5\n--- /dev/null\n+++ b/reftable/merged.c\n@@ -0,0 +1,366 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"constants.h\"\n+#include \"iter.h\"\n+#include \"pq.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-error.h\"\n+#include \"system.h\"\n+\n+static int merged_iter_init(struct merged_iter *mi)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\t\tint err = iterator_next(&mi->stack[i], &rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t\t\treftable_record_destroy(&rec);\n+\t\t} else {\n+\t\t\tstruct pq_entry e = {\n+\t\t\t\t.rec = rec,\n+\t\t\t\t.index = i,\n+\t\t\t};\n+\t\t\tmerged_iter_pqueue_add(&mi->pq, e);\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void merged_iter_close(void *p)\n+{\n+\tstruct merged_iter *mi = (struct merged_iter *)p;\n+\tint i = 0;\n+\tmerged_iter_pqueue_release(&mi->pq);\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t}\n+\treftable_free(mi->stack);\n+}\n+\n+static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,\n+\t\t\t\t\t size_t idx)\n+{\n+\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\tstruct pq_entry e = {\n+\t\t.rec = rec,\n+\t\t.index = idx,\n+\t};\n+\tint err = iterator_next(&mi->stack[idx], &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (err > 0) {\n+\t\treftable_iterator_destroy(&mi->stack[idx]);\n+\t\treftable_record_destroy(&rec);\n+\t\treturn 0;\n+\t}\n+\n+\tmerged_iter_pqueue_add(&mi->pq, e);\n+\treturn 0;\n+}\n+\n+static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)\n+{\n+\tif (iterator_is_null(&mi->stack[idx]))\n+\t\treturn 0;\n+\treturn merged_iter_advance_nonnull_subiter(mi, idx);\n+}\n+\n+static int merged_iter_next_entry(struct merged_iter *mi,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct strbuf entry_key = STRBUF_INIT;\n+\tstruct pq_entry entry = { 0 };\n+\tint err = 0;\n+\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\tentry = merged_iter_pqueue_remove(&mi->pq);\n+\terr = merged_iter_advance_subiter(mi, entry.index);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\t/*\n+\t One can also use reftable as datacenter-local storage, where the ref\n+\t database is maintained in globally consistent database (eg.\n+\t CockroachDB or Spanner). In this scenario, replication delays together\n+\t with compaction may cause newer tables to contain older entries. In\n+\t such a deployment, the loop below must be changed to collect all\n+\t entries for the same key, and return new the newest one.\n+\t*/\n+\treftable_record_key(&entry.rec, &entry_key);\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq)) {\n+\t\tstruct pq_entry top = merged_iter_pqueue_top(mi->pq);\n+\t\tstruct strbuf k = STRBUF_INIT;\n+\t\tint err = 0, cmp = 0;\n+\n+\t\treftable_record_key(&top.rec, &k);\n+\n+\t\tcmp = strbuf_cmp(&k, &entry_key);\n+\t\tstrbuf_release(&k);\n+\n+\t\tif (cmp > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\t\terr = merged_iter_advance_subiter(mi, top.index);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_record_destroy(&top.rec);\n+\t}\n+\n+\treftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));\n+\treftable_record_destroy(&entry.rec);\n+\tstrbuf_release(&entry_key);\n+\treturn 0;\n+}\n+\n+static int merged_iter_next(struct merged_iter *mi, struct reftable_record *rec)\n+{\n+\twhile (1) {\n+\t\tint err = merged_iter_next_entry(mi, rec);\n+\t\tif (err == 0 && mi->suppress_deletions &&\n+\t\t reftable_record_is_deletion(rec)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\treturn err;\n+\t}\n+}\n+\n+static int merged_iter_next_void(void *p, struct reftable_record *rec)\n+{\n+\tstruct merged_iter *mi = (struct merged_iter *)p;\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\treturn merged_iter_next(mi, rec);\n+}\n+\n+static struct reftable_iterator_vtable merged_iter_vtable = {\n+\t.next = &merged_iter_next_void,\n+\t.close = &merged_iter_close,\n+};\n+\n+static void iterator_from_merged_iter(struct reftable_iterator *it,\n+\t\t\t\t struct merged_iter *mi)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = mi;\n+\tit->ops = &merged_iter_vtable;\n+}\n+\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id)\n+{\n+\tstruct reftable_merged_table *m = NULL;\n+\tuint64_t last_max = 0;\n+\tuint64_t first_min = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t min = reftable_table_min_update_index(&stack[i]);\n+\t\tuint64_t max = reftable_table_max_update_index(&stack[i]);\n+\n+\t\tif (reftable_table_hash_id(&stack[i]) != hash_id) {\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\t\t}\n+\t\tif (i == 0 || min < first_min) {\n+\t\t\tfirst_min = min;\n+\t\t}\n+\t\tif (i == 0 || max > last_max) {\n+\t\t\tlast_max = max;\n+\t\t}\n+\t}\n+\n+\tm = (struct reftable_merged_table *)reftable_calloc(\n+\t\tsizeof(struct reftable_merged_table));\n+\tm->stack = stack;\n+\tm->stack_len = n;\n+\tm->min = first_min;\n+\tm->max = last_max;\n+\tm->hash_id = hash_id;\n+\t*dest = m;\n+\treturn 0;\n+}\n+\n+/* clears the list of subtable, without affecting the readers themselves. */\n+void merged_table_release(struct reftable_merged_table *mt)\n+{\n+\tFREE_AND_NULL(mt->stack);\n+\tmt->stack_len = 0;\n+}\n+\n+void reftable_merged_table_free(struct reftable_merged_table *mt)\n+{\n+\tif (mt == NULL) {\n+\t\treturn;\n+\t}\n+\tmerged_table_release(mt);\n+\treftable_free(mt);\n+}\n+\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->max;\n+}\n+\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->min;\n+}\n+\n+static int reftable_table_seek_record(struct reftable_table *tab,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn tab->ops->seek_record(tab->table_arg, it, rec);\n+}\n+\n+static int merged_table_seek_record(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_iterator *iters = reftable_calloc(\n+\t\tsizeof(struct reftable_iterator) * mt->stack_len);\n+\tstruct merged_iter merged = {\n+\t\t.stack = iters,\n+\t\t.typ = reftable_record_type(rec),\n+\t\t.hash_id = mt->hash_id,\n+\t\t.suppress_deletions = mt->suppress_deletions,\n+\t};\n+\tint n = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < mt->stack_len && err == 0; i++) {\n+\t\tint e = reftable_table_seek_record(&mt->stack[i], &iters[n],\n+\t\t\t\t\t\t rec);\n+\t\tif (e < 0) {\n+\t\t\terr = e;\n+\t\t}\n+\t\tif (e == 0) {\n+\t\t\tn++;\n+\t\t}\n+\t}\n+\tif (err < 0) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < n; i++) {\n+\t\t\treftable_iterator_destroy(&iters[i]);\n+\t\t}\n+\t\treftable_free(iters);\n+\t\treturn err;\n+\t}\n+\n+\tmerged.stack_len = n;\n+\terr = merged_iter_init(&merged);\n+\tif (err < 0) {\n+\t\tmerged_iter_close(&merged);\n+\t\treturn err;\n+\t} else {\n+\t\tstruct merged_iter *p =\n+\t\t\treftable_malloc(sizeof(struct merged_iter));\n+\t\t*p = merged;\n+\t\titerator_from_merged_iter(it, p);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_merged_table_seek_log_at(mt, it, name, max);\n+}\n+\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\n+{\n+\treturn mt->hash_id;\n+}\n+\n+static int reftable_merged_table_seek_void(void *tab,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn merged_table_seek_record((struct reftable_merged_table *)tab, it,\n+\t\t\t\t\trec);\n+}\n+\n+static uint32_t reftable_merged_table_hash_id_void(void *tab)\n+{\n+\treturn reftable_merged_table_hash_id(\n+\t\t(struct reftable_merged_table *)tab);\n+}\n+\n+static uint64_t reftable_merged_table_min_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_min_update_index(\n+\t\t(struct reftable_merged_table *)tab);\n+}\n+\n+static uint64_t reftable_merged_table_max_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_max_update_index(\n+\t\t(struct reftable_merged_table *)tab);\n+}\n+\n+static struct reftable_table_vtable merged_table_vtable = {\n+\t.seek_record = reftable_merged_table_seek_void,\n+\t.hash_id = reftable_merged_table_hash_id_void,\n+\t.min_update_index = reftable_merged_table_min_update_index_void,\n+\t.max_update_index = reftable_merged_table_max_update_index_void,\n+};\n+\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *merged)\n+{\n+\tassert(tab->ops == NULL);\n+\ttab->ops = &merged_table_vtable;\n+\ttab->table_arg = merged;\n+}\ndiff --git a/reftable/merged.h b/reftable/merged.h\nnew file mode 100644\nindex 0000000000..8c4d4d58d7\n--- /dev/null\n+++ b/reftable/merged.h\n@@ -0,0 +1,35 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef MERGED_H\n+#define MERGED_H\n+\n+#include \"pq.h\"\n+\n+struct reftable_merged_table {\n+\tstruct reftable_table *stack;\n+\tsize_t stack_len;\n+\tuint32_t hash_id;\n+\tint suppress_deletions;\n+\n+\tuint64_t min;\n+\tuint64_t max;\n+};\n+\n+struct merged_iter {\n+\tstruct reftable_iterator *stack;\n+\tuint32_t hash_id;\n+\tsize_t stack_len;\n+\tuint8_t typ;\n+\tint suppress_deletions;\n+\tstruct merged_iter_pqueue pq;\n+};\n+\n+void merged_table_release(struct reftable_merged_table *mt);\n+\n+#endif\ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nnew file mode 100644\nindex 0000000000..fbb2e5ac10\n--- /dev/null\n+++ b/reftable/merged_test.c\n@@ -0,0 +1,333 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"pq.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-generic.h\"\n+#include \"reftable-stack.h\"\n+\n+static void test_pq(void)\n+{\n+\tchar *names[54] = { NULL };\n+\tint N = ARRAY_SIZE(names) - 1;\n+\n+\tstruct merged_iter_pqueue pq = { NULL };\n+\tconst char *last = NULL;\n+\n+\tint i = 0;\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tsnprintf(name, sizeof(name), \"%02d\", i);\n+\t\tnames[i] = xstrdup(name);\n+\t}\n+\n+\ti = 1;\n+\tdo {\n+\t\tstruct reftable_record rec =\n+\t\t\treftable_new_record(BLOCK_TYPE_REF);\n+\t\tstruct pq_entry e = { 0 };\n+\n+\t\treftable_record_as_ref(&rec)->refname = names[i];\n+\t\te.rec = rec;\n+\t\tmerged_iter_pqueue_add(&pq, e);\n+\t\tmerged_iter_pqueue_check(pq);\n+\t\ti = (i * 7) % N;\n+\t} while (i != 1);\n+\n+\twhile (!merged_iter_pqueue_is_empty(pq)) {\n+\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct reftable_ref_record *ref =\n+\t\t\treftable_record_as_ref(&e.rec);\n+\n+\t\tmerged_iter_pqueue_check(pq);\n+\n+\t\tif (last != NULL) {\n+\t\t\tassert(strcmp(last, ref->refname) < 0);\n+\t\t}\n+\t\tlast = ref->refname;\n+\t\tref->refname = NULL;\n+\t\treftable_free(ref);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\n+\tmerged_iter_pqueue_release(&pq);\n+}\n+\n+static void write_test_table(struct strbuf *buf,\n+\t\t\t struct reftable_ref_record refs[], int n)\n+{\n+\tint min = 0xffffffff;\n+\tint max = 0;\n+\tint i = 0;\n+\tint err;\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_writer *w = NULL;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t ui = refs[i].update_index;\n+\t\tif (ui > max) {\n+\t\t\tmax = ui;\n+\t\t}\n+\t\tif (ui < min) {\n+\t\t\tmin = ui;\n+\t\t}\n+\t}\n+\n+\tw = reftable_new_writer(&strbuf_add_void, buf, &opts);\n+\treftable_writer_set_limits(w, min, max);\n+\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t before = refs[i].update_index;\n+\t\tint n = reftable_writer_add_ref(w, &refs[i]);\n+\t\tassert(n == 0);\n+\t\tassert(before == refs[i].update_index);\n+\t}\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\n+\treftable_writer_free(w);\n+}\n+\n+static struct reftable_merged_table *\n+merged_table_from_records(struct reftable_ref_record **refs,\n+\t\t\t struct reftable_block_source **source,\n+\t\t\t struct reftable_reader ***readers, int *sizes,\n+\t\t\t struct strbuf *buf, int n)\n+{\n+\tint i = 0;\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err;\n+\tstruct reftable_table *tabs =\n+\t\treftable_calloc(n * sizeof(struct reftable_table));\n+\t*readers = reftable_calloc(n * sizeof(struct reftable_reader *));\n+\t*source = reftable_calloc(n * sizeof(**source));\n+\tfor (i = 0; i < n; i++) {\n+\t\twrite_test_table(&buf[i], refs[i], sizes[i]);\n+\t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n+\n+\t\terr = reftable_new_reader(&(*readers)[i], &(*source)[i],\n+\t\t\t\t\t \"name\");\n+\t\tEXPECT_ERR(err);\n+\t\treftable_table_from_reader(&tabs[i], (*readers)[i]);\n+\t}\n+\n+\terr = reftable_new_merged_table(&mt, tabs, n, SHA1_ID);\n+\tEXPECT_ERR(err);\n+\treturn mt;\n+}\n+\n+static void readers_destroy(struct reftable_reader **readers, size_t n)\n+{\n+\tint i = 0;\n+\tfor (; i < n; i++)\n+\t\treftable_reader_free(readers[i]);\n+\treftable_free(readers);\n+}\n+\n+static void test_merged_between(void)\n+{\n+\tuint8_t hash1[SHA1_SIZE] = { 1, 2, 3, 0 };\n+\n+\tstruct reftable_ref_record r1[] = { {\n+\t\t.refname = \"b\",\n+\t\t.update_index = 1,\n+\t\t.value = hash1,\n+\t} };\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t} };\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2 };\n+\tint sizes[] = { 1, 1 };\n+\tstruct strbuf bufs[2] = { STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 2);\n+\tint i;\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT_ERR(err);\n+\tEXPECT(ref.update_index == 2);\n+\treftable_ref_record_release(&ref);\n+\treftable_iterator_destroy(&it);\n+\treaders_destroy(readers, 2);\n+\treftable_merged_table_free(mt);\n+\tfor (i = 0; i < ARRAY_SIZE(bufs); i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treftable_free(bs);\n+}\n+\n+static void test_merged(void)\n+{\n+\tuint8_t hash1[SHA1_SIZE] = { 1 };\n+\tuint8_t hash2[SHA1_SIZE] = { 2 };\n+\tstruct reftable_ref_record r1[] = { {\n+\t\t\t\t\t\t .refname = \"a\",\n+\t\t\t\t\t\t .update_index = 1,\n+\t\t\t\t\t\t .value = hash1,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .refname = \"b\",\n+\t\t\t\t\t\t .update_index = 1,\n+\t\t\t\t\t\t .value = hash1,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .refname = \"c\",\n+\t\t\t\t\t\t .update_index = 1,\n+\t\t\t\t\t\t .value = hash1,\n+\t\t\t\t\t } };\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t} };\n+\tstruct reftable_ref_record r3[] = {\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value = hash2,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"d\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value = hash1,\n+\t\t},\n+\t};\n+\n+\tstruct reftable_ref_record want[] = {\n+\t\tr2[0],\n+\t\tr1[1],\n+\t\tr3[0],\n+\t\tr3[1],\n+\t};\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2, r3 };\n+\tint sizes[3] = { 3, 1, 2 };\n+\tstruct strbuf bufs[3] = { STRBUF_INIT, STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 3);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tstruct reftable_ref_record *out = NULL;\n+\tsize_t len = 0;\n+\tsize_t cap = 0;\n+\tint i = 0;\n+\n+\tEXPECT_ERR(err);\n+\twhile (len < 100) { /* cap loops/recursion. */\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tout = reftable_realloc(\n+\t\t\t\tout, sizeof(struct reftable_ref_record) * cap);\n+\t\t}\n+\t\tout[len++] = ref;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\tassert(ARRAY_SIZE(want) == len);\n+\tfor (i = 0; i < len; i++) {\n+\t\tassert(reftable_ref_record_equal(&want[i], &out[i], SHA1_SIZE));\n+\t}\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_release(&out[i]);\n+\t}\n+\treftable_free(out);\n+\n+\tfor (i = 0; i < 3; i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treaders_destroy(readers, 3);\n+\treftable_merged_table_free(mt);\n+\treftable_free(bs);\n+}\n+\n+static void test_default_write_opts(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"master\",\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_table *tab = reftable_calloc(sizeof(*tab) * 1);\n+\tuint32_t hash_id;\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_merged_table *merged = NULL;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\thash_id = reftable_reader_hash_id(rd);\n+\tassert(hash_id == SHA1_ID);\n+\n+\treftable_table_from_reader(&tab[0], rd);\n+\terr = reftable_new_merged_table(&merged, tab, 1, SHA1_ID);\n+\tEXPECT_ERR(err);\n+\n+\treftable_reader_free(rd);\n+\treftable_merged_table_free(merged);\n+\tstrbuf_release(&buf);\n+}\n+\n+/* XXX test refs_for(oid) */\n+\n+int merged_test_main(int argc, const char *argv[])\n+{\n+\ttest_merged_between();\n+\ttest_pq();\n+\ttest_merged();\n+\ttest_default_write_opts();\n+\treturn 0;\n+}\ndiff --git a/reftable/pq.c b/reftable/pq.c\nnew file mode 100644\nindex 0000000000..8918d158e2\n--- /dev/null\n+++ b/reftable/pq.c\n@@ -0,0 +1,115 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"pq.h\"\n+\n+#include \"reftable-record.h\"\n+#include \"system.h\"\n+#include \"basics.h\"\n+\n+static int pq_less(struct pq_entry a, struct pq_entry b)\n+{\n+\tstruct strbuf ak = STRBUF_INIT;\n+\tstruct strbuf bk = STRBUF_INIT;\n+\tint cmp = 0;\n+\treftable_record_key(&a.rec, &ak);\n+\treftable_record_key(&b.rec, &bk);\n+\n+\tcmp = strbuf_cmp(&ak, &bk);\n+\n+\tstrbuf_release(&ak);\n+\tstrbuf_release(&bk);\n+\n+\tif (cmp == 0)\n+\t\treturn a.index > b.index;\n+\n+\treturn cmp < 0;\n+}\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.heap[0];\n+}\n+\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.len == 0;\n+}\n+\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq)\n+{\n+\tint i = 0;\n+\tfor (i = 1; i < pq.len; i++) {\n+\t\tint parent = (i - 1) / 2;\n+\n+\t\tassert(pq_less(pq.heap[parent], pq.heap[i]));\n+\t}\n+}\n+\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tstruct pq_entry e = pq->heap[0];\n+\tpq->heap[0] = pq->heap[pq->len - 1];\n+\tpq->len--;\n+\n+\ti = 0;\n+\twhile (i < pq->len) {\n+\t\tint min = i;\n+\t\tint j = 2 * i + 1;\n+\t\tint k = 2 * i + 2;\n+\t\tif (j < pq->len && pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tmin = j;\n+\t\t}\n+\t\tif (k < pq->len && pq_less(pq->heap[k], pq->heap[min])) {\n+\t\t\tmin = k;\n+\t\t}\n+\n+\t\tif (min == i) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[i], pq->heap[min]);\n+\t\ti = min;\n+\t}\n+\n+\treturn e;\n+}\n+\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)\n+{\n+\tint i = 0;\n+\tif (pq->len == pq->cap) {\n+\t\tpq->cap = 2 * pq->cap + 1;\n+\t\tpq->heap = reftable_realloc(pq->heap,\n+\t\t\t\t\t pq->cap * sizeof(struct pq_entry));\n+\t}\n+\n+\tpq->heap[pq->len++] = e;\n+\ti = pq->len - 1;\n+\twhile (i > 0) {\n+\t\tint j = (i - 1) / 2;\n+\t\tif (pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[j], pq->heap[i]);\n+\n+\t\ti = j;\n+\t}\n+}\n+\n+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < pq->len; i++) {\n+\t\treftable_record_destroy(&pq->heap[i].rec);\n+\t}\n+\tFREE_AND_NULL(pq->heap);\n+\tpq->len = pq->cap = 0;\n+}\ndiff --git a/reftable/pq.h b/reftable/pq.h\nnew file mode 100644\nindex 0000000000..385d2fb139\n--- /dev/null\n+++ b/reftable/pq.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef PQ_H\n+#define PQ_H\n+\n+#include \"record.h\"\n+\n+struct pq_entry {\n+\tint index;\n+\tstruct reftable_record rec;\n+};\n+\n+struct merged_iter_pqueue {\n+\tstruct pq_entry *heap;\n+\tsize_t len;\n+\tsize_t cap;\n+};\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq);\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq);\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq);\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e);\n+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n+\n+#endif\ndiff --git a/reftable/refname.c b/reftable/refname.c\nnew file mode 100644\nindex 0000000000..0f4eb3b292\n--- /dev/null\n+++ b/reftable/refname.c\n@@ -0,0 +1,209 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"reftable-error.h\"\n+#include \"basics.h\"\n+#include \"refname.h\"\n+#include \"reftable-iterator.h\"\n+\n+struct find_arg {\n+\tchar **names;\n+\tconst char *want;\n+};\n+\n+static int find_name(size_t k, void *arg)\n+{\n+\tstruct find_arg *f_arg = (struct find_arg *)arg;\n+\treturn strcmp(f_arg->names[k], f_arg->want) >= 0;\n+}\n+\n+static int modification_has_ref(struct modification *mod, const char *name)\n+{\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len && !strcmp(mod->add[idx], name)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tif (mod->del_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->del,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\tif (idx < mod->del_len && !strcmp(mod->del[idx], name)) {\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\terr = reftable_table_read_ref(&mod->tab, name, &ref);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+static void modification_release(struct modification *mod)\n+{\n+\t/* don't delete the strings themselves; they're owned by ref records.\n+\t */\n+\tFREE_AND_NULL(mod->add);\n+\tFREE_AND_NULL(mod->del);\n+\tmod->add_len = 0;\n+\tmod->del_len = 0;\n+}\n+\n+static int modification_has_ref_with_prefix(struct modification *mod,\n+\t\t\t\t\t const char *prefix)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = prefix,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len &&\n+\t\t !strncmp(prefix, mod->add[idx], strlen(prefix)))\n+\t\t\tgoto done;\n+\t}\n+\terr = reftable_table_seek_ref(&mod->tab, &it, prefix);\n+\tif (err)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\n+\t\tif (mod->del_len > 0) {\n+\t\t\tstruct find_arg arg = {\n+\t\t\t\t.names = mod->del,\n+\t\t\t\t.want = ref.refname,\n+\t\t\t};\n+\t\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\t\tif (idx < mod->del_len &&\n+\t\t\t !strcmp(ref.refname, mod->del[idx])) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (strncmp(ref.refname, prefix, strlen(prefix))) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_ref_record_release(&ref);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int validate_refname(const char *name)\n+{\n+\twhile (1) {\n+\t\tchar *next = strchr(name, '/');\n+\t\tif (!*name) {\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\t}\n+\t\tif (!next) {\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (next - name == 0 || (next - name == 1 && *name == '.') ||\n+\t\t (next - name == 2 && name[0] == '.' && name[1] == '.'))\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\tname = next + 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz)\n+{\n+\tstruct modification mod = {\n+\t\t.tab = tab,\n+\t\t.add = reftable_calloc(sizeof(char *) * sz),\n+\t\t.del = reftable_calloc(sizeof(char *) * sz),\n+\t};\n+\tint i = 0;\n+\tint err = 0;\n+\tfor (; i < sz; i++) {\n+\t\tif (reftable_ref_record_is_deletion(&recs[i])) {\n+\t\t\tmod.del[mod.del_len++] = recs[i].refname;\n+\t\t} else {\n+\t\t\tmod.add[mod.add_len++] = recs[i].refname;\n+\t\t}\n+\t}\n+\n+\terr = modification_validate(&mod);\n+\tmodification_release(&mod);\n+\treturn err;\n+}\n+\n+static void strbuf_trim_component(struct strbuf *sl)\n+{\n+\twhile (sl->len > 0) {\n+\t\tint is_slash = (sl->buf[sl->len - 1] == '/');\n+\t\tstrbuf_setlen(sl, sl->len - 1);\n+\t\tif (is_slash)\n+\t\t\tbreak;\n+\t}\n+}\n+\n+int modification_validate(struct modification *mod)\n+{\n+\tstruct strbuf slashed = STRBUF_INIT;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (; i < mod->add_len; i++) {\n+\t\terr = validate_refname(mod->add[i]);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\tstrbuf_addstr(&slashed, \"/\");\n+\n+\t\terr = modification_has_ref_with_prefix(mod, slashed.buf);\n+\t\tif (err == 0) {\n+\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\twhile (slashed.len) {\n+\t\t\tstrbuf_trim_component(&slashed);\n+\t\t\terr = modification_has_ref(mod, slashed.buf);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\terr = 0;\n+done:\n+\tstrbuf_release(&slashed);\n+\treturn err;\n+}\ndiff --git a/reftable/refname.h b/reftable/refname.h\nnew file mode 100644\nindex 0000000000..a24b40fcb4\n--- /dev/null\n+++ b/reftable/refname.h\n@@ -0,0 +1,29 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+#ifndef REFNAME_H\n+#define REFNAME_H\n+\n+#include \"reftable-record.h\"\n+#include \"reftable-generic.h\"\n+\n+struct modification {\n+\tstruct reftable_table tab;\n+\n+\tchar **add;\n+\tsize_t add_len;\n+\n+\tchar **del;\n+\tsize_t del_len;\n+};\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz);\n+\n+int modification_validate(struct modification *mod);\n+\n+#endif\ndiff --git a/reftable/refname_test.c b/reftable/refname_test.c\nnew file mode 100644\nindex 0000000000..af04bc9a59\n--- /dev/null\n+++ b/reftable/refname_test.c\n@@ -0,0 +1,100 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"refname.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-writer.h\"\n+#include \"system.h\"\n+\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct testcase {\n+\tchar *add;\n+\tchar *del;\n+\tint error_code;\n+};\n+\n+static void test_conflict(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"a/b\",\n+\t\t.target = \"destination\", /* make sure it's not a symref. */\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tint i;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct testcase cases[] = {\n+\t\t{ \"a/b/c\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"b\", NULL, 0 },\n+\t\t{ \"a\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"a\", \"a/b\", 0 },\n+\n+\t\t{ \"p/\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p//q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/./q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/../q\", NULL, REFTABLE_REFNAME_ERROR },\n+\n+\t\t{ \"a/b/c\", \"a/b\", 0 },\n+\t\t{ NULL, \"a//b\", 0 },\n+\t};\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\treftable_table_from_reader(&tab, rd);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct modification mod = {\n+\t\t\t.tab = tab,\n+\t\t};\n+\n+\t\tif (cases[i].add != NULL) {\n+\t\t\tmod.add = &cases[i].add;\n+\t\t\tmod.add_len = 1;\n+\t\t}\n+\t\tif (cases[i].del != NULL) {\n+\t\t\tmod.del = &cases[i].del;\n+\t\t\tmod.del_len = 1;\n+\t\t}\n+\n+\t\terr = modification_validate(&mod);\n+\t\tEXPECT(err == cases[i].error_code);\n+\t}\n+\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+int refname_test_main(int argc, const char *argv[])\n+{\n+\ttest_conflict();\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-generic.h b/reftable/reftable-generic.h\nnew file mode 100644\nindex 0000000000..964d25f040\n--- /dev/null\n+++ b/reftable/reftable-generic.h\n@@ -0,0 +1,49 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_GENERIC_H\n+#define REFTABLE_GENERIC_H\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+#include \"reftable-merged.h\"\n+\n+/*\n+ Provides a unified API for reading tables, either merged tables, or single\n+ readers.\n+*/\n+struct reftable_table {\n+\tstruct reftable_table_vtable *ops;\n+\tvoid *table_arg;\n+};\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader);\n+\n+/* returns the hash ID from a generic reftable_table */\n+uint32_t reftable_table_hash_id(struct reftable_table *tab);\n+\n+/* create a generic table from reftable_merged_table */\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *table);\n+\n+/* returns the max update_index covered by this table. */\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab);\n+\n+/* returns the min update_index covered by this table. */\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0\n+ for success, and 1 if ref not found. */\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+#endif\ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nnew file mode 100644\nindex 0000000000..5ee16aa02a\n--- /dev/null\n+++ b/reftable/reftable-merged.h\n@@ -0,0 +1,68 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_MERGED_H\n+#define REFTABLE_MERGED_H\n+\n+#include \"reftable-iterator.h\"\n+\n+/*\n+ Merged tables\n+\n+ A ref database kept in a sequence of table files. The merged_table presents a\n+ unified view to reading (seeking, iterating) a sequence of immutable tables.\n+\n+ The merged tables are on purpose kept disconnected from their actual storage\n+ (eg. files on disk), because it is useful to merge tables aren't files. For\n+ example, the per-workspace and global ref namespace can be implemented as a\n+ merged table of two stacks of file-backed reftables.\n+*/\n+\n+/* A merged table is implements seeking/iterating over a stack of tables. */\n+struct reftable_merged_table;\n+\n+/* A generic reftable; see below. */\n+struct reftable_table;\n+\n+/* reftable_new_merged_table creates a new merged table. It takes ownership of\n+ the stack array.\n+*/\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id);\n+\n+/* returns an iterator positioned just before 'name' */\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns an iterator for log entry, at given update_index */\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index);\n+\n+/* like reftable_merged_table_seek_log_at but look for the newest entry. */\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns the max update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt);\n+\n+/* returns the min update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt);\n+\n+/* releases memory for the merged_table */\n+void reftable_merged_table_free(struct reftable_merged_table *m);\n+\n+/* return the hash ID of the merged table. */\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m);\n+\n+#endif\ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nnew file mode 100644\nindex 0000000000..140a28b790\n--- /dev/null\n+++ b/reftable/reftable-stack.h\n@@ -0,0 +1,116 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_STACK_H\n+#define REFTABLE_STACK_H\n+\n+#include \"reftable-writer.h\"\n+\n+/*\n+ The stack presents an interface to a mutable sequence of reftables.\n+\n+ A stack can be mutated by pushing a table to the top of the stack.\n+\n+ The reftable_stack automatically compacts files on disk to ensure good\n+ amortized performance.\n+*/\n+struct reftable_stack;\n+\n+/* open a new reftable stack. The tables along with the table list will be\n+ stored in 'dir'. Typically, this should be .git/reftables.\n+*/\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config);\n+\n+/* returns the update_index at which a next table should be written. */\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\n+\n+/* holds a transaction to add tables at the top of a stack. */\n+struct reftable_addition;\n+\n+/*\n+ returns a new transaction to add reftables to the given stack. As a side\n+ effect, the ref database is locked.\n+*/\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st);\n+\n+/* Adds a reftable to transaction. */\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+\n+/* Commits the transaction, releasing the lock. */\n+int reftable_addition_commit(struct reftable_addition *add);\n+\n+/* Release all non-committed data from the transaction, and deallocate the\n+ transaction. Releases the lock if held. */\n+void reftable_addition_destroy(struct reftable_addition *add);\n+\n+/* add a new table to the stack. The write_table function must call\n+ reftable_writer_set_limits, add refs and return an error value. */\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *write_arg),\n+\t\t void *write_arg);\n+\n+/* returns the merged_table for seeking. This table is valid until the\n+ next write or reload, and should not be closed or deleted.\n+*/\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st);\n+\n+/* frees all resources associated with the stack. */\n+void reftable_stack_destroy(struct reftable_stack *st);\n+\n+/* Reloads the stack if necessary. This is very cheap to run if the stack was up\n+ * to date */\n+int reftable_stack_reload(struct reftable_stack *st);\n+\n+/* Policy for expiring reflog entries. */\n+struct reftable_log_expiry_config {\n+\t/* Drop entries older than this timestamp */\n+\tuint64_t time;\n+\n+\t/* Drop older entries */\n+\tuint64_t min_update_index;\n+};\n+\n+/* compacts all reftables into a giant table. Expire reflog entries if config is\n+ * non-NULL */\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config);\n+\n+/* heuristically compact unbalanced table stack. */\n+int reftable_stack_auto_compact(struct reftable_stack *st);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0\n+ for success, and 1 if ref not found. */\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* convenience function to read a single log. Returns < 0 for error, 0\n+ for success, and 1 if ref not found. */\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* statistics on past compactions. */\n+struct reftable_compaction_stats {\n+\tuint64_t bytes; /* total number of bytes written */\n+\tuint64_t entries_written; /* total number of entries written, including\n+\t\t\t\t failures. */\n+\tint attempts; /* how often we tried to compact */\n+\tint failures; /* failures happen on concurrent updates */\n+};\n+\n+/* return statistics for compaction up till now. */\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st);\n+\n+#endif\ndiff --git a/reftable/reftable.c b/reftable/reftable.c\nnew file mode 100644\nindex 0000000000..dc4fd03d5b\n--- /dev/null\n+++ b/reftable/reftable.c\n@@ -0,0 +1,98 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"record.h\"\n+#include \"reader.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+static int reftable_reader_seek_void(void *tab, struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn reader_seek((struct reftable_reader *)tab, it, rec);\n+}\n+\n+static uint32_t reftable_reader_hash_id_void(void *tab)\n+{\n+\treturn reftable_reader_hash_id((struct reftable_reader *)tab);\n+}\n+\n+static uint64_t reftable_reader_min_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_min_update_index((struct reftable_reader *)tab);\n+}\n+\n+static uint64_t reftable_reader_max_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_max_update_index((struct reftable_reader *)tab);\n+}\n+\n+static struct reftable_table_vtable reader_vtable = {\n+\t.seek_record = reftable_reader_seek_void,\n+\t.hash_id = reftable_reader_hash_id_void,\n+\t.min_update_index = reftable_reader_min_update_index_void,\n+\t.max_update_index = reftable_reader_max_update_index_void,\n+};\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader)\n+{\n+\tassert(tab->ops == NULL);\n+\ttab->ops = &reader_vtable;\n+\ttab->table_arg = reader;\n+}\n+\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_table_seek_ref(tab, &it, name);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_ref(&it, ref);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref->refname, name) ||\n+\t reftable_ref_record_is_deletion(ref)) {\n+\t\treftable_ref_record_release(ref);\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->max_update_index(tab->table_arg);\n+}\n+\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->min_update_index(tab->table_arg);\n+}\n+\n+uint32_t reftable_table_hash_id(struct reftable_table *tab)\n+{\n+\treturn tab->ops->hash_id(tab->table_arg);\n+}\ndiff --git a/reftable/stack.c b/reftable/stack.c\nnew file mode 100644\nindex 0000000000..1d632937d7\n--- /dev/null\n+++ b/reftable/stack.c\n@@ -0,0 +1,1247 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+#include \"merged.h\"\n+#include \"reader.h\"\n+#include \"refname.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-record.h\"\n+#include \"writer.h\"\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config);\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name);\n+static void reftable_addition_close(struct reftable_addition *add);\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open);\n+\n+static int reftable_fd_write(void *arg, const void *data, size_t sz)\n+{\n+\tint *fdp = (int *)arg;\n+\treturn write(*fdp, data, sz);\n+}\n+\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config)\n+{\n+\tstruct reftable_stack *p =\n+\t\treftable_calloc(sizeof(struct reftable_stack));\n+\tstruct strbuf list_file_name = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tif (config.hash_id == 0) {\n+\t\tconfig.hash_id = SHA1_ID;\n+\t}\n+\n+\t*dest = NULL;\n+\n+\tstrbuf_reset(&list_file_name);\n+\tstrbuf_addstr(&list_file_name, dir);\n+\tstrbuf_addstr(&list_file_name, \"/tables.list\");\n+\n+\tp->list_file = strbuf_detach(&list_file_name, NULL);\n+\tp->reftable_dir = xstrdup(dir);\n+\tp->config = config;\n+\n+\terr = reftable_stack_reload_maybe_reuse(p, 1);\n+\tif (err < 0) {\n+\t\treftable_stack_destroy(p);\n+\t} else {\n+\t\t*dest = p;\n+\t}\n+\treturn err;\n+}\n+\n+static int fd_read_lines(int fd, char ***namesp)\n+{\n+\toff_t size = lseek(fd, 0, SEEK_END);\n+\tchar *buf = NULL;\n+\tint err = 0;\n+\tif (size < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\terr = lseek(fd, 0, SEEK_SET);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tbuf = reftable_malloc(size + 1);\n+\tif (read(fd, buf, size) != size) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\tbuf[size] = 0;\n+\n+\tparse_names(buf, size, namesp);\n+\n+done:\n+\treftable_free(buf);\n+\treturn err;\n+}\n+\n+int read_lines(const char *filename, char ***namesp)\n+{\n+\tint fd = open(filename, O_RDONLY, 0644);\n+\tint err = 0;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\t*namesp = reftable_calloc(sizeof(char *));\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\treturn REFTABLE_IO_ERROR;\n+\t}\n+\terr = fd_read_lines(fd, namesp);\n+\tclose(fd);\n+\treturn err;\n+}\n+\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st)\n+{\n+\treturn st->merged;\n+}\n+\n+/* Close and free the stack */\n+void reftable_stack_destroy(struct reftable_stack *st)\n+{\n+\tif (st->merged != NULL) {\n+\t\treftable_merged_table_free(st->merged);\n+\t\tst->merged = NULL;\n+\t}\n+\n+\tif (st->readers != NULL) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < st->readers_len; i++) {\n+\t\t\treftable_reader_free(st->readers[i]);\n+\t\t}\n+\t\tst->readers_len = 0;\n+\t\tFREE_AND_NULL(st->readers);\n+\t}\n+\tFREE_AND_NULL(st->list_file);\n+\tFREE_AND_NULL(st->reftable_dir);\n+\treftable_free(st);\n+}\n+\n+static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n+\t\t\t\t\t\t int cur_len)\n+{\n+\tstruct reftable_reader **cur =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * cur_len);\n+\tint i = 0;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tcur[i] = st->readers[i];\n+\t}\n+\treturn cur;\n+}\n+\n+static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n+\t\t\t\t int reuse_open)\n+{\n+\tint cur_len = st->merged == NULL ? 0 : st->merged->stack_len;\n+\tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n+\tint err = 0;\n+\tint names_len = names_length(names);\n+\tstruct reftable_reader **new_readers =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * names_len);\n+\tstruct reftable_table *new_tables =\n+\t\treftable_calloc(sizeof(struct reftable_table) * names_len);\n+\tint new_readers_len = 0;\n+\tstruct reftable_merged_table *new_merged = NULL;\n+\tint i;\n+\n+\twhile (*names) {\n+\t\tstruct reftable_reader *rd = NULL;\n+\t\tchar *name = *names++;\n+\n+\t\t/* this is linear; we assume compaction keeps the number of\n+\t\t tables under control so this is not quadratic. */\n+\t\tint j = 0;\n+\t\tfor (j = 0; reuse_open && j < cur_len; j++) {\n+\t\t\tif (cur[j] != NULL && 0 == strcmp(cur[j]->name, name)) {\n+\t\t\t\trd = cur[j];\n+\t\t\t\tcur[j] = NULL;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (rd == NULL) {\n+\t\t\tstruct reftable_block_source src = { NULL };\n+\t\t\tstruct strbuf table_path = STRBUF_INIT;\n+\t\t\tstrbuf_addstr(&table_path, st->reftable_dir);\n+\t\t\tstrbuf_addstr(&table_path, \"/\");\n+\t\t\tstrbuf_addstr(&table_path, name);\n+\n+\t\t\terr = reftable_block_source_from_file(&src,\n+\t\t\t\t\t\t\t table_path.buf);\n+\t\t\tstrbuf_release(&table_path);\n+\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\n+\t\t\terr = reftable_new_reader(&rd, &src, name);\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\n+\t\tnew_readers[new_readers_len] = rd;\n+\t\treftable_table_from_reader(&new_tables[new_readers_len], rd);\n+\t\tnew_readers_len++;\n+\t}\n+\n+\t/* success! */\n+\terr = reftable_new_merged_table(&new_merged, new_tables,\n+\t\t\t\t\tnew_readers_len, st->config.hash_id);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tnew_tables = NULL;\n+\tst->readers_len = new_readers_len;\n+\tif (st->merged != NULL) {\n+\t\tmerged_table_release(st->merged);\n+\t\treftable_merged_table_free(st->merged);\n+\t}\n+\tif (st->readers != NULL) {\n+\t\treftable_free(st->readers);\n+\t}\n+\tst->readers = new_readers;\n+\tnew_readers = NULL;\n+\tnew_readers_len = 0;\n+\n+\tnew_merged->suppress_deletions = 1;\n+\tst->merged = new_merged;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tif (cur[i] != NULL) {\n+\t\t\treader_close(cur[i]);\n+\t\t\treftable_reader_free(cur[i]);\n+\t\t}\n+\t}\n+\n+done:\n+\tfor (i = 0; i < new_readers_len; i++) {\n+\t\treader_close(new_readers[i]);\n+\t\treftable_reader_free(new_readers[i]);\n+\t}\n+\treftable_free(new_readers);\n+\treftable_free(new_tables);\n+\treftable_free(cur);\n+\treturn err;\n+}\n+\n+/* return negative if a before b. */\n+static int tv_cmp(struct timeval *a, struct timeval *b)\n+{\n+\ttime_t diff = a->tv_sec - b->tv_sec;\n+\tint udiff = a->tv_usec - b->tv_usec;\n+\n+\tif (diff != 0)\n+\t\treturn diff;\n+\n+\treturn udiff;\n+}\n+\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open)\n+{\n+\tstruct timeval deadline = { 0 };\n+\tint err = gettimeofday(&deadline, NULL);\n+\tint64_t delay = 0;\n+\tint tries = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tdeadline.tv_sec += 3;\n+\twhile (1) {\n+\t\tchar **names = NULL;\n+\t\tchar **names_after = NULL;\n+\t\tstruct timeval now = { 0 };\n+\t\tint err = gettimeofday(&now, NULL);\n+\t\tint err2 = 0;\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* Only look at deadlines after the first few times. This\n+\t\t simplifies debugging in GDB */\n+\t\ttries++;\n+\t\tif (tries > 3 && tv_cmp(&now, &deadline) >= 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = read_lines(st->list_file, &names);\n+\t\tif (err < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\t\terr = reftable_stack_reload_once(st, names, reuse_open);\n+\t\tif (err == 0) {\n+\t\t\tfree_names(names);\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err != REFTABLE_NOT_EXIST_ERROR) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* err == REFTABLE_NOT_EXIST_ERROR can be caused by a concurrent\n+\t\t writer. Check if there was one by checking if the name list\n+\t\t changed.\n+\t\t*/\n+\t\terr2 = read_lines(st->list_file, &names_after);\n+\t\tif (err2 < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err2;\n+\t\t}\n+\n+\t\tif (names_equal(names_after, names)) {\n+\t\t\tfree_names(names);\n+\t\t\tfree_names(names_after);\n+\t\t\treturn err;\n+\t\t}\n+\t\tfree_names(names);\n+\t\tfree_names(names_after);\n+\n+\t\tdelay = delay + (delay * rand()) / RAND_MAX + 1;\n+\t\tsleep_millisec(delay);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/* -1 = error\n+ 0 = up to date\n+ 1 = changed. */\n+static int stack_uptodate(struct reftable_stack *st)\n+{\n+\tchar **names = NULL;\n+\tint err = read_lines(st->list_file, &names);\n+\tint i = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tfor (i = 0; i < st->readers_len; i++) {\n+\t\tif (names[i] == NULL) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (strcmp(st->readers[i]->name, names[i])) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (names[st->merged->stack_len] != NULL) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tfree_names(names);\n+\treturn err;\n+}\n+\n+int reftable_stack_reload(struct reftable_stack *st)\n+{\n+\tint err = stack_uptodate(st);\n+\tif (err > 0)\n+\t\treturn reftable_stack_reload_maybe_reuse(st, 1);\n+\treturn err;\n+}\n+\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write)(struct reftable_writer *wr, void *arg),\n+\t\t void *arg)\n+{\n+\tint err = stack_try_add(st, write, arg);\n+\tif (err < 0) {\n+\t\tif (err == REFTABLE_LOCK_ERROR) {\n+\t\t\t/* Ignore error return, we want to propagate\n+\t\t\t REFTABLE_LOCK_ERROR.\n+\t\t\t*/\n+\t\t\treftable_stack_reload(st);\n+\t\t}\n+\t\treturn err;\n+\t}\n+\n+\tif (!st->disable_auto_compact)\n+\t\treturn reftable_stack_auto_compact(st);\n+\n+\treturn 0;\n+}\n+\n+static void format_name(struct strbuf *dest, uint64_t min, uint64_t max)\n+{\n+\tchar buf[100];\n+\tsnprintf(buf, sizeof(buf), \"0x%012\" PRIx64 \"-0x%012\" PRIx64, min, max);\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, buf);\n+}\n+\n+struct reftable_addition {\n+\tint lock_file_fd;\n+\tstruct strbuf lock_file_name;\n+\tstruct reftable_stack *stack;\n+\tchar **names;\n+\tchar **new_tables;\n+\tint new_tables_len;\n+\tuint64_t next_update_index;\n+};\n+\n+#define REFTABLE_ADDITION_INIT \\\n+\t{ \\\n+\t\t.lock_file_name = STRBUF_INIT \\\n+\t}\n+\n+static int reftable_stack_init_addition(struct reftable_addition *add,\n+\t\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tadd->stack = st;\n+\n+\tstrbuf_reset(&add->lock_file_name);\n+\tstrbuf_addstr(&add->lock_file_name, st->list_file);\n+\tstrbuf_addstr(&add->lock_file_name, \".lock\");\n+\n+\tadd->lock_file_fd = open(add->lock_file_name.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (add->lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = REFTABLE_LOCK_ERROR;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\terr = stack_uptodate(st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 1) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->next_update_index = reftable_stack_next_update_index(st);\n+done:\n+\tif (err) {\n+\t\treftable_addition_close(add);\n+\t}\n+\treturn err;\n+}\n+\n+static void reftable_addition_close(struct reftable_addition *add)\n+{\n+\tint i = 0;\n+\tstruct strbuf nm = STRBUF_INIT;\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstrbuf_reset(&nm);\n+\t\tstrbuf_addstr(&nm, add->stack->list_file);\n+\t\tstrbuf_addstr(&nm, \"/\");\n+\t\tstrbuf_addstr(&nm, add->new_tables[i]);\n+\t\tunlink(nm.buf);\n+\t\treftable_free(add->new_tables[i]);\n+\t\tadd->new_tables[i] = NULL;\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\tif (add->lock_file_fd > 0) {\n+\t\tclose(add->lock_file_fd);\n+\t\tadd->lock_file_fd = 0;\n+\t}\n+\tif (add->lock_file_name.len > 0) {\n+\t\tunlink(add->lock_file_name.buf);\n+\t\tstrbuf_release(&add->lock_file_name);\n+\t}\n+\n+\tfree_names(add->names);\n+\tadd->names = NULL;\n+\tstrbuf_release(&nm);\n+}\n+\n+void reftable_addition_destroy(struct reftable_addition *add)\n+{\n+\tif (add == NULL) {\n+\t\treturn;\n+\t}\n+\treftable_addition_close(add);\n+\treftable_free(add);\n+}\n+\n+int reftable_addition_commit(struct reftable_addition *add)\n+{\n+\tstruct strbuf table_list = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\tif (add->new_tables_len == 0)\n+\t\tgoto done;\n+\n+\tfor (i = 0; i < add->stack->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->stack->readers[i]->name);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->new_tables[i]);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\n+\terr = write(add->lock_file_fd, table_list.buf, table_list.len);\n+\tstrbuf_release(&table_list);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = close(add->lock_file_fd);\n+\tadd->lock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(add->lock_file_name.buf, add->stack->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(add->stack);\n+\n+done:\n+\treftable_addition_close(add);\n+\treturn err;\n+}\n+\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n+\t*dest = reftable_calloc(sizeof(**dest));\n+\t**dest = empty;\n+\terr = reftable_stack_init_addition(*dest, st);\n+\tif (err) {\n+\t\treftable_free(*dest);\n+\t\t*dest = NULL;\n+\t}\n+\treturn err;\n+}\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct reftable_addition add = REFTABLE_ADDITION_INIT;\n+\tint err = reftable_stack_init_addition(&add, st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tif (err > 0) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_addition_add(&add, write_table, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_addition_commit(&add);\n+done:\n+\treftable_addition_close(&add);\n+\treturn err;\n+}\n+\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf tab_file_name = STRBUF_INIT;\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\tint tab_fd = 0;\n+\n+\tstrbuf_reset(&next_name);\n+\tformat_name(&next_name, add->next_update_index, add->next_update_index);\n+\n+\tstrbuf_addstr(&temp_tab_file_name, add->stack->reftable_dir);\n+\tstrbuf_addstr(&temp_tab_file_name, \"/\");\n+\tstrbuf_addbuf(&temp_tab_file_name, &next_name);\n+\tstrbuf_addstr(&temp_tab_file_name, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab_file_name.buf);\n+\tif (tab_fd < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd,\n+\t\t\t\t &add->stack->config);\n+\terr = write_table(wr, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_writer_close(wr);\n+\tif (err == REFTABLE_EMPTY_TABLE_ERROR) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = stack_check_addition(add->stack, temp_tab_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (wr->min_update_index < add->next_update_index) {\n+\t\terr = REFTABLE_API_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tformat_name(&next_name, wr->min_update_index, wr->max_update_index);\n+\tstrbuf_addstr(&next_name, \".ref\");\n+\n+\tstrbuf_addstr(&tab_file_name, add->stack->reftable_dir);\n+\tstrbuf_addstr(&tab_file_name, \"/\");\n+\tstrbuf_addbuf(&tab_file_name, &next_name);\n+\n+\t/* TODO: should check destination out of paranoia */\n+\terr = rename(temp_tab_file_name.buf, tab_file_name.buf);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->new_tables = reftable_realloc(add->new_tables,\n+\t\t\t\t\t sizeof(*add->new_tables) *\n+\t\t\t\t\t\t (add->new_tables_len + 1));\n+\tadd->new_tables[add->new_tables_len] = strbuf_detach(&next_name, NULL);\n+\tadd->new_tables_len++;\n+done:\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (temp_tab_file_name.len > 0) {\n+\t\tunlink(temp_tab_file_name.buf);\n+\t}\n+\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&tab_file_name);\n+\tstrbuf_release(&next_name);\n+\treftable_writer_free(wr);\n+\treturn err;\n+}\n+\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st)\n+{\n+\tint sz = st->merged->stack_len;\n+\tif (sz > 0)\n+\t\treturn reftable_reader_max_update_index(st->readers[sz - 1]) +\n+\t\t 1;\n+\treturn 1;\n+}\n+\n+static int stack_compact_locked(struct reftable_stack *st, int first, int last,\n+\t\t\t\tstruct strbuf *temp_tab,\n+\t\t\t\tstruct reftable_log_expiry_config *config)\n+{\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tint tab_fd = -1;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\n+\tformat_name(&next_name,\n+\t\t reftable_reader_min_update_index(st->readers[first]),\n+\t\t reftable_reader_max_update_index(st->readers[first]));\n+\n+\tstrbuf_reset(temp_tab);\n+\tstrbuf_addstr(temp_tab, st->reftable_dir);\n+\tstrbuf_addstr(temp_tab, \"/\");\n+\tstrbuf_addbuf(temp_tab, &next_name);\n+\tstrbuf_addstr(temp_tab, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab->buf);\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd, &st->config);\n+\n+\terr = stack_write_compact(st, wr, first, last, config);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = reftable_writer_close(wr);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\n+done:\n+\treftable_writer_free(wr);\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (err != 0 && temp_tab->len > 0) {\n+\t\tunlink(temp_tab->buf);\n+\t\tstrbuf_release(temp_tab);\n+\t}\n+\tstrbuf_release(&next_name);\n+\treturn err;\n+}\n+\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint subtabs_len = last - first + 1;\n+\tstruct reftable_table *subtabs = reftable_calloc(\n+\t\tsizeof(struct reftable_table) * (last - first + 1));\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tuint64_t entries = 0;\n+\n+\tint i = 0, j = 0;\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct reftable_reader *t = st->readers[i];\n+\t\treftable_table_from_reader(&subtabs[j++], t);\n+\t\tst->stats.bytes += t->size;\n+\t}\n+\treftable_writer_set_limits(wr, st->readers[first]->min_update_index,\n+\t\t\t\t st->readers[last]->max_update_index);\n+\n+\terr = reftable_new_merged_table(&mt, subtabs, subtabs_len,\n+\t\t\t\t\tst->config.hash_id);\n+\tif (err < 0) {\n+\t\treftable_free(subtabs);\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_merged_table_seek_ref(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (first == 0 && reftable_ref_record_is_deletion(&ref)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_ref(wr, &ref);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_merged_table_seek_log(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (first == 0 && reftable_log_record_is_deletion(&log)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config != NULL && config->time > 0 &&\n+\t\t log.time < config->time) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config != NULL && config->min_update_index > 0 &&\n+\t\t log.update_index < config->min_update_index) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_log(wr, &log);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\tif (mt != NULL) {\n+\t\tmerged_table_release(mt);\n+\t\treftable_merged_table_free(mt);\n+\t}\n+\treftable_ref_record_release(&ref);\n+\treftable_log_record_release(&log);\n+\tst->stats.entries_written += entries;\n+\treturn err;\n+}\n+\n+/* < 0: error. 0 == OK, > 0 attempt failed; could retry. */\n+static int stack_compact_range(struct reftable_stack *st, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *expiry)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf new_table_name = STRBUF_INIT;\n+\tstruct strbuf lock_file_name = STRBUF_INIT;\n+\tstruct strbuf ref_list_contents = STRBUF_INIT;\n+\tstruct strbuf new_table_path = STRBUF_INIT;\n+\tint err = 0;\n+\tint have_lock = 0;\n+\tint lock_file_fd = 0;\n+\tint compact_count = last - first + 1;\n+\tchar **listp = NULL;\n+\tchar **delete_on_success =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tchar **subtable_locks =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tint i = 0;\n+\tint j = 0;\n+\tint is_empty_table = 0;\n+\n+\tif (first > last || (expiry == NULL && first == last)) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\tst->stats.attempts++;\n+\n+\tstrbuf_reset(&lock_file_name);\n+\tstrbuf_addstr(&lock_file_name, st->list_file);\n+\tstrbuf_addstr(&lock_file_name, \".lock\");\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\t/* Don't want to write to the lock for now. */\n+\tclose(lock_file_fd);\n+\tlock_file_fd = 0;\n+\n+\thave_lock = 1;\n+\terr = stack_uptodate(st);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct strbuf subtab_file_name = STRBUF_INIT;\n+\t\tstruct strbuf subtab_lock = STRBUF_INIT;\n+\t\tint sublock_file_fd = -1;\n+\n+\t\tstrbuf_addstr(&subtab_file_name, st->reftable_dir);\n+\t\tstrbuf_addstr(&subtab_file_name, \"/\");\n+\t\tstrbuf_addstr(&subtab_file_name, reader_name(st->readers[i]));\n+\n+\t\tstrbuf_reset(&subtab_lock);\n+\t\tstrbuf_addbuf(&subtab_lock, &subtab_file_name);\n+\t\tstrbuf_addstr(&subtab_lock, \".lock\");\n+\n+\t\tsublock_file_fd = open(subtab_lock.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\t\tif (sublock_file_fd > 0) {\n+\t\t\tclose(sublock_file_fd);\n+\t\t} else if (sublock_file_fd < 0) {\n+\t\t\tif (errno == EEXIST) {\n+\t\t\t\terr = 1;\n+\t\t\t} else {\n+\t\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\t}\n+\t\t}\n+\n+\t\tsubtable_locks[j] = subtab_lock.buf;\n+\t\tdelete_on_success[j] = subtab_file_name.buf;\n+\t\tj++;\n+\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\terr = unlink(lock_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\thave_lock = 0;\n+\n+\terr = stack_compact_locked(st, first, last, &temp_tab_file_name,\n+\t\t\t\t expiry);\n+\t/* Compaction + tombstones can create an empty table out of non-empty\n+\t * tables. */\n+\tis_empty_table = (err == REFTABLE_EMPTY_TABLE_ERROR);\n+\tif (is_empty_table) {\n+\t\terr = 0;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\thave_lock = 1;\n+\n+\tformat_name(&new_table_name, st->readers[first]->min_update_index,\n+\t\t st->readers[last]->max_update_index);\n+\tstrbuf_addstr(&new_table_name, \".ref\");\n+\n+\tstrbuf_reset(&new_table_path);\n+\tstrbuf_addstr(&new_table_path, st->reftable_dir);\n+\tstrbuf_addstr(&new_table_path, \"/\");\n+\tstrbuf_addbuf(&new_table_path, &new_table_name);\n+\n+\tif (!is_empty_table) {\n+\t\terr = rename(temp_tab_file_name.buf, new_table_path.buf);\n+\t\tif (err < 0) {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < first; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tif (!is_empty_table) {\n+\t\tstrbuf_addbuf(&ref_list_contents, &new_table_name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tfor (i = last + 1; i < st->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\n+\terr = write(lock_file_fd, ref_list_contents.buf, ref_list_contents.len);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\terr = close(lock_file_fd);\n+\tlock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(lock_file_name.buf, st->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\thave_lock = 0;\n+\n+\t/* Reload the stack before deleting. On windows, we can only delete the\n+\t files after we closed them.\n+\t*/\n+\terr = reftable_stack_reload_maybe_reuse(st, first < last);\n+\n+\tlistp = delete_on_success;\n+\twhile (*listp) {\n+\t\tif (strcmp(*listp, new_table_path.buf)) {\n+\t\t\tunlink(*listp);\n+\t\t}\n+\t\tlistp++;\n+\t}\n+\n+done:\n+\tfree_names(delete_on_success);\n+\n+\tlistp = subtable_locks;\n+\twhile (*listp) {\n+\t\tunlink(*listp);\n+\t\tlistp++;\n+\t}\n+\tfree_names(subtable_locks);\n+\tif (lock_file_fd > 0) {\n+\t\tclose(lock_file_fd);\n+\t\tlock_file_fd = 0;\n+\t}\n+\tif (have_lock) {\n+\t\tunlink(lock_file_name.buf);\n+\t}\n+\tstrbuf_release(&new_table_name);\n+\tstrbuf_release(&new_table_path);\n+\tstrbuf_release(&ref_list_contents);\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&lock_file_name);\n+\treturn err;\n+}\n+\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\treturn stack_compact_range(st, 0, st->merged->stack_len - 1, config);\n+}\n+\n+static int stack_compact_range_stats(struct reftable_stack *st, int first,\n+\t\t\t\t int last,\n+\t\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint err = stack_compact_range(st, first, last, config);\n+\tif (err > 0) {\n+\t\tst->stats.failures++;\n+\t}\n+\treturn err;\n+}\n+\n+static int segment_size(struct segment *s)\n+{\n+\treturn s->end - s->start;\n+}\n+\n+int fastlog2(uint64_t sz)\n+{\n+\tint l = 0;\n+\tif (sz == 0)\n+\t\treturn 0;\n+\tfor (; sz; sz /= 2) {\n+\t\tl++;\n+\t}\n+\treturn l - 1;\n+}\n+\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n+{\n+\tstruct segment *segs = reftable_calloc(sizeof(struct segment) * n);\n+\tint next = 0;\n+\tstruct segment cur = { 0 };\n+\tint i = 0;\n+\n+\tif (n == 0) {\n+\t\t*seglen = 0;\n+\t\treturn segs;\n+\t}\n+\tfor (i = 0; i < n; i++) {\n+\t\tint log = fastlog2(sizes[i]);\n+\t\tif (cur.log != log && cur.bytes > 0) {\n+\t\t\tstruct segment fresh = {\n+\t\t\t\t.start = i,\n+\t\t\t};\n+\n+\t\t\tsegs[next++] = cur;\n+\t\t\tcur = fresh;\n+\t\t}\n+\n+\t\tcur.log = log;\n+\t\tcur.end = i + 1;\n+\t\tcur.bytes += sizes[i];\n+\t}\n+\tsegs[next++] = cur;\n+\t*seglen = next;\n+\treturn segs;\n+}\n+\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, sizes, n);\n+\tstruct segment min_seg = {\n+\t\t.log = 64,\n+\t};\n+\tint i = 0;\n+\tfor (i = 0; i < seglen; i++) {\n+\t\tif (segment_size(&segs[i]) == 1) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (segs[i].log < min_seg.log) {\n+\t\t\tmin_seg = segs[i];\n+\t\t}\n+\t}\n+\n+\twhile (min_seg.start > 0) {\n+\t\tint prev = min_seg.start - 1;\n+\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmin_seg.start = prev;\n+\t\tmin_seg.bytes += sizes[prev];\n+\t}\n+\n+\treftable_free(segs);\n+\treturn min_seg;\n+}\n+\n+static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes =\n+\t\treftable_calloc(sizeof(uint64_t) * st->merged->stack_len);\n+\tint version = (st->config.hash_id == SHA1_ID) ? 1 : 2;\n+\tint overhead = header_size(version) - 1;\n+\tint i = 0;\n+\tfor (i = 0; i < st->merged->stack_len; i++) {\n+\t\tsizes[i] = st->readers[i]->size - overhead;\n+\t}\n+\treturn sizes;\n+}\n+\n+int reftable_stack_auto_compact(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n+\tstruct segment seg =\n+\t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n+\treftable_free(sizes);\n+\tif (segment_size(&seg) > 0)\n+\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n+\t\t\t\t\t\t NULL);\n+\n+\treturn 0;\n+}\n+\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st)\n+{\n+\treturn &st->stats;\n+}\n+\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_table tab = { NULL };\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\treturn reftable_table_read_ref(&tab, refname, ref);\n+}\n+\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(st);\n+\tint err = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_log(&it, log);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(log->refname, refname) ||\n+\t reftable_log_record_is_deletion(log)) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tif (err) {\n+\t\treftable_log_record_release(log);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name)\n+{\n+\tint err = 0;\n+\tstruct reftable_block_source src = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct reftable_ref_record *refs = NULL;\n+\tstruct reftable_iterator it = { NULL };\n+\tint cap = 0;\n+\tint len = 0;\n+\tint i = 0;\n+\n+\tif (st->config.skip_name_check)\n+\t\treturn 0;\n+\n+\terr = reftable_block_source_from_file(&src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&rd, &src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tif (err > 0) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (len >= cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\trefs = reftable_realloc(refs, cap * sizeof(refs[0]));\n+\t\t}\n+\n+\t\trefs[len++] = ref;\n+\t}\n+\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\n+\terr = validate_ref_record_addition(tab, refs, len);\n+\n+done:\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t}\n+\n+\tfree(refs);\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\treturn err;\n+}\ndiff --git a/reftable/stack.h b/reftable/stack.h\nnew file mode 100644\nindex 0000000000..f57005846e\n--- /dev/null\n+++ b/reftable/stack.h\n@@ -0,0 +1,41 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef STACK_H\n+#define STACK_H\n+\n+#include \"system.h\"\n+#include \"reftable-writer.h\"\n+#include \"reftable-stack.h\"\n+\n+struct reftable_stack {\n+\tchar *list_file;\n+\tchar *reftable_dir;\n+\tint disable_auto_compact;\n+\n+\tstruct reftable_write_options config;\n+\n+\tstruct reftable_reader **readers;\n+\tsize_t readers_len;\n+\tstruct reftable_merged_table *merged;\n+\tstruct reftable_compaction_stats stats;\n+};\n+\n+int read_lines(const char *filename, char ***lines);\n+\n+struct segment {\n+\tint start, end;\n+\tint log;\n+\tuint64_t bytes;\n+};\n+\n+int fastlog2(uint64_t sz);\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n);\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n);\n+\n+#endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nnew file mode 100644\nindex 0000000000..11d3d30799\n--- /dev/null\n+++ b/reftable/stack_test.c\n@@ -0,0 +1,781 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+\n+#include \"merged.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+#include \n+#include \n+\n+static void clear_dir(const char *dirname)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstrbuf_addstr(&path, dirname);\n+\tremove_dir_recursively(&path, 0);\n+\tstrbuf_release(&path);\n+}\n+\n+static void test_read_file(void)\n+{\n+\tchar fn[256] = \"/tmp/stack.test_read_file.XXXXXX\";\n+\tint fd = mkstemp(fn);\n+\tchar out[1024] = \"line1\\n\\nline2\\nline3\";\n+\tint n, err;\n+\tchar **names = NULL;\n+\tchar *want[] = { \"line1\", \"line2\", \"line3\" };\n+\tint i = 0;\n+\n+\tEXPECT(fd > 0);\n+\tn = write(fd, out, strlen(out));\n+\tEXPECT(n == strlen(out));\n+\terr = close(fd);\n+\tEXPECT(err >= 0);\n+\n+\terr = read_lines(fn, &names);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; names[i] != NULL; i++) {\n+\t\tEXPECT(0 == strcmp(want[i], names[i]));\n+\t}\n+\tfree_names(names);\n+\tremove(fn);\n+}\n+\n+static void test_parse_names(void)\n+{\n+\tchar buf[] = \"line\\n\";\n+\tchar **names = NULL;\n+\tparse_names(buf, strlen(buf), &names);\n+\n+\tEXPECT(NULL != names[0]);\n+\tEXPECT(0 == strcmp(names[0], \"line\"));\n+\tEXPECT(NULL == names[1]);\n+\tfree_names(names);\n+}\n+\n+static void test_names_equal(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", \"c\", NULL };\n+\tchar *b[] = { \"a\", \"b\", \"d\", NULL };\n+\tchar *c[] = { \"a\", \"b\", NULL };\n+\n+\tEXPECT(names_equal(a, a));\n+\tEXPECT(!names_equal(a, b));\n+\tEXPECT(!names_equal(a, c));\n+}\n+\n+static int write_test_ref(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct reftable_ref_record *ref = arg;\n+\treftable_writer_set_limits(wr, ref->update_index, ref->update_index);\n+\treturn reftable_writer_add_ref(wr, ref);\n+}\n+\n+struct write_log_arg {\n+\tstruct reftable_log_record *log;\n+\tuint64_t update_index;\n+};\n+\n+static int write_test_log(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct write_log_arg *wla = arg;\n+\n+\treftable_writer_set_limits(wr, wla->update_index, wla->update_index);\n+\treturn reftable_writer_add_log(wr, wla->log);\n+}\n+\n+static void test_reftable_stack_add_one(void)\n+{\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.target = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(\"master\", dest.target));\n+\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_uptodate(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL;\n+\tstruct reftable_stack *st2 = NULL;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.target = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"branch2\",\n+\t\t.update_index = 2,\n+\t\t.target = \"master\",\n+\t};\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st1, &write_test_ref, &ref1);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tEXPECT(err == REFTABLE_LOCK_ERROR);\n+\n+\terr = reftable_stack_reload(st2);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tEXPECT_ERR(err);\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_transaction_api(void)\n+{\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_addition *add = NULL;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.target = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_new_addition(&add, st);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_addition_add(add, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_addition_commit(add);\n+\tEXPECT_ERR(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(\"master\", dest.target));\n+\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_validate_refname(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tint i;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"a/b\",\n+\t\t.update_index = 1,\n+\t\t.target = \"master\",\n+\t};\n+\tchar *additions[] = { \"a\", \"a/b/c\" };\n+\n+\tEXPECT(mkdtemp(dir));\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(additions); i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = additions[i],\n+\t\t\t.update_index = 1,\n+\t\t\t.target = \"master\",\n+\t\t};\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tEXPECT(err == REFTABLE_NAME_CONFLICT);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static int write_error(struct reftable_writer *wr, void *arg)\n+{\n+\treturn *((int *)arg);\n+}\n+\n+static void test_reftable_stack_update_index_check(void)\n+{\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"name1\",\n+\t\t.update_index = 1,\n+\t\t.target = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"name2\",\n+\t\t.update_index = 1,\n+\t\t.target = \"master\",\n+\t};\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref1);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref2);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_lock_failure(void)\n+{\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err, i;\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tfor (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) {\n+\t\terr = reftable_stack_add(st, &write_error, &i);\n+\t\tEXPECT(err == i);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_add(void)\n+{\n+\tint i = 0;\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t.exact_log_message = 1,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tst->disable_auto_compact = 1;\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].value = reftable_malloc(SHA1_SIZE);\n+\t\trefs[i].update_index = i + 1;\n+\t\tset_test_hash(refs[i].value, i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = N + i + 1;\n+\t\tlogs[i].new_hash = reftable_malloc(SHA1_SIZE);\n+\t\tlogs[i].email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].new_hash, i);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_ref_record dest = { NULL };\n+\n+\t\tint err = reftable_stack_read_ref(st, refs[i].refname, &dest);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(reftable_ref_record_equal(&dest, refs + i, SHA1_SIZE));\n+\t\treftable_ref_record_release(&dest);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_log_record dest = { NULL };\n+\t\tint err = reftable_stack_read_log(st, refs[i].refname, &dest);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(reftable_log_record_equal(&dest, logs + i, SHA1_SIZE));\n+\t\treftable_log_record_release(&dest);\n+\t}\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_log_normalize(void)\n+{\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t0,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\n+\tuint8_t h1[SHA1_SIZE] = { 0x01 }, h2[SHA1_SIZE] = { 0x02 };\n+\n+\tstruct reftable_log_record input = {\n+\t\t.refname = \"branch\",\n+\t\t.update_index = 1,\n+\t\t.new_hash = h1,\n+\t\t.old_hash = h2,\n+\t};\n+\tstruct reftable_log_record dest = {\n+\t\t.update_index = 0,\n+\t};\n+\tstruct write_log_arg arg = {\n+\t\t.log = &input,\n+\t\t.update_index = 1,\n+\t};\n+\n+\tEXPECT(mkdtemp(dir));\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tinput.message = \"one\\ntwo\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\n+\tinput.message = \"one\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(dest.message, \"one\\n\"));\n+\n+\tinput.message = \"two\\n\";\n+\targ.update_index = 2;\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT_ERR(err);\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(dest.message, \"two\\n\"));\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\treftable_log_record_release(&dest);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_tombstone(void)\n+{\n+\tint i = 0;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\tstruct reftable_ref_record dest = { NULL };\n+\tstruct reftable_log_record log_dest = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tconst char *buf = \"branch\";\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\tif (i % 2 == 0) {\n+\t\t\trefs[i].value = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_test_hash(refs[i].value, i);\n+\t\t}\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\t/* update_index is part of the key. */\n+\t\tlogs[i].update_index = 42;\n+\t\tif (i % 2 == 0) {\n+\t\t\tlogs[i].new_hash = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_test_hash(logs[i].new_hash, i);\n+\t\t\tlogs[i].email = xstrdup(\"identity@invalid\");\n+\t\t}\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tEXPECT(err == 1);\n+\treftable_ref_record_release(&dest);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tEXPECT(err == 1);\n+\treftable_log_record_release(&log_dest);\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tEXPECT(err == 1);\n+\treftable_ref_record_release(&dest);\n+\treftable_log_record_release(&log_dest);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_hash_id(void)\n+{\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"master\",\n+\t\t.target = \"target\",\n+\t\t.update_index = 1,\n+\t};\n+\tstruct reftable_write_options cfg32 = { .hash_id = SHA256_ID };\n+\tstruct reftable_stack *st32 = NULL;\n+\tstruct reftable_write_options cfg_default = { 0 };\n+\tstruct reftable_stack *st_default = NULL;\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\t/* can't read it with the wrong hash ID. */\n+\terr = reftable_new_stack(&st32, dir, cfg32);\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+\n+\t/* check that we can read it back with default config too. */\n+\terr = reftable_new_stack(&st_default, dir, cfg_default);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n+\tEXPECT_ERR(err);\n+\n+\tEXPECT(!strcmp(dest.target, ref.target));\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st_default);\n+\tclear_dir(dir);\n+}\n+\n+static void test_log2(void)\n+{\n+\tEXPECT(1 == fastlog2(3));\n+\tEXPECT(2 == fastlog2(4));\n+\tEXPECT(2 == fastlog2(5));\n+}\n+\n+static void test_sizes_to_segments(void)\n+{\n+\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n+\t/* .................0 1 2 3 4 5 */\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(segs[2].log == 3);\n+\tEXPECT(segs[2].start == 5);\n+\tEXPECT(segs[2].end == 6);\n+\n+\tEXPECT(segs[1].log == 2);\n+\tEXPECT(segs[1].start == 2);\n+\tEXPECT(segs[1].end == 5);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_empty(void)\n+{\n+\tuint64_t sizes[0];\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(seglen == 0);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_all_equal(void)\n+{\n+\tuint64_t sizes[] = { 5, 5 };\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(seglen == 1);\n+\tEXPECT(segs[0].start == 0);\n+\tEXPECT(segs[0].end == 2);\n+\treftable_free(segs);\n+}\n+\n+static void test_suggest_compaction_segment(void)\n+{\n+\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n+\t/* .................0 1 2 3 4 5 6 */\n+\tstruct segment min =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(min.start == 2);\n+\tEXPECT(min.end == 7);\n+}\n+\n+static void test_suggest_compaction_segment_nothing(void)\n+{\n+\tuint64_t sizes[] = { 64, 32, 16, 8, 4, 2 };\n+\tstruct segment result =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(result.start == result.end);\n+}\n+\n+static void test_reflog_expire(void)\n+{\n+\tchar dir[256] = \"/tmp/stack.test_reflog_expire.XXXXXX\";\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tstruct reftable_log_record logs[20] = { { NULL } };\n+\tint N = ARRAY_SIZE(logs) - 1;\n+\tint i = 0;\n+\tint err;\n+\tstruct reftable_log_expiry_config expiry = {\n+\t\t.time = 10,\n+\t};\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = i;\n+\t\tlogs[i].time = i;\n+\t\tlogs[i].new_hash = reftable_malloc(SHA1_SIZE);\n+\t\tlogs[i].email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].new_hash, i);\n+\t}\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, logs[9].refname, &log);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[11].refname, &log);\n+\tEXPECT_ERR(err);\n+\n+\texpiry.min_update_index = 15;\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, logs[14].refname, &log);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[16].refname, &log);\n+\tEXPECT_ERR(err);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i <= N; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+\treftable_log_record_release(&log);\n+}\n+\n+static int write_nothing(struct reftable_writer *wr, void *arg)\n+{\n+\treftable_writer_set_limits(wr, 1, 1);\n+\treturn 0;\n+}\n+\n+static void test_empty_add(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tstruct reftable_stack *st2 = NULL;\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_nothing, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tclear_dir(dir);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st2);\n+}\n+\n+static void test_reftable_stack_auto_compaction(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n+\tint err, i;\n+\tint N = 100;\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t\t.target = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\n+\t\tEXPECT(i < 3 || st->merged->stack_len < 2 * fastlog2(i));\n+\t}\n+\n+\tEXPECT(reftable_stack_compaction_stats(st)->entries_written <\n+\t (uint64_t)(N * fastlog2(N)));\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+int stack_test_main(int argc, const char *argv[])\n+{\n+\ttest_reftable_stack_uptodate();\n+\ttest_reftable_stack_transaction_api();\n+\ttest_reftable_stack_hash_id();\n+\ttest_sizes_to_segments_all_equal();\n+\ttest_reftable_stack_auto_compaction();\n+\ttest_reftable_stack_validate_refname();\n+\ttest_reftable_stack_update_index_check();\n+\ttest_reftable_stack_lock_failure();\n+\ttest_reftable_stack_log_normalize();\n+\ttest_reftable_stack_tombstone();\n+\ttest_reftable_stack_add_one();\n+\ttest_empty_add();\n+\ttest_reflog_expire();\n+\ttest_suggest_compaction_segment();\n+\ttest_suggest_compaction_segment_nothing();\n+\ttest_sizes_to_segments();\n+\ttest_sizes_to_segments_empty();\n+\ttest_log2();\n+\ttest_parse_names();\n+\ttest_read_file();\n+\ttest_names_equal();\n+\ttest_reftable_stack_add();\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex fdf9258673..3b702f4855 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -5,8 +5,11 @@ int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n+\tmerged_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\trefname_test_main(argc, argv);\n \treftable_test_main(argc, argv);\n+\tstack_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"410896","messageId":"6229da992e43f4a2d6f4ccaccf2dbbdf11bd5a4a.1606419752.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v3.git.git.1606419752.gitgitgadget@gmail.com","subject":"[PATCH v3 16/16] Add \"test-tool dump-reftable\" command.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-11-26T19:42:31Z","receivedAt":"2020-11-26T19:42:57Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis command dumps individual tables or a stack of of tables.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/dump.c | 31 +++++++++++++++++++------------\n t/helper/test-reftable.c | 5 +++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 5 files changed, 27 insertions(+), 12 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex ec91fa14f4..6154db5615 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2409,6 +2409,7 @@ REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/dump.o\n REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/refname_test.o\ndiff --git a/reftable/dump.c b/reftable/dump.c\nindex 00b444e8c9..0bd7a1cfa6 100644\n--- a/reftable/dump.c\n+++ b/reftable/dump.c\n@@ -12,18 +12,25 @@ license that can be found in the LICENSE file or at\n #include \n #include \n \n-#include \"reftable.h\"\n+#include \"reftable-blocksource.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-record.h\"\n #include \"reftable-tests.h\"\n+#include \"reftable-writer.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+#include \"reftable-stack.h\"\n \n static uint32_t hash_id;\n \n static int dump_table(const char *tablename)\n {\n-\tstruct reftable_block_source src = { 0 };\n+\tstruct reftable_block_source src = { NULL };\n \tint err = reftable_block_source_from_file(&src, tablename);\n-\tstruct reftable_iterator it = { 0 };\n-\tstruct reftable_ref_record ref = { 0 };\n-\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n \tstruct reftable_reader *r = NULL;\n \n \tif (err < 0)\n@@ -49,7 +56,7 @@ static int dump_table(const char *tablename)\n \t\treftable_ref_record_print(&ref, hash_id);\n \t}\n \treftable_iterator_destroy(&it);\n-\treftable_ref_record_clear(&ref);\n+\treftable_ref_record_release(&ref);\n \n \terr = reftable_reader_seek_log(r, &it, \"\");\n \tif (err < 0) {\n@@ -66,7 +73,7 @@ static int dump_table(const char *tablename)\n \t\treftable_log_record_print(&log, hash_id);\n \t}\n \treftable_iterator_destroy(&it);\n-\treftable_log_record_clear(&log);\n+\treftable_log_record_release(&log);\n \n \treftable_reader_free(r);\n \treturn 0;\n@@ -95,9 +102,9 @@ static int dump_stack(const char *stackdir)\n {\n \tstruct reftable_stack *stack = NULL;\n \tstruct reftable_write_options cfg = {};\n-\tstruct reftable_iterator it = { 0 };\n-\tstruct reftable_ref_record ref = { 0 };\n-\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n \tstruct reftable_merged_table *merged = NULL;\n \n \tint err = reftable_new_stack(&stack, stackdir, cfg);\n@@ -122,7 +129,7 @@ static int dump_stack(const char *stackdir)\n \t\treftable_ref_record_print(&ref, hash_id);\n \t}\n \treftable_iterator_destroy(&it);\n-\treftable_ref_record_clear(&ref);\n+\treftable_ref_record_release(&ref);\n \n \terr = reftable_merged_table_seek_log(merged, &it, \"\");\n \tif (err < 0) {\n@@ -139,7 +146,7 @@ static int dump_stack(const char *stackdir)\n \t\treftable_log_record_print(&log, hash_id);\n \t}\n \treftable_iterator_destroy(&it);\n-\treftable_log_record_clear(&log);\n+\treftable_log_record_release(&log);\n \n \treftable_stack_destroy(stack);\n \treturn 0;\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 3b702f4855..c1ba131c3a 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -13,3 +13,8 @@ int cmd__reftable(int argc, const char **argv)\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n+\n+int cmd__dump_reftable(int argc, const char **argv)\n+{\n+\treturn reftable_dump_main(argc, (char *const *)argv);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 0208a0a41c..f064440a31 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -56,6 +56,7 @@ static struct test_cmd cmds[] = {\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"reftable\", cmd__reftable },\n+\t{ \"dump-reftable\", cmd__dump_reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 1de39ce5b5..226af8c6b8 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -18,6 +18,7 @@ int cmd__dump_cache_tree(int argc, const char **argv);\n int cmd__dump_fsmonitor(int argc, const char **argv);\n int cmd__dump_split_index(int argc, const char **argv);\n int cmd__dump_untracked_cache(int argc, const char **argv);\n+int cmd__dump_reftable(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n-- \ngitgitgadget\n"},{"id":"410899","messageId":"a590865a7008f065b3f4a75deabbf6dc06807623.1606419752.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v3.git.git.1606419752.gitgitgadget@gmail.com","subject":"[PATCH v3 14/16] Reftable support for git-core","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-11-26T19:42:29Z","receivedAt":"2020-11-26T19:42:57Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nFor background, see the previous commit introducing the library.\n\nThis introduces the file refs/reftable-backend.c containing a reftable-powered\nref storage backend.\n\nIt can be activated by passing --ref-storage=reftable to \"git init\", or setting\nGIT_TEST_REFTABLE in the environment.\n\nExample use: see t/t0031-reftable.sh\n\nSigned-off-by: Han-Wen Nienhuys \nSigned-off-by: Johannes Schindelin \nHelped-by: Johannes Schindelin \nCo-authored-by: Jeff King \n---\n .../technical/repository-version.txt | 7 +\n Makefile | 4 +\n builtin/clone.c | 5 +-\n builtin/init-db.c | 55 +-\n builtin/worktree.c | 27 +-\n cache.h | 8 +-\n config.mak.uname | 2 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n refs.c | 27 +-\n refs.h | 3 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1418 +++++++++++++++++\n repository.c | 2 +\n repository.h | 3 +\n setup.c | 9 +-\n t/t0031-reftable.sh | 199 +++\n t/t1409-avoid-packing-refs.sh | 6 +\n t/t1450-fsck.sh | 6 +\n t/t3210-pack-refs.sh | 6 +\n t/test-lib.sh | 5 +\n 20 files changed, 1771 insertions(+), 33 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100755 t/t0031-reftable.sh\n\ndiff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\nindex 7844ef30ff..7257623583 100644\n--- a/Documentation/technical/repository-version.txt\n+++ b/Documentation/technical/repository-version.txt\n@@ -100,3 +100,10 @@ If set, by default \"git config\" reads from both \"config\" and\n multiple working directory mode, \"config\" file is shared while\n \"config.worktree\" is per-working directory (i.e., it's in\n GIT_COMMON_DIR/worktrees//config.worktree)\n+\n+==== `refStorage`\n+\n+Specifies the file format for the ref database. Values are `files`\n+(for the traditional packed + loose ref format) and `reftable` for the\n+binary reftable format. See https://github.com/google/reftable for\n+more information.\ndiff --git a/Makefile b/Makefile\nindex 278c489da4..ec91fa14f4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -978,6 +978,7 @@ LIB_OBJS += reflog-walk.o\n LIB_OBJS += refs.o\n LIB_OBJS += refs/debug.o\n LIB_OBJS += refs/files-backend.o\n+LIB_OBJS += refs/reftable-backend.o\n LIB_OBJS += refs/iterator.o\n LIB_OBJS += refs/packed-backend.o\n LIB_OBJS += refs/ref-cache.o\n@@ -2408,8 +2409,11 @@ REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n+REFTABLE_TEST_OBJS += reftable/refname_test.o\n REFTABLE_TEST_OBJS += reftable/reftable_test.o\n+REFTABLE_TEST_OBJS += reftable/stack_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex a0841923cf..974a374e9c 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1138,7 +1138,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t}\n \n \tinit_db(git_dir, real_git_dir, option_template, GIT_HASH_UNKNOWN, NULL,\n-\t\tINIT_DB_QUIET);\n+\t\tdefault_ref_storage(), INIT_DB_QUIET);\n \n \tif (real_git_dir)\n \t\tgit_dir = real_git_dir;\n@@ -1273,7 +1273,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t * Now that we know what algorithm the remote side is using,\n \t\t * let's set ours to the same thing.\n \t\t */\n-\t\tinitialize_repository_version(hash_algo, 1);\n+\t\tinitialize_repository_version(hash_algo, 1,\n+\t\t\t\t\t default_ref_storage());\n \t\trepo_set_hash_algo(the_repository, hash_algo);\n \n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 5c8c67fec6..8b5873ec56 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -179,12 +179,14 @@ static int needs_work_tree_config(const char *git_dir, const char *work_tree)\n \treturn 1;\n }\n \n-void initialize_repository_version(int hash_algo, int reinit)\n+void initialize_repository_version(int hash_algo, int reinit,\n+\t\t\t\t const char *ref_storage_format)\n {\n \tchar repo_version_string[10];\n \tint repo_version = GIT_REPO_VERSION;\n \n-\tif (hash_algo != GIT_HASH_SHA1)\n+\tif (hash_algo != GIT_HASH_SHA1 ||\n+\t !strcmp(ref_storage_format, \"reftable\"))\n \t\trepo_version = GIT_REPO_VERSION_READ;\n \n \t/* This forces creation of new config file */\n@@ -237,6 +239,7 @@ static int create_default_files(const char *template_path,\n \tis_bare_repository_cfg = init_is_bare_repository;\n \tif (init_shared_repository != -1)\n \t\tset_shared_repository(init_shared_repository);\n+\tthe_repository->ref_storage_format = xstrdup(fmt->ref_storage);\n \n \t/*\n \t * We would have created the above under user's umask -- under\n@@ -246,6 +249,24 @@ static int create_default_files(const char *template_path,\n \t\tadjust_shared_perm(get_git_dir());\n \t}\n \n+\t/*\n+\t * Check to see if .git/HEAD exists; this must happen before\n+\t * initializing the ref db, because we want to see if there is an\n+\t * existing HEAD.\n+\t */\n+\tpath = git_path_buf(&buf, \"HEAD\");\n+\treinit = (!access(path, R_OK) ||\n+\t\t readlink(path, junk, sizeof(junk) - 1) != -1);\n+\n+\t/*\n+\t * refs/heads is a file when using reftable. We can't reinitialize with\n+\t * a reftable because it will overwrite HEAD\n+\t */\n+\tif (reinit && (!strcmp(fmt->ref_storage, \"reftable\")) ==\n+\t\t\t is_directory(git_path_buf(&buf, \"refs/heads\"))) {\n+\t\tdie(\"cannot switch ref storage format.\");\n+\t}\n+\n \t/*\n \t * We need to create a \"refs\" dir in any case so that older\n \t * versions of git can tell that this is a repository.\n@@ -260,9 +281,6 @@ static int create_default_files(const char *template_path,\n \t * Point the HEAD symref to the initial branch with if HEAD does\n \t * not yet exist.\n \t */\n-\tpath = git_path_buf(&buf, \"HEAD\");\n-\treinit = (!access(path, R_OK)\n-\t\t || readlink(path, junk, sizeof(junk)-1) != -1);\n \tif (!reinit) {\n \t\tchar *ref;\n \n@@ -279,7 +297,7 @@ static int create_default_files(const char *template_path,\n \t\tfree(ref);\n \t}\n \n-\tinitialize_repository_version(fmt->hash_algo, 0);\n+\tinitialize_repository_version(fmt->hash_algo, 0, fmt->ref_storage);\n \n \t/* Check filemode trustability */\n \tpath = git_path_buf(&buf, \"config\");\n@@ -396,7 +414,7 @@ static void validate_hash_algorithm(struct repository_format *repo_fmt, int hash\n \n int init_db(const char *git_dir, const char *real_git_dir,\n \t const char *template_dir, int hash, const char *initial_branch,\n-\t unsigned int flags)\n+\t const char *ref_storage_format, unsigned int flags)\n {\n \tint reinit;\n \tint exist_ok = flags & INIT_DB_EXIST_OK;\n@@ -435,6 +453,7 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \t * is an attempt to reinitialize new repository with an old tool.\n \t */\n \tcheck_repository_format(&repo_fmt);\n+\trepo_fmt.ref_storage = xstrdup(ref_storage_format);\n \n \tvalidate_hash_algorithm(&repo_fmt, hash);\n \n@@ -467,6 +486,9 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \t\tgit_config_set(\"receive.denyNonFastforwards\", \"true\");\n \t}\n \n+\tif (!strcmp(ref_storage_format, \"reftable\"))\n+\t\tgit_config_set(\"extensions.refStorage\", ref_storage_format);\n+\n \tif (!(flags & INIT_DB_QUIET)) {\n \t\tint len = strlen(git_dir);\n \n@@ -540,6 +562,7 @@ static const char *const init_db_usage[] = {\n int cmd_init_db(int argc, const char **argv, const char *prefix)\n {\n \tconst char *git_dir;\n+\tconst char *ref_storage_format = default_ref_storage();\n \tconst char *real_git_dir = NULL;\n \tconst char *work_tree;\n \tconst char *template_dir = NULL;\n@@ -548,15 +571,18 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n \tconst char *initial_branch = NULL;\n \tint hash_algo = GIT_HASH_UNKNOWN;\n \tconst struct option init_db_options[] = {\n-\t\tOPT_STRING(0, \"template\", &template_dir, N_(\"template-directory\"),\n-\t\t\t\tN_(\"directory from which templates will be used\")),\n+\t\tOPT_STRING(0, \"template\", &template_dir,\n+\t\t\t N_(\"template-directory\"),\n+\t\t\t N_(\"directory from which templates will be used\")),\n \t\tOPT_SET_INT(0, \"bare\", &is_bare_repository_cfg,\n-\t\t\t\tN_(\"create a bare repository\"), 1),\n+\t\t\t N_(\"create a bare repository\"), 1),\n \t\t{ OPTION_CALLBACK, 0, \"shared\", &init_shared_repository,\n-\t\t\tN_(\"permissions\"),\n-\t\t\tN_(\"specify that the git repository is to be shared amongst several users\"),\n-\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0},\n+\t\t N_(\"permissions\"),\n+\t\t N_(\"specify that the git repository is to be shared amongst several users\"),\n+\t\t PARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0 },\n \t\tOPT_BIT('q', \"quiet\", &flags, N_(\"be quiet\"), INIT_DB_QUIET),\n+\t\tOPT_STRING(0, \"ref-storage\", &ref_storage_format, N_(\"backend\"),\n+\t\t\t N_(\"the ref storage format to use\")),\n \t\tOPT_STRING(0, \"separate-git-dir\", &real_git_dir, N_(\"gitdir\"),\n \t\t\t N_(\"separate git dir from working tree\")),\n \t\tOPT_STRING('b', \"initial-branch\", &initial_branch, N_(\"name\"),\n@@ -697,10 +723,11 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n \t}\n \n \tUNLEAK(real_git_dir);\n+\tUNLEAK(ref_storage_format);\n \tUNLEAK(git_dir);\n \tUNLEAK(work_tree);\n \n \tflags |= INIT_DB_EXIST_OK;\n \treturn init_db(git_dir, real_git_dir, template_dir, hash_algo,\n-\t\t initial_branch, flags);\n+\t\t initial_branch, ref_storage_format, flags);\n }\ndiff --git a/builtin/worktree.c b/builtin/worktree.c\nindex ce56fdaaa9..7931620788 100644\n--- a/builtin/worktree.c\n+++ b/builtin/worktree.c\n@@ -12,6 +12,7 @@\n #include \"submodule.h\"\n #include \"utf8.h\"\n #include \"worktree.h\"\n+#include \"../refs/refs-internal.h\"\n \n static const char * const worktree_usage[] = {\n \tN_(\"git worktree add [] []\"),\n@@ -402,9 +403,29 @@ static int add_worktree(const char *path, const char *refname,\n \t * worktree.\n \t */\n \tstrbuf_reset(&sb);\n-\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n-\twrite_file(sb.buf, \"%s\", oid_to_hex(&null_oid));\n-\tstrbuf_reset(&sb);\n+\tif (get_main_ref_store(the_repository)->be == &refs_be_reftable) {\n+\t\t/* XXX this is cut & paste from reftable_init_db. */\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"%s\", \"ref: refs/heads/.invalid\\n\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs/heads\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\t} else {\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"%s\", oid_to_hex(&null_oid));\n+\t\tstrbuf_reset(&sb);\n+\t}\n+\n \tstrbuf_addf(&sb, \"%s/commondir\", sb_repo.buf);\n \twrite_file(sb.buf, \"../..\");\n \ndiff --git a/cache.h b/cache.h\nindex e986cf4ea9..545d2b7260 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -627,9 +627,10 @@ int path_inside_repo(const char *prefix, const char *path);\n #define INIT_DB_EXIST_OK 0x0002\n \n int init_db(const char *git_dir, const char *real_git_dir,\n-\t const char *template_dir, int hash_algo,\n-\t const char *initial_branch, unsigned int flags);\n-void initialize_repository_version(int hash_algo, int reinit);\n+\t const char *template_dir, int hash_algo, const char *initial_branch,\n+\t const char *ref_storage_format, unsigned int flags);\n+void initialize_repository_version(int hash_algo, int reinit,\n+\t\t\t\t const char *ref_storage_format);\n \n void sanitize_stdfds(void);\n int daemonize(void);\n@@ -1043,6 +1044,7 @@ struct repository_format {\n \tint is_bare;\n \tint hash_algo;\n \tchar *work_tree;\n+\tchar *ref_storage;\n \tstruct string_list unknown_extensions;\n \tstruct string_list v1_only_extensions;\n };\ndiff --git a/config.mak.uname b/config.mak.uname\nindex c7eba69e54..ae4e25a1a4 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -709,7 +709,7 @@ vcxproj:\n \t# Make .vcxproj files and add them\n \tunset QUIET_GEN QUIET_BUILT_IN; \\\n \tperl contrib/buildsystems/generate -g Vcxproj\n-\tgit add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj\n+\tgit add -f git.sln {*,*/lib,*/libreftable,t/helper/*}/*.vcxproj\n \n \t# Generate the LinkOrCopyBuiltins.targets and LinkOrCopyRemoteHttp.targets file\n \t(echo '' && \\\ndiff --git a/contrib/buildsystems/Generators/Vcxproj.pm b/contrib/buildsystems/Generators/Vcxproj.pm\nindex d2584450ba..1a25789d28 100644\n--- a/contrib/buildsystems/Generators/Vcxproj.pm\n+++ b/contrib/buildsystems/Generators/Vcxproj.pm\n@@ -77,7 +77,7 @@ sub createProject {\n my $libs_release = \"\\n \";\n my $libs_debug = \"\\n \";\n if (!$static_library) {\n- $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n+ $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib|reftable\\/libreftable\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n $libs_debug = $libs_release;\n $libs_debug =~ s/zlib\\.lib/zlibd\\.lib/g;\n $libs_debug =~ s/libexpat\\.lib/libexpatd\\.lib/g;\n@@ -232,6 +232,7 @@ sub createProject {\n EOM\n if (!$static_library || $target =~ 'vcs-svn' || $target =~ 'xdiff') {\n my $uuid_libgit = $$build_structure{\"LIBS_libgit_GUID\"};\n+ my $uuid_libreftable = $$build_structure{\"LIBS_reftable/libreftable_GUID\"};\n my $uuid_xdiff_lib = $$build_structure{\"LIBS_xdiff/lib_GUID\"};\n \n print F << \"EOM\";\n@@ -241,6 +242,14 @@ sub createProject {\n false\n \n EOM\n+ if (!($name =~ /xdiff|libreftable/)) {\n+ print F << \"EOM\";\n+ \n+ $uuid_libreftable\n+ false\n+ \n+EOM\n+ }\n if (!($name =~ 'xdiff')) {\n print F << \"EOM\";\n \ndiff --git a/refs.c b/refs.c\nindex 392f0bbf68..1b874db334 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -19,10 +19,16 @@\n #include \"repository.h\"\n #include \"sigchain.h\"\n \n+const char *default_ref_storage(void)\n+{\n+\tconst char *test = getenv(\"GIT_TEST_REFTABLE\");\n+\treturn test ? \"reftable\" : \"files\";\n+}\n+\n /*\n * List of all available backends\n */\n-static struct ref_storage_be *refs_backends = &refs_be_files;\n+static struct ref_storage_be *refs_backends = &refs_be_reftable;\n \n static struct ref_storage_be *find_ref_storage_backend(const char *name)\n {\n@@ -1754,13 +1760,13 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,\n * Create, record, and return a ref_store instance for the specified\n * gitdir.\n */\n-static struct ref_store *ref_store_init(const char *gitdir,\n+static struct ref_store *ref_store_init(const char *gitdir, const char *be_name,\n \t\t\t\t\tunsigned int flags)\n {\n-\tconst char *be_name = \"files\";\n-\tstruct ref_storage_be *be = find_ref_storage_backend(be_name);\n+\tstruct ref_storage_be *be;\n \tstruct ref_store *refs;\n \n+\tbe = find_ref_storage_backend(be_name);\n \tif (!be)\n \t\tBUG(\"reference backend %s is unknown\", be_name);\n \n@@ -1776,7 +1782,11 @@ struct ref_store *get_main_ref_store(struct repository *r)\n \tif (!r->gitdir)\n \t\tBUG(\"attempting to get main_ref_store outside of repository\");\n \n-\tr->refs_private = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);\n+\tr->refs_private = ref_store_init(r->gitdir,\n+\t\t\t\t\t r->ref_storage_format ?\n+\t\t\t\t\t\t r->ref_storage_format :\n+\t\t\t\t\t\t default_ref_storage(),\n+\t\t\t\t\t REF_STORE_ALL_CAPS);\n \tr->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private);\n \treturn r->refs_private;\n }\n@@ -1832,7 +1842,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \t\tgoto done;\n \n \t/* assume that add_submodule_odb() has been called */\n-\trefs = ref_store_init(submodule_sb.buf,\n+\trefs = ref_store_init(submodule_sb.buf, default_ref_storage(),\n \t\t\t REF_STORE_READ | REF_STORE_ODB);\n \tregister_ref_store_map(&submodule_ref_stores, \"submodule\",\n \t\t\t refs, submodule);\n@@ -1846,6 +1856,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \n struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n {\n+\tconst char *format = default_ref_storage();\n \tstruct ref_store *refs;\n \tconst char *id;\n \n@@ -1859,9 +1870,9 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n \n \tif (wt->id)\n \t\trefs = ref_store_init(git_common_path(\"worktrees/%s\", wt->id),\n-\t\t\t\t REF_STORE_ALL_CAPS);\n+\t\t\t\t format, REF_STORE_ALL_CAPS);\n \telse\n-\t\trefs = ref_store_init(get_git_common_dir(),\n+\t\trefs = ref_store_init(get_git_common_dir(), format,\n \t\t\t\t REF_STORE_ALL_CAPS);\n \n \tif (refs)\ndiff --git a/refs.h b/refs.h\nindex 6695518156..7dc60472c9 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -11,6 +11,9 @@ struct string_list;\n struct string_list_item;\n struct worktree;\n \n+/* Returns the ref storage backend to use by default. */\n+const char *default_ref_storage(void);\n+\n /*\n * Resolve a reference, recursively following symbolic refererences.\n *\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 467f4b3c93..28166bf1f8 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -669,6 +669,7 @@ struct ref_storage_be {\n };\n \n extern struct ref_storage_be refs_be_files;\n+extern struct ref_storage_be refs_be_reftable;\n extern struct ref_storage_be refs_be_packed;\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nnew file mode 100644\nindex 0000000000..894c72ddc4\n--- /dev/null\n+++ b/refs/reftable-backend.c\n@@ -0,0 +1,1418 @@\n+#include \"../cache.h\"\n+#include \"../chdir-notify.h\"\n+#include \"../config.h\"\n+#include \"../iterator.h\"\n+#include \"../lockfile.h\"\n+#include \"../refs.h\"\n+#include \"../reftable/reftable-stack.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-blocksource.h\"\n+#include \"../reftable/reftable-reader.h\"\n+#include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-merged.h\"\n+#include \"../reftable/reftable-generic.h\"\n+#include \"../worktree.h\"\n+#include \"refs-internal.h\"\n+\n+extern struct ref_storage_be refs_be_reftable;\n+\n+struct git_reftable_ref_store {\n+\tstruct ref_store base;\n+\tunsigned int store_flags;\n+\n+\tint err;\n+\tchar *repo_dir;\n+\n+\tchar *reftable_dir;\n+\tchar *worktree_reftable_dir;\n+\n+\tstruct reftable_stack *main_stack;\n+\tstruct reftable_stack *worktree_stack;\n+};\n+\n+static struct reftable_stack *stack_for(struct git_reftable_ref_store *store,\n+\t\t\t\t\tconst char *refname)\n+{\n+\tif (store->worktree_stack == NULL)\n+\t\treturn store->main_stack;\n+\n+\tswitch (ref_type(refname)) {\n+\tcase REF_TYPE_PER_WORKTREE:\n+\tcase REF_TYPE_PSEUDOREF:\n+\tcase REF_TYPE_OTHER_PSEUDOREF:\n+\t\treturn store->worktree_stack;\n+\tdefault:\n+\tcase REF_TYPE_MAIN_PSEUDOREF:\n+\tcase REF_TYPE_NORMAL:\n+\t\treturn store->main_stack;\n+\t}\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type);\n+\n+static void clear_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tlog->old_hash = NULL;\n+\tlog->new_hash = NULL;\n+\tlog->message = NULL;\n+\tlog->refname = NULL;\n+\treftable_log_record_release(log);\n+}\n+\n+static void fill_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tconst char *info = git_committer_info(0);\n+\tstruct ident_split split = { NULL };\n+\tint result = split_ident_line(&split, info, strlen(info));\n+\tint sign = 1;\n+\tassert(0 == result);\n+\n+\treftable_log_record_release(log);\n+\tlog->name =\n+\t\txstrndup(split.name_begin, split.name_end - split.name_begin);\n+\tlog->email =\n+\t\txstrndup(split.mail_begin, split.mail_end - split.mail_begin);\n+\tlog->time = atol(split.date_begin);\n+\tif (*split.tz_begin == '-') {\n+\t\tsign = -1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\tif (*split.tz_begin == '+') {\n+\t\tsign = 1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\n+\tlog->tz_offset = sign * atoi(split.tz_begin);\n+}\n+\n+static int has_suffix(struct strbuf *b, const char *suffix)\n+{\n+\tsize_t len = strlen(suffix);\n+\n+\tif (len > b->len) {\n+\t\treturn 0;\n+\t}\n+\n+\treturn 0 == strncmp(b->buf + b->len - len, suffix, len);\n+}\n+\n+/* trims the last path component of b. Returns -1 if it is not\n+ * present, or 0 on success\n+ */\n+static int trim_component(struct strbuf *b)\n+{\n+\tchar *last;\n+\tlast = strrchr(b->buf, '/');\n+\tif (!last)\n+\t\treturn -1;\n+\tstrbuf_setlen(b, last - b->buf);\n+\treturn 0;\n+}\n+\n+/* Returns whether `b` is a worktree path, trimming it to the gitdir\n+ */\n+static int is_worktree(struct strbuf *b)\n+{\n+\tif (trim_component(b) < 0) {\n+\t\treturn 0;\n+\t}\n+\tif (!has_suffix(b, \"/worktrees\")) {\n+\t\treturn 0;\n+\t}\n+\ttrim_component(b);\n+\treturn 1;\n+}\n+\n+static struct ref_store *git_reftable_ref_store_create(const char *path,\n+\t\t\t\t\t\t unsigned int store_flags)\n+{\n+\tstruct git_reftable_ref_store *refs = xcalloc(1, sizeof(*refs));\n+\tstruct ref_store *ref_store = (struct ref_store *)refs;\n+\tstruct reftable_write_options cfg = {\n+\t\t.block_size = 4096,\n+\t\t.hash_id = the_hash_algo->format_id,\n+\t};\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tconst char *gitdir = path;\n+\tstruct strbuf wt_buf = STRBUF_INIT;\n+\tint wt = 0;\n+\n+\tstrbuf_addstr(&wt_buf, path);\n+\n+\t/* this is clumsy, but the official worktree functions (eg.\n+\t * get_worktrees()) function will try to initialize a ref storage\n+\t * backend, leading to infinite recursion. */\n+\twt = is_worktree(&wt_buf);\n+\tif (wt) {\n+\t\tgitdir = wt_buf.buf;\n+\t}\n+\n+\tbase_ref_store_init(ref_store, &refs_be_reftable);\n+\tref_store->gitdir = xstrdup(gitdir);\n+\trefs->store_flags = store_flags;\n+\tstrbuf_addf(&sb, \"%s/reftable\", gitdir);\n+\trefs->reftable_dir = xstrdup(sb.buf);\n+\tstrbuf_reset(&sb);\n+\n+\trefs->err =\n+\t\treftable_new_stack(&refs->main_stack, refs->reftable_dir, cfg);\n+\tassert(refs->err != REFTABLE_API_ERROR);\n+\n+\tif (refs->err == 0 && wt) {\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", path);\n+\t\trefs->worktree_reftable_dir = xstrdup(sb.buf);\n+\n+\t\trefs->err = reftable_new_stack(&refs->worktree_stack,\n+\t\t\t\t\t refs->worktree_reftable_dir,\n+\t\t\t\t\t cfg);\n+\t\tassert(refs->err != REFTABLE_API_ERROR);\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\tstrbuf_release(&wt_buf);\n+\treturn ref_store;\n+}\n+\n+static int git_reftable_init_db(struct ref_store *ref_store, struct strbuf *err)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\tsafe_create_dir(refs->reftable_dir, 1);\n+\tassert(refs->worktree_reftable_dir == NULL);\n+\n+\tstrbuf_addf(&sb, \"%s/HEAD\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"ref: refs/heads/.invalid\");\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs\", refs->base.gitdir);\n+\tsafe_create_dir(sb.buf, 1);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs/heads\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\n+\treturn 0;\n+}\n+\n+struct git_reftable_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_ref_record ref;\n+\tstruct object_id oid;\n+\tstruct ref_store *ref_store;\n+\n+\t/* In case we must iterate over 2 stacks, this is non-null. */\n+\tstruct reftable_merged_table *merged;\n+\tunsigned int flags;\n+\tint err;\n+\tconst char *prefix;\n+};\n+\n+static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\twhile (ri->err == 0) {\n+\t\tri->err = reftable_iterator_next_ref(&ri->iter, &ri->ref);\n+\t\tif (ri->err) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (ref_type(ri->ref.refname) == REF_TYPE_PSEUDOREF) {\n+\t\t\t/*\n+\t\t\t pseudorefs, eg. HEAD, FETCH_HEAD should not be\n+\t\t\t produced, by default.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\t\tri->base.refname = ri->ref.refname;\n+\t\tif (ri->prefix != NULL &&\n+\t\t strncmp(ri->prefix, ri->ref.refname, strlen(ri->prefix))) {\n+\t\t\tri->err = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (ri->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n+\t\t ref_type(ri->base.refname) != REF_TYPE_PER_WORKTREE)\n+\t\t\tcontinue;\n+\n+\t\tri->base.flags = 0;\n+\t\tif (ri->ref.value != NULL) {\n+\t\t\thashcpy(ri->oid.hash, ri->ref.value);\n+\t\t} else if (ri->ref.target != NULL) {\n+\t\t\tint out_flags = 0;\n+\t\t\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\t\t\tri->ref_store, ri->ref.refname,\n+\t\t\t\tRESOLVE_REF_READING, &ri->oid, &out_flags);\n+\t\t\tri->base.flags = out_flags;\n+\t\t\tif (resolved == NULL &&\n+\t\t\t !(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t\t (ri->base.flags & REF_ISBROKEN)) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tri->base.oid = &ri->oid;\n+\t\tif (!(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t !ref_resolves_to_object(ri->base.refname, ri->base.oid,\n+\t\t\t\t\t ri->base.flags)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tbreak;\n+\t}\n+\n+\tif (ri->err > 0) {\n+\t\treturn ITER_DONE;\n+\t}\n+\tif (ri->err < 0) {\n+\t\treturn ITER_ERROR;\n+\t}\n+\n+\treturn ITER_OK;\n+}\n+\n+static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\tif (ri->ref.target_value != NULL) {\n+\t\thashcpy(peeled->hash, ri->ref.target_value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\n+\n+static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\treftable_ref_record_release(&ri->ref);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged) {\n+\t\treftable_merged_table_free(ri->merged);\n+\t}\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n+\treftable_ref_iterator_advance, reftable_ref_iterator_peel,\n+\treftable_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,\n+\t\t\t\tunsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct git_reftable_iterator *ri = xcalloc(1, sizeof(*ri));\n+\n+\tif (refs->err < 0) {\n+\t\tri->err = refs->err;\n+\t} else if (refs->worktree_stack == NULL) {\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tri->err = reftable_merged_table_seek_ref(mt, &ri->iter, prefix);\n+\t} else {\n+\t\tstruct reftable_merged_table *mt1 =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tstruct reftable_merged_table *mt2 =\n+\t\t\treftable_stack_merged_table(refs->worktree_stack);\n+\t\tstruct reftable_table *tabs =\n+\t\t\txcalloc(2, sizeof(struct reftable_table));\n+\t\treftable_table_from_merged_table(&tabs[0], mt1);\n+\t\treftable_table_from_merged_table(&tabs[1], mt2);\n+\t\tri->err = reftable_new_merged_table(&ri->merged, tabs, 2,\n+\t\t\t\t\t\t the_hash_algo->format_id);\n+\t\tif (ri->err == 0)\n+\t\t\tri->err = reftable_merged_table_seek_ref(\n+\t\t\t\tri->merged, &ri->iter, prefix);\n+\t}\n+\n+\tbase_ref_iterator_init(&ri->base, &reftable_ref_iterator_vtable, 1);\n+\tri->prefix = prefix;\n+\tri->base.oid = &ri->oid;\n+\tri->flags = flags;\n+\tri->ref_store = ref_store;\n+\treturn &ri->base;\n+}\n+\n+static int fixup_symrefs(struct ref_store *ref_store,\n+\t\t\t struct ref_transaction *transaction)\n+{\n+\tstruct strbuf referent = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\t\tstruct object_id old_oid;\n+\n+\t\terr = git_reftable_read_raw_ref(ref_store, update->refname,\n+\t\t\t\t\t\t&old_oid, &referent,\n+\t\t\t\t\t\t/* mutate input, like\n+\t\t\t\t\t\t files-backend.c */\n+\t\t\t\t\t\t&update->type);\n+\t\tif (err < 0 && errno == ENOENT &&\n+\t\t is_null_oid(&update->old_oid)) {\n+\t\t\terr = 0;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (!(update->type & REF_ISSYMREF))\n+\t\t\tcontinue;\n+\n+\t\tif (update->flags & REF_NO_DEREF) {\n+\t\t\t/* what should happen here? See files-backend.c\n+\t\t\t * lock_ref_for_update. */\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t If we are updating a symref (eg. HEAD), we should also\n+\t\t\t update the branch that the symref points to.\n+\n+\t\t\t This is generic functionality, and would be better\n+\t\t\t done in refs.c, but the current implementation is\n+\t\t\t intertwined with the locking in files-backend.c.\n+\t\t\t*/\n+\t\t\tint new_flags = update->flags;\n+\t\t\tstruct ref_update *new_update = NULL;\n+\n+\t\t\t/* if this is an update for HEAD, should also record a\n+\t\t\t log entry for HEAD? See files-backend.c,\n+\t\t\t split_head_update()\n+\t\t\t*/\n+\t\t\tnew_update = ref_transaction_add_update(\n+\t\t\t\ttransaction, referent.buf, new_flags,\n+\t\t\t\t&update->new_oid, &update->old_oid,\n+\t\t\t\tupdate->msg);\n+\t\t\tnew_update->parent_update = update;\n+\n+\t\t\t/* files-backend sets REF_LOG_ONLY here. */\n+\t\t\tupdate->flags |= REF_NO_DEREF | REF_LOG_ONLY;\n+\t\t\tupdate->flags &= ~REF_HAVE_OLD;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tstrbuf_release(&referent);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_prepare(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errbuf)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_addition *add = NULL;\n+\tstruct reftable_stack *stack =\n+\t\ttransaction->nr ?\n+\t\t\t stack_for(refs, transaction->updates[0]->refname) :\n+\t\t\t refs->main_stack;\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_new_addition(&add, stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = fixup_symrefs(ref_store, transaction);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\ttransaction->backend_data = add;\n+\ttransaction->state = REF_TRANSACTION_PREPARED;\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tif (err < 0) {\n+\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\t\tstrbuf_addf(errbuf, \"reftable: transaction prepare: %s\",\n+\t\t\t reftable_error_str(err));\n+\t}\n+\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_abort(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\treftable_addition_destroy(add);\n+\ttransaction->backend_data = NULL;\n+\treturn 0;\n+}\n+\n+static int reftable_check_old_oid(struct ref_store *refs, const char *refname,\n+\t\t\t\t struct object_id *want_oid)\n+{\n+\tstruct object_id out_oid;\n+\tint out_flags = 0;\n+\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\trefs, refname, RESOLVE_REF_READING, &out_oid, &out_flags);\n+\tif (is_null_oid(want_oid) != (resolved == NULL)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\tif (resolved != NULL && !oideq(&out_oid, want_oid)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int ref_update_cmp(const void *a, const void *b)\n+{\n+\treturn strcmp((*(struct ref_update **)a)->refname,\n+\t\t (*(struct ref_update **)b)->refname);\n+}\n+\n+static int write_transaction_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct ref_transaction *transaction = (struct ref_transaction *)arg;\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)transaction->ref_store;\n+\tstruct reftable_stack *stack =\n+\t\tstack_for(refs, transaction->updates[0]->refname);\n+\tuint64_t ts = reftable_stack_next_update_index(stack);\n+\tint err = 0;\n+\tint i = 0;\n+\tstruct reftable_log_record *logs =\n+\t\tcalloc(transaction->nr, sizeof(*logs));\n+\tstruct ref_update **sorted =\n+\t\tmalloc(transaction->nr * sizeof(struct ref_update *));\n+\tCOPY_ARRAY(sorted, transaction->updates, transaction->nr);\n+\tQSORT(sorted, transaction->nr, ref_update_cmp);\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = sorted[i];\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tfill_reftable_log_record(log);\n+\t\tlog->refname = (char *)u->refname;\n+\t\tlog->old_hash = u->old_oid.hash;\n+\t\tlog->new_hash = u->new_oid.hash;\n+\t\tlog->update_index = ts;\n+\t\tlog->message = u->msg;\n+\n+\t\tif (u->flags & REF_LOG_ONLY) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (u->flags & REF_HAVE_NEW) {\n+\t\t\tstruct reftable_ref_record ref = { NULL };\n+\t\t\tstruct object_id peeled;\n+\n+\t\t\tint peel_error = peel_object(&u->new_oid, &peeled);\n+\t\t\tref.refname = (char *)u->refname;\n+\n+\t\t\tif (!is_null_oid(&u->new_oid)) {\n+\t\t\t\tref.value = u->new_oid.hash;\n+\t\t\t}\n+\t\t\tref.update_index = ts;\n+\t\t\tif (!peel_error) {\n+\t\t\t\tref.target_value = peeled.hash;\n+\t\t\t}\n+\n+\t\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\terr = reftable_writer_add_log(writer, &logs[i]);\n+\t\tclear_reftable_log_record(&logs[i]);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tfree(logs);\n+\tfree(sorted);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_finish(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errmsg)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\tint err = 0;\n+\tint i;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = transaction->updates[i];\n+\t\tif (u->flags & REF_HAVE_OLD) {\n+\t\t\terr = reftable_check_old_oid(transaction->ref_store,\n+\t\t\t\t\t\t u->refname, &u->old_oid);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (transaction->nr) {\n+\t\terr = reftable_addition_add(add, &write_transaction_table,\n+\t\t\t\t\t transaction);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\terr = reftable_addition_commit(add);\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_addition_destroy(add);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\ttransaction->backend_data = NULL;\n+\tif (err) {\n+\t\tstrbuf_addf(errmsg, \"reftable: transaction failure: %s\",\n+\t\t\t reftable_error_str(err));\n+\t\treturn -1;\n+\t}\n+\treturn err;\n+}\n+\n+static int\n+git_reftable_transaction_initial_commit(struct ref_store *ref_store,\n+\t\t\t\t\tstruct ref_transaction *transaction,\n+\t\t\t\t\tstruct strbuf *errmsg)\n+{\n+\tint err = git_reftable_transaction_prepare(ref_store, transaction,\n+\t\t\t\t\t\t errmsg);\n+\tif (err)\n+\t\treturn err;\n+\n+\treturn git_reftable_transaction_finish(ref_store, transaction, errmsg);\n+}\n+\n+struct write_delete_refs_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct string_list *refnames;\n+\tconst char *logmsg;\n+\tunsigned int flags;\n+};\n+\n+static int write_delete_refs_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_delete_refs_arg *arg =\n+\t\t(struct write_delete_refs_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint err = 0;\n+\tint i = 0;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = (char *)arg->refnames->items[i].string,\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_log_record log = {\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\tstruct reftable_ref_record current = { NULL };\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.message = xstrdup(arg->logmsg);\n+\t\tlog.new_hash = NULL;\n+\t\tlog.old_hash = NULL;\n+\t\tlog.update_index = ts;\n+\t\tlog.refname = (char *)arg->refnames->items[i].string;\n+\n+\t\tif (reftable_stack_read_ref(arg->stack, log.refname,\n+\t\t\t\t\t ¤t) == 0) {\n+\t\t\tlog.old_hash = current.value;\n+\t\t}\n+\t\terr = reftable_writer_add_log(writer, &log);\n+\t\tlog.old_hash = NULL;\n+\t\treftable_ref_record_release(¤t);\n+\n+\t\tclear_reftable_log_record(&log);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int git_reftable_delete_refs(struct ref_store *ref_store,\n+\t\t\t\t const char *msg,\n+\t\t\t\t struct string_list *refnames,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack =\n+\t\tstack_for(refs, refnames->items[0].string);\n+\tstruct write_delete_refs_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refnames = refnames,\n+\t\t.logmsg = msg,\n+\t\t.flags = flags,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tstring_list_sort(refnames);\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_delete_refs_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_pack_refs(struct ref_store *ref_store,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\terr = reftable_stack_compact_all(refs->main_stack, NULL);\n+\tif (err == 0 && refs->worktree_stack != NULL)\n+\t\terr = reftable_stack_compact_all(refs->worktree_stack, NULL);\n+\treturn err;\n+}\n+\n+struct write_create_symref_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+\tconst char *target;\n+\tconst char *logmsg;\n+};\n+\n+static int write_create_symref_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct write_create_symref_arg *create =\n+\t\t(struct write_create_symref_arg *)arg;\n+\tuint64_t ts = reftable_stack_next_update_index(create->stack);\n+\tint err = 0;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)create->refname,\n+\t\t.target = (char *)create->target,\n+\t\t.update_index = ts,\n+\t};\n+\treftable_writer_set_limits(writer, ts, ts);\n+\terr = reftable_writer_add_ref(writer, &ref);\n+\tif (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tstruct object_id new_oid;\n+\t\tstruct object_id old_oid;\n+\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.refname = (char *)create->refname;\n+\t\tlog.message = (char *)create->logmsg;\n+\t\tlog.update_index = ts;\n+\t\tif (refs_resolve_ref_unsafe(\n+\t\t\t (struct ref_store *)create->refs, create->refname,\n+\t\t\t RESOLVE_REF_READING, &old_oid, NULL) != NULL) {\n+\t\t\tlog.old_hash = old_oid.hash;\n+\t\t}\n+\n+\t\tif (refs_resolve_ref_unsafe((struct ref_store *)create->refs,\n+\t\t\t\t\t create->target, RESOLVE_REF_READING,\n+\t\t\t\t\t &new_oid, NULL) != NULL) {\n+\t\t\tlog.new_hash = new_oid.hash;\n+\t\t}\n+\n+\t\tif (log.old_hash != NULL || log.new_hash != NULL) {\n+\t\t\terr = reftable_writer_add_log(writer, &log);\n+\t\t}\n+\t\tlog.refname = NULL;\n+\t\tlog.message = NULL;\n+\t\tlog.old_hash = NULL;\n+\t\tlog.new_hash = NULL;\n+\t\tclear_reftable_log_record(&log);\n+\t}\n+\treturn err;\n+}\n+\n+static int git_reftable_create_symref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, const char *target,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_create_symref_arg arg = { .refs = refs,\n+\t\t\t\t\t .stack = stack,\n+\t\t\t\t\t .refname = refname,\n+\t\t\t\t\t .target = target,\n+\t\t\t\t\t .logmsg = logmsg };\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_create_symref_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct write_rename_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *oldname;\n+\tconst char *newname;\n+\tconst char *logmsg;\n+};\n+\n+static int write_rename_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &ref);\n+\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* XXX do ref renames overwrite the target? */\n+\tif (reftable_stack_read_ref(arg->stack, arg->newname, &ref) == 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tfree(ref.refname);\n+\tref.refname = strdup(arg->newname);\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tref.update_index = ts;\n+\n+\t{\n+\t\tstruct reftable_ref_record todo[2] = { { NULL } };\n+\t\ttodo[0].refname = (char *)arg->oldname;\n+\t\ttodo[0].update_index = ts;\n+\t\t/* leave todo[0] empty */\n+\t\ttodo[1] = ref;\n+\t\ttodo[1].update_index = ts;\n+\n+\t\terr = reftable_writer_add_refs(writer, todo, 2);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (ref.value != NULL) {\n+\t\tstruct reftable_log_record todo[2] = { { NULL } };\n+\t\tfill_reftable_log_record(&todo[0]);\n+\t\tfill_reftable_log_record(&todo[1]);\n+\n+\t\ttodo[0].refname = (char *)arg->oldname;\n+\t\ttodo[0].update_index = ts;\n+\t\ttodo[0].message = (char *)arg->logmsg;\n+\t\ttodo[0].old_hash = ref.value;\n+\t\ttodo[0].new_hash = NULL;\n+\n+\t\ttodo[1].refname = (char *)arg->newname;\n+\t\ttodo[1].update_index = ts;\n+\t\ttodo[1].old_hash = NULL;\n+\t\ttodo[1].new_hash = ref.value;\n+\t\ttodo[1].message = (char *)arg->logmsg;\n+\n+\t\terr = reftable_writer_add_logs(writer, todo, 2);\n+\n+\t\tclear_reftable_log_record(&todo[0]);\n+\t\tclear_reftable_log_record(&todo[1]);\n+\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t} else {\n+\t\t/* XXX symrefs? */\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+static int git_reftable_rename_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname,\n+\t\t\t\t const char *newrefname, const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n+\tstruct write_rename_arg arg = {\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_rename_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_copy_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname, const char *newrefname,\n+\t\t\t\t const char *logmsg)\n+{\n+\tBUG(\"reftable reference store does not support copying references\");\n+}\n+\n+struct git_reftable_reflog_ref_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_log_record log;\n+\tstruct object_id oid;\n+\n+\t/* Used when iterating over worktree & main */\n+\tstruct reftable_merged_table *merged;\n+\tchar *last_name;\n+};\n+\n+static int\n+git_reftable_reflog_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&ri->iter, &ri->log);\n+\t\tif (err > 0) {\n+\t\t\treturn ITER_DONE;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn ITER_ERROR;\n+\t\t}\n+\n+\t\tri->base.refname = ri->log.refname;\n+\t\tif (ri->last_name != NULL &&\n+\t\t !strcmp(ri->log.refname, ri->last_name)) {\n+\t\t\t/* we want the refnames that we have reflogs for, so we\n+\t\t\t * skip if we've already produced this name. This could\n+\t\t\t * be faster by seeking directly to\n+\t\t\t * reflog@update_index==0.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfree(ri->last_name);\n+\t\tri->last_name = xstrdup(ri->log.refname);\n+\t\thashcpy(ri->oid.hash, ri->log.new_hash);\n+\t\treturn ITER_OK;\n+\t}\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tBUG(\"not supported.\");\n+\treturn -1;\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\treftable_log_record_release(&ri->log);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged)\n+\t\treftable_merged_table_free(ri->merged);\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable git_reftable_reflog_ref_iterator_vtable = {\n+\tgit_reftable_reflog_ref_iterator_advance,\n+\tgit_reftable_reflog_ref_iterator_peel,\n+\tgit_reftable_reflog_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_reflog_iterator_begin(struct ref_store *ref_store)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri = xcalloc(sizeof(*ri), 1);\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\n+\tif (refs->worktree_stack == NULL) {\n+\t\tstruct reftable_stack *stack = refs->main_stack;\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(stack);\n+\t\tint err = reftable_merged_table_seek_log(mt, &ri->iter, \"\");\n+\t\tif (err < 0) {\n+\t\t\tfree(ri);\n+\t\t\t/* XXX is this allowed? */\n+\t\t\treturn NULL;\n+\t\t}\n+\t} else {\n+\t\tstruct reftable_merged_table *mt1 =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tstruct reftable_merged_table *mt2 =\n+\t\t\treftable_stack_merged_table(refs->worktree_stack);\n+\t\tstruct reftable_table *tabs =\n+\t\t\txcalloc(2, sizeof(struct reftable_table));\n+\t\tint err = 0;\n+\t\treftable_table_from_merged_table(&tabs[0], mt1);\n+\t\treftable_table_from_merged_table(&tabs[1], mt2);\n+\t\terr = reftable_new_merged_table(&ri->merged, tabs, 2,\n+\t\t\t\t\t\tthe_hash_algo->format_id);\n+\t\tif (err < 0) {\n+\t\t\tfree(tabs);\n+\t\t\t/* XXX see above */\n+\t\t\treturn NULL;\n+\t\t}\n+\t\terr = reftable_merged_table_seek_ref(ri->merged, &ri->iter, \"\");\n+\t\tif (err < 0) {\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\tbase_ref_iterator_init(&ri->base,\n+\t\t\t &git_reftable_reflog_ref_iterator_vtable, 1);\n+\tri->base.oid = &ri->oid;\n+\n+\treturn (struct ref_iterator *)ri;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_newest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (err == 0) {\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\thashcpy(old_oid.hash, log.old_hash);\n+\t\thashcpy(new_oid.hash, log.new_hash);\n+\n+\t\tfull_committer = fmt_ident(log.name, log.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT,\n+\t\t\t\t\t /*date*/ NULL, IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer, log.time,\n+\t\t\t log.tz_offset, log.message, cb_data);\n+\t\tif (err)\n+\t\t\tbreak;\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_oldest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record *logs = NULL;\n+\tint cap = 0;\n+\tint len = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\n+\twhile (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tlogs = realloc(logs, cap * sizeof(*logs));\n+\t\t}\n+\n+\t\tlogs[len++] = log;\n+\t}\n+\n+\tfor (i = len; i--;) {\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\thashcpy(old_oid.hash, log->old_hash);\n+\t\thashcpy(new_oid.hash, log->new_hash);\n+\n+\t\tfull_committer = fmt_ident(log->name, log->email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT, NULL,\n+\t\t\t\t\t IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer, log->time,\n+\t\t\t log->tz_offset, log->message, cb_data);\n+\t\tif (err) {\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tfree(logs);\n+\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_reflog_exists(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record log = { NULL };\n+\tint err = refs->err;\n+\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (strcmp(log.refname, refname)) {\n+\t\terr = 1;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\treturn !err;\n+}\n+\n+static int git_reftable_create_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, int force_create,\n+\t\t\t\t struct strbuf *err)\n+{\n+\treturn 0;\n+}\n+\n+static int git_reftable_delete_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\treturn 0;\n+}\n+\n+struct reflog_expiry_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_log_record *tombstones;\n+\tint len;\n+\tint cap;\n+};\n+\n+static void clear_log_tombstones(struct reflog_expiry_arg *arg)\n+{\n+\tint i = 0;\n+\tfor (; i < arg->len; i++) {\n+\t\treftable_log_record_release(&arg->tombstones[i]);\n+\t}\n+\n+\tFREE_AND_NULL(arg->tombstones);\n+}\n+\n+static void add_log_tombstone(struct reflog_expiry_arg *arg,\n+\t\t\t const char *refname, uint64_t ts)\n+{\n+\tstruct reftable_log_record tombstone = {\n+\t\t.refname = xstrdup(refname),\n+\t\t.update_index = ts,\n+\t};\n+\tif (arg->len == arg->cap) {\n+\t\targ->cap = 2 * arg->cap + 1;\n+\t\targ->tombstones =\n+\t\t\trealloc(arg->tombstones, arg->cap * sizeof(tombstone));\n+\t}\n+\targ->tombstones[arg->len++] = tombstone;\n+}\n+\n+static int write_reflog_expiry_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct reflog_expiry_arg *arg = (struct reflog_expiry_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint i = 0;\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->len; i++) {\n+\t\tint err = reftable_writer_add_log(writer, &arg->tombstones[i]);\n+\t\tif (err) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int\n+git_reftable_reflog_expire(struct ref_store *ref_store, const char *refname,\n+\t\t\t const struct object_id *oid, unsigned int flags,\n+\t\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t void *policy_cb_data)\n+{\n+\t/*\n+\t For log expiry, we write tombstones in place of the expired entries,\n+\t This means that the entries are still retrievable by delving into the\n+\t stack, and expiring entries paradoxically takes extra memory.\n+\n+\t This memory is only reclaimed when some operation issues a\n+\t git_reftable_pack_refs(), which will compact the entire stack and get\n+\t rid of deletion entries.\n+\n+\t It would be better if the refs backend supported an API that sets a\n+\t criterion for all refs, passing the criterion to pack_refs().\n+\t*/\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reflog_expiry_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refs = refs,\n+\t};\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = 0;\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\twhile (1) {\n+\t\tstruct object_id ooid;\n+\t\tstruct object_id noid;\n+\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (err > 0 || strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\t\thashcpy(ooid.hash, log.old_hash);\n+\t\thashcpy(noid.hash, log.new_hash);\n+\n+\t\tif (should_prune_fn(&ooid, &noid, log.email,\n+\t\t\t\t (timestamp_t)log.time, log.tz_offset,\n+\t\t\t\t log.message, policy_cb_data)) {\n+\t\t\tadd_log_tombstone(&arg, refname, log.update_index);\n+\t\t}\n+\t}\n+\terr = reftable_stack_add(stack, &write_reflog_expiry_table, &arg);\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\tclear_log_tombstones(&arg);\n+\treturn err;\n+}\n+\n+static int reftable_error_to_errno(int err)\n+{\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn EIO;\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn EFAULT;\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn ENOENT;\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn EBUSY;\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn EINVAL;\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn EDOM;\n+\tdefault:\n+\t\treturn ERANGE;\n+\t}\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\n+\t/* This is usually not needed, but Git doesn't signal to ref backend if\n+\t a subprocess updated the ref DB. So we always check.\n+\t*/\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (err > 0) {\n+\t\terrno = ENOENT;\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0) {\n+\t\terrno = reftable_error_to_errno(err);\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\tif (ref.target != NULL) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.target);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (ref.value != NULL) {\n+\t\thashcpy(oid->hash, ref.value);\n+\t} else {\n+\t\t*type |= REF_ISBROKEN;\n+\t\terrno = EINVAL;\n+\t\terr = -1;\n+\t}\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+struct ref_storage_be refs_be_reftable = {\n+\t&refs_be_files,\n+\t\"reftable\",\n+\tgit_reftable_ref_store_create,\n+\tgit_reftable_init_db,\n+\tgit_reftable_transaction_prepare,\n+\tgit_reftable_transaction_finish,\n+\tgit_reftable_transaction_abort,\n+\tgit_reftable_transaction_initial_commit,\n+\n+\tgit_reftable_pack_refs,\n+\tgit_reftable_create_symref,\n+\tgit_reftable_delete_refs,\n+\tgit_reftable_rename_ref,\n+\tgit_reftable_copy_ref,\n+\n+\tgit_reftable_ref_iterator_begin,\n+\tgit_reftable_read_raw_ref,\n+\n+\tgit_reftable_reflog_iterator_begin,\n+\tgit_reftable_for_each_reflog_ent_oldest_first,\n+\tgit_reftable_for_each_reflog_ent_newest_first,\n+\tgit_reftable_reflog_exists,\n+\tgit_reftable_create_reflog,\n+\tgit_reftable_delete_reflog,\n+\tgit_reftable_reflog_expire,\n+};\ndiff --git a/repository.c b/repository.c\nindex a4174ddb06..ff0988dac8 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -174,6 +174,8 @@ int repo_init(struct repository *repo,\n \tif (worktree)\n \t\trepo_set_worktree(repo, worktree);\n \n+\trepo->ref_storage_format = xstrdup_or_null(format.ref_storage);\n+\n \tclear_repository_format(&format);\n \treturn 0;\n \ndiff --git a/repository.h b/repository.h\nindex b385ca3c94..8019a7d0a1 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -78,6 +78,9 @@ struct repository {\n \t */\n \tstruct ref_store *refs_private;\n \n+\t/* The format to use for the ref database. */\n+\tchar *ref_storage_format;\n+\n \t/*\n \t * Contains path to often used file names.\n \t */\ndiff --git a/setup.c b/setup.c\nindex c04cd25a30..c6b57efd03 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -500,6 +500,9 @@ static enum extension_result handle_extension(const char *var,\n \t\t\treturn error(\"invalid value for 'extensions.objectformat'\");\n \t\tdata->hash_algo = format;\n \t\treturn EXTENSION_OK;\n+\t} else if (!strcmp(ext, \"refstorage\")) {\n+\t\tdata->ref_storage = xstrdup(value);\n+\t\treturn EXTENSION_OK;\n \t}\n \treturn EXTENSION_UNKNOWN;\n }\n@@ -651,6 +654,7 @@ void clear_repository_format(struct repository_format *format)\n \tstring_list_clear(&format->v1_only_extensions, 0);\n \tfree(format->work_tree);\n \tfree(format->partial_clone);\n+\tfree(format->ref_storage);\n \tinit_repository_format(format);\n }\n \n@@ -1308,8 +1312,11 @@ const char *setup_git_directory_gently(int *nongit_ok)\n \t\t\t\tgitdir = DEFAULT_GIT_DIR_ENVIRONMENT;\n \t\t\tsetup_git_env(gitdir);\n \t\t}\n-\t\tif (startup_info->have_repository)\n+\t\tif (startup_info->have_repository) {\n \t\t\trepo_set_hash_algo(the_repository, repo_fmt.hash_algo);\n+\t\t\tthe_repository->ref_storage_format =\n+\t\t\t\txstrdup_or_null(repo_fmt.ref_storage);\n+\t\t}\n \t}\n \n \tstrbuf_release(&dir);\ndiff --git a/t/t0031-reftable.sh b/t/t0031-reftable.sh\nnew file mode 100755\nindex 0000000000..58c7d5d4bc\n--- /dev/null\n+++ b/t/t0031-reftable.sh\n@@ -0,0 +1,199 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable basics'\n+\n+. ./test-lib.sh\n+\n+INVALID_SHA1=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n+\n+initialize () {\n+\trm -rf .git &&\n+\tgit init --ref-storage=reftable &&\n+\tmv .git/hooks .git/hooks-disabled\n+}\n+\n+test_expect_success 'SHA256 support, env' '\n+\trm -rf .git &&\n+\tGIT_DEFAULT_HASH=sha256 && export GIT_DEFAULT_HASH &&\n+\tgit init --ref-storage=reftable &&\n+\tmv .git/hooks .git/hooks-disabled &&\n+\ttest_commit file\n+'\n+\n+test_expect_success 'SHA256 support, option' '\n+\trm -rf .git &&\n+\tgit init --ref-storage=reftable --object-format=sha256 &&\n+\tmv .git/hooks .git/hooks-disabled &&\n+\ttest_commit file\n+'\n+\n+test_expect_success 'delete ref' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\tSHA=$(git show-ref -s --verify HEAD) &&\n+\ttest_write_lines \"$SHA refs/heads/master\" \"$SHA refs/tags/file\" >expect &&\n+\tgit show-ref > actual &&\n+\t! git update-ref -d refs/tags/file $INVALID_SHA1 &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d refs/tags/file $SHA &&\n+\ttest_write_lines \"$SHA refs/heads/master\" >expect &&\n+\tgit show-ref > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+\n+test_expect_success 'clone calls transaction_initial_commit' '\n+\ttest_commit message1 file1 &&\n+\tgit clone . cloned &&\n+\t(test -f cloned/file1 || echo \"Fixme.\")\n+'\n+\n+test_expect_success 'basic operation of reftable storage: commit, show-ref' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_write_lines refs/heads/master refs/tags/file >expect &&\n+\tgit show-ref &&\n+\tgit show-ref | cut -f2 -d\" \" > actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'reflog, repack' '\n+\tinitialize &&\n+\tfor count in $(test_seq 1 10)\n+\tdo\n+\t\ttest_commit \"number $count\" file.t $count number-$count ||\n+\t\treturn 1\n+\tdone &&\n+\tgit pack-refs &&\n+\tls -1 .git/reftable >table-files &&\n+\ttest_line_count = 2 table-files &&\n+\tgit reflog refs/heads/master >output &&\n+\ttest_line_count = 10 output &&\n+\tgrep \"commit (initial): number 1\" output &&\n+\tgrep \"commit: number 10\" output &&\n+\tgit gc &&\n+\tgit reflog refs/heads/master >output &&\n+\ttest_line_count = 0 output\n+'\n+\n+test_expect_success 'branch switch in reflog output' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tgit checkout -b branch1 &&\n+\ttest_commit file2 &&\n+\tgit checkout -b branch2 &&\n+\tgit switch - &&\n+\tgit rev-parse --symbolic-full-name HEAD > actual &&\n+\techo refs/heads/branch1 > expect &&\n+\ttest_cmp actual expect\n+'\n+\n+\n+# This matches show-ref's output\n+print_ref() {\n+\techo \"$(git rev-parse \"$1\") $1\"\n+}\n+\n+test_expect_success 'peeled tags are stored' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\tgit tag -m \"annotated tag\" test_tag HEAD &&\n+\t{\n+\t\tprint_ref \"refs/heads/master\" &&\n+\t\tprint_ref \"refs/tags/file\" &&\n+\t\tprint_ref \"refs/tags/test_tag\" &&\n+\t\tprint_ref \"refs/tags/test_tag^{}\"\n+\t} >expect &&\n+\tgit show-ref -d >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'show-ref works on fresh repo' '\n+\tinitialize &&\n+\trm -rf .git &&\n+\tgit init --ref-storage=reftable &&\n+\t>expect &&\n+\t! git show-ref > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'checkout unborn branch' '\n+\tinitialize &&\n+\tgit checkout -b master\n+'\n+\n+\n+test_expect_success 'dir/file conflict' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\t! git branch master/forbidden\n+'\n+\n+\n+test_expect_success 'do not clobber existing repo' '\n+\trm -rf .git &&\n+\tgit init --ref-storage=files &&\n+\tcat .git/HEAD > expect &&\n+\ttest_commit file &&\n+\t(git init --ref-storage=reftable || true) &&\n+\tcat .git/HEAD > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# cherry-pick uses a pseudo ref.\n+test_expect_success 'pseudo refs' '\n+\tinitialize &&\n+\ttest_commit message1 file1 &&\n+\ttest_commit message2 file2 &&\n+\tgit branch source &&\n+\tgit checkout HEAD^ &&\n+\ttest_commit message3 file3 &&\n+\tgit cherry-pick source &&\n+\ttest -f file2\n+'\n+\n+# cherry-pick uses a pseudo ref.\n+test_expect_success 'rebase' '\n+\tinitialize &&\n+\ttest_commit message1 file1 &&\n+\ttest_commit message2 file2 &&\n+\tgit branch source &&\n+\tgit checkout HEAD^ &&\n+\ttest_commit message3 file3 &&\n+\tgit rebase source &&\n+\ttest -f file2\n+'\n+\n+test_expect_success 'worktrees' '\n+\tgit init --ref-storage=reftable start &&\n+\t(cd start && test_commit file1 && git checkout -b branch1 &&\n+\tgit checkout -b branch2 &&\n+\tgit worktree add ../wt\n+\t) &&\n+\tcd wt &&\n+\tgit checkout branch1 &&\n+\tgit branch\n+'\n+\n+test_expect_success 'worktrees 2' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tmkdir existing_empty &&\n+\tgit worktree add --detach existing_empty master\n+'\n+\n+test_expect_success 'FETCH_HEAD' '\n+\tinitialize &&\n+\ttest_commit one &&\n+\t(git init sub && cd sub && test_commit two) &&\n+\tgit --git-dir sub/.git rev-parse HEAD >expect &&\n+\tgit fetch sub &&\n+\tgit checkout FETCH_HEAD &&\n+\tgit rev-parse HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/t1409-avoid-packing-refs.sh b/t/t1409-avoid-packing-refs.sh\nindex be12fb6350..c6f7832556 100755\n--- a/t/t1409-avoid-packing-refs.sh\n+++ b/t/t1409-avoid-packing-refs.sh\n@@ -4,6 +4,12 @@ test_description='avoid rewriting packed-refs unnecessarily'\n \n . ./test-lib.sh\n \n+if test_have_prereq REFTABLE\n+then\n+ skip_all='skipping pack-refs tests; incompatible with reftable'\n+ test_done\n+fi\n+\n # Add an identifying mark to the packed-refs file header line. This\n # shouldn't upset readers, and it should be omitted if the file is\n # ever rewritten.\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex b17f5c21fb..cc5d01571a 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -8,6 +8,12 @@ test_description='git fsck random collection of tests\n \n . ./test-lib.sh\n \n+if test_have_prereq REFTABLE\n+then\n+ skip_all='skipping tests; incompatible with reftable'\n+ test_done\n+fi\n+\n test_expect_success setup '\n \tgit config gc.auto 0 &&\n \tgit config i18n.commitencoding ISO-8859-1 &&\ndiff --git a/t/t3210-pack-refs.sh b/t/t3210-pack-refs.sh\nindex f41b2afb99..edaef2c175 100755\n--- a/t/t3210-pack-refs.sh\n+++ b/t/t3210-pack-refs.sh\n@@ -11,6 +11,12 @@ semantic is still the same.\n '\n . ./test-lib.sh\n \n+if test_have_prereq REFTABLE\n+then\n+ skip_all='skipping pack-refs tests; incompatible with reftable'\n+ test_done\n+fi\n+\n test_expect_success 'enable reflogs' '\n \tgit config core.logallrefupdates true\n '\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex a863ccee7e..7b638e0b8c 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1520,6 +1520,11 @@ parisc* | hppa*)\n \t;;\n esac\n \n+if test -n \"$GIT_TEST_REFTABLE\"\n+then\n+ test_set_prereq REFTABLE\n+fi\n+\n ( COLUMNS=1 && test $COLUMNS = 1 ) && test_set_prereq COLUMNS_CAN_BE_1\n test -z \"$NO_PERL\" && test_set_prereq PERL\n test -z \"$NO_PTHREADS\" && test_set_prereq PTHREADS\n-- \ngitgitgadget\n\n"},{"id":"410922","messageId":"CAMP44s1EhqDf4_T1zLC8iXA32P1atVMW-AnP=EJ3eXTzUbnU8w@mail.gmail.com","threadId":"54245","inReplyTo":"2aa30f536fb7ce5501d1ecf0315cbcb1c1c5ce38.1606419752.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 04/16] reftable: add error related functionality","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2020-11-27T09:13:20Z","receivedAt":"2020-11-27T09:13:39Z","isPatch":true,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Fri, Nov 27, 2020 at 2:42 AM Han-Wen Nienhuys via GitGitGadget\n wrote:\n\n> +/*\n> + Errors in reftable calls are signaled with negative integer return values. 0\n> + means success.\n> +*/\n\nThis is the format of multi-line comments:\n\n/*\n* A very long\n* multi-line comment.\n*/\n\n> +enum reftable_error {\n> + /* Unexpected file system behavior */\n> + REFTABLE_IO_ERROR = -2,\n> +\n> + /* Format inconsistency on reading data\n> + */\n\nNo need for a multi-line comment here.\n\n> + REFTABLE_FORMAT_ERROR = -3,\n> +\n> + /* File does not exist. Returned from block_source_from_file(), because\n> + it needs special handling in stack.\n> + */\n\nOnce again, and for the rest of the file.\n\n-- \nFelipe Contreras\n"},{"id":"410923","messageId":"CAMP44s1EEwmJtQiPx=NkZdJ-7LOm+S55Hdukf+Oz1aBV1MREEg@mail.gmail.com","threadId":"54245","inReplyTo":"987d1d186fd68d604268f9a66118b7f276094daf.1606419752.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 05/16] reftable: utility functions","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2020-11-27T10:18:04Z","receivedAt":"2020-11-27T10:18:19Z","isPatch":true,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Fri, Nov 27, 2020 at 2:42 AM Han-Wen Nienhuys via GitGitGadget\n wrote:\n\n> +int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n> +{\n> + size_t lo = 0;\n> + size_t hi = sz;\n\nIs there anything wrong with:\n\n size_t lo = 0, hi = sz;\n\n> + /* invariant: (hi == sz) || f(hi) == true\n> + (lo == 0 && f(0) == true) || fi(lo) == false\n> + */\n\nThis doesn't follow the multi-line format. And perhaps English would be better.\n\n> + while (hi - lo > 1) {\n> + size_t mid = lo + (hi - lo) / 2;\n> +\n> + int val = f(mid, args);\n> + if (val) {\n> + hi = mid;\n> + } else {\n> + lo = mid;\n> + }\n\nUnnecessary braces are frowned upon. Also, why store the value if you\nare not going to use it later?\n\nif (f(mid, args))\n hi = mid;\nelse\n lo = mid;\n\n> + }\n> +\n> + if (lo == 0) {\n\nif (!lo)\n\n> + if (f(0, args)) {\n> + return 0;\n> + } else {\n> + return 1;\n> + }\n\nreturn f(0, args) ? 0 : 1;\n\nOr just:\n\nreturn !f(0, args);\n\n> + }\n> +\n> + return hi;\n\nIt would be much clearer to just do:\n\nif (lo)\n return hi;\nelse\n return !f(0, args);\n\n> +}\n> +\n> +void free_names(char **a)\n> +{\n> + char **p = a;\n> + if (p == NULL) {\n> + return;\n> + }\n\nOnce again:\n\n if (!p)\n return;\n\n> + while (*p) {\n> + reftable_free(*p);\n> + p++;\n> + }\n\nThis is the same thing as:\n\n if (!a)\n return;\n\n for (p = a; *p; p++)\n reftable_free(*p);\n\n> +int names_length(char **names)\n> +{\n> + int len = 0;\n> + char **p = names;\n> + while (*p) {\n> + p++;\n> + len++;\n> + }\n> + return len;\n> +}\n\nIs there any need to keep track of len? It's always p - names.\n\nThis should do it:\n\n char **p;\n for (p = names; *p; p++);\n return p - names;\n\n> +void parse_names(char *buf, int size, char ***namesp)\n> +{\n> + char **names = NULL;\n> + size_t names_cap = 0;\n> + size_t names_len = 0;\n> +\n> + char *p = buf;\n> + char *end = buf + size;\n> + while (p < end) {\n\nYet another while that can be a for:\n\n for (p = buf; p < end; p = next + 1)\n\n> + char *next = strchr(p, '\\n');\n> + if (next != NULL) {\n> + *next = 0;\n> + } else {\n> + next = end;\n> + }\n\nif (next)\n *next = 0;\nelse\n next = end;\n\n> + if (p < next) {\n\nOr just:\n\n if (p >= next)\n continue;\n\n> + if (names_len == names_cap) {\n> + names_cap = 2 * names_cap + 1;\n> + names = reftable_realloc(\n> + names, names_cap * sizeof(char *));\n\nsizeof(*names);\n\n> + }\n> + names[names_len++] = xstrdup(p);\n> + }\n> + p = next + 1;\n> + }\n> +\n> + if (names_len == names_cap) {\n> + names_cap = 2 * names_cap + 1;\n> + names = reftable_realloc(names, names_cap * sizeof(char *));\n> + }\n\nThis is repeated, maybe a macro like ALLOC_GROW(), or literally ALLOC_GROW().\n\n> + names[names_len] = NULL;\n> + *namesp = names;\n> +}\n> +\n> +int names_equal(char **a, char **b)\n> +{\n> + while (*a && *b) {\n> + if (strcmp(*a, *b)) {\n> + return 0;\n> + }\n> +\n> + a++;\n> + b++;\n> + }\n\nAgain:\n\n for (; *a && *b, a++, b++) {\n if (strcmp(*a, *b)\n return 0;\n }\n\nBut instead of moving multiple pointers:\n\n for (i = 0; a[i] && b[i], i++) {\n if (strcmp(a[i], b[i])\n return 0;\n }\n\n> +int common_prefix_size(struct strbuf *a, struct strbuf *b)\n> +{\n> + int p = 0;\n> + while (p < a->len && p < b->len) {\n\nOnce again, this can easily be a for.\n\n> + if (a->buf[p] != b->buf[p]) {\n> + break;\n> + }\n> + p++;\n> + }\n> +\n> + return p;\n> +}\n\nCheers.\n\n-- \nFelipe Contreras\n"},{"id":"410924","messageId":"87o8jjnlot.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"cf667653dff437cee4c0d04e8c061a8b9ccc46a7.1606419752.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 02/16] init-db: set the_repository->hash_algo early on","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2020-11-27T10:22:42Z","receivedAt":"2020-11-27T10:22:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Nov 26 2020, Han-Wen Nienhuys via GitGitGadget wrote:\n\n> From: Han-Wen Nienhuys \n>\n> The reftable backend needs to know the hash algorithm for writing the\n> initialization hash table.\n>\n> The initial reftable contains a symref HEAD => \"main\" (or \"master\"), which is\n> agnostic to the size of hash value, but this is an exceptional circumstance, and\n> the reftable library does not cater for this exceptional. It insists that all\n\nrephrase: \"does not cater to this exception\" maybe?\n\n"},{"id":"410925","messageId":"87lfennln8.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"91c3ac2449edc45db643b0ba6cba66ef31176600.1606419752.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 03/16] reftable: add LICENSE","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2020-11-27T10:23:39Z","receivedAt":"2020-11-27T10:24:07Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Nov 26 2020, Han-Wen Nienhuys via GitGitGadget wrote:\n\n> From: Han-Wen Nienhuys \n>\n> TODO: relicense?\n\nOn the topics of TODOs, it would be nice to have an answer/some idea the\n\"I am concerned...\" question in about\nhttps://lore.kernel.org/git/873625i9tc.fsf@evledraar.gmail.com/ relevant\nto the license etc.\n"},{"id":"410926","messageId":"87im9rnlji.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"2aa30f536fb7ce5501d1ecf0315cbcb1c1c5ce38.1606419752.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 04/16] reftable: add error related functionality","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2020-11-27T10:25:53Z","receivedAt":"2020-11-27T10:26:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Nov 26 2020, Han-Wen Nienhuys via GitGitGadget wrote:\n\n> The reftable/ directory is structured as a library, so it cannot\n> crash on misuse. Instead, it returns an error codes.\n>\n> In addition, the error code can be used to signal conditions from lower levels\n> of the library to be handled by higher levels of the library. For example, a\n> transaction might legitimately write an empty reftable file, but in that case,\n> we'd want to shortcut the transaction overhead.\n> [...]\n> +\tstatic char buf[250];\n> +\tswitch (err) {\n> +\tcase REFTABLE_IO_ERROR:\n> +\t\treturn \"I/O error\";\n> +\tcase REFTABLE_FORMAT_ERROR:\n> +\t\treturn \"corrupt reftable file\";\n> +\tcase REFTABLE_NOT_EXIST_ERROR:\n> +\t\treturn \"file does not exist\";\n> +\tcase REFTABLE_LOCK_ERROR:\n> +\t\treturn \"data is outdated\";\n> +\tcase REFTABLE_API_ERROR:\n> +\t\treturn \"misuse of the reftable API\";\n> +\tcase REFTABLE_ZLIB_ERROR:\n> +\t\treturn \"zlib failure\";\n> +\tcase REFTABLE_NAME_CONFLICT:\n> +\t\treturn \"file/directory conflict\";\n> [...]\n\nNot for this series, but I wonder if the eventual integration into the\ni18n framework in git needs some thought, seems all the (including\npresumably stuff show to users) UI strings are hardcoded English at the\nmoment.\n"},{"id":"410927","messageId":"87ft4vnl7j.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"987d1d186fd68d604268f9a66118b7f276094daf.1606419752.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 05/16] reftable: utility functions","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2020-11-27T10:33:04Z","receivedAt":"2020-11-27T10:33:25Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Nov 26 2020, Han-Wen Nienhuys via GitGitGadget wrote:\n\n> From: Han-Wen Nienhuys \n>\n> This commit provides basic utility classes for the reftable library.\n\nRe Johannes's comments in v2\n(https://lore.kernel.org/git/nycvar.QRO.7.76.6.2010021557570.50@tvgsbejvaqbjf.bet/)\nabout strbuf.c etc. being duplicated I see that's gone now.\n\nIt seems you did some version of my suggestion in\nhttps://lore.kernel.org/git/873625i9tc.fsf@evledraar.gmail.com/ of\nfactoring out that code, except here:\n\n> [...]\n> +#ifdef REFTABLE_STANDALONE\n> +\n> +/* functions that git-core provides, for standalone compilation */\n> +#include \n> +\n> +uint64_t get_be64(void *in);\n> +void put_be64(void *out, uint64_t i);\n\nIt's unclear whether that's an omission, or needed for some reason (is\ngit.git ever going to have REFTABLE_STANDALONE=true?\n\nThe update shellscript is also gone from v2->v3, is that out of tree now\nsomewhere? How does one test with the upstream code & presumably do some\nmunging on it to make it diff-able with what's going to be in git.git?\n\nIt would be really useful to have that sort of info/summary in the cover\nletter between serieses. It's not that I mind e.g. having some of this\nREFTABLE_STANDALONE code here. Just having to do archaeology trying to\nfigure out changes in this rather large series...\n"},{"id":"410928","messageId":"87czzznk03.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"a590865a7008f065b3f4a75deabbf6dc06807623.1606419752.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 14/16] Reftable support for git-core","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2020-11-27T10:59:08Z","receivedAt":"2020-11-27T10:59:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Nov 26 2020, Han-Wen Nienhuys via GitGitGadget wrote:\n\n> From: Han-Wen Nienhuys \n>\n> For background, see the previous commit introducing the library.\n>\n> This introduces the file refs/reftable-backend.c containing a reftable-powered\n> ref storage backend.\n>\n> It can be activated by passing --ref-storage=reftable to \"git init\", or setting\n> GIT_TEST_REFTABLE in the environment.\n>\n> Example use: see t/t0031-reftable.sh\n>\n> Signed-off-by: Han-Wen Nienhuys \n> Signed-off-by: Johannes Schindelin \n> Helped-by: Johannes Schindelin \n> Co-authored-by: Jeff King \n> ---\n> .../technical/repository-version.txt | 7 +\n> Makefile | 4 +\n> builtin/clone.c | 5 +-\n> builtin/init-db.c | 55 +-\n> builtin/worktree.c | 27 +-\n> cache.h | 8 +-\n> config.mak.uname | 2 +-\n> contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n> refs.c | 27 +-\n> refs.h | 3 +\n> refs/refs-internal.h | 1 +\n> refs/reftable-backend.c | 1418 +++++++++++++++++\n> repository.c | 2 +\n> repository.h | 3 +\n> setup.c | 9 +-\n> t/t0031-reftable.sh | 199 +++\n> t/t1409-avoid-packing-refs.sh | 6 +\n> t/t1450-fsck.sh | 6 +\n> t/t3210-pack-refs.sh | 6 +\n> t/test-lib.sh | 5 +\n> 20 files changed, 1771 insertions(+), 33 deletions(-)\n> create mode 100644 refs/reftable-backend.c\n> create mode 100755 t/t0031-reftable.sh\n>\n> diff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\n> index 7844ef30ff..7257623583 100644\n> --- a/Documentation/technical/repository-version.txt\n> +++ b/Documentation/technical/repository-version.txt\n> @@ -100,3 +100,10 @@ If set, by default \"git config\" reads from both \"config\" and\n> multiple working directory mode, \"config\" file is shared while\n> \"config.worktree\" is per-working directory (i.e., it's in\n> GIT_COMMON_DIR/worktrees//config.worktree)\n> +\n> +==== `refStorage`\n> +\n> +Specifies the file format for the ref database. Values are `files`\n> +(for the traditional packed + loose ref format) and `reftable` for the\n> +binary reftable format. See https://github.com/google/reftable for\n> +more information.\n\nShould also be documented in Documentation/config/extensions.txt,\nincluding maybe that setting this will hard die on older git\nversions.\n\nThen if you init with reftable and set it to \"files\" it'll say \"warning:\nignoring broken ref refs/heads. Maybe amend that message to say the\nconfig is wrong/detect it's a reftable file?\n\n> +\t/*\n> +\t * Check to see if .git/HEAD exists; this must happen before\n> +\t * initializing the ref db, because we want to see if there is an\n> +\t * existing HEAD.\n> +\t */\n> +\tpath = git_path_buf(&buf, \"HEAD\");\n> +\treinit = (!access(path, R_OK) ||\n> +\t\t readlink(path, junk, sizeof(junk) - 1) != -1);\n> +\n> +\t/*\n> +\t * refs/heads is a file when using reftable. We can't reinitialize with\n> +\t * a reftable because it will overwrite HEAD\n> +\t */\n> +\tif (reinit && (!strcmp(fmt->ref_storage, \"reftable\")) ==\n> +\t\t\t is_directory(git_path_buf(&buf, \"refs/heads\"))) {\n> +\t\tdie(\"cannot switch ref storage format.\");\n> +\t}\n> +\n\nI found this a bit weird, why would creating .git/refs/heads as a file\nhave to do with .git/HEAD, but reading on what this comment means is\nthat part of the initialization is to put \"ref: refs/heads/.invalid\" in\nHEAD\n\nAnd through further dissecting (nothing in this series documents this\nAFAICT) that value is chosen because if you stray much from it git will\ndie with \"not a git repository\", i.e. it's invalid content, but not\n*too* invalid.\n\nI wonder if to achive that aim this isn't more useful &\nself-documenting:\n\n $ cat .git/HEAD\n ref: refs/heads/[this repository uses a reftable for ref storage!]\n\nAlso since older versions of git will do this:\n\n $ __git_ps1\n ([this repository uses a reftable for ref storage!])\n\n> \t/*\n> \t * We need to create a \"refs\" dir in any case so that older\n> \t * versions of git can tell that this is a repository.\n> @@ -260,9 +281,6 @@ static int create_default_files(const char *template_path,\n> \t * Point the HEAD symref to the initial branch with if HEAD does\n> \t * not yet exist.\n> \t */\n> -\tpath = git_path_buf(&buf, \"HEAD\");\n> -\treinit = (!access(path, R_OK)\n> -\t\t || readlink(path, junk, sizeof(junk)-1) != -1);\n> \tif (!reinit) {\n> \t\tchar *ref;\n> \n> @@ -279,7 +297,7 @@ static int create_default_files(const char *template_path,\n> \t\tfree(ref);\n> \t}\n> \n> -\tinitialize_repository_version(fmt->hash_algo, 0);\n> +\tinitialize_repository_version(fmt->hash_algo, 0, fmt->ref_storage);\n> \n> \t/* Check filemode trustability */\n> \tpath = git_path_buf(&buf, \"config\");\n> @@ -396,7 +414,7 @@ static void validate_hash_algorithm(struct repository_format *repo_fmt, int hash\n> \n> int init_db(const char *git_dir, const char *real_git_dir,\n> \t const char *template_dir, int hash, const char *initial_branch,\n> -\t unsigned int flags)\n> +\t const char *ref_storage_format, unsigned int flags)\n> {\n> \tint reinit;\n> \tint exist_ok = flags & INIT_DB_EXIST_OK;\n> @@ -435,6 +453,7 @@ int init_db(const char *git_dir, const char *real_git_dir,\n> \t * is an attempt to reinitialize new repository with an old tool.\n> \t */\n> \tcheck_repository_format(&repo_fmt);\n> +\trepo_fmt.ref_storage = xstrdup(ref_storage_format);\n> \n> \tvalidate_hash_algorithm(&repo_fmt, hash);\n> \n> @@ -467,6 +486,9 @@ int init_db(const char *git_dir, const char *real_git_dir,\n> \t\tgit_config_set(\"receive.denyNonFastforwards\", \"true\");\n> \t}\n> \n> +\tif (!strcmp(ref_storage_format, \"reftable\"))\n> +\t\tgit_config_set(\"extensions.refStorage\", ref_storage_format);\n> +\n> \tif (!(flags & INIT_DB_QUIET)) {\n> \t\tint len = strlen(git_dir);\n> \n> @@ -540,6 +562,7 @@ static const char *const init_db_usage[] = {\n> int cmd_init_db(int argc, const char **argv, const char *prefix)\n> {\n> \tconst char *git_dir;\n> +\tconst char *ref_storage_format = default_ref_storage();\n> \tconst char *real_git_dir = NULL;\n> \tconst char *work_tree;\n> \tconst char *template_dir = NULL;\n> @@ -548,15 +571,18 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n> \tconst char *initial_branch = NULL;\n> \tint hash_algo = GIT_HASH_UNKNOWN;\n> \tconst struct option init_db_options[] = {\n> -\t\tOPT_STRING(0, \"template\", &template_dir, N_(\"template-directory\"),\n> -\t\t\t\tN_(\"directory from which templates will be used\")),\n> +\t\tOPT_STRING(0, \"template\", &template_dir,\n> +\t\t\t N_(\"template-directory\"),\n> +\t\t\t N_(\"directory from which templates will be used\")),\n> \t\tOPT_SET_INT(0, \"bare\", &is_bare_repository_cfg,\n> -\t\t\t\tN_(\"create a bare repository\"), 1),\n> +\t\t\t N_(\"create a bare repository\"), 1),\n> \t\t{ OPTION_CALLBACK, 0, \"shared\", &init_shared_repository,\n> -\t\t\tN_(\"permissions\"),\n> -\t\t\tN_(\"specify that the git repository is to be shared amongst several users\"),\n> -\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0},\n> +\t\t N_(\"permissions\"),\n> +\t\t N_(\"specify that the git repository is to be shared amongst several users\"),\n> +\t\t PARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0 },\n> \t\tOPT_BIT('q', \"quiet\", &flags, N_(\"be quiet\"), INIT_DB_QUIET),\n> +\t\tOPT_STRING(0, \"ref-storage\", &ref_storage_format, N_(\"backend\"),\n> +\t\t\t N_(\"the ref storage format to use\")),\n> \t\tOPT_STRING(0, \"separate-git-dir\", &real_git_dir, N_(\"gitdir\"),\n> \t\t\t N_(\"separate git dir from working tree\")),\n> \t\tOPT_STRING('b', \"initial-branch\", &initial_branch, N_(\"name\"),\n> @@ -697,10 +723,11 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n> \t}\n> \n> \tUNLEAK(real_git_dir);\n> +\tUNLEAK(ref_storage_format);\n> \tUNLEAK(git_dir);\n> \tUNLEAK(work_tree);\n> \n> \tflags |= INIT_DB_EXIST_OK;\n> \treturn init_db(git_dir, real_git_dir, template_dir, hash_algo,\n> -\t\t initial_branch, flags);\n> +\t\t initial_branch, ref_storage_format, flags);\n> }\n> diff --git a/builtin/worktree.c b/builtin/worktree.c\n> index ce56fdaaa9..7931620788 100644\n> --- a/builtin/worktree.c\n> +++ b/builtin/worktree.c\n> @@ -12,6 +12,7 @@\n> #include \"submodule.h\"\n> #include \"utf8.h\"\n> #include \"worktree.h\"\n> +#include \"../refs/refs-internal.h\"\n> \n> static const char * const worktree_usage[] = {\n> \tN_(\"git worktree add [] []\"),\n> @@ -402,9 +403,29 @@ static int add_worktree(const char *path, const char *refname,\n> \t * worktree.\n> \t */\n> \tstrbuf_reset(&sb);\n> -\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n> -\twrite_file(sb.buf, \"%s\", oid_to_hex(&null_oid));\n> -\tstrbuf_reset(&sb);\n> +\tif (get_main_ref_store(the_repository)->be == &refs_be_reftable) {\n> +\t\t/* XXX this is cut & paste from reftable_init_db. */\n> +\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n> +\t\twrite_file(sb.buf, \"%s\", \"ref: refs/heads/.invalid\\n\");\n> +\t\tstrbuf_reset(&sb);\n> +\n> +\t\tstrbuf_addf(&sb, \"%s/refs\", sb_repo.buf);\n> +\t\tsafe_create_dir(sb.buf, 1);\n> +\t\tstrbuf_reset(&sb);\n> +\n> +\t\tstrbuf_addf(&sb, \"%s/refs/heads\", sb_repo.buf);\n> +\t\twrite_file(sb.buf, \"this repository uses the reftable format\");\n> +\t\tstrbuf_reset(&sb);\n> +\n> +\t\tstrbuf_addf(&sb, \"%s/reftable\", sb_repo.buf);\n> +\t\tsafe_create_dir(sb.buf, 1);\n> +\t\tstrbuf_reset(&sb);\n> +\t} else {\n> +\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n> +\t\twrite_file(sb.buf, \"%s\", oid_to_hex(&null_oid));\n> +\t\tstrbuf_reset(&sb);\n> +\t}\n> +\n> \tstrbuf_addf(&sb, \"%s/commondir\", sb_repo.buf);\n> \twrite_file(sb.buf, \"../..\");\n> \n> diff --git a/cache.h b/cache.h\n> index e986cf4ea9..545d2b7260 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -627,9 +627,10 @@ int path_inside_repo(const char *prefix, const char *path);\n> #define INIT_DB_EXIST_OK 0x0002\n> \n> int init_db(const char *git_dir, const char *real_git_dir,\n> -\t const char *template_dir, int hash_algo,\n> -\t const char *initial_branch, unsigned int flags);\n> -void initialize_repository_version(int hash_algo, int reinit);\n> +\t const char *template_dir, int hash_algo, const char *initial_branch,\n> +\t const char *ref_storage_format, unsigned int flags);\n> +void initialize_repository_version(int hash_algo, int reinit,\n> +\t\t\t\t const char *ref_storage_format);\n> \n> void sanitize_stdfds(void);\n> int daemonize(void);\n> @@ -1043,6 +1044,7 @@ struct repository_format {\n> \tint is_bare;\n> \tint hash_algo;\n> \tchar *work_tree;\n> +\tchar *ref_storage;\n> \tstruct string_list unknown_extensions;\n> \tstruct string_list v1_only_extensions;\n> };\n> diff --git a/config.mak.uname b/config.mak.uname\n> index c7eba69e54..ae4e25a1a4 100644\n> --- a/config.mak.uname\n> +++ b/config.mak.uname\n> @@ -709,7 +709,7 @@ vcxproj:\n> \t# Make .vcxproj files and add them\n> \tunset QUIET_GEN QUIET_BUILT_IN; \\\n> \tperl contrib/buildsystems/generate -g Vcxproj\n> -\tgit add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj\n> +\tgit add -f git.sln {*,*/lib,*/libreftable,t/helper/*}/*.vcxproj\n> \n> \t# Generate the LinkOrCopyBuiltins.targets and LinkOrCopyRemoteHttp.targets file\n> \t(echo '' && \\\n> diff --git a/contrib/buildsystems/Generators/Vcxproj.pm b/contrib/buildsystems/Generators/Vcxproj.pm\n> index d2584450ba..1a25789d28 100644\n> --- a/contrib/buildsystems/Generators/Vcxproj.pm\n> +++ b/contrib/buildsystems/Generators/Vcxproj.pm\n> @@ -77,7 +77,7 @@ sub createProject {\n> my $libs_release = \"\\n \";\n> my $libs_debug = \"\\n \";\n> if (!$static_library) {\n> - $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n> + $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib|reftable\\/libreftable\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n> $libs_debug = $libs_release;\n> $libs_debug =~ s/zlib\\.lib/zlibd\\.lib/g;\n> $libs_debug =~ s/libexpat\\.lib/libexpatd\\.lib/g;\n> @@ -232,6 +232,7 @@ sub createProject {\n> EOM\n> if (!$static_library || $target =~ 'vcs-svn' || $target =~ 'xdiff') {\n> my $uuid_libgit = $$build_structure{\"LIBS_libgit_GUID\"};\n> + my $uuid_libreftable = $$build_structure{\"LIBS_reftable/libreftable_GUID\"};\n> my $uuid_xdiff_lib = $$build_structure{\"LIBS_xdiff/lib_GUID\"};\n> \n> print F << \"EOM\";\n> @@ -241,6 +242,14 @@ sub createProject {\n> false\n> \n> EOM\n> + if (!($name =~ /xdiff|libreftable/)) {\n> + print F << \"EOM\";\n> + \n> + $uuid_libreftable\n> + false\n> + \n> +EOM\n> + }\n> if (!($name =~ 'xdiff')) {\n> print F << \"EOM\";\n> \n> diff --git a/refs.c b/refs.c\n> index 392f0bbf68..1b874db334 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -19,10 +19,16 @@\n> #include \"repository.h\"\n> #include \"sigchain.h\"\n> \n> +const char *default_ref_storage(void)\n> +{\n> +\tconst char *test = getenv(\"GIT_TEST_REFTABLE\");\n> +\treturn test ? \"reftable\" : \"files\";\n> +}\n> +\n> /*\n> * List of all available backends\n> */\n> -static struct ref_storage_be *refs_backends = &refs_be_files;\n> +static struct ref_storage_be *refs_backends = &refs_be_reftable;\n> \n> static struct ref_storage_be *find_ref_storage_backend(const char *name)\n> {\n> @@ -1754,13 +1760,13 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,\n> * Create, record, and return a ref_store instance for the specified\n> * gitdir.\n> */\n> -static struct ref_store *ref_store_init(const char *gitdir,\n> +static struct ref_store *ref_store_init(const char *gitdir, const char *be_name,\n> \t\t\t\t\tunsigned int flags)\n> {\n> -\tconst char *be_name = \"files\";\n> -\tstruct ref_storage_be *be = find_ref_storage_backend(be_name);\n> +\tstruct ref_storage_be *be;\n> \tstruct ref_store *refs;\n> \n> +\tbe = find_ref_storage_backend(be_name);\n> \tif (!be)\n> \t\tBUG(\"reference backend %s is unknown\", be_name);\n> \n> @@ -1776,7 +1782,11 @@ struct ref_store *get_main_ref_store(struct repository *r)\n> \tif (!r->gitdir)\n> \t\tBUG(\"attempting to get main_ref_store outside of repository\");\n> \n> -\tr->refs_private = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);\n> +\tr->refs_private = ref_store_init(r->gitdir,\n> +\t\t\t\t\t r->ref_storage_format ?\n> +\t\t\t\t\t\t r->ref_storage_format :\n> +\t\t\t\t\t\t default_ref_storage(),\n> +\t\t\t\t\t REF_STORE_ALL_CAPS);\n> \tr->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private);\n> \treturn r->refs_private;\n> }\n> @@ -1832,7 +1842,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n> \t\tgoto done;\n> \n> \t/* assume that add_submodule_odb() has been called */\n> -\trefs = ref_store_init(submodule_sb.buf,\n> +\trefs = ref_store_init(submodule_sb.buf, default_ref_storage(),\n> \t\t\t REF_STORE_READ | REF_STORE_ODB);\n> \tregister_ref_store_map(&submodule_ref_stores, \"submodule\",\n> \t\t\t refs, submodule);\n> @@ -1846,6 +1856,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n> \n> struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n> {\n> +\tconst char *format = default_ref_storage();\n> \tstruct ref_store *refs;\n> \tconst char *id;\n> \n> @@ -1859,9 +1870,9 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n> \n> \tif (wt->id)\n> \t\trefs = ref_store_init(git_common_path(\"worktrees/%s\", wt->id),\n> -\t\t\t\t REF_STORE_ALL_CAPS);\n> +\t\t\t\t format, REF_STORE_ALL_CAPS);\n> \telse\n> -\t\trefs = ref_store_init(get_git_common_dir(),\n> +\t\trefs = ref_store_init(get_git_common_dir(), format,\n> \t\t\t\t REF_STORE_ALL_CAPS);\n> \n> \tif (refs)\n> diff --git a/refs.h b/refs.h\n> index 6695518156..7dc60472c9 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -11,6 +11,9 @@ struct string_list;\n> struct string_list_item;\n> struct worktree;\n> \n> +/* Returns the ref storage backend to use by default. */\n> +const char *default_ref_storage(void);\n> +\n> /*\n> * Resolve a reference, recursively following symbolic refererences.\n> *\n> diff --git a/refs/refs-internal.h b/refs/refs-internal.h\n> index 467f4b3c93..28166bf1f8 100644\n> --- a/refs/refs-internal.h\n> +++ b/refs/refs-internal.h\n> @@ -669,6 +669,7 @@ struct ref_storage_be {\n> };\n> \n> extern struct ref_storage_be refs_be_files;\n> +extern struct ref_storage_be refs_be_reftable;\n> extern struct ref_storage_be refs_be_packed;\n> \n> /*\n> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> new file mode 100644\n> index 0000000000..894c72ddc4\n> --- /dev/null\n> +++ b/refs/reftable-backend.c\n> @@ -0,0 +1,1418 @@\n> +#include \"../cache.h\"\n> +#include \"../chdir-notify.h\"\n> +#include \"../config.h\"\n> +#include \"../iterator.h\"\n> +#include \"../lockfile.h\"\n> +#include \"../refs.h\"\n> +#include \"../reftable/reftable-stack.h\"\n> +#include \"../reftable/reftable-record.h\"\n> +#include \"../reftable/reftable-error.h\"\n> +#include \"../reftable/reftable-blocksource.h\"\n> +#include \"../reftable/reftable-reader.h\"\n> +#include \"../reftable/reftable-iterator.h\"\n> +#include \"../reftable/reftable-merged.h\"\n> +#include \"../reftable/reftable-generic.h\"\n> +#include \"../worktree.h\"\n> +#include \"refs-internal.h\"\n> +\n> +extern struct ref_storage_be refs_be_reftable;\n> +\n> +struct git_reftable_ref_store {\n> +\tstruct ref_store base;\n> +\tunsigned int store_flags;\n> +\n> +\tint err;\n> +\tchar *repo_dir;\n> +\n> +\tchar *reftable_dir;\n> +\tchar *worktree_reftable_dir;\n> +\n> +\tstruct reftable_stack *main_stack;\n> +\tstruct reftable_stack *worktree_stack;\n> +};\n> +\n> +static struct reftable_stack *stack_for(struct git_reftable_ref_store *store,\n> +\t\t\t\t\tconst char *refname)\n> +{\n> +\tif (store->worktree_stack == NULL)\n> +\t\treturn store->main_stack;\n> +\n> +\tswitch (ref_type(refname)) {\n> +\tcase REF_TYPE_PER_WORKTREE:\n> +\tcase REF_TYPE_PSEUDOREF:\n> +\tcase REF_TYPE_OTHER_PSEUDOREF:\n> +\t\treturn store->worktree_stack;\n> +\tdefault:\n> +\tcase REF_TYPE_MAIN_PSEUDOREF:\n> +\tcase REF_TYPE_NORMAL:\n> +\t\treturn store->main_stack;\n> +\t}\n> +}\n> +\n> +static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n> +\t\t\t\t const char *refname, struct object_id *oid,\n> +\t\t\t\t struct strbuf *referent,\n> +\t\t\t\t unsigned int *type);\n> +\n> +static void clear_reftable_log_record(struct reftable_log_record *log)\n> +{\n> +\tlog->old_hash = NULL;\n> +\tlog->new_hash = NULL;\n> +\tlog->message = NULL;\n> +\tlog->refname = NULL;\n> +\treftable_log_record_release(log);\n> +}\n> +\n> +static void fill_reftable_log_record(struct reftable_log_record *log)\n> +{\n> +\tconst char *info = git_committer_info(0);\n> +\tstruct ident_split split = { NULL };\n> +\tint result = split_ident_line(&split, info, strlen(info));\n> +\tint sign = 1;\n> +\tassert(0 == result);\n> +\n> +\treftable_log_record_release(log);\n> +\tlog->name =\n> +\t\txstrndup(split.name_begin, split.name_end - split.name_begin);\n> +\tlog->email =\n> +\t\txstrndup(split.mail_begin, split.mail_end - split.mail_begin);\n"},{"id":"410992","messageId":"CAFQ2z_NEkJQM6xtppW9kkAyjFrBeEiXmhhiPB8A7pmssjXHhKg@mail.gmail.com","threadId":"54245","inReplyTo":"87lfennln8.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 03/16] reftable: add LICENSE","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2020-11-30T11:26:47Z","receivedAt":"2020-11-30T11:28:02Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Fri, Nov 27, 2020 at 11:23 AM Ævar Arnfjörð Bjarmason\n wrote:\n>\n>\n> On Thu, Nov 26 2020, Han-Wen Nienhuys via GitGitGadget wrote:\n>\n> > From: Han-Wen Nienhuys \n> >\n> > TODO: relicense?\n>\n> On the topics of TODOs, it would be nice to have an answer/some idea the\n> \"I am concerned...\" question in about\n> https://lore.kernel.org/git/873625i9tc.fsf@evledraar.gmail.com/ relevant\n> to the license etc.\n\n\nLong term objective is for this to live in git proper, but remains\nindependently compilable, so it can be used by libgit2 as well.\n\nThis is achieved by having the code only depend on functions in\nstrbuf.h and git-compat-util.h.\n\nI would like to enforce the separation by having an actual standalone\ncompile, but the discussion over the code to do so has caused more\nheat than light, so I have deferred this by reworking standalone code\nto to be on file level, and only importing the non-standalone code. In\none of the other patches, you found a place where I had overlooked a\nREFTABLE_STANDALONE #ifdef; I'll remove that. I'll also remove the\nupdate.sh script.\n\nUntil the reftable code is actually merged into git, I work on the\nreftable library itself at github.com/google/reftable. I've asked our\nopensource team if we could switch off the CLA enforcement for this\nrepo, to facilitate back & forth imports of fixes.\n\nThe current version of the patchseries mainly addresses overall\nfeedback from the Git team at Google to split the public API along\ncommits too, and updates the series so it applies to master again.\n\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"410993","messageId":"CAFQ2z_PWM3zTYEQ9r-Ky_NwFk_KdtZMvx2s-kLyzuos==A7iMw@mail.gmail.com","threadId":"54245","inReplyTo":"87im9rnlji.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 04/16] reftable: add error related functionality","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2020-11-30T11:27:25Z","receivedAt":"2020-11-30T11:28:58Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Fri, Nov 27, 2020 at 11:25 AM Ævar Arnfjörð Bjarmason\n wrote:\n> Not for this series, but I wonder if the eventual integration into the\n> i18n framework in git needs some thought, seems all the (including\n> presumably stuff show to users) UI strings are hardcoded English at the\n> moment.\n\nAFAIK, these are the only strings that potentially show up in the UI.\n\nIf these strings need to be i18n'd. It is probably easier for\ngit-core to provide its own version of reftable_error_str() ? Or we\ncould remove reftable_error_str from the official API?\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"410999","messageId":"CAFQ2z_Oz9KZAVwYVWGBGZG4dB7BG0OBwrj+j67qSyNDrtPHiOw@mail.gmail.com","threadId":"54245","inReplyTo":"20201009211300.GA2774782@google.com","subject":"Re: [PATCH v2 02/13] reftable: define the public API","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2020-11-30T14:44:35Z","receivedAt":"2020-11-30T14:45:57Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Fri, Oct 9, 2020 at 11:13 PM Emily Shaffer wrote:\n>\n> On Thu, Oct 01, 2020 at 08:58:51PM -0700, Jonathan Nieder wrote:\n> >\n> > Hi,\n> >\n> > Han-Wen Nienhuys wrote:\n> >\n> > > reftable/reftable.h | 585 ++++++++++++++++++++++++++++++++++++++++++++\n> > > 1 file changed, 585 insertions(+)\n> > > create mode 100644 reftable/reftable.h\n> >\n> > Adding a header in a separate patch from the implementation doesn't\n> > match the usual practice. Can we add declarations in the same patch\n> > as the functions being declared instead?\n> >\n> > We could still introduce the header early in its own patch if we want,\n> > but it would be a skeleton of a header that gets filled out by later\n> > patches.\n>\n> To poke a little deeper into what I think Jonathan is trying to say:\n>\n> Of course, taking a completed project - like your initial reftable\n> submission - and then chopping it up into a cute story of commits is a\n> pain in the ***. Doing it twice - or more - is just aggravating. So I\n> wonder whether we can bikeshed what story would look nice before you\n> even pick up your 'git rebase -i'? Doing that bikeshedding here on list\n\nLooks like this didn't happen. I went with Jonathan's suggestion, and\nchopped the public API header in smaller bits. PTAL.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"411000","messageId":"CAFQ2z_PNn499nxv5h-_Q6bzR15S6e8CaFxoR+yL6DJkTj6=bgA@mail.gmail.com","threadId":"54245","inReplyTo":"20201012165730.GC3740546@google.com","subject":"Re: [PATCH v2 02/13] reftable: define the public API","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2020-11-30T14:55:57Z","receivedAt":"2020-11-30T14:57:06Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Mon, Oct 12, 2020 at 6:57 PM Jonathan Nieder wrote:\n> > We could, but it would considerably complicate work on this patch\n> > series, as the commit boundary then doesn't fall on file boundaries\n> > anymore. Would you be open to having multiple headers for the public\n> > interface? eg. reftable-record.h, reftable-reader.h etc. ?\n>\n> I'd be more than open to it: I think that would make for a clearer API.\n\nDone.\n\n> >> Should these call BUG()? Or is it useful for some callers to be able\n> >> to recover from these errors?\n> >\n> > Since this was written as a standalone library, it's up to the caller\n> > to decide what should be done.\n>\n> I'm not strongly opinionated about this, but just a quick note: this\n> implies that the library would need to make sure it is producing a\n> valid state in error cases.\n\nThis is what it already does. I tried to clarify this in some places.\n\n> Does the API documentation describe what state a handle is in after\n> an error --- e.g., what operations are permitted after that?\n\nAPI_ERROR should not result in state changes.\n\n> >>> +int reftable_error_to_errno(int err);\n> >>\n> >> What is the intended use of this function?\n> >\n> > The read_raw_ref method in the ref backend API uses errno values as\n> > out-of-band communication mechanism.\n>\n> Could we change that? It sounds error-prone.\n\nI moved it into refs/reftable-backend.c; I'd rather not also have to\nclean up the files backend as part of this work, though.\n\n> >> Do I pass in the 'struct block_source *' as the source arg? If so, why\n> >> are these declared as void *?\n> >\n> > you pass in block_source->arg. Should struct implementations of\n> > polymorphic types carry a pointer to their own vtable instead?\n>\n> I think a pointer to block_source would make it more self-explanatory,\n> yes.\n\nThis has a number of downsides, though. It means I have to introduce\nseparate structs strbuf_blocksource and file_blocksource, along with\nfunctions to create and deallocate them, so it makes the whole thing\nmore unwieldy to use.\n\n> >> Is the reason this manages the buffer instead of requiring a\n> >> caller-supplied buffer to support zero-copy?\n> >\n> > Log blocks are compressed, so the caller doesn't know the correct size\n> > to supply. By letting the block source handle the management, we can\n> > swap out the block read from the file for a block managed by malloc on\n> > decompressing a log block.\n>\n> Hm, my naive assumption would have been that we'd use different\n> buffers for the compressed and uncompressed data.\n\nMaybe you can comment on how to solve this differently when we get to\nthe implementation of block reading/writing in the commit series.\n\n> >>> + Generic tables\n> >>> +\n> >>> + A unified API for reading tables, either merged tables, or single readers.\n> >>\n> >> Are there callers/helpers that don't know whether they want one or the\n> >> other?\n> >\n> > The setup with per-worktree refs means that there are two reftable\n> > stacks in a .git repo. In order to iterate over the entire ref space,\n> > you have to merge two merged tables, but the stack itslef merges a set\n> > of simple reftables.\n>\n> I see. That's subtle, so it seems worth documenting for the next\n> person reading this documentation and wondering why.\n\nDone.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"411013","messageId":"CAFQ2z_MXF0wO76tVTcBuJJp0CwjVoVfE11BOYYQ7Y7h1h_VwJg@mail.gmail.com","threadId":"54245","inReplyTo":"CAFQ2z_NEkJQM6xtppW9kkAyjFrBeEiXmhhiPB8A7pmssjXHhKg@mail.gmail.com","subject":"Re: [PATCH v3 03/16] reftable: add LICENSE","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2020-11-30T20:25:45Z","receivedAt":"2020-11-30T20:26:54Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Mon, Nov 30, 2020 at 12:26 PM Han-Wen Nienhuys wrote:\n> Until the reftable code is actually merged into git, I work on the\n> reftable library itself at github.com/google/reftable. I've asked our\n> opensource team if we could switch off the CLA enforcement for this\n> repo, to facilitate back & forth imports of fixes.\n\nI've asked our lawyers, but this is not currently possible, unfortunately.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"411014","messageId":"CAMP44s0zoekLZYQ=r516jjdsGGQ2Y7m1fXpwW2BBJsGjQ5sseA@mail.gmail.com","threadId":"54245","inReplyTo":"CAFQ2z_MXF0wO76tVTcBuJJp0CwjVoVfE11BOYYQ7Y7h1h_VwJg@mail.gmail.com","subject":"Re: [PATCH v3 03/16] reftable: add LICENSE","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2020-11-30T21:21:28Z","receivedAt":"2020-11-30T21:22:48Z","isPatch":true,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Mon, Nov 30, 2020 at 2:28 PM Han-Wen Nienhuys wrote:\n>\n> On Mon, Nov 30, 2020 at 12:26 PM Han-Wen Nienhuys wrote:\n> > Until the reftable code is actually merged into git, I work on the\n> > reftable library itself at github.com/google/reftable. I've asked our\n> > opensource team if we could switch off the CLA enforcement for this\n> > repo, to facilitate back & forth imports of fixes.\n>\n> I've asked our lawyers, but this is not currently possible, unfortunately.\n\nMaybe I'm not understanding something correctly.\n\nGit's license is GPLv2. When you add your Signed-off-by line you are\nsaying you are authorized to provide this code to the Git project\nunder its open source license.\n\nTherefore, I should be able to take your patches, and rework them in\nany way I want as long as I keep Git's license.\n\nI don't need your permission, you are supposedly giving it by adding\nyour Signed-off-by line.\n\nRead: https://developercertificate.org/\n\nIf you are not authorized to provide these patches under an open\nsource license, then you should not add your Signed-off-by line.\n\nAnd if these patches cannot be used by the Git project, nor can we\nwork on them in a collaborative manner, then what's the point in\nproviding them?\n\nSounds to me Google has not made its mind about actually contributing\nthese changes.\n\nOr am I missing something?\n\nCheers.\n\n-- \nFelipe Contreras\n"},{"id":"411043","messageId":"CAFQ2z_PaPdiJC71WGdePFVgxyJFFHZY_DFbmLg9sRFm=vc7XYw@mail.gmail.com","threadId":"54245","inReplyTo":"CAMP44s0zoekLZYQ=r516jjdsGGQ2Y7m1fXpwW2BBJsGjQ5sseA@mail.gmail.com","subject":"Re: [PATCH v3 03/16] reftable: add LICENSE","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2020-12-01T09:51:41Z","receivedAt":"2020-12-01T09:52:35Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Mon, Nov 30, 2020 at 10:21 PM Felipe Contreras\n wrote:\n>\n> On Mon, Nov 30, 2020 at 2:28 PM Han-Wen Nienhuys wrote:\n> >\n> > On Mon, Nov 30, 2020 at 12:26 PM Han-Wen Nienhuys wrote:\n> > > Until the reftable code is actually merged into git, I work on the\n> > > reftable library itself at github.com/google/reftable. I've asked our\n> > > opensource team if we could switch off the CLA enforcement for this\n> > > repo, to facilitate back & forth imports of fixes.\n> >\n> > I've asked our lawyers, but this is not currently possible, unfortunately.\n>\n> Maybe I'm not understanding something correctly.\n\n> Sounds to me Google has not made its mind about actually contributing\n> these changes.\n>\n> Or am I missing something?\n\nThe restrictions are not about the patches themselves. They are about\nrestricting what gets posted under github.com/google/ .\n\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"411053","messageId":"CAMP44s0v92C+uUwoeBJxRv2MPCd2RuCPn8FVBUztNM9VOdjJgA@mail.gmail.com","threadId":"54245","inReplyTo":"CAFQ2z_PaPdiJC71WGdePFVgxyJFFHZY_DFbmLg9sRFm=vc7XYw@mail.gmail.com","subject":"Re: [PATCH v3 03/16] reftable: add LICENSE","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2020-12-01T10:38:08Z","receivedAt":"2020-12-01T10:39:05Z","isPatch":true,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Tue, Dec 1, 2020 at 3:51 AM Han-Wen Nienhuys wrote:\n> On Mon, Nov 30, 2020 at 10:21 PM Felipe Contreras\n> wrote:\n> > On Mon, Nov 30, 2020 at 2:28 PM Han-Wen Nienhuys wrote:\n\n> > Sounds to me Google has not made its mind about actually contributing\n> > these changes.\n> >\n> > Or am I missing something?\n>\n> The restrictions are not about the patches themselves. They are about\n> restricting what gets posted under github.com/google/ .\n\nI see.\n\nBut it doesn't have to be posted on github.com/google/, it doesn't\nhave to be posted on github.com at all. If the code is under an open\nsource license, you (or anyone) can post it anywhere.\n\n-- \nFelipe Contreras\n"},{"id":"411060","messageId":"87wny1vjg0.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"CAMP44s0v92C+uUwoeBJxRv2MPCd2RuCPn8FVBUztNM9VOdjJgA@mail.gmail.com","subject":"Re: [PATCH v3 03/16] reftable: add LICENSE","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2020-12-01T11:45:19Z","receivedAt":"2020-12-01T11:46:22Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Dec 01 2020, Felipe Contreras wrote:\n\n> On Tue, Dec 1, 2020 at 3:51 AM Han-Wen Nienhuys wrote:\n>> On Mon, Nov 30, 2020 at 10:21 PM Felipe Contreras\n>> wrote:\n>> > On Mon, Nov 30, 2020 at 2:28 PM Han-Wen Nienhuys wrote:\n>\n>> > Sounds to me Google has not made its mind about actually contributing\n>> > these changes.\n>> >\n>> > Or am I missing something?\n>>\n>> The restrictions are not about the patches themselves. They are about\n>> restricting what gets posted under github.com/google/ .\n>\n> I see.\n>\n> But it doesn't have to be posted on github.com/google/, it doesn't\n> have to be posted on github.com at all. If the code is under an open\n> source license, you (or anyone) can post it anywhere.\n\nThe libgit2.git and git.git codebases are under different\nlicenses. Therefore if the goal is to have code that's used in both it's\nnot viable to e.g. store it in git.git under our current contribution\npolicies.\n\nThe first contributor to submit a fix to it under GPLv2 as opposed to\n\"any GPL or LGPL version\" or whatever will preclude its subsequent\nimport into libgit2.\n\nAssigning copyright to Google is a way around that. They own your work,\nand then they re-license it under whatever license those two projects\nuse.\n\nBut as I pointed out in [1] it requires contributors to git.git's\nreftable/ directory & Junio to play along with that scheme, least the\nwhole process come to a halt. Maybe that's worth it, maybe not. But\nseems like something the series should very explicitly highlight and\ndocument e.g. in Documentation/SubmittingPatches.\n\nWe do have some in-tree code in that state already, but it's\nmostly/entirely one-off imports of GNU/FSF code like compat/regex/ or\nsh-i18n--envsubst.c that we're not actively working on in any way, and\nisn't core to git like the reftable would be.\n\nThere's individual contributors who just don't like the idea of\ncopyright assignment and wouldn't do it for that reason, but there's\nalso contributors who do paid-for work for their employers on git.git\nsometimes.\n\nE.g. I've done such work under terms that were basically \"you can\ncontribute to open source as part of your job, and under the license the\nrelevant project uses\". Going through the legal process of changing that\nto \"we, your employer, who own copyright to your work, agree to license\nit to this third party project/company/whatever for the purposes of an\nopen source contribution\" would in my experience be *very much* an\nuphill battle.\n\nSo I think even if the individual contributor wouldn't mind the\ncopyright assignment (I'd personally be fine with that), e.g. I've been\nin employment relationships where I'd just forget about sending a patch\nif it needed assignment, because there's no way I'd be getting it past\nlegal, in the same way Han-Wen seems to be having an uphill battle with\nGoogle's lawyers.\n\n1. https://lore.kernel.org/git/873625i9tc.fsf@evledraar.gmail.com/\n"},{"id":"411068","messageId":"CAFQ2z_OJQf3+b0TT6BmAV9q9G9c2icbLK0EqrEjpYmpi8g9Fsg@mail.gmail.com","threadId":"54245","inReplyTo":"87wny1vjg0.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 03/16] reftable: add LICENSE","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2020-12-01T13:34:57Z","receivedAt":"2020-12-01T13:36:06Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Tue, Dec 1, 2020 at 12:45 PM Ævar Arnfjörð Bjarmason\n wrote:\n>\n>\n> On Tue, Dec 01 2020, Felipe Contreras wrote:\n>\n> > On Tue, Dec 1, 2020 at 3:51 AM Han-Wen Nienhuys wrote:\n> >> On Mon, Nov 30, 2020 at 10:21 PM Felipe Contreras\n> >> wrote:\n> >> > On Mon, Nov 30, 2020 at 2:28 PM Han-Wen Nienhuys wrote:\n> >\n> >> > Sounds to me Google has not made its mind about actually contributing\n> >> > these changes.\n> >> >\n> >> > Or am I missing something?\n> >>\n> >> The restrictions are not about the patches themselves. They are about\n> >> restricting what gets posted under github.com/google/ .\n> >\n> > I see.\n> >\n> > But it doesn't have to be posted on github.com/google/, it doesn't\n> > have to be posted on github.com at all. If the code is under an open\n> > source license, you (or anyone) can post it anywhere.\n\nI am also bound by my employment contract to follow company\ninstructions. While you (Felipe) can post code where you want, the\nsame is not true for me.\n\n> The libgit2.git and git.git codebases are under different\n> licenses. Therefore if the goal is to have code that's used in both it's\n> not viable to e.g. store it in git.git under our current contribution\n> policies.\n> [..]\n\nThanks for sketching the dilemmas here. Me and Jun are talking to the\nopen source folks here at Google, so I am optimistic that we will find\nsome solution. Please give us a few days to sort this out.\n\n> The first contributor to submit a fix to it under GPLv2 as opposed to\n> \"any GPL or LGPL version\" or whatever will preclude its subsequent\n> import into libgit2.\n>\n> Assigning copyright to Google is a way around that. They own your work,\n> and then they re-license it under whatever license those two projects\n> use.\n>\n> But as I pointed out in [1] it requires contributors to git.git's\n> reftable/ directory & Junio to play along with that scheme, least the\n> whole process come to a halt. Maybe that's worth it, maybe not. But\n> seems like something the series should very explicitly highlight and\n> document e.g. in Documentation/SubmittingPatches.\n\nFWIW, the plan of record is still for this code to have its source of\ntruth in the Git project. Once it's there, there is no need for Google\nto have CLAs for contributions to the reftable/ directory.\n\nThere is still the open question of how to set the license terms for\nthis code, so it is OK for it to be copied into libgit2. I picked the\ncurrent license (BSD) because it was the most liberal I could find.\n(Maybe I should document that in the commit message.)\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"411117","messageId":"CAMP44s1u6KENgb5vfp0s97cDoJD5On3_AN0H8Re5Kn3YgBJAXQ@mail.gmail.com","threadId":"54245","inReplyTo":"87wny1vjg0.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 03/16] reftable: add LICENSE","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2020-12-01T23:03:48Z","receivedAt":"2020-12-01T23:04:43Z","isPatch":true,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Tue, Dec 1, 2020 at 5:45 AM Ævar Arnfjörð Bjarmason wrote:\n\n> Assigning copyright to Google is a way around that. They own your work,\n> and then they re-license it under whatever license those two projects\n> use.\n\nThat's the default. Unless otherwise stated; you retain the copyright\nwhile contributing code to open source projects.\n\nGenerally when open source projects want to change license they have\nto get the agreement from the majority of copyright holders.\n\nSome projects do ask you to give away your copyright, but I'm not\ncomfortable with that. If I want to contribute my code with GPLv2,\nthat's the license I want. If a project I contribute to wants to\nswitch to GPLv3, then they have to remove my code (or do something\nillegal).\n\nSo I see nothing wrong with Google owning the copyright and\ncontrolling under which license(s) their code is distributed. In fact,\nI would see something wrong with the opposite; them giving away their\ncopyright.\n\nCheers.\n\n-- \nFelipe Contreras\n"},{"id":"411119","messageId":"CAMP44s1cKGsOtHNVsquznSQZ3x0ri7dj-PQ=4LGN4VTAFe=tow@mail.gmail.com","threadId":"54245","inReplyTo":"CAFQ2z_OJQf3+b0TT6BmAV9q9G9c2icbLK0EqrEjpYmpi8g9Fsg@mail.gmail.com","subject":"Re: [PATCH v3 03/16] reftable: add LICENSE","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2020-12-01T23:13:56Z","receivedAt":"2020-12-01T23:14:51Z","isPatch":true,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Tue, Dec 1, 2020 at 7:35 AM Han-Wen Nienhuys wrote:\n> > On Tue, Dec 01 2020, Felipe Contreras wrote:\n\n> > > But it doesn't have to be posted on github.com/google/, it doesn't\n> > > have to be posted on github.com at all. If the code is under an open\n> > > source license, you (or anyone) can post it anywhere.\n>\n> I am also bound by my employment contract to follow company\n> instructions. While you (Felipe) can post code where you want, the\n> same is not true for me.\n\nI understand that, but what if a stubborn person decides to do that,\nand the code starts diverging from github.com/google/? So essentially\na fork is created.\n\nI don't think that's in the best interest of Google, and that would\nforce their hand to either a) allow you to contribute to the open\nrepo, or b) allow open source people to contribute to the google repo.\n\nI find it silly that these thought experiments have to be explained to\ncorporate lawyers, or worse; they actually have to be done so the\nlawyers get the point. But alas, that's what working in a big\ncorporation entails.\n\nGood luck with that. (No sarcasm intended)\n\n-- \nFelipe Contreras\n"},{"id":"411849","messageId":"798f680b800b6ef23c220fef88f82b7064b796ff.1607522429.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v4.git.git.1607522429.gitgitgadget@gmail.com","subject":"[PATCH v4 03/15] reftable: add error related functionality","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-12-09T14:00:17Z","receivedAt":"2020-12-09T14:01:34Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable/ directory is structured as a library, so it cannot\ncrash on misuse. Instead, it returns an error codes.\n\nIn addition, the error code can be used to signal conditions from lower levels\nof the library to be handled by higher levels of the library. For example, a\ntransaction might legitimately write an empty reftable file, but in that case,\nwe'd want to shortcut the transaction overhead.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/error.c | 41 ++++++++++++++++++++++++++\n reftable/reftable-error.h | 62 +++++++++++++++++++++++++++++++++++++++\n 2 files changed, 103 insertions(+)\n create mode 100644 reftable/error.c\n create mode 100644 reftable/reftable-error.h\n\ndiff --git a/reftable/error.c b/reftable/error.c\nnew file mode 100644\nindex 00000000000..f6f16def921\n--- /dev/null\n+++ b/reftable/error.c\n@@ -0,0 +1,41 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-error.h\"\n+\n+#include \n+\n+const char *reftable_error_str(int err)\n+{\n+\tstatic char buf[250];\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn \"I/O error\";\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn \"corrupt reftable file\";\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn \"file does not exist\";\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn \"data is outdated\";\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn \"misuse of the reftable API\";\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn \"zlib failure\";\n+\tcase REFTABLE_NAME_CONFLICT:\n+\t\treturn \"file/directory conflict\";\n+\tcase REFTABLE_EMPTY_TABLE_ERROR:\n+\t\treturn \"wrote empty table\";\n+\tcase REFTABLE_REFNAME_ERROR:\n+\t\treturn \"invalid refname\";\n+\tcase -1:\n+\t\treturn \"general error\";\n+\tdefault:\n+\t\tsnprintf(buf, sizeof(buf), \"unknown error code %d\", err);\n+\t\treturn buf;\n+\t}\n+}\ndiff --git a/reftable/reftable-error.h b/reftable/reftable-error.h\nnew file mode 100644\nindex 00000000000..6f89bedf1a5\n--- /dev/null\n+++ b/reftable/reftable-error.h\n@@ -0,0 +1,62 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_ERROR_H\n+#define REFTABLE_ERROR_H\n+\n+/*\n+ * Errors in reftable calls are signaled with negative integer return values. 0\n+ * means success.\n+ */\n+enum reftable_error {\n+\t/* Unexpected file system behavior */\n+\tREFTABLE_IO_ERROR = -2,\n+\n+\t/* Format inconsistency on reading data */\n+\tREFTABLE_FORMAT_ERROR = -3,\n+\n+\t/* File does not exist. Returned from block_source_from_file(), because\n+\t * it needs special handling in stack.\n+\t */\n+\tREFTABLE_NOT_EXIST_ERROR = -4,\n+\n+\t/* Trying to write out-of-date data. */\n+\tREFTABLE_LOCK_ERROR = -5,\n+\n+\t/* Misuse of the API:\n+\t * - on writing a record with NULL refname.\n+\t * - on writing a reftable_ref_record outside the table limits\n+\t * - on writing a ref or log record before the stack's\n+\t * next_update_inde*x\n+\t * - on writing a log record with multiline message with\n+\t * exact_log_message unset\n+\t * - on reading a reftable_ref_record from log iterator, or vice versa.\n+\t *\n+\t * When a call misuses the API, the internal state of the library is\n+\t * kept unchanged.\n+\t */\n+\tREFTABLE_API_ERROR = -6,\n+\n+\t/* Decompression error */\n+\tREFTABLE_ZLIB_ERROR = -7,\n+\n+\t/* Wrote a table without blocks. */\n+\tREFTABLE_EMPTY_TABLE_ERROR = -8,\n+\n+\t/* Dir/file conflict. */\n+\tREFTABLE_NAME_CONFLICT = -9,\n+\n+\t/* Invalid ref name. */\n+\tREFTABLE_REFNAME_ERROR = -10,\n+};\n+\n+/* convert the numeric error code to a string. The string should not be\n+ * deallocated. */\n+const char *reftable_error_str(int err);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"411850","messageId":"pull.847.v4.git.git.1607522429.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v3.git.git.1606419752.gitgitgadget@gmail.com","subject":"[PATCH v4 00/15] reftable library","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-12-09T14:00:14Z","receivedAt":"2020-12-09T14:01:34Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"This splits the giant commit from\nhttps://github.com/gitgitgadget/git/pull/539 into a series of smaller\ncommits, which build and have unittests.\n\nThe final commit should also be split up, but I want to wait until we have\nconsensus that the bottom commits look good.\n\nversion 8 Dec 2020:\n\n * fixes for windows: add random suffix to table names\n * tagged unions for reftable_ref_record\n * some fixes by dscho\n * some tweaks in response to Felipe Contreras\n * switched to github.com/hanwen/reftable (which does not need CLA)\n\nBased on github.com/hanwen/reftable at\nc33a40ad23dd622d8c72139c41a2be1cfa063e5f Add random suffix to table names\n\nHan-Wen Nienhuys (14):\n init-db: set the_repository->hash_algo early on\n reftable: add LICENSE\n reftable: add error related functionality\n reftable: utility functions\n reftable: add blocksource, an abstraction for random access reads\n reftable: (de)serialization for the polymorphic record type.\n reftable: reading/writing blocks\n reftable: a generic binary tree implementation\n reftable: write reftable files\n reftable: read reftable files\n reftable: reftable file level tests\n reftable: rest of library\n Reftable support for git-core\n Add \"test-tool dump-reftable\" command.\n\nSZEDER Gábor (1):\n git-prompt: prepare for reftable refs backend\n\n Documentation/config/extensions.txt | 9 +\n .../technical/repository-version.txt | 7 +\n Makefile | 46 +-\n builtin/clone.c | 5 +-\n builtin/init-db.c | 76 +-\n builtin/worktree.c | 27 +-\n cache.h | 8 +-\n config.mak.uname | 2 +-\n contrib/buildsystems/CMakeLists.txt | 14 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n contrib/completion/git-prompt.sh | 7 +-\n refs.c | 27 +-\n refs.h | 3 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1435 +++++++++++++++++\n reftable/.gitattributes | 1 +\n reftable/LICENSE | 31 +\n reftable/VERSION | 1 +\n reftable/basics.c | 128 ++\n reftable/basics.h | 60 +\n reftable/basics_test.c | 98 ++\n reftable/block.c | 440 +++++\n reftable/block.h | 127 ++\n reftable/block_test.c | 121 ++\n reftable/blocksource.c | 148 ++\n reftable/blocksource.h | 22 +\n reftable/constants.h | 21 +\n reftable/dump.c | 213 +++\n reftable/error.c | 41 +\n reftable/iter.c | 248 +++\n reftable/iter.h | 75 +\n reftable/merged.c | 366 +++++\n reftable/merged.h | 35 +\n reftable/merged_test.c | 343 ++++\n reftable/pq.c | 115 ++\n reftable/pq.h | 32 +\n reftable/publicbasics.c | 58 +\n reftable/reader.c | 733 +++++++++\n reftable/reader.h | 75 +\n reftable/record.c | 1157 +++++++++++++\n reftable/record.h | 139 ++\n reftable/record_test.c | 398 +++++\n reftable/refname.c | 209 +++\n reftable/refname.h | 29 +\n reftable/refname_test.c | 102 ++\n reftable/reftable-blocksource.h | 49 +\n reftable/reftable-error.h | 62 +\n reftable/reftable-generic.h | 48 +\n reftable/reftable-iterator.h | 37 +\n reftable/reftable-malloc.h | 18 +\n reftable/reftable-merged.h | 68 +\n reftable/reftable-reader.h | 89 +\n reftable/reftable-record.h | 100 ++\n reftable/reftable-stack.h | 120 ++\n reftable/reftable-tests.h | 22 +\n reftable/reftable-writer.h | 147 ++\n reftable/reftable.c | 98 ++\n reftable/reftable_test.c | 580 +++++++\n reftable/stack.c | 1260 +++++++++++++++\n reftable/stack.h | 41 +\n reftable/stack_test.c | 803 +++++++++\n reftable/system.h | 32 +\n reftable/test_framework.c | 23 +\n reftable/test_framework.h | 53 +\n reftable/tree.c | 63 +\n reftable/tree.h | 34 +\n reftable/tree_test.c | 61 +\n reftable/writer.c | 681 ++++++++\n reftable/writer.h | 50 +\n reftable/zlib-compat.c | 92 ++\n repository.c | 2 +\n repository.h | 3 +\n setup.c | 9 +-\n t/helper/test-reftable.c | 20 +\n t/helper/test-tool.c | 4 +-\n t/helper/test-tool.h | 2 +\n t/t0031-reftable.sh | 199 +++\n t/t0032-reftable-unittest.sh | 15 +\n t/t1409-avoid-packing-refs.sh | 6 +\n t/t1450-fsck.sh | 6 +\n t/t3210-pack-refs.sh | 6 +\n t/test-lib.sh | 5 +\n 82 files changed, 12112 insertions(+), 40 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100644 reftable/.gitattributes\n create mode 100644 reftable/LICENSE\n create mode 100644 reftable/VERSION\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/basics_test.c\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/dump.c\n create mode 100644 reftable/error.c\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n create mode 100644 reftable/reftable-blocksource.h\n create mode 100644 reftable/reftable-error.h\n create mode 100644 reftable/reftable-generic.h\n create mode 100644 reftable/reftable-iterator.h\n create mode 100644 reftable/reftable-malloc.h\n create mode 100644 reftable/reftable-merged.h\n create mode 100644 reftable/reftable-reader.h\n create mode 100644 reftable/reftable-record.h\n create mode 100644 reftable/reftable-stack.h\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/reftable-writer.h\n create mode 100644 reftable/reftable.c\n create mode 100644 reftable/reftable_test.c\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n create mode 100644 reftable/zlib-compat.c\n create mode 100644 t/helper/test-reftable.c\n create mode 100755 t/t0031-reftable.sh\n create mode 100755 t/t0032-reftable-unittest.sh\n\n\nbase-commit: 3a0b884caba2752da0af626fb2de7d597c844e8b\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-847%2Fhanwen%2Flibreftable-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-847/hanwen/libreftable-v4\nPull-Request: https://github.com/git/git/pull/847\n\nRange-diff vs v3:\n\n 1: 030998a732a < -: ----------- move sleep_millisec to git-compat-util.h\n 2: cf667653dff < -: ----------- init-db: set the_repository->hash_algo early on\n -: ----------- > 1: 40ac041d0ef init-db: set the_repository->hash_algo early on\n 3: 91c3ac2449e = 2: cc0fe58bd33 reftable: add LICENSE\n 4: 2aa30f536fb ! 3: 798f680b800 reftable: add error related functionality\n @@ reftable/error.c (new)\n +\t\treturn \"zlib failure\";\n +\tcase REFTABLE_NAME_CONFLICT:\n +\t\treturn \"file/directory conflict\";\n ++\tcase REFTABLE_EMPTY_TABLE_ERROR:\n ++\t\treturn \"wrote empty table\";\n +\tcase REFTABLE_REFNAME_ERROR:\n +\t\treturn \"invalid refname\";\n +\tcase -1:\n @@ reftable/reftable-error.h (new)\n +#define REFTABLE_ERROR_H\n +\n +/*\n -+ Errors in reftable calls are signaled with negative integer return values. 0\n -+ means success.\n -+*/\n ++ * Errors in reftable calls are signaled with negative integer return values. 0\n ++ * means success.\n ++ */\n +enum reftable_error {\n +\t/* Unexpected file system behavior */\n +\tREFTABLE_IO_ERROR = -2,\n +\n -+\t/* Format inconsistency on reading data\n -+\t */\n ++\t/* Format inconsistency on reading data */\n +\tREFTABLE_FORMAT_ERROR = -3,\n +\n -+\t/* File does not exist. Returned from block_source_from_file(), because\n -+\t it needs special handling in stack.\n -+\t*/\n ++\t/* File does not exist. Returned from block_source_from_file(), because\n ++\t * it needs special handling in stack.\n ++\t */\n +\tREFTABLE_NOT_EXIST_ERROR = -4,\n +\n +\t/* Trying to write out-of-date data. */\n +\tREFTABLE_LOCK_ERROR = -5,\n +\n +\t/* Misuse of the API:\n -+\t - on writing a record with NULL refname.\n -+\t - on writing a reftable_ref_record outside the table limits\n -+\t - on writing a ref or log record before the stack's next_update_index\n -+\t - on writing a log record with multiline message with\n -+\t exact_log_message unset\n -+\t - on reading a reftable_ref_record from log iterator, or vice versa.\n -+\n -+\t When a call misuses the API, the internal state of the library is kept\n -+\t unchanged.\n -+\t*/\n ++\t * - on writing a record with NULL refname.\n ++\t * - on writing a reftable_ref_record outside the table limits\n ++\t * - on writing a ref or log record before the stack's\n ++\t * next_update_inde*x\n ++\t * - on writing a log record with multiline message with\n ++\t * exact_log_message unset\n ++\t * - on reading a reftable_ref_record from log iterator, or vice versa.\n ++\t *\n ++\t * When a call misuses the API, the internal state of the library is\n ++\t * kept unchanged.\n ++\t */\n +\tREFTABLE_API_ERROR = -6,\n +\n +\t/* Decompression error */\n @@ reftable/reftable-error.h (new)\n +\t/* Dir/file conflict. */\n +\tREFTABLE_NAME_CONFLICT = -9,\n +\n -+\t/* Illegal ref name. */\n ++\t/* Invalid ref name. */\n +\tREFTABLE_REFNAME_ERROR = -10,\n +};\n +\n 5: 987d1d186fd ! 4: 2f55ff6d240 reftable: utility functions\n @@ Commit message\n This commit provides basic utility classes for the reftable library.\n \n Signed-off-by: Han-Wen Nienhuys \n + Helped-by: Johannes Schindelin \n \n ## Makefile ##\n @@ Makefile: TEST_BUILTINS_OBJS += test-read-cache.o\n @@ Makefile: cocciclean:\n \t$(RM) $(TEST_PROGRAMS)\n \t$(RM) $(FUZZ_PROGRAMS)\n \n + ## contrib/buildsystems/CMakeLists.txt ##\n +@@ contrib/buildsystems/CMakeLists.txt: parse_makefile_for_sources(libxdiff_SOURCES \"XDIFF_OBJS\")\n + list(TRANSFORM libxdiff_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n + add_library(xdiff STATIC ${libxdiff_SOURCES})\n + \n ++#reftable\n ++parse_makefile_for_sources(reftable_SOURCES \"REFTABLE_OBJS\")\n ++\n ++list(TRANSFORM reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n ++add_library(reftable STATIC ${reftable_SOURCES})\n ++\n + if(WIN32)\n + \tif(NOT MSVC)#use windres when compiling with gcc and clang\n + \t\tadd_custom_command(OUTPUT ${CMAKE_BINARY_DIR}/git.res\n +@@ contrib/buildsystems/CMakeLists.txt: endif()\n + #link all required libraries to common-main\n + add_library(common-main OBJECT ${CMAKE_SOURCE_DIR}/common-main.c)\n + \n +-target_link_libraries(common-main libgit xdiff ${ZLIB_LIBRARIES})\n ++target_link_libraries(common-main libgit xdiff reftable ${ZLIB_LIBRARIES})\n + if(Intl_FOUND)\n + \ttarget_link_libraries(common-main ${Intl_LIBRARIES})\n + endif()\n +@@ contrib/buildsystems/CMakeLists.txt: if(BUILD_TESTING)\n + add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)\n + target_link_libraries(test-fake-ssh common-main)\n + \n ++#reftable-tests\n ++parse_makefile_for_sources(test-reftable_SOURCES \"REFTABLE_TEST_OBJS\")\n ++list(TRANSFORM test-reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n ++\n + #test-tool\n + parse_makefile_for_sources(test-tool_SOURCES \"TEST_BUILTINS_OBJS\")\n + \n + list(TRANSFORM test-tool_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/t/helper/\")\n +-add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES})\n ++add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES} ${test-reftable_SOURCES})\n + target_link_libraries(test-tool common-main)\n + \n + set_target_properties(test-fake-ssh test-tool\n +\n ## reftable/basics.c (new) ##\n @@\n +/*\n @@ reftable/basics.c (new)\n +\tsize_t lo = 0;\n +\tsize_t hi = sz;\n +\n -+\t/* invariant: (hi == sz) || f(hi) == true\n -+\t (lo == 0 && f(0) == true) || fi(lo) == false\n ++\t/* Invariants:\n ++\t *\n ++\t * (hi == sz) || f(hi) == true\n ++\t * (lo == 0 && f(0) == true) || fi(lo) == false\n +\t */\n +\twhile (hi - lo > 1) {\n +\t\tsize_t mid = lo + (hi - lo) / 2;\n +\n -+\t\tint val = f(mid, args);\n -+\t\tif (val) {\n ++\t\tif (f(mid, args))\n +\t\t\thi = mid;\n -+\t\t} else {\n ++\t\telse\n +\t\t\tlo = mid;\n -+\t\t}\n +\t}\n +\n -+\tif (lo == 0) {\n -+\t\tif (f(0, args)) {\n -+\t\t\treturn 0;\n -+\t\t} else {\n -+\t\t\treturn 1;\n -+\t\t}\n -+\t}\n ++\tif (lo)\n ++\t\treturn hi;\n +\n -+\treturn hi;\n ++\treturn f(0, args) ? 0 : 1;\n +}\n +\n +void free_names(char **a)\n +{\n -+\tchar **p = a;\n -+\tif (p == NULL) {\n ++\tchar **p;\n ++\tif (a == NULL) {\n +\t\treturn;\n +\t}\n -+\twhile (*p) {\n ++\tfor (p = a; *p; p++) {\n +\t\treftable_free(*p);\n -+\t\tp++;\n +\t}\n +\treftable_free(a);\n +}\n +\n +int names_length(char **names)\n +{\n -+\tint len = 0;\n +\tchar **p = names;\n -+\twhile (*p) {\n -+\t\tp++;\n -+\t\tlen++;\n ++\tfor (; *p; p++) {\n ++\t\t/* empty */\n +\t}\n -+\treturn len;\n ++\treturn p - names;\n +}\n +\n +void parse_names(char *buf, int size, char ***namesp)\n @@ reftable/basics.c (new)\n +\tchar *end = buf + size;\n +\twhile (p < end) {\n +\t\tchar *next = strchr(p, '\\n');\n -+\t\tif (next != NULL) {\n ++\t\tif (next && next < end) {\n +\t\t\t*next = 0;\n +\t\t} else {\n +\t\t\tnext = end;\n @@ reftable/basics.c (new)\n +\t\t\tif (names_len == names_cap) {\n +\t\t\t\tnames_cap = 2 * names_cap + 1;\n +\t\t\t\tnames = reftable_realloc(\n -+\t\t\t\t\tnames, names_cap * sizeof(char *));\n ++\t\t\t\t\tnames, names_cap * sizeof(*names));\n +\t\t\t}\n +\t\t\tnames[names_len++] = xstrdup(p);\n +\t\t}\n +\t\tp = next + 1;\n +\t}\n +\n -+\tif (names_len == names_cap) {\n -+\t\tnames_cap = 2 * names_cap + 1;\n -+\t\tnames = reftable_realloc(names, names_cap * sizeof(char *));\n -+\t}\n -+\n ++\tnames = reftable_realloc(names, (names_len + 1) * sizeof(*names));\n +\tnames[names_len] = NULL;\n +\t*namesp = names;\n +}\n +\n +int names_equal(char **a, char **b)\n +{\n -+\twhile (*a && *b) {\n -+\t\tif (strcmp(*a, *b)) {\n ++\tint i = 0;\n ++\tfor (; a[i] && b[i]; i++) {\n ++\t\tif (strcmp(a[i], b[i])) {\n +\t\t\treturn 0;\n +\t\t}\n -+\n -+\t\ta++;\n -+\t\tb++;\n +\t}\n +\n -+\treturn *a == *b;\n ++\treturn a[i] == b[i];\n +}\n +\n +int common_prefix_size(struct strbuf *a, struct strbuf *b)\n +{\n +\tint p = 0;\n -+\twhile (p < a->len && p < b->len) {\n -+\t\tif (a->buf[p] != b->buf[p]) {\n ++\tfor (; p < a->len && p < b->len; p++) {\n ++\t\tif (a->buf[p] != b->buf[p])\n +\t\t\tbreak;\n -+\t\t}\n -+\t\tp++;\n +\t}\n +\n +\treturn p;\n @@ reftable/basics.h (new)\n +void put_be16(uint8_t *out, uint16_t i);\n +\n +/*\n -+ find smallest index i in [0, sz) at which f(i) is true, assuming\n -+ that f is ascending. Return sz if f(i) is false for all indices.\n -+*/\n ++ * find smallest index i in [0, sz) at which f(i) is true, assuming\n ++ * that f is ascending. Return sz if f(i) is false for all indices.\n ++ *\n ++ * Contrary to bsearch(3), this returns something useful if the argument is not\n ++ * found.\n ++ */\n +int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args);\n +\n +/*\n -+ Frees a NULL terminated array of malloced strings. The array itself is also\n -+ freed.\n ++ * Frees a NULL terminated array of malloced strings. The array itself is also\n ++ * freed.\n + */\n +void free_names(char **a);\n +\n -+/* parse a newline separated list of names. Empty names are discarded. */\n ++/* parse a newline separated list of names. `size` is the length of the buffer,\n ++ * without terminating '\\0'. Empty names are discarded. */\n +void parse_names(char *buf, int size, char ***namesp);\n +\n +/* compares two NULL-terminated arrays of strings. */\n @@ reftable/basics_test.c (new)\n +\t}\n +}\n +\n -+int basics_test_main(int argc, const char *argv[])\n ++static void test_names_length(void)\n +{\n -+\ttest_binsearch();\n -+\treturn 0;\n ++\tchar *a[] = { \"a\", \"b\", NULL };\n ++\tEXPECT(names_length(a) == 2);\n +}\n -\n - ## reftable/compat.h (new) ##\n -@@\n -+/*\n -+Copyright 2020 Google LLC\n -+\n -+Use of this source code is governed by a BSD-style\n -+license that can be found in the LICENSE file or at\n -+https://developers.google.com/open-source/licenses/bsd\n -+*/\n -+\n -+#ifndef COMPAT_H\n -+#define COMPAT_H\n +\n -+#include \"system.h\"\n -+\n -+#ifdef REFTABLE_STANDALONE\n -+\n -+/* functions that git-core provides, for standalone compilation */\n -+#include \n -+\n -+uint64_t get_be64(void *in);\n -+void put_be64(void *out, uint64_t i);\n -+\n -+void put_be32(void *out, uint32_t i);\n -+uint32_t get_be32(uint8_t *in);\n -+\n -+uint16_t get_be16(uint8_t *in);\n -+\n -+#define ARRAY_SIZE(a) sizeof((a)) / sizeof((a)[0])\n -+#define FREE_AND_NULL(x) \\\n -+\tdo { \\\n -+\t\treftable_free(x); \\\n -+\t\t(x) = NULL; \\\n -+\t} while (0)\n -+#define QSORT(arr, n, cmp) qsort(arr, n, sizeof(arr[0]), cmp)\n -+#define SWAP(a, b) \\\n -+\t{ \\\n -+\t\tchar tmp[sizeof(a)]; \\\n -+\t\tassert(sizeof(a) == sizeof(b)); \\\n -+\t\tmemcpy(&tmp[0], &a, sizeof(a)); \\\n -+\t\tmemcpy(&a, &b, sizeof(a)); \\\n -+\t\tmemcpy(&b, &tmp[0], sizeof(a)); \\\n -+\t}\n ++static void test_parse_names_normal(void)\n ++{\n ++\tchar in[] = \"a\\nb\\n\";\n ++\tchar **out = NULL;\n ++\tparse_names(in, strlen(in), &out);\n ++\tEXPECT(!strcmp(out[0], \"a\"));\n ++\tEXPECT(!strcmp(out[1], \"b\"));\n ++\tEXPECT(out[2] == NULL);\n ++\tfree_names(out);\n ++}\n +\n -+char *xstrdup(const char *s);\n ++static void test_parse_names_drop_empty(void)\n ++{\n ++\tchar in[] = \"a\\n\\n\";\n ++\tchar **out = NULL;\n ++\tparse_names(in, strlen(in), &out);\n ++\tEXPECT(!strcmp(out[0], \"a\"));\n ++\tEXPECT(out[1] == NULL);\n ++\tfree_names(out);\n ++}\n +\n -+void sleep_millisec(int millisecs);\n ++static void test_common_prefix(void)\n ++{\n ++\tstruct strbuf s1 = STRBUF_INIT;\n ++\tstruct strbuf s2 = STRBUF_INIT;\n ++\tstrbuf_addstr(&s1, \"abcdef\");\n ++\tstrbuf_addstr(&s2, \"abc\");\n ++\tEXPECT(common_prefix_size(&s1, &s2) == 3);\n ++\tstrbuf_release(&s1);\n ++\tstrbuf_release(&s2);\n ++}\n +\n -+#endif\n -+#endif\n ++int basics_test_main(int argc, const char *argv[])\n ++{\n ++\tRUN_TEST(test_common_prefix);\n ++\tRUN_TEST(test_parse_names_normal);\n ++\tRUN_TEST(test_parse_names_drop_empty);\n ++\tRUN_TEST(test_binsearch);\n ++\tRUN_TEST(test_names_length);\n ++\treturn 0;\n ++}\n \n ## reftable/publicbasics.c (new) ##\n @@\n @@ reftable/reftable-malloc.h (new)\n +\n +#include \n +\n -+/*\n -+ Overrides the functions to use for memory management.\n -+ */\n ++/* Overrides the functions to use for memory management. */\n +void reftable_set_alloc(void *(*malloc)(size_t),\n +\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n +\n @@ reftable/test_framework.h (new)\n +\t\tabort(); \\\n +\t}\n +\n ++#define RUN_TEST(f) \\\n ++\tfprintf(stderr, \"running %s\\n\", #f); \\\n ++\tfflush(stderr); \\\n ++\tf();\n ++\n +void set_test_hash(uint8_t *p, int i);\n +\n +/* Like strbuf_add, but suitable for passing to reftable_new_writer\n @@ t/t0032-reftable-unittest.sh (new)\n +. ./test-lib.sh\n +\n +test_expect_success 'unittests' '\n -+ test-tool reftable\n ++\tTMPDIR=$(pwd) && export TMPDIR &&\n ++\ttest-tool reftable\n +'\n +\n +test_done\n 6: f71e82b995d ! 5: aaa817f8598 reftable: add blocksource, an abstraction for random access reads\n @@ reftable/reftable-blocksource.h (new)\n +};\n +\n +/* a contiguous segment of bytes. It keeps track of its generating block_source\n -+ so it can return itself into the pool.\n -+*/\n ++ * so it can return itself into the pool. */\n +struct reftable_block {\n +\tuint8_t *data;\n +\tint len;\n 7: bbe2ed71734 ! 6: e13afe31525 reftable: (de)serialization for the polymorphic record type.\n @@ reftable/record.c (new)\n +\treturn 0;\n +}\n +\n ++uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec)\n ++{\n ++\tswitch (rec->value_type) {\n ++\tcase REFTABLE_REF_VAL1:\n ++\t\treturn rec->value.val1;\n ++\tcase REFTABLE_REF_VAL2:\n ++\t\treturn rec->value.val2.value;\n ++\tdefault:\n ++\t\treturn NULL;\n ++\t}\n ++}\n ++\n ++uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec)\n ++{\n ++\tswitch (rec->value_type) {\n ++\tcase REFTABLE_REF_VAL2:\n ++\t\treturn rec->value.val2.target_value;\n ++\tdefault:\n ++\t\treturn NULL;\n ++\t}\n ++}\n ++\n +static int decode_string(struct strbuf *dest, struct string_view in)\n +{\n +\tint start_len = in.len;\n @@ reftable/record.c (new)\n +\tassert(hash_size > 0);\n +\n +\t/* This is simple and correct, but we could probably reuse the hash\n -+\t fields. */\n ++\t * fields. */\n +\treftable_ref_record_release(ref);\n +\tif (src->refname != NULL) {\n +\t\tref->refname = xstrdup(src->refname);\n +\t}\n -+\n -+\tif (src->target != NULL) {\n -+\t\tref->target = xstrdup(src->target);\n -+\t}\n -+\n -+\tif (src->target_value != NULL) {\n -+\t\tref->target_value = reftable_malloc(hash_size);\n -+\t\tmemcpy(ref->target_value, src->target_value, hash_size);\n -+\t}\n -+\n -+\tif (src->value != NULL) {\n -+\t\tref->value = reftable_malloc(hash_size);\n -+\t\tmemcpy(ref->value, src->value, hash_size);\n -+\t}\n +\tref->update_index = src->update_index;\n ++\tref->value_type = src->value_type;\n ++\tswitch (src->value_type) {\n ++\tcase REFTABLE_REF_DELETION:\n ++\t\tbreak;\n ++\tcase REFTABLE_REF_VAL1:\n ++\t\tref->value.val1 = reftable_malloc(hash_size);\n ++\t\tmemcpy(ref->value.val1, src->value.val1, hash_size);\n ++\t\tbreak;\n ++\tcase REFTABLE_REF_VAL2:\n ++\t\tref->value.val2.value = reftable_malloc(hash_size);\n ++\t\tmemcpy(ref->value.val2.value, src->value.val2.value, hash_size);\n ++\t\tref->value.val2.target_value = reftable_malloc(hash_size);\n ++\t\tmemcpy(ref->value.val2.target_value,\n ++\t\t src->value.val2.target_value, hash_size);\n ++\t\tbreak;\n ++\tcase REFTABLE_REF_SYMREF:\n ++\t\tref->value.symref = xstrdup(src->value.symref);\n ++\t\tbreak;\n ++\t}\n +}\n +\n +static char hexdigit(int c)\n @@ reftable/record.c (new)\n +void reftable_ref_record_print(struct reftable_ref_record *ref,\n +\t\t\t uint32_t hash_id)\n +{\n -+\tchar hex[SHA256_SIZE + 1] = { 0 };\n ++\tchar hex[2 * SHA256_SIZE + 1] = { 0 }; /* BUG */\n +\tprintf(\"ref{%s(%\" PRIu64 \") \", ref->refname, ref->update_index);\n -+\tif (ref->value != NULL) {\n -+\t\thex_format(hex, ref->value, hash_size(hash_id));\n -+\t\tprintf(\"%s\", hex);\n -+\t}\n -+\tif (ref->target_value != NULL) {\n -+\t\thex_format(hex, ref->target_value, hash_size(hash_id));\n -+\t\tprintf(\" (T %s)\", hex);\n -+\t}\n -+\tif (ref->target != NULL) {\n -+\t\tprintf(\"=> %s\", ref->target);\n ++\tswitch (ref->value_type) {\n ++\tcase REFTABLE_REF_SYMREF:\n ++\t\tprintf(\"=> %s\", ref->value.symref);\n ++\t\tbreak;\n ++\tcase REFTABLE_REF_VAL2:\n ++\t\thex_format(hex, ref->value.val2.value, hash_size(hash_id));\n ++\t\tprintf(\"val 2 %s\", hex);\n ++\t\thex_format(hex, ref->value.val2.target_value,\n ++\t\t\t hash_size(hash_id));\n ++\t\tprintf(\"(T %s)\", hex);\n ++\t\tbreak;\n ++\tcase REFTABLE_REF_VAL1:\n ++\t\thex_format(hex, ref->value.val1, hash_size(hash_id));\n ++\t\tprintf(\"val 1 %s\", hex);\n ++\t\tbreak;\n ++\tcase REFTABLE_REF_DELETION:\n ++\t\tprintf(\"delete\");\n ++\t\tbreak;\n +\t}\n +\tprintf(\"}\\n\");\n +}\n @@ reftable/record.c (new)\n +\n +void reftable_ref_record_release(struct reftable_ref_record *ref)\n +{\n ++\tswitch (ref->value_type) {\n ++\tcase REFTABLE_REF_SYMREF:\n ++\t\treftable_free(ref->value.symref);\n ++\t\tbreak;\n ++\tcase REFTABLE_REF_VAL2:\n ++\t\treftable_free(ref->value.val2.target_value);\n ++\t\treftable_free(ref->value.val2.value);\n ++\t\tbreak;\n ++\tcase REFTABLE_REF_VAL1:\n ++\t\treftable_free(ref->value.val1);\n ++\t\tbreak;\n ++\tcase REFTABLE_REF_DELETION:\n ++\t\tbreak;\n ++\tdefault:\n ++\t\tabort();\n ++\t}\n ++\n +\treftable_free(ref->refname);\n -+\treftable_free(ref->target);\n -+\treftable_free(ref->target_value);\n -+\treftable_free(ref->value);\n +\tmemset(ref, 0, sizeof(struct reftable_ref_record));\n +}\n +\n @@ reftable/record.c (new)\n +{\n +\tconst struct reftable_ref_record *r =\n +\t\t(const struct reftable_ref_record *)rec;\n -+\tif (r->value != NULL) {\n -+\t\tif (r->target_value != NULL) {\n -+\t\t\treturn 2;\n -+\t\t} else {\n -+\t\t\treturn 1;\n -+\t\t}\n -+\t} else if (r->target != NULL)\n -+\t\treturn 3;\n -+\treturn 0;\n ++\treturn r->value_type;\n +}\n +\n +static int reftable_ref_record_encode(const void *rec, struct string_view s,\n @@ reftable/record.c (new)\n +\t\treturn -1;\n +\tstring_view_consume(&s, n);\n +\n -+\tif (r->value != NULL) {\n -+\t\tif (s.len < hash_size) {\n ++\tswitch (r->value_type) {\n ++\tcase REFTABLE_REF_SYMREF:\n ++\t\tn = encode_string(r->value.symref, s);\n ++\t\tif (n < 0) {\n +\t\t\treturn -1;\n +\t\t}\n -+\t\tmemcpy(s.buf, r->value, hash_size);\n -+\t\tstring_view_consume(&s, hash_size);\n -+\t}\n -+\n -+\tif (r->target_value != NULL) {\n -+\t\tif (s.len < hash_size) {\n ++\t\tstring_view_consume(&s, n);\n ++\t\tbreak;\n ++\tcase REFTABLE_REF_VAL2:\n ++\t\tif (s.len < 2 * hash_size) {\n +\t\t\treturn -1;\n +\t\t}\n -+\t\tmemcpy(s.buf, r->target_value, hash_size);\n ++\t\tmemcpy(s.buf, r->value.val2.value, hash_size);\n +\t\tstring_view_consume(&s, hash_size);\n -+\t}\n -+\n -+\tif (r->target != NULL) {\n -+\t\tint n = encode_string(r->target, s);\n -+\t\tif (n < 0) {\n ++\t\tmemcpy(s.buf, r->value.val2.target_value, hash_size);\n ++\t\tstring_view_consume(&s, hash_size);\n ++\t\tbreak;\n ++\tcase REFTABLE_REF_VAL1:\n ++\t\tif (s.len < hash_size) {\n +\t\t\treturn -1;\n +\t\t}\n -+\t\tstring_view_consume(&s, n);\n ++\t\tmemcpy(s.buf, r->value.val1, hash_size);\n ++\t\tstring_view_consume(&s, hash_size);\n ++\t\tbreak;\n ++\tcase REFTABLE_REF_DELETION:\n ++\t\tbreak;\n ++\tdefault:\n ++\t\tabort();\n +\t}\n +\n +\treturn start.len - s.len;\n @@ reftable/record.c (new)\n +{\n +\tstruct reftable_ref_record *r = (struct reftable_ref_record *)rec;\n +\tstruct string_view start = in;\n -+\tint seen_value = 0;\n -+\tint seen_target_value = 0;\n -+\tint seen_target = 0;\n -+\n -+\tint n = get_var_int(&r->update_index, &in);\n ++\tuint64_t update_index = 0;\n ++\tint n = get_var_int(&update_index, &in);\n +\tif (n < 0)\n +\t\treturn n;\n -+\tassert(hash_size > 0);\n -+\n +\tstring_view_consume(&in, n);\n +\n ++\treftable_ref_record_release(r);\n ++\n ++\tassert(hash_size > 0);\n ++\n +\tr->refname = reftable_realloc(r->refname, key.len + 1);\n +\tmemcpy(r->refname, key.buf, key.len);\n ++\tr->update_index = update_index;\n +\tr->refname[key.len] = 0;\n -+\n ++\tr->value_type = val_type;\n +\tswitch (val_type) {\n -+\tcase 1:\n -+\tcase 2:\n ++\tcase REFTABLE_REF_VAL1:\n +\t\tif (in.len < hash_size) {\n +\t\t\treturn -1;\n +\t\t}\n +\n -+\t\tif (r->value == NULL) {\n -+\t\t\tr->value = reftable_malloc(hash_size);\n -+\t\t}\n -+\t\tseen_value = 1;\n -+\t\tmemcpy(r->value, in.buf, hash_size);\n ++\t\tr->value.val1 = reftable_malloc(hash_size);\n ++\t\tmemcpy(r->value.val1, in.buf, hash_size);\n +\t\tstring_view_consume(&in, hash_size);\n -+\t\tif (val_type == 1) {\n -+\t\t\tbreak;\n -+\t\t}\n -+\t\tif (r->target_value == NULL) {\n -+\t\t\tr->target_value = reftable_malloc(hash_size);\n ++\t\tbreak;\n ++\n ++\tcase REFTABLE_REF_VAL2:\n ++\t\tif (in.len < 2 * hash_size) {\n ++\t\t\treturn -1;\n +\t\t}\n -+\t\tseen_target_value = 1;\n -+\t\tmemcpy(r->target_value, in.buf, hash_size);\n ++\n ++\t\tr->value.val2.value = reftable_malloc(hash_size);\n ++\t\tmemcpy(r->value.val2.value, in.buf, hash_size);\n ++\t\tstring_view_consume(&in, hash_size);\n ++\n ++\t\tr->value.val2.target_value = reftable_malloc(hash_size);\n ++\t\tmemcpy(r->value.val2.target_value, in.buf, hash_size);\n +\t\tstring_view_consume(&in, hash_size);\n +\t\tbreak;\n -+\tcase 3: {\n ++\n ++\tcase REFTABLE_REF_SYMREF: {\n +\t\tstruct strbuf dest = STRBUF_INIT;\n +\t\tint n = decode_string(&dest, in);\n +\t\tif (n < 0) {\n +\t\t\treturn -1;\n +\t\t}\n +\t\tstring_view_consume(&in, n);\n -+\t\tseen_target = 1;\n -+\t\tif (r->target != NULL) {\n -+\t\t\treftable_free(r->target);\n -+\t\t}\n -+\t\tr->target = dest.buf;\n ++\t\tr->value.symref = dest.buf;\n +\t} break;\n +\n -+\tcase 0:\n ++\tcase REFTABLE_REF_DELETION:\n +\t\tbreak;\n +\tdefault:\n +\t\tabort();\n +\t\tbreak;\n +\t}\n +\n -+\tif (!seen_target && r->target != NULL) {\n -+\t\tFREE_AND_NULL(r->target);\n -+\t}\n -+\tif (!seen_target_value && r->target_value != NULL) {\n -+\t\tFREE_AND_NULL(r->target_value);\n -+\t}\n -+\tif (!seen_value && r->value != NULL) {\n -+\t\tFREE_AND_NULL(r->value);\n -+\t}\n -+\n +\treturn start.len - in.len;\n +}\n +\n @@ reftable/record.c (new)\n +\treturn a == b;\n +}\n +\n -+static int str_equal(char *a, char *b)\n -+{\n -+\tif (a != NULL && b != NULL)\n -+\t\treturn 0 == strcmp(a, b);\n -+\n -+\treturn a == b;\n -+}\n -+\n +int reftable_ref_record_equal(struct reftable_ref_record *a,\n +\t\t\t struct reftable_ref_record *b, int hash_size)\n +{\n +\tassert(hash_size > 0);\n -+\treturn 0 == strcmp(a->refname, b->refname) &&\n -+\t a->update_index == b->update_index &&\n -+\t hash_equal(a->value, b->value, hash_size) &&\n -+\t hash_equal(a->target_value, b->target_value, hash_size) &&\n -+\t str_equal(a->target, b->target);\n ++\tif (!(0 == strcmp(a->refname, b->refname) &&\n ++\t a->update_index == b->update_index &&\n ++\t a->value_type == b->value_type))\n ++\t\treturn 0;\n ++\n ++\tswitch (a->value_type) {\n ++\tcase REFTABLE_REF_SYMREF:\n ++\t\treturn !strcmp(a->value.symref, b->value.symref);\n ++\tcase REFTABLE_REF_VAL2:\n ++\t\treturn hash_equal(a->value.val2.value, b->value.val2.value,\n ++\t\t\t\t hash_size) &&\n ++\t\t hash_equal(a->value.val2.target_value,\n ++\t\t\t\t b->value.val2.target_value, hash_size);\n ++\tcase REFTABLE_REF_VAL1:\n ++\t\treturn hash_equal(a->value.val1, b->value.val1, hash_size);\n ++\tcase REFTABLE_REF_DELETION:\n ++\t\treturn 1;\n ++\tdefault:\n ++\t\tabort();\n ++\t}\n +}\n +\n +int reftable_ref_record_compare_name(const void *a, const void *b)\n @@ reftable/record.c (new)\n +\n +int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref)\n +{\n -+\treturn ref->value == NULL && ref->target == NULL &&\n -+\t ref->target_value == NULL;\n ++\treturn ref->value_type == REFTABLE_REF_DELETION;\n +}\n +\n +int reftable_log_record_compare_key(const void *a, const void *b)\n @@ reftable/record.h (new)\n +#include \"reftable-record.h\"\n +\n +/*\n -+ A substring of existing string data. This structure takes no responsibility\n -+ for the lifetime of the data it points to.\n -+*/\n ++ * A substring of existing string data. This structure takes no responsibility\n ++ * for the lifetime of the data it points to.\n ++ */\n +struct string_view {\n +\tuint8_t *buf;\n +\tsize_t len;\n @@ reftable/record.h (new)\n +struct reftable_record reftable_new_record(uint8_t typ);\n +\n +/* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns\n -+ number of bytes written. */\n ++ * number of bytes written. */\n +int reftable_encode_key(int *is_restart, struct string_view dest,\n +\t\t\tstruct strbuf prev_key, struct strbuf key,\n +\t\t\tuint8_t extra);\n @@ reftable/record_test.c (new)\n +{\n +\tint i = 0;\n +\n -+\tfor (i = 0; i <= 3; i++) {\n ++\tfor (i = REFTABLE_REF_DELETION; i < REFTABLE_NR_REF_VALUETYPES; i++) {\n +\t\tstruct reftable_ref_record in = { NULL };\n -+\t\tstruct reftable_ref_record out = {\n -+\t\t\t.refname = xstrdup(\"old name\"),\n -+\t\t\t.value = reftable_calloc(SHA1_SIZE),\n -+\t\t\t.target_value = reftable_calloc(SHA1_SIZE),\n -+\t\t\t.target = xstrdup(\"old value\"),\n -+\t\t};\n ++\t\tstruct reftable_ref_record out = { NULL };\n +\t\tstruct reftable_record rec_out = { NULL };\n +\t\tstruct strbuf key = STRBUF_INIT;\n +\t\tstruct reftable_record rec = { NULL };\n @@ reftable/record_test.c (new)\n +\n +\t\tint n, m;\n +\n ++\t\tin.value_type = i;\n +\t\tswitch (i) {\n -+\t\tcase 0:\n ++\t\tcase REFTABLE_REF_DELETION:\n +\t\t\tbreak;\n -+\t\tcase 1:\n -+\t\t\tin.value = reftable_malloc(SHA1_SIZE);\n -+\t\t\tset_hash(in.value, 1);\n ++\t\tcase REFTABLE_REF_VAL1:\n ++\t\t\tin.value.val1 = reftable_malloc(SHA1_SIZE);\n ++\t\t\tset_hash(in.value.val1, 1);\n +\t\t\tbreak;\n -+\t\tcase 2:\n -+\t\t\tin.value = reftable_malloc(SHA1_SIZE);\n -+\t\t\tset_hash(in.value, 1);\n -+\t\t\tin.target_value = reftable_malloc(SHA1_SIZE);\n -+\t\t\tset_hash(in.target_value, 2);\n ++\t\tcase REFTABLE_REF_VAL2:\n ++\t\t\tin.value.val2.value = reftable_malloc(SHA1_SIZE);\n ++\t\t\tset_hash(in.value.val2.value, 1);\n ++\t\t\tin.value.val2.target_value = reftable_malloc(SHA1_SIZE);\n ++\t\t\tset_hash(in.value.val2.target_value, 2);\n +\t\t\tbreak;\n -+\t\tcase 3:\n -+\t\t\tin.target = xstrdup(\"target\");\n ++\t\tcase REFTABLE_REF_SYMREF:\n ++\t\t\tin.value.symref = xstrdup(\"target\");\n +\t\t\tbreak;\n +\t\t}\n +\t\tin.refname = xstrdup(\"refs/heads/master\");\n @@ reftable/record_test.c (new)\n +\t\tm = reftable_record_decode(&rec_out, key, i, dest, SHA1_SIZE);\n +\t\tEXPECT(n == m);\n +\n -+\t\tEXPECT((out.value != NULL) == (in.value != NULL));\n -+\t\tEXPECT((out.target_value != NULL) == (in.target_value != NULL));\n -+\t\tEXPECT((out.target != NULL) == (in.target != NULL));\n ++\t\tEXPECT(reftable_ref_record_equal(&in, &out, SHA1_SIZE));\n +\t\treftable_record_release(&rec_out);\n +\n +\t\tstrbuf_release(&key);\n @@ reftable/record_test.c (new)\n +\n +int record_test_main(int argc, const char *argv[])\n +{\n -+\ttest_reftable_log_record_equal();\n -+\ttest_reftable_log_record_roundtrip();\n -+\ttest_reftable_ref_record_roundtrip();\n -+\ttest_varint_roundtrip();\n -+\ttest_key_roundtrip();\n -+\ttest_common_prefix();\n -+\ttest_reftable_obj_record_roundtrip();\n -+\ttest_reftable_index_record_roundtrip();\n -+\ttest_u24_roundtrip();\n ++\tRUN_TEST(test_reftable_log_record_equal);\n ++\tRUN_TEST(test_reftable_log_record_roundtrip);\n ++\tRUN_TEST(test_reftable_ref_record_roundtrip);\n ++\tRUN_TEST(test_varint_roundtrip);\n ++\tRUN_TEST(test_key_roundtrip);\n ++\tRUN_TEST(test_common_prefix);\n ++\tRUN_TEST(test_reftable_obj_record_roundtrip);\n ++\tRUN_TEST(test_reftable_index_record_roundtrip);\n ++\tRUN_TEST(test_u24_roundtrip);\n +\treturn 0;\n +}\n \n @@ reftable/reftable-record.h (new)\n +#include \n +\n +/*\n -+ Basic data types\n -+\n -+ Reftables store the state of each ref in struct reftable_ref_record, and they\n -+ store a sequence of reflog updates in struct reftable_log_record.\n -+*/\n ++ * Basic data types\n ++ *\n ++ * Reftables store the state of each ref in struct reftable_ref_record, and they\n ++ * store a sequence of reflog updates in struct reftable_log_record.\n ++ */\n +\n +/* reftable_ref_record holds a ref database entry target_value */\n +struct reftable_ref_record {\n +\tchar *refname; /* Name of the ref, malloced. */\n +\tuint64_t update_index; /* Logical timestamp at which this value is\n -+\t\t\t\t written */\n -+\tuint8_t *value; /* SHA1, or NULL. malloced. */\n -+\tuint8_t *target_value; /* peeled annotated tag, or NULL. malloced. */\n -+\tchar *target; /* symref, or NULL. malloced. */\n ++\t\t\t\t* written */\n ++\n ++\tenum {\n ++\t\t/* tombstone to hide deletions from earlier tables */\n ++\t\tREFTABLE_REF_DELETION = 0x0,\n ++\n ++\t\t/* a simple ref */\n ++\t\tREFTABLE_REF_VAL1 = 0x1,\n ++\t\t/* a tag, plus its peeled hash */\n ++\t\tREFTABLE_REF_VAL2 = 0x2,\n ++\n ++\t\t/* a symbolic reference */\n ++\t\tREFTABLE_REF_SYMREF = 0x3,\n ++#define REFTABLE_NR_REF_VALUETYPES 4\n ++\t} value_type;\n ++\tunion {\n ++\t\tuint8_t *val1; /* malloced hash. */\n ++\t\tstruct {\n ++\t\t\tuint8_t *value; /* first value, malloced hash */\n ++\t\t\tuint8_t *target_value; /* second value, malloced hash */\n ++\t\t} val2;\n ++\t\tchar *symref; /* referent, malloced 0-terminated string */\n ++\t} value;\n +};\n +\n ++/* Returns the first hash, or NULL if `rec` is not of type\n ++ * REFTABLE_REF_VAL1 or REFTABLE_REF_VAL2. */\n ++uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec);\n ++\n ++/* Returns the second hash, or NULL if `rec` is not of type\n ++ * REFTABLE_REF_VAL2. */\n ++uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec);\n ++\n +/* returns whether 'ref' represents a deletion */\n +int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);\n +\n -+/* prints a reftable_ref_record onto stdout */\n ++/* prints a reftable_ref_record onto stdout. Useful for debugging. */\n +void reftable_ref_record_print(struct reftable_ref_record *ref,\n +\t\t\t uint32_t hash_id);\n +\n -+/* frees and nulls all pointer values. */\n ++/* frees and nulls all pointer values inside `ref`. */\n +void reftable_ref_record_release(struct reftable_ref_record *ref);\n +\n -+/* returns whether two reftable_ref_records are the same */\n ++/* returns whether two reftable_ref_records are the same. Useful for testing. */\n +int reftable_ref_record_equal(struct reftable_ref_record *a,\n +\t\t\t struct reftable_ref_record *b, int hash_size);\n +\n 8: a358b052d56 ! 7: a48b9937642 reftable: reading/writing blocks\n @@ reftable/block.h (new)\n +#include \"reftable-blocksource.h\"\n +\n +/*\n -+ Writes reftable blocks. The block_writer is reused across blocks to minimize\n -+ allocation overhead.\n -+*/\n ++ * Writes reftable blocks. The block_writer is reused across blocks to minimize\n ++ * allocation overhead.\n ++ */\n +struct block_writer {\n +\tuint8_t *buf;\n +\tuint32_t block_size;\n @@ reftable/block.h (new)\n +};\n +\n +/*\n -+ initializes the blockwriter to write `typ` entries, using `buf` as temporary\n -+ storage. `buf` is not owned by the block_writer. */\n ++ * initializes the blockwriter to write `typ` entries, using `buf` as temporary\n ++ * storage. `buf` is not owned by the block_writer. */\n +void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n +\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n +\n -+/*\n -+ returns the block type (eg. 'r' for ref records.\n -+*/\n ++/* returns the block type (eg. 'r' for ref records. */\n +uint8_t block_writer_type(struct block_writer *bw);\n +\n +/* appends the record, or -1 if it doesn't fit. */\n @@ reftable/block_test.c (new)\n +\t\tmemset(hash, i, sizeof(hash));\n +\n +\t\tref.refname = name;\n -+\t\tref.value = hash;\n ++\t\tref.value_type = REFTABLE_REF_VAL1;\n ++\t\tref.value.val1 = hash;\n ++\n +\t\tnames[i] = xstrdup(name);\n +\t\tn = block_writer_add(&bw, &rec);\n +\t\tref.refname = NULL;\n -+\t\tref.value = NULL;\n ++\t\tref.value_type = REFTABLE_REF_DELETION;\n +\t\tEXPECT(n == 0);\n +\t}\n +\n @@ reftable/block_test.c (new)\n +\n +int block_test_main(int argc, const char *argv[])\n +{\n -+\ttest_block_read_write();\n ++\tRUN_TEST(test_block_read_write);\n +\treturn 0;\n +}\n \n 9: 24afac9c91a ! 8: 6968dbc3828 reftable: a generic binary tree implementation\n @@ reftable/tree.h (new)\n +};\n +\n +/* looks for `key` in `rootp` using `compare` as comparison function. If insert\n -+ is set, insert the key if it's not found. Else, return NULL.\n -+*/\n ++ * is set, insert the key if it's not found. Else, return NULL.\n ++ */\n +struct tree_node *tree_search(void *key, struct tree_node **rootp,\n +\t\t\t int (*compare)(const void *, const void *),\n +\t\t\t int insert);\n @@ reftable/tree.h (new)\n +\t\tvoid *arg);\n +\n +/*\n -+ deallocates the tree nodes recursively. Keys should be deallocated separately\n -+ by walking over the tree. */\n ++ * deallocates the tree nodes recursively. Keys should be deallocated separately\n ++ * by walking over the tree. */\n +void tree_free(struct tree_node *t);\n +\n +#endif\n @@ reftable/tree_test.c (new)\n +\n +int tree_test_main(int argc, const char *argv[])\n +{\n -+\ttest_tree();\n ++\tRUN_TEST(test_tree);\n +\treturn 0;\n +}\n \n 10: 5f0b7c55925 ! 9: ff5b424d12f reftable: write reftable files\n @@ reftable/reftable-writer.h (new)\n +\n +#include \"reftable-record.h\"\n +\n -+/*\n -+ Writing single reftables\n -+*/\n ++/* Writing single reftables */\n +\n +/* reftable_write_options sets options for writing a single reftable. */\n +struct reftable_write_options {\n @@ reftable/writer.c (new)\n +\n +\treftable_record_from_ref(&rec, ©);\n +\tcopy.update_index -= w->min_update_index;\n ++\n +\terr = writer_add_record(w, &rec);\n +\tif (err < 0)\n +\t\treturn err;\n +\n -+\tif (!w->opts.skip_index_objects && ref->value != NULL) {\n ++\tif (!w->opts.skip_index_objects &&\n ++\t reftable_ref_record_val1(ref) != NULL) {\n +\t\tstruct strbuf h = STRBUF_INIT;\n -+\t\tstrbuf_add(&h, (char *)ref->value, hash_size(w->opts.hash_id));\n ++\t\tstrbuf_add(&h, (char *)reftable_ref_record_val1(ref),\n ++\t\t\t hash_size(w->opts.hash_id));\n +\t\twriter_index_hash(w, &h);\n +\t\tstrbuf_release(&h);\n +\t}\n +\n -+\tif (!w->opts.skip_index_objects && ref->target_value != NULL) {\n ++\tif (!w->opts.skip_index_objects &&\n ++\t reftable_ref_record_val2(ref) != NULL) {\n +\t\tstruct strbuf h = STRBUF_INIT;\n -+\t\tstrbuf_add(&h, ref->target_value, hash_size(w->opts.hash_id));\n ++\t\tstrbuf_add(&h, reftable_ref_record_val2(ref),\n ++\t\t\t hash_size(w->opts.hash_id));\n +\t\twriter_index_hash(w, &h);\n +\t\tstrbuf_release(&h);\n +\t}\n @@ reftable/writer.h (new)\n +\tsize_t index_cap;\n +\n +\t/*\n -+\t tree for use with tsearch; used to populate the 'o' inverse OID\n -+\t map */\n ++\t * tree for use with tsearch; used to populate the 'o' inverse OID\n ++\t * map */\n +\tstruct tree_node *obj_index_tree;\n +\n +\tstruct reftable_stats stats;\n 11: 9aa2caade8c ! 10: 3c2de7dcc65 reftable: read reftable files\n @@ reftable/iter.c (new)\n +\t\t\t}\n +\t\t}\n +\n -+\t\tif ((ref->target_value != NULL &&\n -+\t\t !memcmp(fri->oid.buf, ref->target_value, fri->oid.len)) ||\n -+\t\t (ref->value != NULL &&\n -+\t\t !memcmp(fri->oid.buf, ref->value, fri->oid.len))) {\n ++\t\tif (ref->value_type == REFTABLE_REF_VAL2 &&\n ++\t\t (!memcmp(fri->oid.buf, ref->value.val2.target_value,\n ++\t\t\t fri->oid.len) ||\n ++\t\t !memcmp(fri->oid.buf, ref->value.val2.value,\n ++\t\t\t fri->oid.len)))\n ++\t\t\treturn 0;\n ++\n ++\t\tif (ref->value_type == REFTABLE_REF_VAL1 &&\n ++\t\t !memcmp(fri->oid.buf, ref->value.val1, fri->oid.len)) {\n +\t\t\treturn 0;\n +\t\t}\n +\t}\n @@ reftable/iter.c (new)\n +\t\t\t}\n +\t\t\tcontinue;\n +\t\t}\n -+\n -+\t\tif (!memcmp(it->oid.buf, ref->target_value, it->oid.len) ||\n -+\t\t !memcmp(it->oid.buf, ref->value, it->oid.len)) {\n ++\t\t/* BUG */\n ++\t\tif (!memcmp(it->oid.buf, ref->value.val2.target_value,\n ++\t\t\t it->oid.len) ||\n ++\t\t !memcmp(it->oid.buf, ref->value.val2.value, it->oid.len)) {\n +\t\t\treturn 0;\n +\t\t}\n +\t}\n @@ reftable/iter.h (new)\n +int iterator_next(struct reftable_iterator *it, struct reftable_record *rec);\n +\n +/* Returns true for a zeroed out iterator, such as the one returned from\n -+ iterator_destroy. */\n ++ * iterator_destroy. */\n +int iterator_is_null(struct reftable_iterator *it);\n +\n +/* iterator that produces only ref records that point to `oid` */\n @@ reftable/iter.h (new)\n +\t\t\t\t\t struct filtering_ref_iterator *);\n +\n +/* iterator that produces only ref records that point to `oid`,\n -+ but using the object index.\n ++ * but using the object index.\n + */\n +struct indexed_table_ref_iter {\n +\tstruct reftable_reader *r;\n @@ reftable/reftable-iterator.h (new)\n +#include \"reftable-record.h\"\n +\n +/* iterator is the generic interface for walking over data stored in a\n -+ reftable.\n -+*/\n ++ * reftable.\n ++ */\n +struct reftable_iterator {\n +\tstruct reftable_iterator_vtable *ops;\n +\tvoid *iter_arg;\n +};\n +\n +/* reads the next reftable_ref_record. Returns < 0 for error, 0 for OK and > 0:\n -+ end of iteration.\n -+*/\n ++ * end of iteration.\n ++ */\n +int reftable_iterator_next_ref(struct reftable_iterator *it,\n +\t\t\t struct reftable_ref_record *ref);\n +\n +/* reads the next reftable_log_record. Returns < 0 for error, 0 for OK and > 0:\n -+ end of iteration.\n -+*/\n ++ * end of iteration.\n ++ */\n +int reftable_iterator_next_log(struct reftable_iterator *it,\n +\t\t\t struct reftable_log_record *log);\n +\n @@ reftable/reftable-reader.h (new)\n +#include \"reftable-blocksource.h\"\n +\n +/*\n -+ Reading single tables\n -+\n -+ The follow routines are for reading single files. For an application-level\n -+ interface, skip ahead to struct reftable_merged_table and struct\n -+ reftable_stack.\n -+*/\n ++ * Reading single tables\n ++ *\n ++ * The follow routines are for reading single files. For an application-level\n ++ * interface, skip ahead to struct reftable_merged_table and struct\n ++ * reftable_stack.\n ++ */\n +\n +/* The reader struct is a handle to an open reftable file. */\n +struct reftable_reader;\n 12: 581a4200d64 ! 11: 03681b820d1 reftable: reftable file level tests\n @@ reftable/reftable_test.c (new)\n +\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n +\n +\t\tref.refname = name;\n -+\t\tref.value = hash;\n +\t\tref.update_index = update_index;\n ++\t\tref.value_type = REFTABLE_REF_VAL1;\n ++\t\tref.value.val1 = hash;\n +\t\t(*names)[i] = xstrdup(name);\n +\n +\t\tn = reftable_writer_add_ref(w, &ref);\n @@ reftable/reftable_test.c (new)\n +\t\terr = reftable_iterator_next_ref(&it, &ref);\n +\t\tEXPECT_ERR(err);\n +\t\tEXPECT(0 == strcmp(names[i], ref.refname));\n -+\t\tEXPECT(i == ref.value[0]);\n ++\t\tEXPECT(REFTABLE_REF_VAL1 == ref.value_type);\n ++\t\tEXPECT(i == ref.value.val1[0]);\n +\n +\t\treftable_ref_record_release(&ref);\n +\t\treftable_iterator_destroy(&it);\n @@ reftable/reftable_test.c (new)\n +\n +\t\tset_test_hash(hash1, i / 4);\n +\t\tset_test_hash(hash2, 3 + i / 4);\n -+\t\tref.value = hash1;\n -+\t\tref.target_value = hash2;\n ++\t\tref.value_type = REFTABLE_REF_VAL2;\n ++\t\tref.value.val2.value = hash1;\n ++\t\tref.value.val2.target_value = hash2;\n +\n +\t\t/* 80 bytes / entry, so 3 entries per block. Yields 17\n +\t\t */\n @@ reftable/reftable_test.c (new)\n +\n +int reftable_test_main(int argc, const char *argv[])\n +{\n -+\ttest_log_write_read();\n -+\ttest_table_read_write_seek_linear_sha256();\n -+\ttest_log_buffer_size();\n -+\ttest_table_write_small_table();\n -+\ttest_buffer();\n -+\ttest_table_read_api();\n -+\ttest_table_read_write_sequential();\n -+\ttest_table_read_write_seek_linear();\n -+\ttest_table_read_write_seek_index();\n -+\ttest_table_refs_for_no_index();\n -+\ttest_table_refs_for_obj_index();\n -+\ttest_table_empty();\n ++\tRUN_TEST(test_log_write_read);\n ++\tRUN_TEST(test_table_read_write_seek_linear_sha256);\n ++\tRUN_TEST(test_log_buffer_size);\n ++\tRUN_TEST(test_table_write_small_table);\n ++\tRUN_TEST(test_buffer);\n ++\tRUN_TEST(test_table_read_api);\n ++\tRUN_TEST(test_table_read_write_sequential);\n ++\tRUN_TEST(test_table_read_write_seek_linear);\n ++\tRUN_TEST(test_table_read_write_seek_index);\n ++\tRUN_TEST(test_table_refs_for_no_index);\n ++\tRUN_TEST(test_table_refs_for_obj_index);\n ++\tRUN_TEST(test_table_empty);\n +\treturn 0;\n +}\n \n 13: a26fb180add ! 12: 557183d3e3e reftable: rest of library\n @@ Makefile: REFTABLE_OBJS += reftable/error.o\n \n ## reftable/VERSION (new) ##\n @@\n -+b8ec0f74c74cb6752eb2033ad8e755a9c19aad15 C: add missing header\n ++9b4a54059db9a05c270c0a0587f245bc6868d576 C: use rand() rather than cobbled together random generator.\n \n ## reftable/dump.c (new) ##\n @@\n @@ reftable/merged_test.c (new)\n +\tstruct reftable_ref_record r1[] = { {\n +\t\t.refname = \"b\",\n +\t\t.update_index = 1,\n -+\t\t.value = hash1,\n ++\t\t.value_type = REFTABLE_REF_VAL1,\n ++\t\t.value.val1 = hash1,\n +\t} };\n +\tstruct reftable_ref_record r2[] = { {\n +\t\t.refname = \"a\",\n +\t\t.update_index = 2,\n ++\t\t.value_type = REFTABLE_REF_DELETION,\n +\t} };\n +\n +\tstruct reftable_ref_record *refs[] = { r1, r2 };\n @@ reftable/merged_test.c (new)\n +{\n +\tuint8_t hash1[SHA1_SIZE] = { 1 };\n +\tuint8_t hash2[SHA1_SIZE] = { 2 };\n -+\tstruct reftable_ref_record r1[] = { {\n -+\t\t\t\t\t\t .refname = \"a\",\n -+\t\t\t\t\t\t .update_index = 1,\n -+\t\t\t\t\t\t .value = hash1,\n -+\t\t\t\t\t },\n -+\t\t\t\t\t {\n -+\t\t\t\t\t\t .refname = \"b\",\n -+\t\t\t\t\t\t .update_index = 1,\n -+\t\t\t\t\t\t .value = hash1,\n -+\t\t\t\t\t },\n -+\t\t\t\t\t {\n -+\t\t\t\t\t\t .refname = \"c\",\n -+\t\t\t\t\t\t .update_index = 1,\n -+\t\t\t\t\t\t .value = hash1,\n -+\t\t\t\t\t } };\n ++\tstruct reftable_ref_record r1[] = {\n ++\t\t{\n ++\t\t\t.refname = \"a\",\n ++\t\t\t.update_index = 1,\n ++\t\t\t.value_type = REFTABLE_REF_VAL1,\n ++\t\t\t.value.val1 = hash1,\n ++\t\t},\n ++\t\t{\n ++\t\t\t.refname = \"b\",\n ++\t\t\t.update_index = 1,\n ++\t\t\t.value_type = REFTABLE_REF_VAL1,\n ++\t\t\t.value.val1 = hash1,\n ++\t\t},\n ++\t\t{\n ++\t\t\t.refname = \"c\",\n ++\t\t\t.update_index = 1,\n ++\t\t\t.value_type = REFTABLE_REF_VAL1,\n ++\t\t\t.value.val1 = hash1,\n ++\t\t}\n ++\t};\n +\tstruct reftable_ref_record r2[] = { {\n +\t\t.refname = \"a\",\n +\t\t.update_index = 2,\n ++\t\t.value_type = REFTABLE_REF_DELETION,\n +\t} };\n +\tstruct reftable_ref_record r3[] = {\n +\t\t{\n +\t\t\t.refname = \"c\",\n +\t\t\t.update_index = 3,\n -+\t\t\t.value = hash2,\n ++\t\t\t.value_type = REFTABLE_REF_VAL1,\n ++\t\t\t.value.val1 = hash2,\n +\t\t},\n +\t\t{\n +\t\t\t.refname = \"d\",\n +\t\t\t.update_index = 3,\n -+\t\t\t.value = hash1,\n ++\t\t\t.value_type = REFTABLE_REF_VAL1,\n ++\t\t\t.value.val1 = hash1,\n +\t\t},\n +\t};\n +\n @@ reftable/merged_test.c (new)\n +\n +int merged_test_main(int argc, const char *argv[])\n +{\n -+\ttest_merged_between();\n -+\ttest_pq();\n -+\ttest_merged();\n -+\ttest_default_write_opts();\n ++\tRUN_TEST(test_merged_between);\n ++\tRUN_TEST(test_pq);\n ++\tRUN_TEST(test_merged);\n ++\tRUN_TEST(test_default_write_opts);\n +\treturn 0;\n +}\n \n @@ reftable/refname_test.c (new)\n +\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n +\tstruct reftable_ref_record rec = {\n +\t\t.refname = \"a/b\",\n -+\t\t.target = \"destination\", /* make sure it's not a symref. */\n ++\t\t.value_type = REFTABLE_REF_SYMREF,\n ++\t\t.value.symref = \"destination\", /* make sure it's not a symref.\n ++\t\t\t\t\t\t*/\n +\t\t.update_index = 1,\n +\t};\n +\tint err;\n @@ reftable/refname_test.c (new)\n +\n +int refname_test_main(int argc, const char *argv[])\n +{\n -+\ttest_conflict();\n ++\tRUN_TEST(test_conflict);\n +\treturn 0;\n +}\n \n @@ reftable/reftable-generic.h (new)\n +#include \"reftable-merged.h\"\n +\n +/*\n -+ Provides a unified API for reading tables, either merged tables, or single\n -+ readers.\n -+*/\n ++ * Provides a unified API for reading tables, either merged tables, or single\n ++ * readers. */\n +struct reftable_table {\n +\tstruct reftable_table_vtable *ops;\n +\tvoid *table_arg;\n @@ reftable/reftable-merged.h (new)\n +#include \"reftable-iterator.h\"\n +\n +/*\n -+ Merged tables\n -+\n -+ A ref database kept in a sequence of table files. The merged_table presents a\n -+ unified view to reading (seeking, iterating) a sequence of immutable tables.\n -+\n -+ The merged tables are on purpose kept disconnected from their actual storage\n -+ (eg. files on disk), because it is useful to merge tables aren't files. For\n -+ example, the per-workspace and global ref namespace can be implemented as a\n -+ merged table of two stacks of file-backed reftables.\n -+*/\n ++ * Merged tables\n ++ *\n ++ * A ref database kept in a sequence of table files. The merged_table presents a\n ++ * unified view to reading (seeking, iterating) a sequence of immutable tables.\n ++ *\n ++ * The merged tables are on purpose kept disconnected from their actual storage\n ++ * (eg. files on disk), because it is useful to merge tables aren't files. For\n ++ * example, the per-workspace and global ref namespace can be implemented as a\n ++ * merged table of two stacks of file-backed reftables.\n ++ */\n +\n +/* A merged table is implements seeking/iterating over a stack of tables. */\n +struct reftable_merged_table;\n @@ reftable/reftable-stack.h (new)\n +#include \"reftable-writer.h\"\n +\n +/*\n -+ The stack presents an interface to a mutable sequence of reftables.\n ++ * The stack presents an interface to a mutable sequence of reftables.\n +\n -+ A stack can be mutated by pushing a table to the top of the stack.\n ++ * A stack can be mutated by pushing a table to the top of the stack.\n +\n -+ The reftable_stack automatically compacts files on disk to ensure good\n -+ amortized performance.\n -+*/\n ++ * The reftable_stack automatically compacts files on disk to ensure good\n ++ * amortized performance.\n ++ *\n ++ * For windows and other platforms that cannot have open files as rename\n ++ * destinations, concurrent access from multiple processes needs the rand()\n ++ * random seed to be randomized.\n ++ */\n +struct reftable_stack;\n +\n +/* open a new reftable stack. The tables along with the table list will be\n -+ stored in 'dir'. Typically, this should be .git/reftables.\n -+*/\n ++ * stored in 'dir'. Typically, this should be .git/reftables.\n ++ */\n +int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n +\t\t struct reftable_write_options config);\n +\n @@ reftable/reftable-stack.h (new)\n +struct reftable_addition;\n +\n +/*\n -+ returns a new transaction to add reftables to the given stack. As a side\n -+ effect, the ref database is locked.\n -+*/\n ++ * returns a new transaction to add reftables to the given stack. As a side\n ++ * effect, the ref database is locked.\n ++ */\n +int reftable_stack_new_addition(struct reftable_addition **dest,\n +\t\t\t\tstruct reftable_stack *st);\n +\n @@ reftable/reftable-stack.h (new)\n +int reftable_addition_commit(struct reftable_addition *add);\n +\n +/* Release all non-committed data from the transaction, and deallocate the\n -+ transaction. Releases the lock if held. */\n ++ * transaction. Releases the lock if held. */\n +void reftable_addition_destroy(struct reftable_addition *add);\n +\n +/* add a new table to the stack. The write_table function must call\n -+ reftable_writer_set_limits, add refs and return an error value. */\n ++ * reftable_writer_set_limits, add refs and return an error value. */\n +int reftable_stack_add(struct reftable_stack *st,\n +\t\t int (*write_table)(struct reftable_writer *wr,\n +\t\t\t\t\t void *write_arg),\n +\t\t void *write_arg);\n +\n +/* returns the merged_table for seeking. This table is valid until the\n -+ next write or reload, and should not be closed or deleted.\n -+*/\n ++ * next write or reload, and should not be closed or deleted.\n ++ */\n +struct reftable_merged_table *\n +reftable_stack_merged_table(struct reftable_stack *st);\n +\n @@ reftable/reftable-stack.h (new)\n +/* heuristically compact unbalanced table stack. */\n +int reftable_stack_auto_compact(struct reftable_stack *st);\n +\n -+/* convenience function to read a single ref. Returns < 0 for error, 0\n -+ for success, and 1 if ref not found. */\n ++/* convenience function to read a single ref. Returns < 0 for error, 0 for\n ++ * success, and 1 if ref not found. */\n +int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n +\t\t\t struct reftable_ref_record *ref);\n +\n -+/* convenience function to read a single log. Returns < 0 for error, 0\n -+ for success, and 1 if ref not found. */\n ++/* convenience function to read a single log. Returns < 0 for error, 0 for\n ++ * success, and 1 if ref not found. */\n +int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n +\t\t\t struct reftable_log_record *log);\n +\n @@ reftable/stack.c (new)\n +static void format_name(struct strbuf *dest, uint64_t min, uint64_t max)\n +{\n +\tchar buf[100];\n -+\tsnprintf(buf, sizeof(buf), \"0x%012\" PRIx64 \"-0x%012\" PRIx64, min, max);\n ++\tuint32_t rnd = (uint32_t)rand();\n ++\tsnprintf(buf, sizeof(buf), \"0x%012\" PRIx64 \"-0x%012\" PRIx64 \"-%08x\",\n ++\t\t min, max, rnd);\n +\tstrbuf_reset(dest);\n +\tstrbuf_addstr(dest, buf);\n +}\n @@ reftable/stack.c (new)\n +\tint lock_file_fd;\n +\tstruct strbuf lock_file_name;\n +\tstruct reftable_stack *stack;\n -+\tchar **names;\n ++\n +\tchar **new_tables;\n +\tint new_tables_len;\n +\tuint64_t next_update_index;\n @@ reftable/stack.c (new)\n +\tstruct strbuf nm = STRBUF_INIT;\n +\tfor (i = 0; i < add->new_tables_len; i++) {\n +\t\tstrbuf_reset(&nm);\n -+\t\tstrbuf_addstr(&nm, add->stack->list_file);\n ++\t\tstrbuf_addstr(&nm, add->stack->reftable_dir);\n +\t\tstrbuf_addstr(&nm, \"/\");\n +\t\tstrbuf_addstr(&nm, add->new_tables[i]);\n +\t\tunlink(nm.buf);\n @@ reftable/stack.c (new)\n +\t\tstrbuf_release(&add->lock_file_name);\n +\t}\n +\n -+\tfree_names(add->names);\n -+\tadd->names = NULL;\n +\tstrbuf_release(&nm);\n +}\n +\n @@ reftable/stack.c (new)\n +\t\tgoto done;\n +\t}\n +\n -+\terr = reftable_stack_reload(add->stack);\n ++\t/* success, no more state to clean up. */\n ++\tstrbuf_release(&add->lock_file_name);\n ++\tfor (i = 0; i < add->new_tables_len; i++) {\n ++\t\treftable_free(add->new_tables[i]);\n ++\t}\n ++\treftable_free(add->new_tables);\n ++\tadd->new_tables = NULL;\n ++\tadd->new_tables_len = 0;\n +\n ++\terr = reftable_stack_reload(add->stack);\n +done:\n +\treftable_addition_close(add);\n +\treturn err;\n @@ reftable/stack.c (new)\n +\tstrbuf_addstr(&tab_file_name, \"/\");\n +\tstrbuf_addbuf(&tab_file_name, &next_name);\n +\n -+\t/* TODO: should check destination out of paranoia */\n ++\t/*\n ++\t On windows, this relies on rand() picking a unique destination name.\n ++\t Maybe we should do retry loop as well?\n ++\t */\n +\terr = rename(temp_tab_file_name.buf, tab_file_name.buf);\n +\tif (err < 0) {\n +\t\terr = REFTABLE_IO_ERROR;\n @@ reftable/stack.c (new)\n +\n +\tformat_name(&next_name,\n +\t\t reftable_reader_min_update_index(st->readers[first]),\n -+\t\t reftable_reader_max_update_index(st->readers[first]));\n ++\t\t reftable_reader_max_update_index(st->readers[last]));\n +\n +\tstrbuf_reset(temp_tab);\n +\tstrbuf_addstr(temp_tab, st->reftable_dir);\n @@ reftable/stack.c (new)\n +\t\tif (err < 0) {\n +\t\t\tbreak;\n +\t\t}\n ++\n +\t\tif (first == 0 && reftable_ref_record_is_deletion(&ref)) {\n +\t\t\tcontinue;\n +\t\t}\n @@ reftable/stack.c (new)\n +\tstrbuf_addbuf(&new_table_path, &new_table_name);\n +\n +\tif (!is_empty_table) {\n ++\t\t/* retry? */\n +\t\terr = rename(temp_tab_file_name.buf, new_table_path.buf);\n +\t\tif (err < 0) {\n +\t\t\terr = REFTABLE_IO_ERROR;\n @@ reftable/stack_test.c (new)\n +\tstrbuf_release(&path);\n +}\n +\n ++static char *get_tmp_template(const char *prefix)\n ++{\n ++\tconst char *tmp = getenv(\"TMPDIR\");\n ++\tstatic char template[1024];\n ++\tsnprintf(template, sizeof(template) - 1, \"%s/%s.XXXXXX\",\n ++\t\t tmp ? tmp : \"/tmp\", prefix);\n ++\treturn template;\n ++}\n ++\n +static void test_read_file(void)\n +{\n -+\tchar fn[256] = \"/tmp/stack.test_read_file.XXXXXX\";\n ++\tchar *fn = get_tmp_template(__FUNCTION__);\n +\tint fd = mkstemp(fn);\n +\tchar out[1024] = \"line1\\n\\nline2\\nline3\";\n +\tint n, err;\n @@ reftable/stack_test.c (new)\n +\n +static void test_reftable_stack_add_one(void)\n +{\n -+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n ++\tchar *dir = get_tmp_template(__FUNCTION__);\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tint err;\n +\tstruct reftable_ref_record ref = {\n +\t\t.refname = \"HEAD\",\n +\t\t.update_index = 1,\n -+\t\t.target = \"master\",\n ++\t\t.value_type = REFTABLE_REF_SYMREF,\n ++\t\t.value.symref = \"master\",\n +\t};\n +\tstruct reftable_ref_record dest = { NULL };\n +\n @@ reftable/stack_test.c (new)\n +\n +\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n +\tEXPECT_ERR(err);\n -+\tEXPECT(0 == strcmp(\"master\", dest.target));\n ++\tEXPECT(0 == strcmp(\"master\", dest.value.symref));\n +\n +\treftable_ref_record_release(&dest);\n +\treftable_stack_destroy(st);\n @@ reftable/stack_test.c (new)\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st1 = NULL;\n +\tstruct reftable_stack *st2 = NULL;\n -+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n ++\tchar *dir = get_tmp_template(__FUNCTION__);\n +\tint err;\n +\tstruct reftable_ref_record ref1 = {\n +\t\t.refname = \"HEAD\",\n +\t\t.update_index = 1,\n -+\t\t.target = \"master\",\n ++\t\t.value_type = REFTABLE_REF_SYMREF,\n ++\t\t.value.symref = \"master\",\n +\t};\n +\tstruct reftable_ref_record ref2 = {\n +\t\t.refname = \"branch2\",\n +\t\t.update_index = 2,\n -+\t\t.target = \"master\",\n ++\t\t.value_type = REFTABLE_REF_SYMREF,\n ++\t\t.value.symref = \"master\",\n +\t};\n +\n +\tEXPECT(mkdtemp(dir));\n +\n ++\t/* simulate multi-process access to the same stack\n ++\t by creating two stacks for the same directory.\n ++\t */\n +\terr = reftable_new_stack(&st1, dir, cfg);\n +\tEXPECT_ERR(err);\n +\n @@ reftable/stack_test.c (new)\n +\n +static void test_reftable_stack_transaction_api(void)\n +{\n -+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n ++\tchar *dir = get_tmp_template(__FUNCTION__);\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tint err;\n @@ reftable/stack_test.c (new)\n +\tstruct reftable_ref_record ref = {\n +\t\t.refname = \"HEAD\",\n +\t\t.update_index = 1,\n -+\t\t.target = \"master\",\n ++\t\t.value_type = REFTABLE_REF_SYMREF,\n ++\t\t.value.symref = \"master\",\n +\t};\n +\tstruct reftable_ref_record dest = { NULL };\n +\n @@ reftable/stack_test.c (new)\n +\n +\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n +\tEXPECT_ERR(err);\n -+\tEXPECT(0 == strcmp(\"master\", dest.target));\n ++\tEXPECT(REFTABLE_REF_SYMREF == dest.value_type);\n ++\tEXPECT(0 == strcmp(\"master\", dest.value.symref));\n +\n +\treftable_ref_record_release(&dest);\n +\treftable_stack_destroy(st);\n @@ reftable/stack_test.c (new)\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tint err;\n -+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n ++\tchar *dir = get_tmp_template(__FUNCTION__);\n +\tint i;\n +\tstruct reftable_ref_record ref = {\n +\t\t.refname = \"a/b\",\n +\t\t.update_index = 1,\n -+\t\t.target = \"master\",\n ++\t\t.value_type = REFTABLE_REF_SYMREF,\n ++\t\t.value.symref = \"master\",\n +\t};\n +\tchar *additions[] = { \"a\", \"a/b/c\" };\n +\n @@ reftable/stack_test.c (new)\n +\t\tstruct reftable_ref_record ref = {\n +\t\t\t.refname = additions[i],\n +\t\t\t.update_index = 1,\n -+\t\t\t.target = \"master\",\n ++\t\t\t.value_type = REFTABLE_REF_SYMREF,\n ++\t\t\t.value.symref = \"master\",\n +\t\t};\n +\n +\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n @@ reftable/stack_test.c (new)\n +\n +static void test_reftable_stack_update_index_check(void)\n +{\n -+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n ++\tchar *dir = get_tmp_template(__FUNCTION__);\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tint err;\n +\tstruct reftable_ref_record ref1 = {\n +\t\t.refname = \"name1\",\n +\t\t.update_index = 1,\n -+\t\t.target = \"master\",\n ++\t\t.value_type = REFTABLE_REF_SYMREF,\n ++\t\t.value.symref = \"master\",\n +\t};\n +\tstruct reftable_ref_record ref2 = {\n +\t\t.refname = \"name2\",\n +\t\t.update_index = 1,\n -+\t\t.target = \"master\",\n ++\t\t.value_type = REFTABLE_REF_SYMREF,\n ++\t\t.value.symref = \"master\",\n +\t};\n +\tEXPECT(mkdtemp(dir));\n +\n @@ reftable/stack_test.c (new)\n +\n +static void test_reftable_stack_lock_failure(void)\n +{\n -+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n ++\tchar *dir = get_tmp_template(__FUNCTION__);\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tint err, i;\n @@ reftable/stack_test.c (new)\n +\t\t.exact_log_message = 1,\n +\t};\n +\tstruct reftable_stack *st = NULL;\n -+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n ++\tchar *dir = get_tmp_template(__FUNCTION__);\n +\tstruct reftable_ref_record refs[2] = { { NULL } };\n +\tstruct reftable_log_record logs[2] = { { NULL } };\n +\tint N = ARRAY_SIZE(refs);\n @@ reftable/stack_test.c (new)\n +\t\tchar buf[256];\n +\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n +\t\trefs[i].refname = xstrdup(buf);\n -+\t\trefs[i].value = reftable_malloc(SHA1_SIZE);\n +\t\trefs[i].update_index = i + 1;\n -+\t\tset_test_hash(refs[i].value, i);\n ++\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n ++\t\trefs[i].value.val1 = reftable_malloc(SHA1_SIZE);\n ++\t\tset_test_hash(refs[i].value.val1, i);\n +\n +\t\tlogs[i].refname = xstrdup(buf);\n +\t\tlogs[i].update_index = N + i + 1;\n @@ reftable/stack_test.c (new)\n +\t\t0,\n +\t};\n +\tstruct reftable_stack *st = NULL;\n -+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n ++\tchar *dir = get_tmp_template(__FUNCTION__);\n +\n +\tuint8_t h1[SHA1_SIZE] = { 0x01 }, h2[SHA1_SIZE] = { 0x02 };\n +\n @@ reftable/stack_test.c (new)\n +static void test_reftable_stack_tombstone(void)\n +{\n +\tint i = 0;\n -+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n ++\tchar *dir = get_tmp_template(__FUNCTION__);\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tint err;\n @@ reftable/stack_test.c (new)\n +\t\trefs[i].refname = xstrdup(buf);\n +\t\trefs[i].update_index = i + 1;\n +\t\tif (i % 2 == 0) {\n -+\t\t\trefs[i].value = reftable_malloc(SHA1_SIZE);\n -+\t\t\tset_test_hash(refs[i].value, i);\n ++\t\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n ++\t\t\trefs[i].value.val1 = reftable_malloc(SHA1_SIZE);\n ++\t\t\tset_test_hash(refs[i].value.val1, i);\n +\t\t}\n +\t\tlogs[i].refname = xstrdup(buf);\n +\t\t/* update_index is part of the key. */\n @@ reftable/stack_test.c (new)\n +\n +static void test_reftable_stack_hash_id(void)\n +{\n -+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n ++\tchar *dir = get_tmp_template(__FUNCTION__);\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tint err;\n +\n +\tstruct reftable_ref_record ref = {\n +\t\t.refname = \"master\",\n -+\t\t.target = \"target\",\n ++\t\t.value_type = REFTABLE_REF_SYMREF,\n ++\t\t.value.symref = \"target\",\n +\t\t.update_index = 1,\n +\t};\n +\tstruct reftable_write_options cfg32 = { .hash_id = SHA256_ID };\n @@ reftable/stack_test.c (new)\n +\terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n +\tEXPECT_ERR(err);\n +\n -+\tEXPECT(!strcmp(dest.target, ref.target));\n ++\tEXPECT(reftable_ref_record_equal(&ref, &dest, SHA1_SIZE));\n +\treftable_ref_record_release(&dest);\n +\treftable_stack_destroy(st);\n +\treftable_stack_destroy(st_default);\n @@ reftable/stack_test.c (new)\n +\n +static void test_sizes_to_segments_empty(void)\n +{\n -+\tuint64_t sizes[0];\n -+\n +\tint seglen = 0;\n -+\tstruct segment *segs =\n -+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n ++\tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n +\tEXPECT(seglen == 0);\n +\treftable_free(segs);\n +}\n @@ reftable/stack_test.c (new)\n +\n +static void test_reflog_expire(void)\n +{\n -+\tchar dir[256] = \"/tmp/stack.test_reflog_expire.XXXXXX\";\n ++\tchar *dir = get_tmp_template(__FUNCTION__);\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tstruct reftable_log_record logs[20] = { { NULL } };\n @@ reftable/stack_test.c (new)\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tint err;\n -+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n ++\tchar *dir = get_tmp_template(__FUNCTION__);\n +\tstruct reftable_stack *st2 = NULL;\n +\n +\tEXPECT(mkdtemp(dir));\n @@ reftable/stack_test.c (new)\n +{\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n -+\tchar dir[256] = \"/tmp/stack_test.XXXXXX\";\n ++\tchar *dir = get_tmp_template(__FUNCTION__);\n +\tint err, i;\n +\tint N = 100;\n +\tEXPECT(mkdtemp(dir));\n @@ reftable/stack_test.c (new)\n +\t\tstruct reftable_ref_record ref = {\n +\t\t\t.refname = name,\n +\t\t\t.update_index = reftable_stack_next_update_index(st),\n -+\t\t\t.target = \"master\",\n ++\t\t\t.value_type = REFTABLE_REF_SYMREF,\n ++\t\t\t.value.symref = \"master\",\n +\t\t};\n +\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n +\n @@ reftable/stack_test.c (new)\n +\n +int stack_test_main(int argc, const char *argv[])\n +{\n -+\ttest_reftable_stack_uptodate();\n -+\ttest_reftable_stack_transaction_api();\n -+\ttest_reftable_stack_hash_id();\n -+\ttest_sizes_to_segments_all_equal();\n -+\ttest_reftable_stack_auto_compaction();\n -+\ttest_reftable_stack_validate_refname();\n -+\ttest_reftable_stack_update_index_check();\n -+\ttest_reftable_stack_lock_failure();\n -+\ttest_reftable_stack_log_normalize();\n -+\ttest_reftable_stack_tombstone();\n -+\ttest_reftable_stack_add_one();\n -+\ttest_empty_add();\n -+\ttest_reflog_expire();\n -+\ttest_suggest_compaction_segment();\n -+\ttest_suggest_compaction_segment_nothing();\n -+\ttest_sizes_to_segments();\n -+\ttest_sizes_to_segments_empty();\n -+\ttest_log2();\n -+\ttest_parse_names();\n -+\ttest_read_file();\n -+\ttest_names_equal();\n -+\ttest_reftable_stack_add();\n ++\tRUN_TEST(test_reftable_stack_uptodate);\n ++\tRUN_TEST(test_reftable_stack_transaction_api);\n ++\tRUN_TEST(test_reftable_stack_hash_id);\n ++\tRUN_TEST(test_sizes_to_segments_all_equal);\n ++\tRUN_TEST(test_reftable_stack_auto_compaction);\n ++\tRUN_TEST(test_reftable_stack_validate_refname);\n ++\tRUN_TEST(test_reftable_stack_update_index_check);\n ++\tRUN_TEST(test_reftable_stack_lock_failure);\n ++\tRUN_TEST(test_reftable_stack_log_normalize);\n ++\tRUN_TEST(test_reftable_stack_tombstone);\n ++\tRUN_TEST(test_reftable_stack_add_one);\n ++\tRUN_TEST(test_empty_add);\n ++\tRUN_TEST(test_reflog_expire);\n ++\tRUN_TEST(test_suggest_compaction_segment);\n ++\tRUN_TEST(test_suggest_compaction_segment_nothing);\n ++\tRUN_TEST(test_sizes_to_segments);\n ++\tRUN_TEST(test_sizes_to_segments_empty);\n ++\tRUN_TEST(test_log2);\n ++\tRUN_TEST(test_parse_names);\n ++\tRUN_TEST(test_read_file);\n ++\tRUN_TEST(test_names_equal);\n ++\tRUN_TEST(test_reftable_stack_add);\n +\treturn 0;\n +}\n \n 14: a590865a700 ! 13: d57023d9f13 Reftable support for git-core\n @@ Commit message\n Helped-by: Johannes Schindelin \n Co-authored-by: Jeff King \n \n + ## Documentation/config/extensions.txt ##\n +@@ Documentation/config/extensions.txt: extensions.objectFormat::\n + Note that this setting should only be set by linkgit:git-init[1] or\n + linkgit:git-clone[1]. Trying to change it after initialization will not\n + work and will produce hard-to-diagnose issues.\n +++\n ++extensions.refStorage::\n ++\tSpecify the ref storage mechanism to use. The acceptable values are `files` and\n ++\t`reftable`. If not specified, `files` is assumed. It is an error to specify\n ++\tthis key unless `core.repositoryFormatVersion` is 1.\n +++\n ++Note that this setting should only be set by linkgit:git-init[1] or\n ++linkgit:git-clone[1]. Trying to change it after initialization will not\n ++work and will produce hard-to-diagnose issues.\n +\n ## Documentation/technical/repository-version.txt ##\n @@ Documentation/technical/repository-version.txt: If set, by default \"git config\" reads from both \"config\" and\n multiple working directory mode, \"config\" file is shared while\n @@ refs/reftable-backend.c (new)\n +\t\t\tcontinue;\n +\n +\t\tri->base.flags = 0;\n -+\t\tif (ri->ref.value != NULL) {\n -+\t\t\thashcpy(ri->oid.hash, ri->ref.value);\n -+\t\t} else if (ri->ref.target != NULL) {\n ++\t\tswitch (ri->ref.value_type) {\n ++\t\tcase REFTABLE_REF_VAL1:\n ++\t\t\thashcpy(ri->oid.hash, ri->ref.value.val1);\n ++\t\t\tbreak;\n ++\t\tcase REFTABLE_REF_VAL2:\n ++\t\t\thashcpy(ri->oid.hash, ri->ref.value.val2.value);\n ++\t\t\tbreak;\n ++\t\tcase REFTABLE_REF_SYMREF: {\n +\t\t\tint out_flags = 0;\n +\t\t\tconst char *resolved = refs_resolve_ref_unsafe(\n +\t\t\t\tri->ref_store, ri->ref.refname,\n @@ refs/reftable-backend.c (new)\n +\t\t\t (ri->base.flags & REF_ISBROKEN)) {\n +\t\t\t\tcontinue;\n +\t\t\t}\n ++\t\t\tbreak;\n ++\t\t}\n ++\t\tdefault:\n ++\t\t\tabort();\n +\t\t}\n +\n +\t\tri->base.oid = &ri->oid;\n @@ refs/reftable-backend.c (new)\n +{\n +\tstruct git_reftable_iterator *ri =\n +\t\t(struct git_reftable_iterator *)ref_iterator;\n -+\tif (ri->ref.target_value != NULL) {\n -+\t\thashcpy(peeled->hash, ri->ref.target_value);\n ++\tif (ri->ref.value_type == REFTABLE_REF_VAL2) {\n ++\t\thashcpy(peeled->hash, ri->ref.value.val2.target_value);\n +\t\treturn 0;\n +\t}\n +\n @@ refs/reftable-backend.c (new)\n +\n +\t\t\tint peel_error = peel_object(&u->new_oid, &peeled);\n +\t\t\tref.refname = (char *)u->refname;\n -+\n -+\t\t\tif (!is_null_oid(&u->new_oid)) {\n -+\t\t\t\tref.value = u->new_oid.hash;\n -+\t\t\t}\n +\t\t\tref.update_index = ts;\n ++\n +\t\t\tif (!peel_error) {\n -+\t\t\t\tref.target_value = peeled.hash;\n ++\t\t\t\tref.value_type = REFTABLE_REF_VAL2;\n ++\t\t\t\tref.value.val2.target_value = peeled.hash;\n ++\t\t\t\tref.value.val2.value = u->new_oid.hash;\n ++\t\t\t} else if (!is_null_oid(&u->new_oid)) {\n ++\t\t\t\tref.value_type = REFTABLE_REF_VAL1;\n ++\t\t\t\tref.value.val1 = u->new_oid.hash;\n +\t\t\t}\n +\n +\t\t\terr = reftable_writer_add_ref(writer, &ref);\n @@ refs/reftable-backend.c (new)\n +\tfor (i = 0; i < arg->refnames->nr; i++) {\n +\t\tstruct reftable_ref_record ref = {\n +\t\t\t.refname = (char *)arg->refnames->items[i].string,\n ++\t\t\t.value_type = REFTABLE_REF_DELETION,\n +\t\t\t.update_index = ts,\n +\t\t};\n +\t\terr = reftable_writer_add_ref(writer, &ref);\n @@ refs/reftable-backend.c (new)\n +\n +\t\tif (reftable_stack_read_ref(arg->stack, log.refname,\n +\t\t\t\t\t ¤t) == 0) {\n -+\t\t\tlog.old_hash = current.value;\n ++\t\t\tlog.old_hash = reftable_ref_record_val1(¤t);\n +\t\t}\n +\t\terr = reftable_writer_add_log(writer, &log);\n +\t\tlog.old_hash = NULL;\n @@ refs/reftable-backend.c (new)\n +\n +\tstruct reftable_ref_record ref = {\n +\t\t.refname = (char *)create->refname,\n -+\t\t.target = (char *)create->target,\n ++\t\t.value_type = REFTABLE_REF_SYMREF,\n ++\t\t.value.symref = (char *)create->target,\n +\t\t.update_index = ts,\n +\t};\n +\treftable_writer_set_limits(writer, ts, ts);\n @@ refs/reftable-backend.c (new)\n +\t\tgoto done;\n +\t}\n +\n -+\tfree(ref.refname);\n -+\tref.refname = strdup(arg->newname);\n +\treftable_writer_set_limits(writer, ts, ts);\n -+\tref.update_index = ts;\n +\n +\t{\n -+\t\tstruct reftable_ref_record todo[2] = { { NULL } };\n -+\t\ttodo[0].refname = (char *)arg->oldname;\n -+\t\ttodo[0].update_index = ts;\n -+\t\t/* leave todo[0] empty */\n -+\t\ttodo[1] = ref;\n ++\t\tstruct reftable_ref_record todo[2] = {\n ++\t\t\t{\n ++\t\t\t\t.refname = (char *)arg->oldname,\n ++\t\t\t\t.update_index = ts,\n ++\t\t\t\t.value_type = REFTABLE_REF_DELETION,\n ++\t\t\t},\n ++\t\t\tref,\n ++\t\t};\n +\t\ttodo[1].update_index = ts;\n ++\t\tfree(todo[1].refname);\n ++\t\ttodo[1].refname = strdup(arg->newname);\n +\n +\t\terr = reftable_writer_add_refs(writer, todo, 2);\n +\t\tif (err < 0) {\n @@ refs/reftable-backend.c (new)\n +\t\t}\n +\t}\n +\n -+\tif (ref.value != NULL) {\n ++\tif (reftable_ref_record_val1(&ref)) {\n ++\t\tuint8_t *val1 = reftable_ref_record_val1(&ref);\n +\t\tstruct reftable_log_record todo[2] = { { NULL } };\n +\t\tfill_reftable_log_record(&todo[0]);\n +\t\tfill_reftable_log_record(&todo[1]);\n @@ refs/reftable-backend.c (new)\n +\t\ttodo[0].refname = (char *)arg->oldname;\n +\t\ttodo[0].update_index = ts;\n +\t\ttodo[0].message = (char *)arg->logmsg;\n -+\t\ttodo[0].old_hash = ref.value;\n ++\t\ttodo[0].old_hash = val1;\n +\t\ttodo[0].new_hash = NULL;\n +\n +\t\ttodo[1].refname = (char *)arg->newname;\n +\t\ttodo[1].update_index = ts;\n +\t\ttodo[1].old_hash = NULL;\n -+\t\ttodo[1].new_hash = ref.value;\n ++\t\ttodo[1].new_hash = val1;\n +\t\ttodo[1].message = (char *)arg->logmsg;\n +\n +\t\terr = reftable_writer_add_logs(writer, todo, 2);\n @@ refs/reftable-backend.c (new)\n +\t\terr = -1;\n +\t\tgoto done;\n +\t}\n -+\tif (ref.target != NULL) {\n ++\n ++\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n +\t\tstrbuf_reset(referent);\n -+\t\tstrbuf_addstr(referent, ref.target);\n ++\t\tstrbuf_addstr(referent, ref.value.symref);\n +\t\t*type |= REF_ISSYMREF;\n -+\t} else if (ref.value != NULL) {\n -+\t\thashcpy(oid->hash, ref.value);\n ++\t} else if (reftable_ref_record_val1(&ref) != NULL) {\n ++\t\thashcpy(oid->hash, reftable_ref_record_val1(&ref));\n +\t} else {\n +\t\t*type |= REF_ISBROKEN;\n +\t\terrno = EINVAL;\n 15: 57626bfe2d4 = 14: 9df5bc69f97 git-prompt: prepare for reftable refs backend\n 16: 6229da992e4 ! 15: 4076ef5d20b Add \"test-tool dump-reftable\" command.\n @@ reftable/dump.c: static int dump_table(const char *tablename)\n \n \treftable_reader_free(r);\n \treturn 0;\n -@@ reftable/dump.c: static int dump_stack(const char *stackdir)\n +@@ reftable/dump.c: static int dump_table(const char *tablename)\n + static int compact_stack(const char *stackdir)\n {\n \tstruct reftable_stack *stack = NULL;\n - \tstruct reftable_write_options cfg = {};\n +-\tstruct reftable_write_options cfg = {};\n ++\tstruct reftable_write_options cfg = { 0 };\n + \n + \tint err = reftable_new_stack(&stack, stackdir, cfg);\n + \tif (err < 0)\n +@@ reftable/dump.c: static int compact_stack(const char *stackdir)\n + static int dump_stack(const char *stackdir)\n + {\n + \tstruct reftable_stack *stack = NULL;\n +-\tstruct reftable_write_options cfg = {};\n -\tstruct reftable_iterator it = { 0 };\n -\tstruct reftable_ref_record ref = { 0 };\n -\tstruct reftable_log_record log = { 0 };\n ++\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_iterator it = { NULL };\n +\tstruct reftable_ref_record ref = { NULL };\n +\tstruct reftable_log_record log = { NULL };\n @@ reftable/dump.c: static int dump_stack(const char *stackdir)\n \n \treftable_stack_destroy(stack);\n \treturn 0;\n +@@ reftable/dump.c: static void print_help(void)\n + int reftable_dump_main(int argc, char *const *argv)\n + {\n + \tint err = 0;\n +-\tint opt;\n + \tint opt_dump_table = 0;\n + \tint opt_dump_stack = 0;\n + \tint opt_compact = 0;\n +-\tconst char *arg = NULL;\n +-\twhile ((opt = getopt(argc, argv, \"2chts\")) != -1) {\n +-\t\tswitch (opt) {\n +-\t\tcase '2':\n +-\t\t\thash_id = 0x73323536;\n ++\tconst char *arg = NULL, *argv0 = argv[0];\n ++\n ++\tfor (; argc > 1; argv++, argc--)\n ++\t\tif (*argv[1] != '-')\n + \t\t\tbreak;\n +-\t\tcase 't':\n ++\t\telse if (!strcmp(\"-2\", argv[1]))\n ++\t\t\thash_id = 0x73323536;\n ++\t\telse if (!strcmp(\"-t\", argv[1]))\n + \t\t\topt_dump_table = 1;\n +-\t\t\tbreak;\n +-\t\tcase 's':\n ++\t\telse if (!strcmp(\"-s\", argv[1]))\n + \t\t\topt_dump_stack = 1;\n +-\t\t\tbreak;\n +-\t\tcase 'c':\n ++\t\telse if (!strcmp(\"-c\", argv[1]))\n + \t\t\topt_compact = 1;\n +-\t\t\tbreak;\n +-\t\tcase '?':\n +-\t\tcase 'h':\n ++\t\telse if (!strcmp(\"-?\", argv[1]) || !strcmp(\"-h\", argv[1])) {\n + \t\t\tprint_help();\n + \t\t\treturn 2;\n +-\t\t\tbreak;\n + \t\t}\n +-\t}\n + \n +-\tif (argv[optind] == NULL) {\n ++\tif (argc != 2) {\n + \t\tfprintf(stderr, \"need argument\\n\");\n + \t\tprint_help();\n + \t\treturn 2;\n + \t}\n + \n +-\targ = argv[optind];\n ++\targ = argv[1];\n + \n + \tif (opt_dump_table) {\n + \t\terr = dump_table(arg);\n +@@ reftable/dump.c: int reftable_dump_main(int argc, char *const *argv)\n + \t}\n + \n + \tif (err < 0) {\n +-\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv[0], arg,\n ++\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv0, arg,\n + \t\t\treftable_error_str(err));\n + \t\treturn 1;\n + \t}\n \n ## t/helper/test-reftable.c ##\n @@ t/helper/test-reftable.c: int cmd__reftable(int argc, const char **argv)\n\n-- \ngitgitgadget\n"},{"id":"411851","messageId":"cc0fe58bd3337d63377949db9dd4c754db660774.1607522429.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v4.git.git.1607522429.gitgitgadget@gmail.com","subject":"[PATCH v4 02/15] reftable: add LICENSE","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-12-09T14:00:16Z","receivedAt":"2020-12-09T14:02:07Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nTODO: relicense?\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/LICENSE | 31 +++++++++++++++++++++++++++++++\n 1 file changed, 31 insertions(+)\n create mode 100644 reftable/LICENSE\n\ndiff --git a/reftable/LICENSE b/reftable/LICENSE\nnew file mode 100644\nindex 00000000000..402e0f9356b\n--- /dev/null\n+++ b/reftable/LICENSE\n@@ -0,0 +1,31 @@\n+BSD License\n+\n+Copyright (c) 2020, Google LLC\n+All rights reserved.\n+\n+Redistribution and use in source and binary forms, with or without\n+modification, are permitted provided that the following conditions are\n+met:\n+\n+* Redistributions of source code must retain the above copyright notice,\n+this list of conditions and the following disclaimer.\n+\n+* Redistributions in binary form must reproduce the above copyright\n+notice, this list of conditions and the following disclaimer in the\n+documentation and/or other materials provided with the distribution.\n+\n+* Neither the name of Google LLC nor the names of its contributors may\n+be used to endorse or promote products derived from this software\n+without specific prior written permission.\n+\n+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n+\"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n-- \ngitgitgadget\n\n"},{"id":"411853","messageId":"40ac041d0efef7a7baec56354265176235137444.1607522429.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v4.git.git.1607522429.gitgitgadget@gmail.com","subject":"[PATCH v4 01/15] init-db: set the_repository->hash_algo early on","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-12-09T14:00:15Z","receivedAt":"2020-12-09T14:02:07Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable backend needs to know the hash algorithm for writing the\ninitialization hash table.\n\nThe initial reftable contains a symref HEAD => \"main\" (or \"master\"), which is\nagnostic to the size of hash value, but this is an exceptional circumstance, and\nthe reftable library does not cater to this exception. It insists that all\ntables in the stack have a consistent format ID for the hash algorithm.\n\nCall set_repo_hash_algo directly after calling validate_hash_algorithm() (which\nreads $GIT_DEFAULT_HASH).\n\nHelped-by: Junio C Hamano \nSigned-off-by: Han-Wen Nienhuys \n---\n builtin/init-db.c | 21 +++++++++++++++++++++\n 1 file changed, 21 insertions(+)\n\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 01bc648d416..dcb7015db48 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -437,6 +437,27 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \n \tvalidate_hash_algorithm(&repo_fmt, hash);\n \n+\t/*\n+\t * At this point, the_repository we have in-core does not look\n+\t * anything like one that we would see initialized in an already\n+\t * working repository after calling setup_git_directory().\n+\t *\n+\t * Calling repository.c::initialize_the_repository() may have\n+\t * prepared the .index .objects and .parsed_objects members, but\n+\t * other members like .gitdir, .commondir, etc. have not been\n+\t * initialized.\n+\t *\n+\t * Many API functions assume they are working with the_repository\n+\t * that has sensibly been initialized, but because we haven't\n+\t * really read from an existing repository, we need to hand-craft\n+\t * the necessary members of the structure to get out of this\n+\t * chicken-and-egg situation.\n+\t *\n+\t * For now, we update the hash algorithm member to what the\n+\t * validate_hash_algorithm() call decided for us.\n+\t */\n+\trepo_set_hash_algo(the_repository, repo_fmt.hash_algo);\n+\n \treinit = create_default_files(template_dir, original_git_dir,\n \t\t\t\t initial_branch, &repo_fmt);\n \tif (reinit && initial_branch)\n-- \ngitgitgadget\n\n"},{"id":"411854","messageId":"9df5bc69f971dc2b51d519a2db9a2b0f22f9d87b.1607522430.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v4.git.git.1607522429.gitgitgadget@gmail.com","subject":"[PATCH v4 14/15] git-prompt: prepare for reftable refs backend","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-12-09T14:00:28Z","receivedAt":"2020-12-09T14:02:38Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nIn our git-prompt script we strive to use Bash builtins wherever\npossible, because fork()-ing subshells for command substitutions and\nfork()+exec()-ing Git commands are expensive on some platforms. We\neven read and parse '.git/HEAD' using Bash builtins to get the name of\nthe current branch [1]. However, the upcoming reftable refs backend\nwon't use '.git/HEAD' at all, but will write an invalid refname as\nplaceholder for backwards compatibility instead, which will break our\ngit-prompt script.\n\nUpdate the git-prompt script to recognize the placeholder '.git/HEAD'\nwritten by the reftable backend (its content is specified in the\nreftable specs), and then fall back to use 'git symbolic-ref' to get\nthe name of the current branch.\n\n[1] 3a43c4b5bd (bash prompt: use bash builtins to find out current\n branch, 2011-03-31)\n\nSigned-off-by: SZEDER Gábor \n---\n contrib/completion/git-prompt.sh | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/completion/git-prompt.sh b/contrib/completion/git-prompt.sh\nindex 4640a1535d1..2e5a5d80271 100644\n--- a/contrib/completion/git-prompt.sh\n+++ b/contrib/completion/git-prompt.sh\n@@ -478,10 +478,15 @@ __git_ps1 ()\n \t\t\tif ! __git_eread \"$g/HEAD\" head; then\n \t\t\t\treturn $exit\n \t\t\tfi\n-\t\t\t# is it a symbolic ref?\n \t\t\tb=\"${head#ref: }\"\n \t\t\tif [ \"$head\" = \"$b\" ]; then\n \t\t\t\tdetached=yes\n+\t\t\telif [ \"$b\" = \"refs/heads/.invalid\" ]; then\n+\t\t\t\t# Reftable\n+\t\t\t\tb=\"$(git symbolic-ref HEAD 2>/dev/null)\" ||\n+\t\t\t\tdetached=yes\n+\t\t\tfi\n+\t\t\tif [ \"$detached\" = yes ]; then\n \t\t\t\tb=\"$(\n \t\t\t\tcase \"${GIT_PS1_DESCRIBE_STYLE-}\" in\n \t\t\t\t(contains)\n-- \ngitgitgadget\n\n"},{"id":"411855","messageId":"03681b820d132a270a08a52040c1ad55c12b8f4b.1607522429.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v4.git.git.1607522429.gitgitgadget@gmail.com","subject":"[PATCH v4 11/15] reftable: reftable file level tests","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-12-09T14:00:25Z","receivedAt":"2020-12-09T14:02:38Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nWith support for reading and writing files in place, we can construct files (in\nmemory) and attempt to read them back.\n\nBecause some sections of the format are optional (eg. indices, log entries), we\nhave to exercise this code using multiple sizes of input data\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable_test.c | 580 +++++++++++++++++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 3 files changed, 582 insertions(+)\n create mode 100644 reftable/reftable_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 2bafcb1b62a..c913e90b643 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2405,6 +2405,7 @@ REFTABLE_OBJS += reftable/zlib-compat.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n+REFTABLE_TEST_OBJS += reftable/reftable_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/reftable_test.c b/reftable/reftable_test.c\nnew file mode 100644\nindex 00000000000..c58f7061833\n--- /dev/null\n+++ b/reftable/reftable_test.c\n@@ -0,0 +1,580 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-stack.h\"\n+\n+static const int update_index = 5;\n+\n+static void test_buffer(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_block out = { NULL };\n+\tint n;\n+\tuint8_t in[] = \"hello\";\n+\tstrbuf_add(&buf, in, sizeof(in));\n+\tblock_source_from_strbuf(&source, &buf);\n+\tEXPECT(block_source_size(&source) == 6);\n+\tn = block_source_read_block(&source, &out, 0, sizeof(in));\n+\tEXPECT(n == sizeof(in));\n+\tEXPECT(!memcmp(in, out.data, n));\n+\treftable_block_done(&out);\n+\n+\tn = block_source_read_block(&source, &out, 1, 2);\n+\tEXPECT(n == 2);\n+\tEXPECT(!memcmp(out.data, \"el\", 2));\n+\n+\treftable_block_done(&out);\n+\tblock_source_close(&source);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void write_table(char ***names, struct strbuf *buf, int N,\n+\t\t\tint block_size, uint32_t hash_id)\n+{\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = block_size,\n+\t\t.hash_id = hash_id,\n+\t};\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, buf, &opts);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0, n;\n+\tstruct reftable_log_record log = { NULL };\n+\tconst struct reftable_stats *stats = NULL;\n+\t*names = reftable_calloc(sizeof(char *) * (N + 1));\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[SHA256_SIZE] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tref.refname = name;\n+\t\tref.update_index = update_index;\n+\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\tref.value.val1 = hash;\n+\t\t(*names)[i] = xstrdup(name);\n+\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[SHA256_SIZE] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tlog.refname = name;\n+\t\tlog.new_hash = hash;\n+\t\tlog.update_index = update_index;\n+\t\tlog.message = \"message\";\n+\n+\t\tn = reftable_writer_add_log(w, &log);\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tfor (i = 0; i < stats->ref_stats.blocks; i++) {\n+\t\tint off = i * opts.block_size;\n+\t\tif (off == 0) {\n+\t\t\toff = header_size((hash_id == SHA256_ID) ? 2 : 1);\n+\t\t}\n+\t\tEXPECT(buf->buf[off] == 'r');\n+\t}\n+\n+\tEXPECT(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+}\n+\n+static void test_log_buffer_size(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 4096,\n+\t};\n+\tint err;\n+\tstruct reftable_log_record log = {\n+\t\t.refname = \"refs/heads/master\",\n+\t\t.name = \"Han-Wen Nienhuys\",\n+\t\t.email = \"hanwen@google.com\",\n+\t\t.tz_offset = 100,\n+\t\t.time = 0x5e430672,\n+\t\t.update_index = 0xa,\n+\t\t.message = \"commit: 9\\n\",\n+\t};\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\t/* This tests buffer extension for log compression. Must use a random\n+\t hash, to ensure that the compressed part is larger than the original.\n+\t*/\n+\tuint8_t hash1[SHA1_SIZE], hash2[SHA1_SIZE];\n+\tfor (int i = 0; i < SHA1_SIZE; i++) {\n+\t\thash1[i] = (uint8_t)(rand() % 256);\n+\t\thash2[i] = (uint8_t)(rand() % 256);\n+\t}\n+\tlog.old_hash = hash1;\n+\tlog.new_hash = hash2;\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\terr = reftable_writer_add_log(w, &log);\n+\tEXPECT_ERR(err);\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_log_write_read(void)\n+{\n+\tint N = 2;\n+\tchar **names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint err;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\tint n;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tconst struct reftable_stats *stats = NULL;\n+\treftable_writer_set_limits(w, 0, N);\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[256];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tsnprintf(name, sizeof(name), \"b%02d%0*d\", i, 130, 7);\n+\t\tnames[i] = xstrdup(name);\n+\t\tref.refname = name;\n+\t\tref.update_index = i;\n+\n+\t\terr = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash1[SHA1_SIZE], hash2[SHA1_SIZE];\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tset_test_hash(hash1, i);\n+\t\tset_test_hash(hash2, i + 1);\n+\n+\t\tlog.refname = names[i];\n+\t\tlog.update_index = i;\n+\t\tlog.old_hash = hash1;\n+\t\tlog.new_hash = hash2;\n+\n+\t\terr = reftable_writer_add_log(w, &log);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tEXPECT(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[N - 1]);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT_ERR(err);\n+\n+\t/* end of iteration. */\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT(0 < err);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_release(&ref);\n+\n+\terr = reftable_reader_seek_log(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\ti = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT_STREQ(names[i], log.refname);\n+\t\tEXPECT(i == log.update_index);\n+\t\ti++;\n+\t\treftable_log_record_release(&log);\n+\t}\n+\n+\tEXPECT(i == N);\n+\treftable_iterator_destroy(&it);\n+\n+\t/* cleanup. */\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_sequential(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tint err = 0;\n+\tint j = 0;\n+\n+\twrite_table(&names, &buf, N, 256, SHA1_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint r = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tEXPECT(0 == strcmp(names[j], ref.refname));\n+\t\tEXPECT(update_index == ref.update_index);\n+\n+\t\tj++;\n+\t\treftable_ref_record_release(&ref);\n+\t}\n+\tEXPECT(j == N);\n+\treftable_iterator_destroy(&it);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\n+\treader_close(&rd);\n+}\n+\n+static void test_table_write_small_table(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 1;\n+\twrite_table(&names, &buf, N, 4096, SHA1_ID);\n+\tEXPECT(buf.len < 200);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+}\n+\n+static void test_table_read_api(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i;\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, SHA1_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[0]);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_free(names);\n+\treader_close(&rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_table_read_write_seek(int index, int hash_id)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i = 0;\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct strbuf pastLast = STRBUF_INIT;\n+\tstruct reftable_ref_record ref = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, hash_id);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\tEXPECT(hash_id == reftable_reader_hash_id(&rd));\n+\n+\tif (!index) {\n+\t\trd.ref_offsets.index_offset = 0;\n+\t} else {\n+\t\tEXPECT(rd.ref_offsets.index_offset > 0);\n+\t}\n+\n+\tfor (i = 1; i < N; i++) {\n+\t\tint err = reftable_reader_seek_ref(&rd, &it, names[i]);\n+\t\tEXPECT_ERR(err);\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(0 == strcmp(names[i], ref.refname));\n+\t\tEXPECT(REFTABLE_REF_VAL1 == ref.value_type);\n+\t\tEXPECT(i == ref.value.val1[0]);\n+\n+\t\treftable_ref_record_release(&ref);\n+\t\treftable_iterator_destroy(&it);\n+\t}\n+\n+\tstrbuf_addstr(&pastLast, names[N - 1]);\n+\tstrbuf_addstr(&pastLast, \"/\");\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, pastLast.buf);\n+\tif (err == 0) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(err > 0);\n+\t} else {\n+\t\tEXPECT(err > 0);\n+\t}\n+\n+\tstrbuf_release(&pastLast);\n+\treftable_iterator_destroy(&it);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_free(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_seek_linear(void)\n+{\n+\ttest_table_read_write_seek(0, SHA1_ID);\n+}\n+\n+static void test_table_read_write_seek_linear_sha256(void)\n+{\n+\ttest_table_read_write_seek(0, SHA256_ID);\n+}\n+\n+static void test_table_read_write_seek_index(void)\n+{\n+\ttest_table_read_write_seek(1, SHA1_ID);\n+}\n+\n+static void test_table_refs_for(int indexed)\n+{\n+\tint N = 50;\n+\tchar **want_names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint want_names_len = 0;\n+\tuint8_t want_hash[SHA1_SIZE];\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tint err;\n+\tstruct reftable_reader rd;\n+\tstruct reftable_block_source source = { NULL };\n+\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint j;\n+\n+\tset_test_hash(want_hash, 4);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[SHA1_SIZE];\n+\t\tchar fill[51] = { 0 };\n+\t\tchar name[100];\n+\t\tuint8_t hash1[SHA1_SIZE];\n+\t\tuint8_t hash2[SHA1_SIZE];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\n+\t\tmemset(hash, i, sizeof(hash));\n+\t\tmemset(fill, 'x', 50);\n+\t\t/* Put the variable part in the start */\n+\t\tsnprintf(name, sizeof(name), \"br%02d%s\", i, fill);\n+\t\tname[40] = 0;\n+\t\tref.refname = name;\n+\n+\t\tset_test_hash(hash1, i / 4);\n+\t\tset_test_hash(hash2, 3 + i / 4);\n+\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\tref.value.val2.value = hash1;\n+\t\tref.value.val2.target_value = hash2;\n+\n+\t\t/* 80 bytes / entry, so 3 entries per block. Yields 17\n+\t\t */\n+\t\t/* blocks. */\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT(n == 0);\n+\n+\t\tif (!memcmp(hash1, want_hash, SHA1_SIZE) ||\n+\t\t !memcmp(hash2, want_hash, SHA1_SIZE)) {\n+\t\t\twant_names[want_names_len++] = xstrdup(name);\n+\t\t}\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\tif (!indexed) {\n+\t\trd.obj_offsets.is_present = 0;\n+\t}\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_reader_refs_for(&rd, &it, want_hash);\n+\tEXPECT_ERR(err);\n+\n+\tj = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(err >= 0);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tEXPECT(j < want_names_len);\n+\t\tEXPECT(0 == strcmp(ref.refname, want_names[j]));\n+\t\tj++;\n+\t\treftable_ref_record_release(&ref);\n+\t}\n+\tEXPECT(j == want_names_len);\n+\n+\tstrbuf_release(&buf);\n+\tfree_names(want_names);\n+\treftable_iterator_destroy(&it);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_refs_for_no_index(void)\n+{\n+\ttest_table_refs_for(0);\n+}\n+\n+static void test_table_refs_for_obj_index(void)\n+{\n+\ttest_table_refs_for(1);\n+}\n+\n+static void test_table_empty(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_ref_record rec = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT(err == REFTABLE_EMPTY_TABLE_ERROR);\n+\treftable_writer_free(w);\n+\n+\tEXPECT(buf.len == header_size(1) + footer_size(1));\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tEXPECT(err > 0);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+int reftable_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_log_write_read);\n+\tRUN_TEST(test_table_read_write_seek_linear_sha256);\n+\tRUN_TEST(test_log_buffer_size);\n+\tRUN_TEST(test_table_write_small_table);\n+\tRUN_TEST(test_buffer);\n+\tRUN_TEST(test_table_read_api);\n+\tRUN_TEST(test_table_read_write_sequential);\n+\tRUN_TEST(test_table_read_write_seek_linear);\n+\tRUN_TEST(test_table_read_write_seek_index);\n+\tRUN_TEST(test_table_refs_for_no_index);\n+\tRUN_TEST(test_table_refs_for_obj_index);\n+\tRUN_TEST(test_table_empty);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 050551fa698..fdf92586737 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,6 +6,7 @@ int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\treftable_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"411856","messageId":"d57023d9f13d178dc95d7a74751923b80d1a5939.1607522429.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v4.git.git.1607522429.gitgitgadget@gmail.com","subject":"[PATCH v4 13/15] Reftable support for git-core","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-12-09T14:00:27Z","receivedAt":"2020-12-09T14:02:38Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nFor background, see the previous commit introducing the library.\n\nThis introduces the file refs/reftable-backend.c containing a reftable-powered\nref storage backend.\n\nIt can be activated by passing --ref-storage=reftable to \"git init\", or setting\nGIT_TEST_REFTABLE in the environment.\n\nExample use: see t/t0031-reftable.sh\n\nSigned-off-by: Han-Wen Nienhuys \nSigned-off-by: Johannes Schindelin \nHelped-by: Johannes Schindelin \nCo-authored-by: Jeff King \n---\n Documentation/config/extensions.txt | 9 +\n .../technical/repository-version.txt | 7 +\n Makefile | 4 +\n builtin/clone.c | 5 +-\n builtin/init-db.c | 55 +-\n builtin/worktree.c | 27 +-\n cache.h | 8 +-\n config.mak.uname | 2 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n refs.c | 27 +-\n refs.h | 3 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1435 +++++++++++++++++\n repository.c | 2 +\n repository.h | 3 +\n setup.c | 9 +-\n t/t0031-reftable.sh | 199 +++\n t/t1409-avoid-packing-refs.sh | 6 +\n t/t1450-fsck.sh | 6 +\n t/t3210-pack-refs.sh | 6 +\n t/test-lib.sh | 5 +\n 21 files changed, 1797 insertions(+), 33 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100755 t/t0031-reftable.sh\n\ndiff --git a/Documentation/config/extensions.txt b/Documentation/config/extensions.txt\nindex 4e23d73cdca..82c5940f143 100644\n--- a/Documentation/config/extensions.txt\n+++ b/Documentation/config/extensions.txt\n@@ -6,3 +6,12 @@ extensions.objectFormat::\n Note that this setting should only be set by linkgit:git-init[1] or\n linkgit:git-clone[1]. Trying to change it after initialization will not\n work and will produce hard-to-diagnose issues.\n++\n+extensions.refStorage::\n+\tSpecify the ref storage mechanism to use. The acceptable values are `files` and\n+\t`reftable`. If not specified, `files` is assumed. It is an error to specify\n+\tthis key unless `core.repositoryFormatVersion` is 1.\n++\n+Note that this setting should only be set by linkgit:git-init[1] or\n+linkgit:git-clone[1]. Trying to change it after initialization will not\n+work and will produce hard-to-diagnose issues.\ndiff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\nindex 7844ef30ffd..72576235833 100644\n--- a/Documentation/technical/repository-version.txt\n+++ b/Documentation/technical/repository-version.txt\n@@ -100,3 +100,10 @@ If set, by default \"git config\" reads from both \"config\" and\n multiple working directory mode, \"config\" file is shared while\n \"config.worktree\" is per-working directory (i.e., it's in\n GIT_COMMON_DIR/worktrees//config.worktree)\n+\n+==== `refStorage`\n+\n+Specifies the file format for the ref database. Values are `files`\n+(for the traditional packed + loose ref format) and `reftable` for the\n+binary reftable format. See https://github.com/google/reftable for\n+more information.\ndiff --git a/Makefile b/Makefile\nindex 18cc18c2153..6b7c8a165f4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -977,6 +977,7 @@ LIB_OBJS += reflog-walk.o\n LIB_OBJS += refs.o\n LIB_OBJS += refs/debug.o\n LIB_OBJS += refs/files-backend.o\n+LIB_OBJS += refs/reftable-backend.o\n LIB_OBJS += refs/iterator.o\n LIB_OBJS += refs/packed-backend.o\n LIB_OBJS += refs/ref-cache.o\n@@ -2408,8 +2409,11 @@ REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n+REFTABLE_TEST_OBJS += reftable/refname_test.o\n REFTABLE_TEST_OBJS += reftable/reftable_test.o\n+REFTABLE_TEST_OBJS += reftable/stack_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex a0841923cfe..974a374e9c9 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1138,7 +1138,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t}\n \n \tinit_db(git_dir, real_git_dir, option_template, GIT_HASH_UNKNOWN, NULL,\n-\t\tINIT_DB_QUIET);\n+\t\tdefault_ref_storage(), INIT_DB_QUIET);\n \n \tif (real_git_dir)\n \t\tgit_dir = real_git_dir;\n@@ -1273,7 +1273,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t * Now that we know what algorithm the remote side is using,\n \t\t * let's set ours to the same thing.\n \t\t */\n-\t\tinitialize_repository_version(hash_algo, 1);\n+\t\tinitialize_repository_version(hash_algo, 1,\n+\t\t\t\t\t default_ref_storage());\n \t\trepo_set_hash_algo(the_repository, hash_algo);\n \n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex dcb7015db48..e10dc712b8d 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -179,12 +179,14 @@ static int needs_work_tree_config(const char *git_dir, const char *work_tree)\n \treturn 1;\n }\n \n-void initialize_repository_version(int hash_algo, int reinit)\n+void initialize_repository_version(int hash_algo, int reinit,\n+\t\t\t\t const char *ref_storage_format)\n {\n \tchar repo_version_string[10];\n \tint repo_version = GIT_REPO_VERSION;\n \n-\tif (hash_algo != GIT_HASH_SHA1)\n+\tif (hash_algo != GIT_HASH_SHA1 ||\n+\t !strcmp(ref_storage_format, \"reftable\"))\n \t\trepo_version = GIT_REPO_VERSION_READ;\n \n \t/* This forces creation of new config file */\n@@ -237,6 +239,7 @@ static int create_default_files(const char *template_path,\n \tis_bare_repository_cfg = init_is_bare_repository;\n \tif (init_shared_repository != -1)\n \t\tset_shared_repository(init_shared_repository);\n+\tthe_repository->ref_storage_format = xstrdup(fmt->ref_storage);\n \n \t/*\n \t * We would have created the above under user's umask -- under\n@@ -246,6 +249,24 @@ static int create_default_files(const char *template_path,\n \t\tadjust_shared_perm(get_git_dir());\n \t}\n \n+\t/*\n+\t * Check to see if .git/HEAD exists; this must happen before\n+\t * initializing the ref db, because we want to see if there is an\n+\t * existing HEAD.\n+\t */\n+\tpath = git_path_buf(&buf, \"HEAD\");\n+\treinit = (!access(path, R_OK) ||\n+\t\t readlink(path, junk, sizeof(junk) - 1) != -1);\n+\n+\t/*\n+\t * refs/heads is a file when using reftable. We can't reinitialize with\n+\t * a reftable because it will overwrite HEAD\n+\t */\n+\tif (reinit && (!strcmp(fmt->ref_storage, \"reftable\")) ==\n+\t\t\t is_directory(git_path_buf(&buf, \"refs/heads\"))) {\n+\t\tdie(\"cannot switch ref storage format.\");\n+\t}\n+\n \t/*\n \t * We need to create a \"refs\" dir in any case so that older\n \t * versions of git can tell that this is a repository.\n@@ -260,9 +281,6 @@ static int create_default_files(const char *template_path,\n \t * Point the HEAD symref to the initial branch with if HEAD does\n \t * not yet exist.\n \t */\n-\tpath = git_path_buf(&buf, \"HEAD\");\n-\treinit = (!access(path, R_OK)\n-\t\t || readlink(path, junk, sizeof(junk)-1) != -1);\n \tif (!reinit) {\n \t\tchar *ref;\n \n@@ -279,7 +297,7 @@ static int create_default_files(const char *template_path,\n \t\tfree(ref);\n \t}\n \n-\tinitialize_repository_version(fmt->hash_algo, 0);\n+\tinitialize_repository_version(fmt->hash_algo, 0, fmt->ref_storage);\n \n \t/* Check filemode trustability */\n \tpath = git_path_buf(&buf, \"config\");\n@@ -395,7 +413,7 @@ static void validate_hash_algorithm(struct repository_format *repo_fmt, int hash\n \n int init_db(const char *git_dir, const char *real_git_dir,\n \t const char *template_dir, int hash, const char *initial_branch,\n-\t unsigned int flags)\n+\t const char *ref_storage_format, unsigned int flags)\n {\n \tint reinit;\n \tint exist_ok = flags & INIT_DB_EXIST_OK;\n@@ -434,6 +452,7 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \t * is an attempt to reinitialize new repository with an old tool.\n \t */\n \tcheck_repository_format(&repo_fmt);\n+\trepo_fmt.ref_storage = xstrdup(ref_storage_format);\n \n \tvalidate_hash_algorithm(&repo_fmt, hash);\n \n@@ -487,6 +506,9 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \t\tgit_config_set(\"receive.denyNonFastforwards\", \"true\");\n \t}\n \n+\tif (!strcmp(ref_storage_format, \"reftable\"))\n+\t\tgit_config_set(\"extensions.refStorage\", ref_storage_format);\n+\n \tif (!(flags & INIT_DB_QUIET)) {\n \t\tint len = strlen(git_dir);\n \n@@ -560,6 +582,7 @@ static const char *const init_db_usage[] = {\n int cmd_init_db(int argc, const char **argv, const char *prefix)\n {\n \tconst char *git_dir;\n+\tconst char *ref_storage_format = default_ref_storage();\n \tconst char *real_git_dir = NULL;\n \tconst char *work_tree;\n \tconst char *template_dir = NULL;\n@@ -568,15 +591,18 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n \tconst char *initial_branch = NULL;\n \tint hash_algo = GIT_HASH_UNKNOWN;\n \tconst struct option init_db_options[] = {\n-\t\tOPT_STRING(0, \"template\", &template_dir, N_(\"template-directory\"),\n-\t\t\t\tN_(\"directory from which templates will be used\")),\n+\t\tOPT_STRING(0, \"template\", &template_dir,\n+\t\t\t N_(\"template-directory\"),\n+\t\t\t N_(\"directory from which templates will be used\")),\n \t\tOPT_SET_INT(0, \"bare\", &is_bare_repository_cfg,\n-\t\t\t\tN_(\"create a bare repository\"), 1),\n+\t\t\t N_(\"create a bare repository\"), 1),\n \t\t{ OPTION_CALLBACK, 0, \"shared\", &init_shared_repository,\n-\t\t\tN_(\"permissions\"),\n-\t\t\tN_(\"specify that the git repository is to be shared amongst several users\"),\n-\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0},\n+\t\t N_(\"permissions\"),\n+\t\t N_(\"specify that the git repository is to be shared amongst several users\"),\n+\t\t PARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0 },\n \t\tOPT_BIT('q', \"quiet\", &flags, N_(\"be quiet\"), INIT_DB_QUIET),\n+\t\tOPT_STRING(0, \"ref-storage\", &ref_storage_format, N_(\"backend\"),\n+\t\t\t N_(\"the ref storage format to use\")),\n \t\tOPT_STRING(0, \"separate-git-dir\", &real_git_dir, N_(\"gitdir\"),\n \t\t\t N_(\"separate git dir from working tree\")),\n \t\tOPT_STRING('b', \"initial-branch\", &initial_branch, N_(\"name\"),\n@@ -717,10 +743,11 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n \t}\n \n \tUNLEAK(real_git_dir);\n+\tUNLEAK(ref_storage_format);\n \tUNLEAK(git_dir);\n \tUNLEAK(work_tree);\n \n \tflags |= INIT_DB_EXIST_OK;\n \treturn init_db(git_dir, real_git_dir, template_dir, hash_algo,\n-\t\t initial_branch, flags);\n+\t\t initial_branch, ref_storage_format, flags);\n }\ndiff --git a/builtin/worktree.c b/builtin/worktree.c\nindex 197fd24a555..a102aedfca7 100644\n--- a/builtin/worktree.c\n+++ b/builtin/worktree.c\n@@ -12,6 +12,7 @@\n #include \"submodule.h\"\n #include \"utf8.h\"\n #include \"worktree.h\"\n+#include \"../refs/refs-internal.h\"\n \n static const char * const worktree_usage[] = {\n \tN_(\"git worktree add [] []\"),\n@@ -402,9 +403,29 @@ static int add_worktree(const char *path, const char *refname,\n \t * worktree.\n \t */\n \tstrbuf_reset(&sb);\n-\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n-\twrite_file(sb.buf, \"%s\", oid_to_hex(&null_oid));\n-\tstrbuf_reset(&sb);\n+\tif (get_main_ref_store(the_repository)->be == &refs_be_reftable) {\n+\t\t/* XXX this is cut & paste from reftable_init_db. */\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"%s\", \"ref: refs/heads/.invalid\\n\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs/heads\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\t} else {\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"%s\", oid_to_hex(&null_oid));\n+\t\tstrbuf_reset(&sb);\n+\t}\n+\n \tstrbuf_addf(&sb, \"%s/commondir\", sb_repo.buf);\n \twrite_file(sb.buf, \"../..\");\n \ndiff --git a/cache.h b/cache.h\nindex e986cf4ea9c..545d2b72607 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -627,9 +627,10 @@ int path_inside_repo(const char *prefix, const char *path);\n #define INIT_DB_EXIST_OK 0x0002\n \n int init_db(const char *git_dir, const char *real_git_dir,\n-\t const char *template_dir, int hash_algo,\n-\t const char *initial_branch, unsigned int flags);\n-void initialize_repository_version(int hash_algo, int reinit);\n+\t const char *template_dir, int hash_algo, const char *initial_branch,\n+\t const char *ref_storage_format, unsigned int flags);\n+void initialize_repository_version(int hash_algo, int reinit,\n+\t\t\t\t const char *ref_storage_format);\n \n void sanitize_stdfds(void);\n int daemonize(void);\n@@ -1043,6 +1044,7 @@ struct repository_format {\n \tint is_bare;\n \tint hash_algo;\n \tchar *work_tree;\n+\tchar *ref_storage;\n \tstruct string_list unknown_extensions;\n \tstruct string_list v1_only_extensions;\n };\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 5b30a9154ac..4ab0191f5e6 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -701,7 +701,7 @@ vcxproj:\n \t# Make .vcxproj files and add them\n \tunset QUIET_GEN QUIET_BUILT_IN; \\\n \tperl contrib/buildsystems/generate -g Vcxproj\n-\tgit add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj\n+\tgit add -f git.sln {*,*/lib,*/libreftable,t/helper/*}/*.vcxproj\n \n \t# Generate the LinkOrCopyBuiltins.targets and LinkOrCopyRemoteHttp.targets file\n \t(echo '' && \\\ndiff --git a/contrib/buildsystems/Generators/Vcxproj.pm b/contrib/buildsystems/Generators/Vcxproj.pm\nindex d2584450ba1..1a25789d285 100644\n--- a/contrib/buildsystems/Generators/Vcxproj.pm\n+++ b/contrib/buildsystems/Generators/Vcxproj.pm\n@@ -77,7 +77,7 @@ sub createProject {\n my $libs_release = \"\\n \";\n my $libs_debug = \"\\n \";\n if (!$static_library) {\n- $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n+ $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib|reftable\\/libreftable\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n $libs_debug = $libs_release;\n $libs_debug =~ s/zlib\\.lib/zlibd\\.lib/g;\n $libs_debug =~ s/libexpat\\.lib/libexpatd\\.lib/g;\n@@ -232,6 +232,7 @@ sub createProject {\n EOM\n if (!$static_library || $target =~ 'vcs-svn' || $target =~ 'xdiff') {\n my $uuid_libgit = $$build_structure{\"LIBS_libgit_GUID\"};\n+ my $uuid_libreftable = $$build_structure{\"LIBS_reftable/libreftable_GUID\"};\n my $uuid_xdiff_lib = $$build_structure{\"LIBS_xdiff/lib_GUID\"};\n \n print F << \"EOM\";\n@@ -241,6 +242,14 @@ sub createProject {\n false\n \n EOM\n+ if (!($name =~ /xdiff|libreftable/)) {\n+ print F << \"EOM\";\n+ \n+ $uuid_libreftable\n+ false\n+ \n+EOM\n+ }\n if (!($name =~ 'xdiff')) {\n print F << \"EOM\";\n \ndiff --git a/refs.c b/refs.c\nindex 392f0bbf68b..1b874db3345 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -19,10 +19,16 @@\n #include \"repository.h\"\n #include \"sigchain.h\"\n \n+const char *default_ref_storage(void)\n+{\n+\tconst char *test = getenv(\"GIT_TEST_REFTABLE\");\n+\treturn test ? \"reftable\" : \"files\";\n+}\n+\n /*\n * List of all available backends\n */\n-static struct ref_storage_be *refs_backends = &refs_be_files;\n+static struct ref_storage_be *refs_backends = &refs_be_reftable;\n \n static struct ref_storage_be *find_ref_storage_backend(const char *name)\n {\n@@ -1754,13 +1760,13 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,\n * Create, record, and return a ref_store instance for the specified\n * gitdir.\n */\n-static struct ref_store *ref_store_init(const char *gitdir,\n+static struct ref_store *ref_store_init(const char *gitdir, const char *be_name,\n \t\t\t\t\tunsigned int flags)\n {\n-\tconst char *be_name = \"files\";\n-\tstruct ref_storage_be *be = find_ref_storage_backend(be_name);\n+\tstruct ref_storage_be *be;\n \tstruct ref_store *refs;\n \n+\tbe = find_ref_storage_backend(be_name);\n \tif (!be)\n \t\tBUG(\"reference backend %s is unknown\", be_name);\n \n@@ -1776,7 +1782,11 @@ struct ref_store *get_main_ref_store(struct repository *r)\n \tif (!r->gitdir)\n \t\tBUG(\"attempting to get main_ref_store outside of repository\");\n \n-\tr->refs_private = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);\n+\tr->refs_private = ref_store_init(r->gitdir,\n+\t\t\t\t\t r->ref_storage_format ?\n+\t\t\t\t\t\t r->ref_storage_format :\n+\t\t\t\t\t\t default_ref_storage(),\n+\t\t\t\t\t REF_STORE_ALL_CAPS);\n \tr->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private);\n \treturn r->refs_private;\n }\n@@ -1832,7 +1842,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \t\tgoto done;\n \n \t/* assume that add_submodule_odb() has been called */\n-\trefs = ref_store_init(submodule_sb.buf,\n+\trefs = ref_store_init(submodule_sb.buf, default_ref_storage(),\n \t\t\t REF_STORE_READ | REF_STORE_ODB);\n \tregister_ref_store_map(&submodule_ref_stores, \"submodule\",\n \t\t\t refs, submodule);\n@@ -1846,6 +1856,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \n struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n {\n+\tconst char *format = default_ref_storage();\n \tstruct ref_store *refs;\n \tconst char *id;\n \n@@ -1859,9 +1870,9 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n \n \tif (wt->id)\n \t\trefs = ref_store_init(git_common_path(\"worktrees/%s\", wt->id),\n-\t\t\t\t REF_STORE_ALL_CAPS);\n+\t\t\t\t format, REF_STORE_ALL_CAPS);\n \telse\n-\t\trefs = ref_store_init(get_git_common_dir(),\n+\t\trefs = ref_store_init(get_git_common_dir(), format,\n \t\t\t\t REF_STORE_ALL_CAPS);\n \n \tif (refs)\ndiff --git a/refs.h b/refs.h\nindex 66955181569..7dc60472c96 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -11,6 +11,9 @@ struct string_list;\n struct string_list_item;\n struct worktree;\n \n+/* Returns the ref storage backend to use by default. */\n+const char *default_ref_storage(void);\n+\n /*\n * Resolve a reference, recursively following symbolic refererences.\n *\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 467f4b3c936..28166bf1f82 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -669,6 +669,7 @@ struct ref_storage_be {\n };\n \n extern struct ref_storage_be refs_be_files;\n+extern struct ref_storage_be refs_be_reftable;\n extern struct ref_storage_be refs_be_packed;\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nnew file mode 100644\nindex 00000000000..b7d12ca91de\n--- /dev/null\n+++ b/refs/reftable-backend.c\n@@ -0,0 +1,1435 @@\n+#include \"../cache.h\"\n+#include \"../chdir-notify.h\"\n+#include \"../config.h\"\n+#include \"../iterator.h\"\n+#include \"../lockfile.h\"\n+#include \"../refs.h\"\n+#include \"../reftable/reftable-stack.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-blocksource.h\"\n+#include \"../reftable/reftable-reader.h\"\n+#include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-merged.h\"\n+#include \"../reftable/reftable-generic.h\"\n+#include \"../worktree.h\"\n+#include \"refs-internal.h\"\n+\n+extern struct ref_storage_be refs_be_reftable;\n+\n+struct git_reftable_ref_store {\n+\tstruct ref_store base;\n+\tunsigned int store_flags;\n+\n+\tint err;\n+\tchar *repo_dir;\n+\n+\tchar *reftable_dir;\n+\tchar *worktree_reftable_dir;\n+\n+\tstruct reftable_stack *main_stack;\n+\tstruct reftable_stack *worktree_stack;\n+};\n+\n+static struct reftable_stack *stack_for(struct git_reftable_ref_store *store,\n+\t\t\t\t\tconst char *refname)\n+{\n+\tif (store->worktree_stack == NULL)\n+\t\treturn store->main_stack;\n+\n+\tswitch (ref_type(refname)) {\n+\tcase REF_TYPE_PER_WORKTREE:\n+\tcase REF_TYPE_PSEUDOREF:\n+\tcase REF_TYPE_OTHER_PSEUDOREF:\n+\t\treturn store->worktree_stack;\n+\tdefault:\n+\tcase REF_TYPE_MAIN_PSEUDOREF:\n+\tcase REF_TYPE_NORMAL:\n+\t\treturn store->main_stack;\n+\t}\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type);\n+\n+static void clear_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tlog->old_hash = NULL;\n+\tlog->new_hash = NULL;\n+\tlog->message = NULL;\n+\tlog->refname = NULL;\n+\treftable_log_record_release(log);\n+}\n+\n+static void fill_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tconst char *info = git_committer_info(0);\n+\tstruct ident_split split = { NULL };\n+\tint result = split_ident_line(&split, info, strlen(info));\n+\tint sign = 1;\n+\tassert(0 == result);\n+\n+\treftable_log_record_release(log);\n+\tlog->name =\n+\t\txstrndup(split.name_begin, split.name_end - split.name_begin);\n+\tlog->email =\n+\t\txstrndup(split.mail_begin, split.mail_end - split.mail_begin);\n+\tlog->time = atol(split.date_begin);\n+\tif (*split.tz_begin == '-') {\n+\t\tsign = -1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\tif (*split.tz_begin == '+') {\n+\t\tsign = 1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\n+\tlog->tz_offset = sign * atoi(split.tz_begin);\n+}\n+\n+static int has_suffix(struct strbuf *b, const char *suffix)\n+{\n+\tsize_t len = strlen(suffix);\n+\n+\tif (len > b->len) {\n+\t\treturn 0;\n+\t}\n+\n+\treturn 0 == strncmp(b->buf + b->len - len, suffix, len);\n+}\n+\n+/* trims the last path component of b. Returns -1 if it is not\n+ * present, or 0 on success\n+ */\n+static int trim_component(struct strbuf *b)\n+{\n+\tchar *last;\n+\tlast = strrchr(b->buf, '/');\n+\tif (!last)\n+\t\treturn -1;\n+\tstrbuf_setlen(b, last - b->buf);\n+\treturn 0;\n+}\n+\n+/* Returns whether `b` is a worktree path, trimming it to the gitdir\n+ */\n+static int is_worktree(struct strbuf *b)\n+{\n+\tif (trim_component(b) < 0) {\n+\t\treturn 0;\n+\t}\n+\tif (!has_suffix(b, \"/worktrees\")) {\n+\t\treturn 0;\n+\t}\n+\ttrim_component(b);\n+\treturn 1;\n+}\n+\n+static struct ref_store *git_reftable_ref_store_create(const char *path,\n+\t\t\t\t\t\t unsigned int store_flags)\n+{\n+\tstruct git_reftable_ref_store *refs = xcalloc(1, sizeof(*refs));\n+\tstruct ref_store *ref_store = (struct ref_store *)refs;\n+\tstruct reftable_write_options cfg = {\n+\t\t.block_size = 4096,\n+\t\t.hash_id = the_hash_algo->format_id,\n+\t};\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tconst char *gitdir = path;\n+\tstruct strbuf wt_buf = STRBUF_INIT;\n+\tint wt = 0;\n+\n+\tstrbuf_addstr(&wt_buf, path);\n+\n+\t/* this is clumsy, but the official worktree functions (eg.\n+\t * get_worktrees()) function will try to initialize a ref storage\n+\t * backend, leading to infinite recursion. */\n+\twt = is_worktree(&wt_buf);\n+\tif (wt) {\n+\t\tgitdir = wt_buf.buf;\n+\t}\n+\n+\tbase_ref_store_init(ref_store, &refs_be_reftable);\n+\tref_store->gitdir = xstrdup(gitdir);\n+\trefs->store_flags = store_flags;\n+\tstrbuf_addf(&sb, \"%s/reftable\", gitdir);\n+\trefs->reftable_dir = xstrdup(sb.buf);\n+\tstrbuf_reset(&sb);\n+\n+\trefs->err =\n+\t\treftable_new_stack(&refs->main_stack, refs->reftable_dir, cfg);\n+\tassert(refs->err != REFTABLE_API_ERROR);\n+\n+\tif (refs->err == 0 && wt) {\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", path);\n+\t\trefs->worktree_reftable_dir = xstrdup(sb.buf);\n+\n+\t\trefs->err = reftable_new_stack(&refs->worktree_stack,\n+\t\t\t\t\t refs->worktree_reftable_dir,\n+\t\t\t\t\t cfg);\n+\t\tassert(refs->err != REFTABLE_API_ERROR);\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\tstrbuf_release(&wt_buf);\n+\treturn ref_store;\n+}\n+\n+static int git_reftable_init_db(struct ref_store *ref_store, struct strbuf *err)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\tsafe_create_dir(refs->reftable_dir, 1);\n+\tassert(refs->worktree_reftable_dir == NULL);\n+\n+\tstrbuf_addf(&sb, \"%s/HEAD\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"ref: refs/heads/.invalid\");\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs\", refs->base.gitdir);\n+\tsafe_create_dir(sb.buf, 1);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs/heads\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\n+\treturn 0;\n+}\n+\n+struct git_reftable_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_ref_record ref;\n+\tstruct object_id oid;\n+\tstruct ref_store *ref_store;\n+\n+\t/* In case we must iterate over 2 stacks, this is non-null. */\n+\tstruct reftable_merged_table *merged;\n+\tunsigned int flags;\n+\tint err;\n+\tconst char *prefix;\n+};\n+\n+static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\twhile (ri->err == 0) {\n+\t\tri->err = reftable_iterator_next_ref(&ri->iter, &ri->ref);\n+\t\tif (ri->err) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (ref_type(ri->ref.refname) == REF_TYPE_PSEUDOREF) {\n+\t\t\t/*\n+\t\t\t pseudorefs, eg. HEAD, FETCH_HEAD should not be\n+\t\t\t produced, by default.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\t\tri->base.refname = ri->ref.refname;\n+\t\tif (ri->prefix != NULL &&\n+\t\t strncmp(ri->prefix, ri->ref.refname, strlen(ri->prefix))) {\n+\t\t\tri->err = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (ri->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n+\t\t ref_type(ri->base.refname) != REF_TYPE_PER_WORKTREE)\n+\t\t\tcontinue;\n+\n+\t\tri->base.flags = 0;\n+\t\tswitch (ri->ref.value_type) {\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\thashcpy(ri->oid.hash, ri->ref.value.val1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\thashcpy(ri->oid.hash, ri->ref.value.val2.value);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF: {\n+\t\t\tint out_flags = 0;\n+\t\t\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\t\t\tri->ref_store, ri->ref.refname,\n+\t\t\t\tRESOLVE_REF_READING, &ri->oid, &out_flags);\n+\t\t\tri->base.flags = out_flags;\n+\t\t\tif (resolved == NULL &&\n+\t\t\t !(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t\t (ri->base.flags & REF_ISBROKEN)) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\t}\n+\t\tdefault:\n+\t\t\tabort();\n+\t\t}\n+\n+\t\tri->base.oid = &ri->oid;\n+\t\tif (!(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t !ref_resolves_to_object(ri->base.refname, ri->base.oid,\n+\t\t\t\t\t ri->base.flags)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tbreak;\n+\t}\n+\n+\tif (ri->err > 0) {\n+\t\treturn ITER_DONE;\n+\t}\n+\tif (ri->err < 0) {\n+\t\treturn ITER_ERROR;\n+\t}\n+\n+\treturn ITER_OK;\n+}\n+\n+static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\tif (ri->ref.value_type == REFTABLE_REF_VAL2) {\n+\t\thashcpy(peeled->hash, ri->ref.value.val2.target_value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\n+\n+static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\treftable_ref_record_release(&ri->ref);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged) {\n+\t\treftable_merged_table_free(ri->merged);\n+\t}\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n+\treftable_ref_iterator_advance, reftable_ref_iterator_peel,\n+\treftable_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,\n+\t\t\t\tunsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct git_reftable_iterator *ri = xcalloc(1, sizeof(*ri));\n+\n+\tif (refs->err < 0) {\n+\t\tri->err = refs->err;\n+\t} else if (refs->worktree_stack == NULL) {\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tri->err = reftable_merged_table_seek_ref(mt, &ri->iter, prefix);\n+\t} else {\n+\t\tstruct reftable_merged_table *mt1 =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tstruct reftable_merged_table *mt2 =\n+\t\t\treftable_stack_merged_table(refs->worktree_stack);\n+\t\tstruct reftable_table *tabs =\n+\t\t\txcalloc(2, sizeof(struct reftable_table));\n+\t\treftable_table_from_merged_table(&tabs[0], mt1);\n+\t\treftable_table_from_merged_table(&tabs[1], mt2);\n+\t\tri->err = reftable_new_merged_table(&ri->merged, tabs, 2,\n+\t\t\t\t\t\t the_hash_algo->format_id);\n+\t\tif (ri->err == 0)\n+\t\t\tri->err = reftable_merged_table_seek_ref(\n+\t\t\t\tri->merged, &ri->iter, prefix);\n+\t}\n+\n+\tbase_ref_iterator_init(&ri->base, &reftable_ref_iterator_vtable, 1);\n+\tri->prefix = prefix;\n+\tri->base.oid = &ri->oid;\n+\tri->flags = flags;\n+\tri->ref_store = ref_store;\n+\treturn &ri->base;\n+}\n+\n+static int fixup_symrefs(struct ref_store *ref_store,\n+\t\t\t struct ref_transaction *transaction)\n+{\n+\tstruct strbuf referent = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\t\tstruct object_id old_oid;\n+\n+\t\terr = git_reftable_read_raw_ref(ref_store, update->refname,\n+\t\t\t\t\t\t&old_oid, &referent,\n+\t\t\t\t\t\t/* mutate input, like\n+\t\t\t\t\t\t files-backend.c */\n+\t\t\t\t\t\t&update->type);\n+\t\tif (err < 0 && errno == ENOENT &&\n+\t\t is_null_oid(&update->old_oid)) {\n+\t\t\terr = 0;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (!(update->type & REF_ISSYMREF))\n+\t\t\tcontinue;\n+\n+\t\tif (update->flags & REF_NO_DEREF) {\n+\t\t\t/* what should happen here? See files-backend.c\n+\t\t\t * lock_ref_for_update. */\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t If we are updating a symref (eg. HEAD), we should also\n+\t\t\t update the branch that the symref points to.\n+\n+\t\t\t This is generic functionality, and would be better\n+\t\t\t done in refs.c, but the current implementation is\n+\t\t\t intertwined with the locking in files-backend.c.\n+\t\t\t*/\n+\t\t\tint new_flags = update->flags;\n+\t\t\tstruct ref_update *new_update = NULL;\n+\n+\t\t\t/* if this is an update for HEAD, should also record a\n+\t\t\t log entry for HEAD? See files-backend.c,\n+\t\t\t split_head_update()\n+\t\t\t*/\n+\t\t\tnew_update = ref_transaction_add_update(\n+\t\t\t\ttransaction, referent.buf, new_flags,\n+\t\t\t\t&update->new_oid, &update->old_oid,\n+\t\t\t\tupdate->msg);\n+\t\t\tnew_update->parent_update = update;\n+\n+\t\t\t/* files-backend sets REF_LOG_ONLY here. */\n+\t\t\tupdate->flags |= REF_NO_DEREF | REF_LOG_ONLY;\n+\t\t\tupdate->flags &= ~REF_HAVE_OLD;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tstrbuf_release(&referent);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_prepare(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errbuf)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_addition *add = NULL;\n+\tstruct reftable_stack *stack =\n+\t\ttransaction->nr ?\n+\t\t\t stack_for(refs, transaction->updates[0]->refname) :\n+\t\t\t refs->main_stack;\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_new_addition(&add, stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = fixup_symrefs(ref_store, transaction);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\ttransaction->backend_data = add;\n+\ttransaction->state = REF_TRANSACTION_PREPARED;\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tif (err < 0) {\n+\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\t\tstrbuf_addf(errbuf, \"reftable: transaction prepare: %s\",\n+\t\t\t reftable_error_str(err));\n+\t}\n+\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_abort(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\treftable_addition_destroy(add);\n+\ttransaction->backend_data = NULL;\n+\treturn 0;\n+}\n+\n+static int reftable_check_old_oid(struct ref_store *refs, const char *refname,\n+\t\t\t\t struct object_id *want_oid)\n+{\n+\tstruct object_id out_oid;\n+\tint out_flags = 0;\n+\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\trefs, refname, RESOLVE_REF_READING, &out_oid, &out_flags);\n+\tif (is_null_oid(want_oid) != (resolved == NULL)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\tif (resolved != NULL && !oideq(&out_oid, want_oid)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int ref_update_cmp(const void *a, const void *b)\n+{\n+\treturn strcmp((*(struct ref_update **)a)->refname,\n+\t\t (*(struct ref_update **)b)->refname);\n+}\n+\n+static int write_transaction_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct ref_transaction *transaction = (struct ref_transaction *)arg;\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)transaction->ref_store;\n+\tstruct reftable_stack *stack =\n+\t\tstack_for(refs, transaction->updates[0]->refname);\n+\tuint64_t ts = reftable_stack_next_update_index(stack);\n+\tint err = 0;\n+\tint i = 0;\n+\tstruct reftable_log_record *logs =\n+\t\tcalloc(transaction->nr, sizeof(*logs));\n+\tstruct ref_update **sorted =\n+\t\tmalloc(transaction->nr * sizeof(struct ref_update *));\n+\tCOPY_ARRAY(sorted, transaction->updates, transaction->nr);\n+\tQSORT(sorted, transaction->nr, ref_update_cmp);\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = sorted[i];\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tfill_reftable_log_record(log);\n+\t\tlog->refname = (char *)u->refname;\n+\t\tlog->old_hash = u->old_oid.hash;\n+\t\tlog->new_hash = u->new_oid.hash;\n+\t\tlog->update_index = ts;\n+\t\tlog->message = u->msg;\n+\n+\t\tif (u->flags & REF_LOG_ONLY) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (u->flags & REF_HAVE_NEW) {\n+\t\t\tstruct reftable_ref_record ref = { NULL };\n+\t\t\tstruct object_id peeled;\n+\n+\t\t\tint peel_error = peel_object(&u->new_oid, &peeled);\n+\t\t\tref.refname = (char *)u->refname;\n+\t\t\tref.update_index = ts;\n+\n+\t\t\tif (!peel_error) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\t\t\tref.value.val2.target_value = peeled.hash;\n+\t\t\t\tref.value.val2.value = u->new_oid.hash;\n+\t\t\t} else if (!is_null_oid(&u->new_oid)) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\t\t\tref.value.val1 = u->new_oid.hash;\n+\t\t\t}\n+\n+\t\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\terr = reftable_writer_add_log(writer, &logs[i]);\n+\t\tclear_reftable_log_record(&logs[i]);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tfree(logs);\n+\tfree(sorted);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_finish(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errmsg)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\tint err = 0;\n+\tint i;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = transaction->updates[i];\n+\t\tif (u->flags & REF_HAVE_OLD) {\n+\t\t\terr = reftable_check_old_oid(transaction->ref_store,\n+\t\t\t\t\t\t u->refname, &u->old_oid);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (transaction->nr) {\n+\t\terr = reftable_addition_add(add, &write_transaction_table,\n+\t\t\t\t\t transaction);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\terr = reftable_addition_commit(add);\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_addition_destroy(add);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\ttransaction->backend_data = NULL;\n+\tif (err) {\n+\t\tstrbuf_addf(errmsg, \"reftable: transaction failure: %s\",\n+\t\t\t reftable_error_str(err));\n+\t\treturn -1;\n+\t}\n+\treturn err;\n+}\n+\n+static int\n+git_reftable_transaction_initial_commit(struct ref_store *ref_store,\n+\t\t\t\t\tstruct ref_transaction *transaction,\n+\t\t\t\t\tstruct strbuf *errmsg)\n+{\n+\tint err = git_reftable_transaction_prepare(ref_store, transaction,\n+\t\t\t\t\t\t errmsg);\n+\tif (err)\n+\t\treturn err;\n+\n+\treturn git_reftable_transaction_finish(ref_store, transaction, errmsg);\n+}\n+\n+struct write_delete_refs_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct string_list *refnames;\n+\tconst char *logmsg;\n+\tunsigned int flags;\n+};\n+\n+static int write_delete_refs_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_delete_refs_arg *arg =\n+\t\t(struct write_delete_refs_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint err = 0;\n+\tint i = 0;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = (char *)arg->refnames->items[i].string,\n+\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_log_record log = {\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\tstruct reftable_ref_record current = { NULL };\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.message = xstrdup(arg->logmsg);\n+\t\tlog.new_hash = NULL;\n+\t\tlog.old_hash = NULL;\n+\t\tlog.update_index = ts;\n+\t\tlog.refname = (char *)arg->refnames->items[i].string;\n+\n+\t\tif (reftable_stack_read_ref(arg->stack, log.refname,\n+\t\t\t\t\t ¤t) == 0) {\n+\t\t\tlog.old_hash = reftable_ref_record_val1(¤t);\n+\t\t}\n+\t\terr = reftable_writer_add_log(writer, &log);\n+\t\tlog.old_hash = NULL;\n+\t\treftable_ref_record_release(¤t);\n+\n+\t\tclear_reftable_log_record(&log);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int git_reftable_delete_refs(struct ref_store *ref_store,\n+\t\t\t\t const char *msg,\n+\t\t\t\t struct string_list *refnames,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack =\n+\t\tstack_for(refs, refnames->items[0].string);\n+\tstruct write_delete_refs_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refnames = refnames,\n+\t\t.logmsg = msg,\n+\t\t.flags = flags,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tstring_list_sort(refnames);\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_delete_refs_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_pack_refs(struct ref_store *ref_store,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\terr = reftable_stack_compact_all(refs->main_stack, NULL);\n+\tif (err == 0 && refs->worktree_stack != NULL)\n+\t\terr = reftable_stack_compact_all(refs->worktree_stack, NULL);\n+\treturn err;\n+}\n+\n+struct write_create_symref_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+\tconst char *target;\n+\tconst char *logmsg;\n+};\n+\n+static int write_create_symref_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct write_create_symref_arg *create =\n+\t\t(struct write_create_symref_arg *)arg;\n+\tuint64_t ts = reftable_stack_next_update_index(create->stack);\n+\tint err = 0;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)create->refname,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = (char *)create->target,\n+\t\t.update_index = ts,\n+\t};\n+\treftable_writer_set_limits(writer, ts, ts);\n+\terr = reftable_writer_add_ref(writer, &ref);\n+\tif (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tstruct object_id new_oid;\n+\t\tstruct object_id old_oid;\n+\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.refname = (char *)create->refname;\n+\t\tlog.message = (char *)create->logmsg;\n+\t\tlog.update_index = ts;\n+\t\tif (refs_resolve_ref_unsafe(\n+\t\t\t (struct ref_store *)create->refs, create->refname,\n+\t\t\t RESOLVE_REF_READING, &old_oid, NULL) != NULL) {\n+\t\t\tlog.old_hash = old_oid.hash;\n+\t\t}\n+\n+\t\tif (refs_resolve_ref_unsafe((struct ref_store *)create->refs,\n+\t\t\t\t\t create->target, RESOLVE_REF_READING,\n+\t\t\t\t\t &new_oid, NULL) != NULL) {\n+\t\t\tlog.new_hash = new_oid.hash;\n+\t\t}\n+\n+\t\tif (log.old_hash != NULL || log.new_hash != NULL) {\n+\t\t\terr = reftable_writer_add_log(writer, &log);\n+\t\t}\n+\t\tlog.refname = NULL;\n+\t\tlog.message = NULL;\n+\t\tlog.old_hash = NULL;\n+\t\tlog.new_hash = NULL;\n+\t\tclear_reftable_log_record(&log);\n+\t}\n+\treturn err;\n+}\n+\n+static int git_reftable_create_symref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, const char *target,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_create_symref_arg arg = { .refs = refs,\n+\t\t\t\t\t .stack = stack,\n+\t\t\t\t\t .refname = refname,\n+\t\t\t\t\t .target = target,\n+\t\t\t\t\t .logmsg = logmsg };\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_create_symref_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct write_rename_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *oldname;\n+\tconst char *newname;\n+\tconst char *logmsg;\n+};\n+\n+static int write_rename_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &ref);\n+\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* XXX do ref renames overwrite the target? */\n+\tif (reftable_stack_read_ref(arg->stack, arg->newname, &ref) == 0) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\t{\n+\t\tstruct reftable_ref_record todo[2] = {\n+\t\t\t{\n+\t\t\t\t.refname = (char *)arg->oldname,\n+\t\t\t\t.update_index = ts,\n+\t\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t},\n+\t\t\tref,\n+\t\t};\n+\t\ttodo[1].update_index = ts;\n+\t\tfree(todo[1].refname);\n+\t\ttodo[1].refname = strdup(arg->newname);\n+\n+\t\terr = reftable_writer_add_refs(writer, todo, 2);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (reftable_ref_record_val1(&ref)) {\n+\t\tuint8_t *val1 = reftable_ref_record_val1(&ref);\n+\t\tstruct reftable_log_record todo[2] = { { NULL } };\n+\t\tfill_reftable_log_record(&todo[0]);\n+\t\tfill_reftable_log_record(&todo[1]);\n+\n+\t\ttodo[0].refname = (char *)arg->oldname;\n+\t\ttodo[0].update_index = ts;\n+\t\ttodo[0].message = (char *)arg->logmsg;\n+\t\ttodo[0].old_hash = val1;\n+\t\ttodo[0].new_hash = NULL;\n+\n+\t\ttodo[1].refname = (char *)arg->newname;\n+\t\ttodo[1].update_index = ts;\n+\t\ttodo[1].old_hash = NULL;\n+\t\ttodo[1].new_hash = val1;\n+\t\ttodo[1].message = (char *)arg->logmsg;\n+\n+\t\terr = reftable_writer_add_logs(writer, todo, 2);\n+\n+\t\tclear_reftable_log_record(&todo[0]);\n+\t\tclear_reftable_log_record(&todo[1]);\n+\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t} else {\n+\t\t/* XXX symrefs? */\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+static int git_reftable_rename_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname,\n+\t\t\t\t const char *newrefname, const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n+\tstruct write_rename_arg arg = {\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_rename_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_copy_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname, const char *newrefname,\n+\t\t\t\t const char *logmsg)\n+{\n+\tBUG(\"reftable reference store does not support copying references\");\n+}\n+\n+struct git_reftable_reflog_ref_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_log_record log;\n+\tstruct object_id oid;\n+\n+\t/* Used when iterating over worktree & main */\n+\tstruct reftable_merged_table *merged;\n+\tchar *last_name;\n+};\n+\n+static int\n+git_reftable_reflog_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&ri->iter, &ri->log);\n+\t\tif (err > 0) {\n+\t\t\treturn ITER_DONE;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn ITER_ERROR;\n+\t\t}\n+\n+\t\tri->base.refname = ri->log.refname;\n+\t\tif (ri->last_name != NULL &&\n+\t\t !strcmp(ri->log.refname, ri->last_name)) {\n+\t\t\t/* we want the refnames that we have reflogs for, so we\n+\t\t\t * skip if we've already produced this name. This could\n+\t\t\t * be faster by seeking directly to\n+\t\t\t * reflog@update_index==0.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfree(ri->last_name);\n+\t\tri->last_name = xstrdup(ri->log.refname);\n+\t\thashcpy(ri->oid.hash, ri->log.new_hash);\n+\t\treturn ITER_OK;\n+\t}\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tBUG(\"not supported.\");\n+\treturn -1;\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\treftable_log_record_release(&ri->log);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged)\n+\t\treftable_merged_table_free(ri->merged);\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable git_reftable_reflog_ref_iterator_vtable = {\n+\tgit_reftable_reflog_ref_iterator_advance,\n+\tgit_reftable_reflog_ref_iterator_peel,\n+\tgit_reftable_reflog_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_reflog_iterator_begin(struct ref_store *ref_store)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri = xcalloc(sizeof(*ri), 1);\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\n+\tif (refs->worktree_stack == NULL) {\n+\t\tstruct reftable_stack *stack = refs->main_stack;\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(stack);\n+\t\tint err = reftable_merged_table_seek_log(mt, &ri->iter, \"\");\n+\t\tif (err < 0) {\n+\t\t\tfree(ri);\n+\t\t\t/* XXX is this allowed? */\n+\t\t\treturn NULL;\n+\t\t}\n+\t} else {\n+\t\tstruct reftable_merged_table *mt1 =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tstruct reftable_merged_table *mt2 =\n+\t\t\treftable_stack_merged_table(refs->worktree_stack);\n+\t\tstruct reftable_table *tabs =\n+\t\t\txcalloc(2, sizeof(struct reftable_table));\n+\t\tint err = 0;\n+\t\treftable_table_from_merged_table(&tabs[0], mt1);\n+\t\treftable_table_from_merged_table(&tabs[1], mt2);\n+\t\terr = reftable_new_merged_table(&ri->merged, tabs, 2,\n+\t\t\t\t\t\tthe_hash_algo->format_id);\n+\t\tif (err < 0) {\n+\t\t\tfree(tabs);\n+\t\t\t/* XXX see above */\n+\t\t\treturn NULL;\n+\t\t}\n+\t\terr = reftable_merged_table_seek_ref(ri->merged, &ri->iter, \"\");\n+\t\tif (err < 0) {\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\tbase_ref_iterator_init(&ri->base,\n+\t\t\t &git_reftable_reflog_ref_iterator_vtable, 1);\n+\tri->base.oid = &ri->oid;\n+\n+\treturn (struct ref_iterator *)ri;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_newest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (err == 0) {\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\thashcpy(old_oid.hash, log.old_hash);\n+\t\thashcpy(new_oid.hash, log.new_hash);\n+\n+\t\tfull_committer = fmt_ident(log.name, log.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT,\n+\t\t\t\t\t /*date*/ NULL, IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer, log.time,\n+\t\t\t log.tz_offset, log.message, cb_data);\n+\t\tif (err)\n+\t\t\tbreak;\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_oldest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record *logs = NULL;\n+\tint cap = 0;\n+\tint len = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\n+\twhile (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tlogs = realloc(logs, cap * sizeof(*logs));\n+\t\t}\n+\n+\t\tlogs[len++] = log;\n+\t}\n+\n+\tfor (i = len; i--;) {\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\thashcpy(old_oid.hash, log->old_hash);\n+\t\thashcpy(new_oid.hash, log->new_hash);\n+\n+\t\tfull_committer = fmt_ident(log->name, log->email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT, NULL,\n+\t\t\t\t\t IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer, log->time,\n+\t\t\t log->tz_offset, log->message, cb_data);\n+\t\tif (err) {\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tfree(logs);\n+\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_reflog_exists(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record log = { NULL };\n+\tint err = refs->err;\n+\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (strcmp(log.refname, refname)) {\n+\t\terr = 1;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\treturn !err;\n+}\n+\n+static int git_reftable_create_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, int force_create,\n+\t\t\t\t struct strbuf *err)\n+{\n+\treturn 0;\n+}\n+\n+static int git_reftable_delete_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\treturn 0;\n+}\n+\n+struct reflog_expiry_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_log_record *tombstones;\n+\tint len;\n+\tint cap;\n+};\n+\n+static void clear_log_tombstones(struct reflog_expiry_arg *arg)\n+{\n+\tint i = 0;\n+\tfor (; i < arg->len; i++) {\n+\t\treftable_log_record_release(&arg->tombstones[i]);\n+\t}\n+\n+\tFREE_AND_NULL(arg->tombstones);\n+}\n+\n+static void add_log_tombstone(struct reflog_expiry_arg *arg,\n+\t\t\t const char *refname, uint64_t ts)\n+{\n+\tstruct reftable_log_record tombstone = {\n+\t\t.refname = xstrdup(refname),\n+\t\t.update_index = ts,\n+\t};\n+\tif (arg->len == arg->cap) {\n+\t\targ->cap = 2 * arg->cap + 1;\n+\t\targ->tombstones =\n+\t\t\trealloc(arg->tombstones, arg->cap * sizeof(tombstone));\n+\t}\n+\targ->tombstones[arg->len++] = tombstone;\n+}\n+\n+static int write_reflog_expiry_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct reflog_expiry_arg *arg = (struct reflog_expiry_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint i = 0;\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->len; i++) {\n+\t\tint err = reftable_writer_add_log(writer, &arg->tombstones[i]);\n+\t\tif (err) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int\n+git_reftable_reflog_expire(struct ref_store *ref_store, const char *refname,\n+\t\t\t const struct object_id *oid, unsigned int flags,\n+\t\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t void *policy_cb_data)\n+{\n+\t/*\n+\t For log expiry, we write tombstones in place of the expired entries,\n+\t This means that the entries are still retrievable by delving into the\n+\t stack, and expiring entries paradoxically takes extra memory.\n+\n+\t This memory is only reclaimed when some operation issues a\n+\t git_reftable_pack_refs(), which will compact the entire stack and get\n+\t rid of deletion entries.\n+\n+\t It would be better if the refs backend supported an API that sets a\n+\t criterion for all refs, passing the criterion to pack_refs().\n+\t*/\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reflog_expiry_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refs = refs,\n+\t};\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = 0;\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\twhile (1) {\n+\t\tstruct object_id ooid;\n+\t\tstruct object_id noid;\n+\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (err > 0 || strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\t\thashcpy(ooid.hash, log.old_hash);\n+\t\thashcpy(noid.hash, log.new_hash);\n+\n+\t\tif (should_prune_fn(&ooid, &noid, log.email,\n+\t\t\t\t (timestamp_t)log.time, log.tz_offset,\n+\t\t\t\t log.message, policy_cb_data)) {\n+\t\t\tadd_log_tombstone(&arg, refname, log.update_index);\n+\t\t}\n+\t}\n+\terr = reftable_stack_add(stack, &write_reflog_expiry_table, &arg);\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\tclear_log_tombstones(&arg);\n+\treturn err;\n+}\n+\n+static int reftable_error_to_errno(int err)\n+{\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn EIO;\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn EFAULT;\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn ENOENT;\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn EBUSY;\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn EINVAL;\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn EDOM;\n+\tdefault:\n+\t\treturn ERANGE;\n+\t}\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\n+\t/* This is usually not needed, but Git doesn't signal to ref backend if\n+\t a subprocess updated the ref DB. So we always check.\n+\t*/\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (err > 0) {\n+\t\terrno = ENOENT;\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0) {\n+\t\terrno = reftable_error_to_errno(err);\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\n+\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (reftable_ref_record_val1(&ref) != NULL) {\n+\t\thashcpy(oid->hash, reftable_ref_record_val1(&ref));\n+\t} else {\n+\t\t*type |= REF_ISBROKEN;\n+\t\terrno = EINVAL;\n+\t\terr = -1;\n+\t}\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+struct ref_storage_be refs_be_reftable = {\n+\t&refs_be_files,\n+\t\"reftable\",\n+\tgit_reftable_ref_store_create,\n+\tgit_reftable_init_db,\n+\tgit_reftable_transaction_prepare,\n+\tgit_reftable_transaction_finish,\n+\tgit_reftable_transaction_abort,\n+\tgit_reftable_transaction_initial_commit,\n+\n+\tgit_reftable_pack_refs,\n+\tgit_reftable_create_symref,\n+\tgit_reftable_delete_refs,\n+\tgit_reftable_rename_ref,\n+\tgit_reftable_copy_ref,\n+\n+\tgit_reftable_ref_iterator_begin,\n+\tgit_reftable_read_raw_ref,\n+\n+\tgit_reftable_reflog_iterator_begin,\n+\tgit_reftable_for_each_reflog_ent_oldest_first,\n+\tgit_reftable_for_each_reflog_ent_newest_first,\n+\tgit_reftable_reflog_exists,\n+\tgit_reftable_create_reflog,\n+\tgit_reftable_delete_reflog,\n+\tgit_reftable_reflog_expire,\n+};\ndiff --git a/repository.c b/repository.c\nindex a4174ddb062..ff0988dac84 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -174,6 +174,8 @@ int repo_init(struct repository *repo,\n \tif (worktree)\n \t\trepo_set_worktree(repo, worktree);\n \n+\trepo->ref_storage_format = xstrdup_or_null(format.ref_storage);\n+\n \tclear_repository_format(&format);\n \treturn 0;\n \ndiff --git a/repository.h b/repository.h\nindex b385ca3c94b..8019a7d0a1f 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -78,6 +78,9 @@ struct repository {\n \t */\n \tstruct ref_store *refs_private;\n \n+\t/* The format to use for the ref database. */\n+\tchar *ref_storage_format;\n+\n \t/*\n \t * Contains path to often used file names.\n \t */\ndiff --git a/setup.c b/setup.c\nindex c04cd25a30d..c6b57efd031 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -500,6 +500,9 @@ static enum extension_result handle_extension(const char *var,\n \t\t\treturn error(\"invalid value for 'extensions.objectformat'\");\n \t\tdata->hash_algo = format;\n \t\treturn EXTENSION_OK;\n+\t} else if (!strcmp(ext, \"refstorage\")) {\n+\t\tdata->ref_storage = xstrdup(value);\n+\t\treturn EXTENSION_OK;\n \t}\n \treturn EXTENSION_UNKNOWN;\n }\n@@ -651,6 +654,7 @@ void clear_repository_format(struct repository_format *format)\n \tstring_list_clear(&format->v1_only_extensions, 0);\n \tfree(format->work_tree);\n \tfree(format->partial_clone);\n+\tfree(format->ref_storage);\n \tinit_repository_format(format);\n }\n \n@@ -1308,8 +1312,11 @@ const char *setup_git_directory_gently(int *nongit_ok)\n \t\t\t\tgitdir = DEFAULT_GIT_DIR_ENVIRONMENT;\n \t\t\tsetup_git_env(gitdir);\n \t\t}\n-\t\tif (startup_info->have_repository)\n+\t\tif (startup_info->have_repository) {\n \t\t\trepo_set_hash_algo(the_repository, repo_fmt.hash_algo);\n+\t\t\tthe_repository->ref_storage_format =\n+\t\t\t\txstrdup_or_null(repo_fmt.ref_storage);\n+\t\t}\n \t}\n \n \tstrbuf_release(&dir);\ndiff --git a/t/t0031-reftable.sh b/t/t0031-reftable.sh\nnew file mode 100755\nindex 00000000000..58c7d5d4bcd\n--- /dev/null\n+++ b/t/t0031-reftable.sh\n@@ -0,0 +1,199 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable basics'\n+\n+. ./test-lib.sh\n+\n+INVALID_SHA1=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n+\n+initialize () {\n+\trm -rf .git &&\n+\tgit init --ref-storage=reftable &&\n+\tmv .git/hooks .git/hooks-disabled\n+}\n+\n+test_expect_success 'SHA256 support, env' '\n+\trm -rf .git &&\n+\tGIT_DEFAULT_HASH=sha256 && export GIT_DEFAULT_HASH &&\n+\tgit init --ref-storage=reftable &&\n+\tmv .git/hooks .git/hooks-disabled &&\n+\ttest_commit file\n+'\n+\n+test_expect_success 'SHA256 support, option' '\n+\trm -rf .git &&\n+\tgit init --ref-storage=reftable --object-format=sha256 &&\n+\tmv .git/hooks .git/hooks-disabled &&\n+\ttest_commit file\n+'\n+\n+test_expect_success 'delete ref' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\tSHA=$(git show-ref -s --verify HEAD) &&\n+\ttest_write_lines \"$SHA refs/heads/master\" \"$SHA refs/tags/file\" >expect &&\n+\tgit show-ref > actual &&\n+\t! git update-ref -d refs/tags/file $INVALID_SHA1 &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d refs/tags/file $SHA &&\n+\ttest_write_lines \"$SHA refs/heads/master\" >expect &&\n+\tgit show-ref > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+\n+test_expect_success 'clone calls transaction_initial_commit' '\n+\ttest_commit message1 file1 &&\n+\tgit clone . cloned &&\n+\t(test -f cloned/file1 || echo \"Fixme.\")\n+'\n+\n+test_expect_success 'basic operation of reftable storage: commit, show-ref' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_write_lines refs/heads/master refs/tags/file >expect &&\n+\tgit show-ref &&\n+\tgit show-ref | cut -f2 -d\" \" > actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'reflog, repack' '\n+\tinitialize &&\n+\tfor count in $(test_seq 1 10)\n+\tdo\n+\t\ttest_commit \"number $count\" file.t $count number-$count ||\n+\t\treturn 1\n+\tdone &&\n+\tgit pack-refs &&\n+\tls -1 .git/reftable >table-files &&\n+\ttest_line_count = 2 table-files &&\n+\tgit reflog refs/heads/master >output &&\n+\ttest_line_count = 10 output &&\n+\tgrep \"commit (initial): number 1\" output &&\n+\tgrep \"commit: number 10\" output &&\n+\tgit gc &&\n+\tgit reflog refs/heads/master >output &&\n+\ttest_line_count = 0 output\n+'\n+\n+test_expect_success 'branch switch in reflog output' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tgit checkout -b branch1 &&\n+\ttest_commit file2 &&\n+\tgit checkout -b branch2 &&\n+\tgit switch - &&\n+\tgit rev-parse --symbolic-full-name HEAD > actual &&\n+\techo refs/heads/branch1 > expect &&\n+\ttest_cmp actual expect\n+'\n+\n+\n+# This matches show-ref's output\n+print_ref() {\n+\techo \"$(git rev-parse \"$1\") $1\"\n+}\n+\n+test_expect_success 'peeled tags are stored' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\tgit tag -m \"annotated tag\" test_tag HEAD &&\n+\t{\n+\t\tprint_ref \"refs/heads/master\" &&\n+\t\tprint_ref \"refs/tags/file\" &&\n+\t\tprint_ref \"refs/tags/test_tag\" &&\n+\t\tprint_ref \"refs/tags/test_tag^{}\"\n+\t} >expect &&\n+\tgit show-ref -d >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'show-ref works on fresh repo' '\n+\tinitialize &&\n+\trm -rf .git &&\n+\tgit init --ref-storage=reftable &&\n+\t>expect &&\n+\t! git show-ref > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'checkout unborn branch' '\n+\tinitialize &&\n+\tgit checkout -b master\n+'\n+\n+\n+test_expect_success 'dir/file conflict' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\t! git branch master/forbidden\n+'\n+\n+\n+test_expect_success 'do not clobber existing repo' '\n+\trm -rf .git &&\n+\tgit init --ref-storage=files &&\n+\tcat .git/HEAD > expect &&\n+\ttest_commit file &&\n+\t(git init --ref-storage=reftable || true) &&\n+\tcat .git/HEAD > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# cherry-pick uses a pseudo ref.\n+test_expect_success 'pseudo refs' '\n+\tinitialize &&\n+\ttest_commit message1 file1 &&\n+\ttest_commit message2 file2 &&\n+\tgit branch source &&\n+\tgit checkout HEAD^ &&\n+\ttest_commit message3 file3 &&\n+\tgit cherry-pick source &&\n+\ttest -f file2\n+'\n+\n+# cherry-pick uses a pseudo ref.\n+test_expect_success 'rebase' '\n+\tinitialize &&\n+\ttest_commit message1 file1 &&\n+\ttest_commit message2 file2 &&\n+\tgit branch source &&\n+\tgit checkout HEAD^ &&\n+\ttest_commit message3 file3 &&\n+\tgit rebase source &&\n+\ttest -f file2\n+'\n+\n+test_expect_success 'worktrees' '\n+\tgit init --ref-storage=reftable start &&\n+\t(cd start && test_commit file1 && git checkout -b branch1 &&\n+\tgit checkout -b branch2 &&\n+\tgit worktree add ../wt\n+\t) &&\n+\tcd wt &&\n+\tgit checkout branch1 &&\n+\tgit branch\n+'\n+\n+test_expect_success 'worktrees 2' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tmkdir existing_empty &&\n+\tgit worktree add --detach existing_empty master\n+'\n+\n+test_expect_success 'FETCH_HEAD' '\n+\tinitialize &&\n+\ttest_commit one &&\n+\t(git init sub && cd sub && test_commit two) &&\n+\tgit --git-dir sub/.git rev-parse HEAD >expect &&\n+\tgit fetch sub &&\n+\tgit checkout FETCH_HEAD &&\n+\tgit rev-parse HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/t1409-avoid-packing-refs.sh b/t/t1409-avoid-packing-refs.sh\nindex be12fb63506..c6f78325563 100755\n--- a/t/t1409-avoid-packing-refs.sh\n+++ b/t/t1409-avoid-packing-refs.sh\n@@ -4,6 +4,12 @@ test_description='avoid rewriting packed-refs unnecessarily'\n \n . ./test-lib.sh\n \n+if test_have_prereq REFTABLE\n+then\n+ skip_all='skipping pack-refs tests; incompatible with reftable'\n+ test_done\n+fi\n+\n # Add an identifying mark to the packed-refs file header line. This\n # shouldn't upset readers, and it should be omitted if the file is\n # ever rewritten.\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex b17f5c21fbc..cc5d01571a4 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -8,6 +8,12 @@ test_description='git fsck random collection of tests\n \n . ./test-lib.sh\n \n+if test_have_prereq REFTABLE\n+then\n+ skip_all='skipping tests; incompatible with reftable'\n+ test_done\n+fi\n+\n test_expect_success setup '\n \tgit config gc.auto 0 &&\n \tgit config i18n.commitencoding ISO-8859-1 &&\ndiff --git a/t/t3210-pack-refs.sh b/t/t3210-pack-refs.sh\nindex f41b2afb996..edaef2c175a 100755\n--- a/t/t3210-pack-refs.sh\n+++ b/t/t3210-pack-refs.sh\n@@ -11,6 +11,12 @@ semantic is still the same.\n '\n . ./test-lib.sh\n \n+if test_have_prereq REFTABLE\n+then\n+ skip_all='skipping pack-refs tests; incompatible with reftable'\n+ test_done\n+fi\n+\n test_expect_success 'enable reflogs' '\n \tgit config core.logallrefupdates true\n '\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex a863ccee7e9..7b638e0b8c8 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1520,6 +1520,11 @@ parisc* | hppa*)\n \t;;\n esac\n \n+if test -n \"$GIT_TEST_REFTABLE\"\n+then\n+ test_set_prereq REFTABLE\n+fi\n+\n ( COLUMNS=1 && test $COLUMNS = 1 ) && test_set_prereq COLUMNS_CAN_BE_1\n test -z \"$NO_PERL\" && test_set_prereq PERL\n test -z \"$NO_PTHREADS\" && test_set_prereq PTHREADS\n-- \ngitgitgadget\n\n"},{"id":"411857","messageId":"557183d3e3e252932ce2f8b7c96d7378e295e2dd.1607522429.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v4.git.git.1607522429.gitgitgadget@gmail.com","subject":"[PATCH v4 12/15] reftable: rest of library","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-12-09T14:00:26Z","receivedAt":"2020-12-09T14:02:39Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 4 +\n reftable/VERSION | 1 +\n reftable/dump.c | 212 ++++++\n reftable/merged.c | 366 ++++++++++\n reftable/merged.h | 35 +\n reftable/merged_test.c | 343 ++++++++++\n reftable/pq.c | 115 ++++\n reftable/pq.h | 32 +\n reftable/refname.c | 209 ++++++\n reftable/refname.h | 29 +\n reftable/refname_test.c | 102 +++\n reftable/reftable-generic.h | 48 ++\n reftable/reftable-merged.h | 68 ++\n reftable/reftable-stack.h | 120 ++++\n reftable/reftable.c | 98 +++\n reftable/stack.c | 1260 +++++++++++++++++++++++++++++++++++\n reftable/stack.h | 41 ++\n reftable/stack_test.c | 803 ++++++++++++++++++++++\n t/helper/test-reftable.c | 3 +\n 19 files changed, 3889 insertions(+)\n create mode 100644 reftable/VERSION\n create mode 100644 reftable/dump.c\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n create mode 100644 reftable/reftable-generic.h\n create mode 100644 reftable/reftable-merged.h\n create mode 100644 reftable/reftable-stack.h\n create mode 100644 reftable/reftable.c\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n\ndiff --git a/Makefile b/Makefile\nindex c913e90b643..18cc18c2153 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2394,10 +2394,14 @@ REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n+REFTABLE_OBJS += reftable/merged.o\n+REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/refname.o\n REFTABLE_OBJS += reftable/reftable.o\n+REFTABLE_OBJS += reftable/stack.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n REFTABLE_OBJS += reftable/zlib-compat.o\ndiff --git a/reftable/VERSION b/reftable/VERSION\nnew file mode 100644\nindex 00000000000..a67c0682e1b\n--- /dev/null\n+++ b/reftable/VERSION\n@@ -0,0 +1 @@\n+9b4a54059db9a05c270c0a0587f245bc6868d576 C: use rand() rather than cobbled together random generator.\ndiff --git a/reftable/dump.c b/reftable/dump.c\nnew file mode 100644\nindex 00000000000..00b444e8c9b\n--- /dev/null\n+++ b/reftable/dump.c\n@@ -0,0 +1,212 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+\n+#include \"reftable.h\"\n+#include \"reftable-tests.h\"\n+\n+static uint32_t hash_id;\n+\n+static int dump_table(const char *tablename)\n+{\n+\tstruct reftable_block_source src = { 0 };\n+\tint err = reftable_block_source_from_file(&src, tablename);\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_ref_record ref = { 0 };\n+\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_reader *r = NULL;\n+\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\terr = reftable_new_reader(&r, &src, tablename);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\terr = reftable_reader_seek_ref(r, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_ref_record_print(&ref, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_clear(&ref);\n+\n+\terr = reftable_reader_seek_log(r, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_log_record_print(&log, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_clear(&log);\n+\n+\treftable_reader_free(r);\n+\treturn 0;\n+}\n+\n+static int compact_stack(const char *stackdir)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = {};\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_stack_compact_all(stack, NULL);\n+\tif (err < 0)\n+\t\tgoto done;\n+done:\n+\tif (stack != NULL) {\n+\t\treftable_stack_destroy(stack);\n+\t}\n+\treturn err;\n+}\n+\n+static int dump_stack(const char *stackdir)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = {};\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_ref_record ref = { 0 };\n+\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_merged_table *merged = NULL;\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tmerged = reftable_stack_merged_table(stack);\n+\n+\terr = reftable_merged_table_seek_ref(merged, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_ref_record_print(&ref, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_clear(&ref);\n+\n+\terr = reftable_merged_table_seek_log(merged, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_log_record_print(&log, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_clear(&log);\n+\n+\treftable_stack_destroy(stack);\n+\treturn 0;\n+}\n+\n+static void print_help(void)\n+{\n+\tprintf(\"usage: dump [-cst] arg\\n\\n\"\n+\t \"options: \\n\"\n+\t \" -c compact\\n\"\n+\t \" -t dump table\\n\"\n+\t \" -s dump stack\\n\"\n+\t \" -h this help\\n\"\n+\t \" -2 use SHA256\\n\"\n+\t \"\\n\");\n+}\n+\n+int reftable_dump_main(int argc, char *const *argv)\n+{\n+\tint err = 0;\n+\tint opt;\n+\tint opt_dump_table = 0;\n+\tint opt_dump_stack = 0;\n+\tint opt_compact = 0;\n+\tconst char *arg = NULL;\n+\twhile ((opt = getopt(argc, argv, \"2chts\")) != -1) {\n+\t\tswitch (opt) {\n+\t\tcase '2':\n+\t\t\thash_id = 0x73323536;\n+\t\t\tbreak;\n+\t\tcase 't':\n+\t\t\topt_dump_table = 1;\n+\t\t\tbreak;\n+\t\tcase 's':\n+\t\t\topt_dump_stack = 1;\n+\t\t\tbreak;\n+\t\tcase 'c':\n+\t\t\topt_compact = 1;\n+\t\t\tbreak;\n+\t\tcase '?':\n+\t\tcase 'h':\n+\t\t\tprint_help();\n+\t\t\treturn 2;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (argv[optind] == NULL) {\n+\t\tfprintf(stderr, \"need argument\\n\");\n+\t\tprint_help();\n+\t\treturn 2;\n+\t}\n+\n+\targ = argv[optind];\n+\n+\tif (opt_dump_table) {\n+\t\terr = dump_table(arg);\n+\t} else if (opt_dump_stack) {\n+\t\terr = dump_stack(arg);\n+\t} else if (opt_compact) {\n+\t\terr = compact_stack(arg);\n+\t}\n+\n+\tif (err < 0) {\n+\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv[0], arg,\n+\t\t\treftable_error_str(err));\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\ndiff --git a/reftable/merged.c b/reftable/merged.c\nnew file mode 100644\nindex 00000000000..ed02625bb54\n--- /dev/null\n+++ b/reftable/merged.c\n@@ -0,0 +1,366 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"constants.h\"\n+#include \"iter.h\"\n+#include \"pq.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-error.h\"\n+#include \"system.h\"\n+\n+static int merged_iter_init(struct merged_iter *mi)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\t\tint err = iterator_next(&mi->stack[i], &rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t\t\treftable_record_destroy(&rec);\n+\t\t} else {\n+\t\t\tstruct pq_entry e = {\n+\t\t\t\t.rec = rec,\n+\t\t\t\t.index = i,\n+\t\t\t};\n+\t\t\tmerged_iter_pqueue_add(&mi->pq, e);\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void merged_iter_close(void *p)\n+{\n+\tstruct merged_iter *mi = (struct merged_iter *)p;\n+\tint i = 0;\n+\tmerged_iter_pqueue_release(&mi->pq);\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t}\n+\treftable_free(mi->stack);\n+}\n+\n+static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,\n+\t\t\t\t\t size_t idx)\n+{\n+\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\tstruct pq_entry e = {\n+\t\t.rec = rec,\n+\t\t.index = idx,\n+\t};\n+\tint err = iterator_next(&mi->stack[idx], &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (err > 0) {\n+\t\treftable_iterator_destroy(&mi->stack[idx]);\n+\t\treftable_record_destroy(&rec);\n+\t\treturn 0;\n+\t}\n+\n+\tmerged_iter_pqueue_add(&mi->pq, e);\n+\treturn 0;\n+}\n+\n+static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)\n+{\n+\tif (iterator_is_null(&mi->stack[idx]))\n+\t\treturn 0;\n+\treturn merged_iter_advance_nonnull_subiter(mi, idx);\n+}\n+\n+static int merged_iter_next_entry(struct merged_iter *mi,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct strbuf entry_key = STRBUF_INIT;\n+\tstruct pq_entry entry = { 0 };\n+\tint err = 0;\n+\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\tentry = merged_iter_pqueue_remove(&mi->pq);\n+\terr = merged_iter_advance_subiter(mi, entry.index);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\t/*\n+\t One can also use reftable as datacenter-local storage, where the ref\n+\t database is maintained in globally consistent database (eg.\n+\t CockroachDB or Spanner). In this scenario, replication delays together\n+\t with compaction may cause newer tables to contain older entries. In\n+\t such a deployment, the loop below must be changed to collect all\n+\t entries for the same key, and return new the newest one.\n+\t*/\n+\treftable_record_key(&entry.rec, &entry_key);\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq)) {\n+\t\tstruct pq_entry top = merged_iter_pqueue_top(mi->pq);\n+\t\tstruct strbuf k = STRBUF_INIT;\n+\t\tint err = 0, cmp = 0;\n+\n+\t\treftable_record_key(&top.rec, &k);\n+\n+\t\tcmp = strbuf_cmp(&k, &entry_key);\n+\t\tstrbuf_release(&k);\n+\n+\t\tif (cmp > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\t\terr = merged_iter_advance_subiter(mi, top.index);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_record_destroy(&top.rec);\n+\t}\n+\n+\treftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));\n+\treftable_record_destroy(&entry.rec);\n+\tstrbuf_release(&entry_key);\n+\treturn 0;\n+}\n+\n+static int merged_iter_next(struct merged_iter *mi, struct reftable_record *rec)\n+{\n+\twhile (1) {\n+\t\tint err = merged_iter_next_entry(mi, rec);\n+\t\tif (err == 0 && mi->suppress_deletions &&\n+\t\t reftable_record_is_deletion(rec)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\treturn err;\n+\t}\n+}\n+\n+static int merged_iter_next_void(void *p, struct reftable_record *rec)\n+{\n+\tstruct merged_iter *mi = (struct merged_iter *)p;\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\treturn merged_iter_next(mi, rec);\n+}\n+\n+static struct reftable_iterator_vtable merged_iter_vtable = {\n+\t.next = &merged_iter_next_void,\n+\t.close = &merged_iter_close,\n+};\n+\n+static void iterator_from_merged_iter(struct reftable_iterator *it,\n+\t\t\t\t struct merged_iter *mi)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = mi;\n+\tit->ops = &merged_iter_vtable;\n+}\n+\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id)\n+{\n+\tstruct reftable_merged_table *m = NULL;\n+\tuint64_t last_max = 0;\n+\tuint64_t first_min = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t min = reftable_table_min_update_index(&stack[i]);\n+\t\tuint64_t max = reftable_table_max_update_index(&stack[i]);\n+\n+\t\tif (reftable_table_hash_id(&stack[i]) != hash_id) {\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\t\t}\n+\t\tif (i == 0 || min < first_min) {\n+\t\t\tfirst_min = min;\n+\t\t}\n+\t\tif (i == 0 || max > last_max) {\n+\t\t\tlast_max = max;\n+\t\t}\n+\t}\n+\n+\tm = (struct reftable_merged_table *)reftable_calloc(\n+\t\tsizeof(struct reftable_merged_table));\n+\tm->stack = stack;\n+\tm->stack_len = n;\n+\tm->min = first_min;\n+\tm->max = last_max;\n+\tm->hash_id = hash_id;\n+\t*dest = m;\n+\treturn 0;\n+}\n+\n+/* clears the list of subtable, without affecting the readers themselves. */\n+void merged_table_release(struct reftable_merged_table *mt)\n+{\n+\tFREE_AND_NULL(mt->stack);\n+\tmt->stack_len = 0;\n+}\n+\n+void reftable_merged_table_free(struct reftable_merged_table *mt)\n+{\n+\tif (mt == NULL) {\n+\t\treturn;\n+\t}\n+\tmerged_table_release(mt);\n+\treftable_free(mt);\n+}\n+\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->max;\n+}\n+\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->min;\n+}\n+\n+static int reftable_table_seek_record(struct reftable_table *tab,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn tab->ops->seek_record(tab->table_arg, it, rec);\n+}\n+\n+static int merged_table_seek_record(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_iterator *iters = reftable_calloc(\n+\t\tsizeof(struct reftable_iterator) * mt->stack_len);\n+\tstruct merged_iter merged = {\n+\t\t.stack = iters,\n+\t\t.typ = reftable_record_type(rec),\n+\t\t.hash_id = mt->hash_id,\n+\t\t.suppress_deletions = mt->suppress_deletions,\n+\t};\n+\tint n = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < mt->stack_len && err == 0; i++) {\n+\t\tint e = reftable_table_seek_record(&mt->stack[i], &iters[n],\n+\t\t\t\t\t\t rec);\n+\t\tif (e < 0) {\n+\t\t\terr = e;\n+\t\t}\n+\t\tif (e == 0) {\n+\t\t\tn++;\n+\t\t}\n+\t}\n+\tif (err < 0) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < n; i++) {\n+\t\t\treftable_iterator_destroy(&iters[i]);\n+\t\t}\n+\t\treftable_free(iters);\n+\t\treturn err;\n+\t}\n+\n+\tmerged.stack_len = n;\n+\terr = merged_iter_init(&merged);\n+\tif (err < 0) {\n+\t\tmerged_iter_close(&merged);\n+\t\treturn err;\n+\t} else {\n+\t\tstruct merged_iter *p =\n+\t\t\treftable_malloc(sizeof(struct merged_iter));\n+\t\t*p = merged;\n+\t\titerator_from_merged_iter(it, p);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_merged_table_seek_log_at(mt, it, name, max);\n+}\n+\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\n+{\n+\treturn mt->hash_id;\n+}\n+\n+static int reftable_merged_table_seek_void(void *tab,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn merged_table_seek_record((struct reftable_merged_table *)tab, it,\n+\t\t\t\t\trec);\n+}\n+\n+static uint32_t reftable_merged_table_hash_id_void(void *tab)\n+{\n+\treturn reftable_merged_table_hash_id(\n+\t\t(struct reftable_merged_table *)tab);\n+}\n+\n+static uint64_t reftable_merged_table_min_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_min_update_index(\n+\t\t(struct reftable_merged_table *)tab);\n+}\n+\n+static uint64_t reftable_merged_table_max_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_max_update_index(\n+\t\t(struct reftable_merged_table *)tab);\n+}\n+\n+static struct reftable_table_vtable merged_table_vtable = {\n+\t.seek_record = reftable_merged_table_seek_void,\n+\t.hash_id = reftable_merged_table_hash_id_void,\n+\t.min_update_index = reftable_merged_table_min_update_index_void,\n+\t.max_update_index = reftable_merged_table_max_update_index_void,\n+};\n+\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *merged)\n+{\n+\tassert(tab->ops == NULL);\n+\ttab->ops = &merged_table_vtable;\n+\ttab->table_arg = merged;\n+}\ndiff --git a/reftable/merged.h b/reftable/merged.h\nnew file mode 100644\nindex 00000000000..8c4d4d58d77\n--- /dev/null\n+++ b/reftable/merged.h\n@@ -0,0 +1,35 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef MERGED_H\n+#define MERGED_H\n+\n+#include \"pq.h\"\n+\n+struct reftable_merged_table {\n+\tstruct reftable_table *stack;\n+\tsize_t stack_len;\n+\tuint32_t hash_id;\n+\tint suppress_deletions;\n+\n+\tuint64_t min;\n+\tuint64_t max;\n+};\n+\n+struct merged_iter {\n+\tstruct reftable_iterator *stack;\n+\tuint32_t hash_id;\n+\tsize_t stack_len;\n+\tuint8_t typ;\n+\tint suppress_deletions;\n+\tstruct merged_iter_pqueue pq;\n+};\n+\n+void merged_table_release(struct reftable_merged_table *mt);\n+\n+#endif\ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nnew file mode 100644\nindex 00000000000..09a70aa0fda\n--- /dev/null\n+++ b/reftable/merged_test.c\n@@ -0,0 +1,343 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"pq.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-generic.h\"\n+#include \"reftable-stack.h\"\n+\n+static void test_pq(void)\n+{\n+\tchar *names[54] = { NULL };\n+\tint N = ARRAY_SIZE(names) - 1;\n+\n+\tstruct merged_iter_pqueue pq = { NULL };\n+\tconst char *last = NULL;\n+\n+\tint i = 0;\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tsnprintf(name, sizeof(name), \"%02d\", i);\n+\t\tnames[i] = xstrdup(name);\n+\t}\n+\n+\ti = 1;\n+\tdo {\n+\t\tstruct reftable_record rec =\n+\t\t\treftable_new_record(BLOCK_TYPE_REF);\n+\t\tstruct pq_entry e = { 0 };\n+\n+\t\treftable_record_as_ref(&rec)->refname = names[i];\n+\t\te.rec = rec;\n+\t\tmerged_iter_pqueue_add(&pq, e);\n+\t\tmerged_iter_pqueue_check(pq);\n+\t\ti = (i * 7) % N;\n+\t} while (i != 1);\n+\n+\twhile (!merged_iter_pqueue_is_empty(pq)) {\n+\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct reftable_ref_record *ref =\n+\t\t\treftable_record_as_ref(&e.rec);\n+\n+\t\tmerged_iter_pqueue_check(pq);\n+\n+\t\tif (last != NULL) {\n+\t\t\tassert(strcmp(last, ref->refname) < 0);\n+\t\t}\n+\t\tlast = ref->refname;\n+\t\tref->refname = NULL;\n+\t\treftable_free(ref);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\n+\tmerged_iter_pqueue_release(&pq);\n+}\n+\n+static void write_test_table(struct strbuf *buf,\n+\t\t\t struct reftable_ref_record refs[], int n)\n+{\n+\tint min = 0xffffffff;\n+\tint max = 0;\n+\tint i = 0;\n+\tint err;\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_writer *w = NULL;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t ui = refs[i].update_index;\n+\t\tif (ui > max) {\n+\t\t\tmax = ui;\n+\t\t}\n+\t\tif (ui < min) {\n+\t\t\tmin = ui;\n+\t\t}\n+\t}\n+\n+\tw = reftable_new_writer(&strbuf_add_void, buf, &opts);\n+\treftable_writer_set_limits(w, min, max);\n+\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t before = refs[i].update_index;\n+\t\tint n = reftable_writer_add_ref(w, &refs[i]);\n+\t\tassert(n == 0);\n+\t\tassert(before == refs[i].update_index);\n+\t}\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\n+\treftable_writer_free(w);\n+}\n+\n+static struct reftable_merged_table *\n+merged_table_from_records(struct reftable_ref_record **refs,\n+\t\t\t struct reftable_block_source **source,\n+\t\t\t struct reftable_reader ***readers, int *sizes,\n+\t\t\t struct strbuf *buf, int n)\n+{\n+\tint i = 0;\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err;\n+\tstruct reftable_table *tabs =\n+\t\treftable_calloc(n * sizeof(struct reftable_table));\n+\t*readers = reftable_calloc(n * sizeof(struct reftable_reader *));\n+\t*source = reftable_calloc(n * sizeof(**source));\n+\tfor (i = 0; i < n; i++) {\n+\t\twrite_test_table(&buf[i], refs[i], sizes[i]);\n+\t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n+\n+\t\terr = reftable_new_reader(&(*readers)[i], &(*source)[i],\n+\t\t\t\t\t \"name\");\n+\t\tEXPECT_ERR(err);\n+\t\treftable_table_from_reader(&tabs[i], (*readers)[i]);\n+\t}\n+\n+\terr = reftable_new_merged_table(&mt, tabs, n, SHA1_ID);\n+\tEXPECT_ERR(err);\n+\treturn mt;\n+}\n+\n+static void readers_destroy(struct reftable_reader **readers, size_t n)\n+{\n+\tint i = 0;\n+\tfor (; i < n; i++)\n+\t\treftable_reader_free(readers[i]);\n+\treftable_free(readers);\n+}\n+\n+static void test_merged_between(void)\n+{\n+\tuint8_t hash1[SHA1_SIZE] = { 1, 2, 3, 0 };\n+\n+\tstruct reftable_ref_record r1[] = { {\n+\t\t.refname = \"b\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t.value.val1 = hash1,\n+\t} };\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_DELETION,\n+\t} };\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2 };\n+\tint sizes[] = { 1, 1 };\n+\tstruct strbuf bufs[2] = { STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 2);\n+\tint i;\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT_ERR(err);\n+\tEXPECT(ref.update_index == 2);\n+\treftable_ref_record_release(&ref);\n+\treftable_iterator_destroy(&it);\n+\treaders_destroy(readers, 2);\n+\treftable_merged_table_free(mt);\n+\tfor (i = 0; i < ARRAY_SIZE(bufs); i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treftable_free(bs);\n+}\n+\n+static void test_merged(void)\n+{\n+\tuint8_t hash1[SHA1_SIZE] = { 1 };\n+\tuint8_t hash2[SHA1_SIZE] = { 2 };\n+\tstruct reftable_ref_record r1[] = {\n+\t\t{\n+\t\t\t.refname = \"a\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"b\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t}\n+\t};\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_DELETION,\n+\t} };\n+\tstruct reftable_ref_record r3[] = {\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash2,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"d\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t};\n+\n+\tstruct reftable_ref_record want[] = {\n+\t\tr2[0],\n+\t\tr1[1],\n+\t\tr3[0],\n+\t\tr3[1],\n+\t};\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2, r3 };\n+\tint sizes[3] = { 3, 1, 2 };\n+\tstruct strbuf bufs[3] = { STRBUF_INIT, STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 3);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tstruct reftable_ref_record *out = NULL;\n+\tsize_t len = 0;\n+\tsize_t cap = 0;\n+\tint i = 0;\n+\n+\tEXPECT_ERR(err);\n+\twhile (len < 100) { /* cap loops/recursion. */\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tout = reftable_realloc(\n+\t\t\t\tout, sizeof(struct reftable_ref_record) * cap);\n+\t\t}\n+\t\tout[len++] = ref;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\tassert(ARRAY_SIZE(want) == len);\n+\tfor (i = 0; i < len; i++) {\n+\t\tassert(reftable_ref_record_equal(&want[i], &out[i], SHA1_SIZE));\n+\t}\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_release(&out[i]);\n+\t}\n+\treftable_free(out);\n+\n+\tfor (i = 0; i < 3; i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treaders_destroy(readers, 3);\n+\treftable_merged_table_free(mt);\n+\treftable_free(bs);\n+}\n+\n+static void test_default_write_opts(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"master\",\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_table *tab = reftable_calloc(sizeof(*tab) * 1);\n+\tuint32_t hash_id;\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_merged_table *merged = NULL;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\thash_id = reftable_reader_hash_id(rd);\n+\tassert(hash_id == SHA1_ID);\n+\n+\treftable_table_from_reader(&tab[0], rd);\n+\terr = reftable_new_merged_table(&merged, tab, 1, SHA1_ID);\n+\tEXPECT_ERR(err);\n+\n+\treftable_reader_free(rd);\n+\treftable_merged_table_free(merged);\n+\tstrbuf_release(&buf);\n+}\n+\n+/* XXX test refs_for(oid) */\n+\n+int merged_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_merged_between);\n+\tRUN_TEST(test_pq);\n+\tRUN_TEST(test_merged);\n+\tRUN_TEST(test_default_write_opts);\n+\treturn 0;\n+}\ndiff --git a/reftable/pq.c b/reftable/pq.c\nnew file mode 100644\nindex 00000000000..8918d158e2d\n--- /dev/null\n+++ b/reftable/pq.c\n@@ -0,0 +1,115 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"pq.h\"\n+\n+#include \"reftable-record.h\"\n+#include \"system.h\"\n+#include \"basics.h\"\n+\n+static int pq_less(struct pq_entry a, struct pq_entry b)\n+{\n+\tstruct strbuf ak = STRBUF_INIT;\n+\tstruct strbuf bk = STRBUF_INIT;\n+\tint cmp = 0;\n+\treftable_record_key(&a.rec, &ak);\n+\treftable_record_key(&b.rec, &bk);\n+\n+\tcmp = strbuf_cmp(&ak, &bk);\n+\n+\tstrbuf_release(&ak);\n+\tstrbuf_release(&bk);\n+\n+\tif (cmp == 0)\n+\t\treturn a.index > b.index;\n+\n+\treturn cmp < 0;\n+}\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.heap[0];\n+}\n+\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.len == 0;\n+}\n+\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq)\n+{\n+\tint i = 0;\n+\tfor (i = 1; i < pq.len; i++) {\n+\t\tint parent = (i - 1) / 2;\n+\n+\t\tassert(pq_less(pq.heap[parent], pq.heap[i]));\n+\t}\n+}\n+\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tstruct pq_entry e = pq->heap[0];\n+\tpq->heap[0] = pq->heap[pq->len - 1];\n+\tpq->len--;\n+\n+\ti = 0;\n+\twhile (i < pq->len) {\n+\t\tint min = i;\n+\t\tint j = 2 * i + 1;\n+\t\tint k = 2 * i + 2;\n+\t\tif (j < pq->len && pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tmin = j;\n+\t\t}\n+\t\tif (k < pq->len && pq_less(pq->heap[k], pq->heap[min])) {\n+\t\t\tmin = k;\n+\t\t}\n+\n+\t\tif (min == i) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[i], pq->heap[min]);\n+\t\ti = min;\n+\t}\n+\n+\treturn e;\n+}\n+\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)\n+{\n+\tint i = 0;\n+\tif (pq->len == pq->cap) {\n+\t\tpq->cap = 2 * pq->cap + 1;\n+\t\tpq->heap = reftable_realloc(pq->heap,\n+\t\t\t\t\t pq->cap * sizeof(struct pq_entry));\n+\t}\n+\n+\tpq->heap[pq->len++] = e;\n+\ti = pq->len - 1;\n+\twhile (i > 0) {\n+\t\tint j = (i - 1) / 2;\n+\t\tif (pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[j], pq->heap[i]);\n+\n+\t\ti = j;\n+\t}\n+}\n+\n+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < pq->len; i++) {\n+\t\treftable_record_destroy(&pq->heap[i].rec);\n+\t}\n+\tFREE_AND_NULL(pq->heap);\n+\tpq->len = pq->cap = 0;\n+}\ndiff --git a/reftable/pq.h b/reftable/pq.h\nnew file mode 100644\nindex 00000000000..385d2fb139a\n--- /dev/null\n+++ b/reftable/pq.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef PQ_H\n+#define PQ_H\n+\n+#include \"record.h\"\n+\n+struct pq_entry {\n+\tint index;\n+\tstruct reftable_record rec;\n+};\n+\n+struct merged_iter_pqueue {\n+\tstruct pq_entry *heap;\n+\tsize_t len;\n+\tsize_t cap;\n+};\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq);\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq);\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq);\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e);\n+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n+\n+#endif\ndiff --git a/reftable/refname.c b/reftable/refname.c\nnew file mode 100644\nindex 00000000000..0f4eb3b292d\n--- /dev/null\n+++ b/reftable/refname.c\n@@ -0,0 +1,209 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"reftable-error.h\"\n+#include \"basics.h\"\n+#include \"refname.h\"\n+#include \"reftable-iterator.h\"\n+\n+struct find_arg {\n+\tchar **names;\n+\tconst char *want;\n+};\n+\n+static int find_name(size_t k, void *arg)\n+{\n+\tstruct find_arg *f_arg = (struct find_arg *)arg;\n+\treturn strcmp(f_arg->names[k], f_arg->want) >= 0;\n+}\n+\n+static int modification_has_ref(struct modification *mod, const char *name)\n+{\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len && !strcmp(mod->add[idx], name)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tif (mod->del_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->del,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\tif (idx < mod->del_len && !strcmp(mod->del[idx], name)) {\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\terr = reftable_table_read_ref(&mod->tab, name, &ref);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+static void modification_release(struct modification *mod)\n+{\n+\t/* don't delete the strings themselves; they're owned by ref records.\n+\t */\n+\tFREE_AND_NULL(mod->add);\n+\tFREE_AND_NULL(mod->del);\n+\tmod->add_len = 0;\n+\tmod->del_len = 0;\n+}\n+\n+static int modification_has_ref_with_prefix(struct modification *mod,\n+\t\t\t\t\t const char *prefix)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = prefix,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len &&\n+\t\t !strncmp(prefix, mod->add[idx], strlen(prefix)))\n+\t\t\tgoto done;\n+\t}\n+\terr = reftable_table_seek_ref(&mod->tab, &it, prefix);\n+\tif (err)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\n+\t\tif (mod->del_len > 0) {\n+\t\t\tstruct find_arg arg = {\n+\t\t\t\t.names = mod->del,\n+\t\t\t\t.want = ref.refname,\n+\t\t\t};\n+\t\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\t\tif (idx < mod->del_len &&\n+\t\t\t !strcmp(ref.refname, mod->del[idx])) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (strncmp(ref.refname, prefix, strlen(prefix))) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_ref_record_release(&ref);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int validate_refname(const char *name)\n+{\n+\twhile (1) {\n+\t\tchar *next = strchr(name, '/');\n+\t\tif (!*name) {\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\t}\n+\t\tif (!next) {\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (next - name == 0 || (next - name == 1 && *name == '.') ||\n+\t\t (next - name == 2 && name[0] == '.' && name[1] == '.'))\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\tname = next + 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz)\n+{\n+\tstruct modification mod = {\n+\t\t.tab = tab,\n+\t\t.add = reftable_calloc(sizeof(char *) * sz),\n+\t\t.del = reftable_calloc(sizeof(char *) * sz),\n+\t};\n+\tint i = 0;\n+\tint err = 0;\n+\tfor (; i < sz; i++) {\n+\t\tif (reftable_ref_record_is_deletion(&recs[i])) {\n+\t\t\tmod.del[mod.del_len++] = recs[i].refname;\n+\t\t} else {\n+\t\t\tmod.add[mod.add_len++] = recs[i].refname;\n+\t\t}\n+\t}\n+\n+\terr = modification_validate(&mod);\n+\tmodification_release(&mod);\n+\treturn err;\n+}\n+\n+static void strbuf_trim_component(struct strbuf *sl)\n+{\n+\twhile (sl->len > 0) {\n+\t\tint is_slash = (sl->buf[sl->len - 1] == '/');\n+\t\tstrbuf_setlen(sl, sl->len - 1);\n+\t\tif (is_slash)\n+\t\t\tbreak;\n+\t}\n+}\n+\n+int modification_validate(struct modification *mod)\n+{\n+\tstruct strbuf slashed = STRBUF_INIT;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (; i < mod->add_len; i++) {\n+\t\terr = validate_refname(mod->add[i]);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\tstrbuf_addstr(&slashed, \"/\");\n+\n+\t\terr = modification_has_ref_with_prefix(mod, slashed.buf);\n+\t\tif (err == 0) {\n+\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\twhile (slashed.len) {\n+\t\t\tstrbuf_trim_component(&slashed);\n+\t\t\terr = modification_has_ref(mod, slashed.buf);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\terr = 0;\n+done:\n+\tstrbuf_release(&slashed);\n+\treturn err;\n+}\ndiff --git a/reftable/refname.h b/reftable/refname.h\nnew file mode 100644\nindex 00000000000..a24b40fcb42\n--- /dev/null\n+++ b/reftable/refname.h\n@@ -0,0 +1,29 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+#ifndef REFNAME_H\n+#define REFNAME_H\n+\n+#include \"reftable-record.h\"\n+#include \"reftable-generic.h\"\n+\n+struct modification {\n+\tstruct reftable_table tab;\n+\n+\tchar **add;\n+\tsize_t add_len;\n+\n+\tchar **del;\n+\tsize_t del_len;\n+};\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz);\n+\n+int modification_validate(struct modification *mod);\n+\n+#endif\ndiff --git a/reftable/refname_test.c b/reftable/refname_test.c\nnew file mode 100644\nindex 00000000000..5e005d6af31\n--- /dev/null\n+++ b/reftable/refname_test.c\n@@ -0,0 +1,102 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"refname.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-writer.h\"\n+#include \"system.h\"\n+\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct testcase {\n+\tchar *add;\n+\tchar *del;\n+\tint error_code;\n+};\n+\n+static void test_conflict(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"a/b\",\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"destination\", /* make sure it's not a symref.\n+\t\t\t\t\t\t*/\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tint i;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct testcase cases[] = {\n+\t\t{ \"a/b/c\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"b\", NULL, 0 },\n+\t\t{ \"a\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"a\", \"a/b\", 0 },\n+\n+\t\t{ \"p/\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p//q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/./q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/../q\", NULL, REFTABLE_REFNAME_ERROR },\n+\n+\t\t{ \"a/b/c\", \"a/b\", 0 },\n+\t\t{ NULL, \"a//b\", 0 },\n+\t};\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\treftable_table_from_reader(&tab, rd);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct modification mod = {\n+\t\t\t.tab = tab,\n+\t\t};\n+\n+\t\tif (cases[i].add != NULL) {\n+\t\t\tmod.add = &cases[i].add;\n+\t\t\tmod.add_len = 1;\n+\t\t}\n+\t\tif (cases[i].del != NULL) {\n+\t\t\tmod.del = &cases[i].del;\n+\t\t\tmod.del_len = 1;\n+\t\t}\n+\n+\t\terr = modification_validate(&mod);\n+\t\tEXPECT(err == cases[i].error_code);\n+\t}\n+\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+int refname_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_conflict);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-generic.h b/reftable/reftable-generic.h\nnew file mode 100644\nindex 00000000000..77eca3b4eb0\n--- /dev/null\n+++ b/reftable/reftable-generic.h\n@@ -0,0 +1,48 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_GENERIC_H\n+#define REFTABLE_GENERIC_H\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+#include \"reftable-merged.h\"\n+\n+/*\n+ * Provides a unified API for reading tables, either merged tables, or single\n+ * readers. */\n+struct reftable_table {\n+\tstruct reftable_table_vtable *ops;\n+\tvoid *table_arg;\n+};\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader);\n+\n+/* returns the hash ID from a generic reftable_table */\n+uint32_t reftable_table_hash_id(struct reftable_table *tab);\n+\n+/* create a generic table from reftable_merged_table */\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *table);\n+\n+/* returns the max update_index covered by this table. */\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab);\n+\n+/* returns the min update_index covered by this table. */\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0\n+ for success, and 1 if ref not found. */\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+#endif\ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nnew file mode 100644\nindex 00000000000..0e8ebe5a995\n--- /dev/null\n+++ b/reftable/reftable-merged.h\n@@ -0,0 +1,68 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_MERGED_H\n+#define REFTABLE_MERGED_H\n+\n+#include \"reftable-iterator.h\"\n+\n+/*\n+ * Merged tables\n+ *\n+ * A ref database kept in a sequence of table files. The merged_table presents a\n+ * unified view to reading (seeking, iterating) a sequence of immutable tables.\n+ *\n+ * The merged tables are on purpose kept disconnected from their actual storage\n+ * (eg. files on disk), because it is useful to merge tables aren't files. For\n+ * example, the per-workspace and global ref namespace can be implemented as a\n+ * merged table of two stacks of file-backed reftables.\n+ */\n+\n+/* A merged table is implements seeking/iterating over a stack of tables. */\n+struct reftable_merged_table;\n+\n+/* A generic reftable; see below. */\n+struct reftable_table;\n+\n+/* reftable_new_merged_table creates a new merged table. It takes ownership of\n+ the stack array.\n+*/\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id);\n+\n+/* returns an iterator positioned just before 'name' */\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns an iterator for log entry, at given update_index */\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index);\n+\n+/* like reftable_merged_table_seek_log_at but look for the newest entry. */\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns the max update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt);\n+\n+/* returns the min update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt);\n+\n+/* releases memory for the merged_table */\n+void reftable_merged_table_free(struct reftable_merged_table *m);\n+\n+/* return the hash ID of the merged table. */\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m);\n+\n+#endif\ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nnew file mode 100644\nindex 00000000000..b7060f111e8\n--- /dev/null\n+++ b/reftable/reftable-stack.h\n@@ -0,0 +1,120 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_STACK_H\n+#define REFTABLE_STACK_H\n+\n+#include \"reftable-writer.h\"\n+\n+/*\n+ * The stack presents an interface to a mutable sequence of reftables.\n+\n+ * A stack can be mutated by pushing a table to the top of the stack.\n+\n+ * The reftable_stack automatically compacts files on disk to ensure good\n+ * amortized performance.\n+ *\n+ * For windows and other platforms that cannot have open files as rename\n+ * destinations, concurrent access from multiple processes needs the rand()\n+ * random seed to be randomized.\n+ */\n+struct reftable_stack;\n+\n+/* open a new reftable stack. The tables along with the table list will be\n+ * stored in 'dir'. Typically, this should be .git/reftables.\n+ */\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config);\n+\n+/* returns the update_index at which a next table should be written. */\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\n+\n+/* holds a transaction to add tables at the top of a stack. */\n+struct reftable_addition;\n+\n+/*\n+ * returns a new transaction to add reftables to the given stack. As a side\n+ * effect, the ref database is locked.\n+ */\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st);\n+\n+/* Adds a reftable to transaction. */\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+\n+/* Commits the transaction, releasing the lock. */\n+int reftable_addition_commit(struct reftable_addition *add);\n+\n+/* Release all non-committed data from the transaction, and deallocate the\n+ * transaction. Releases the lock if held. */\n+void reftable_addition_destroy(struct reftable_addition *add);\n+\n+/* add a new table to the stack. The write_table function must call\n+ * reftable_writer_set_limits, add refs and return an error value. */\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *write_arg),\n+\t\t void *write_arg);\n+\n+/* returns the merged_table for seeking. This table is valid until the\n+ * next write or reload, and should not be closed or deleted.\n+ */\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st);\n+\n+/* frees all resources associated with the stack. */\n+void reftable_stack_destroy(struct reftable_stack *st);\n+\n+/* Reloads the stack if necessary. This is very cheap to run if the stack was up\n+ * to date */\n+int reftable_stack_reload(struct reftable_stack *st);\n+\n+/* Policy for expiring reflog entries. */\n+struct reftable_log_expiry_config {\n+\t/* Drop entries older than this timestamp */\n+\tuint64_t time;\n+\n+\t/* Drop older entries */\n+\tuint64_t min_update_index;\n+};\n+\n+/* compacts all reftables into a giant table. Expire reflog entries if config is\n+ * non-NULL */\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config);\n+\n+/* heuristically compact unbalanced table stack. */\n+int reftable_stack_auto_compact(struct reftable_stack *st);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0 for\n+ * success, and 1 if ref not found. */\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* convenience function to read a single log. Returns < 0 for error, 0 for\n+ * success, and 1 if ref not found. */\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* statistics on past compactions. */\n+struct reftable_compaction_stats {\n+\tuint64_t bytes; /* total number of bytes written */\n+\tuint64_t entries_written; /* total number of entries written, including\n+\t\t\t\t failures. */\n+\tint attempts; /* how often we tried to compact */\n+\tint failures; /* failures happen on concurrent updates */\n+};\n+\n+/* return statistics for compaction up till now. */\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st);\n+\n+#endif\ndiff --git a/reftable/reftable.c b/reftable/reftable.c\nnew file mode 100644\nindex 00000000000..dc4fd03d5b2\n--- /dev/null\n+++ b/reftable/reftable.c\n@@ -0,0 +1,98 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"record.h\"\n+#include \"reader.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+static int reftable_reader_seek_void(void *tab, struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn reader_seek((struct reftable_reader *)tab, it, rec);\n+}\n+\n+static uint32_t reftable_reader_hash_id_void(void *tab)\n+{\n+\treturn reftable_reader_hash_id((struct reftable_reader *)tab);\n+}\n+\n+static uint64_t reftable_reader_min_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_min_update_index((struct reftable_reader *)tab);\n+}\n+\n+static uint64_t reftable_reader_max_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_max_update_index((struct reftable_reader *)tab);\n+}\n+\n+static struct reftable_table_vtable reader_vtable = {\n+\t.seek_record = reftable_reader_seek_void,\n+\t.hash_id = reftable_reader_hash_id_void,\n+\t.min_update_index = reftable_reader_min_update_index_void,\n+\t.max_update_index = reftable_reader_max_update_index_void,\n+};\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader)\n+{\n+\tassert(tab->ops == NULL);\n+\ttab->ops = &reader_vtable;\n+\ttab->table_arg = reader;\n+}\n+\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_table_seek_ref(tab, &it, name);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_ref(&it, ref);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref->refname, name) ||\n+\t reftable_ref_record_is_deletion(ref)) {\n+\t\treftable_ref_record_release(ref);\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->max_update_index(tab->table_arg);\n+}\n+\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->min_update_index(tab->table_arg);\n+}\n+\n+uint32_t reftable_table_hash_id(struct reftable_table *tab)\n+{\n+\treturn tab->ops->hash_id(tab->table_arg);\n+}\ndiff --git a/reftable/stack.c b/reftable/stack.c\nnew file mode 100644\nindex 00000000000..10608089347\n--- /dev/null\n+++ b/reftable/stack.c\n@@ -0,0 +1,1260 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+#include \"merged.h\"\n+#include \"reader.h\"\n+#include \"refname.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-record.h\"\n+#include \"writer.h\"\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config);\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name);\n+static void reftable_addition_close(struct reftable_addition *add);\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open);\n+\n+static int reftable_fd_write(void *arg, const void *data, size_t sz)\n+{\n+\tint *fdp = (int *)arg;\n+\treturn write(*fdp, data, sz);\n+}\n+\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config)\n+{\n+\tstruct reftable_stack *p =\n+\t\treftable_calloc(sizeof(struct reftable_stack));\n+\tstruct strbuf list_file_name = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tif (config.hash_id == 0) {\n+\t\tconfig.hash_id = SHA1_ID;\n+\t}\n+\n+\t*dest = NULL;\n+\n+\tstrbuf_reset(&list_file_name);\n+\tstrbuf_addstr(&list_file_name, dir);\n+\tstrbuf_addstr(&list_file_name, \"/tables.list\");\n+\n+\tp->list_file = strbuf_detach(&list_file_name, NULL);\n+\tp->reftable_dir = xstrdup(dir);\n+\tp->config = config;\n+\n+\terr = reftable_stack_reload_maybe_reuse(p, 1);\n+\tif (err < 0) {\n+\t\treftable_stack_destroy(p);\n+\t} else {\n+\t\t*dest = p;\n+\t}\n+\treturn err;\n+}\n+\n+static int fd_read_lines(int fd, char ***namesp)\n+{\n+\toff_t size = lseek(fd, 0, SEEK_END);\n+\tchar *buf = NULL;\n+\tint err = 0;\n+\tif (size < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\terr = lseek(fd, 0, SEEK_SET);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tbuf = reftable_malloc(size + 1);\n+\tif (read(fd, buf, size) != size) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\tbuf[size] = 0;\n+\n+\tparse_names(buf, size, namesp);\n+\n+done:\n+\treftable_free(buf);\n+\treturn err;\n+}\n+\n+int read_lines(const char *filename, char ***namesp)\n+{\n+\tint fd = open(filename, O_RDONLY, 0644);\n+\tint err = 0;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\t*namesp = reftable_calloc(sizeof(char *));\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\treturn REFTABLE_IO_ERROR;\n+\t}\n+\terr = fd_read_lines(fd, namesp);\n+\tclose(fd);\n+\treturn err;\n+}\n+\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st)\n+{\n+\treturn st->merged;\n+}\n+\n+/* Close and free the stack */\n+void reftable_stack_destroy(struct reftable_stack *st)\n+{\n+\tif (st->merged != NULL) {\n+\t\treftable_merged_table_free(st->merged);\n+\t\tst->merged = NULL;\n+\t}\n+\n+\tif (st->readers != NULL) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < st->readers_len; i++) {\n+\t\t\treftable_reader_free(st->readers[i]);\n+\t\t}\n+\t\tst->readers_len = 0;\n+\t\tFREE_AND_NULL(st->readers);\n+\t}\n+\tFREE_AND_NULL(st->list_file);\n+\tFREE_AND_NULL(st->reftable_dir);\n+\treftable_free(st);\n+}\n+\n+static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n+\t\t\t\t\t\t int cur_len)\n+{\n+\tstruct reftable_reader **cur =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * cur_len);\n+\tint i = 0;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tcur[i] = st->readers[i];\n+\t}\n+\treturn cur;\n+}\n+\n+static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n+\t\t\t\t int reuse_open)\n+{\n+\tint cur_len = st->merged == NULL ? 0 : st->merged->stack_len;\n+\tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n+\tint err = 0;\n+\tint names_len = names_length(names);\n+\tstruct reftable_reader **new_readers =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * names_len);\n+\tstruct reftable_table *new_tables =\n+\t\treftable_calloc(sizeof(struct reftable_table) * names_len);\n+\tint new_readers_len = 0;\n+\tstruct reftable_merged_table *new_merged = NULL;\n+\tint i;\n+\n+\twhile (*names) {\n+\t\tstruct reftable_reader *rd = NULL;\n+\t\tchar *name = *names++;\n+\n+\t\t/* this is linear; we assume compaction keeps the number of\n+\t\t tables under control so this is not quadratic. */\n+\t\tint j = 0;\n+\t\tfor (j = 0; reuse_open && j < cur_len; j++) {\n+\t\t\tif (cur[j] != NULL && 0 == strcmp(cur[j]->name, name)) {\n+\t\t\t\trd = cur[j];\n+\t\t\t\tcur[j] = NULL;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (rd == NULL) {\n+\t\t\tstruct reftable_block_source src = { NULL };\n+\t\t\tstruct strbuf table_path = STRBUF_INIT;\n+\t\t\tstrbuf_addstr(&table_path, st->reftable_dir);\n+\t\t\tstrbuf_addstr(&table_path, \"/\");\n+\t\t\tstrbuf_addstr(&table_path, name);\n+\n+\t\t\terr = reftable_block_source_from_file(&src,\n+\t\t\t\t\t\t\t table_path.buf);\n+\t\t\tstrbuf_release(&table_path);\n+\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\n+\t\t\terr = reftable_new_reader(&rd, &src, name);\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\n+\t\tnew_readers[new_readers_len] = rd;\n+\t\treftable_table_from_reader(&new_tables[new_readers_len], rd);\n+\t\tnew_readers_len++;\n+\t}\n+\n+\t/* success! */\n+\terr = reftable_new_merged_table(&new_merged, new_tables,\n+\t\t\t\t\tnew_readers_len, st->config.hash_id);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tnew_tables = NULL;\n+\tst->readers_len = new_readers_len;\n+\tif (st->merged != NULL) {\n+\t\tmerged_table_release(st->merged);\n+\t\treftable_merged_table_free(st->merged);\n+\t}\n+\tif (st->readers != NULL) {\n+\t\treftable_free(st->readers);\n+\t}\n+\tst->readers = new_readers;\n+\tnew_readers = NULL;\n+\tnew_readers_len = 0;\n+\n+\tnew_merged->suppress_deletions = 1;\n+\tst->merged = new_merged;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tif (cur[i] != NULL) {\n+\t\t\treader_close(cur[i]);\n+\t\t\treftable_reader_free(cur[i]);\n+\t\t}\n+\t}\n+\n+done:\n+\tfor (i = 0; i < new_readers_len; i++) {\n+\t\treader_close(new_readers[i]);\n+\t\treftable_reader_free(new_readers[i]);\n+\t}\n+\treftable_free(new_readers);\n+\treftable_free(new_tables);\n+\treftable_free(cur);\n+\treturn err;\n+}\n+\n+/* return negative if a before b. */\n+static int tv_cmp(struct timeval *a, struct timeval *b)\n+{\n+\ttime_t diff = a->tv_sec - b->tv_sec;\n+\tint udiff = a->tv_usec - b->tv_usec;\n+\n+\tif (diff != 0)\n+\t\treturn diff;\n+\n+\treturn udiff;\n+}\n+\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open)\n+{\n+\tstruct timeval deadline = { 0 };\n+\tint err = gettimeofday(&deadline, NULL);\n+\tint64_t delay = 0;\n+\tint tries = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tdeadline.tv_sec += 3;\n+\twhile (1) {\n+\t\tchar **names = NULL;\n+\t\tchar **names_after = NULL;\n+\t\tstruct timeval now = { 0 };\n+\t\tint err = gettimeofday(&now, NULL);\n+\t\tint err2 = 0;\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* Only look at deadlines after the first few times. This\n+\t\t simplifies debugging in GDB */\n+\t\ttries++;\n+\t\tif (tries > 3 && tv_cmp(&now, &deadline) >= 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = read_lines(st->list_file, &names);\n+\t\tif (err < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\t\terr = reftable_stack_reload_once(st, names, reuse_open);\n+\t\tif (err == 0) {\n+\t\t\tfree_names(names);\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err != REFTABLE_NOT_EXIST_ERROR) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* err == REFTABLE_NOT_EXIST_ERROR can be caused by a concurrent\n+\t\t writer. Check if there was one by checking if the name list\n+\t\t changed.\n+\t\t*/\n+\t\terr2 = read_lines(st->list_file, &names_after);\n+\t\tif (err2 < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err2;\n+\t\t}\n+\n+\t\tif (names_equal(names_after, names)) {\n+\t\t\tfree_names(names);\n+\t\t\tfree_names(names_after);\n+\t\t\treturn err;\n+\t\t}\n+\t\tfree_names(names);\n+\t\tfree_names(names_after);\n+\n+\t\tdelay = delay + (delay * rand()) / RAND_MAX + 1;\n+\t\tsleep_millisec(delay);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/* -1 = error\n+ 0 = up to date\n+ 1 = changed. */\n+static int stack_uptodate(struct reftable_stack *st)\n+{\n+\tchar **names = NULL;\n+\tint err = read_lines(st->list_file, &names);\n+\tint i = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tfor (i = 0; i < st->readers_len; i++) {\n+\t\tif (names[i] == NULL) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (strcmp(st->readers[i]->name, names[i])) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (names[st->merged->stack_len] != NULL) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tfree_names(names);\n+\treturn err;\n+}\n+\n+int reftable_stack_reload(struct reftable_stack *st)\n+{\n+\tint err = stack_uptodate(st);\n+\tif (err > 0)\n+\t\treturn reftable_stack_reload_maybe_reuse(st, 1);\n+\treturn err;\n+}\n+\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write)(struct reftable_writer *wr, void *arg),\n+\t\t void *arg)\n+{\n+\tint err = stack_try_add(st, write, arg);\n+\tif (err < 0) {\n+\t\tif (err == REFTABLE_LOCK_ERROR) {\n+\t\t\t/* Ignore error return, we want to propagate\n+\t\t\t REFTABLE_LOCK_ERROR.\n+\t\t\t*/\n+\t\t\treftable_stack_reload(st);\n+\t\t}\n+\t\treturn err;\n+\t}\n+\n+\tif (!st->disable_auto_compact)\n+\t\treturn reftable_stack_auto_compact(st);\n+\n+\treturn 0;\n+}\n+\n+static void format_name(struct strbuf *dest, uint64_t min, uint64_t max)\n+{\n+\tchar buf[100];\n+\tuint32_t rnd = (uint32_t)rand();\n+\tsnprintf(buf, sizeof(buf), \"0x%012\" PRIx64 \"-0x%012\" PRIx64 \"-%08x\",\n+\t\t min, max, rnd);\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, buf);\n+}\n+\n+struct reftable_addition {\n+\tint lock_file_fd;\n+\tstruct strbuf lock_file_name;\n+\tstruct reftable_stack *stack;\n+\n+\tchar **new_tables;\n+\tint new_tables_len;\n+\tuint64_t next_update_index;\n+};\n+\n+#define REFTABLE_ADDITION_INIT \\\n+\t{ \\\n+\t\t.lock_file_name = STRBUF_INIT \\\n+\t}\n+\n+static int reftable_stack_init_addition(struct reftable_addition *add,\n+\t\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tadd->stack = st;\n+\n+\tstrbuf_reset(&add->lock_file_name);\n+\tstrbuf_addstr(&add->lock_file_name, st->list_file);\n+\tstrbuf_addstr(&add->lock_file_name, \".lock\");\n+\n+\tadd->lock_file_fd = open(add->lock_file_name.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (add->lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = REFTABLE_LOCK_ERROR;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\terr = stack_uptodate(st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 1) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->next_update_index = reftable_stack_next_update_index(st);\n+done:\n+\tif (err) {\n+\t\treftable_addition_close(add);\n+\t}\n+\treturn err;\n+}\n+\n+static void reftable_addition_close(struct reftable_addition *add)\n+{\n+\tint i = 0;\n+\tstruct strbuf nm = STRBUF_INIT;\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstrbuf_reset(&nm);\n+\t\tstrbuf_addstr(&nm, add->stack->reftable_dir);\n+\t\tstrbuf_addstr(&nm, \"/\");\n+\t\tstrbuf_addstr(&nm, add->new_tables[i]);\n+\t\tunlink(nm.buf);\n+\t\treftable_free(add->new_tables[i]);\n+\t\tadd->new_tables[i] = NULL;\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\tif (add->lock_file_fd > 0) {\n+\t\tclose(add->lock_file_fd);\n+\t\tadd->lock_file_fd = 0;\n+\t}\n+\tif (add->lock_file_name.len > 0) {\n+\t\tunlink(add->lock_file_name.buf);\n+\t\tstrbuf_release(&add->lock_file_name);\n+\t}\n+\n+\tstrbuf_release(&nm);\n+}\n+\n+void reftable_addition_destroy(struct reftable_addition *add)\n+{\n+\tif (add == NULL) {\n+\t\treturn;\n+\t}\n+\treftable_addition_close(add);\n+\treftable_free(add);\n+}\n+\n+int reftable_addition_commit(struct reftable_addition *add)\n+{\n+\tstruct strbuf table_list = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\tif (add->new_tables_len == 0)\n+\t\tgoto done;\n+\n+\tfor (i = 0; i < add->stack->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->stack->readers[i]->name);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->new_tables[i]);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\n+\terr = write(add->lock_file_fd, table_list.buf, table_list.len);\n+\tstrbuf_release(&table_list);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = close(add->lock_file_fd);\n+\tadd->lock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(add->lock_file_name.buf, add->stack->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\t/* success, no more state to clean up. */\n+\tstrbuf_release(&add->lock_file_name);\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\treftable_free(add->new_tables[i]);\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\terr = reftable_stack_reload(add->stack);\n+done:\n+\treftable_addition_close(add);\n+\treturn err;\n+}\n+\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n+\t*dest = reftable_calloc(sizeof(**dest));\n+\t**dest = empty;\n+\terr = reftable_stack_init_addition(*dest, st);\n+\tif (err) {\n+\t\treftable_free(*dest);\n+\t\t*dest = NULL;\n+\t}\n+\treturn err;\n+}\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct reftable_addition add = REFTABLE_ADDITION_INIT;\n+\tint err = reftable_stack_init_addition(&add, st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tif (err > 0) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_addition_add(&add, write_table, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_addition_commit(&add);\n+done:\n+\treftable_addition_close(&add);\n+\treturn err;\n+}\n+\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf tab_file_name = STRBUF_INIT;\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\tint tab_fd = 0;\n+\n+\tstrbuf_reset(&next_name);\n+\tformat_name(&next_name, add->next_update_index, add->next_update_index);\n+\n+\tstrbuf_addstr(&temp_tab_file_name, add->stack->reftable_dir);\n+\tstrbuf_addstr(&temp_tab_file_name, \"/\");\n+\tstrbuf_addbuf(&temp_tab_file_name, &next_name);\n+\tstrbuf_addstr(&temp_tab_file_name, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab_file_name.buf);\n+\tif (tab_fd < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd,\n+\t\t\t\t &add->stack->config);\n+\terr = write_table(wr, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_writer_close(wr);\n+\tif (err == REFTABLE_EMPTY_TABLE_ERROR) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = stack_check_addition(add->stack, temp_tab_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (wr->min_update_index < add->next_update_index) {\n+\t\terr = REFTABLE_API_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tformat_name(&next_name, wr->min_update_index, wr->max_update_index);\n+\tstrbuf_addstr(&next_name, \".ref\");\n+\n+\tstrbuf_addstr(&tab_file_name, add->stack->reftable_dir);\n+\tstrbuf_addstr(&tab_file_name, \"/\");\n+\tstrbuf_addbuf(&tab_file_name, &next_name);\n+\n+\t/*\n+\t On windows, this relies on rand() picking a unique destination name.\n+\t Maybe we should do retry loop as well?\n+\t */\n+\terr = rename(temp_tab_file_name.buf, tab_file_name.buf);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->new_tables = reftable_realloc(add->new_tables,\n+\t\t\t\t\t sizeof(*add->new_tables) *\n+\t\t\t\t\t\t (add->new_tables_len + 1));\n+\tadd->new_tables[add->new_tables_len] = strbuf_detach(&next_name, NULL);\n+\tadd->new_tables_len++;\n+done:\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (temp_tab_file_name.len > 0) {\n+\t\tunlink(temp_tab_file_name.buf);\n+\t}\n+\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&tab_file_name);\n+\tstrbuf_release(&next_name);\n+\treftable_writer_free(wr);\n+\treturn err;\n+}\n+\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st)\n+{\n+\tint sz = st->merged->stack_len;\n+\tif (sz > 0)\n+\t\treturn reftable_reader_max_update_index(st->readers[sz - 1]) +\n+\t\t 1;\n+\treturn 1;\n+}\n+\n+static int stack_compact_locked(struct reftable_stack *st, int first, int last,\n+\t\t\t\tstruct strbuf *temp_tab,\n+\t\t\t\tstruct reftable_log_expiry_config *config)\n+{\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tint tab_fd = -1;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\n+\tformat_name(&next_name,\n+\t\t reftable_reader_min_update_index(st->readers[first]),\n+\t\t reftable_reader_max_update_index(st->readers[last]));\n+\n+\tstrbuf_reset(temp_tab);\n+\tstrbuf_addstr(temp_tab, st->reftable_dir);\n+\tstrbuf_addstr(temp_tab, \"/\");\n+\tstrbuf_addbuf(temp_tab, &next_name);\n+\tstrbuf_addstr(temp_tab, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab->buf);\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd, &st->config);\n+\n+\terr = stack_write_compact(st, wr, first, last, config);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = reftable_writer_close(wr);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\n+done:\n+\treftable_writer_free(wr);\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (err != 0 && temp_tab->len > 0) {\n+\t\tunlink(temp_tab->buf);\n+\t\tstrbuf_release(temp_tab);\n+\t}\n+\tstrbuf_release(&next_name);\n+\treturn err;\n+}\n+\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint subtabs_len = last - first + 1;\n+\tstruct reftable_table *subtabs = reftable_calloc(\n+\t\tsizeof(struct reftable_table) * (last - first + 1));\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tuint64_t entries = 0;\n+\n+\tint i = 0, j = 0;\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct reftable_reader *t = st->readers[i];\n+\t\treftable_table_from_reader(&subtabs[j++], t);\n+\t\tst->stats.bytes += t->size;\n+\t}\n+\treftable_writer_set_limits(wr, st->readers[first]->min_update_index,\n+\t\t\t\t st->readers[last]->max_update_index);\n+\n+\terr = reftable_new_merged_table(&mt, subtabs, subtabs_len,\n+\t\t\t\t\tst->config.hash_id);\n+\tif (err < 0) {\n+\t\treftable_free(subtabs);\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_merged_table_seek_ref(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (first == 0 && reftable_ref_record_is_deletion(&ref)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_ref(wr, &ref);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_merged_table_seek_log(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (first == 0 && reftable_log_record_is_deletion(&log)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config != NULL && config->time > 0 &&\n+\t\t log.time < config->time) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config != NULL && config->min_update_index > 0 &&\n+\t\t log.update_index < config->min_update_index) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_log(wr, &log);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\tif (mt != NULL) {\n+\t\tmerged_table_release(mt);\n+\t\treftable_merged_table_free(mt);\n+\t}\n+\treftable_ref_record_release(&ref);\n+\treftable_log_record_release(&log);\n+\tst->stats.entries_written += entries;\n+\treturn err;\n+}\n+\n+/* < 0: error. 0 == OK, > 0 attempt failed; could retry. */\n+static int stack_compact_range(struct reftable_stack *st, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *expiry)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf new_table_name = STRBUF_INIT;\n+\tstruct strbuf lock_file_name = STRBUF_INIT;\n+\tstruct strbuf ref_list_contents = STRBUF_INIT;\n+\tstruct strbuf new_table_path = STRBUF_INIT;\n+\tint err = 0;\n+\tint have_lock = 0;\n+\tint lock_file_fd = 0;\n+\tint compact_count = last - first + 1;\n+\tchar **listp = NULL;\n+\tchar **delete_on_success =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tchar **subtable_locks =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tint i = 0;\n+\tint j = 0;\n+\tint is_empty_table = 0;\n+\n+\tif (first > last || (expiry == NULL && first == last)) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\tst->stats.attempts++;\n+\n+\tstrbuf_reset(&lock_file_name);\n+\tstrbuf_addstr(&lock_file_name, st->list_file);\n+\tstrbuf_addstr(&lock_file_name, \".lock\");\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\t/* Don't want to write to the lock for now. */\n+\tclose(lock_file_fd);\n+\tlock_file_fd = 0;\n+\n+\thave_lock = 1;\n+\terr = stack_uptodate(st);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct strbuf subtab_file_name = STRBUF_INIT;\n+\t\tstruct strbuf subtab_lock = STRBUF_INIT;\n+\t\tint sublock_file_fd = -1;\n+\n+\t\tstrbuf_addstr(&subtab_file_name, st->reftable_dir);\n+\t\tstrbuf_addstr(&subtab_file_name, \"/\");\n+\t\tstrbuf_addstr(&subtab_file_name, reader_name(st->readers[i]));\n+\n+\t\tstrbuf_reset(&subtab_lock);\n+\t\tstrbuf_addbuf(&subtab_lock, &subtab_file_name);\n+\t\tstrbuf_addstr(&subtab_lock, \".lock\");\n+\n+\t\tsublock_file_fd = open(subtab_lock.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\t\tif (sublock_file_fd > 0) {\n+\t\t\tclose(sublock_file_fd);\n+\t\t} else if (sublock_file_fd < 0) {\n+\t\t\tif (errno == EEXIST) {\n+\t\t\t\terr = 1;\n+\t\t\t} else {\n+\t\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\t}\n+\t\t}\n+\n+\t\tsubtable_locks[j] = subtab_lock.buf;\n+\t\tdelete_on_success[j] = subtab_file_name.buf;\n+\t\tj++;\n+\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\terr = unlink(lock_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\thave_lock = 0;\n+\n+\terr = stack_compact_locked(st, first, last, &temp_tab_file_name,\n+\t\t\t\t expiry);\n+\t/* Compaction + tombstones can create an empty table out of non-empty\n+\t * tables. */\n+\tis_empty_table = (err == REFTABLE_EMPTY_TABLE_ERROR);\n+\tif (is_empty_table) {\n+\t\terr = 0;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\thave_lock = 1;\n+\n+\tformat_name(&new_table_name, st->readers[first]->min_update_index,\n+\t\t st->readers[last]->max_update_index);\n+\tstrbuf_addstr(&new_table_name, \".ref\");\n+\n+\tstrbuf_reset(&new_table_path);\n+\tstrbuf_addstr(&new_table_path, st->reftable_dir);\n+\tstrbuf_addstr(&new_table_path, \"/\");\n+\tstrbuf_addbuf(&new_table_path, &new_table_name);\n+\n+\tif (!is_empty_table) {\n+\t\t/* retry? */\n+\t\terr = rename(temp_tab_file_name.buf, new_table_path.buf);\n+\t\tif (err < 0) {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < first; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tif (!is_empty_table) {\n+\t\tstrbuf_addbuf(&ref_list_contents, &new_table_name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tfor (i = last + 1; i < st->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\n+\terr = write(lock_file_fd, ref_list_contents.buf, ref_list_contents.len);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\terr = close(lock_file_fd);\n+\tlock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(lock_file_name.buf, st->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\thave_lock = 0;\n+\n+\t/* Reload the stack before deleting. On windows, we can only delete the\n+\t files after we closed them.\n+\t*/\n+\terr = reftable_stack_reload_maybe_reuse(st, first < last);\n+\n+\tlistp = delete_on_success;\n+\twhile (*listp) {\n+\t\tif (strcmp(*listp, new_table_path.buf)) {\n+\t\t\tunlink(*listp);\n+\t\t}\n+\t\tlistp++;\n+\t}\n+\n+done:\n+\tfree_names(delete_on_success);\n+\n+\tlistp = subtable_locks;\n+\twhile (*listp) {\n+\t\tunlink(*listp);\n+\t\tlistp++;\n+\t}\n+\tfree_names(subtable_locks);\n+\tif (lock_file_fd > 0) {\n+\t\tclose(lock_file_fd);\n+\t\tlock_file_fd = 0;\n+\t}\n+\tif (have_lock) {\n+\t\tunlink(lock_file_name.buf);\n+\t}\n+\tstrbuf_release(&new_table_name);\n+\tstrbuf_release(&new_table_path);\n+\tstrbuf_release(&ref_list_contents);\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&lock_file_name);\n+\treturn err;\n+}\n+\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\treturn stack_compact_range(st, 0, st->merged->stack_len - 1, config);\n+}\n+\n+static int stack_compact_range_stats(struct reftable_stack *st, int first,\n+\t\t\t\t int last,\n+\t\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint err = stack_compact_range(st, first, last, config);\n+\tif (err > 0) {\n+\t\tst->stats.failures++;\n+\t}\n+\treturn err;\n+}\n+\n+static int segment_size(struct segment *s)\n+{\n+\treturn s->end - s->start;\n+}\n+\n+int fastlog2(uint64_t sz)\n+{\n+\tint l = 0;\n+\tif (sz == 0)\n+\t\treturn 0;\n+\tfor (; sz; sz /= 2) {\n+\t\tl++;\n+\t}\n+\treturn l - 1;\n+}\n+\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n+{\n+\tstruct segment *segs = reftable_calloc(sizeof(struct segment) * n);\n+\tint next = 0;\n+\tstruct segment cur = { 0 };\n+\tint i = 0;\n+\n+\tif (n == 0) {\n+\t\t*seglen = 0;\n+\t\treturn segs;\n+\t}\n+\tfor (i = 0; i < n; i++) {\n+\t\tint log = fastlog2(sizes[i]);\n+\t\tif (cur.log != log && cur.bytes > 0) {\n+\t\t\tstruct segment fresh = {\n+\t\t\t\t.start = i,\n+\t\t\t};\n+\n+\t\t\tsegs[next++] = cur;\n+\t\t\tcur = fresh;\n+\t\t}\n+\n+\t\tcur.log = log;\n+\t\tcur.end = i + 1;\n+\t\tcur.bytes += sizes[i];\n+\t}\n+\tsegs[next++] = cur;\n+\t*seglen = next;\n+\treturn segs;\n+}\n+\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, sizes, n);\n+\tstruct segment min_seg = {\n+\t\t.log = 64,\n+\t};\n+\tint i = 0;\n+\tfor (i = 0; i < seglen; i++) {\n+\t\tif (segment_size(&segs[i]) == 1) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (segs[i].log < min_seg.log) {\n+\t\t\tmin_seg = segs[i];\n+\t\t}\n+\t}\n+\n+\twhile (min_seg.start > 0) {\n+\t\tint prev = min_seg.start - 1;\n+\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmin_seg.start = prev;\n+\t\tmin_seg.bytes += sizes[prev];\n+\t}\n+\n+\treftable_free(segs);\n+\treturn min_seg;\n+}\n+\n+static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes =\n+\t\treftable_calloc(sizeof(uint64_t) * st->merged->stack_len);\n+\tint version = (st->config.hash_id == SHA1_ID) ? 1 : 2;\n+\tint overhead = header_size(version) - 1;\n+\tint i = 0;\n+\tfor (i = 0; i < st->merged->stack_len; i++) {\n+\t\tsizes[i] = st->readers[i]->size - overhead;\n+\t}\n+\treturn sizes;\n+}\n+\n+int reftable_stack_auto_compact(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n+\tstruct segment seg =\n+\t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n+\treftable_free(sizes);\n+\tif (segment_size(&seg) > 0)\n+\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n+\t\t\t\t\t\t NULL);\n+\n+\treturn 0;\n+}\n+\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st)\n+{\n+\treturn &st->stats;\n+}\n+\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_table tab = { NULL };\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\treturn reftable_table_read_ref(&tab, refname, ref);\n+}\n+\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(st);\n+\tint err = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_log(&it, log);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(log->refname, refname) ||\n+\t reftable_log_record_is_deletion(log)) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tif (err) {\n+\t\treftable_log_record_release(log);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name)\n+{\n+\tint err = 0;\n+\tstruct reftable_block_source src = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct reftable_ref_record *refs = NULL;\n+\tstruct reftable_iterator it = { NULL };\n+\tint cap = 0;\n+\tint len = 0;\n+\tint i = 0;\n+\n+\tif (st->config.skip_name_check)\n+\t\treturn 0;\n+\n+\terr = reftable_block_source_from_file(&src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&rd, &src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tif (err > 0) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (len >= cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\trefs = reftable_realloc(refs, cap * sizeof(refs[0]));\n+\t\t}\n+\n+\t\trefs[len++] = ref;\n+\t}\n+\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\n+\terr = validate_ref_record_addition(tab, refs, len);\n+\n+done:\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t}\n+\n+\tfree(refs);\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\treturn err;\n+}\ndiff --git a/reftable/stack.h b/reftable/stack.h\nnew file mode 100644\nindex 00000000000..f57005846e5\n--- /dev/null\n+++ b/reftable/stack.h\n@@ -0,0 +1,41 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef STACK_H\n+#define STACK_H\n+\n+#include \"system.h\"\n+#include \"reftable-writer.h\"\n+#include \"reftable-stack.h\"\n+\n+struct reftable_stack {\n+\tchar *list_file;\n+\tchar *reftable_dir;\n+\tint disable_auto_compact;\n+\n+\tstruct reftable_write_options config;\n+\n+\tstruct reftable_reader **readers;\n+\tsize_t readers_len;\n+\tstruct reftable_merged_table *merged;\n+\tstruct reftable_compaction_stats stats;\n+};\n+\n+int read_lines(const char *filename, char ***lines);\n+\n+struct segment {\n+\tint start, end;\n+\tint log;\n+\tuint64_t bytes;\n+};\n+\n+int fastlog2(uint64_t sz);\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n);\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n);\n+\n+#endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nnew file mode 100644\nindex 00000000000..f88a18e51fb\n--- /dev/null\n+++ b/reftable/stack_test.c\n@@ -0,0 +1,803 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+\n+#include \"merged.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+#include \n+#include \n+\n+static void clear_dir(const char *dirname)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstrbuf_addstr(&path, dirname);\n+\tremove_dir_recursively(&path, 0);\n+\tstrbuf_release(&path);\n+}\n+\n+static char *get_tmp_template(const char *prefix)\n+{\n+\tconst char *tmp = getenv(\"TMPDIR\");\n+\tstatic char template[1024];\n+\tsnprintf(template, sizeof(template) - 1, \"%s/%s.XXXXXX\",\n+\t\t tmp ? tmp : \"/tmp\", prefix);\n+\treturn template;\n+}\n+\n+static void test_read_file(void)\n+{\n+\tchar *fn = get_tmp_template(__FUNCTION__);\n+\tint fd = mkstemp(fn);\n+\tchar out[1024] = \"line1\\n\\nline2\\nline3\";\n+\tint n, err;\n+\tchar **names = NULL;\n+\tchar *want[] = { \"line1\", \"line2\", \"line3\" };\n+\tint i = 0;\n+\n+\tEXPECT(fd > 0);\n+\tn = write(fd, out, strlen(out));\n+\tEXPECT(n == strlen(out));\n+\terr = close(fd);\n+\tEXPECT(err >= 0);\n+\n+\terr = read_lines(fn, &names);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; names[i] != NULL; i++) {\n+\t\tEXPECT(0 == strcmp(want[i], names[i]));\n+\t}\n+\tfree_names(names);\n+\tremove(fn);\n+}\n+\n+static void test_parse_names(void)\n+{\n+\tchar buf[] = \"line\\n\";\n+\tchar **names = NULL;\n+\tparse_names(buf, strlen(buf), &names);\n+\n+\tEXPECT(NULL != names[0]);\n+\tEXPECT(0 == strcmp(names[0], \"line\"));\n+\tEXPECT(NULL == names[1]);\n+\tfree_names(names);\n+}\n+\n+static void test_names_equal(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", \"c\", NULL };\n+\tchar *b[] = { \"a\", \"b\", \"d\", NULL };\n+\tchar *c[] = { \"a\", \"b\", NULL };\n+\n+\tEXPECT(names_equal(a, a));\n+\tEXPECT(!names_equal(a, b));\n+\tEXPECT(!names_equal(a, c));\n+}\n+\n+static int write_test_ref(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct reftable_ref_record *ref = arg;\n+\treftable_writer_set_limits(wr, ref->update_index, ref->update_index);\n+\treturn reftable_writer_add_ref(wr, ref);\n+}\n+\n+struct write_log_arg {\n+\tstruct reftable_log_record *log;\n+\tuint64_t update_index;\n+};\n+\n+static int write_test_log(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct write_log_arg *wla = arg;\n+\n+\treftable_writer_set_limits(wr, wla->update_index, wla->update_index);\n+\treturn reftable_writer_add_log(wr, wla->log);\n+}\n+\n+static void test_reftable_stack_add_one(void)\n+{\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(\"master\", dest.value.symref));\n+\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_uptodate(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL;\n+\tstruct reftable_stack *st2 = NULL;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"branch2\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\t/* simulate multi-process access to the same stack\n+\t by creating two stacks for the same directory.\n+\t */\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st1, &write_test_ref, &ref1);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tEXPECT(err == REFTABLE_LOCK_ERROR);\n+\n+\terr = reftable_stack_reload(st2);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tEXPECT_ERR(err);\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_transaction_api(void)\n+{\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_addition *add = NULL;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_new_addition(&add, st);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_addition_add(add, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_addition_commit(add);\n+\tEXPECT_ERR(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(REFTABLE_REF_SYMREF == dest.value_type);\n+\tEXPECT(0 == strcmp(\"master\", dest.value.symref));\n+\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_validate_refname(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tint i;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"a/b\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tchar *additions[] = { \"a\", \"a/b/c\" };\n+\n+\tEXPECT(mkdtemp(dir));\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(additions); i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = additions[i],\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tEXPECT(err == REFTABLE_NAME_CONFLICT);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static int write_error(struct reftable_writer *wr, void *arg)\n+{\n+\treturn *((int *)arg);\n+}\n+\n+static void test_reftable_stack_update_index_check(void)\n+{\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"name1\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"name2\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref1);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref2);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_lock_failure(void)\n+{\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err, i;\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tfor (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) {\n+\t\terr = reftable_stack_add(st, &write_error, &i);\n+\t\tEXPECT(err == i);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_add(void)\n+{\n+\tint i = 0;\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t.exact_log_message = 1,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tst->disable_auto_compact = 1;\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n+\t\trefs[i].value.val1 = reftable_malloc(SHA1_SIZE);\n+\t\tset_test_hash(refs[i].value.val1, i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = N + i + 1;\n+\t\tlogs[i].new_hash = reftable_malloc(SHA1_SIZE);\n+\t\tlogs[i].email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].new_hash, i);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_ref_record dest = { NULL };\n+\n+\t\tint err = reftable_stack_read_ref(st, refs[i].refname, &dest);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(reftable_ref_record_equal(&dest, refs + i, SHA1_SIZE));\n+\t\treftable_ref_record_release(&dest);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_log_record dest = { NULL };\n+\t\tint err = reftable_stack_read_log(st, refs[i].refname, &dest);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(reftable_log_record_equal(&dest, logs + i, SHA1_SIZE));\n+\t\treftable_log_record_release(&dest);\n+\t}\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_log_normalize(void)\n+{\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t0,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\n+\tuint8_t h1[SHA1_SIZE] = { 0x01 }, h2[SHA1_SIZE] = { 0x02 };\n+\n+\tstruct reftable_log_record input = {\n+\t\t.refname = \"branch\",\n+\t\t.update_index = 1,\n+\t\t.new_hash = h1,\n+\t\t.old_hash = h2,\n+\t};\n+\tstruct reftable_log_record dest = {\n+\t\t.update_index = 0,\n+\t};\n+\tstruct write_log_arg arg = {\n+\t\t.log = &input,\n+\t\t.update_index = 1,\n+\t};\n+\n+\tEXPECT(mkdtemp(dir));\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tinput.message = \"one\\ntwo\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\n+\tinput.message = \"one\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(dest.message, \"one\\n\"));\n+\n+\tinput.message = \"two\\n\";\n+\targ.update_index = 2;\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT_ERR(err);\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(dest.message, \"two\\n\"));\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\treftable_log_record_release(&dest);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_tombstone(void)\n+{\n+\tint i = 0;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\tstruct reftable_ref_record dest = { NULL };\n+\tstruct reftable_log_record log_dest = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tconst char *buf = \"branch\";\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\tif (i % 2 == 0) {\n+\t\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n+\t\t\trefs[i].value.val1 = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_test_hash(refs[i].value.val1, i);\n+\t\t}\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\t/* update_index is part of the key. */\n+\t\tlogs[i].update_index = 42;\n+\t\tif (i % 2 == 0) {\n+\t\t\tlogs[i].new_hash = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_test_hash(logs[i].new_hash, i);\n+\t\t\tlogs[i].email = xstrdup(\"identity@invalid\");\n+\t\t}\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tEXPECT(err == 1);\n+\treftable_ref_record_release(&dest);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tEXPECT(err == 1);\n+\treftable_log_record_release(&log_dest);\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tEXPECT(err == 1);\n+\treftable_ref_record_release(&dest);\n+\treftable_log_record_release(&log_dest);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_hash_id(void)\n+{\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"master\",\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"target\",\n+\t\t.update_index = 1,\n+\t};\n+\tstruct reftable_write_options cfg32 = { .hash_id = SHA256_ID };\n+\tstruct reftable_stack *st32 = NULL;\n+\tstruct reftable_write_options cfg_default = { 0 };\n+\tstruct reftable_stack *st_default = NULL;\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\t/* can't read it with the wrong hash ID. */\n+\terr = reftable_new_stack(&st32, dir, cfg32);\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+\n+\t/* check that we can read it back with default config too. */\n+\terr = reftable_new_stack(&st_default, dir, cfg_default);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n+\tEXPECT_ERR(err);\n+\n+\tEXPECT(reftable_ref_record_equal(&ref, &dest, SHA1_SIZE));\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st_default);\n+\tclear_dir(dir);\n+}\n+\n+static void test_log2(void)\n+{\n+\tEXPECT(1 == fastlog2(3));\n+\tEXPECT(2 == fastlog2(4));\n+\tEXPECT(2 == fastlog2(5));\n+}\n+\n+static void test_sizes_to_segments(void)\n+{\n+\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n+\t/* .................0 1 2 3 4 5 */\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(segs[2].log == 3);\n+\tEXPECT(segs[2].start == 5);\n+\tEXPECT(segs[2].end == 6);\n+\n+\tEXPECT(segs[1].log == 2);\n+\tEXPECT(segs[1].start == 2);\n+\tEXPECT(segs[1].end == 5);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_empty(void)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n+\tEXPECT(seglen == 0);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_all_equal(void)\n+{\n+\tuint64_t sizes[] = { 5, 5 };\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(seglen == 1);\n+\tEXPECT(segs[0].start == 0);\n+\tEXPECT(segs[0].end == 2);\n+\treftable_free(segs);\n+}\n+\n+static void test_suggest_compaction_segment(void)\n+{\n+\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n+\t/* .................0 1 2 3 4 5 6 */\n+\tstruct segment min =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(min.start == 2);\n+\tEXPECT(min.end == 7);\n+}\n+\n+static void test_suggest_compaction_segment_nothing(void)\n+{\n+\tuint64_t sizes[] = { 64, 32, 16, 8, 4, 2 };\n+\tstruct segment result =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(result.start == result.end);\n+}\n+\n+static void test_reflog_expire(void)\n+{\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tstruct reftable_log_record logs[20] = { { NULL } };\n+\tint N = ARRAY_SIZE(logs) - 1;\n+\tint i = 0;\n+\tint err;\n+\tstruct reftable_log_expiry_config expiry = {\n+\t\t.time = 10,\n+\t};\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = i;\n+\t\tlogs[i].time = i;\n+\t\tlogs[i].new_hash = reftable_malloc(SHA1_SIZE);\n+\t\tlogs[i].email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].new_hash, i);\n+\t}\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, logs[9].refname, &log);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[11].refname, &log);\n+\tEXPECT_ERR(err);\n+\n+\texpiry.min_update_index = 15;\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, logs[14].refname, &log);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[16].refname, &log);\n+\tEXPECT_ERR(err);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i <= N; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+\treftable_log_record_release(&log);\n+}\n+\n+static int write_nothing(struct reftable_writer *wr, void *arg)\n+{\n+\treftable_writer_set_limits(wr, 1, 1);\n+\treturn 0;\n+}\n+\n+static void test_empty_add(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_stack *st2 = NULL;\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_nothing, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tclear_dir(dir);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st2);\n+}\n+\n+static void test_reftable_stack_auto_compaction(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tint err, i;\n+\tint N = 100;\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\n+\t\tEXPECT(i < 3 || st->merged->stack_len < 2 * fastlog2(i));\n+\t}\n+\n+\tEXPECT(reftable_stack_compaction_stats(st)->entries_written <\n+\t (uint64_t)(N * fastlog2(N)));\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+int stack_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_reftable_stack_uptodate);\n+\tRUN_TEST(test_reftable_stack_transaction_api);\n+\tRUN_TEST(test_reftable_stack_hash_id);\n+\tRUN_TEST(test_sizes_to_segments_all_equal);\n+\tRUN_TEST(test_reftable_stack_auto_compaction);\n+\tRUN_TEST(test_reftable_stack_validate_refname);\n+\tRUN_TEST(test_reftable_stack_update_index_check);\n+\tRUN_TEST(test_reftable_stack_lock_failure);\n+\tRUN_TEST(test_reftable_stack_log_normalize);\n+\tRUN_TEST(test_reftable_stack_tombstone);\n+\tRUN_TEST(test_reftable_stack_add_one);\n+\tRUN_TEST(test_empty_add);\n+\tRUN_TEST(test_reflog_expire);\n+\tRUN_TEST(test_suggest_compaction_segment);\n+\tRUN_TEST(test_suggest_compaction_segment_nothing);\n+\tRUN_TEST(test_sizes_to_segments);\n+\tRUN_TEST(test_sizes_to_segments_empty);\n+\tRUN_TEST(test_log2);\n+\tRUN_TEST(test_parse_names);\n+\tRUN_TEST(test_read_file);\n+\tRUN_TEST(test_names_equal);\n+\tRUN_TEST(test_reftable_stack_add);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex fdf92586737..3b702f4855e 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -5,8 +5,11 @@ int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n+\tmerged_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\trefname_test_main(argc, argv);\n \treftable_test_main(argc, argv);\n+\tstack_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"411858","messageId":"aaa817f8598bd5eee3b3f0fa35d613a8687cdd01.1607522429.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v4.git.git.1607522429.gitgitgadget@gmail.com","subject":"[PATCH v4 05/15] reftable: add blocksource, an abstraction for random access reads","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-12-09T14:00:19Z","receivedAt":"2020-12-09T14:03:07Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is usually used with files for storage. However, we abstract\naway this using the blocksource data structure. This has two advantages:\n\n* log blocks are zlib compressed, and handling them is simplified if we can\n discard byte segments from within the block layer.\n\n* for unittests, it is useful to read and write in-memory. The blocksource\n allows us to abstract the data away from on-disk files.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/blocksource.c | 148 ++++++++++++++++++++++++++++++++\n reftable/blocksource.h | 22 +++++\n reftable/reftable-blocksource.h | 49 +++++++++++\n 4 files changed, 220 insertions(+)\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/reftable-blocksource.h\n\ndiff --git a/Makefile b/Makefile\nindex 9b6d84af8f6..f623611bde2 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2391,6 +2391,7 @@ XDIFF_OBJS += xdiff/xutils.o\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n \n REFTABLE_TEST_OBJS += reftable/test_framework.o\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nnew file mode 100644\nindex 00000000000..25d4d95b52b\n--- /dev/null\n+++ b/reftable/blocksource.c\n@@ -0,0 +1,148 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"reftable-blocksource.h\"\n+#include \"reftable-error.h\"\n+\n+static void strbuf_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void strbuf_close(void *b)\n+{\n+}\n+\n+static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct strbuf *b = (struct strbuf *)v;\n+\tassert(off + size <= b->len);\n+\tdest->data = reftable_calloc(size);\n+\tmemcpy(dest->data, b->buf + off, size);\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static uint64_t strbuf_size(void *b)\n+{\n+\treturn ((struct strbuf *)b)->len;\n+}\n+\n+static struct reftable_block_source_vtable strbuf_vtable = {\n+\t.size = &strbuf_size,\n+\t.read_block = &strbuf_read_block,\n+\t.return_block = &strbuf_return_block,\n+\t.close = &strbuf_close,\n+};\n+\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf)\n+{\n+\tassert(bs->ops == NULL);\n+\tbs->ops = &strbuf_vtable;\n+\tbs->arg = buf;\n+}\n+\n+static void malloc_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static struct reftable_block_source_vtable malloc_vtable = {\n+\t.return_block = &malloc_return_block,\n+};\n+\n+static struct reftable_block_source malloc_block_source_instance = {\n+\t.ops = &malloc_vtable,\n+};\n+\n+struct reftable_block_source malloc_block_source(void)\n+{\n+\treturn malloc_block_source_instance;\n+}\n+\n+struct file_block_source {\n+\tint fd;\n+\tuint64_t size;\n+};\n+\n+static uint64_t file_size(void *b)\n+{\n+\treturn ((struct file_block_source *)b)->size;\n+}\n+\n+static void file_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void file_close(void *b)\n+{\n+\tint fd = ((struct file_block_source *)b)->fd;\n+\tif (fd > 0) {\n+\t\tclose(fd);\n+\t\t((struct file_block_source *)b)->fd = 0;\n+\t}\n+\n+\treftable_free(b);\n+}\n+\n+static int file_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct file_block_source *b = (struct file_block_source *)v;\n+\tassert(off + size <= b->size);\n+\tdest->data = reftable_malloc(size);\n+\tif (pread(b->fd, dest->data, size, off) != size)\n+\t\treturn -1;\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static struct reftable_block_source_vtable file_vtable = {\n+\t.size = &file_size,\n+\t.read_block = &file_read_block,\n+\t.return_block = &file_return_block,\n+\t.close = &file_close,\n+};\n+\n+int reftable_block_source_from_file(struct reftable_block_source *bs,\n+\t\t\t\t const char *name)\n+{\n+\tstruct stat st = { 0 };\n+\tint err = 0;\n+\tint fd = open(name, O_RDONLY);\n+\tstruct file_block_source *p = NULL;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\treturn REFTABLE_NOT_EXIST_ERROR;\n+\t\t}\n+\t\treturn -1;\n+\t}\n+\n+\terr = fstat(fd, &st);\n+\tif (err < 0)\n+\t\treturn -1;\n+\n+\tp = reftable_calloc(sizeof(struct file_block_source));\n+\tp->size = st.st_size;\n+\tp->fd = fd;\n+\n+\tassert(bs->ops == NULL);\n+\tbs->ops = &file_vtable;\n+\tbs->arg = p;\n+\treturn 0;\n+}\ndiff --git a/reftable/blocksource.h b/reftable/blocksource.h\nnew file mode 100644\nindex 00000000000..072e2727ad2\n--- /dev/null\n+++ b/reftable/blocksource.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCKSOURCE_H\n+#define BLOCKSOURCE_H\n+\n+#include \"system.h\"\n+\n+struct reftable_block_source;\n+\n+/* Create an in-memory block source for reading reftables */\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf);\n+\n+struct reftable_block_source malloc_block_source(void);\n+\n+#endif\ndiff --git a/reftable/reftable-blocksource.h b/reftable/reftable-blocksource.h\nnew file mode 100644\nindex 00000000000..5aa3990a573\n--- /dev/null\n+++ b/reftable/reftable-blocksource.h\n@@ -0,0 +1,49 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_BLOCKSOURCE_H\n+#define REFTABLE_BLOCKSOURCE_H\n+\n+#include \n+\n+/* block_source is a generic wrapper for a seekable readable file.\n+ */\n+struct reftable_block_source {\n+\tstruct reftable_block_source_vtable *ops;\n+\tvoid *arg;\n+};\n+\n+/* a contiguous segment of bytes. It keeps track of its generating block_source\n+ * so it can return itself into the pool. */\n+struct reftable_block {\n+\tuint8_t *data;\n+\tint len;\n+\tstruct reftable_block_source source;\n+};\n+\n+/* block_source_vtable are the operations that make up block_source */\n+struct reftable_block_source_vtable {\n+\t/* returns the size of a block source */\n+\tuint64_t (*size)(void *source);\n+\n+\t/* reads a segment from the block source. It is an error to read\n+\t beyond the end of the block */\n+\tint (*read_block)(void *source, struct reftable_block *dest,\n+\t\t\t uint64_t off, uint32_t size);\n+\t/* mark the block as read; may return the data back to malloc */\n+\tvoid (*return_block)(void *source, struct reftable_block *blockp);\n+\n+\t/* release all resources associated with the block source */\n+\tvoid (*close)(void *source);\n+};\n+\n+/* opens a file on the file system as a block_source */\n+int reftable_block_source_from_file(struct reftable_block_source *block_src,\n+\t\t\t\t const char *name);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"411859","messageId":"4076ef5d20b4986cf857e70fffe1de9da3bb12e3.1607522430.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v4.git.git.1607522429.gitgitgadget@gmail.com","subject":"[PATCH v4 15/15] Add \"test-tool dump-reftable\" command.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-12-09T14:00:29Z","receivedAt":"2020-12-09T14:03:07Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis command dumps individual tables or a stack of of tables.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/dump.c | 67 ++++++++++++++++++++--------------------\n t/helper/test-reftable.c | 5 +++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 5 files changed, 42 insertions(+), 33 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 6b7c8a165f4..2f9bf61fffb 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2409,6 +2409,7 @@ REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/dump.o\n REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/refname_test.o\ndiff --git a/reftable/dump.c b/reftable/dump.c\nindex 00b444e8c9b..7d620a3cf0f 100644\n--- a/reftable/dump.c\n+++ b/reftable/dump.c\n@@ -12,18 +12,25 @@ license that can be found in the LICENSE file or at\n #include \n #include \n \n-#include \"reftable.h\"\n+#include \"reftable-blocksource.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-record.h\"\n #include \"reftable-tests.h\"\n+#include \"reftable-writer.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+#include \"reftable-stack.h\"\n \n static uint32_t hash_id;\n \n static int dump_table(const char *tablename)\n {\n-\tstruct reftable_block_source src = { 0 };\n+\tstruct reftable_block_source src = { NULL };\n \tint err = reftable_block_source_from_file(&src, tablename);\n-\tstruct reftable_iterator it = { 0 };\n-\tstruct reftable_ref_record ref = { 0 };\n-\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n \tstruct reftable_reader *r = NULL;\n \n \tif (err < 0)\n@@ -49,7 +56,7 @@ static int dump_table(const char *tablename)\n \t\treftable_ref_record_print(&ref, hash_id);\n \t}\n \treftable_iterator_destroy(&it);\n-\treftable_ref_record_clear(&ref);\n+\treftable_ref_record_release(&ref);\n \n \terr = reftable_reader_seek_log(r, &it, \"\");\n \tif (err < 0) {\n@@ -66,7 +73,7 @@ static int dump_table(const char *tablename)\n \t\treftable_log_record_print(&log, hash_id);\n \t}\n \treftable_iterator_destroy(&it);\n-\treftable_log_record_clear(&log);\n+\treftable_log_record_release(&log);\n \n \treftable_reader_free(r);\n \treturn 0;\n@@ -75,7 +82,7 @@ static int dump_table(const char *tablename)\n static int compact_stack(const char *stackdir)\n {\n \tstruct reftable_stack *stack = NULL;\n-\tstruct reftable_write_options cfg = {};\n+\tstruct reftable_write_options cfg = { 0 };\n \n \tint err = reftable_new_stack(&stack, stackdir, cfg);\n \tif (err < 0)\n@@ -94,10 +101,10 @@ static int compact_stack(const char *stackdir)\n static int dump_stack(const char *stackdir)\n {\n \tstruct reftable_stack *stack = NULL;\n-\tstruct reftable_write_options cfg = {};\n-\tstruct reftable_iterator it = { 0 };\n-\tstruct reftable_ref_record ref = { 0 };\n-\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n \tstruct reftable_merged_table *merged = NULL;\n \n \tint err = reftable_new_stack(&stack, stackdir, cfg);\n@@ -122,7 +129,7 @@ static int dump_stack(const char *stackdir)\n \t\treftable_ref_record_print(&ref, hash_id);\n \t}\n \treftable_iterator_destroy(&it);\n-\treftable_ref_record_clear(&ref);\n+\treftable_ref_record_release(&ref);\n \n \terr = reftable_merged_table_seek_log(merged, &it, \"\");\n \tif (err < 0) {\n@@ -139,7 +146,7 @@ static int dump_stack(const char *stackdir)\n \t\treftable_log_record_print(&log, hash_id);\n \t}\n \treftable_iterator_destroy(&it);\n-\treftable_log_record_clear(&log);\n+\treftable_log_record_release(&log);\n \n \treftable_stack_destroy(stack);\n \treturn 0;\n@@ -160,40 +167,34 @@ static void print_help(void)\n int reftable_dump_main(int argc, char *const *argv)\n {\n \tint err = 0;\n-\tint opt;\n \tint opt_dump_table = 0;\n \tint opt_dump_stack = 0;\n \tint opt_compact = 0;\n-\tconst char *arg = NULL;\n-\twhile ((opt = getopt(argc, argv, \"2chts\")) != -1) {\n-\t\tswitch (opt) {\n-\t\tcase '2':\n-\t\t\thash_id = 0x73323536;\n+\tconst char *arg = NULL, *argv0 = argv[0];\n+\n+\tfor (; argc > 1; argv++, argc--)\n+\t\tif (*argv[1] != '-')\n \t\t\tbreak;\n-\t\tcase 't':\n+\t\telse if (!strcmp(\"-2\", argv[1]))\n+\t\t\thash_id = 0x73323536;\n+\t\telse if (!strcmp(\"-t\", argv[1]))\n \t\t\topt_dump_table = 1;\n-\t\t\tbreak;\n-\t\tcase 's':\n+\t\telse if (!strcmp(\"-s\", argv[1]))\n \t\t\topt_dump_stack = 1;\n-\t\t\tbreak;\n-\t\tcase 'c':\n+\t\telse if (!strcmp(\"-c\", argv[1]))\n \t\t\topt_compact = 1;\n-\t\t\tbreak;\n-\t\tcase '?':\n-\t\tcase 'h':\n+\t\telse if (!strcmp(\"-?\", argv[1]) || !strcmp(\"-h\", argv[1])) {\n \t\t\tprint_help();\n \t\t\treturn 2;\n-\t\t\tbreak;\n \t\t}\n-\t}\n \n-\tif (argv[optind] == NULL) {\n+\tif (argc != 2) {\n \t\tfprintf(stderr, \"need argument\\n\");\n \t\tprint_help();\n \t\treturn 2;\n \t}\n \n-\targ = argv[optind];\n+\targ = argv[1];\n \n \tif (opt_dump_table) {\n \t\terr = dump_table(arg);\n@@ -204,7 +205,7 @@ int reftable_dump_main(int argc, char *const *argv)\n \t}\n \n \tif (err < 0) {\n-\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv[0], arg,\n+\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv0, arg,\n \t\t\treftable_error_str(err));\n \t\treturn 1;\n \t}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 3b702f4855e..c1ba131c3a4 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -13,3 +13,8 @@ int cmd__reftable(int argc, const char **argv)\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n+\n+int cmd__dump_reftable(int argc, const char **argv)\n+{\n+\treturn reftable_dump_main(argc, (char *const *)argv);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 0208a0a41cf..f064440a319 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -56,6 +56,7 @@ static struct test_cmd cmds[] = {\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"reftable\", cmd__reftable },\n+\t{ \"dump-reftable\", cmd__dump_reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 1de39ce5b58..226af8c6b89 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -18,6 +18,7 @@ int cmd__dump_cache_tree(int argc, const char **argv);\n int cmd__dump_fsmonitor(int argc, const char **argv);\n int cmd__dump_split_index(int argc, const char **argv);\n int cmd__dump_untracked_cache(int argc, const char **argv);\n+int cmd__dump_reftable(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n-- \ngitgitgadget\n"},{"id":"411860","messageId":"2f55ff6d2406612c6fea16e0abefc41299642305.1607522429.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v4.git.git.1607522429.gitgitgadget@gmail.com","subject":"[PATCH v4 04/15] reftable: utility functions","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-12-09T14:00:18Z","receivedAt":"2020-12-09T14:03:07Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis commit provides basic utility classes for the reftable library.\n\nSigned-off-by: Han-Wen Nienhuys \nHelped-by: Johannes Schindelin \n---\n Makefile | 24 +++++-\n contrib/buildsystems/CMakeLists.txt | 14 ++-\n reftable/basics.c | 128 ++++++++++++++++++++++++++++\n reftable/basics.h | 60 +++++++++++++\n reftable/basics_test.c | 98 +++++++++++++++++++++\n reftable/publicbasics.c | 58 +++++++++++++\n reftable/reftable-malloc.h | 18 ++++\n reftable/reftable-tests.h | 22 +++++\n reftable/system.h | 32 +++++++\n reftable/test_framework.c | 23 +++++\n reftable/test_framework.h | 53 ++++++++++++\n t/helper/test-reftable.c | 9 ++\n t/helper/test-tool.c | 3 +-\n t/helper/test-tool.h | 1 +\n t/t0032-reftable-unittest.sh | 15 ++++\n 15 files changed, 552 insertions(+), 6 deletions(-)\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/basics_test.c\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-malloc.h\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 t/helper/test-reftable.c\n create mode 100755 t/t0032-reftable-unittest.sh\n\ndiff --git a/Makefile b/Makefile\nindex 45bce31016b..9b6d84af8f6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -731,6 +731,7 @@ TEST_BUILTINS_OBJS += test-read-cache.o\n TEST_BUILTINS_OBJS += test-read-graph.o\n TEST_BUILTINS_OBJS += test-read-midx.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n+TEST_BUILTINS_OBJS += test-reftable.o\n TEST_BUILTINS_OBJS += test-regex.o\n TEST_BUILTINS_OBJS += test-repository.o\n TEST_BUILTINS_OBJS += test-revision-walking.o\n@@ -820,6 +821,8 @@ TEST_SHELL_PATH = $(SHELL_PATH)\n \n LIB_FILE = libgit.a\n XDIFF_LIB = xdiff/lib.a\n+REFTABLE_LIB = reftable/libreftable.a\n+REFTABLE_TEST_LIB = reftable/libreftable_test.a\n \n GENERATED_H += command-list.h\n GENERATED_H += config-list.h\n@@ -1185,7 +1188,7 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n-GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB)\n+GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB)\n EXTLIBS =\n \n GIT_USER_AGENT = git/$(GIT_VERSION)\n@@ -2386,10 +2389,19 @@ XDIFF_OBJS += xdiff/xpatience.o\n XDIFF_OBJS += xdiff/xprepare.o\n XDIFF_OBJS += xdiff/xutils.o\n \n+REFTABLE_OBJS += reftable/basics.o\n+REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/publicbasics.o\n+\n+REFTABLE_TEST_OBJS += reftable/test_framework.o\n+REFTABLE_TEST_OBJS += reftable/basics_test.o\n+\n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \\\n \t$(XDIFF_OBJS) \\\n \t$(FUZZ_OBJS) \\\n+\t$(REFTABLE_OBJS) \\\n+\t$(REFTABLE_TEST_OBJS) \\\n \tcommon-main.o \\\n \tgit.o\n ifndef NO_CURL\n@@ -2541,6 +2553,12 @@ $(LIB_FILE): $(LIB_OBJS)\n $(XDIFF_LIB): $(XDIFF_OBJS)\n \t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n \n+$(REFTABLE_LIB): $(REFTABLE_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n+$(REFTABLE_TEST_LIB): $(REFTABLE_TEST_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n export DEFAULT_EDITOR DEFAULT_PAGER\n \n Documentation/GIT-EXCLUDED-PROGRAMS: FORCE\n@@ -2821,7 +2839,7 @@ perf: all\n \n t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n \n-t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS)\n+t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) $(REFTABLE_TEST_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n \n check-sha1:: t/helper/test-tool$X\n@@ -3150,7 +3168,7 @@ cocciclean:\n clean: profile-clean coverage-clean cocciclean\n \t$(RM) *.res\n \t$(RM) $(OBJECTS)\n-\t$(RM) $(LIB_FILE) $(XDIFF_LIB)\n+\t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) git$X\n \t$(RM) $(TEST_PROGRAMS)\n \t$(RM) $(FUZZ_PROGRAMS)\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex df539a44fa0..f3a2fd35616 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -591,6 +591,12 @@ parse_makefile_for_sources(libxdiff_SOURCES \"XDIFF_OBJS\")\n list(TRANSFORM libxdiff_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n add_library(xdiff STATIC ${libxdiff_SOURCES})\n \n+#reftable\n+parse_makefile_for_sources(reftable_SOURCES \"REFTABLE_OBJS\")\n+\n+list(TRANSFORM reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n+add_library(reftable STATIC ${reftable_SOURCES})\n+\n if(WIN32)\n \tif(NOT MSVC)#use windres when compiling with gcc and clang\n \t\tadd_custom_command(OUTPUT ${CMAKE_BINARY_DIR}/git.res\n@@ -613,7 +619,7 @@ endif()\n #link all required libraries to common-main\n add_library(common-main OBJECT ${CMAKE_SOURCE_DIR}/common-main.c)\n \n-target_link_libraries(common-main libgit xdiff ${ZLIB_LIBRARIES})\n+target_link_libraries(common-main libgit xdiff reftable ${ZLIB_LIBRARIES})\n if(Intl_FOUND)\n \ttarget_link_libraries(common-main ${Intl_LIBRARIES})\n endif()\n@@ -848,11 +854,15 @@ if(BUILD_TESTING)\n add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)\n target_link_libraries(test-fake-ssh common-main)\n \n+#reftable-tests\n+parse_makefile_for_sources(test-reftable_SOURCES \"REFTABLE_TEST_OBJS\")\n+list(TRANSFORM test-reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n+\n #test-tool\n parse_makefile_for_sources(test-tool_SOURCES \"TEST_BUILTINS_OBJS\")\n \n list(TRANSFORM test-tool_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/t/helper/\")\n-add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES})\n+add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES} ${test-reftable_SOURCES})\n target_link_libraries(test-tool common-main)\n \n set_target_properties(test-fake-ssh test-tool\ndiff --git a/reftable/basics.c b/reftable/basics.c\nnew file mode 100644\nindex 00000000000..abd027b9888\n--- /dev/null\n+++ b/reftable/basics.c\n@@ -0,0 +1,128 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+\n+void put_be24(uint8_t *out, uint32_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 16) & 0xff);\n+\tout[1] = (uint8_t)((i >> 8) & 0xff);\n+\tout[2] = (uint8_t)(i & 0xff);\n+}\n+\n+uint32_t get_be24(uint8_t *in)\n+{\n+\treturn (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 |\n+\t (uint32_t)(in[2]);\n+}\n+\n+void put_be16(uint8_t *out, uint16_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 8) & 0xff);\n+\tout[1] = (uint8_t)(i & 0xff);\n+}\n+\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n+{\n+\tsize_t lo = 0;\n+\tsize_t hi = sz;\n+\n+\t/* Invariants:\n+\t *\n+\t * (hi == sz) || f(hi) == true\n+\t * (lo == 0 && f(0) == true) || fi(lo) == false\n+\t */\n+\twhile (hi - lo > 1) {\n+\t\tsize_t mid = lo + (hi - lo) / 2;\n+\n+\t\tif (f(mid, args))\n+\t\t\thi = mid;\n+\t\telse\n+\t\t\tlo = mid;\n+\t}\n+\n+\tif (lo)\n+\t\treturn hi;\n+\n+\treturn f(0, args) ? 0 : 1;\n+}\n+\n+void free_names(char **a)\n+{\n+\tchar **p;\n+\tif (a == NULL) {\n+\t\treturn;\n+\t}\n+\tfor (p = a; *p; p++) {\n+\t\treftable_free(*p);\n+\t}\n+\treftable_free(a);\n+}\n+\n+int names_length(char **names)\n+{\n+\tchar **p = names;\n+\tfor (; *p; p++) {\n+\t\t/* empty */\n+\t}\n+\treturn p - names;\n+}\n+\n+void parse_names(char *buf, int size, char ***namesp)\n+{\n+\tchar **names = NULL;\n+\tsize_t names_cap = 0;\n+\tsize_t names_len = 0;\n+\n+\tchar *p = buf;\n+\tchar *end = buf + size;\n+\twhile (p < end) {\n+\t\tchar *next = strchr(p, '\\n');\n+\t\tif (next && next < end) {\n+\t\t\t*next = 0;\n+\t\t} else {\n+\t\t\tnext = end;\n+\t\t}\n+\t\tif (p < next) {\n+\t\t\tif (names_len == names_cap) {\n+\t\t\t\tnames_cap = 2 * names_cap + 1;\n+\t\t\t\tnames = reftable_realloc(\n+\t\t\t\t\tnames, names_cap * sizeof(*names));\n+\t\t\t}\n+\t\t\tnames[names_len++] = xstrdup(p);\n+\t\t}\n+\t\tp = next + 1;\n+\t}\n+\n+\tnames = reftable_realloc(names, (names_len + 1) * sizeof(*names));\n+\tnames[names_len] = NULL;\n+\t*namesp = names;\n+}\n+\n+int names_equal(char **a, char **b)\n+{\n+\tint i = 0;\n+\tfor (; a[i] && b[i]; i++) {\n+\t\tif (strcmp(a[i], b[i])) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treturn a[i] == b[i];\n+}\n+\n+int common_prefix_size(struct strbuf *a, struct strbuf *b)\n+{\n+\tint p = 0;\n+\tfor (; p < a->len && p < b->len; p++) {\n+\t\tif (a->buf[p] != b->buf[p])\n+\t\t\tbreak;\n+\t}\n+\n+\treturn p;\n+}\ndiff --git a/reftable/basics.h b/reftable/basics.h\nnew file mode 100644\nindex 00000000000..096b36862b9\n--- /dev/null\n+++ b/reftable/basics.h\n@@ -0,0 +1,60 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BASICS_H\n+#define BASICS_H\n+\n+/*\n+ * miscellaneous utilities that are not provided by Git.\n+ */\n+\n+#include \"system.h\"\n+\n+/* Bigendian en/decoding of integers */\n+\n+void put_be24(uint8_t *out, uint32_t i);\n+uint32_t get_be24(uint8_t *in);\n+void put_be16(uint8_t *out, uint16_t i);\n+\n+/*\n+ * find smallest index i in [0, sz) at which f(i) is true, assuming\n+ * that f is ascending. Return sz if f(i) is false for all indices.\n+ *\n+ * Contrary to bsearch(3), this returns something useful if the argument is not\n+ * found.\n+ */\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args);\n+\n+/*\n+ * Frees a NULL terminated array of malloced strings. The array itself is also\n+ * freed.\n+ */\n+void free_names(char **a);\n+\n+/* parse a newline separated list of names. `size` is the length of the buffer,\n+ * without terminating '\\0'. Empty names are discarded. */\n+void parse_names(char *buf, int size, char ***namesp);\n+\n+/* compares two NULL-terminated arrays of strings. */\n+int names_equal(char **a, char **b);\n+\n+/* returns the array size of a NULL-terminated array of strings. */\n+int names_length(char **names);\n+\n+/* Allocation routines; they invoke the functions set through\n+ * reftable_set_alloc() */\n+void *reftable_malloc(size_t sz);\n+void *reftable_realloc(void *p, size_t sz);\n+void reftable_free(void *p);\n+void *reftable_calloc(size_t sz);\n+\n+/* Find the longest shared prefix size of `a` and `b` */\n+struct strbuf;\n+int common_prefix_size(struct strbuf *a, struct strbuf *b);\n+\n+#endif\ndiff --git a/reftable/basics_test.c b/reftable/basics_test.c\nnew file mode 100644\nindex 00000000000..6d52f0f9d5a\n--- /dev/null\n+++ b/reftable/basics_test.c\n@@ -0,0 +1,98 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct binsearch_args {\n+\tint key;\n+\tint *arr;\n+};\n+\n+static int binsearch_func(size_t i, void *void_args)\n+{\n+\tstruct binsearch_args *args = (struct binsearch_args *)void_args;\n+\n+\treturn args->key < args->arr[i];\n+}\n+\n+static void test_binsearch(void)\n+{\n+\tint arr[] = { 2, 4, 6, 8, 10 };\n+\tsize_t sz = ARRAY_SIZE(arr);\n+\tstruct binsearch_args args = {\n+\t\t.arr = arr,\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 1; i < 11; i++) {\n+\t\tint res;\n+\t\targs.key = i;\n+\t\tres = binsearch(sz, &binsearch_func, &args);\n+\n+\t\tif (res < sz) {\n+\t\t\tEXPECT(args.key < arr[res]);\n+\t\t\tif (res > 0) {\n+\t\t\t\tEXPECT(args.key >= arr[res - 1]);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tEXPECT(args.key == 10 || args.key == 11);\n+\t\t}\n+\t}\n+}\n+\n+static void test_names_length(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", NULL };\n+\tEXPECT(names_length(a) == 2);\n+}\n+\n+static void test_parse_names_normal(void)\n+{\n+\tchar in[] = \"a\\nb\\n\";\n+\tchar **out = NULL;\n+\tparse_names(in, strlen(in), &out);\n+\tEXPECT(!strcmp(out[0], \"a\"));\n+\tEXPECT(!strcmp(out[1], \"b\"));\n+\tEXPECT(out[2] == NULL);\n+\tfree_names(out);\n+}\n+\n+static void test_parse_names_drop_empty(void)\n+{\n+\tchar in[] = \"a\\n\\n\";\n+\tchar **out = NULL;\n+\tparse_names(in, strlen(in), &out);\n+\tEXPECT(!strcmp(out[0], \"a\"));\n+\tEXPECT(out[1] == NULL);\n+\tfree_names(out);\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct strbuf s1 = STRBUF_INIT;\n+\tstruct strbuf s2 = STRBUF_INIT;\n+\tstrbuf_addstr(&s1, \"abcdef\");\n+\tstrbuf_addstr(&s2, \"abc\");\n+\tEXPECT(common_prefix_size(&s1, &s2) == 3);\n+\tstrbuf_release(&s1);\n+\tstrbuf_release(&s2);\n+}\n+\n+int basics_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_common_prefix);\n+\tRUN_TEST(test_parse_names_normal);\n+\tRUN_TEST(test_parse_names_drop_empty);\n+\tRUN_TEST(test_binsearch);\n+\tRUN_TEST(test_names_length);\n+\treturn 0;\n+}\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\nnew file mode 100644\nindex 00000000000..25639f61af6\n--- /dev/null\n+++ b/reftable/publicbasics.c\n@@ -0,0 +1,58 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-malloc.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+static void *(*reftable_malloc_ptr)(size_t sz) = &malloc;\n+static void *(*reftable_realloc_ptr)(void *, size_t) = &realloc;\n+static void (*reftable_free_ptr)(void *) = &free;\n+\n+void *reftable_malloc(size_t sz)\n+{\n+\treturn (*reftable_malloc_ptr)(sz);\n+}\n+\n+void *reftable_realloc(void *p, size_t sz)\n+{\n+\treturn (*reftable_realloc_ptr)(p, sz);\n+}\n+\n+void reftable_free(void *p)\n+{\n+\treftable_free_ptr(p);\n+}\n+\n+void *reftable_calloc(size_t sz)\n+{\n+\tvoid *p = reftable_malloc(sz);\n+\tmemset(p, 0, sz);\n+\treturn p;\n+}\n+\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n+{\n+\treftable_malloc_ptr = malloc;\n+\treftable_realloc_ptr = realloc;\n+\treftable_free_ptr = free;\n+}\n+\n+int hash_size(uint32_t id)\n+{\n+\tswitch (id) {\n+\tcase 0:\n+\tcase SHA1_ID:\n+\t\treturn SHA1_SIZE;\n+\tcase SHA256_ID:\n+\t\treturn SHA256_SIZE;\n+\t}\n+\tabort();\n+}\ndiff --git a/reftable/reftable-malloc.h b/reftable/reftable-malloc.h\nnew file mode 100644\nindex 00000000000..5f2185f1f34\n--- /dev/null\n+++ b/reftable/reftable-malloc.h\n@@ -0,0 +1,18 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_H\n+#define REFTABLE_H\n+\n+#include \n+\n+/* Overrides the functions to use for memory management. */\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n+\n+#endif\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nnew file mode 100644\nindex 00000000000..5e7698ae654\n--- /dev/null\n+++ b/reftable/reftable-tests.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_TESTS_H\n+#define REFTABLE_TESTS_H\n+\n+int basics_test_main(int argc, const char **argv);\n+int block_test_main(int argc, const char **argv);\n+int merged_test_main(int argc, const char **argv);\n+int record_test_main(int argc, const char **argv);\n+int refname_test_main(int argc, const char **argv);\n+int reftable_test_main(int argc, const char **argv);\n+int stack_test_main(int argc, const char **argv);\n+int tree_test_main(int argc, const char **argv);\n+int reftable_dump_main(int argc, char *const *argv);\n+\n+#endif\ndiff --git a/reftable/system.h b/reftable/system.h\nnew file mode 100644\nindex 00000000000..07277ca0627\n--- /dev/null\n+++ b/reftable/system.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef SYSTEM_H\n+#define SYSTEM_H\n+\n+#include \"git-compat-util.h\"\n+#include \"strbuf.h\"\n+\n+#include \n+\n+struct strbuf;\n+/* In git, this is declared in dir.h */\n+int remove_dir_recursively(struct strbuf *path, int flags);\n+\n+#define SHA1_ID 0x73686131\n+#define SHA256_ID 0x73323536\n+#define SHA1_SIZE 20\n+#define SHA256_SIZE 32\n+\n+/* This is uncompress2, which is only available in zlib as of 2017.\n+ */\n+int uncompress_return_consumed(Bytef *dest, uLongf *destLen,\n+\t\t\t const Bytef *source, uLong *sourceLen);\n+int hash_size(uint32_t id);\n+\n+#endif\ndiff --git a/reftable/test_framework.c b/reftable/test_framework.c\nnew file mode 100644\nindex 00000000000..a5ff4e2a2d2\n--- /dev/null\n+++ b/reftable/test_framework.c\n@@ -0,0 +1,23 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"test_framework.h\"\n+\n+#include \"basics.h\"\n+\n+void set_test_hash(uint8_t *p, int i)\n+{\n+\tmemset(p, (uint8_t)i, hash_size(SHA1_ID));\n+}\n+\n+int strbuf_add_void(void *b, const void *data, size_t sz)\n+{\n+\tstrbuf_add((struct strbuf *)b, data, sz);\n+\treturn sz;\n+}\ndiff --git a/reftable/test_framework.h b/reftable/test_framework.h\nnew file mode 100644\nindex 00000000000..5fdc9519a5a\n--- /dev/null\n+++ b/reftable/test_framework.h\n@@ -0,0 +1,53 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TEST_FRAMEWORK_H\n+#define TEST_FRAMEWORK_H\n+\n+#include \"system.h\"\n+#include \"reftable-error.h\"\n+\n+#define EXPECT_ERR(c) \\\n+\tif (c != 0) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: error == %d (%s), want 0\\n\", \\\n+\t\t\t__FILE__, __LINE__, c, reftable_error_str(c)); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define EXPECT_STREQ(a, b) \\\n+\tif (strcmp(a, b)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s:%d: %s (%s) != %s (%s)\\n\", __FILE__, \\\n+\t\t\t__LINE__, #a, a, #b, b); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define EXPECT(c) \\\n+\tif (!(c)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: failed assertion %s\\n\", __FILE__, \\\n+\t\t\t__LINE__, #c); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define RUN_TEST(f) \\\n+\tfprintf(stderr, \"running %s\\n\", #f); \\\n+\tfflush(stderr); \\\n+\tf();\n+\n+void set_test_hash(uint8_t *p, int i);\n+\n+/* Like strbuf_add, but suitable for passing to reftable_new_writer\n+ */\n+int strbuf_add_void(void *b, const void *data, size_t sz);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nnew file mode 100644\nindex 00000000000..3b58e423e7b\n--- /dev/null\n+++ b/t/helper/test-reftable.c\n@@ -0,0 +1,9 @@\n+#include \"reftable/reftable-tests.h\"\n+#include \"test-tool.h\"\n+\n+int cmd__reftable(int argc, const char **argv)\n+{\n+\tbasics_test_main(argc, argv);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 9d6d14d9293..0208a0a41cf 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -48,13 +48,14 @@ static struct test_cmd cmds[] = {\n \t{ \"path-utils\", cmd__path_utils },\n \t{ \"pkt-line\", cmd__pkt_line },\n \t{ \"prio-queue\", cmd__prio_queue },\n-\t{ \"proc-receive\", cmd__proc_receive},\n+\t{ \"proc-receive\", cmd__proc_receive },\n \t{ \"progress\", cmd__progress },\n \t{ \"reach\", cmd__reach },\n \t{ \"read-cache\", cmd__read_cache },\n \t{ \"read-graph\", cmd__read_graph },\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n+\t{ \"reftable\", cmd__reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex a6470ff62c4..1de39ce5b58 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -44,6 +44,7 @@ int cmd__read_cache(int argc, const char **argv);\n int cmd__read_graph(int argc, const char **argv);\n int cmd__read_midx(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n+int cmd__reftable(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__repository(int argc, const char **argv);\n int cmd__revision_walking(int argc, const char **argv);\ndiff --git a/t/t0032-reftable-unittest.sh b/t/t0032-reftable-unittest.sh\nnew file mode 100755\nindex 00000000000..0ed14971a58\n--- /dev/null\n+++ b/t/t0032-reftable-unittest.sh\n@@ -0,0 +1,15 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable unittests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'unittests' '\n+\tTMPDIR=$(pwd) && export TMPDIR &&\n+\ttest-tool reftable\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"411861","messageId":"3c2de7dcc65c6ce7b3aa1cab5c99b21e53f5f99e.1607522429.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v4.git.git.1607522429.gitgitgadget@gmail.com","subject":"[PATCH v4 10/15] reftable: read reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-12-09T14:00:24Z","receivedAt":"2020-12-09T14:03:07Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis supports reading a single reftable file.\n\nThe commit introduces an abstract iterator type, which captures the usecases\nboth of reading individual refs, and iterating over a segment of the ref\nnamespace.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 3 +\n reftable/iter.c | 248 ++++++++++++\n reftable/iter.h | 75 ++++\n reftable/reader.c | 733 +++++++++++++++++++++++++++++++++++\n reftable/reader.h | 75 ++++\n reftable/reftable-iterator.h | 37 ++\n reftable/reftable-reader.h | 89 +++++\n 7 files changed, 1260 insertions(+)\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/reftable-iterator.h\n create mode 100644 reftable/reftable-reader.h\n\ndiff --git a/Makefile b/Makefile\nindex e78908c976a..2bafcb1b62a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2393,8 +2393,11 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n+REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/reftable.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n REFTABLE_OBJS += reftable/zlib-compat.o\ndiff --git a/reftable/iter.c b/reftable/iter.c\nnew file mode 100644\nindex 00000000000..b8c091ffd57\n--- /dev/null\n+++ b/reftable/iter.c\n@@ -0,0 +1,248 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"iter.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"reftable-error.h\"\n+\n+int iterator_is_null(struct reftable_iterator *it)\n+{\n+\treturn it->ops == NULL;\n+}\n+\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n+{\n+\treturn 1;\n+}\n+\n+static void empty_iterator_close(void *arg)\n+{\n+}\n+\n+static struct reftable_iterator_vtable empty_vtable = {\n+\t.next = &empty_iterator_next,\n+\t.close = &empty_iterator_close,\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = NULL;\n+\tit->ops = &empty_vtable;\n+}\n+\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n+{\n+\treturn it->ops->next(it->iter_arg, rec);\n+}\n+\n+void reftable_iterator_destroy(struct reftable_iterator *it)\n+{\n+\tif (it->ops == NULL) {\n+\t\treturn;\n+\t}\n+\tit->ops->close(it->iter_arg);\n+\tit->ops = NULL;\n+\tFREE_AND_NULL(it->iter_arg);\n+}\n+\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, ref);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, log);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+static void filtering_ref_iterator_close(void *iter_arg)\n+{\n+\tstruct filtering_ref_iterator *fri =\n+\t\t(struct filtering_ref_iterator *)iter_arg;\n+\tstrbuf_release(&fri->oid);\n+\treftable_iterator_destroy(&fri->it);\n+}\n+\n+static int filtering_ref_iterator_next(void *iter_arg,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct filtering_ref_iterator *fri =\n+\t\t(struct filtering_ref_iterator *)iter_arg;\n+\tstruct reftable_ref_record *ref =\n+\t\t(struct reftable_ref_record *)rec->data;\n+\tint err = 0;\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&fri->it, ref);\n+\t\tif (err != 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (fri->double_check) {\n+\t\t\tstruct reftable_iterator it = { NULL };\n+\n+\t\t\terr = reftable_table_seek_ref(&fri->tab, &it,\n+\t\t\t\t\t\t ref->refname);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = reftable_iterator_next_ref(&it, ref);\n+\t\t\t}\n+\n+\t\t\treftable_iterator_destroy(&it);\n+\n+\t\t\tif (err < 0) {\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (err > 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (ref->value_type == REFTABLE_REF_VAL2 &&\n+\t\t (!memcmp(fri->oid.buf, ref->value.val2.target_value,\n+\t\t\t fri->oid.len) ||\n+\t\t !memcmp(fri->oid.buf, ref->value.val2.value,\n+\t\t\t fri->oid.len)))\n+\t\t\treturn 0;\n+\n+\t\tif (ref->value_type == REFTABLE_REF_VAL1 &&\n+\t\t !memcmp(fri->oid.buf, ref->value.val1, fri->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treftable_ref_record_release(ref);\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable filtering_ref_iterator_vtable = {\n+\t.next = &filtering_ref_iterator_next,\n+\t.close = &filtering_ref_iterator_close,\n+};\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *it,\n+\t\t\t\t\t struct filtering_ref_iterator *fri)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = fri;\n+\tit->ops = &filtering_ref_iterator_vtable;\n+}\n+\n+static void indexed_table_ref_iter_close(void *p)\n+{\n+\tstruct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p;\n+\tblock_iter_close(&it->cur);\n+\treftable_block_done(&it->block_reader.block);\n+\tstrbuf_release(&it->oid);\n+}\n+\n+static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n+{\n+\tuint64_t off;\n+\tint err = 0;\n+\tif (it->offset_idx == it->offset_len) {\n+\t\tit->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\n+\treftable_block_done(&it->block_reader.block);\n+\n+\toff = it->offsets[it->offset_idx++];\n+\terr = reader_init_block_reader(it->r, &it->block_reader, off,\n+\t\t\t\t BLOCK_TYPE_REF);\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\tif (err > 0) {\n+\t\t/* indexed block does not exist. */\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\t}\n+\tblock_reader_start(&it->block_reader, &it->cur);\n+\treturn 0;\n+}\n+\n+static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\n+{\n+\tstruct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p;\n+\tstruct reftable_ref_record *ref =\n+\t\t(struct reftable_ref_record *)rec->data;\n+\n+\twhile (1) {\n+\t\tint err = block_iter_next(&it->cur, rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\terr = indexed_table_ref_iter_next_block(it);\n+\t\t\tif (err < 0) {\n+\t\t\t\treturn err;\n+\t\t\t}\n+\n+\t\t\tif (it->is_finished) {\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\t\t/* BUG */\n+\t\tif (!memcmp(it->oid.buf, ref->value.val2.target_value,\n+\t\t\t it->oid.len) ||\n+\t\t !memcmp(it->oid.buf, ref->value.val2.value, it->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+}\n+\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len)\n+{\n+\tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n+\tstruct indexed_table_ref_iter *itr =\n+\t\treftable_calloc(sizeof(struct indexed_table_ref_iter));\n+\tint err = 0;\n+\n+\t*itr = empty;\n+\titr->r = r;\n+\tstrbuf_add(&itr->oid, oid, oid_len);\n+\n+\titr->offsets = offsets;\n+\titr->offset_len = offset_len;\n+\n+\terr = indexed_table_ref_iter_next_block(itr);\n+\tif (err < 0) {\n+\t\treftable_free(itr);\n+\t} else {\n+\t\t*dest = itr;\n+\t}\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable indexed_table_ref_iter_vtable = {\n+\t.next = &indexed_table_ref_iter_next,\n+\t.close = &indexed_table_ref_iter_close,\n+};\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = itr;\n+\tit->ops = &indexed_table_ref_iter_vtable;\n+}\ndiff --git a/reftable/iter.h b/reftable/iter.h\nnew file mode 100644\nindex 00000000000..2ba964c29a3\n--- /dev/null\n+++ b/reftable/iter.h\n@@ -0,0 +1,75 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef ITER_H\n+#define ITER_H\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"record.h\"\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+struct reftable_iterator_vtable {\n+\tint (*next)(void *iter_arg, struct reftable_record *rec);\n+\tvoid (*close)(void *iter_arg);\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it);\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec);\n+\n+/* Returns true for a zeroed out iterator, such as the one returned from\n+ * iterator_destroy. */\n+int iterator_is_null(struct reftable_iterator *it);\n+\n+/* iterator that produces only ref records that point to `oid` */\n+struct filtering_ref_iterator {\n+\tint double_check;\n+\tstruct reftable_table tab;\n+\tstruct strbuf oid;\n+\tstruct reftable_iterator it;\n+};\n+#define FILTERING_REF_ITERATOR_INIT \\\n+\t{ \\\n+\t\t.oid = STRBUF_INIT \\\n+\t}\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *,\n+\t\t\t\t\t struct filtering_ref_iterator *);\n+\n+/* iterator that produces only ref records that point to `oid`,\n+ * but using the object index.\n+ */\n+struct indexed_table_ref_iter {\n+\tstruct reftable_reader *r;\n+\tstruct strbuf oid;\n+\n+\t/* mutable */\n+\tuint64_t *offsets;\n+\n+\t/* Points to the next offset to read. */\n+\tint offset_idx;\n+\tint offset_len;\n+\tstruct block_reader block_reader;\n+\tstruct block_iter cur;\n+\tint is_finished;\n+};\n+\n+#define INDEXED_TABLE_REF_ITER_INIT \\\n+\t{ \\\n+\t\t.cur = { .last_key = STRBUF_INIT }, .oid = STRBUF_INIT, \\\n+\t}\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr);\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len);\n+\n+#endif\ndiff --git a/reftable/reader.c b/reftable/reader.c\nnew file mode 100644\nindex 00000000000..a5b7b1d4fef\n--- /dev/null\n+++ b/reftable/reader.c\n@@ -0,0 +1,733 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reader.h\"\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"iter.h\"\n+#include \"record.h\"\n+#include \"reftable-error.h\"\n+#include \"tree.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source)\n+{\n+\treturn source->ops->size(source->arg);\n+}\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tint result = source->ops->read_block(source->arg, dest, off, size);\n+\tdest->source = *source;\n+\treturn result;\n+}\n+\n+void block_source_close(struct reftable_block_source *source)\n+{\n+\tif (source->ops == NULL) {\n+\t\treturn;\n+\t}\n+\n+\tsource->ops->close(source->arg);\n+\tsource->ops = NULL;\n+}\n+\n+static struct reftable_reader_offsets *\n+reader_offsets_for(struct reftable_reader *r, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\t\treturn &r->ref_offsets;\n+\tcase BLOCK_TYPE_LOG:\n+\t\treturn &r->log_offsets;\n+\tcase BLOCK_TYPE_OBJ:\n+\t\treturn &r->obj_offsets;\n+\t}\n+\tabort();\n+}\n+\n+static int reader_get_block(struct reftable_reader *r,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t sz)\n+{\n+\tif (off >= r->size)\n+\t\treturn 0;\n+\n+\tif (off + sz > r->size) {\n+\t\tsz = r->size - off;\n+\t}\n+\n+\treturn block_source_read_block(&r->source, dest, off, sz);\n+}\n+\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r)\n+{\n+\treturn r->hash_id;\n+}\n+\n+const char *reader_name(struct reftable_reader *r)\n+{\n+\treturn r->name;\n+}\n+\n+static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n+\t\t\tuint8_t *header)\n+{\n+\tuint8_t *f = footer;\n+\tuint8_t first_block_typ;\n+\tint err = 0;\n+\tuint32_t computed_crc;\n+\tuint32_t file_crc;\n+\n+\tif (memcmp(f, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tf += 4;\n+\n+\tif (memcmp(footer, header, header_size(r->version))) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tf++;\n+\tr->block_size = get_be24(f);\n+\n+\tf += 3;\n+\tr->min_update_index = get_be64(f);\n+\tf += 8;\n+\tr->max_update_index = get_be64(f);\n+\tf += 8;\n+\n+\tif (r->version == 1) {\n+\t\tr->hash_id = SHA1_ID;\n+\t} else {\n+\t\tr->hash_id = get_be32(f);\n+\t\tswitch (r->hash_id) {\n+\t\tcase SHA1_ID:\n+\t\t\tbreak;\n+\t\tcase SHA256_ID:\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tf += 4;\n+\t}\n+\n+\tr->ref_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->obj_offsets.offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1);\n+\tr->obj_offsets.offset >>= 5;\n+\n+\tr->obj_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tcomputed_crc = crc32(0, footer, f - footer);\n+\tfile_crc = get_be32(f);\n+\tf += 4;\n+\tif (computed_crc != file_crc) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tfirst_block_typ = header[header_size(r->version)];\n+\tr->ref_offsets.is_present = (first_block_typ == BLOCK_TYPE_REF);\n+\tr->ref_offsets.offset = 0;\n+\tr->log_offsets.is_present = (first_block_typ == BLOCK_TYPE_LOG ||\n+\t\t\t\t r->log_offsets.offset > 0);\n+\tr->obj_offsets.is_present = r->obj_offsets.offset > 0;\n+\terr = 0;\n+done:\n+\treturn err;\n+}\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name)\n+{\n+\tstruct reftable_block footer = { NULL };\n+\tstruct reftable_block header = { NULL };\n+\tint err = 0;\n+\n+\tmemset(r, 0, sizeof(struct reftable_reader));\n+\n+\t/* Need +1 to read type of first block. */\n+\terr = block_source_read_block(source, &header, 0, header_size(2) + 1);\n+\tif (err != header_size(2) + 1) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tif (memcmp(header.data, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tr->version = header.data[4];\n+\tif (r->version != 1 && r->version != 2) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tr->size = block_source_size(source) - footer_size(r->version);\n+\tr->source = *source;\n+\tr->name = xstrdup(name);\n+\tr->hash_id = 0;\n+\n+\terr = block_source_read_block(source, &footer, r->size,\n+\t\t\t\t footer_size(r->version));\n+\tif (err != footer_size(r->version)) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = parse_footer(r, footer.data, header.data);\n+done:\n+\treftable_block_done(&footer);\n+\treftable_block_done(&header);\n+\treturn err;\n+}\n+\n+struct table_iter {\n+\tstruct reftable_reader *r;\n+\tuint8_t typ;\n+\tuint64_t block_off;\n+\tstruct block_iter bi;\n+\tint is_finished;\n+};\n+#define TABLE_ITER_INIT \\\n+\t{ \\\n+\t\t.bi = {.last_key = STRBUF_INIT } \\\n+\t}\n+\n+static void table_iter_copy_from(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = src->block_off;\n+\tdest->is_finished = src->is_finished;\n+\tblock_iter_copy_from(&dest->bi, &src->bi);\n+}\n+\n+static int table_iter_next_in_block(struct table_iter *ti,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tint res = block_iter_next(&ti->bi, rec);\n+\tif (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {\n+\t\t((struct reftable_ref_record *)rec->data)->update_index +=\n+\t\t\tti->r->min_update_index;\n+\t}\n+\n+\treturn res;\n+}\n+\n+static void table_iter_block_done(struct table_iter *ti)\n+{\n+\tif (ti->bi.br == NULL) {\n+\t\treturn;\n+\t}\n+\treftable_block_done(&ti->bi.br->block);\n+\tFREE_AND_NULL(ti->bi.br);\n+\n+\tti->bi.last_key.len = 0;\n+\tti->bi.next_off = 0;\n+}\n+\n+static int32_t extract_block_size(uint8_t *data, uint8_t *typ, uint64_t off,\n+\t\t\t\t int version)\n+{\n+\tint32_t result = 0;\n+\n+\tif (off == 0) {\n+\t\tdata += header_size(version);\n+\t}\n+\n+\t*typ = data[0];\n+\tif (reftable_is_block_type(*typ)) {\n+\t\tresult = get_be24(data + 1);\n+\t}\n+\treturn result;\n+}\n+\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ)\n+{\n+\tint32_t guess_block_size = r->block_size ? r->block_size :\n+\t\t\t\t\t\t\t DEFAULT_BLOCK_SIZE;\n+\tstruct reftable_block block = { NULL };\n+\tuint8_t block_typ = 0;\n+\tint err = 0;\n+\tuint32_t header_off = next_off ? 0 : header_size(r->version);\n+\tint32_t block_size = 0;\n+\n+\tif (next_off >= r->size)\n+\t\treturn 1;\n+\n+\terr = reader_get_block(r, &block, next_off, guess_block_size);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tblock_size = extract_block_size(block.data, &block_typ, next_off,\n+\t\t\t\t\tr->version);\n+\tif (block_size < 0)\n+\t\treturn block_size;\n+\n+\tif (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {\n+\t\treftable_block_done(&block);\n+\t\treturn 1;\n+\t}\n+\n+\tif (block_size > guess_block_size) {\n+\t\treftable_block_done(&block);\n+\t\terr = reader_get_block(r, &block, next_off, block_size);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\treturn block_reader_init(br, &block, header_off, r->block_size,\n+\t\t\t\t hash_size(r->hash_id));\n+}\n+\n+static int table_iter_next_block(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tuint64_t next_block_off = src->block_off + src->bi.br->full_block_size;\n+\tstruct block_reader br = { 0 };\n+\tint err = 0;\n+\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = next_block_off;\n+\n+\terr = reader_init_block_reader(src->r, &br, next_block_off, src->typ);\n+\tif (err > 0) {\n+\t\tdest->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\tif (err != 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct block_reader *brp =\n+\t\t\treftable_malloc(sizeof(struct block_reader));\n+\t\t*brp = br;\n+\n+\t\tdest->is_finished = 0;\n+\t\tblock_reader_start(brp, &dest->bi);\n+\t}\n+\treturn 0;\n+}\n+\n+static int table_iter_next(struct table_iter *ti, struct reftable_record *rec)\n+{\n+\tif (reftable_record_type(rec) != ti->typ)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\twhile (1) {\n+\t\tstruct table_iter next = TABLE_ITER_INIT;\n+\t\tint err = 0;\n+\t\tif (ti->is_finished) {\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\terr = table_iter_next_in_block(ti, rec);\n+\t\tif (err <= 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err != 0) {\n+\t\t\tti->is_finished = 1;\n+\t\t}\n+\t\ttable_iter_block_done(ti);\n+\t\tif (err != 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\ttable_iter_copy_from(ti, &next);\n+\t\tblock_iter_close(&next.bi);\n+\t}\n+}\n+\n+static int table_iter_next_void(void *ti, struct reftable_record *rec)\n+{\n+\treturn table_iter_next((struct table_iter *)ti, rec);\n+}\n+\n+static void table_iter_close(void *p)\n+{\n+\tstruct table_iter *ti = (struct table_iter *)p;\n+\ttable_iter_block_done(ti);\n+\tblock_iter_close(&ti->bi);\n+}\n+\n+static struct reftable_iterator_vtable table_iter_vtable = {\n+\t.next = &table_iter_next_void,\n+\t.close = &table_iter_close,\n+};\n+\n+static void iterator_from_table_iter(struct reftable_iterator *it,\n+\t\t\t\t struct table_iter *ti)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = ti;\n+\tit->ops = &table_iter_vtable;\n+}\n+\n+static int reader_table_iter_at(struct reftable_reader *r,\n+\t\t\t\tstruct table_iter *ti, uint64_t off,\n+\t\t\t\tuint8_t typ)\n+{\n+\tstruct block_reader br = { 0 };\n+\tstruct block_reader *brp = NULL;\n+\n+\tint err = reader_init_block_reader(r, &br, off, typ);\n+\tif (err != 0)\n+\t\treturn err;\n+\n+\tbrp = reftable_malloc(sizeof(struct block_reader));\n+\t*brp = br;\n+\tti->r = r;\n+\tti->typ = block_reader_type(brp);\n+\tti->block_off = off;\n+\tblock_reader_start(brp, &ti->bi);\n+\treturn 0;\n+}\n+\n+static int reader_start(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\tuint8_t typ, int index)\n+{\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tuint64_t off = offs->offset;\n+\tif (index) {\n+\t\toff = offs->index_offset;\n+\t\tif (off == 0) {\n+\t\t\treturn 1;\n+\t\t}\n+\t\ttyp = BLOCK_TYPE_INDEX;\n+\t}\n+\n+\treturn reader_table_iter_at(r, ti, off, typ);\n+}\n+\n+static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\t struct reftable_record *want)\n+{\n+\tstruct reftable_record rec =\n+\t\treftable_new_record(reftable_record_type(want));\n+\tstruct strbuf want_key = STRBUF_INIT;\n+\tstruct strbuf got_key = STRBUF_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = -1;\n+\n+\treftable_record_key(want, &want_key);\n+\n+\twhile (1) {\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = block_reader_first_key(next.bi.br, &got_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (strbuf_cmp(&got_key, &want_key) > 0) {\n+\t\t\ttable_iter_block_done(&next);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_block_done(ti);\n+\t\ttable_iter_copy_from(ti, &next);\n+\t}\n+\n+\terr = block_iter_seek(&ti->bi, &want_key);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = 0;\n+\n+done:\n+\tblock_iter_close(&next.bi);\n+\treftable_record_destroy(&rec);\n+\tstrbuf_release(&want_key);\n+\tstrbuf_release(&got_key);\n+\treturn err;\n+}\n+\n+static int reader_seek_indexed(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_index_record want_index = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record want_index_rec = { NULL };\n+\tstruct reftable_index_record index_result = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record index_result_rec = { NULL };\n+\tstruct table_iter index_iter = TABLE_ITER_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = 0;\n+\n+\treftable_record_key(rec, &want_index.last_key);\n+\treftable_record_from_index(&want_index_rec, &want_index);\n+\treftable_record_from_index(&index_result_rec, &index_result);\n+\n+\terr = reader_start(r, &index_iter, reftable_record_type(rec), 1);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reader_seek_linear(r, &index_iter, &want_index_rec);\n+\twhile (1) {\n+\t\terr = table_iter_next(&index_iter, &index_result_rec);\n+\t\ttable_iter_block_done(&index_iter);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = reader_table_iter_at(r, &next, index_result.offset, 0);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = block_iter_seek(&next.bi, &want_index.last_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (next.typ == reftable_record_type(rec)) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (next.typ != BLOCK_TYPE_INDEX) {\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_copy_from(&index_iter, &next);\n+\t}\n+\n+\tif (err == 0) {\n+\t\tstruct table_iter empty = TABLE_ITER_INIT;\n+\t\tstruct table_iter *malloced =\n+\t\t\treftable_calloc(sizeof(struct table_iter));\n+\t\t*malloced = empty;\n+\t\ttable_iter_copy_from(malloced, &next);\n+\t\titerator_from_table_iter(it, malloced);\n+\t}\n+done:\n+\tblock_iter_close(&next.bi);\n+\ttable_iter_close(&index_iter);\n+\treftable_record_release(&want_index_rec);\n+\treftable_record_release(&index_result_rec);\n+\treturn err;\n+}\n+\n+static int reader_seek_internal(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it,\n+\t\t\t\tstruct reftable_record *rec)\n+{\n+\tstruct reftable_reader_offsets *offs =\n+\t\treader_offsets_for(r, reftable_record_type(rec));\n+\tuint64_t idx = offs->index_offset;\n+\tstruct table_iter ti = TABLE_ITER_INIT;\n+\tint err = 0;\n+\tif (idx > 0)\n+\t\treturn reader_seek_indexed(r, it, rec);\n+\n+\terr = reader_start(r, &ti, reftable_record_type(rec), 0);\n+\tif (err < 0)\n+\t\treturn err;\n+\terr = reader_seek_linear(r, &ti, rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct table_iter *p =\n+\t\t\treftable_malloc(sizeof(struct table_iter));\n+\t\t*p = ti;\n+\t\titerator_from_table_iter(it, p);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec)\n+{\n+\tuint8_t typ = reftable_record_type(rec);\n+\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tif (!offs->is_present) {\n+\t\titerator_set_empty(it);\n+\t\treturn 0;\n+\t}\n+\n+\treturn reader_seek_internal(r, it, rec);\n+}\n+\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_reader_seek_log_at(r, it, name, max);\n+}\n+\n+void reader_close(struct reftable_reader *r)\n+{\n+\tblock_source_close(&r->source);\n+\tFREE_AND_NULL(r->name);\n+}\n+\n+int reftable_new_reader(struct reftable_reader **p,\n+\t\t\tstruct reftable_block_source *src, char const *name)\n+{\n+\tstruct reftable_reader *rd =\n+\t\treftable_calloc(sizeof(struct reftable_reader));\n+\tint err = init_reader(rd, src, name);\n+\tif (err == 0) {\n+\t\t*p = rd;\n+\t} else {\n+\t\tblock_source_close(src);\n+\t\treftable_free(rd);\n+\t}\n+\treturn err;\n+}\n+\n+void reftable_reader_free(struct reftable_reader *r)\n+{\n+\treader_close(r);\n+\treftable_free(r);\n+}\n+\n+static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct reftable_obj_record want = {\n+\t\t.hash_prefix = oid,\n+\t\t.hash_prefix_len = r->object_id_len,\n+\t};\n+\tstruct reftable_record want_rec = { NULL };\n+\tstruct reftable_iterator oit = { NULL };\n+\tstruct reftable_obj_record got = { NULL };\n+\tstruct reftable_record got_rec = { NULL };\n+\tint err = 0;\n+\tstruct indexed_table_ref_iter *itr = NULL;\n+\n+\t/* Look through the reverse index. */\n+\treftable_record_from_obj(&want_rec, &want);\n+\terr = reader_seek(r, &oit, &want_rec);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\t/* read out the reftable_obj_record */\n+\treftable_record_from_obj(&got_rec, &got);\n+\terr = iterator_next(&oit, &got_rec);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 0 ||\n+\t memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {\n+\t\t/* didn't find it; return empty iterator */\n+\t\titerator_set_empty(it);\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),\n+\t\t\t\t\t got.offsets, got.offset_len);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tgot.offsets = NULL;\n+\titerator_from_indexed_table_ref_iter(it, itr);\n+\n+done:\n+\treftable_iterator_destroy(&oit);\n+\treftable_record_release(&got_rec);\n+\treturn err;\n+}\n+\n+static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct table_iter ti_empty = TABLE_ITER_INIT;\n+\tstruct table_iter *ti = reftable_calloc(sizeof(struct table_iter));\n+\tstruct filtering_ref_iterator *filter = NULL;\n+\tstruct filtering_ref_iterator empty = FILTERING_REF_ITERATOR_INIT;\n+\tint oid_len = hash_size(r->hash_id);\n+\tint err;\n+\n+\t*ti = ti_empty;\n+\terr = reader_start(r, ti, BLOCK_TYPE_REF, 0);\n+\tif (err < 0) {\n+\t\treftable_free(ti);\n+\t\treturn err;\n+\t}\n+\n+\tfilter = reftable_malloc(sizeof(struct filtering_ref_iterator));\n+\t*filter = empty;\n+\n+\tstrbuf_add(&filter->oid, oid, oid_len);\n+\treftable_table_from_reader(&filter->tab, r);\n+\tfilter->double_check = 0;\n+\titerator_from_table_iter(&filter->it, ti);\n+\n+\titerator_from_filtering_ref_iterator(it, filter);\n+\treturn 0;\n+}\n+\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid)\n+{\n+\tif (r->obj_offsets.is_present)\n+\t\treturn reftable_reader_refs_for_indexed(r, it, oid);\n+\treturn reftable_reader_refs_for_unindexed(r, it, oid);\n+}\n+\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r)\n+{\n+\treturn r->max_update_index;\n+}\n+\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r)\n+{\n+\treturn r->min_update_index;\n+}\ndiff --git a/reftable/reader.h b/reftable/reader.h\nnew file mode 100644\nindex 00000000000..6d4927e1c5d\n--- /dev/null\n+++ b/reftable/reader.h\n@@ -0,0 +1,75 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef READER_H\n+#define READER_H\n+\n+#include \"block.h\"\n+#include \"record.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source);\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size);\n+void block_source_close(struct reftable_block_source *source);\n+\n+/* metadata for a block type */\n+struct reftable_reader_offsets {\n+\tint is_present;\n+\tuint64_t offset;\n+\tuint64_t index_offset;\n+};\n+\n+/* The state for reading a reftable file. */\n+struct reftable_reader {\n+\t/* for convience, associate a name with the instance. */\n+\tchar *name;\n+\tstruct reftable_block_source source;\n+\n+\t/* Size of the file, excluding the footer. */\n+\tuint64_t size;\n+\n+\t/* 'sha1' for SHA1, 's256' for SHA-256 */\n+\tuint32_t hash_id;\n+\n+\tuint32_t block_size;\n+\tuint64_t min_update_index;\n+\tuint64_t max_update_index;\n+\t/* Length of the OID keys in the 'o' section */\n+\tint object_id_len;\n+\tint version;\n+\n+\tstruct reftable_reader_offsets ref_offsets;\n+\tstruct reftable_reader_offsets obj_offsets;\n+\tstruct reftable_reader_offsets log_offsets;\n+};\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name);\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec);\n+void reader_close(struct reftable_reader *r);\n+const char *reader_name(struct reftable_reader *r);\n+\n+/* initialize a block reader to read from `r` */\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ);\n+\n+/* generic interface to reftables */\n+struct reftable_table_vtable {\n+\tint (*seek_record)(void *tab, struct reftable_iterator *it,\n+\t\t\t struct reftable_record *);\n+\tuint32_t (*hash_id)(void *tab);\n+\tuint64_t (*min_update_index)(void *tab);\n+\tuint64_t (*max_update_index)(void *tab);\n+};\n+\n+#endif\ndiff --git a/reftable/reftable-iterator.h b/reftable/reftable-iterator.h\nnew file mode 100644\nindex 00000000000..d8251a45ba1\n--- /dev/null\n+++ b/reftable/reftable-iterator.h\n@@ -0,0 +1,37 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_ITERATOR_H\n+#define REFTABLE_ITERATOR_H\n+\n+#include \"reftable-record.h\"\n+\n+/* iterator is the generic interface for walking over data stored in a\n+ * reftable.\n+ */\n+struct reftable_iterator {\n+\tstruct reftable_iterator_vtable *ops;\n+\tvoid *iter_arg;\n+};\n+\n+/* reads the next reftable_ref_record. Returns < 0 for error, 0 for OK and > 0:\n+ * end of iteration.\n+ */\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* reads the next reftable_log_record. Returns < 0 for error, 0 for OK and > 0:\n+ * end of iteration.\n+ */\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* releases resources associated with an iterator. */\n+void reftable_iterator_destroy(struct reftable_iterator *it);\n+\n+#endif\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nnew file mode 100644\nindex 00000000000..804e28543b3\n--- /dev/null\n+++ b/reftable/reftable-reader.h\n@@ -0,0 +1,89 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_READER_H\n+#define REFTABLE_READER_H\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-blocksource.h\"\n+\n+/*\n+ * Reading single tables\n+ *\n+ * The follow routines are for reading single files. For an application-level\n+ * interface, skip ahead to struct reftable_merged_table and struct\n+ * reftable_stack.\n+ */\n+\n+/* The reader struct is a handle to an open reftable file. */\n+struct reftable_reader;\n+\n+/* reftable_new_reader opens a reftable for reading. If successful, returns 0\n+ * code and sets pp. The name is used for creating a stack. Typically, it is the\n+ * basename of the file. The block source `src` is owned by the reader, and is\n+ * closed on calling reftable_reader_destroy().\n+ */\n+int reftable_new_reader(struct reftable_reader **pp,\n+\t\t\tstruct reftable_block_source *src, const char *name);\n+\n+/* reftable_reader_seek_ref returns an iterator where 'name' would be inserted\n+ in the table. To seek to the start of the table, use name = \"\".\n+\n+ example:\n+\n+ struct reftable_reader *r = NULL;\n+ int err = reftable_new_reader(&r, &src, \"filename\");\n+ if (err < 0) { ... }\n+ struct reftable_iterator it = {0};\n+ err = reftable_reader_seek_ref(r, &it, \"refs/heads/master\");\n+ if (err < 0) { ... }\n+ struct reftable_ref_record ref = {0};\n+ while (1) {\n+ err = reftable_iterator_next_ref(&it, &ref);\n+ if (err > 0) {\n+ break;\n+ }\n+ if (err < 0) {\n+ ..error handling..\n+ }\n+ ..found..\n+ }\n+ reftable_iterator_destroy(&it);\n+ reftable_ref_record_release(&ref);\n+ */\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* returns the hash ID used in this table. */\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r);\n+\n+/* seek to logs for the given name, older than update_index. To seek to the\n+ start of the table, use name = \"\".\n+ */\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index);\n+\n+/* seek to newest log entry for given name. */\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* closes and deallocates a reader. */\n+void reftable_reader_free(struct reftable_reader *);\n+\n+/* return an iterator for the refs pointing to `oid`. */\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid);\n+\n+/* return the max_update_index for a table */\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r);\n+\n+/* return the min_update_index for a table */\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"411862","messageId":"6968dbc3828f22369cc92bf56e3a7855769415fa.1607522429.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v4.git.git.1607522429.gitgitgadget@gmail.com","subject":"[PATCH v4 08/15] reftable: a generic binary tree implementation","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-12-09T14:00:22Z","receivedAt":"2020-12-09T14:03:37Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format includes support for an (OID => ref) map. This map can speed\nup visibility and reachability checks. In particular, various operations along\nthe fetch/push path within Gerrit have ben sped up by using this structure.\n\nThe map is constructed with help of a binary tree. Object IDs are hashes, so\nthey are uniformly distributed. Hence, the tree does not attempt forced\nrebalancing.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 4 ++-\n reftable/tree.c | 63 ++++++++++++++++++++++++++++++++++++++++\n reftable/tree.h | 34 ++++++++++++++++++++++\n reftable/tree_test.c | 61 ++++++++++++++++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 162 insertions(+), 1 deletion(-)\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n\ndiff --git a/Makefile b/Makefile\nindex abe0dda5777..734c05655b2 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2395,12 +2395,14 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/zlib-compat.o\n \n+REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n-REFTABLE_TEST_OBJS += reftable/basics_test.o\n+REFTABLE_TEST_OBJS += reftable/tree_test.o\n \n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \\\ndiff --git a/reftable/tree.c b/reftable/tree.c\nnew file mode 100644\nindex 00000000000..0061d14e306\n--- /dev/null\n+++ b/reftable/tree.c\n@@ -0,0 +1,63 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert)\n+{\n+\tint res;\n+\tif (*rootp == NULL) {\n+\t\tif (!insert) {\n+\t\t\treturn NULL;\n+\t\t} else {\n+\t\t\tstruct tree_node *n =\n+\t\t\t\treftable_calloc(sizeof(struct tree_node));\n+\t\t\tn->key = key;\n+\t\t\t*rootp = n;\n+\t\t\treturn *rootp;\n+\t\t}\n+\t}\n+\n+\tres = compare(key, (*rootp)->key);\n+\tif (res < 0)\n+\t\treturn tree_search(key, &(*rootp)->left, compare, insert);\n+\telse if (res > 0)\n+\t\treturn tree_search(key, &(*rootp)->right, compare, insert);\n+\treturn *rootp;\n+}\n+\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg)\n+{\n+\tif (t->left != NULL) {\n+\t\tinfix_walk(t->left, action, arg);\n+\t}\n+\taction(arg, t->key);\n+\tif (t->right != NULL) {\n+\t\tinfix_walk(t->right, action, arg);\n+\t}\n+}\n+\n+void tree_free(struct tree_node *t)\n+{\n+\tif (t == NULL) {\n+\t\treturn;\n+\t}\n+\tif (t->left != NULL) {\n+\t\ttree_free(t->left);\n+\t}\n+\tif (t->right != NULL) {\n+\t\ttree_free(t->right);\n+\t}\n+\treftable_free(t);\n+}\ndiff --git a/reftable/tree.h b/reftable/tree.h\nnew file mode 100644\nindex 00000000000..fbdd002e23a\n--- /dev/null\n+++ b/reftable/tree.h\n@@ -0,0 +1,34 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TREE_H\n+#define TREE_H\n+\n+/* tree_node is a generic binary search tree. */\n+struct tree_node {\n+\tvoid *key;\n+\tstruct tree_node *left, *right;\n+};\n+\n+/* looks for `key` in `rootp` using `compare` as comparison function. If insert\n+ * is set, insert the key if it's not found. Else, return NULL.\n+ */\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert);\n+\n+/* performs an infix walk of the tree. */\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg);\n+\n+/*\n+ * deallocates the tree nodes recursively. Keys should be deallocated separately\n+ * by walking over the tree. */\n+void tree_free(struct tree_node *t);\n+\n+#endif\ndiff --git a/reftable/tree_test.c b/reftable/tree_test.c\nnew file mode 100644\nindex 00000000000..26d1e694252\n--- /dev/null\n+++ b/reftable/tree_test.c\n@@ -0,0 +1,61 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static int test_compare(const void *a, const void *b)\n+{\n+\treturn (char *)a - (char *)b;\n+}\n+\n+struct curry {\n+\tvoid *last;\n+};\n+\n+static void check_increasing(void *arg, void *key)\n+{\n+\tstruct curry *c = (struct curry *)arg;\n+\tif (c->last != NULL) {\n+\t\tassert(test_compare(c->last, key) < 0);\n+\t}\n+\tc->last = key;\n+}\n+\n+static void test_tree(void)\n+{\n+\tstruct tree_node *root = NULL;\n+\n+\tvoid *values[11] = { NULL };\n+\tstruct tree_node *nodes[11] = { NULL };\n+\tint i = 1;\n+\tstruct curry c = { NULL };\n+\tdo {\n+\t\tnodes[i] = tree_search(values + i, &root, &test_compare, 1);\n+\t\ti = (i * 7) % 11;\n+\t} while (i != 1);\n+\n+\tfor (i = 1; i < ARRAY_SIZE(nodes); i++) {\n+\t\tassert(values + i == nodes[i]->key);\n+\t\tassert(nodes[i] ==\n+\t\t tree_search(values + i, &root, &test_compare, 0));\n+\t}\n+\n+\tinfix_walk(root, check_increasing, &c);\n+\ttree_free(root);\n+}\n+\n+int tree_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_tree);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex c9deeaf08c7..050551fa698 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,5 +6,6 @@ int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"411863","messageId":"e13afe3152585d95a4020fb462c4fface1a38d4b.1607522429.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v4.git.git.1607522429.gitgitgadget@gmail.com","subject":"[PATCH v4 06/15] reftable: (de)serialization for the polymorphic record type.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-12-09T14:00:20Z","receivedAt":"2020-12-09T14:03:38Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is structured as a sequence of blocks, and each block\ncontains a sequence of prefix-compressed key-value records. There are 4 types of\nrecords, and they have similarities in how they must be handled. This is\nachieved by introducing a polymorphic 'record' type that encapsulates ref, log,\nindex and object records.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/constants.h | 21 +\n reftable/record.c | 1157 ++++++++++++++++++++++++++++++++++++\n reftable/record.h | 139 +++++\n reftable/record_test.c | 398 +++++++++++++\n reftable/reftable-record.h | 100 ++++\n t/helper/test-reftable.c | 2 +-\n 7 files changed, 1818 insertions(+), 1 deletion(-)\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/reftable-record.h\n\ndiff --git a/Makefile b/Makefile\nindex f623611bde2..ce3052169b4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2393,7 +2393,9 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/record.o\n \n+REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n \ndiff --git a/reftable/constants.h b/reftable/constants.h\nnew file mode 100644\nindex 00000000000..5eee72c4c11\n--- /dev/null\n+++ b/reftable/constants.h\n@@ -0,0 +1,21 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef CONSTANTS_H\n+#define CONSTANTS_H\n+\n+#define BLOCK_TYPE_LOG 'g'\n+#define BLOCK_TYPE_INDEX 'i'\n+#define BLOCK_TYPE_REF 'r'\n+#define BLOCK_TYPE_OBJ 'o'\n+#define BLOCK_TYPE_ANY 0\n+\n+#define MAX_RESTARTS ((1 << 16) - 1)\n+#define DEFAULT_BLOCK_SIZE 4096\n+\n+#endif\ndiff --git a/reftable/record.c b/reftable/record.c\nnew file mode 100644\nindex 00000000000..5381e9aaa89\n--- /dev/null\n+++ b/reftable/record.c\n@@ -0,0 +1,1157 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+/* record.c - methods for different types of records. */\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"constants.h\"\n+#include \"reftable-error.h\"\n+#include \"basics.h\"\n+\n+int get_var_int(uint64_t *dest, struct string_view *in)\n+{\n+\tint ptr = 0;\n+\tuint64_t val;\n+\n+\tif (in->len == 0)\n+\t\treturn -1;\n+\tval = in->buf[ptr] & 0x7f;\n+\n+\twhile (in->buf[ptr] & 0x80) {\n+\t\tptr++;\n+\t\tif (ptr > in->len) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tval = (val + 1) << 7 | (uint64_t)(in->buf[ptr] & 0x7f);\n+\t}\n+\n+\t*dest = val;\n+\treturn ptr + 1;\n+}\n+\n+int put_var_int(struct string_view *dest, uint64_t val)\n+{\n+\tuint8_t buf[10] = { 0 };\n+\tint i = 9;\n+\tint n = 0;\n+\tbuf[i] = (uint8_t)(val & 0x7f);\n+\ti--;\n+\twhile (1) {\n+\t\tval >>= 7;\n+\t\tif (!val) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tval--;\n+\t\tbuf[i] = 0x80 | (uint8_t)(val & 0x7f);\n+\t\ti--;\n+\t}\n+\n+\tn = sizeof(buf) - i - 1;\n+\tif (dest->len < n)\n+\t\treturn -1;\n+\tmemcpy(dest->buf, &buf[i + 1], n);\n+\treturn n;\n+}\n+\n+int reftable_is_block_type(uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\tcase BLOCK_TYPE_LOG:\n+\tcase BLOCK_TYPE_OBJ:\n+\tcase BLOCK_TYPE_INDEX:\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec)\n+{\n+\tswitch (rec->value_type) {\n+\tcase REFTABLE_REF_VAL1:\n+\t\treturn rec->value.val1;\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn rec->value.val2.value;\n+\tdefault:\n+\t\treturn NULL;\n+\t}\n+}\n+\n+uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec)\n+{\n+\tswitch (rec->value_type) {\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn rec->value.val2.target_value;\n+\tdefault:\n+\t\treturn NULL;\n+\t}\n+}\n+\n+static int decode_string(struct strbuf *dest, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t tsize = 0;\n+\tint n = get_var_int(&tsize, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\tif (in.len < tsize)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, in.buf, tsize);\n+\tstring_view_consume(&in, tsize);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static int encode_string(char *str, struct string_view s)\n+{\n+\tstruct string_view start = s;\n+\tint l = strlen(str);\n+\tint n = put_var_int(&s, l);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\tif (s.len < l)\n+\t\treturn -1;\n+\tmemcpy(s.buf, str, l);\n+\tstring_view_consume(&s, l);\n+\n+\treturn start.len - s.len;\n+}\n+\n+int reftable_encode_key(int *restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra)\n+{\n+\tstruct string_view start = dest;\n+\tint prefix_len = common_prefix_size(&prev_key, &key);\n+\tuint64_t suffix_len = key.len - prefix_len;\n+\tint n = put_var_int(&dest, (uint64_t)prefix_len);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\t*restart = (prefix_len == 0);\n+\n+\tn = put_var_int(&dest, suffix_len << 3 | (uint64_t)extra);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\tif (dest.len < suffix_len)\n+\t\treturn -1;\n+\tmemcpy(dest.buf, key.buf + prefix_len, suffix_len);\n+\tstring_view_consume(&dest, suffix_len);\n+\n+\treturn start.len - dest.len;\n+}\n+\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t prefix_len = 0;\n+\tuint64_t suffix_len = 0;\n+\tint n = get_var_int(&prefix_len, &in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tif (prefix_len > last_key.len)\n+\t\treturn -1;\n+\n+\tn = get_var_int(&suffix_len, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\t*extra = (uint8_t)(suffix_len & 0x7);\n+\tsuffix_len >>= 3;\n+\n+\tif (in.len < suffix_len)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(key);\n+\tstrbuf_add(key, last_key.buf, prefix_len);\n+\tstrbuf_add(key, in.buf, suffix_len);\n+\tstring_view_consume(&in, suffix_len);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static void reftable_ref_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_ref_record *rec =\n+\t\t(const struct reftable_ref_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, rec->refname);\n+}\n+\n+static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *ref = (struct reftable_ref_record *)rec;\n+\tstruct reftable_ref_record *src = (struct reftable_ref_record *)src_rec;\n+\tassert(hash_size > 0);\n+\n+\t/* This is simple and correct, but we could probably reuse the hash\n+\t * fields. */\n+\treftable_ref_record_release(ref);\n+\tif (src->refname != NULL) {\n+\t\tref->refname = xstrdup(src->refname);\n+\t}\n+\tref->update_index = src->update_index;\n+\tref->value_type = src->value_type;\n+\tswitch (src->value_type) {\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\tref->value.val1 = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val1, src->value.val1, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\tref->value.val2.value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val2.value, src->value.val2.value, hash_size);\n+\t\tref->value.val2.target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val2.target_value,\n+\t\t src->value.val2.target_value, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tref->value.symref = xstrdup(src->value.symref);\n+\t\tbreak;\n+\t}\n+}\n+\n+static char hexdigit(int c)\n+{\n+\tif (c <= 9)\n+\t\treturn '0' + c;\n+\treturn 'a' + (c - 10);\n+}\n+\n+static void hex_format(char *dest, uint8_t *src, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (src != NULL) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < hash_size; i++) {\n+\t\t\tdest[2 * i] = hexdigit(src[i] >> 4);\n+\t\t\tdest[2 * i + 1] = hexdigit(src[i] & 0xf);\n+\t\t}\n+\t\tdest[2 * hash_size] = 0;\n+\t}\n+}\n+\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[2 * SHA256_SIZE + 1] = { 0 }; /* BUG */\n+\tprintf(\"ref{%s(%\" PRIu64 \") \", ref->refname, ref->update_index);\n+\tswitch (ref->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tprintf(\"=> %s\", ref->value.symref);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\thex_format(hex, ref->value.val2.value, hash_size(hash_id));\n+\t\tprintf(\"val 2 %s\", hex);\n+\t\thex_format(hex, ref->value.val2.target_value,\n+\t\t\t hash_size(hash_id));\n+\t\tprintf(\"(T %s)\", hex);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\thex_format(hex, ref->value.val1, hash_size(hash_id));\n+\t\tprintf(\"val 1 %s\", hex);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tprintf(\"delete\");\n+\t\tbreak;\n+\t}\n+\tprintf(\"}\\n\");\n+}\n+\n+static void reftable_ref_record_release_void(void *rec)\n+{\n+\treftable_ref_record_release((struct reftable_ref_record *)rec);\n+}\n+\n+void reftable_ref_record_release(struct reftable_ref_record *ref)\n+{\n+\tswitch (ref->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\treftable_free(ref->value.symref);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\treftable_free(ref->value.val2.target_value);\n+\t\treftable_free(ref->value.val2.value);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\treftable_free(ref->value.val1);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t}\n+\n+\treftable_free(ref->refname);\n+\tmemset(ref, 0, sizeof(struct reftable_ref_record));\n+}\n+\n+static uint8_t reftable_ref_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\treturn r->value_type;\n+}\n+\n+static int reftable_ref_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\tstruct string_view start = s;\n+\tint n = put_var_int(&s, r->update_index);\n+\tassert(hash_size > 0);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tswitch (r->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tn = encode_string(r->value.symref, s);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\tif (s.len < 2 * hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value.val2.value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tmemcpy(s.buf, r->value.val2.target_value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\tif (s.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value.val1, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t}\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_ref_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *r = (struct reftable_ref_record *)rec;\n+\tstruct string_view start = in;\n+\tuint64_t update_index = 0;\n+\tint n = get_var_int(&update_index, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\treftable_ref_record_release(r);\n+\n+\tassert(hash_size > 0);\n+\n+\tr->refname = reftable_realloc(r->refname, key.len + 1);\n+\tmemcpy(r->refname, key.buf, key.len);\n+\tr->update_index = update_index;\n+\tr->refname[key.len] = 0;\n+\tr->value_type = val_type;\n+\tswitch (val_type) {\n+\tcase REFTABLE_REF_VAL1:\n+\t\tif (in.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tr->value.val1 = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val1, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\n+\tcase REFTABLE_REF_VAL2:\n+\t\tif (in.len < 2 * hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tr->value.val2.value = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val2.value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\n+\t\tr->value.val2.target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val2.target_value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\n+\tcase REFTABLE_REF_SYMREF: {\n+\t\tstruct strbuf dest = STRBUF_INIT;\n+\t\tint n = decode_string(&dest, in);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\t\tr->value.symref = dest.buf;\n+\t} break;\n+\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t\tbreak;\n+\t}\n+\n+\treturn start.len - in.len;\n+}\n+\n+static int reftable_ref_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_ref_record_is_deletion(\n+\t\t(const struct reftable_ref_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_ref_record_vtable = {\n+\t.key = &reftable_ref_record_key,\n+\t.type = BLOCK_TYPE_REF,\n+\t.copy_from = &reftable_ref_record_copy_from,\n+\t.val_type = &reftable_ref_record_val_type,\n+\t.encode = &reftable_ref_record_encode,\n+\t.decode = &reftable_ref_record_decode,\n+\t.release = &reftable_ref_record_release_void,\n+\t.is_deletion = &reftable_ref_record_is_deletion_void,\n+};\n+\n+static void reftable_obj_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_obj_record *rec =\n+\t\t(const struct reftable_obj_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, rec->hash_prefix, rec->hash_prefix_len);\n+}\n+\n+static void reftable_obj_record_release(void *rec)\n+{\n+\tstruct reftable_obj_record *obj = (struct reftable_obj_record *)rec;\n+\tFREE_AND_NULL(obj->hash_prefix);\n+\tFREE_AND_NULL(obj->offsets);\n+\tmemset(obj, 0, sizeof(struct reftable_obj_record));\n+}\n+\n+static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_obj_record *obj = (struct reftable_obj_record *)rec;\n+\tconst struct reftable_obj_record *src =\n+\t\t(const struct reftable_obj_record *)src_rec;\n+\tint olen;\n+\n+\treftable_obj_record_release(obj);\n+\t*obj = *src;\n+\tobj->hash_prefix = reftable_malloc(obj->hash_prefix_len);\n+\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n+\n+\tolen = obj->offset_len * sizeof(uint64_t);\n+\tobj->offsets = reftable_malloc(olen);\n+\tmemcpy(obj->offsets, src->offsets, olen);\n+}\n+\n+static uint8_t reftable_obj_record_val_type(const void *rec)\n+{\n+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n+\tif (r->offset_len > 0 && r->offset_len < 8)\n+\t\treturn r->offset_len;\n+\treturn 0;\n+}\n+\n+static int reftable_obj_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n+\tstruct string_view start = s;\n+\tint i = 0;\n+\tint n = 0;\n+\tuint64_t last = 0;\n+\tif (r->offset_len == 0 || r->offset_len >= 8) {\n+\t\tn = put_var_int(&s, r->offset_len);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t}\n+\tif (r->offset_len == 0)\n+\t\treturn start.len - s.len;\n+\tn = put_var_int(&s, r->offsets[0]);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tlast = r->offsets[0];\n+\tfor (i = 1; i < r->offset_len; i++) {\n+\t\tint n = put_var_int(&s, r->offsets[i] - last);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tlast = r->offsets[i];\n+\t}\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_obj_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n+\tuint64_t count = val_type;\n+\tint n = 0;\n+\tuint64_t last;\n+\tint j;\n+\tr->hash_prefix = reftable_malloc(key.len);\n+\tmemcpy(r->hash_prefix, key.buf, key.len);\n+\tr->hash_prefix_len = key.len;\n+\n+\tif (val_type == 0) {\n+\t\tn = get_var_int(&count, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\n+\t\tstring_view_consume(&in, n);\n+\t}\n+\n+\tr->offsets = NULL;\n+\tr->offset_len = 0;\n+\tif (count == 0)\n+\t\treturn start.len - in.len;\n+\n+\tr->offsets = reftable_malloc(count * sizeof(uint64_t));\n+\tr->offset_len = count;\n+\n+\tn = get_var_int(&r->offsets[0], &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\tlast = r->offsets[0];\n+\tj = 1;\n+\twhile (j < count) {\n+\t\tuint64_t delta = 0;\n+\t\tint n = get_var_int(&delta, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\n+\t\tlast = r->offsets[j] = (delta + last);\n+\t\tj++;\n+\t}\n+\treturn start.len - in.len;\n+}\n+\n+static int not_a_deletion(const void *p)\n+{\n+\treturn 0;\n+}\n+\n+static struct reftable_record_vtable reftable_obj_record_vtable = {\n+\t.key = &reftable_obj_record_key,\n+\t.type = BLOCK_TYPE_OBJ,\n+\t.copy_from = &reftable_obj_record_copy_from,\n+\t.val_type = &reftable_obj_record_val_type,\n+\t.encode = &reftable_obj_record_encode,\n+\t.decode = &reftable_obj_record_decode,\n+\t.release = &reftable_obj_record_release,\n+\t.is_deletion = not_a_deletion,\n+};\n+\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[SHA256_SIZE + 1] = { 0 };\n+\n+\tprintf(\"log{%s(%\" PRIu64 \") %s <%s> %\" PRIu64 \" %04d\\n\", log->refname,\n+\t log->update_index, log->name, log->email, log->time,\n+\t log->tz_offset);\n+\thex_format(hex, log->old_hash, hash_size(hash_id));\n+\tprintf(\"%s => \", hex);\n+\thex_format(hex, log->new_hash, hash_size(hash_id));\n+\tprintf(\"%s\\n\\n%s\\n}\\n\", hex, log->message);\n+}\n+\n+static void reftable_log_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_log_record *rec =\n+\t\t(const struct reftable_log_record *)r;\n+\tint len = strlen(rec->refname);\n+\tuint8_t i64[8];\n+\tuint64_t ts = 0;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, (uint8_t *)rec->refname, len + 1);\n+\n+\tts = (~ts) - rec->update_index;\n+\tput_be64(&i64[0], ts);\n+\tstrbuf_add(dest, i64, sizeof(i64));\n+}\n+\n+static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_log_record *dst = (struct reftable_log_record *)rec;\n+\tconst struct reftable_log_record *src =\n+\t\t(const struct reftable_log_record *)src_rec;\n+\n+\treftable_log_record_release(dst);\n+\t*dst = *src;\n+\tif (dst->refname != NULL) {\n+\t\tdst->refname = xstrdup(dst->refname);\n+\t}\n+\tif (dst->email != NULL) {\n+\t\tdst->email = xstrdup(dst->email);\n+\t}\n+\tif (dst->name != NULL) {\n+\t\tdst->name = xstrdup(dst->name);\n+\t}\n+\tif (dst->message != NULL) {\n+\t\tdst->message = xstrdup(dst->message);\n+\t}\n+\n+\tif (dst->new_hash != NULL) {\n+\t\tdst->new_hash = reftable_malloc(hash_size);\n+\t\tmemcpy(dst->new_hash, src->new_hash, hash_size);\n+\t}\n+\tif (dst->old_hash != NULL) {\n+\t\tdst->old_hash = reftable_malloc(hash_size);\n+\t\tmemcpy(dst->old_hash, src->old_hash, hash_size);\n+\t}\n+}\n+\n+static void reftable_log_record_release_void(void *rec)\n+{\n+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n+\treftable_log_record_release(r);\n+}\n+\n+void reftable_log_record_release(struct reftable_log_record *r)\n+{\n+\treftable_free(r->refname);\n+\treftable_free(r->new_hash);\n+\treftable_free(r->old_hash);\n+\treftable_free(r->name);\n+\treftable_free(r->email);\n+\treftable_free(r->message);\n+\tmemset(r, 0, sizeof(struct reftable_log_record));\n+}\n+\n+static uint8_t reftable_log_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_log_record *log =\n+\t\t(const struct reftable_log_record *)rec;\n+\n+\treturn reftable_log_record_is_deletion(log) ? 0 : 1;\n+}\n+\n+static uint8_t zero[SHA256_SIZE] = { 0 };\n+\n+static int reftable_log_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n+\tstruct string_view start = s;\n+\tint n = 0;\n+\tuint8_t *oldh = r->old_hash;\n+\tuint8_t *newh = r->new_hash;\n+\tif (reftable_log_record_is_deletion(r))\n+\t\treturn 0;\n+\n+\tif (oldh == NULL) {\n+\t\toldh = zero;\n+\t}\n+\tif (newh == NULL) {\n+\t\tnewh = zero;\n+\t}\n+\n+\tif (s.len < 2 * hash_size)\n+\t\treturn -1;\n+\n+\tmemcpy(s.buf, oldh, hash_size);\n+\tmemcpy(s.buf + hash_size, newh, hash_size);\n+\tstring_view_consume(&s, 2 * hash_size);\n+\n+\tn = encode_string(r->name ? r->name : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = encode_string(r->email ? r->email : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = put_var_int(&s, r->time);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tif (s.len < 2)\n+\t\treturn -1;\n+\n+\tput_be16(s.buf, r->tz_offset);\n+\tstring_view_consume(&s, 2);\n+\n+\tn = encode_string(r->message ? r->message : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_log_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n+\tuint64_t max = 0;\n+\tuint64_t ts = 0;\n+\tstruct strbuf dest = STRBUF_INIT;\n+\tint n;\n+\n+\tif (key.len <= 9 || key.buf[key.len - 9] != 0)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->refname = reftable_realloc(r->refname, key.len - 8);\n+\tmemcpy(r->refname, key.buf, key.len - 8);\n+\tts = get_be64(key.buf + key.len - 8);\n+\n+\tr->update_index = (~max) - ts;\n+\n+\tif (val_type == 0) {\n+\t\tFREE_AND_NULL(r->old_hash);\n+\t\tFREE_AND_NULL(r->new_hash);\n+\t\tFREE_AND_NULL(r->message);\n+\t\tFREE_AND_NULL(r->email);\n+\t\tFREE_AND_NULL(r->name);\n+\t\treturn 0;\n+\t}\n+\n+\tif (in.len < 2 * hash_size)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->old_hash = reftable_realloc(r->old_hash, hash_size);\n+\tr->new_hash = reftable_realloc(r->new_hash, hash_size);\n+\n+\tmemcpy(r->old_hash, in.buf, hash_size);\n+\tmemcpy(r->new_hash, in.buf + hash_size, hash_size);\n+\n+\tstring_view_consume(&in, 2 * hash_size);\n+\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->name = reftable_realloc(r->name, dest.len + 1);\n+\tmemcpy(r->name, dest.buf, dest.len);\n+\tr->name[dest.len] = 0;\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->email = reftable_realloc(r->email, dest.len + 1);\n+\tmemcpy(r->email, dest.buf, dest.len);\n+\tr->email[dest.len] = 0;\n+\n+\tts = 0;\n+\tn = get_var_int(&ts, &in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\tr->time = ts;\n+\tif (in.len < 2)\n+\t\tgoto done;\n+\n+\tr->tz_offset = get_be16(in.buf);\n+\tstring_view_consume(&in, 2);\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->message = reftable_realloc(r->message, dest.len + 1);\n+\tmemcpy(r->message, dest.buf, dest.len);\n+\tr->message[dest.len] = 0;\n+\n+\tstrbuf_release(&dest);\n+\treturn start.len - in.len;\n+\n+done:\n+\tstrbuf_release(&dest);\n+\treturn REFTABLE_FORMAT_ERROR;\n+}\n+\n+static int null_streq(char *a, char *b)\n+{\n+\tchar *empty = \"\";\n+\tif (a == NULL)\n+\t\ta = empty;\n+\n+\tif (b == NULL)\n+\t\tb = empty;\n+\n+\treturn 0 == strcmp(a, b);\n+}\n+\n+static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)\n+{\n+\tif (a == NULL)\n+\t\ta = zero;\n+\n+\tif (b == NULL)\n+\t\tb = zero;\n+\n+\treturn !memcmp(a, b, sz);\n+}\n+\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size)\n+{\n+\treturn null_streq(a->name, b->name) && null_streq(a->email, b->email) &&\n+\t null_streq(a->message, b->message) &&\n+\t zero_hash_eq(a->old_hash, b->old_hash, hash_size) &&\n+\t zero_hash_eq(a->new_hash, b->new_hash, hash_size) &&\n+\t a->time == b->time && a->tz_offset == b->tz_offset &&\n+\t a->update_index == b->update_index;\n+}\n+\n+static int reftable_log_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_log_record_is_deletion(\n+\t\t(const struct reftable_log_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_log_record_vtable = {\n+\t.key = &reftable_log_record_key,\n+\t.type = BLOCK_TYPE_LOG,\n+\t.copy_from = &reftable_log_record_copy_from,\n+\t.val_type = &reftable_log_record_val_type,\n+\t.encode = &reftable_log_record_encode,\n+\t.decode = &reftable_log_record_decode,\n+\t.release = &reftable_log_record_release_void,\n+\t.is_deletion = &reftable_log_record_is_deletion_void,\n+};\n+\n+struct reftable_record reftable_new_record(uint8_t typ)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF: {\n+\t\tstruct reftable_ref_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_ref_record));\n+\t\treftable_record_from_ref(&rec, r);\n+\t\treturn rec;\n+\t}\n+\n+\tcase BLOCK_TYPE_OBJ: {\n+\t\tstruct reftable_obj_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_obj_record));\n+\t\treftable_record_from_obj(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_LOG: {\n+\t\tstruct reftable_log_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_log_record));\n+\t\treftable_record_from_log(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_INDEX: {\n+\t\tstruct reftable_index_record empty = { .last_key =\n+\t\t\t\t\t\t\t STRBUF_INIT };\n+\t\tstruct reftable_index_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_index_record));\n+\t\t*r = empty;\n+\t\treftable_record_from_index(&rec, r);\n+\t\treturn rec;\n+\t}\n+\t}\n+\tabort();\n+\treturn rec;\n+}\n+\n+/* clear out the record, yielding the reftable_record data that was\n+ * encapsulated. */\n+static void *reftable_record_yield(struct reftable_record *rec)\n+{\n+\tvoid *p = rec->data;\n+\trec->data = NULL;\n+\treturn p;\n+}\n+\n+void reftable_record_destroy(struct reftable_record *rec)\n+{\n+\treftable_record_release(rec);\n+\treftable_free(reftable_record_yield(rec));\n+}\n+\n+static void reftable_index_record_key(const void *r, struct strbuf *dest)\n+{\n+\tstruct reftable_index_record *rec = (struct reftable_index_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addbuf(dest, &rec->last_key);\n+}\n+\n+static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_index_record *dst = (struct reftable_index_record *)rec;\n+\tstruct reftable_index_record *src =\n+\t\t(struct reftable_index_record *)src_rec;\n+\n+\tstrbuf_reset(&dst->last_key);\n+\tstrbuf_addbuf(&dst->last_key, &src->last_key);\n+\tdst->offset = src->offset;\n+}\n+\n+static void reftable_index_record_release(void *rec)\n+{\n+\tstruct reftable_index_record *idx = (struct reftable_index_record *)rec;\n+\tstrbuf_release(&idx->last_key);\n+}\n+\n+static uint8_t reftable_index_record_val_type(const void *rec)\n+{\n+\treturn 0;\n+}\n+\n+static int reftable_index_record_encode(const void *rec, struct string_view out,\n+\t\t\t\t\tint hash_size)\n+{\n+\tconst struct reftable_index_record *r =\n+\t\t(const struct reftable_index_record *)rec;\n+\tstruct string_view start = out;\n+\n+\tint n = put_var_int(&out, r->offset);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&out, n);\n+\n+\treturn start.len - out.len;\n+}\n+\n+static int reftable_index_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t\tuint8_t val_type, struct string_view in,\n+\t\t\t\t\tint hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_index_record *r = (struct reftable_index_record *)rec;\n+\tint n = 0;\n+\n+\tstrbuf_reset(&r->last_key);\n+\tstrbuf_addbuf(&r->last_key, &key);\n+\n+\tn = get_var_int(&r->offset, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&in, n);\n+\treturn start.len - in.len;\n+}\n+\n+static struct reftable_record_vtable reftable_index_record_vtable = {\n+\t.key = &reftable_index_record_key,\n+\t.type = BLOCK_TYPE_INDEX,\n+\t.copy_from = &reftable_index_record_copy_from,\n+\t.val_type = &reftable_index_record_val_type,\n+\t.encode = &reftable_index_record_encode,\n+\t.decode = &reftable_index_record_decode,\n+\t.release = &reftable_index_record_release,\n+\t.is_deletion = ¬_a_deletion,\n+};\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)\n+{\n+\trec->ops->key(rec->data, dest);\n+}\n+\n+uint8_t reftable_record_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->type;\n+}\n+\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size)\n+{\n+\treturn rec->ops->encode(rec->data, dest, hash_size);\n+}\n+\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size)\n+{\n+\tassert(src->ops->type == rec->ops->type);\n+\n+\trec->ops->copy_from(rec->data, src->data, hash_size);\n+}\n+\n+uint8_t reftable_record_val_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->val_type(rec->data);\n+}\n+\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src, int hash_size)\n+{\n+\treturn rec->ops->decode(rec->data, key, extra, src, hash_size);\n+}\n+\n+void reftable_record_release(struct reftable_record *rec)\n+{\n+\trec->ops->release(rec->data);\n+}\n+\n+int reftable_record_is_deletion(struct reftable_record *rec)\n+{\n+\treturn rec->ops->is_deletion(rec->data);\n+}\n+\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *ref_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = ref_rec;\n+\trec->ops = &reftable_ref_record_vtable;\n+}\n+\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *obj_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = obj_rec;\n+\trec->ops = &reftable_obj_record_vtable;\n+}\n+\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *index_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = index_rec;\n+\trec->ops = &reftable_index_record_vtable;\n+}\n+\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *log_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = log_rec;\n+\trec->ops = &reftable_log_record_vtable;\n+}\n+\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_REF);\n+\treturn (struct reftable_ref_record *)rec->data;\n+}\n+\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_LOG);\n+\treturn (struct reftable_log_record *)rec->data;\n+}\n+\n+static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)\n+{\n+\tif (a != NULL && b != NULL)\n+\t\treturn !memcmp(a, b, hash_size);\n+\n+\treturn a == b;\n+}\n+\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (!(0 == strcmp(a->refname, b->refname) &&\n+\t a->update_index == b->update_index &&\n+\t a->value_type == b->value_type))\n+\t\treturn 0;\n+\n+\tswitch (a->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\treturn !strcmp(a->value.symref, b->value.symref);\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn hash_equal(a->value.val2.value, b->value.val2.value,\n+\t\t\t\t hash_size) &&\n+\t\t hash_equal(a->value.val2.target_value,\n+\t\t\t\t b->value.val2.target_value, hash_size);\n+\tcase REFTABLE_REF_VAL1:\n+\t\treturn hash_equal(a->value.val1, b->value.val1, hash_size);\n+\tcase REFTABLE_REF_DELETION:\n+\t\treturn 1;\n+\tdefault:\n+\t\tabort();\n+\t}\n+}\n+\n+int reftable_ref_record_compare_name(const void *a, const void *b)\n+{\n+\treturn strcmp(((struct reftable_ref_record *)a)->refname,\n+\t\t ((struct reftable_ref_record *)b)->refname);\n+}\n+\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref)\n+{\n+\treturn ref->value_type == REFTABLE_REF_DELETION;\n+}\n+\n+int reftable_log_record_compare_key(const void *a, const void *b)\n+{\n+\tstruct reftable_log_record *la = (struct reftable_log_record *)a;\n+\tstruct reftable_log_record *lb = (struct reftable_log_record *)b;\n+\n+\tint cmp = strcmp(la->refname, lb->refname);\n+\tif (cmp)\n+\t\treturn cmp;\n+\tif (la->update_index > lb->update_index)\n+\t\treturn -1;\n+\treturn (la->update_index < lb->update_index) ? 1 : 0;\n+}\n+\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log)\n+{\n+\treturn (log->new_hash == NULL && log->old_hash == NULL &&\n+\t\tlog->name == NULL && log->email == NULL &&\n+\t\tlog->message == NULL && log->time == 0 && log->tz_offset == 0 &&\n+\t\tlog->message == NULL);\n+}\n+\n+void string_view_consume(struct string_view *s, int n)\n+{\n+\ts->buf += n;\n+\ts->len -= n;\n+}\ndiff --git a/reftable/record.h b/reftable/record.h\nnew file mode 100644\nindex 00000000000..498e8c50bf4\n--- /dev/null\n+++ b/reftable/record.h\n@@ -0,0 +1,139 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef RECORD_H\n+#define RECORD_H\n+\n+#include \"system.h\"\n+\n+#include \n+\n+#include \"reftable-record.h\"\n+\n+/*\n+ * A substring of existing string data. This structure takes no responsibility\n+ * for the lifetime of the data it points to.\n+ */\n+struct string_view {\n+\tuint8_t *buf;\n+\tsize_t len;\n+};\n+\n+/* Advance `s.buf` by `n`, and decrease length. */\n+void string_view_consume(struct string_view *s, int n);\n+\n+/* utilities for de/encoding varints */\n+\n+int get_var_int(uint64_t *dest, struct string_view *in);\n+int put_var_int(struct string_view *dest, uint64_t val);\n+\n+/* Methods for records. */\n+struct reftable_record_vtable {\n+\t/* encode the key of to a uint8_t strbuf. */\n+\tvoid (*key)(const void *rec, struct strbuf *dest);\n+\n+\t/* The record type of ('r' for ref). */\n+\tuint8_t type;\n+\n+\tvoid (*copy_from)(void *dest, const void *src, int hash_size);\n+\n+\t/* a value of [0..7], indicating record subvariants (eg. ref vs. symref\n+\t * vs ref deletion) */\n+\tuint8_t (*val_type)(const void *rec);\n+\n+\t/* encodes rec into dest, returning how much space was used. */\n+\tint (*encode)(const void *rec, struct string_view dest, int hash_size);\n+\n+\t/* decode data from `src` into the record. */\n+\tint (*decode)(void *rec, struct strbuf key, uint8_t extra,\n+\t\t struct string_view src, int hash_size);\n+\n+\t/* deallocate and null the record. */\n+\tvoid (*release)(void *rec);\n+\n+\t/* is this a tombstone? */\n+\tint (*is_deletion)(const void *rec);\n+};\n+\n+/* record is a generic wrapper for different types of records. */\n+struct reftable_record {\n+\tvoid *data;\n+\tstruct reftable_record_vtable *ops;\n+};\n+\n+/* returns true for recognized block types. Block start with the block type. */\n+int reftable_is_block_type(uint8_t typ);\n+\n+/* creates a malloced record of the given type. Dispose with record_destroy */\n+struct reftable_record reftable_new_record(uint8_t typ);\n+\n+/* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns\n+ * number of bytes written. */\n+int reftable_encode_key(int *is_restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra);\n+\n+/* Decode into `key` and `extra` from `in` */\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in);\n+\n+/* reftable_index_record are used internally to speed up lookups. */\n+struct reftable_index_record {\n+\tuint64_t offset; /* Offset of block */\n+\tstruct strbuf last_key; /* Last key of the block. */\n+};\n+\n+/* reftable_obj_record stores an object ID => ref mapping. */\n+struct reftable_obj_record {\n+\tuint8_t *hash_prefix; /* leading bytes of the object ID */\n+\tint hash_prefix_len; /* number of leading bytes. Constant\n+\t\t\t * across a single table. */\n+\tuint64_t *offsets; /* a vector of file offsets. */\n+\tint offset_len;\n+};\n+\n+/* see struct record_vtable */\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);\n+uint8_t reftable_record_type(struct reftable_record *rec);\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size);\n+uint8_t reftable_record_val_type(struct reftable_record *rec);\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size);\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src,\n+\t\t\t int hash_size);\n+int reftable_record_is_deletion(struct reftable_record *rec);\n+\n+/* zeroes out the embedded record */\n+void reftable_record_release(struct reftable_record *rec);\n+\n+/* clear and deallocate embedded record, and zero `rec`. */\n+void reftable_record_destroy(struct reftable_record *rec);\n+\n+/* initialize generic records from concrete records. The generic record should\n+ * be zeroed out. */\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *objrec);\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *idxrec);\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *refrec);\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *logrec);\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *ref);\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *ref);\n+\n+/* for qsort. */\n+int reftable_ref_record_compare_name(const void *a, const void *b);\n+\n+/* for qsort. */\n+int reftable_log_record_compare_key(const void *a, const void *b);\n+\n+#endif\ndiff --git a/reftable/record_test.c b/reftable/record_test.c\nnew file mode 100644\nindex 00000000000..1bfeafe7f44\n--- /dev/null\n+++ b/reftable/record_test.c\n@@ -0,0 +1,398 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_copy(struct reftable_record *rec)\n+{\n+\tstruct reftable_record copy =\n+\t\treftable_new_record(reftable_record_type(rec));\n+\treftable_record_copy_from(©, rec, SHA1_SIZE);\n+\t/* do it twice to catch memory leaks */\n+\treftable_record_copy_from(©, rec, SHA1_SIZE);\n+\tswitch (reftable_record_type(©)) {\n+\tcase BLOCK_TYPE_REF:\n+\t\tEXPECT(reftable_ref_record_equal(reftable_record_as_ref(©),\n+\t\t\t\t\t\t reftable_record_as_ref(rec),\n+\t\t\t\t\t\t SHA1_SIZE));\n+\t\tbreak;\n+\tcase BLOCK_TYPE_LOG:\n+\t\tEXPECT(reftable_log_record_equal(reftable_record_as_log(©),\n+\t\t\t\t\t\t reftable_record_as_log(rec),\n+\t\t\t\t\t\t SHA1_SIZE));\n+\t\tbreak;\n+\t}\n+\treftable_record_destroy(©);\n+}\n+\n+static void test_varint_roundtrip(void)\n+{\n+\tuint64_t inputs[] = { 0,\n+\t\t\t 1,\n+\t\t\t 27,\n+\t\t\t 127,\n+\t\t\t 128,\n+\t\t\t 257,\n+\t\t\t 4096,\n+\t\t\t ((uint64_t)1 << 63),\n+\t\t\t ((uint64_t)1 << 63) + ((uint64_t)1 << 63) - 1 };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(inputs); i++) {\n+\t\tuint8_t dest[10];\n+\n+\t\tstruct string_view out = {\n+\t\t\t.buf = dest,\n+\t\t\t.len = sizeof(dest),\n+\t\t};\n+\t\tuint64_t in = inputs[i];\n+\t\tint n = put_var_int(&out, in);\n+\t\tuint64_t got = 0;\n+\n+\t\tEXPECT(n > 0);\n+\t\tout.len = n;\n+\t\tn = get_var_int(&got, &out);\n+\t\tEXPECT(n > 0);\n+\n+\t\tEXPECT(got == in);\n+\t}\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct {\n+\t\tconst char *a, *b;\n+\t\tint want;\n+\t} cases[] = {\n+\t\t{ \"abc\", \"ab\", 2 },\n+\t\t{ \"\", \"abc\", 0 },\n+\t\t{ \"abc\", \"abd\", 2 },\n+\t\t{ \"abc\", \"pqr\", 0 },\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct strbuf a = STRBUF_INIT;\n+\t\tstruct strbuf b = STRBUF_INIT;\n+\t\tstrbuf_addstr(&a, cases[i].a);\n+\t\tstrbuf_addstr(&b, cases[i].b);\n+\t\tEXPECT(common_prefix_size(&a, &b) == cases[i].want);\n+\n+\t\tstrbuf_release(&a);\n+\t\tstrbuf_release(&b);\n+\t}\n+}\n+\n+static void set_hash(uint8_t *h, int j)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < hash_size(SHA1_ID); i++) {\n+\t\th[i] = (j >> i) & 0xff;\n+\t}\n+}\n+\n+static void test_reftable_ref_record_roundtrip(void)\n+{\n+\tint i = 0;\n+\n+\tfor (i = REFTABLE_REF_DELETION; i < REFTABLE_NR_REF_VALUETYPES; i++) {\n+\t\tstruct reftable_ref_record in = { NULL };\n+\t\tstruct reftable_ref_record out = { NULL };\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\n+\t\tint n, m;\n+\n+\t\tin.value_type = i;\n+\t\tswitch (i) {\n+\t\tcase REFTABLE_REF_DELETION:\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\tin.value.val1 = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_hash(in.value.val1, 1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\tin.value.val2.value = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_hash(in.value.val2.value, 1);\n+\t\t\tin.value.val2.target_value = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_hash(in.value.val2.target_value, 2);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF:\n+\t\t\tin.value.symref = xstrdup(\"target\");\n+\t\t\tbreak;\n+\t\t}\n+\t\tin.refname = xstrdup(\"refs/heads/master\");\n+\n+\t\treftable_record_from_ref(&rec, &in);\n+\t\ttest_copy(&rec);\n+\n+\t\tEXPECT(reftable_record_val_type(&rec) == i);\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\t\tEXPECT(n > 0);\n+\n+\t\t/* decode into a non-zero reftable_record to test for leaks. */\n+\n+\t\treftable_record_from_ref(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, i, dest, SHA1_SIZE);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(reftable_ref_record_equal(&in, &out, SHA1_SIZE));\n+\t\treftable_record_release(&rec_out);\n+\n+\t\tstrbuf_release(&key);\n+\t\treftable_ref_record_release(&in);\n+\t}\n+}\n+\n+static void test_reftable_log_record_equal(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 42,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t}\n+\t};\n+\n+\tEXPECT(!reftable_log_record_equal(&in[0], &in[1], SHA1_SIZE));\n+\tin[1].update_index = in[0].update_index;\n+\tEXPECT(reftable_log_record_equal(&in[0], &in[1], SHA1_SIZE));\n+\treftable_log_record_release(&in[0]);\n+\treftable_log_record_release(&in[1]);\n+}\n+\n+static void test_reftable_log_record_roundtrip(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.old_hash = reftable_malloc(SHA1_SIZE),\n+\t\t\t.new_hash = reftable_malloc(SHA1_SIZE),\n+\t\t\t.name = xstrdup(\"han-wen\"),\n+\t\t\t.email = xstrdup(\"hanwen@google.com\"),\n+\t\t\t.message = xstrdup(\"test\"),\n+\t\t\t.update_index = 42,\n+\t\t\t.time = 1577123507,\n+\t\t\t.tz_offset = 100,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t}\n+\t};\n+\tset_test_hash(in[0].new_hash, 1);\n+\tset_test_hash(in[0].old_hash, 2);\n+\tfor (int i = 0; i < ARRAY_SIZE(in); i++) {\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\t/* populate out, to check for leaks. */\n+\t\tstruct reftable_log_record out = {\n+\t\t\t.refname = xstrdup(\"old name\"),\n+\t\t\t.new_hash = reftable_calloc(SHA1_SIZE),\n+\t\t\t.old_hash = reftable_calloc(SHA1_SIZE),\n+\t\t\t.name = xstrdup(\"old name\"),\n+\t\t\t.email = xstrdup(\"old@email\"),\n+\t\t\t.message = xstrdup(\"old message\"),\n+\t\t};\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m, valtype;\n+\n+\t\treftable_record_from_log(&rec, &in[i]);\n+\n+\t\ttest_copy(&rec);\n+\n+\t\treftable_record_key(&rec, &key);\n+\n+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\t\tEXPECT(n >= 0);\n+\t\treftable_record_from_log(&rec_out, &out);\n+\t\tvaltype = reftable_record_val_type(&rec);\n+\t\tm = reftable_record_decode(&rec_out, key, valtype, dest,\n+\t\t\t\t\t SHA1_SIZE);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(reftable_log_record_equal(&in[i], &out, SHA1_SIZE));\n+\t\treftable_log_record_release(&in[i]);\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_release(&rec_out);\n+\t}\n+}\n+\n+static void test_u24_roundtrip(void)\n+{\n+\tuint32_t in = 0x112233;\n+\tuint8_t dest[3];\n+\tuint32_t out;\n+\tput_be24(dest, in);\n+\tout = get_be24(dest);\n+\tEXPECT(in == out);\n+}\n+\n+static void test_key_roundtrip(void)\n+{\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct strbuf roundtrip = STRBUF_INIT;\n+\tint restart;\n+\tuint8_t extra;\n+\tint n, m;\n+\tuint8_t rt_extra;\n+\n+\tstrbuf_addstr(&last_key, \"refs/heads/master\");\n+\tstrbuf_addstr(&key, \"refs/tags/bla\");\n+\textra = 6;\n+\tn = reftable_encode_key(&restart, dest, last_key, key, extra);\n+\tEXPECT(!restart);\n+\tEXPECT(n > 0);\n+\n+\tm = reftable_decode_key(&roundtrip, &rt_extra, last_key, dest);\n+\tEXPECT(n == m);\n+\tEXPECT(0 == strbuf_cmp(&key, &roundtrip));\n+\tEXPECT(rt_extra == extra);\n+\n+\tstrbuf_release(&last_key);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&roundtrip);\n+}\n+\n+static void test_reftable_obj_record_roundtrip(void)\n+{\n+\tuint8_t testHash1[SHA1_SIZE] = { 1, 2, 3, 4, 0 };\n+\tuint64_t till9[] = { 1, 2, 3, 4, 500, 600, 700, 800, 9000 };\n+\tstruct reftable_obj_record recs[3] = { {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 3,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 9,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t } };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(recs); i++) {\n+\t\tstruct reftable_obj_record in = recs[i];\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_obj_record out = { NULL };\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m;\n+\t\tuint8_t extra;\n+\n+\t\treftable_record_from_obj(&rec, &in);\n+\t\ttest_copy(&rec);\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\t\tEXPECT(n > 0);\n+\t\textra = reftable_record_val_type(&rec);\n+\t\treftable_record_from_obj(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, extra, dest,\n+\t\t\t\t\t SHA1_SIZE);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(in.hash_prefix_len == out.hash_prefix_len);\n+\t\tEXPECT(in.offset_len == out.offset_len);\n+\n+\t\tEXPECT(!memcmp(in.hash_prefix, out.hash_prefix,\n+\t\t\t in.hash_prefix_len));\n+\t\tEXPECT(0 == memcmp(in.offsets, out.offsets,\n+\t\t\t\t sizeof(uint64_t) * in.offset_len));\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_release(&rec_out);\n+\t}\n+}\n+\n+static void test_reftable_index_record_roundtrip(void)\n+{\n+\tstruct reftable_index_record in = {\n+\t\t.offset = 42,\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_index_record out = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record out_rec = { NULL };\n+\tint n, m;\n+\tuint8_t extra;\n+\n+\tstrbuf_addstr(&in.last_key, \"refs/heads/master\");\n+\treftable_record_from_index(&rec, &in);\n+\treftable_record_key(&rec, &key);\n+\ttest_copy(&rec);\n+\n+\tEXPECT(0 == strbuf_cmp(&key, &in.last_key));\n+\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\tEXPECT(n > 0);\n+\n+\textra = reftable_record_val_type(&rec);\n+\treftable_record_from_index(&out_rec, &out);\n+\tm = reftable_record_decode(&out_rec, key, extra, dest, SHA1_SIZE);\n+\tEXPECT(m == n);\n+\n+\tEXPECT(in.offset == out.offset);\n+\n+\treftable_record_release(&out_rec);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&in.last_key);\n+}\n+\n+int record_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_reftable_log_record_equal);\n+\tRUN_TEST(test_reftable_log_record_roundtrip);\n+\tRUN_TEST(test_reftable_ref_record_roundtrip);\n+\tRUN_TEST(test_varint_roundtrip);\n+\tRUN_TEST(test_key_roundtrip);\n+\tRUN_TEST(test_common_prefix);\n+\tRUN_TEST(test_reftable_obj_record_roundtrip);\n+\tRUN_TEST(test_reftable_index_record_roundtrip);\n+\tRUN_TEST(test_u24_roundtrip);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-record.h b/reftable/reftable-record.h\nnew file mode 100644\nindex 00000000000..7b1bdbf5e5f\n--- /dev/null\n+++ b/reftable/reftable-record.h\n@@ -0,0 +1,100 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_RECORD_H\n+#define REFTABLE_RECORD_H\n+\n+#include \n+\n+/*\n+ * Basic data types\n+ *\n+ * Reftables store the state of each ref in struct reftable_ref_record, and they\n+ * store a sequence of reflog updates in struct reftable_log_record.\n+ */\n+\n+/* reftable_ref_record holds a ref database entry target_value */\n+struct reftable_ref_record {\n+\tchar *refname; /* Name of the ref, malloced. */\n+\tuint64_t update_index; /* Logical timestamp at which this value is\n+\t\t\t\t* written */\n+\n+\tenum {\n+\t\t/* tombstone to hide deletions from earlier tables */\n+\t\tREFTABLE_REF_DELETION = 0x0,\n+\n+\t\t/* a simple ref */\n+\t\tREFTABLE_REF_VAL1 = 0x1,\n+\t\t/* a tag, plus its peeled hash */\n+\t\tREFTABLE_REF_VAL2 = 0x2,\n+\n+\t\t/* a symbolic reference */\n+\t\tREFTABLE_REF_SYMREF = 0x3,\n+#define REFTABLE_NR_REF_VALUETYPES 4\n+\t} value_type;\n+\tunion {\n+\t\tuint8_t *val1; /* malloced hash. */\n+\t\tstruct {\n+\t\t\tuint8_t *value; /* first value, malloced hash */\n+\t\t\tuint8_t *target_value; /* second value, malloced hash */\n+\t\t} val2;\n+\t\tchar *symref; /* referent, malloced 0-terminated string */\n+\t} value;\n+};\n+\n+/* Returns the first hash, or NULL if `rec` is not of type\n+ * REFTABLE_REF_VAL1 or REFTABLE_REF_VAL2. */\n+uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec);\n+\n+/* Returns the second hash, or NULL if `rec` is not of type\n+ * REFTABLE_REF_VAL2. */\n+uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec);\n+\n+/* returns whether 'ref' represents a deletion */\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);\n+\n+/* prints a reftable_ref_record onto stdout. Useful for debugging. */\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id);\n+\n+/* frees and nulls all pointer values inside `ref`. */\n+void reftable_ref_record_release(struct reftable_ref_record *ref);\n+\n+/* returns whether two reftable_ref_records are the same. Useful for testing. */\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size);\n+\n+/* reftable_log_record holds a reflog entry */\n+struct reftable_log_record {\n+\tchar *refname;\n+\tuint64_t update_index; /* logical timestamp of a transactional update.\n+\t\t\t\t*/\n+\tuint8_t *new_hash;\n+\tuint8_t *old_hash;\n+\tchar *name;\n+\tchar *email;\n+\tuint64_t time;\n+\tint16_t tz_offset;\n+\tchar *message;\n+};\n+\n+/* returns whether 'ref' represents the deletion of a log record. */\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log);\n+\n+/* frees and nulls all pointer values. */\n+void reftable_log_record_release(struct reftable_log_record *log);\n+\n+/* returns whether two records are equal. Useful for testing. */\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size);\n+\n+/* dumps a reftable_log_record on stdout, for debugging/testing. */\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 3b58e423e7b..09d4b83ef9b 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,6 +4,6 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n-\n+\trecord_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"411864","messageId":"a48b9937642ec7e91edb3e0917d464143420287a.1607522429.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v4.git.git.1607522429.gitgitgadget@gmail.com","subject":"[PATCH v4 07/15] reftable: reading/writing blocks","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-12-09T14:00:21Z","receivedAt":"2020-12-09T14:03:38Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is structured as a sequence of block. Within a block,\nrecords are prefix compressed, with an index of offsets for fully expand keys to\nenable binary search within blocks.\n\nThis commit provides the logic to read and write these blocks.\n\nIncludes a code snippet copied from zlib\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 3 +\n reftable/.gitattributes | 1 +\n reftable/block.c | 440 +++++++++++++++++++++++++++++++++++++++\n reftable/block.h | 127 +++++++++++\n reftable/block_test.c | 121 +++++++++++\n reftable/zlib-compat.c | 92 ++++++++\n t/helper/test-reftable.c | 1 +\n 7 files changed, 785 insertions(+)\n create mode 100644 reftable/.gitattributes\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n create mode 100644 reftable/zlib-compat.c\n\ndiff --git a/Makefile b/Makefile\nindex ce3052169b4..abe0dda5777 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2391,10 +2391,13 @@ XDIFF_OBJS += xdiff/xutils.o\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/zlib-compat.o\n \n+REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\ndiff --git a/reftable/.gitattributes b/reftable/.gitattributes\nnew file mode 100644\nindex 00000000000..f44451a3795\n--- /dev/null\n+++ b/reftable/.gitattributes\n@@ -0,0 +1 @@\n+/zlib-compat.c\twhitespace=-indent-with-non-tab,-trailing-space\ndiff --git a/reftable/block.c b/reftable/block.c\nnew file mode 100644\nindex 00000000000..5b4e49fb2be\n--- /dev/null\n+++ b/reftable/block.c\n@@ -0,0 +1,440 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"reftable-error.h\"\n+#include \"system.h\"\n+#include \"zlib.h\"\n+\n+int header_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 24;\n+\tcase 2:\n+\t\treturn 28;\n+\t}\n+\tabort();\n+}\n+\n+int footer_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 68;\n+\tcase 2:\n+\t\treturn 72;\n+\t}\n+\tabort();\n+}\n+\n+static int block_writer_register_restart(struct block_writer *w, int n,\n+\t\t\t\t\t int is_restart, struct strbuf *key)\n+{\n+\tint rlen = w->restart_len;\n+\tif (rlen >= MAX_RESTARTS) {\n+\t\tis_restart = 0;\n+\t}\n+\n+\tif (is_restart) {\n+\t\trlen++;\n+\t}\n+\tif (2 + 3 * rlen + n > w->block_size - w->next)\n+\t\treturn -1;\n+\tif (is_restart) {\n+\t\tif (w->restart_len == w->restart_cap) {\n+\t\t\tw->restart_cap = w->restart_cap * 2 + 1;\n+\t\t\tw->restarts = reftable_realloc(\n+\t\t\t\tw->restarts, sizeof(uint32_t) * w->restart_cap);\n+\t\t}\n+\n+\t\tw->restarts[w->restart_len++] = w->next;\n+\t}\n+\n+\tw->next += n;\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, key);\n+\tw->entries++;\n+\treturn 0;\n+}\n+\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n+{\n+\tbw->buf = buf;\n+\tbw->hash_size = hash_size;\n+\tbw->block_size = block_size;\n+\tbw->header_off = header_off;\n+\tbw->buf[header_off] = typ;\n+\tbw->next = header_off + 4;\n+\tbw->restart_interval = 16;\n+\tbw->entries = 0;\n+\tbw->restart_len = 0;\n+\tbw->last_key.len = 0;\n+}\n+\n+uint8_t block_writer_type(struct block_writer *bw)\n+{\n+\treturn bw->buf[bw->header_off];\n+}\n+\n+/* adds the reftable_record to the block. Returns -1 if it does not fit, 0 on\n+ success */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tstruct strbuf last =\n+\t\tw->entries % w->restart_interval == 0 ? empty : w->last_key;\n+\tstruct string_view out = {\n+\t\t.buf = w->buf + w->next,\n+\t\t.len = w->block_size - w->next,\n+\t};\n+\n+\tstruct string_view start = out;\n+\n+\tint is_restart = 0;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint n = 0;\n+\n+\treftable_record_key(rec, &key);\n+\tn = reftable_encode_key(&is_restart, out, last, key,\n+\t\t\t\treftable_record_val_type(rec));\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tn = reftable_record_encode(rec, out, w->hash_size);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tif (block_writer_register_restart(w, start.len - out.len, is_restart,\n+\t\t\t\t\t &key) < 0)\n+\t\tgoto done;\n+\n+\tstrbuf_release(&key);\n+\treturn 0;\n+\n+done:\n+\tstrbuf_release(&key);\n+\treturn -1;\n+}\n+\n+int block_writer_finish(struct block_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->restart_len; i++) {\n+\t\tput_be24(w->buf + w->next, w->restarts[i]);\n+\t\tw->next += 3;\n+\t}\n+\n+\tput_be16(w->buf + w->next, w->restart_len);\n+\tw->next += 2;\n+\tput_be24(w->buf + 1 + w->header_off, w->next);\n+\n+\tif (block_writer_type(w) == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + w->header_off;\n+\t\tuint8_t *compressed = NULL;\n+\t\tint zresult = 0;\n+\t\tuLongf src_len = w->next - block_header_skip;\n+\t\tsize_t dest_cap = src_len;\n+\n+\t\tcompressed = reftable_malloc(dest_cap);\n+\t\twhile (1) {\n+\t\t\tuLongf out_dest_len = dest_cap;\n+\n+\t\t\tzresult = compress2(compressed, &out_dest_len,\n+\t\t\t\t\t w->buf + block_header_skip, src_len,\n+\t\t\t\t\t 9);\n+\t\t\tif (zresult == Z_BUF_ERROR) {\n+\t\t\t\tdest_cap *= 2;\n+\t\t\t\tcompressed =\n+\t\t\t\t\treftable_realloc(compressed, dest_cap);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (Z_OK != zresult) {\n+\t\t\t\treftable_free(compressed);\n+\t\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t\t}\n+\n+\t\t\tmemcpy(w->buf + block_header_skip, compressed,\n+\t\t\t out_dest_len);\n+\t\t\tw->next = out_dest_len + block_header_skip;\n+\t\t\treftable_free(compressed);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\treturn w->next;\n+}\n+\n+uint8_t block_reader_type(struct block_reader *r)\n+{\n+\treturn r->block.data[r->header_off];\n+}\n+\n+int block_reader_init(struct block_reader *br, struct reftable_block *block,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size)\n+{\n+\tuint32_t full_block_size = table_block_size;\n+\tuint8_t typ = block->data[header_off];\n+\tuint32_t sz = get_be24(block->data + header_off + 1);\n+\n+\tuint16_t restart_count = 0;\n+\tuint32_t restart_start = 0;\n+\tuint8_t *restart_bytes = NULL;\n+\n+\tif (!reftable_is_block_type(typ))\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tif (typ == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + header_off;\n+\t\tuLongf dst_len = sz - block_header_skip; /* total size of dest\n+\t\t\t\t\t\t\t buffer. */\n+\t\tuLongf src_len = block->len - block_header_skip;\n+\t\t/* Log blocks specify the *uncompressed* size in their header.\n+\t\t */\n+\t\tuint8_t *uncompressed = reftable_malloc(sz);\n+\n+\t\t/* Copy over the block header verbatim. It's not compressed. */\n+\t\tmemcpy(uncompressed, block->data, block_header_skip);\n+\n+\t\t/* Uncompress */\n+\t\tif (Z_OK != uncompress_return_consumed(\n+\t\t\t\t uncompressed + block_header_skip, &dst_len,\n+\t\t\t\t block->data + block_header_skip,\n+\t\t\t\t &src_len)) {\n+\t\t\treftable_free(uncompressed);\n+\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t}\n+\n+\t\tif (dst_len + block_header_skip != sz)\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\t\t/* We're done with the input data. */\n+\t\treftable_block_done(block);\n+\t\tblock->data = uncompressed;\n+\t\tblock->len = sz;\n+\t\tblock->source = malloc_block_source();\n+\t\tfull_block_size = src_len + block_header_skip;\n+\t} else if (full_block_size == 0) {\n+\t\tfull_block_size = sz;\n+\t} else if (sz < full_block_size && sz < block->len &&\n+\t\t block->data[sz] != 0) {\n+\t\t/* If the block is smaller than the full block size, it is\n+\t\t padded (data followed by '\\0') or the next block is\n+\t\t unaligned. */\n+\t\tfull_block_size = sz;\n+\t}\n+\n+\trestart_count = get_be16(block->data + sz - 2);\n+\trestart_start = sz - 2 - 3 * restart_count;\n+\trestart_bytes = block->data + restart_start;\n+\n+\t/* transfer ownership. */\n+\tbr->block = *block;\n+\tblock->data = NULL;\n+\tblock->len = 0;\n+\n+\tbr->hash_size = hash_size;\n+\tbr->block_len = restart_start;\n+\tbr->full_block_size = full_block_size;\n+\tbr->header_off = header_off;\n+\tbr->restart_count = restart_count;\n+\tbr->restart_bytes = restart_bytes;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t block_reader_restart_offset(struct block_reader *br, int i)\n+{\n+\treturn get_be24(br->restart_bytes + 3 * i);\n+}\n+\n+void block_reader_start(struct block_reader *br, struct block_iter *it)\n+{\n+\tit->br = br;\n+\tstrbuf_reset(&it->last_key);\n+\tit->next_off = br->header_off + 4;\n+}\n+\n+struct restart_find_args {\n+\tint error;\n+\tstruct strbuf key;\n+\tstruct block_reader *r;\n+};\n+\n+static int restart_key_less(size_t idx, void *args)\n+{\n+\tstruct restart_find_args *a = (struct restart_find_args *)args;\n+\tuint32_t off = block_reader_restart_offset(a->r, idx);\n+\tstruct string_view in = {\n+\t\t.buf = a->r->block.data + off,\n+\t\t.len = a->r->block_len - off,\n+\t};\n+\n+\t/* the restart key is verbatim in the block, so this could avoid the\n+\t alloc for decoding the key */\n+\tstruct strbuf rkey = STRBUF_INIT;\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tuint8_t unused_extra;\n+\tint n = reftable_decode_key(&rkey, &unused_extra, last_key, in);\n+\tint result;\n+\tif (n < 0) {\n+\t\ta->error = 1;\n+\t\treturn -1;\n+\t}\n+\n+\tresult = strbuf_cmp(&a->key, &rkey);\n+\tstrbuf_release(&rkey);\n+\treturn result;\n+}\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src)\n+{\n+\tdest->br = src->br;\n+\tdest->next_off = src->next_off;\n+\tstrbuf_reset(&dest->last_key);\n+\tstrbuf_addbuf(&dest->last_key, &src->last_key);\n+}\n+\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec)\n+{\n+\tstruct string_view in = {\n+\t\t.buf = it->br->block.data + it->next_off,\n+\t\t.len = it->br->block_len - it->next_off,\n+\t};\n+\tstruct string_view start = in;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tuint8_t extra = 0;\n+\tint n = 0;\n+\n+\tif (it->next_off >= it->br->block_len)\n+\t\treturn 1;\n+\n+\tn = reftable_decode_key(&key, &extra, it->last_key, in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\n+\tstring_view_consume(&in, n);\n+\tn = reftable_record_decode(rec, key, extra, in, it->br->hash_size);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tstrbuf_reset(&it->last_key);\n+\tstrbuf_addbuf(&it->last_key, &key);\n+\tit->next_off += start.len - in.len;\n+\tstrbuf_release(&key);\n+\treturn 0;\n+}\n+\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tint off = br->header_off + 4;\n+\tstruct string_view in = {\n+\t\t.buf = br->block.data + off,\n+\t\t.len = br->block_len - off,\n+\t};\n+\n+\tuint8_t extra = 0;\n+\tint n = reftable_decode_key(key, &extra, empty, in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\treturn 0;\n+}\n+\n+int block_iter_seek(struct block_iter *it, struct strbuf *want)\n+{\n+\treturn block_reader_seek(it->br, it, want);\n+}\n+\n+void block_iter_close(struct block_iter *it)\n+{\n+\tstrbuf_release(&it->last_key);\n+}\n+\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want)\n+{\n+\tstruct restart_find_args args = {\n+\t\t.key = *want,\n+\t\t.r = br,\n+\t};\n+\tstruct reftable_record rec = reftable_new_record(block_reader_type(br));\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = 0;\n+\tstruct block_iter next = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\n+\tint i = binsearch(br->restart_count, &restart_key_less, &args);\n+\tif (args.error) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tit->br = br;\n+\tif (i > 0) {\n+\t\ti--;\n+\t\tit->next_off = block_reader_restart_offset(br, i);\n+\t} else {\n+\t\tit->next_off = br->header_off + 4;\n+\t}\n+\n+\t/* We're looking for the last entry less/equal than the wanted key, so\n+\t we have to go one entry too far and then back up.\n+\t*/\n+\twhile (1) {\n+\t\tblock_iter_copy_from(&next, it);\n+\t\terr = block_iter_next(&next, &rec);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tif (err > 0 || strbuf_cmp(&key, want) >= 0) {\n+\t\t\terr = 0;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tblock_iter_copy_from(it, &next);\n+\t}\n+\n+done:\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&next.last_key);\n+\treftable_record_destroy(&rec);\n+\n+\treturn err;\n+}\n+\n+void block_writer_release(struct block_writer *bw)\n+{\n+\tFREE_AND_NULL(bw->restarts);\n+\tstrbuf_release(&bw->last_key);\n+\t/* the block is not owned. */\n+}\n+\n+void reftable_block_done(struct reftable_block *blockp)\n+{\n+\tstruct reftable_block_source source = blockp->source;\n+\tif (blockp != NULL && source.ops != NULL)\n+\t\tsource.ops->return_block(source.arg, blockp);\n+\tblockp->data = NULL;\n+\tblockp->len = 0;\n+\tblockp->source.ops = NULL;\n+\tblockp->source.arg = NULL;\n+}\ndiff --git a/reftable/block.h b/reftable/block.h\nnew file mode 100644\nindex 00000000000..e207706a644\n--- /dev/null\n+++ b/reftable/block.h\n@@ -0,0 +1,127 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCK_H\n+#define BLOCK_H\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"reftable-blocksource.h\"\n+\n+/*\n+ * Writes reftable blocks. The block_writer is reused across blocks to minimize\n+ * allocation overhead.\n+ */\n+struct block_writer {\n+\tuint8_t *buf;\n+\tuint32_t block_size;\n+\n+\t/* Offset ofof the global header. Nonzero in the first block only. */\n+\tuint32_t header_off;\n+\n+\t/* How often to restart keys. */\n+\tint restart_interval;\n+\tint hash_size;\n+\n+\t/* Offset of next uint8_t to write. */\n+\tuint32_t next;\n+\tuint32_t *restarts;\n+\tuint32_t restart_len;\n+\tuint32_t restart_cap;\n+\n+\tstruct strbuf last_key;\n+\tint entries;\n+};\n+\n+/*\n+ * initializes the blockwriter to write `typ` entries, using `buf` as temporary\n+ * storage. `buf` is not owned by the block_writer. */\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n+\n+/* returns the block type (eg. 'r' for ref records. */\n+uint8_t block_writer_type(struct block_writer *bw);\n+\n+/* appends the record, or -1 if it doesn't fit. */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec);\n+\n+/* appends the key restarts, and compress the block if necessary. */\n+int block_writer_finish(struct block_writer *w);\n+\n+/* clears out internally allocated block_writer members. */\n+void block_writer_release(struct block_writer *bw);\n+\n+/* Read a block. */\n+struct block_reader {\n+\t/* offset of the block header; nonzero for the first block in a\n+\t * reftable. */\n+\tuint32_t header_off;\n+\n+\t/* the memory block */\n+\tstruct reftable_block block;\n+\tint hash_size;\n+\n+\t/* size of the data, excluding restart data. */\n+\tuint32_t block_len;\n+\tuint8_t *restart_bytes;\n+\tuint16_t restart_count;\n+\n+\t/* size of the data in the file. For log blocks, this is the compressed\n+\t * size. */\n+\tuint32_t full_block_size;\n+};\n+\n+/* Iterate over entries in a block */\n+struct block_iter {\n+\t/* offset within the block of the next entry to read. */\n+\tuint32_t next_off;\n+\tstruct block_reader *br;\n+\n+\t/* key for last entry we read. */\n+\tstruct strbuf last_key;\n+};\n+\n+/* initializes a block reader. */\n+int block_reader_init(struct block_reader *br, struct reftable_block *bl,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size);\n+\n+/* Position `it` at start of the block */\n+void block_reader_start(struct block_reader *br, struct block_iter *it);\n+\n+/* Position `it` to the `want` key in the block */\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want);\n+\n+/* Returns the block type (eg. 'r' for refs) */\n+uint8_t block_reader_type(struct block_reader *r);\n+\n+/* Decodes the first key in the block */\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key);\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src);\n+\n+/* return < 0 for error, 0 for OK, > 0 for EOF. */\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec);\n+\n+/* Seek to `want` with in the block pointed to by `it` */\n+int block_iter_seek(struct block_iter *it, struct strbuf *want);\n+\n+/* deallocate memory for `it`. The block reader and its block is left intact. */\n+void block_iter_close(struct block_iter *it);\n+\n+/* size of file header, depending on format version */\n+int header_size(int version);\n+\n+/* size of file footer, depending on format version */\n+int footer_size(int version);\n+\n+/* returns a block to its source. */\n+void reftable_block_done(struct reftable_block *ret);\n+\n+#endif\ndiff --git a/reftable/block_test.c b/reftable/block_test.c\nnew file mode 100644\nindex 00000000000..75fe198e677\n--- /dev/null\n+++ b/reftable/block_test.c\n@@ -0,0 +1,121 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"system.h\"\n+\n+#include \"blocksource.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_block_read_write(void)\n+{\n+\tconst int header_off = 21; /* random */\n+\tchar *names[30];\n+\tconst int N = ARRAY_SIZE(names);\n+\tconst int block_size = 1024;\n+\tstruct reftable_block block = { NULL };\n+\tstruct block_writer bw = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_record rec = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tstruct block_reader br = { 0 };\n+\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\tint j = 0;\n+\tstruct strbuf want = STRBUF_INIT;\n+\n+\tblock.data = reftable_calloc(block_size);\n+\tblock.len = block_size;\n+\tblock.source = malloc_block_source();\n+\tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n+\t\t\t header_off, hash_size(SHA1_ID));\n+\treftable_record_from_ref(&rec, &ref);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tuint8_t hash[SHA1_SIZE];\n+\t\tsnprintf(name, sizeof(name), \"branch%02d\", i);\n+\t\tmemset(hash, i, sizeof(hash));\n+\n+\t\tref.refname = name;\n+\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\tref.value.val1 = hash;\n+\n+\t\tnames[i] = xstrdup(name);\n+\t\tn = block_writer_add(&bw, &rec);\n+\t\tref.refname = NULL;\n+\t\tref.value_type = REFTABLE_REF_DELETION;\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tn = block_writer_finish(&bw);\n+\tEXPECT(n > 0);\n+\n+\tblock_writer_release(&bw);\n+\n+\tblock_reader_init(&br, &block, header_off, block_size, SHA1_SIZE);\n+\n+\tblock_reader_start(&br, &it);\n+\n+\twhile (1) {\n+\t\tint r = block_iter_next(&it, &rec);\n+\t\tEXPECT(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tEXPECT_STREQ(names[j], ref.refname);\n+\t\tj++;\n+\t}\n+\n+\treftable_record_release(&rec);\n+\tblock_iter_close(&it);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\t\tstrbuf_reset(&want);\n+\t\tstrbuf_addstr(&want, names[i]);\n+\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tEXPECT(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tEXPECT(n == 0);\n+\n+\t\tEXPECT_STREQ(names[i], ref.refname);\n+\n+\t\twant.len--;\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tEXPECT(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tEXPECT(n == 0);\n+\t\tEXPECT_STREQ(names[10 * (i / 10)], ref.refname);\n+\n+\t\tblock_iter_close(&it);\n+\t}\n+\n+\treftable_record_release(&rec);\n+\treftable_block_done(&br.block);\n+\tstrbuf_release(&want);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+}\n+\n+int block_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_block_read_write);\n+\treturn 0;\n+}\ndiff --git a/reftable/zlib-compat.c b/reftable/zlib-compat.c\nnew file mode 100644\nindex 00000000000..3e0b0f24f1c\n--- /dev/null\n+++ b/reftable/zlib-compat.c\n@@ -0,0 +1,92 @@\n+/* taken from zlib's uncompr.c\n+\n+ commit cacf7f1d4e3d44d871b605da3b647f07d718623f\n+ Author: Mark Adler \n+ Date: Sun Jan 15 09:18:46 2017 -0800\n+\n+ zlib 1.2.11\n+\n+*/\n+\n+/*\n+ * Copyright (C) 1995-2003, 2010, 2014, 2016 Jean-loup Gailly, Mark Adler\n+ * For conditions of distribution and use, see copyright notice in zlib.h\n+ */\n+\n+#include \"system.h\"\n+\n+/* clang-format off */\n+\n+/* ===========================================================================\n+ Decompresses the source buffer into the destination buffer. *sourceLen is\n+ the byte length of the source buffer. Upon entry, *destLen is the total size\n+ of the destination buffer, which must be large enough to hold the entire\n+ uncompressed data. (The size of the uncompressed data must have been saved\n+ previously by the compressor and transmitted to the decompressor by some\n+ mechanism outside the scope of this compression library.) Upon exit,\n+ *destLen is the size of the decompressed data and *sourceLen is the number\n+ of source bytes consumed. Upon return, source + *sourceLen points to the\n+ first unused input byte.\n+\n+ uncompress returns Z_OK if success, Z_MEM_ERROR if there was not enough\n+ memory, Z_BUF_ERROR if there was not enough room in the output buffer, or\n+ Z_DATA_ERROR if the input data was corrupted, including if the input data is\n+ an incomplete zlib stream.\n+*/\n+int ZEXPORT uncompress_return_consumed (\n+ Bytef *dest,\n+ uLongf *destLen,\n+ const Bytef *source,\n+ uLong *sourceLen) {\n+ z_stream stream;\n+ int err;\n+ const uInt max = (uInt)-1;\n+ uLong len, left;\n+ Byte buf[1]; /* for detection of incomplete stream when *destLen == 0 */\n+\n+ len = *sourceLen;\n+ if (*destLen) {\n+ left = *destLen;\n+ *destLen = 0;\n+ }\n+ else {\n+ left = 1;\n+ dest = buf;\n+ }\n+\n+ stream.next_in = (z_const Bytef *)source;\n+ stream.avail_in = 0;\n+ stream.zalloc = (alloc_func)0;\n+ stream.zfree = (free_func)0;\n+ stream.opaque = (voidpf)0;\n+\n+ err = inflateInit(&stream);\n+ if (err != Z_OK) return err;\n+\n+ stream.next_out = dest;\n+ stream.avail_out = 0;\n+\n+ do {\n+ if (stream.avail_out == 0) {\n+ stream.avail_out = left > (uLong)max ? max : (uInt)left;\n+ left -= stream.avail_out;\n+ }\n+ if (stream.avail_in == 0) {\n+ stream.avail_in = len > (uLong)max ? max : (uInt)len;\n+ len -= stream.avail_in;\n+ }\n+ err = inflate(&stream, Z_NO_FLUSH);\n+ } while (err == Z_OK);\n+\n+ *sourceLen -= len + stream.avail_in;\n+ if (dest != buf)\n+ *destLen = stream.total_out;\n+ else if (stream.total_out && err == Z_BUF_ERROR)\n+ left = 1;\n+\n+ inflateEnd(&stream);\n+ return err == Z_STREAM_END ? Z_OK :\n+ err == Z_NEED_DICT ? Z_DATA_ERROR :\n+ err == Z_BUF_ERROR && left + stream.avail_out ? Z_DATA_ERROR :\n+ err;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 09d4b83ef9b..c9deeaf08c7 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,6 +4,7 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n+\tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"411865","messageId":"ff5b424d12fb016f6ed6a335d05732cf30305fbb.1607522429.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v4.git.git.1607522429.gitgitgadget@gmail.com","subject":"[PATCH v4 09/15] reftable: write reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-12-09T14:00:23Z","receivedAt":"2020-12-09T14:03:38Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable-writer.h | 147 ++++++++\n reftable/writer.c | 681 +++++++++++++++++++++++++++++++++++++\n reftable/writer.h | 50 +++\n 4 files changed, 879 insertions(+)\n create mode 100644 reftable/reftable-writer.h\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n\ndiff --git a/Makefile b/Makefile\nindex 734c05655b2..e78908c976a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2396,6 +2396,7 @@ REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/tree.o\n+REFTABLE_OBJS += reftable/writer.o\n REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nnew file mode 100644\nindex 00000000000..9d2f8d60555\n--- /dev/null\n+++ b/reftable/reftable-writer.h\n@@ -0,0 +1,147 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_WRITER_H\n+#define REFTABLE_WRITER_H\n+\n+#include \n+\n+#include \"reftable-record.h\"\n+\n+/* Writing single reftables */\n+\n+/* reftable_write_options sets options for writing a single reftable. */\n+struct reftable_write_options {\n+\t/* boolean: do not pad out blocks to block size. */\n+\tunsigned unpadded : 1;\n+\n+\t/* the blocksize. Should be less than 2^24. */\n+\tuint32_t block_size;\n+\n+\t/* boolean: do not generate a SHA1 => ref index. */\n+\tunsigned skip_index_objects : 1;\n+\n+\t/* how often to write complete keys in each block. */\n+\tint restart_interval;\n+\n+\t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n+\t * Defaults to SHA1 if unset\n+\t */\n+\tuint32_t hash_id;\n+\n+\t/* boolean: do not check ref names for validity or dir/file conflicts.\n+\t */\n+\tunsigned skip_name_check : 1;\n+\n+\t/* boolean: copy log messages exactly. If unset, check that the message\n+\t * is a single line, and add '\\n' if missing.\n+\t */\n+\tunsigned exact_log_message : 1;\n+};\n+\n+/* reftable_block_stats holds statistics for a single block type */\n+struct reftable_block_stats {\n+\t/* total number of entries written */\n+\tint entries;\n+\t/* total number of key restarts */\n+\tint restarts;\n+\t/* total number of blocks */\n+\tint blocks;\n+\t/* total number of index blocks */\n+\tint index_blocks;\n+\t/* depth of the index */\n+\tint max_index_level;\n+\n+\t/* offset of the first block for this type */\n+\tuint64_t offset;\n+\t/* offset of the top level index block for this type, or 0 if not\n+\t * present */\n+\tuint64_t index_offset;\n+};\n+\n+/* stats holds overall statistics for a single reftable */\n+struct reftable_stats {\n+\t/* total number of blocks written. */\n+\tint blocks;\n+\t/* stats for ref data */\n+\tstruct reftable_block_stats ref_stats;\n+\t/* stats for the SHA1 to ref map. */\n+\tstruct reftable_block_stats obj_stats;\n+\t/* stats for index blocks */\n+\tstruct reftable_block_stats idx_stats;\n+\t/* stats for log blocks */\n+\tstruct reftable_block_stats log_stats;\n+\n+\t/* disambiguation length of shortened object IDs. */\n+\tint object_id_len;\n+};\n+\n+/* reftable_new_writer creates a new writer */\n+struct reftable_writer *\n+reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts);\n+\n+/* Set the range of update indices for the records we will add. When writing a\n+ table into a stack, the min should be at least\n+ reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned.\n+\n+ For transactional updates to a stack, typically min==max, and the\n+ update_index can be obtained by inspeciting the stack. When converting an\n+ existing ref database into a single reftable, this would be a range of\n+ update-index timestamps.\n+ */\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max);\n+\n+/*\n+ Add a reftable_ref_record. The record should have names that come after\n+ already added records.\n+\n+ The update_index must be within the limits set by\n+ reftable_writer_set_limits(), or REFTABLE_API_ERROR is returned. It is an\n+ REFTABLE_API_ERROR error to write a ref record after a log record.\n+*/\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/*\n+ Convenience function to add multiple reftable_ref_records; the function sorts\n+ the records before adding them, reordering the records array passed in.\n+*/\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n);\n+\n+/*\n+ adds reftable_log_records. Log records are keyed by (refname, decreasing\n+ update_index). The key for the record added must come after the already added\n+ log records.\n+*/\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log);\n+\n+/*\n+ Convenience function to add multiple reftable_log_records; the function sorts\n+ the records before adding them, reordering records array passed in.\n+*/\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n);\n+\n+/* reftable_writer_close finalizes the reftable. The writer is retained so\n+ * statistics can be inspected. */\n+int reftable_writer_close(struct reftable_writer *w);\n+\n+/* writer_stats returns the statistics on the reftable being written.\n+\n+ This struct becomes invalid when the writer is freed.\n+ */\n+const struct reftable_stats *writer_stats(struct reftable_writer *w);\n+\n+/* reftable_writer_free deallocates memory for the writer */\n+void reftable_writer_free(struct reftable_writer *w);\n+\n+#endif\ndiff --git a/reftable/writer.c b/reftable/writer.c\nnew file mode 100644\nindex 00000000000..00a0a11440f\n--- /dev/null\n+++ b/reftable/writer.c\n@@ -0,0 +1,681 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"writer.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"tree.h\"\n+#include \"reftable-error.h\"\n+\n+/* finishes a block, and writes it to storage */\n+static int writer_flush_block(struct reftable_writer *w);\n+\n+/* deallocates memory related to the index */\n+static void writer_clear_index(struct reftable_writer *w);\n+\n+/* finishes writing a 'r' (refs) or 'g' (reflogs) section */\n+static int writer_finish_public_section(struct reftable_writer *w);\n+\n+static struct reftable_block_stats *\n+writer_reftable_block_stats(struct reftable_writer *w, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase 'r':\n+\t\treturn &w->stats.ref_stats;\n+\tcase 'o':\n+\t\treturn &w->stats.obj_stats;\n+\tcase 'i':\n+\t\treturn &w->stats.idx_stats;\n+\tcase 'g':\n+\t\treturn &w->stats.log_stats;\n+\t}\n+\tabort();\n+\treturn NULL;\n+}\n+\n+/* write data, queuing the padding for the next write. Returns negative for\n+ * error. */\n+static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n+\t\t\tint padding)\n+{\n+\tint n = 0;\n+\tif (w->pending_padding > 0) {\n+\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding);\n+\t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n+\t\tif (n < 0)\n+\t\t\treturn n;\n+\n+\t\tw->pending_padding = 0;\n+\t\treftable_free(zeroed);\n+\t}\n+\n+\tw->pending_padding = padding;\n+\tn = w->write(w->write_arg, data, len);\n+\tif (n < 0)\n+\t\treturn n;\n+\tn += padding;\n+\treturn 0;\n+}\n+\n+static void options_set_defaults(struct reftable_write_options *opts)\n+{\n+\tif (opts->restart_interval == 0) {\n+\t\topts->restart_interval = 16;\n+\t}\n+\n+\tif (opts->hash_id == 0) {\n+\t\topts->hash_id = SHA1_ID;\n+\t}\n+\tif (opts->block_size == 0) {\n+\t\topts->block_size = DEFAULT_BLOCK_SIZE;\n+\t}\n+}\n+\n+static int writer_version(struct reftable_writer *w)\n+{\n+\treturn (w->opts.hash_id == 0 || w->opts.hash_id == SHA1_ID) ? 1 : 2;\n+}\n+\n+static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n+{\n+\tmemcpy((char *)dest, \"REFT\", 4);\n+\n+\tdest[4] = writer_version(w);\n+\n+\tput_be24(dest + 5, w->opts.block_size);\n+\tput_be64(dest + 8, w->min_update_index);\n+\tput_be64(dest + 16, w->max_update_index);\n+\tif (writer_version(w) == 2) {\n+\t\tput_be32(dest + 24, w->opts.hash_id);\n+\t}\n+\treturn header_size(writer_version(w));\n+}\n+\n+static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n+{\n+\tint block_start = 0;\n+\tif (w->next == 0) {\n+\t\tblock_start = header_size(writer_version(w));\n+\t}\n+\n+\tstrbuf_release(&w->last_key);\n+\tblock_writer_init(&w->block_writer_data, typ, w->block,\n+\t\t\t w->opts.block_size, block_start,\n+\t\t\t hash_size(w->opts.hash_id));\n+\tw->block_writer = &w->block_writer_data;\n+\tw->block_writer->restart_interval = w->opts.restart_interval;\n+}\n+\n+static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n+\n+struct reftable_writer *\n+reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts)\n+{\n+\tstruct reftable_writer *wp =\n+\t\treftable_calloc(sizeof(struct reftable_writer));\n+\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n+\toptions_set_defaults(opts);\n+\tif (opts->block_size >= (1 << 24)) {\n+\t\t/* TODO - error return? */\n+\t\tabort();\n+\t}\n+\twp->last_key = reftable_empty_strbuf;\n+\twp->block = reftable_calloc(opts->block_size);\n+\twp->write = writer_func;\n+\twp->write_arg = writer_arg;\n+\twp->opts = *opts;\n+\twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n+\n+\treturn wp;\n+}\n+\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max)\n+{\n+\tw->min_update_index = min;\n+\tw->max_update_index = max;\n+}\n+\n+void reftable_writer_free(struct reftable_writer *w)\n+{\n+\treftable_free(w->block);\n+\treftable_free(w);\n+}\n+\n+struct obj_index_tree_node {\n+\tstruct strbuf hash;\n+\tuint64_t *offsets;\n+\tsize_t offset_len;\n+\tsize_t offset_cap;\n+};\n+\n+#define OBJ_INDEX_TREE_NODE_INIT \\\n+\t{ \\\n+\t\t.hash = STRBUF_INIT \\\n+\t}\n+\n+static int obj_index_tree_node_compare(const void *a, const void *b)\n+{\n+\treturn strbuf_cmp(&((const struct obj_index_tree_node *)a)->hash,\n+\t\t\t &((const struct obj_index_tree_node *)b)->hash);\n+}\n+\n+static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n+{\n+\tuint64_t off = w->next;\n+\n+\tstruct obj_index_tree_node want = { .hash = *hash };\n+\n+\tstruct tree_node *node = tree_search(&want, &w->obj_index_tree,\n+\t\t\t\t\t &obj_index_tree_node_compare, 0);\n+\tstruct obj_index_tree_node *key = NULL;\n+\tif (node == NULL) {\n+\t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n+\t\tkey = reftable_malloc(sizeof(struct obj_index_tree_node));\n+\t\t*key = empty;\n+\n+\t\tstrbuf_reset(&key->hash);\n+\t\tstrbuf_addbuf(&key->hash, hash);\n+\t\ttree_search((void *)key, &w->obj_index_tree,\n+\t\t\t &obj_index_tree_node_compare, 1);\n+\t} else {\n+\t\tkey = node->key;\n+\t}\n+\n+\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off) {\n+\t\treturn;\n+\t}\n+\n+\tif (key->offset_len == key->offset_cap) {\n+\t\tkey->offset_cap = 2 * key->offset_cap + 1;\n+\t\tkey->offsets = reftable_realloc(\n+\t\t\tkey->offsets, sizeof(uint64_t) * key->offset_cap);\n+\t}\n+\n+\tkey->offsets[key->offset_len++] = off;\n+}\n+\n+static int writer_add_record(struct reftable_writer *w,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tint result = -1;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = 0;\n+\treftable_record_key(rec, &key);\n+\tif (strbuf_cmp(&w->last_key, &key) >= 0)\n+\t\tgoto done;\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, &key);\n+\tif (w->block_writer == NULL) {\n+\t\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\t}\n+\n+\tassert(block_writer_type(w->block_writer) == reftable_record_type(rec));\n+\n+\tif (block_writer_add(w->block_writer, rec) == 0) {\n+\t\tresult = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0) {\n+\t\tresult = err;\n+\t\tgoto done;\n+\t}\n+\n+\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\terr = block_writer_add(w->block_writer, rec);\n+\tif (err < 0) {\n+\t\tresult = err;\n+\t\tgoto done;\n+\t}\n+\n+\tresult = 0;\n+done:\n+\tstrbuf_release(&key);\n+\treturn result;\n+}\n+\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_ref_record copy = *ref;\n+\tint err = 0;\n+\n+\tif (ref->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\tif (ref->update_index < w->min_update_index ||\n+\t ref->update_index > w->max_update_index)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\treftable_record_from_ref(&rec, ©);\n+\tcopy.update_index -= w->min_update_index;\n+\n+\terr = writer_add_record(w, &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (!w->opts.skip_index_objects &&\n+\t reftable_ref_record_val1(ref) != NULL) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, (char *)reftable_ref_record_val1(ref),\n+\t\t\t hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\n+\tif (!w->opts.skip_index_objects &&\n+\t reftable_ref_record_val2(ref) != NULL) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, reftable_ref_record_val2(ref),\n+\t\t\t hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(refs, n, reftable_ref_record_compare_name);\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_ref(w, &refs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tchar *input_log_message = log->message;\n+\tstruct strbuf cleaned_message = STRBUF_INIT;\n+\tint err;\n+\tif (log->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\tif (w->block_writer != NULL &&\n+\t block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {\n+\t\tint err = writer_finish_public_section(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tif (!w->opts.exact_log_message && log->message != NULL) {\n+\t\tstrbuf_addstr(&cleaned_message, log->message);\n+\t\twhile (cleaned_message.len &&\n+\t\t cleaned_message.buf[cleaned_message.len - 1] == '\\n')\n+\t\t\tstrbuf_setlen(&cleaned_message,\n+\t\t\t\t cleaned_message.len - 1);\n+\t\tif (strchr(cleaned_message.buf, '\\n')) {\n+\t\t\t// multiple lines not allowed.\n+\t\t\terr = REFTABLE_API_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tstrbuf_addstr(&cleaned_message, \"\\n\");\n+\t\tlog->message = cleaned_message.buf;\n+\t}\n+\n+\tw->next -= w->pending_padding;\n+\tw->pending_padding = 0;\n+\n+\treftable_record_from_log(&rec, log);\n+\terr = writer_add_record(w, &rec);\n+\n+done:\n+\tlog->message = input_log_message;\n+\tstrbuf_release(&cleaned_message);\n+\treturn err;\n+}\n+\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(logs, n, reftable_log_record_compare_key);\n+\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_log(w, &logs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+static int writer_finish_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tuint64_t index_start = 0;\n+\tint max_level = 0;\n+\tint threshold = w->opts.unpadded ? 1 : 3;\n+\tint before_blocks = w->stats.idx_stats.blocks;\n+\tint err = writer_flush_block(w);\n+\tint i = 0;\n+\tstruct reftable_block_stats *bstats = NULL;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\twhile (w->index_len > threshold) {\n+\t\tstruct reftable_index_record *idx = NULL;\n+\t\tint idx_len = 0;\n+\n+\t\tmax_level++;\n+\t\tindex_start = w->next;\n+\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\tidx = w->index;\n+\t\tidx_len = w->index_len;\n+\n+\t\tw->index = NULL;\n+\t\tw->index_len = 0;\n+\t\tw->index_cap = 0;\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstruct reftable_record rec = { NULL };\n+\t\t\treftable_record_from_index(&rec, idx + i);\n+\t\t\tif (block_writer_add(w->block_writer, &rec) == 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\terr = writer_flush_block(w);\n+\t\t\tif (err < 0)\n+\t\t\t\treturn err;\n+\n+\t\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\t\terr = block_writer_add(w->block_writer, &rec);\n+\t\t\tif (err != 0) {\n+\t\t\t\t/* write into fresh block should always succeed\n+\t\t\t\t */\n+\t\t\t\tabort();\n+\t\t\t}\n+\t\t}\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstrbuf_release(&idx[i].last_key);\n+\t\t}\n+\t\treftable_free(idx);\n+\t}\n+\n+\twriter_clear_index(w);\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tbstats = writer_reftable_block_stats(w, typ);\n+\tbstats->index_blocks = w->stats.idx_stats.blocks - before_blocks;\n+\tbstats->index_offset = index_start;\n+\tbstats->max_index_level = max_level;\n+\n+\t/* Reinit lastKey, as the next section can start with any key. */\n+\tw->last_key.len = 0;\n+\n+\treturn 0;\n+}\n+\n+struct common_prefix_arg {\n+\tstruct strbuf *last;\n+\tint max;\n+};\n+\n+static void update_common(void *void_arg, void *key)\n+{\n+\tstruct common_prefix_arg *arg = (struct common_prefix_arg *)void_arg;\n+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n+\tif (arg->last != NULL) {\n+\t\tint n = common_prefix_size(&entry->hash, arg->last);\n+\t\tif (n > arg->max) {\n+\t\t\targ->max = n;\n+\t\t}\n+\t}\n+\targ->last = &entry->hash;\n+}\n+\n+struct write_record_arg {\n+\tstruct reftable_writer *w;\n+\tint err;\n+};\n+\n+static void write_object_record(void *void_arg, void *key)\n+{\n+\tstruct write_record_arg *arg = (struct write_record_arg *)void_arg;\n+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n+\tstruct reftable_obj_record obj_rec = {\n+\t\t.hash_prefix = (uint8_t *)entry->hash.buf,\n+\t\t.hash_prefix_len = arg->w->stats.object_id_len,\n+\t\t.offsets = entry->offsets,\n+\t\t.offset_len = entry->offset_len,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\treftable_record_from_obj(&rec, &obj_rec);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\n+\targ->err = writer_flush_block(arg->w);\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\twriter_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\tobj_rec.offset_len = 0;\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\n+\t/* Should be able to write into a fresh block. */\n+\tassert(arg->err == 0);\n+\n+done:;\n+}\n+\n+static void object_record_free(void *void_arg, void *key)\n+{\n+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n+\n+\tFREE_AND_NULL(entry->offsets);\n+\tstrbuf_release(&entry->hash);\n+\treftable_free(entry);\n+}\n+\n+static int writer_dump_object_index(struct reftable_writer *w)\n+{\n+\tstruct write_record_arg closure = { .w = w };\n+\tstruct common_prefix_arg common = { NULL };\n+\tif (w->obj_index_tree != NULL) {\n+\t\tinfix_walk(w->obj_index_tree, &update_common, &common);\n+\t}\n+\tw->stats.object_id_len = common.max + 1;\n+\n+\twriter_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\n+\tif (w->obj_index_tree != NULL) {\n+\t\tinfix_walk(w->obj_index_tree, &write_object_record, &closure);\n+\t}\n+\n+\tif (closure.err < 0)\n+\t\treturn closure.err;\n+\treturn writer_finish_section(w);\n+}\n+\n+static int writer_finish_public_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = 0;\n+\tint err = 0;\n+\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\n+\ttyp = block_writer_type(w->block_writer);\n+\terr = writer_finish_section(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\tif (typ == BLOCK_TYPE_REF && !w->opts.skip_index_objects &&\n+\t w->stats.ref_stats.index_blocks > 0) {\n+\t\terr = writer_dump_object_index(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tif (w->obj_index_tree != NULL) {\n+\t\tinfix_walk(w->obj_index_tree, &object_record_free, NULL);\n+\t\ttree_free(w->obj_index_tree);\n+\t\tw->obj_index_tree = NULL;\n+\t}\n+\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+int reftable_writer_close(struct reftable_writer *w)\n+{\n+\tuint8_t footer[72];\n+\tuint8_t *p = footer;\n+\tint err = writer_finish_public_section(w);\n+\tint empty_table = w->next == 0;\n+\tif (err != 0)\n+\t\tgoto done;\n+\tw->pending_padding = 0;\n+\tif (empty_table) {\n+\t\t/* Empty tables need a header anyway. */\n+\t\tuint8_t header[28];\n+\t\tint n = writer_write_header(w, header);\n+\t\terr = padded_write(w, header, n, 0);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\tp += writer_write_header(w, footer);\n+\tput_be64(p, w->stats.ref_stats.index_offset);\n+\tp += 8;\n+\tput_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n+\tp += 8;\n+\tput_be64(p, w->stats.obj_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be64(p, w->stats.log_stats.offset);\n+\tp += 8;\n+\tput_be64(p, w->stats.log_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be32(p, crc32(0, footer, p - footer));\n+\tp += 4;\n+\n+\terr = padded_write(w, footer, footer_size(writer_version(w)), 0);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (empty_table) {\n+\t\terr = REFTABLE_EMPTY_TABLE_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\t/* free up memory. */\n+\tblock_writer_release(&w->block_writer_data);\n+\twriter_clear_index(w);\n+\tstrbuf_release(&w->last_key);\n+\treturn err;\n+}\n+\n+static void writer_clear_index(struct reftable_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->index_len; i++) {\n+\t\tstrbuf_release(&w->index[i].last_key);\n+\t}\n+\n+\tFREE_AND_NULL(w->index);\n+\tw->index_len = 0;\n+\tw->index_cap = 0;\n+}\n+\n+static const int debug = 0;\n+\n+static int writer_flush_nonempty_block(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tstruct reftable_block_stats *bstats =\n+\t\twriter_reftable_block_stats(w, typ);\n+\tuint64_t block_typ_off = (bstats->blocks == 0) ? w->next : 0;\n+\tint raw_bytes = block_writer_finish(w->block_writer);\n+\tint padding = 0;\n+\tint err = 0;\n+\tstruct reftable_index_record ir = { .last_key = STRBUF_INIT };\n+\tif (raw_bytes < 0)\n+\t\treturn raw_bytes;\n+\n+\tif (!w->opts.unpadded && typ != BLOCK_TYPE_LOG) {\n+\t\tpadding = w->opts.block_size - raw_bytes;\n+\t}\n+\n+\tif (block_typ_off > 0) {\n+\t\tbstats->offset = block_typ_off;\n+\t}\n+\n+\tbstats->entries += w->block_writer->entries;\n+\tbstats->restarts += w->block_writer->restart_len;\n+\tbstats->blocks++;\n+\tw->stats.blocks++;\n+\n+\tif (debug) {\n+\t\tfprintf(stderr, \"block %c off %\" PRIu64 \" sz %d (%d)\\n\", typ,\n+\t\t\tw->next, raw_bytes,\n+\t\t\tget_be24(w->block + w->block_writer->header_off + 1));\n+\t}\n+\n+\tif (w->next == 0) {\n+\t\twriter_write_header(w, w->block);\n+\t}\n+\n+\terr = padded_write(w, w->block, raw_bytes, padding);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (w->index_cap == w->index_len) {\n+\t\tw->index_cap = 2 * w->index_cap + 1;\n+\t\tw->index = reftable_realloc(\n+\t\t\tw->index,\n+\t\t\tsizeof(struct reftable_index_record) * w->index_cap);\n+\t}\n+\n+\tir.offset = w->next;\n+\tstrbuf_reset(&ir.last_key);\n+\tstrbuf_addbuf(&ir.last_key, &w->block_writer->last_key);\n+\tw->index[w->index_len] = ir;\n+\n+\tw->index_len++;\n+\tw->next += padding + raw_bytes;\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+static int writer_flush_block(struct reftable_writer *w)\n+{\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\tif (w->block_writer->entries == 0)\n+\t\treturn 0;\n+\treturn writer_flush_nonempty_block(w);\n+}\n+\n+const struct reftable_stats *writer_stats(struct reftable_writer *w)\n+{\n+\treturn &w->stats;\n+}\ndiff --git a/reftable/writer.h b/reftable/writer.h\nnew file mode 100644\nindex 00000000000..4921c249d06\n--- /dev/null\n+++ b/reftable/writer.h\n@@ -0,0 +1,50 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef WRITER_H\n+#define WRITER_H\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"tree.h\"\n+#include \"reftable-writer.h\"\n+\n+struct reftable_writer {\n+\tint (*write)(void *, const void *, size_t);\n+\tvoid *write_arg;\n+\tint pending_padding;\n+\tstruct strbuf last_key;\n+\n+\t/* offset of next block to write. */\n+\tuint64_t next;\n+\tuint64_t min_update_index, max_update_index;\n+\tstruct reftable_write_options opts;\n+\n+\t/* memory buffer for writing */\n+\tuint8_t *block;\n+\n+\t/* writer for the current section. NULL or points to\n+\t * block_writer_data */\n+\tstruct block_writer *block_writer;\n+\n+\tstruct block_writer block_writer_data;\n+\n+\t/* pending index records for the current section */\n+\tstruct reftable_index_record *index;\n+\tsize_t index_len;\n+\tsize_t index_cap;\n+\n+\t/*\n+\t * tree for use with tsearch; used to populate the 'o' inverse OID\n+\t * map */\n+\tstruct tree_node *obj_index_tree;\n+\n+\tstruct reftable_stats stats;\n+};\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"414908","messageId":"CAFQ2z_Py46k71fHyBHyS2jqjhJJm-N0AV4X8YZZVawoZyVyhAg@mail.gmail.com","threadId":"54245","inReplyTo":"871ree8eto.fsf@evledraar.gmail.com","subject":"Re: [PATCH v4 13/15] Reftable support for git-core","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-01-21T16:14:36Z","receivedAt":"2021-01-21T16:16:41Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Thu, Jan 21, 2021 at 4:55 PM Ævar Arnfjörð Bjarmason\n wrote:\n> I must say I do find it a bit more concerning in the case of the\n> reftable series. I.e. the git_reftable_read_raw_ref() function seems\n> like a faithful copy of the general logic (at least this part) of the\n> refs_resolve_ref_unsafe() function in refs.c.\n>\n> That I can remove error checking/handling from this place means existing\n> general logic was faithfully copied without checking that we have a test\n> for it, or adding one. Or perhaps given the history of this codebase\n> there's out-of-tree tests for it somewhere?\n\nOne of the earlier iterations of this code didn't have the errno\nhandling, and it caused a bug that I tracked down to the errno\nhandling. I'm fairly sure of this, because I remember being perplexed\nand flabbergasted at using errno as an out-of-band API mechanism. The\nfirst iteration of reftable code was about a year ago, so if you\ncheckout the Git source code from that time, and remove the errno\nassignments, you would be able to see which test exercised this code\npath.\n\nOr you could check where errno is read-back (I believe ENOENT is the\ncase that has special meaning)\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"414909","messageId":"CAFQ2z_PPnmCHYvse-V8QG2zG2MouHxuvraMmWD=cv_bRWpakEA@mail.gmail.com","threadId":"54245","inReplyTo":"CAFQ2z_Py46k71fHyBHyS2jqjhJJm-N0AV4X8YZZVawoZyVyhAg@mail.gmail.com","subject":"Re: [PATCH v4 13/15] Reftable support for git-core","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-01-21T16:21:03Z","receivedAt":"2021-01-21T16:22:31Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Thu, Jan 21, 2021 at 5:14 PM Han-Wen Nienhuys wrote:\n>\n> On Thu, Jan 21, 2021 at 4:55 PM Ævar Arnfjörð Bjarmason\n> wrote:\n> > I must say I do find it a bit more concerning in the case of the\n> > reftable series. I.e. the git_reftable_read_raw_ref() function seems\n> > like a faithful copy of the general logic (at least this part) of the\n> > refs_resolve_ref_unsafe() function in refs.c.\n> >\n> > That I can remove error checking/handling from this place means existing\n> > general logic was faithfully copied without checking that we have a test\n> > for it, or adding one. Or perhaps given the history of this codebase\n> > there's out-of-tree tests for it somewhere?\n>\n> One of the earlier iterations of this code didn't have the errno\n> handling, and it caused a bug that I tracked down to the errno\n> handling. I'm fairly sure of this, because I remember being perplexed\n> and flabbergasted at using errno as an out-of-band API mechanism. The\n> first iteration of reftable code was about a year ago, so if you\n> checkout the Git source code from that time, and remove the errno\n> assignments, you would be able to see which test exercised this code\n> path.\n>\n> Or you could check where errno is read-back (I believe ENOENT is the\n> case that has special meaning)\n\nIn refs.c, there is behavior that depends on errno,\n\nif (refs_read_raw_ref(refs, refname,\n oid, &sb_refname, &read_flags)) {\n..\n/*\n* Otherwise a missing ref is OK. But the files backend\n* may show errors besides ENOENT if there are\n* similarly-named refs.\n*/\nif (errno != ENOENT &&\n errno != EISDIR &&\n errno != ENOTDIR)\nreturn NULL;\n\nblame tells me it has something to do with dir/file conflicts and\ncommit a1c1d8170db.\n\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\nRegistergericht und -nummer: Hamburg, HRB 86891\nSitz der Gesellschaft: Hamburg\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"414912","messageId":"871ree8eto.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"d57023d9f13d178dc95d7a74751923b80d1a5939.1607522429.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 13/15] Reftable support for git-core","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-01-21T15:55:31Z","receivedAt":"2021-01-21T16:50:33Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Dec 09 2020, Han-Wen Nienhuys via GitGitGadget wrote:\n\n> From: Han-Wen Nienhuys \n> [...]\n> refs.c | 27 +-\n> refs.h | 3 +\n> [...]\n> refs/reftable-backend.c | 1435 +++++++++++++++++\n> [...]\n> +static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n> +\t\t\t\t const char *refname, struct object_id *oid,\n> +\t\t\t\t struct strbuf *referent,\n> +\t\t\t\t unsigned int *type)\n> +{\n> +\tstruct git_reftable_ref_store *refs =\n> +\t\t(struct git_reftable_ref_store *)ref_store;\n> +\tstruct reftable_stack *stack = stack_for(refs, refname);\n> +\n> +\tstruct reftable_ref_record ref = { NULL };\n> +\tint err = 0;\n> +\tif (refs->err < 0) {\n> +\t\treturn refs->err;\n> +\t}\n> +\n> +\t/* This is usually not needed, but Git doesn't signal to ref backend if\n> +\t a subprocess updated the ref DB. So we always check.\n> +\t*/\n> +\terr = reftable_stack_reload(stack);\n> +\tif (err) {\n> +\t\tgoto done;\n> +\t}\n> +\n> +\terr = reftable_stack_read_ref(stack, refname, &ref);\n> +\tif (err > 0) {\n> +\t\terrno = ENOENT;\n> +\t\terr = -1;\n> +\t\tgoto done;\n> +\t}\n> +\tif (err < 0) {\n> +\t\terrno = reftable_error_to_errno(err);\n> +\t\terr = -1;\n> +\t\tgoto done;\n> +\t}\n> +\n> +\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n> +\t\tstrbuf_reset(referent);\n> +\t\tstrbuf_addstr(referent, ref.value.symref);\n> +\t\t*type |= REF_ISSYMREF;\n> +\t} else if (reftable_ref_record_val1(&ref) != NULL) {\n> +\t\thashcpy(oid->hash, reftable_ref_record_val1(&ref));\n> +\t} else {\n> +\t\t*type |= REF_ISBROKEN;\n> +\t\terrno = EINVAL;\n> +\t\terr = -1;\n\nI've been chasing down some edge cases in refs.c as part of another WIP\nseries I have and found that for this particular \"errno\" stuff we don't\nhave any test coverage. And with master & hanwen/reftable if I apply:\n\n diff --git a/refs.c b/refs.c\n index af0d000082..8ac57908ea 100644\n --- a/refs.c\n +++ b/refs.c\n @@ -1589,3 +1589,2 @@ const char *refs_resolve_ref_unsafe(struct ref_store *refs,\n !refname_is_safe(refname)) {\n - errno = EINVAL;\n return NULL;\n @@ -1649,3 +1648,2 @@ const char *refs_resolve_ref_unsafe(struct ref_store *refs,\n !refname_is_safe(refname)) {\n - errno = EINVAL;\n return NULL;\n @@ -1657,3 +1655,2 @@ const char *refs_resolve_ref_unsafe(struct ref_store *refs,\n \n - errno = ELOOP;\n return NULL;\n diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n index e98d22c826..e09ae21965 100644\n --- a/refs/reftable-backend.c\n +++ b/refs/reftable-backend.c\n @@ -1343,3 +1343,2 @@ static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n if (err < 0) {\n - errno = reftable_error_to_errno(err);\n err = -1;\n @@ -1355,3 +1354,2 @@ static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n *type |= REF_ISBROKEN;\n - errno = EINVAL;\n err = -1;\n\nAll tests pass on both.\n\nAnyway, in the case of the code sitting on \"master\" I find that less\nconcerning. It's some code that's drifted apart over the years blah blah\nblah. Boring details but for various reasons we don't use some/any of\nthat error handling logic anymore, or to the extent that we do I'll need\nto improve the tests.\n\nI must say I do find it a bit more concerning in the case of the\nreftable series. I.e. the git_reftable_read_raw_ref() function seems\nlike a faithful copy of the general logic (at least this part) of the\nrefs_resolve_ref_unsafe() function in refs.c.\n\nThat I can remove error checking/handling from this place means existing\ngeneral logic was faithfully copied without checking that we have a test\nfor it, or adding one. Or perhaps given the history of this codebase\nthere's out-of-tree tests for it somewhere?\n"},{"id":"415282","messageId":"871re77qyw.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"CAFQ2z_PPnmCHYvse-V8QG2zG2MouHxuvraMmWD=cv_bRWpakEA@mail.gmail.com","subject":"Re: [PATCH v4 13/15] Reftable support for git-core","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-01-26T13:44:23Z","receivedAt":"2021-01-26T13:46:47Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jan 21 2021, Han-Wen Nienhuys wrote:\n\n> On Thu, Jan 21, 2021 at 5:14 PM Han-Wen Nienhuys wrote:\n>>\n>> On Thu, Jan 21, 2021 at 4:55 PM Ævar Arnfjörð Bjarmason\n>> wrote:\n>> > I must say I do find it a bit more concerning in the case of the\n>> > reftable series. I.e. the git_reftable_read_raw_ref() function seems\n>> > like a faithful copy of the general logic (at least this part) of the\n>> > refs_resolve_ref_unsafe() function in refs.c.\n>> >\n>> > That I can remove error checking/handling from this place means existing\n>> > general logic was faithfully copied without checking that we have a test\n>> > for it, or adding one. Or perhaps given the history of this codebase\n>> > there's out-of-tree tests for it somewhere?\n>>\n>> One of the earlier iterations of this code didn't have the errno\n>> handling, and it caused a bug that I tracked down to the errno\n>> handling. I'm fairly sure of this, because I remember being perplexed\n>> and flabbergasted at using errno as an out-of-band API mechanism. The\n>> first iteration of reftable code was about a year ago, so if you\n>> checkout the Git source code from that time, and remove the errno\n>> assignments, you would be able to see which test exercised this code\n>> path.\n>>\n>> Or you could check where errno is read-back (I believe ENOENT is the\n>> case that has special meaning)\n>\n> In refs.c, there is behavior that depends on errno,\n>\n> if (refs_read_raw_ref(refs, refname,\n> oid, &sb_refname, &read_flags)) {\n> ..\n> /*\n> * Otherwise a missing ref is OK. But the files backend\n> * may show errors besides ENOENT if there are\n> * similarly-named refs.\n> */\n> if (errno != ENOENT &&\n> errno != EISDIR &&\n> errno != ENOTDIR)\n> return NULL;\n>\n> blame tells me it has something to do with dir/file conflicts and\n> commit a1c1d8170db.\n\nI'm aware mostly of what the code is doing. Yes some of it's used, some\nof it's not.\n\nMy upthread [1] wasn't clear enough, the intended message was \"hey this\nnew major refactoring/replacement of a core git format still has zero\ncoverage on at least some of the code it seeks to replace\" not \"hey\nwhat's this thing in refs.c doing?\".\n\nI.e. lack of test coverage on refactored code as part of a big patch\nseries is a lot scarier than the same lack of coverage on existing\nlong-established code we'll either not update, or in small incremental\nupdates.\n\nSo it was a gentle prod to maybe look into what the state of that in the\nreftable series as a whole is and fix the coverage gaps. I'd like the\nfeature in git.git, but this series has been stalled for a while. I for\none would feel a lot more confident in reviewing it if I knew I'd just\ndiscovered an outlier case.\n\nBut in any case, when I wrote[1] I didn't know about / had forgotten\n\"make cover\". As noted in t/README that can even give pretty HTML\noutput. It takes ages to run so I haven't run it on your series, but I\nran it on master a couple of days ago and it flagged those (and other)\nparts of refs.c.\n\nSo perhaps running it on the reftables series will find some useful\ncases to make up the test difference / potential bugs.\n\nOf course 100% line coverage isn't practical (hitting every BUG(...)),\nand even 100% line coverage != 100% test coverage. But it's a very good\nfirst stop to see glaring omissions. E.g. the refs.c cases I mentioned\nwhere we'resetting flag \"xyz\" in a() and then acting on \"xyz/!xyz\" in\nb().\n\n1. <871ree8eto.fsf@evledraar.gmail.com> \n\n"},{"id":"417461","messageId":"20210222004112.99268-1-stefanbeller@gmail.com","threadId":"54245","inReplyTo":"d57023d9f13d178dc95d7a74751923b80d1a5939.1607522429.git.gitgitgadget@gmail.com","subject":"[PATCH] refs: introduce API function to write invalid null ref","fromName":"Stefan Beller","fromEmail":"stefanbeller@gmail.com","sentAt":"2021-02-22T00:41:12Z","receivedAt":"2021-02-22T00:42:15Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Different ref backends will have different ways to write out the invalid 00..00\nref when starting a new worktree. Encapsulate this into a function and expose\nthe function in the refs API.\n\nSigned-off-by: Stefan Beller \n---\n\nHi Han-Wen,\n\nit's been a while since I looked at git source code, but today is the day!\nI was actually looking how the refs table work progresses and this patch\ncaught my attention. I think the changes in builtin/worktree.c (that \nif/else depending on the actual refs backend used)\ndemonstrate that the refs API layer is leaking implementation details.\n\nWhat do you think about rolling this patch first, and then implementing\nthe following part inside the reftable as a function?\n\n\nint reftable_write_invalid_head_ref(struct *ref_store, const char *dir)\n{\n\tstruct sb = STRBUF_INIT;\n\t/* the following adapted from this patch: */\n\n+\t\t/* XXX this is cut & paste from reftable_init_db. */\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", dir);\n+\t\twrite_file(sb.buf, \"%s\", \"ref: refs/heads/.invalid\\n\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs/heads\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n\n\treturn 0;\n}\n\nWhat do you think?\n\nCheers,\nStefan\n\n\n builtin/worktree.c | 6 +++---\n refs.c | 5 +++++\n refs.h | 3 +++\n refs/debug.c | 10 ++++++++++\n refs/files-backend.c | 13 +++++++++++++\n refs/packed-backend.c | 7 +++++++\n refs/refs-internal.h | 3 +++\n 7 files changed, 44 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/worktree.c b/builtin/worktree.c\nindex 1cd5c2016e..6f5d79e5ec 100644\n--- a/builtin/worktree.c\n+++ b/builtin/worktree.c\n@@ -330,9 +330,9 @@ static int add_worktree(const char *path, const char *refname,\n \t * worktree.\n \t */\n \tstrbuf_reset(&sb);\n-\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n-\twrite_file(sb.buf, \"%s\", oid_to_hex(&null_oid));\n-\tstrbuf_reset(&sb);\n+\n+\twrite_invalid_head(get_main_ref_store(the_repository), sb_repo.buf);\n+\n \tstrbuf_addf(&sb, \"%s/commondir\", sb_repo.buf);\n \twrite_file(sb.buf, \"../..\");\n \ndiff --git a/refs.c b/refs.c\nindex a665ed5e10..a7b415c386 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2454,3 +2454,8 @@ int copy_existing_ref(const char *oldref, const char *newref, const char *logmsg\n {\n \treturn refs_copy_existing_ref(get_main_ref_store(the_repository), oldref, newref, logmsg);\n }\n+\n+int write_invalid_head(struct ref_store *refs, const char *dir)\n+{\n+\treturn refs->be->write_invalid_head_ref(refs, dir);\n+}\ndiff --git a/refs.h b/refs.h\nindex 48970dfc7e..7a64f777d5 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -456,6 +456,9 @@ int refs_delete_refs(struct ref_store *refs, const char *msg,\n int delete_refs(const char *msg, struct string_list *refnames,\n \t\tunsigned int flags);\n \n+\n+int write_invalid_head(struct ref_store *refs, const char *dir);\n+\n /** Delete a reflog */\n int refs_delete_reflog(struct ref_store *refs, const char *refname);\n int delete_reflog(const char *refname);\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 922e64fa6a..9cfc3ca9f3 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -175,6 +175,15 @@ static int debug_copy_ref(struct ref_store *ref_store, const char *oldref,\n \treturn res;\n }\n \n+static int debug_write_invalid_head_ref(struct ref_store *ref_store,\n+\t\t\t\t\tconst char *dir)\n+{\n+\tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n+\tint res = drefs->refs->be->write_invalid_head_ref(drefs->refs, dir);\n+\ttrace_printf_key(&trace_refs, \"write_invalid_head: %s\\n\", dir);\n+\treturn res;\n+}\n+\n struct debug_ref_iterator {\n \tstruct ref_iterator base;\n \tstruct ref_iterator *iter;\n@@ -384,6 +393,7 @@ struct ref_storage_be refs_be_debug = {\n \tdebug_delete_refs,\n \tdebug_rename_ref,\n \tdebug_copy_ref,\n+\tdebug_write_invalid_head_ref,\n \n \tdebug_ref_iterator_begin,\n \tdebug_read_raw_ref,\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 4fdc68810b..b1c82a6f61 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3169,6 +3169,18 @@ static int files_init_db(struct ref_store *ref_store, struct strbuf *err)\n \treturn 0;\n }\n \n+static int files_write_invalid_head_ref(struct ref_store *ref_store,\n+\t\t\t\t\tconst char *dir)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\tstrbuf_addf(&sb, \"%s/HEAD\", dir);\n+\twrite_file(sb.buf, \"%s\", oid_to_hex(&null_oid));\n+\tstrbuf_release(&sb);\n+\n+\treturn 0;\n+}\n+\n struct ref_storage_be refs_be_files = {\n \tNULL,\n \t\"files\",\n@@ -3184,6 +3196,7 @@ struct ref_storage_be refs_be_files = {\n \tfiles_delete_refs,\n \tfiles_rename_ref,\n \tfiles_copy_ref,\n+\tfiles_write_invalid_head_ref,\n \n \tfiles_ref_iterator_begin,\n \tfiles_read_raw_ref,\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex b912f2505f..a1adb81777 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1591,6 +1591,12 @@ static int packed_copy_ref(struct ref_store *ref_store,\n \tBUG(\"packed reference store does not support copying references\");\n }\n \n+static int packed_write_invalid_head_ref(struct ref_store *ref_store,\n+\t\t\t\t\t const char *dir)\n+{\n+\tBUG(\"packed reference store does not support writing an invalid head ref\");\n+}\n+\n static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_store)\n {\n \treturn empty_ref_iterator_begin();\n@@ -1656,6 +1662,7 @@ struct ref_storage_be refs_be_packed = {\n \tpacked_delete_refs,\n \tpacked_rename_ref,\n \tpacked_copy_ref,\n+\tpacked_write_invalid_head_ref,\n \n \tpacked_ref_iterator_begin,\n \tpacked_read_raw_ref,\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 467f4b3c93..5055226f75 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -556,6 +556,8 @@ typedef int rename_ref_fn(struct ref_store *ref_store,\n typedef int copy_ref_fn(struct ref_store *ref_store,\n \t\t\t const char *oldref, const char *newref,\n \t\t\t const char *logmsg);\n+typedef int write_invalid_head_ref_fn(struct ref_store *ref_store,\n+\t\t\t\t const char *dir);\n \n /*\n * Iterate over the references in `ref_store` whose names start with\n@@ -655,6 +657,7 @@ struct ref_storage_be {\n \tdelete_refs_fn *delete_refs;\n \trename_ref_fn *rename_ref;\n \tcopy_ref_fn *copy_ref;\n+\twrite_invalid_head_ref_fn *write_invalid_head_ref;\n \n \tref_iterator_begin_fn *iterator_begin;\n \tread_raw_ref_fn *read_raw_ref;\n-- \n2.26.2.266.ge870325ee8\n\n"},{"id":"417462","messageId":"CAPig+cT9ZUqkOZWZS3+gEd3soh-xyfxu2yvQ_gY-LMgVV-rAiw@mail.gmail.com","threadId":"54245","inReplyTo":"20210222004112.99268-1-stefanbeller@gmail.com","subject":"Re: [PATCH] refs: introduce API function to write invalid null ref","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-02-22T01:20:02Z","receivedAt":"2021-02-22T01:20:57Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Feb 21, 2021 at 7:43 PM Stefan Beller wrote:\n> Different ref backends will have different ways to write out the invalid 00..00\n> ref when starting a new worktree. Encapsulate this into a function and expose\n> the function in the refs API.\n>\n> Signed-off-by: Stefan Beller \n> ---\n> it's been a while since I looked at git source code, but today is the day!\n> I was actually looking how the refs table work progresses and this patch\n> caught my attention. I think the changes in builtin/worktree.c (that\n> if/else depending on the actual refs backend used)\n> demonstrate that the refs API layer is leaking implementation details.\n\nWelcome back. I have a few comments in response to this proposal. See below...\n\n> diff --git a/builtin/worktree.c b/builtin/worktree.c\n> @@ -330,9 +330,9 @@ static int add_worktree(const char *path, const char *refname,\n> strbuf_reset(&sb);\n> - strbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n> - write_file(sb.buf, \"%s\", oid_to_hex(&null_oid));\n> - strbuf_reset(&sb);\n> +\n> + write_invalid_head(get_main_ref_store(the_repository), sb_repo.buf);\n> +\n> strbuf_addf(&sb, \"%s/commondir\", sb_repo.buf);\n> write_file(sb.buf, \"../..\");\n\nNit: This change ends up making the strbuf_reset() unnecessarily\ndistant from the location where it has most meaning (just before we\nstart using it again, which is the pattern in this function):\n\n strbuf_release(&sb);\n\n write_invalid_head(..., sb_repo.buf);\n\n strbuf_add(&sb, ...);\n\nIt would be clearer for the end result to be:\n\n write_invalid_head(..., sb_repo.buf);\n\n strbuf_release(&sb);\n strbuf_add(&sb, ...);\n\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> @@ -3169,6 +3169,18 @@ static int files_init_db(struct ref_store *ref_store, struct strbuf *err)\n> +static int files_write_invalid_head_ref(struct ref_store *ref_store,\n> + const char *dir)\n> +{\n> + struct strbuf sb = STRBUF_INIT;\n> +\n> + strbuf_addf(&sb, \"%s/HEAD\", dir);\n> + write_file(sb.buf, \"%s\", oid_to_hex(&null_oid));\n> + strbuf_release(&sb);\n> +\n> + return 0;\n> +}\n\nI'm not super enthused about the name write_invalid_head_ref(). Even\nif given a different name, I'm not necessarily enthused about it\nunconditionally writing a zero-OID. Do you foresee other cases in\nwhich callers will need to perform this precise operation?\n\nThe reason I ask is that the bit of code in\nbuiltin/worktree.c:add_worktree() which this patch targets is itself a\nhack to work around the shortcoming in which is_git_directory() won't\nconsider the newly-created worktree as being legitimate if it doesn't\nhave a well-formed HEAD ref. It's not visible in the context of this\npatch, but there is a comment just above the changes to\nbuiltin/worktree.c made by this patch:\n\n /*\n * This is to keep resolve_ref() happy. We need a valid HEAD\n * or is_git_directory() will reject the directory. Any value which\n * looks like an object ID will do since it will be immediately\n * replaced by the symbolic-ref or update-ref invocation in the new\n * worktree.\n */\n\nAs mentioned in the comment, it doesn't matter what the value of HEAD\nis; it just needs to be something that looks like a well-formed OID.\n\nSo, my lack of enthusiasm for write_invalid_head_ref() is twofold.\nFirst, this function is being introduced to paper over a hack, which\nleaves the new function smelling funny. Second, the \"invalid head\" in\nthe name and the unconditional zero-OID feel too special-purpose and\nfocus too much on the particular current implementation in\nbuiltin/worktree.c which happens to assign a zero-OID to HEAD. (At an\nearlier time, builtin/worktree.h assigned the value of HEAD from the\ncurrent worktree to HEAD in the newly-created worktree instead. But,\nas noted, the value is arbitrary -- it doesn't matter what it is -- it\njust needs to exist long enough to pacify is_git_directory() and is\nthen immediately overwritten.)\n\nOn the other hand, I could see this as acceptable if \"invalid\" is\nremoved from the function name and if it accepts an OID to write\nrather than unconditionally writing a zero-ID. In that case, it would\nbecome a generally useful function without the bad smells associated\nwith the too-special-purpose write_invalid_head_ref(). (But I haven't\nbeen paying attention to the ref backends work, so perhaps such a\nfunction already exists? I don't know.)\n"},{"id":"417463","messageId":"CAPig+cT9FDZ78mTmuvaOyzhc+JKV3U8DRBfFRezWr2bsW4gmGA@mail.gmail.com","threadId":"54245","inReplyTo":"CAPig+cT9ZUqkOZWZS3+gEd3soh-xyfxu2yvQ_gY-LMgVV-rAiw@mail.gmail.com","subject":"Re: [PATCH] refs: introduce API function to write invalid null ref","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-02-22T03:09:16Z","receivedAt":"2021-02-22T03:10:11Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Feb 21, 2021 at 8:20 PM Eric Sunshine wrote:\n> The reason I ask is that the bit of code in\n> builtin/worktree.c:add_worktree() which this patch targets is itself a\n> hack to work around the shortcoming in which is_git_directory() won't\n> consider the newly-created worktree as being legitimate if it doesn't\n> have a well-formed HEAD ref. [...]\n\nBy the way, the only reason the hack of creating a temporary HEAD\n(containing arbitrary OID) is needed is that add_worktree() shells out\nto invoke one of git-update-ref or git-symbolic-ref. It is that\nshell-out to invoke a Git command which triggers the necessity of\npacifying is_git_directory()...\n\n> On the other hand, I could see this as acceptable if \"invalid\" is\n> removed from the function name and if it accepts an OID to write\n> rather than unconditionally writing a zero-ID. In that case, it would\n> become a generally useful function without the bad smells associated\n> with the too-special-purpose write_invalid_head_ref().\n\n... so, a much cleaner fix would be to stop shelling out to set up\nHEAD in the newly-created worktree, and instead call C API to perform\nthose functions (update-ref and symbolic-ref) directly. If that C API\ndoes not yet exist, then it should be added.\n"},{"id":"417478","messageId":"CAFQ2z_Ob1Ego0vZgW1g3h34LF9H5ZggrLJY4FQHPGaCwVPq8pQ@mail.gmail.com","threadId":"54245","inReplyTo":"20210222004112.99268-1-stefanbeller@gmail.com","subject":"Re: [PATCH] refs: introduce API function to write invalid null ref","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-02-22T18:38:17Z","receivedAt":"2021-02-22T18:39:54Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Mon, Feb 22, 2021 at 1:41 AM Stefan Beller wrote:\n>\n> Different ref backends will have different ways to write out the invalid 00..00\n> ref when starting a new worktree. Encapsulate this into a function and expose\n> the function in the refs API.\n>\n> Signed-off-by: Stefan Beller \n> ---\n>\n> Hi Han-Wen,\n>\n> it's been a while since I looked at git source code, but today is the day!\n> I was actually looking how the refs table work progresses and this patch\n> caught my attention. I think the changes in builtin/worktree.c (that\n> if/else depending on the actual refs backend used)\n> demonstrate that the refs API layer is leaking implementation details.\n>\n> What do you think about rolling this patch first, and then implementing\n> the following part inside the reftable as a function?\n\nThe \"invalid HEAD\" hack is there to avoid confusing historical git\nimplementations. It's not a part of the \"modern\" refs API layer, so I\nthink we shouldn't add it as a method on the ref backend API.\n\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\nRegistergericht und -nummer: Hamburg, HRB 86891\nSitz der Gesellschaft: Hamburg\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"418981","messageId":"1bd136f6a6df9f31e3788f16a70b935e197d674d.1615580397.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v5.git.git.1615580397.gitgitgadget@gmail.com","subject":"[PATCH v5 03/15] reftable: add error related functionality","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-12T20:19:44Z","receivedAt":"2021-03-12T20:20:44Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable/ directory is structured as a library, so it cannot\ncrash on misuse. Instead, it returns an error codes.\n\nIn addition, the error code can be used to signal conditions from lower levels\nof the library to be handled by higher levels of the library. For example, a\ntransaction might legitimately write an empty reftable file, but in that case,\nwe'd want to shortcut the transaction overhead.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/error.c | 41 ++++++++++++++++++++++++++\n reftable/reftable-error.h | 62 +++++++++++++++++++++++++++++++++++++++\n 2 files changed, 103 insertions(+)\n create mode 100644 reftable/error.c\n create mode 100644 reftable/reftable-error.h\n\ndiff --git a/reftable/error.c b/reftable/error.c\nnew file mode 100644\nindex 000000000000..f6f16def9214\n--- /dev/null\n+++ b/reftable/error.c\n@@ -0,0 +1,41 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-error.h\"\n+\n+#include \n+\n+const char *reftable_error_str(int err)\n+{\n+\tstatic char buf[250];\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn \"I/O error\";\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn \"corrupt reftable file\";\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn \"file does not exist\";\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn \"data is outdated\";\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn \"misuse of the reftable API\";\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn \"zlib failure\";\n+\tcase REFTABLE_NAME_CONFLICT:\n+\t\treturn \"file/directory conflict\";\n+\tcase REFTABLE_EMPTY_TABLE_ERROR:\n+\t\treturn \"wrote empty table\";\n+\tcase REFTABLE_REFNAME_ERROR:\n+\t\treturn \"invalid refname\";\n+\tcase -1:\n+\t\treturn \"general error\";\n+\tdefault:\n+\t\tsnprintf(buf, sizeof(buf), \"unknown error code %d\", err);\n+\t\treturn buf;\n+\t}\n+}\ndiff --git a/reftable/reftable-error.h b/reftable/reftable-error.h\nnew file mode 100644\nindex 000000000000..6f89bedf1a58\n--- /dev/null\n+++ b/reftable/reftable-error.h\n@@ -0,0 +1,62 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_ERROR_H\n+#define REFTABLE_ERROR_H\n+\n+/*\n+ * Errors in reftable calls are signaled with negative integer return values. 0\n+ * means success.\n+ */\n+enum reftable_error {\n+\t/* Unexpected file system behavior */\n+\tREFTABLE_IO_ERROR = -2,\n+\n+\t/* Format inconsistency on reading data */\n+\tREFTABLE_FORMAT_ERROR = -3,\n+\n+\t/* File does not exist. Returned from block_source_from_file(), because\n+\t * it needs special handling in stack.\n+\t */\n+\tREFTABLE_NOT_EXIST_ERROR = -4,\n+\n+\t/* Trying to write out-of-date data. */\n+\tREFTABLE_LOCK_ERROR = -5,\n+\n+\t/* Misuse of the API:\n+\t * - on writing a record with NULL refname.\n+\t * - on writing a reftable_ref_record outside the table limits\n+\t * - on writing a ref or log record before the stack's\n+\t * next_update_inde*x\n+\t * - on writing a log record with multiline message with\n+\t * exact_log_message unset\n+\t * - on reading a reftable_ref_record from log iterator, or vice versa.\n+\t *\n+\t * When a call misuses the API, the internal state of the library is\n+\t * kept unchanged.\n+\t */\n+\tREFTABLE_API_ERROR = -6,\n+\n+\t/* Decompression error */\n+\tREFTABLE_ZLIB_ERROR = -7,\n+\n+\t/* Wrote a table without blocks. */\n+\tREFTABLE_EMPTY_TABLE_ERROR = -8,\n+\n+\t/* Dir/file conflict. */\n+\tREFTABLE_NAME_CONFLICT = -9,\n+\n+\t/* Invalid ref name. */\n+\tREFTABLE_REFNAME_ERROR = -10,\n+};\n+\n+/* convert the numeric error code to a string. The string should not be\n+ * deallocated. */\n+const char *reftable_error_str(int err);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"418982","messageId":"502a66befabe1b7c616e9bc0f0ef4843ab4bdf77.1615580397.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v5.git.git.1615580397.gitgitgadget@gmail.com","subject":"[PATCH v5 02/15] reftable: add LICENSE","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-12T20:19:43Z","receivedAt":"2021-03-12T20:20:45Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nTODO: relicense?\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/LICENSE | 31 +++++++++++++++++++++++++++++++\n 1 file changed, 31 insertions(+)\n create mode 100644 reftable/LICENSE\n\ndiff --git a/reftable/LICENSE b/reftable/LICENSE\nnew file mode 100644\nindex 000000000000..402e0f9356ba\n--- /dev/null\n+++ b/reftable/LICENSE\n@@ -0,0 +1,31 @@\n+BSD License\n+\n+Copyright (c) 2020, Google LLC\n+All rights reserved.\n+\n+Redistribution and use in source and binary forms, with or without\n+modification, are permitted provided that the following conditions are\n+met:\n+\n+* Redistributions of source code must retain the above copyright notice,\n+this list of conditions and the following disclaimer.\n+\n+* Redistributions in binary form must reproduce the above copyright\n+notice, this list of conditions and the following disclaimer in the\n+documentation and/or other materials provided with the distribution.\n+\n+* Neither the name of Google LLC nor the names of its contributors may\n+be used to endorse or promote products derived from this software\n+without specific prior written permission.\n+\n+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n+\"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n-- \ngitgitgadget\n\n"},{"id":"418983","messageId":"e1d1d9f498070fa9a55e740ec433630824340525.1615580397.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v5.git.git.1615580397.gitgitgadget@gmail.com","subject":"[PATCH v5 01/15] init-db: set the_repository->hash_algo early on","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-12T20:19:42Z","receivedAt":"2021-03-12T20:20:45Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable backend needs to know the hash algorithm for writing the\ninitialization hash table.\n\nThe initial reftable contains a symref HEAD => \"main\" (or \"master\"), which is\nagnostic to the size of hash value, but this is an exceptional circumstance, and\nthe reftable library does not cater to this exception. It insists that all\ntables in the stack have a consistent format ID for the hash algorithm.\n\nCall set_repo_hash_algo directly after calling validate_hash_algorithm() (which\nreads $GIT_DEFAULT_HASH).\n\nHelped-by: Junio C Hamano \nSigned-off-by: Han-Wen Nienhuys \n---\n builtin/init-db.c | 21 +++++++++++++++++++++\n 1 file changed, 21 insertions(+)\n\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex dcc45bef5148..7766661a4755 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -438,6 +438,27 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \n \tvalidate_hash_algorithm(&repo_fmt, hash);\n \n+\t/*\n+\t * At this point, the_repository we have in-core does not look\n+\t * anything like one that we would see initialized in an already\n+\t * working repository after calling setup_git_directory().\n+\t *\n+\t * Calling repository.c::initialize_the_repository() may have\n+\t * prepared the .index .objects and .parsed_objects members, but\n+\t * other members like .gitdir, .commondir, etc. have not been\n+\t * initialized.\n+\t *\n+\t * Many API functions assume they are working with the_repository\n+\t * that has sensibly been initialized, but because we haven't\n+\t * really read from an existing repository, we need to hand-craft\n+\t * the necessary members of the structure to get out of this\n+\t * chicken-and-egg situation.\n+\t *\n+\t * For now, we update the hash algorithm member to what the\n+\t * validate_hash_algorithm() call decided for us.\n+\t */\n+\trepo_set_hash_algo(the_repository, repo_fmt.hash_algo);\n+\n \treinit = create_default_files(template_dir, original_git_dir,\n \t\t\t\t initial_branch, &repo_fmt,\n \t\t\t\t flags & INIT_DB_QUIET);\n-- \ngitgitgadget\n\n"},{"id":"418984","messageId":"e41fa780882d229995fb69dcc807420a5468ffdb.1615580397.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v5.git.git.1615580397.gitgitgadget@gmail.com","subject":"[PATCH v5 05/15] reftable: add blocksource, an abstraction for random access reads","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-12T20:19:46Z","receivedAt":"2021-03-12T20:20:45Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is usually used with files for storage. However, we abstract\naway this using the blocksource data structure. This has two advantages:\n\n* log blocks are zlib compressed, and handling them is simplified if we can\n discard byte segments from within the block layer.\n\n* for unittests, it is useful to read and write in-memory. The blocksource\n allows us to abstract the data away from on-disk files.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/blocksource.c | 148 ++++++++++++++++++++++++++++++++\n reftable/blocksource.h | 22 +++++\n reftable/reftable-blocksource.h | 49 +++++++++++\n 4 files changed, 220 insertions(+)\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/reftable-blocksource.h\n\ndiff --git a/Makefile b/Makefile\nindex 43ed67fdb9db..82db2ad5e2fd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2384,6 +2384,7 @@ XDIFF_OBJS += xdiff/xutils.o\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n \n REFTABLE_TEST_OBJS += reftable/test_framework.o\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nnew file mode 100644\nindex 000000000000..25d4d95b52b8\n--- /dev/null\n+++ b/reftable/blocksource.c\n@@ -0,0 +1,148 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"reftable-blocksource.h\"\n+#include \"reftable-error.h\"\n+\n+static void strbuf_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void strbuf_close(void *b)\n+{\n+}\n+\n+static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct strbuf *b = (struct strbuf *)v;\n+\tassert(off + size <= b->len);\n+\tdest->data = reftable_calloc(size);\n+\tmemcpy(dest->data, b->buf + off, size);\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static uint64_t strbuf_size(void *b)\n+{\n+\treturn ((struct strbuf *)b)->len;\n+}\n+\n+static struct reftable_block_source_vtable strbuf_vtable = {\n+\t.size = &strbuf_size,\n+\t.read_block = &strbuf_read_block,\n+\t.return_block = &strbuf_return_block,\n+\t.close = &strbuf_close,\n+};\n+\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf)\n+{\n+\tassert(bs->ops == NULL);\n+\tbs->ops = &strbuf_vtable;\n+\tbs->arg = buf;\n+}\n+\n+static void malloc_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static struct reftable_block_source_vtable malloc_vtable = {\n+\t.return_block = &malloc_return_block,\n+};\n+\n+static struct reftable_block_source malloc_block_source_instance = {\n+\t.ops = &malloc_vtable,\n+};\n+\n+struct reftable_block_source malloc_block_source(void)\n+{\n+\treturn malloc_block_source_instance;\n+}\n+\n+struct file_block_source {\n+\tint fd;\n+\tuint64_t size;\n+};\n+\n+static uint64_t file_size(void *b)\n+{\n+\treturn ((struct file_block_source *)b)->size;\n+}\n+\n+static void file_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void file_close(void *b)\n+{\n+\tint fd = ((struct file_block_source *)b)->fd;\n+\tif (fd > 0) {\n+\t\tclose(fd);\n+\t\t((struct file_block_source *)b)->fd = 0;\n+\t}\n+\n+\treftable_free(b);\n+}\n+\n+static int file_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct file_block_source *b = (struct file_block_source *)v;\n+\tassert(off + size <= b->size);\n+\tdest->data = reftable_malloc(size);\n+\tif (pread(b->fd, dest->data, size, off) != size)\n+\t\treturn -1;\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static struct reftable_block_source_vtable file_vtable = {\n+\t.size = &file_size,\n+\t.read_block = &file_read_block,\n+\t.return_block = &file_return_block,\n+\t.close = &file_close,\n+};\n+\n+int reftable_block_source_from_file(struct reftable_block_source *bs,\n+\t\t\t\t const char *name)\n+{\n+\tstruct stat st = { 0 };\n+\tint err = 0;\n+\tint fd = open(name, O_RDONLY);\n+\tstruct file_block_source *p = NULL;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\treturn REFTABLE_NOT_EXIST_ERROR;\n+\t\t}\n+\t\treturn -1;\n+\t}\n+\n+\terr = fstat(fd, &st);\n+\tif (err < 0)\n+\t\treturn -1;\n+\n+\tp = reftable_calloc(sizeof(struct file_block_source));\n+\tp->size = st.st_size;\n+\tp->fd = fd;\n+\n+\tassert(bs->ops == NULL);\n+\tbs->ops = &file_vtable;\n+\tbs->arg = p;\n+\treturn 0;\n+}\ndiff --git a/reftable/blocksource.h b/reftable/blocksource.h\nnew file mode 100644\nindex 000000000000..072e2727ad20\n--- /dev/null\n+++ b/reftable/blocksource.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCKSOURCE_H\n+#define BLOCKSOURCE_H\n+\n+#include \"system.h\"\n+\n+struct reftable_block_source;\n+\n+/* Create an in-memory block source for reading reftables */\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf);\n+\n+struct reftable_block_source malloc_block_source(void);\n+\n+#endif\ndiff --git a/reftable/reftable-blocksource.h b/reftable/reftable-blocksource.h\nnew file mode 100644\nindex 000000000000..5aa3990a5732\n--- /dev/null\n+++ b/reftable/reftable-blocksource.h\n@@ -0,0 +1,49 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_BLOCKSOURCE_H\n+#define REFTABLE_BLOCKSOURCE_H\n+\n+#include \n+\n+/* block_source is a generic wrapper for a seekable readable file.\n+ */\n+struct reftable_block_source {\n+\tstruct reftable_block_source_vtable *ops;\n+\tvoid *arg;\n+};\n+\n+/* a contiguous segment of bytes. It keeps track of its generating block_source\n+ * so it can return itself into the pool. */\n+struct reftable_block {\n+\tuint8_t *data;\n+\tint len;\n+\tstruct reftable_block_source source;\n+};\n+\n+/* block_source_vtable are the operations that make up block_source */\n+struct reftable_block_source_vtable {\n+\t/* returns the size of a block source */\n+\tuint64_t (*size)(void *source);\n+\n+\t/* reads a segment from the block source. It is an error to read\n+\t beyond the end of the block */\n+\tint (*read_block)(void *source, struct reftable_block *dest,\n+\t\t\t uint64_t off, uint32_t size);\n+\t/* mark the block as read; may return the data back to malloc */\n+\tvoid (*return_block)(void *source, struct reftable_block *blockp);\n+\n+\t/* release all resources associated with the block source */\n+\tvoid (*close)(void *source);\n+};\n+\n+/* opens a file on the file system as a block_source */\n+int reftable_block_source_from_file(struct reftable_block_source *block_src,\n+\t\t\t\t const char *name);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"418985","messageId":"b6eed7283aac77d65d9127408b0571ab13e46650.1615580397.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v5.git.git.1615580397.gitgitgadget@gmail.com","subject":"[PATCH v5 08/15] reftable: a generic binary tree implementation","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-12T20:19:49Z","receivedAt":"2021-03-12T20:20:45Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format includes support for an (OID => ref) map. This map can speed\nup visibility and reachability checks. In particular, various operations along\nthe fetch/push path within Gerrit have ben sped up by using this structure.\n\nThe map is constructed with help of a binary tree. Object IDs are hashes, so\nthey are uniformly distributed. Hence, the tree does not attempt forced\nrebalancing.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 4 ++-\n reftable/tree.c | 63 ++++++++++++++++++++++++++++++++++++++++\n reftable/tree.h | 34 ++++++++++++++++++++++\n reftable/tree_test.c | 61 ++++++++++++++++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 162 insertions(+), 1 deletion(-)\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 3499996aa49e..57a61b359ea7 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2388,12 +2388,14 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/zlib-compat.o\n \n+REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n-REFTABLE_TEST_OBJS += reftable/basics_test.o\n+REFTABLE_TEST_OBJS += reftable/tree_test.o\n \n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \\\ndiff --git a/reftable/tree.c b/reftable/tree.c\nnew file mode 100644\nindex 000000000000..0061d14e306c\n--- /dev/null\n+++ b/reftable/tree.c\n@@ -0,0 +1,63 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert)\n+{\n+\tint res;\n+\tif (*rootp == NULL) {\n+\t\tif (!insert) {\n+\t\t\treturn NULL;\n+\t\t} else {\n+\t\t\tstruct tree_node *n =\n+\t\t\t\treftable_calloc(sizeof(struct tree_node));\n+\t\t\tn->key = key;\n+\t\t\t*rootp = n;\n+\t\t\treturn *rootp;\n+\t\t}\n+\t}\n+\n+\tres = compare(key, (*rootp)->key);\n+\tif (res < 0)\n+\t\treturn tree_search(key, &(*rootp)->left, compare, insert);\n+\telse if (res > 0)\n+\t\treturn tree_search(key, &(*rootp)->right, compare, insert);\n+\treturn *rootp;\n+}\n+\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg)\n+{\n+\tif (t->left != NULL) {\n+\t\tinfix_walk(t->left, action, arg);\n+\t}\n+\taction(arg, t->key);\n+\tif (t->right != NULL) {\n+\t\tinfix_walk(t->right, action, arg);\n+\t}\n+}\n+\n+void tree_free(struct tree_node *t)\n+{\n+\tif (t == NULL) {\n+\t\treturn;\n+\t}\n+\tif (t->left != NULL) {\n+\t\ttree_free(t->left);\n+\t}\n+\tif (t->right != NULL) {\n+\t\ttree_free(t->right);\n+\t}\n+\treftable_free(t);\n+}\ndiff --git a/reftable/tree.h b/reftable/tree.h\nnew file mode 100644\nindex 000000000000..fbdd002e23af\n--- /dev/null\n+++ b/reftable/tree.h\n@@ -0,0 +1,34 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TREE_H\n+#define TREE_H\n+\n+/* tree_node is a generic binary search tree. */\n+struct tree_node {\n+\tvoid *key;\n+\tstruct tree_node *left, *right;\n+};\n+\n+/* looks for `key` in `rootp` using `compare` as comparison function. If insert\n+ * is set, insert the key if it's not found. Else, return NULL.\n+ */\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert);\n+\n+/* performs an infix walk of the tree. */\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg);\n+\n+/*\n+ * deallocates the tree nodes recursively. Keys should be deallocated separately\n+ * by walking over the tree. */\n+void tree_free(struct tree_node *t);\n+\n+#endif\ndiff --git a/reftable/tree_test.c b/reftable/tree_test.c\nnew file mode 100644\nindex 000000000000..26d1e694252e\n--- /dev/null\n+++ b/reftable/tree_test.c\n@@ -0,0 +1,61 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static int test_compare(const void *a, const void *b)\n+{\n+\treturn (char *)a - (char *)b;\n+}\n+\n+struct curry {\n+\tvoid *last;\n+};\n+\n+static void check_increasing(void *arg, void *key)\n+{\n+\tstruct curry *c = (struct curry *)arg;\n+\tif (c->last != NULL) {\n+\t\tassert(test_compare(c->last, key) < 0);\n+\t}\n+\tc->last = key;\n+}\n+\n+static void test_tree(void)\n+{\n+\tstruct tree_node *root = NULL;\n+\n+\tvoid *values[11] = { NULL };\n+\tstruct tree_node *nodes[11] = { NULL };\n+\tint i = 1;\n+\tstruct curry c = { NULL };\n+\tdo {\n+\t\tnodes[i] = tree_search(values + i, &root, &test_compare, 1);\n+\t\ti = (i * 7) % 11;\n+\t} while (i != 1);\n+\n+\tfor (i = 1; i < ARRAY_SIZE(nodes); i++) {\n+\t\tassert(values + i == nodes[i]->key);\n+\t\tassert(nodes[i] ==\n+\t\t tree_search(values + i, &root, &test_compare, 0));\n+\t}\n+\n+\tinfix_walk(root, check_increasing, &c);\n+\ttree_free(root);\n+}\n+\n+int tree_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_tree);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex c9deeaf08c7a..050551fa6985 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,5 +6,6 @@ int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"418986","messageId":"pull.847.v5.git.git.1615580397.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v4.git.git.1607522429.gitgitgadget@gmail.com","subject":"[PATCH v5 00/15] reftable library","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-12T20:19:41Z","receivedAt":"2021-03-12T20:20:45Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"This splits the giant commit from\nhttps://github.com/gitgitgadget/git/pull/539 into a series of smaller\ncommits, which build and have unittests.\n\nThe final commit should also be split up, but I want to wait until we have\nconsensus that the bottom commits look good.\n\nversion 12 Mar 2021:\n\n * tagged unions for reftable_log_record\n * some fixups by jun\n\nBased on github.com/hanwen/reftable at\nc33a40ad23dd622d8c72139c41a2be1cfa063e5f Add random suffix to table names\n\nHan-Wen Nienhuys (14):\n init-db: set the_repository->hash_algo early on\n reftable: add LICENSE\n reftable: add error related functionality\n reftable: utility functions\n reftable: add blocksource, an abstraction for random access reads\n reftable: (de)serialization for the polymorphic record type.\n reftable: reading/writing blocks\n reftable: a generic binary tree implementation\n reftable: write reftable files\n reftable: read reftable files\n reftable: reftable file level tests\n reftable: rest of library\n Reftable support for git-core\n Add \"test-tool dump-reftable\" command.\n\nSZEDER Gábor (1):\n git-prompt: prepare for reftable refs backend\n\n Documentation/config/extensions.txt | 9 +\n .../technical/repository-version.txt | 7 +\n Makefile | 46 +-\n builtin/clone.c | 5 +-\n builtin/init-db.c | 76 +-\n builtin/worktree.c | 27 +-\n cache.h | 8 +-\n config.mak.uname | 2 +-\n contrib/buildsystems/CMakeLists.txt | 14 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n contrib/completion/git-prompt.sh | 7 +-\n refs.c | 27 +-\n refs.h | 3 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1446 +++++++++++++++++\n reftable/.gitattributes | 1 +\n reftable/LICENSE | 31 +\n reftable/VERSION | 1 +\n reftable/basics.c | 128 ++\n reftable/basics.h | 60 +\n reftable/basics_test.c | 98 ++\n reftable/block.c | 440 +++++\n reftable/block.h | 127 ++\n reftable/block_test.c | 121 ++\n reftable/blocksource.c | 148 ++\n reftable/blocksource.h | 22 +\n reftable/constants.h | 21 +\n reftable/dump.c | 213 +++\n reftable/error.c | 41 +\n reftable/iter.c | 248 +++\n reftable/iter.h | 75 +\n reftable/merged.c | 366 +++++\n reftable/merged.h | 35 +\n reftable/merged_test.c | 343 ++++\n reftable/pq.c | 115 ++\n reftable/pq.h | 32 +\n reftable/publicbasics.c | 58 +\n reftable/reader.c | 733 +++++++++\n reftable/reader.h | 75 +\n reftable/record.c | 1203 ++++++++++++++\n reftable/record.h | 139 ++\n reftable/record_test.c | 405 +++++\n reftable/refname.c | 209 +++\n reftable/refname.h | 29 +\n reftable/refname_test.c | 102 ++\n reftable/reftable-blocksource.h | 49 +\n reftable/reftable-error.h | 62 +\n reftable/reftable-generic.h | 48 +\n reftable/reftable-iterator.h | 37 +\n reftable/reftable-malloc.h | 18 +\n reftable/reftable-merged.h | 68 +\n reftable/reftable-reader.h | 89 +\n reftable/reftable-record.h | 114 ++\n reftable/reftable-stack.h | 120 ++\n reftable/reftable-tests.h | 22 +\n reftable/reftable-writer.h | 147 ++\n reftable/reftable.c | 98 ++\n reftable/reftable_test.c | 583 +++++++\n reftable/stack.c | 1260 ++++++++++++++\n reftable/stack.h | 41 +\n reftable/stack_test.c | 809 +++++++++\n reftable/system.h | 32 +\n reftable/test_framework.c | 23 +\n reftable/test_framework.h | 53 +\n reftable/tree.c | 63 +\n reftable/tree.h | 34 +\n reftable/tree_test.c | 61 +\n reftable/writer.c | 691 ++++++++\n reftable/writer.h | 50 +\n reftable/zlib-compat.c | 92 ++\n repository.c | 2 +\n repository.h | 3 +\n setup.c | 9 +-\n t/helper/test-reftable.c | 20 +\n t/helper/test-tool.c | 4 +-\n t/helper/test-tool.h | 2 +\n t/t0031-reftable.sh | 203 +++\n t/t0032-reftable-unittest.sh | 15 +\n t/t1409-avoid-packing-refs.sh | 6 +\n t/t1450-fsck.sh | 6 +\n t/t3210-pack-refs.sh | 6 +\n t/test-lib.sh | 5 +\n 82 files changed, 12213 insertions(+), 40 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100644 reftable/.gitattributes\n create mode 100644 reftable/LICENSE\n create mode 100644 reftable/VERSION\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/basics_test.c\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/dump.c\n create mode 100644 reftable/error.c\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n create mode 100644 reftable/reftable-blocksource.h\n create mode 100644 reftable/reftable-error.h\n create mode 100644 reftable/reftable-generic.h\n create mode 100644 reftable/reftable-iterator.h\n create mode 100644 reftable/reftable-malloc.h\n create mode 100644 reftable/reftable-merged.h\n create mode 100644 reftable/reftable-reader.h\n create mode 100644 reftable/reftable-record.h\n create mode 100644 reftable/reftable-stack.h\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/reftable-writer.h\n create mode 100644 reftable/reftable.c\n create mode 100644 reftable/reftable_test.c\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n create mode 100644 reftable/zlib-compat.c\n create mode 100644 t/helper/test-reftable.c\n create mode 100755 t/t0031-reftable.sh\n create mode 100755 t/t0032-reftable-unittest.sh\n\n\nbase-commit: 13d7ab6b5d7929825b626f050b62a11241ea4945\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-847%2Fhanwen%2Flibreftable-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-847/hanwen/libreftable-v5\nPull-Request: https://github.com/git/git/pull/847\n\nRange-diff vs v4:\n\n 1: 40ac041d0efe ! 1: e1d1d9f49807 init-db: set the_repository->hash_algo early on\n @@ builtin/init-db.c: int init_db(const char *git_dir, const char *real_git_dir,\n +\trepo_set_hash_algo(the_repository, repo_fmt.hash_algo);\n +\n \treinit = create_default_files(template_dir, original_git_dir,\n - \t\t\t\t initial_branch, &repo_fmt);\n - \tif (reinit && initial_branch)\n + \t\t\t\t initial_branch, &repo_fmt,\n + \t\t\t\t flags & INIT_DB_QUIET);\n 2: cc0fe58bd333 = 2: 502a66befabe reftable: add LICENSE\n 3: 798f680b800b = 3: 1bd136f6a6df reftable: add error related functionality\n 4: 2f55ff6d2406 ! 4: 5bbbafb428e9 reftable: utility functions\n @@ t/helper/test-reftable.c (new)\n \n ## t/helper/test-tool.c ##\n @@ t/helper/test-tool.c: static struct test_cmd cmds[] = {\n - \t{ \"path-utils\", cmd__path_utils },\n + \t{ \"pcre2-config\", cmd__pcre2_config },\n \t{ \"pkt-line\", cmd__pkt_line },\n \t{ \"prio-queue\", cmd__prio_queue },\n -\t{ \"proc-receive\", cmd__proc_receive},\n 5: aaa817f8598b = 5: e41fa780882d reftable: add blocksource, an abstraction for random access reads\n 6: e13afe315258 ! 6: 390eaca4fc8c reftable: (de)serialization for the polymorphic record type.\n @@ reftable/record.c (new)\n +{\n +\tchar hex[SHA256_SIZE + 1] = { 0 };\n +\n -+\tprintf(\"log{%s(%\" PRIu64 \") %s <%s> %\" PRIu64 \" %04d\\n\", log->refname,\n -+\t log->update_index, log->name, log->email, log->time,\n -+\t log->tz_offset);\n -+\thex_format(hex, log->old_hash, hash_size(hash_id));\n -+\tprintf(\"%s => \", hex);\n -+\thex_format(hex, log->new_hash, hash_size(hash_id));\n -+\tprintf(\"%s\\n\\n%s\\n}\\n\", hex, log->message);\n ++\tswitch (log->value_type) {\n ++\tcase REFTABLE_LOG_DELETION:\n ++\t\tprintf(\"log{%s(%\" PRIu64 \") delete\", log->refname,\n ++\t\t log->update_index);\n ++\t\tbreak;\n ++\tcase REFTABLE_LOG_UPDATE:\n ++\t\tprintf(\"log{%s(%\" PRIu64 \") %s <%s> %\" PRIu64 \" %04d\\n\",\n ++\t\t log->refname, log->update_index, log->update.name,\n ++\t\t log->update.email, log->update.time,\n ++\t\t log->update.tz_offset);\n ++\t\thex_format(hex, log->update.old_hash, hash_size(hash_id));\n ++\t\tprintf(\"%s => \", hex);\n ++\t\thex_format(hex, log->update.new_hash, hash_size(hash_id));\n ++\t\tprintf(\"%s\\n\\n%s\\n}\\n\", hex, log->update.message);\n ++\t\tbreak;\n ++\t}\n +}\n +\n +static void reftable_log_record_key(const void *r, struct strbuf *dest)\n @@ reftable/record.c (new)\n +\tif (dst->refname != NULL) {\n +\t\tdst->refname = xstrdup(dst->refname);\n +\t}\n -+\tif (dst->email != NULL) {\n -+\t\tdst->email = xstrdup(dst->email);\n -+\t}\n -+\tif (dst->name != NULL) {\n -+\t\tdst->name = xstrdup(dst->name);\n -+\t}\n -+\tif (dst->message != NULL) {\n -+\t\tdst->message = xstrdup(dst->message);\n -+\t}\n ++\tswitch (dst->value_type) {\n ++\tcase REFTABLE_LOG_DELETION:\n ++\t\tbreak;\n ++\tcase REFTABLE_LOG_UPDATE:\n ++\t\tif (dst->update.email != NULL) {\n ++\t\t\tdst->update.email = xstrdup(dst->update.email);\n ++\t\t}\n ++\t\tif (dst->update.name != NULL) {\n ++\t\t\tdst->update.name = xstrdup(dst->update.name);\n ++\t\t}\n ++\t\tif (dst->update.message != NULL) {\n ++\t\t\tdst->update.message = xstrdup(dst->update.message);\n ++\t\t}\n +\n -+\tif (dst->new_hash != NULL) {\n -+\t\tdst->new_hash = reftable_malloc(hash_size);\n -+\t\tmemcpy(dst->new_hash, src->new_hash, hash_size);\n -+\t}\n -+\tif (dst->old_hash != NULL) {\n -+\t\tdst->old_hash = reftable_malloc(hash_size);\n -+\t\tmemcpy(dst->old_hash, src->old_hash, hash_size);\n ++\t\tif (dst->update.new_hash != NULL) {\n ++\t\t\tdst->update.new_hash = reftable_malloc(hash_size);\n ++\t\t\tmemcpy(dst->update.new_hash, src->update.new_hash,\n ++\t\t\t hash_size);\n ++\t\t}\n ++\t\tif (dst->update.old_hash != NULL) {\n ++\t\t\tdst->update.old_hash = reftable_malloc(hash_size);\n ++\t\t\tmemcpy(dst->update.old_hash, src->update.old_hash,\n ++\t\t\t hash_size);\n ++\t\t}\n ++\t\tbreak;\n +\t}\n +}\n +\n @@ reftable/record.c (new)\n +void reftable_log_record_release(struct reftable_log_record *r)\n +{\n +\treftable_free(r->refname);\n -+\treftable_free(r->new_hash);\n -+\treftable_free(r->old_hash);\n -+\treftable_free(r->name);\n -+\treftable_free(r->email);\n -+\treftable_free(r->message);\n ++\tswitch (r->value_type) {\n ++\tcase REFTABLE_LOG_DELETION:\n ++\t\tbreak;\n ++\tcase REFTABLE_LOG_UPDATE:\n ++\t\treftable_free(r->update.new_hash);\n ++\t\treftable_free(r->update.old_hash);\n ++\t\treftable_free(r->update.name);\n ++\t\treftable_free(r->update.email);\n ++\t\treftable_free(r->update.message);\n ++\t\tbreak;\n ++\t}\n +\tmemset(r, 0, sizeof(struct reftable_log_record));\n +}\n +\n @@ reftable/record.c (new)\n +\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n +\tstruct string_view start = s;\n +\tint n = 0;\n -+\tuint8_t *oldh = r->old_hash;\n -+\tuint8_t *newh = r->new_hash;\n ++\tuint8_t *oldh = NULL;\n ++\tuint8_t *newh = NULL;\n +\tif (reftable_log_record_is_deletion(r))\n +\t\treturn 0;\n +\n ++\toldh = r->update.old_hash;\n ++\tnewh = r->update.new_hash;\n +\tif (oldh == NULL) {\n +\t\toldh = zero;\n +\t}\n @@ reftable/record.c (new)\n +\tmemcpy(s.buf + hash_size, newh, hash_size);\n +\tstring_view_consume(&s, 2 * hash_size);\n +\n -+\tn = encode_string(r->name ? r->name : \"\", s);\n ++\tn = encode_string(r->update.name ? r->update.name : \"\", s);\n +\tif (n < 0)\n +\t\treturn -1;\n +\tstring_view_consume(&s, n);\n +\n -+\tn = encode_string(r->email ? r->email : \"\", s);\n ++\tn = encode_string(r->update.email ? r->update.email : \"\", s);\n +\tif (n < 0)\n +\t\treturn -1;\n +\tstring_view_consume(&s, n);\n +\n -+\tn = put_var_int(&s, r->time);\n ++\tn = put_var_int(&s, r->update.time);\n +\tif (n < 0)\n +\t\treturn -1;\n +\tstring_view_consume(&s, n);\n @@ reftable/record.c (new)\n +\tif (s.len < 2)\n +\t\treturn -1;\n +\n -+\tput_be16(s.buf, r->tz_offset);\n ++\tput_be16(s.buf, r->update.tz_offset);\n +\tstring_view_consume(&s, 2);\n +\n -+\tn = encode_string(r->message ? r->message : \"\", s);\n ++\tn = encode_string(r->update.message ? r->update.message : \"\", s);\n +\tif (n < 0)\n +\t\treturn -1;\n +\tstring_view_consume(&s, n);\n @@ reftable/record.c (new)\n +\n +\tr->update_index = (~max) - ts;\n +\n -+\tif (val_type == 0) {\n -+\t\tFREE_AND_NULL(r->old_hash);\n -+\t\tFREE_AND_NULL(r->new_hash);\n -+\t\tFREE_AND_NULL(r->message);\n -+\t\tFREE_AND_NULL(r->email);\n -+\t\tFREE_AND_NULL(r->name);\n -+\t\treturn 0;\n ++\tif (val_type != r->value_type) {\n ++\t\tswitch (r->value_type) {\n ++\t\tcase REFTABLE_LOG_UPDATE:\n ++\t\t\tFREE_AND_NULL(r->update.old_hash);\n ++\t\t\tFREE_AND_NULL(r->update.new_hash);\n ++\t\t\tFREE_AND_NULL(r->update.message);\n ++\t\t\tFREE_AND_NULL(r->update.email);\n ++\t\t\tFREE_AND_NULL(r->update.name);\n ++\t\t\tbreak;\n ++\t\tcase REFTABLE_LOG_DELETION:\n ++\t\t\tbreak;\n ++\t\t}\n +\t}\n +\n ++\tr->value_type = val_type;\n ++\tif (val_type == REFTABLE_LOG_DELETION)\n ++\t\treturn 0;\n ++\n +\tif (in.len < 2 * hash_size)\n +\t\treturn REFTABLE_FORMAT_ERROR;\n +\n -+\tr->old_hash = reftable_realloc(r->old_hash, hash_size);\n -+\tr->new_hash = reftable_realloc(r->new_hash, hash_size);\n ++\tr->update.old_hash = reftable_realloc(r->update.old_hash, hash_size);\n ++\tr->update.new_hash = reftable_realloc(r->update.new_hash, hash_size);\n +\n -+\tmemcpy(r->old_hash, in.buf, hash_size);\n -+\tmemcpy(r->new_hash, in.buf + hash_size, hash_size);\n ++\tmemcpy(r->update.old_hash, in.buf, hash_size);\n ++\tmemcpy(r->update.new_hash, in.buf + hash_size, hash_size);\n +\n +\tstring_view_consume(&in, 2 * hash_size);\n +\n @@ reftable/record.c (new)\n +\t\tgoto done;\n +\tstring_view_consume(&in, n);\n +\n -+\tr->name = reftable_realloc(r->name, dest.len + 1);\n -+\tmemcpy(r->name, dest.buf, dest.len);\n -+\tr->name[dest.len] = 0;\n ++\tr->update.name = reftable_realloc(r->update.name, dest.len + 1);\n ++\tmemcpy(r->update.name, dest.buf, dest.len);\n ++\tr->update.name[dest.len] = 0;\n +\n +\tstrbuf_reset(&dest);\n +\tn = decode_string(&dest, in);\n @@ reftable/record.c (new)\n +\t\tgoto done;\n +\tstring_view_consume(&in, n);\n +\n -+\tr->email = reftable_realloc(r->email, dest.len + 1);\n -+\tmemcpy(r->email, dest.buf, dest.len);\n -+\tr->email[dest.len] = 0;\n ++\tr->update.email = reftable_realloc(r->update.email, dest.len + 1);\n ++\tmemcpy(r->update.email, dest.buf, dest.len);\n ++\tr->update.email[dest.len] = 0;\n +\n +\tts = 0;\n +\tn = get_var_int(&ts, &in);\n +\tif (n < 0)\n +\t\tgoto done;\n +\tstring_view_consume(&in, n);\n -+\tr->time = ts;\n ++\tr->update.time = ts;\n +\tif (in.len < 2)\n +\t\tgoto done;\n +\n -+\tr->tz_offset = get_be16(in.buf);\n ++\tr->update.tz_offset = get_be16(in.buf);\n +\tstring_view_consume(&in, 2);\n +\n +\tstrbuf_reset(&dest);\n @@ reftable/record.c (new)\n +\t\tgoto done;\n +\tstring_view_consume(&in, n);\n +\n -+\tr->message = reftable_realloc(r->message, dest.len + 1);\n -+\tmemcpy(r->message, dest.buf, dest.len);\n -+\tr->message[dest.len] = 0;\n ++\tr->update.message = reftable_realloc(r->update.message, dest.len + 1);\n ++\tmemcpy(r->update.message, dest.buf, dest.len);\n ++\tr->update.message[dest.len] = 0;\n +\n +\tstrbuf_release(&dest);\n +\treturn start.len - in.len;\n @@ reftable/record.c (new)\n +int reftable_log_record_equal(struct reftable_log_record *a,\n +\t\t\t struct reftable_log_record *b, int hash_size)\n +{\n -+\treturn null_streq(a->name, b->name) && null_streq(a->email, b->email) &&\n -+\t null_streq(a->message, b->message) &&\n -+\t zero_hash_eq(a->old_hash, b->old_hash, hash_size) &&\n -+\t zero_hash_eq(a->new_hash, b->new_hash, hash_size) &&\n -+\t a->time == b->time && a->tz_offset == b->tz_offset &&\n -+\t a->update_index == b->update_index;\n ++\tif (!(null_streq(a->refname, b->refname) &&\n ++\t a->update_index == b->update_index &&\n ++\t a->value_type == b->value_type))\n ++\t\treturn 0;\n ++\n ++\tswitch (a->value_type) {\n ++\tcase REFTABLE_LOG_DELETION:\n ++\t\treturn 1;\n ++\tcase REFTABLE_LOG_UPDATE:\n ++\t\treturn null_streq(a->update.name, b->update.name) &&\n ++\t\t a->update.time == b->update.time &&\n ++\t\t a->update.tz_offset == b->update.tz_offset &&\n ++\t\t null_streq(a->update.email, b->update.email) &&\n ++\t\t null_streq(a->update.message, b->update.message) &&\n ++\t\t zero_hash_eq(a->update.old_hash, b->update.old_hash,\n ++\t\t\t\t hash_size) &&\n ++\t\t zero_hash_eq(a->update.new_hash, b->update.new_hash,\n ++\t\t\t\t hash_size);\n ++\t}\n ++\n ++\tabort();\n +}\n +\n +static int reftable_log_record_is_deletion_void(const void *p)\n @@ reftable/record.c (new)\n +\n +int reftable_log_record_is_deletion(const struct reftable_log_record *log)\n +{\n -+\treturn (log->new_hash == NULL && log->old_hash == NULL &&\n -+\t\tlog->name == NULL && log->email == NULL &&\n -+\t\tlog->message == NULL && log->time == 0 && log->tz_offset == 0 &&\n -+\t\tlog->message == NULL);\n ++\treturn (log->value_type == REFTABLE_LOG_DELETION);\n +}\n +\n +void string_view_consume(struct string_view *s, int n)\n @@ reftable/record.h (new)\n ## reftable/record_test.c (new) ##\n @@\n +/*\n -+Copyright 2020 Google LLC\n ++ Copyright 2020 Google LLC\n +\n -+Use of this source code is governed by a BSD-style\n -+license that can be found in the LICENSE file or at\n -+https://developers.google.com/open-source/licenses/bsd\n ++ Use of this source code is governed by a BSD-style\n ++ license that can be found in the LICENSE file or at\n ++ https://developers.google.com/open-source/licenses/bsd\n +*/\n +\n +#include \"record.h\"\n @@ reftable/record_test.c (new)\n +\tstruct reftable_log_record in[2] = {\n +\t\t{\n +\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n -+\t\t\t.old_hash = reftable_malloc(SHA1_SIZE),\n -+\t\t\t.new_hash = reftable_malloc(SHA1_SIZE),\n -+\t\t\t.name = xstrdup(\"han-wen\"),\n -+\t\t\t.email = xstrdup(\"hanwen@google.com\"),\n -+\t\t\t.message = xstrdup(\"test\"),\n +\t\t\t.update_index = 42,\n -+\t\t\t.time = 1577123507,\n -+\t\t\t.tz_offset = 100,\n ++\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n ++\t\t\t.update = {\n ++\t\t\t\t.old_hash = reftable_malloc(SHA1_SIZE),\n ++\t\t\t\t.new_hash = reftable_malloc(SHA1_SIZE),\n ++\t\t\t\t.name = xstrdup(\"han-wen\"),\n ++\t\t\t\t.email = xstrdup(\"hanwen@google.com\"),\n ++\t\t\t\t.message = xstrdup(\"test\"),\n ++\t\t\t\t.time = 1577123507,\n ++\t\t\t\t.tz_offset = 100,\n ++\t\t\t}\n +\t\t},\n +\t\t{\n +\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n +\t\t\t.update_index = 22,\n ++\t\t\t.value_type = REFTABLE_LOG_DELETION,\n +\t\t}\n +\t};\n -+\tset_test_hash(in[0].new_hash, 1);\n -+\tset_test_hash(in[0].old_hash, 2);\n ++\tset_test_hash(in[0].update.new_hash, 1);\n ++\tset_test_hash(in[0].update.old_hash, 2);\n +\tfor (int i = 0; i < ARRAY_SIZE(in); i++) {\n +\t\tstruct reftable_record rec = { NULL };\n +\t\tstruct strbuf key = STRBUF_INIT;\n @@ reftable/record_test.c (new)\n +\t\t/* populate out, to check for leaks. */\n +\t\tstruct reftable_log_record out = {\n +\t\t\t.refname = xstrdup(\"old name\"),\n -+\t\t\t.new_hash = reftable_calloc(SHA1_SIZE),\n -+\t\t\t.old_hash = reftable_calloc(SHA1_SIZE),\n -+\t\t\t.name = xstrdup(\"old name\"),\n -+\t\t\t.email = xstrdup(\"old@email\"),\n -+\t\t\t.message = xstrdup(\"old message\"),\n ++\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n ++\t\t\t.update = {\n ++\t\t\t\t.new_hash = reftable_calloc(SHA1_SIZE),\n ++\t\t\t\t.old_hash = reftable_calloc(SHA1_SIZE),\n ++\t\t\t\t.name = xstrdup(\"old name\"),\n ++\t\t\t\t.email = xstrdup(\"old@email\"),\n ++\t\t\t\t.message = xstrdup(\"old message\"),\n ++\t\t\t},\n +\t\t};\n +\t\tstruct reftable_record rec_out = { NULL };\n +\t\tint n, m, valtype;\n @@ reftable/reftable-record.h (new)\n +\tchar *refname;\n +\tuint64_t update_index; /* logical timestamp of a transactional update.\n +\t\t\t\t*/\n -+\tuint8_t *new_hash;\n -+\tuint8_t *old_hash;\n -+\tchar *name;\n -+\tchar *email;\n -+\tuint64_t time;\n -+\tint16_t tz_offset;\n -+\tchar *message;\n ++\n ++\tenum {\n ++\t\t/* tombstone to hide deletions from earlier tables */\n ++\t\tREFTABLE_LOG_DELETION = 0x0,\n ++\n ++\t\t/* a simple update */\n ++\t\tREFTABLE_LOG_UPDATE = 0x1,\n ++#define REFTABLE_NR_LOG_VALUETYPES 2\n ++\t} value_type;\n ++\n ++\tunion {\n ++\t\tstruct {\n ++\t\t\tuint8_t *new_hash;\n ++\t\t\tuint8_t *old_hash;\n ++\t\t\tchar *name;\n ++\t\t\tchar *email;\n ++\t\t\tuint64_t time;\n ++\t\t\tint16_t tz_offset;\n ++\t\t\tchar *message;\n ++\t\t} update;\n ++\t};\n +};\n +\n +/* returns whether 'ref' represents the deletion of a log record. */\n 7: a48b9937642e = 7: b108525009d9 reftable: reading/writing blocks\n 8: 6968dbc3828f = 8: b6eed7283aac reftable: a generic binary tree implementation\n 9: ff5b424d12fb ! 9: f2f005afca19 reftable: write reftable files\n @@ reftable/writer.c (new)\n +\treturn err;\n +}\n +\n -+int reftable_writer_add_log(struct reftable_writer *w,\n -+\t\t\t struct reftable_log_record *log)\n ++static int reftable_writer_add_log_verbatim(struct reftable_writer *w,\n ++\t\t\t\t\t struct reftable_log_record *log)\n +{\n +\tstruct reftable_record rec = { NULL };\n -+\tchar *input_log_message = log->message;\n -+\tstruct strbuf cleaned_message = STRBUF_INIT;\n -+\tint err;\n -+\tif (log->refname == NULL)\n -+\t\treturn REFTABLE_API_ERROR;\n -+\n +\tif (w->block_writer != NULL &&\n +\t block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {\n +\t\tint err = writer_finish_public_section(w);\n @@ reftable/writer.c (new)\n +\t\t\treturn err;\n +\t}\n +\n -+\tif (!w->opts.exact_log_message && log->message != NULL) {\n -+\t\tstrbuf_addstr(&cleaned_message, log->message);\n ++\tw->next -= w->pending_padding;\n ++\tw->pending_padding = 0;\n ++\n ++\treftable_record_from_log(&rec, log);\n ++\treturn writer_add_record(w, &rec);\n ++}\n ++\n ++int reftable_writer_add_log(struct reftable_writer *w,\n ++\t\t\t struct reftable_log_record *log)\n ++{\n ++\tchar *input_log_message = NULL;\n ++\tstruct strbuf cleaned_message = STRBUF_INIT;\n ++\tint err = 0;\n ++\n ++\tif (log->value_type == REFTABLE_LOG_DELETION)\n ++\t\treturn reftable_writer_add_log_verbatim(w, log);\n ++\n ++\tif (log->refname == NULL)\n ++\t\treturn REFTABLE_API_ERROR;\n ++\n ++\tinput_log_message = log->update.message;\n ++\tif (!w->opts.exact_log_message && log->update.message != NULL) {\n ++\t\tstrbuf_addstr(&cleaned_message, log->update.message);\n +\t\twhile (cleaned_message.len &&\n +\t\t cleaned_message.buf[cleaned_message.len - 1] == '\\n')\n +\t\t\tstrbuf_setlen(&cleaned_message,\n @@ reftable/writer.c (new)\n +\t\t\tgoto done;\n +\t\t}\n +\t\tstrbuf_addstr(&cleaned_message, \"\\n\");\n -+\t\tlog->message = cleaned_message.buf;\n ++\t\tlog->update.message = cleaned_message.buf;\n +\t}\n +\n -+\tw->next -= w->pending_padding;\n -+\tw->pending_padding = 0;\n -+\n -+\treftable_record_from_log(&rec, log);\n -+\terr = writer_add_record(w, &rec);\n -+\n ++\terr = reftable_writer_add_log_verbatim(w, log);\n ++\tlog->update.message = input_log_message;\n +done:\n -+\tlog->message = input_log_message;\n +\tstrbuf_release(&cleaned_message);\n +\treturn err;\n +}\n 10: 3c2de7dcc65c = 10: 1f17d5edab52 reftable: read reftable files\n 11: 03681b820d13 ! 11: c916629c562a reftable: reftable file level tests\n @@ reftable/reftable_test.c (new)\n +\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n +\n +\t\tlog.refname = name;\n -+\t\tlog.new_hash = hash;\n +\t\tlog.update_index = update_index;\n -+\t\tlog.message = \"message\";\n ++\t\tlog.value_type = REFTABLE_LOG_UPDATE;\n ++\t\tlog.update.new_hash = hash;\n ++\t\tlog.update.message = \"message\";\n +\n +\t\tn = reftable_writer_add_log(w, &log);\n +\t\tEXPECT(n == 0);\n @@ reftable/reftable_test.c (new)\n +\t\t.block_size = 4096,\n +\t};\n +\tint err;\n -+\tstruct reftable_log_record log = {\n -+\t\t.refname = \"refs/heads/master\",\n -+\t\t.name = \"Han-Wen Nienhuys\",\n -+\t\t.email = \"hanwen@google.com\",\n -+\t\t.tz_offset = 100,\n -+\t\t.time = 0x5e430672,\n -+\t\t.update_index = 0xa,\n -+\t\t.message = \"commit: 9\\n\",\n -+\t};\n ++\tstruct reftable_log_record log = { .refname = \"refs/heads/master\",\n ++\t\t\t\t\t .update_index = 0xa,\n ++\t\t\t\t\t .value_type = REFTABLE_LOG_UPDATE,\n ++\t\t\t\t\t .update = {\n ++\t\t\t\t\t\t .name = \"Han-Wen Nienhuys\",\n ++\t\t\t\t\t\t .email = \"hanwen@google.com\",\n ++\t\t\t\t\t\t .tz_offset = 100,\n ++\t\t\t\t\t\t .time = 0x5e430672,\n ++\t\t\t\t\t\t .message = \"commit: 9\\n\",\n ++\t\t\t\t\t } };\n +\tstruct reftable_writer *w =\n +\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n +\n @@ reftable/reftable_test.c (new)\n +\t\thash1[i] = (uint8_t)(rand() % 256);\n +\t\thash2[i] = (uint8_t)(rand() % 256);\n +\t}\n -+\tlog.old_hash = hash1;\n -+\tlog.new_hash = hash2;\n ++\tlog.update.old_hash = hash1;\n ++\tlog.update.new_hash = hash2;\n +\treftable_writer_set_limits(w, update_index, update_index);\n +\terr = reftable_writer_add_log(w, &log);\n +\tEXPECT_ERR(err);\n @@ reftable/reftable_test.c (new)\n +\n +\t\tlog.refname = names[i];\n +\t\tlog.update_index = i;\n -+\t\tlog.old_hash = hash1;\n -+\t\tlog.new_hash = hash2;\n ++\t\tlog.value_type = REFTABLE_LOG_UPDATE;\n ++\t\tlog.update.old_hash = hash1;\n ++\t\tlog.update.new_hash = hash2;\n +\n +\t\terr = reftable_writer_add_log(w, &log);\n +\t\tEXPECT_ERR(err);\n 12: 557183d3e3e2 ! 12: 28aa69f7bbcb reftable: rest of library\n @@ Makefile: REFTABLE_OBJS += reftable/error.o\n \n ## reftable/VERSION (new) ##\n @@\n -+9b4a54059db9a05c270c0a0587f245bc6868d576 C: use rand() rather than cobbled together random generator.\n ++a337d48d4d42d513d6baa33addc172f0e0e36288 C: use tagged union for reftable_log_record\n \n ## reftable/dump.c (new) ##\n @@\n @@ reftable/stack.c (new)\n +\t\t\tcontinue;\n +\t\t}\n +\n -+\t\tif (config != NULL && config->time > 0 &&\n -+\t\t log.time < config->time) {\n ++\t\tif (config != NULL && config->min_update_index > 0 &&\n ++\t\t log.update_index < config->min_update_index) {\n +\t\t\tcontinue;\n +\t\t}\n +\n -+\t\tif (config != NULL && config->min_update_index > 0 &&\n -+\t\t log.update_index < config->min_update_index) {\n ++\t\tif (config != NULL && config->time > 0 &&\n ++\t\t log.update.time < config->time) {\n +\t\t\tcontinue;\n +\t\t}\n +\n @@ reftable/stack_test.c (new)\n +\n +\t\tlogs[i].refname = xstrdup(buf);\n +\t\tlogs[i].update_index = N + i + 1;\n -+\t\tlogs[i].new_hash = reftable_malloc(SHA1_SIZE);\n -+\t\tlogs[i].email = xstrdup(\"identity@invalid\");\n -+\t\tset_test_hash(logs[i].new_hash, i);\n ++\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n ++\n ++\t\tlogs[i].update.new_hash = reftable_malloc(SHA1_SIZE);\n ++\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n ++\t\tset_test_hash(logs[i].update.new_hash, i);\n +\t}\n +\n +\tfor (i = 0; i < N; i++) {\n @@ reftable/stack_test.c (new)\n +\n +\tuint8_t h1[SHA1_SIZE] = { 0x01 }, h2[SHA1_SIZE] = { 0x02 };\n +\n -+\tstruct reftable_log_record input = {\n -+\t\t.refname = \"branch\",\n -+\t\t.update_index = 1,\n -+\t\t.new_hash = h1,\n -+\t\t.old_hash = h2,\n -+\t};\n ++\tstruct reftable_log_record input = { .refname = \"branch\",\n ++\t\t\t\t\t .update_index = 1,\n ++\t\t\t\t\t .value_type = REFTABLE_LOG_UPDATE,\n ++\t\t\t\t\t .update = {\n ++\t\t\t\t\t\t .new_hash = h1,\n ++\t\t\t\t\t\t .old_hash = h2,\n ++\t\t\t\t\t } };\n +\tstruct reftable_log_record dest = {\n +\t\t.update_index = 0,\n +\t};\n @@ reftable/stack_test.c (new)\n +\terr = reftable_new_stack(&st, dir, cfg);\n +\tEXPECT_ERR(err);\n +\n -+\tinput.message = \"one\\ntwo\";\n ++\tinput.update.message = \"one\\ntwo\";\n +\terr = reftable_stack_add(st, &write_test_log, &arg);\n +\tEXPECT(err == REFTABLE_API_ERROR);\n +\n -+\tinput.message = \"one\";\n ++\tinput.update.message = \"one\";\n +\terr = reftable_stack_add(st, &write_test_log, &arg);\n +\tEXPECT_ERR(err);\n +\n +\terr = reftable_stack_read_log(st, input.refname, &dest);\n +\tEXPECT_ERR(err);\n -+\tEXPECT(0 == strcmp(dest.message, \"one\\n\"));\n ++\tEXPECT(0 == strcmp(dest.update.message, \"one\\n\"));\n +\n -+\tinput.message = \"two\\n\";\n ++\tinput.update.message = \"two\\n\";\n +\targ.update_index = 2;\n +\terr = reftable_stack_add(st, &write_test_log, &arg);\n +\tEXPECT_ERR(err);\n +\terr = reftable_stack_read_log(st, input.refname, &dest);\n +\tEXPECT_ERR(err);\n -+\tEXPECT(0 == strcmp(dest.message, \"two\\n\"));\n ++\tEXPECT(0 == strcmp(dest.update.message, \"two\\n\"));\n +\n +\t/* cleanup */\n +\treftable_stack_destroy(st);\n @@ reftable/stack_test.c (new)\n +\t\t\trefs[i].value.val1 = reftable_malloc(SHA1_SIZE);\n +\t\t\tset_test_hash(refs[i].value.val1, i);\n +\t\t}\n ++\n +\t\tlogs[i].refname = xstrdup(buf);\n +\t\t/* update_index is part of the key. */\n +\t\tlogs[i].update_index = 42;\n +\t\tif (i % 2 == 0) {\n -+\t\t\tlogs[i].new_hash = reftable_malloc(SHA1_SIZE);\n -+\t\t\tset_test_hash(logs[i].new_hash, i);\n -+\t\t\tlogs[i].email = xstrdup(\"identity@invalid\");\n ++\t\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n ++\t\t\tlogs[i].update.new_hash = reftable_malloc(SHA1_SIZE);\n ++\t\t\tset_test_hash(logs[i].update.new_hash, i);\n ++\t\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n +\t\t}\n +\t}\n +\tfor (i = 0; i < N; i++) {\n @@ reftable/stack_test.c (new)\n +\n +\t\tlogs[i].refname = xstrdup(buf);\n +\t\tlogs[i].update_index = i;\n -+\t\tlogs[i].time = i;\n -+\t\tlogs[i].new_hash = reftable_malloc(SHA1_SIZE);\n -+\t\tlogs[i].email = xstrdup(\"identity@invalid\");\n -+\t\tset_test_hash(logs[i].new_hash, i);\n ++\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n ++\t\tlogs[i].update.time = i;\n ++\t\tlogs[i].update.new_hash = reftable_malloc(SHA1_SIZE);\n ++\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n ++\t\tset_test_hash(logs[i].update.new_hash, i);\n +\t}\n +\n +\tfor (i = 1; i <= N; i++) {\n 13: d57023d9f13d ! 13: bdb19af22cc7 Reftable support for git-core\n @@ Commit message\n Signed-off-by: Han-Wen Nienhuys \n Signed-off-by: Johannes Schindelin \n Helped-by: Johannes Schindelin \n + Helped-by: Junio Hamano \n Co-authored-by: Jeff King \n \n ## Documentation/config/extensions.txt ##\n @@ builtin/init-db.c: int cmd_init_db(int argc, const char **argv, const char *pref\n \n ## builtin/worktree.c ##\n @@\n - #include \"submodule.h\"\n #include \"utf8.h\"\n #include \"worktree.h\"\n + #include \"quote.h\"\n +#include \"../refs/refs-internal.h\"\n \n static const char * const worktree_usage[] = {\n @@ refs/reftable-backend.c (new)\n +\n +static void clear_reftable_log_record(struct reftable_log_record *log)\n +{\n -+\tlog->old_hash = NULL;\n -+\tlog->new_hash = NULL;\n -+\tlog->message = NULL;\n +\tlog->refname = NULL;\n ++\tswitch (log->value_type) {\n ++\tcase REFTABLE_LOG_UPDATE:\n ++\t\tlog->update.old_hash = NULL;\n ++\t\tlog->update.new_hash = NULL;\n ++\t\tlog->update.message = NULL;\n ++\t\tbreak;\n ++\tcase REFTABLE_LOG_DELETION:\n ++\t\tbreak;\n ++\t}\n +\treftable_log_record_release(log);\n +}\n +\n @@ refs/reftable-backend.c (new)\n +\tassert(0 == result);\n +\n +\treftable_log_record_release(log);\n -+\tlog->name =\n ++\tlog->value_type = REFTABLE_LOG_UPDATE;\n ++\tlog->update.name =\n +\t\txstrndup(split.name_begin, split.name_end - split.name_begin);\n -+\tlog->email =\n ++\tlog->update.email =\n +\t\txstrndup(split.mail_begin, split.mail_end - split.mail_begin);\n -+\tlog->time = atol(split.date_begin);\n ++\tlog->update.time = atol(split.date_begin);\n +\tif (*split.tz_begin == '-') {\n +\t\tsign = -1;\n +\t\tsplit.tz_begin++;\n @@ refs/reftable-backend.c (new)\n +\t\tsplit.tz_begin++;\n +\t}\n +\n -+\tlog->tz_offset = sign * atoi(split.tz_begin);\n ++\tlog->update.tz_offset = sign * atoi(split.tz_begin);\n +}\n +\n +static int has_suffix(struct strbuf *b, const char *suffix)\n @@ refs/reftable-backend.c (new)\n +\t\tstruct ref_update *u = sorted[i];\n +\t\tstruct reftable_log_record *log = &logs[i];\n +\t\tfill_reftable_log_record(log);\n -+\t\tlog->refname = (char *)u->refname;\n -+\t\tlog->old_hash = u->old_oid.hash;\n -+\t\tlog->new_hash = u->new_oid.hash;\n +\t\tlog->update_index = ts;\n -+\t\tlog->message = u->msg;\n ++\t\tlog->value_type = REFTABLE_LOG_UPDATE;\n ++\t\tlog->refname = (char *)u->refname;\n ++\t\tlog->update.old_hash = u->old_oid.hash;\n ++\t\tlog->update.new_hash = u->new_oid.hash;\n ++\t\tlog->update.message = u->msg;\n +\n +\t\tif (u->flags & REF_LOG_ONLY) {\n +\t\t\tcontinue;\n @@ refs/reftable-backend.c (new)\n +\t\t};\n +\t\tstruct reftable_ref_record current = { NULL };\n +\t\tfill_reftable_log_record(&log);\n -+\t\tlog.message = xstrdup(arg->logmsg);\n -+\t\tlog.new_hash = NULL;\n -+\t\tlog.old_hash = NULL;\n +\t\tlog.update_index = ts;\n +\t\tlog.refname = (char *)arg->refnames->items[i].string;\n +\n ++\t\tlog.update.message = xstrdup(arg->logmsg);\n ++\t\tlog.update.new_hash = NULL;\n ++\t\tlog.update.old_hash = NULL;\n +\t\tif (reftable_stack_read_ref(arg->stack, log.refname,\n +\t\t\t\t\t ¤t) == 0) {\n -+\t\t\tlog.old_hash = reftable_ref_record_val1(¤t);\n ++\t\t\tlog.update.old_hash =\n ++\t\t\t\treftable_ref_record_val1(¤t);\n +\t\t}\n +\t\terr = reftable_writer_add_log(writer, &log);\n -+\t\tlog.old_hash = NULL;\n ++\t\tlog.update.old_hash = NULL;\n +\t\treftable_ref_record_release(¤t);\n +\n +\t\tclear_reftable_log_record(&log);\n @@ refs/reftable-backend.c (new)\n +\n +\t\tfill_reftable_log_record(&log);\n +\t\tlog.refname = (char *)create->refname;\n -+\t\tlog.message = (char *)create->logmsg;\n +\t\tlog.update_index = ts;\n ++\t\tlog.update.message = (char *)create->logmsg;\n +\t\tif (refs_resolve_ref_unsafe(\n +\t\t\t (struct ref_store *)create->refs, create->refname,\n +\t\t\t RESOLVE_REF_READING, &old_oid, NULL) != NULL) {\n -+\t\t\tlog.old_hash = old_oid.hash;\n ++\t\t\tlog.update.old_hash = old_oid.hash;\n +\t\t}\n +\n +\t\tif (refs_resolve_ref_unsafe((struct ref_store *)create->refs,\n +\t\t\t\t\t create->target, RESOLVE_REF_READING,\n +\t\t\t\t\t &new_oid, NULL) != NULL) {\n -+\t\t\tlog.new_hash = new_oid.hash;\n ++\t\t\tlog.update.new_hash = new_oid.hash;\n +\t\t}\n +\n -+\t\tif (log.old_hash != NULL || log.new_hash != NULL) {\n ++\t\tif (log.update.old_hash != NULL ||\n ++\t\t log.update.new_hash != NULL) {\n +\t\t\terr = reftable_writer_add_log(writer, &log);\n +\t\t}\n +\t\tlog.refname = NULL;\n -+\t\tlog.message = NULL;\n -+\t\tlog.old_hash = NULL;\n -+\t\tlog.new_hash = NULL;\n ++\t\tlog.update.message = NULL;\n ++\t\tlog.update.old_hash = NULL;\n ++\t\tlog.update.new_hash = NULL;\n +\t\tclear_reftable_log_record(&log);\n +\t}\n +\treturn err;\n @@ refs/reftable-backend.c (new)\n +\n +\t\ttodo[0].refname = (char *)arg->oldname;\n +\t\ttodo[0].update_index = ts;\n -+\t\ttodo[0].message = (char *)arg->logmsg;\n -+\t\ttodo[0].old_hash = val1;\n -+\t\ttodo[0].new_hash = NULL;\n ++\t\ttodo[0].update.message = (char *)arg->logmsg;\n ++\t\ttodo[0].update.old_hash = val1;\n ++\t\ttodo[0].update.new_hash = NULL;\n +\n +\t\ttodo[1].refname = (char *)arg->newname;\n +\t\ttodo[1].update_index = ts;\n -+\t\ttodo[1].old_hash = NULL;\n -+\t\ttodo[1].new_hash = val1;\n -+\t\ttodo[1].message = (char *)arg->logmsg;\n ++\t\ttodo[1].update.old_hash = NULL;\n ++\t\ttodo[1].update.new_hash = val1;\n ++\t\ttodo[1].update.message = (char *)arg->logmsg;\n +\n +\t\terr = reftable_writer_add_logs(writer, todo, 2);\n +\n @@ refs/reftable-backend.c (new)\n +\n +\t\tfree(ri->last_name);\n +\t\tri->last_name = xstrdup(ri->log.refname);\n -+\t\thashcpy(ri->oid.hash, ri->log.new_hash);\n ++\t\thashcpy(ri->oid.hash, ri->log.update.new_hash);\n +\t\treturn ITER_OK;\n +\t}\n +}\n @@ refs/reftable-backend.c (new)\n +static struct ref_iterator *\n +git_reftable_reflog_iterator_begin(struct ref_store *ref_store)\n +{\n -+\tstruct git_reftable_reflog_ref_iterator *ri = xcalloc(sizeof(*ri), 1);\n ++\tstruct git_reftable_reflog_ref_iterator *ri = xcalloc(1, sizeof(*ri));\n +\tstruct git_reftable_ref_store *refs =\n +\t\t(struct git_reftable_ref_store *)ref_store;\n +\n @@ refs/reftable-backend.c (new)\n +\t\t\tbreak;\n +\t\t}\n +\n -+\t\thashcpy(old_oid.hash, log.old_hash);\n -+\t\thashcpy(new_oid.hash, log.new_hash);\n ++\t\thashcpy(old_oid.hash, log.update.old_hash);\n ++\t\thashcpy(new_oid.hash, log.update.new_hash);\n +\n -+\t\tfull_committer = fmt_ident(log.name, log.email,\n ++\t\tfull_committer = fmt_ident(log.update.name, log.update.email,\n +\t\t\t\t\t WANT_COMMITTER_IDENT,\n +\t\t\t\t\t /*date*/ NULL, IDENT_NO_DATE);\n -+\t\terr = fn(&old_oid, &new_oid, full_committer, log.time,\n -+\t\t\t log.tz_offset, log.message, cb_data);\n ++\t\terr = fn(&old_oid, &new_oid, full_committer, log.update.time,\n ++\t\t\t log.update.tz_offset, log.update.message, cb_data);\n +\t\tif (err)\n +\t\t\tbreak;\n +\t}\n @@ refs/reftable-backend.c (new)\n +\t\tstruct object_id new_oid;\n +\t\tconst char *full_committer = \"\";\n +\n -+\t\thashcpy(old_oid.hash, log->old_hash);\n -+\t\thashcpy(new_oid.hash, log->new_hash);\n ++\t\thashcpy(old_oid.hash, log->update.old_hash);\n ++\t\thashcpy(new_oid.hash, log->update.new_hash);\n +\n -+\t\tfull_committer = fmt_ident(log->name, log->email,\n ++\t\tfull_committer = fmt_ident(log->update.name, log->update.email,\n +\t\t\t\t\t WANT_COMMITTER_IDENT, NULL,\n +\t\t\t\t\t IDENT_NO_DATE);\n -+\t\terr = fn(&old_oid, &new_oid, full_committer, log->time,\n -+\t\t\t log->tz_offset, log->message, cb_data);\n ++\t\terr = fn(&old_oid, &new_oid, full_committer, log->update.time,\n ++\t\t\t log->update.tz_offset, log->update.message, cb_data);\n +\t\tif (err) {\n +\t\t\tbreak;\n +\t\t}\n @@ refs/reftable-backend.c (new)\n +\t\tif (err > 0 || strcmp(log.refname, refname)) {\n +\t\t\tbreak;\n +\t\t}\n -+\t\thashcpy(ooid.hash, log.old_hash);\n -+\t\thashcpy(noid.hash, log.new_hash);\n ++\t\thashcpy(ooid.hash, log.update.old_hash);\n ++\t\thashcpy(noid.hash, log.update.new_hash);\n +\n -+\t\tif (should_prune_fn(&ooid, &noid, log.email,\n -+\t\t\t\t (timestamp_t)log.time, log.tz_offset,\n -+\t\t\t\t log.message, policy_cb_data)) {\n ++\t\tif (should_prune_fn(&ooid, &noid, log.update.email,\n ++\t\t\t\t (timestamp_t)log.update.time,\n ++\t\t\t\t log.update.tz_offset, log.update.message,\n ++\t\t\t\t policy_cb_data)) {\n +\t\t\tadd_log_tombstone(&arg, refname, log.update_index);\n +\t\t}\n +\t}\n @@ t/t0031-reftable.sh (new)\n +\n +INVALID_SHA1=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n +\n ++git_init () {\n ++\tgit init -b primary \"$@\"\n ++}\n ++\n +initialize () {\n +\trm -rf .git &&\n -+\tgit init --ref-storage=reftable &&\n ++\tgit_init --ref-storage=reftable &&\n +\tmv .git/hooks .git/hooks-disabled\n +}\n +\n +test_expect_success 'SHA256 support, env' '\n +\trm -rf .git &&\n +\tGIT_DEFAULT_HASH=sha256 && export GIT_DEFAULT_HASH &&\n -+\tgit init --ref-storage=reftable &&\n ++\tgit_init --ref-storage=reftable &&\n +\tmv .git/hooks .git/hooks-disabled &&\n +\ttest_commit file\n +'\n +\n +test_expect_success 'SHA256 support, option' '\n +\trm -rf .git &&\n -+\tgit init --ref-storage=reftable --object-format=sha256 &&\n ++\tgit_init --ref-storage=reftable --object-format=sha256 &&\n +\tmv .git/hooks .git/hooks-disabled &&\n +\ttest_commit file\n +'\n @@ t/t0031-reftable.sh (new)\n +\tinitialize &&\n +\ttest_commit file &&\n +\tSHA=$(git show-ref -s --verify HEAD) &&\n -+\ttest_write_lines \"$SHA refs/heads/master\" \"$SHA refs/tags/file\" >expect &&\n -+\tgit show-ref > actual &&\n ++\ttest_write_lines \"$SHA refs/heads/primary\" \"$SHA refs/tags/file\" >expect &&\n ++\tgit show-ref >actual &&\n +\t! git update-ref -d refs/tags/file $INVALID_SHA1 &&\n +\ttest_cmp expect actual &&\n +\tgit update-ref -d refs/tags/file $SHA &&\n -+\ttest_write_lines \"$SHA refs/heads/master\" >expect &&\n -+\tgit show-ref > actual &&\n ++\ttest_write_lines \"$SHA refs/heads/primary\" >expect &&\n ++\tgit show-ref >actual &&\n +\ttest_cmp expect actual\n +'\n +\n @@ t/t0031-reftable.sh (new)\n +test_expect_success 'basic operation of reftable storage: commit, show-ref' '\n +\tinitialize &&\n +\ttest_commit file &&\n -+\ttest_write_lines refs/heads/master refs/tags/file >expect &&\n ++\ttest_write_lines refs/heads/primary refs/tags/file >expect &&\n +\tgit show-ref &&\n -+\tgit show-ref | cut -f2 -d\" \" > actual &&\n ++\tgit show-ref | cut -f2 -d\" \" >actual &&\n +\ttest_cmp actual expect\n +'\n +\n @@ t/t0031-reftable.sh (new)\n +\tgit pack-refs &&\n +\tls -1 .git/reftable >table-files &&\n +\ttest_line_count = 2 table-files &&\n -+\tgit reflog refs/heads/master >output &&\n ++\tgit reflog refs/heads/primary >output &&\n +\ttest_line_count = 10 output &&\n +\tgrep \"commit (initial): number 1\" output &&\n +\tgrep \"commit: number 10\" output &&\n +\tgit gc &&\n -+\tgit reflog refs/heads/master >output &&\n ++\tgit reflog refs/heads/primary >output &&\n +\ttest_line_count = 0 output\n +'\n +\n @@ t/t0031-reftable.sh (new)\n +\ttest_commit file2 &&\n +\tgit checkout -b branch2 &&\n +\tgit switch - &&\n -+\tgit rev-parse --symbolic-full-name HEAD > actual &&\n -+\techo refs/heads/branch1 > expect &&\n ++\tgit rev-parse --symbolic-full-name HEAD >actual &&\n ++\techo refs/heads/branch1 >expect &&\n +\ttest_cmp actual expect\n +'\n +\n @@ t/t0031-reftable.sh (new)\n +\ttest_commit file &&\n +\tgit tag -m \"annotated tag\" test_tag HEAD &&\n +\t{\n -+\t\tprint_ref \"refs/heads/master\" &&\n ++\t\tprint_ref \"refs/heads/primary\" &&\n +\t\tprint_ref \"refs/tags/file\" &&\n +\t\tprint_ref \"refs/tags/test_tag\" &&\n +\t\tprint_ref \"refs/tags/test_tag^{}\"\n @@ t/t0031-reftable.sh (new)\n +test_expect_success 'show-ref works on fresh repo' '\n +\tinitialize &&\n +\trm -rf .git &&\n -+\tgit init --ref-storage=reftable &&\n ++\tgit_init --ref-storage=reftable &&\n +\t>expect &&\n -+\t! git show-ref > actual &&\n ++\t! git show-ref >actual &&\n +\ttest_cmp expect actual\n +'\n +\n +test_expect_success 'checkout unborn branch' '\n +\tinitialize &&\n -+\tgit checkout -b master\n ++\tgit checkout -b primary\n +'\n +\n +\n +test_expect_success 'dir/file conflict' '\n +\tinitialize &&\n +\ttest_commit file &&\n -+\t! git branch master/forbidden\n ++\t! git branch primary/forbidden\n +'\n +\n +\n +test_expect_success 'do not clobber existing repo' '\n +\trm -rf .git &&\n -+\tgit init --ref-storage=files &&\n -+\tcat .git/HEAD > expect &&\n ++\tgit_init --ref-storage=files &&\n ++\tcat .git/HEAD >expect &&\n +\ttest_commit file &&\n -+\t(git init --ref-storage=reftable || true) &&\n -+\tcat .git/HEAD > actual &&\n ++\t(git_init --ref-storage=reftable || true) &&\n ++\tcat .git/HEAD >actual &&\n +\ttest_cmp expect actual\n +'\n +\n @@ t/t0031-reftable.sh (new)\n +'\n +\n +test_expect_success 'worktrees' '\n -+\tgit init --ref-storage=reftable start &&\n ++\tgit_init --ref-storage=reftable start &&\n +\t(cd start && test_commit file1 && git checkout -b branch1 &&\n +\tgit checkout -b branch2 &&\n +\tgit worktree add ../wt\n @@ t/t0031-reftable.sh (new)\n +\tinitialize &&\n +\ttest_commit file1 &&\n +\tmkdir existing_empty &&\n -+\tgit worktree add --detach existing_empty master\n ++\tgit worktree add --detach existing_empty primary\n +'\n +\n +test_expect_success 'FETCH_HEAD' '\n +\tinitialize &&\n +\ttest_commit one &&\n -+\t(git init sub && cd sub && test_commit two) &&\n ++\t(git_init sub && cd sub && test_commit two) &&\n +\tgit --git-dir sub/.git rev-parse HEAD >expect &&\n +\tgit fetch sub &&\n +\tgit checkout FETCH_HEAD &&\n @@ t/t1450-fsck.sh: test_description='git fsck random collection of tests\n \tgit config i18n.commitencoding ISO-8859-1 &&\n \n ## t/t3210-pack-refs.sh ##\n -@@ t/t3210-pack-refs.sh: semantic is still the same.\n - '\n +@@ t/t3210-pack-refs.sh: export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n + \n . ./test-lib.sh\n \n +if test_have_prereq REFTABLE\n 14: 9df5bc69f971 = 14: 0d9477941678 git-prompt: prepare for reftable refs backend\n 15: 4076ef5d20b4 = 15: 59f432522230 Add \"test-tool dump-reftable\" command.\n\n-- \ngitgitgadget\n"},{"id":"418987","messageId":"5bbbafb428e9c3f6e95a880abbc1be596cf021fe.1615580397.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v5.git.git.1615580397.gitgitgadget@gmail.com","subject":"[PATCH v5 04/15] reftable: utility functions","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-12T20:19:45Z","receivedAt":"2021-03-12T20:20:45Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis commit provides basic utility classes for the reftable library.\n\nSigned-off-by: Han-Wen Nienhuys \nHelped-by: Johannes Schindelin \n---\n Makefile | 24 +++++-\n contrib/buildsystems/CMakeLists.txt | 14 ++-\n reftable/basics.c | 128 ++++++++++++++++++++++++++++\n reftable/basics.h | 60 +++++++++++++\n reftable/basics_test.c | 98 +++++++++++++++++++++\n reftable/publicbasics.c | 58 +++++++++++++\n reftable/reftable-malloc.h | 18 ++++\n reftable/reftable-tests.h | 22 +++++\n reftable/system.h | 32 +++++++\n reftable/test_framework.c | 23 +++++\n reftable/test_framework.h | 53 ++++++++++++\n t/helper/test-reftable.c | 9 ++\n t/helper/test-tool.c | 3 +-\n t/helper/test-tool.h | 1 +\n t/t0032-reftable-unittest.sh | 15 ++++\n 15 files changed, 552 insertions(+), 6 deletions(-)\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/basics_test.c\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-malloc.h\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 t/helper/test-reftable.c\n create mode 100755 t/t0032-reftable-unittest.sh\n\ndiff --git a/Makefile b/Makefile\nindex dfb0f1000fa3..43ed67fdb9db 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -728,6 +728,7 @@ TEST_BUILTINS_OBJS += test-read-cache.o\n TEST_BUILTINS_OBJS += test-read-graph.o\n TEST_BUILTINS_OBJS += test-read-midx.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n+TEST_BUILTINS_OBJS += test-reftable.o\n TEST_BUILTINS_OBJS += test-regex.o\n TEST_BUILTINS_OBJS += test-repository.o\n TEST_BUILTINS_OBJS += test-revision-walking.o\n@@ -804,6 +805,8 @@ TEST_SHELL_PATH = $(SHELL_PATH)\n \n LIB_FILE = libgit.a\n XDIFF_LIB = xdiff/lib.a\n+REFTABLE_LIB = reftable/libreftable.a\n+REFTABLE_TEST_LIB = reftable/libreftable_test.a\n \n GENERATED_H += command-list.h\n GENERATED_H += config-list.h\n@@ -1172,7 +1175,7 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n-GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB)\n+GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB)\n EXTLIBS =\n \n GIT_USER_AGENT = git/$(GIT_VERSION)\n@@ -2379,10 +2382,19 @@ XDIFF_OBJS += xdiff/xpatience.o\n XDIFF_OBJS += xdiff/xprepare.o\n XDIFF_OBJS += xdiff/xutils.o\n \n+REFTABLE_OBJS += reftable/basics.o\n+REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/publicbasics.o\n+\n+REFTABLE_TEST_OBJS += reftable/test_framework.o\n+REFTABLE_TEST_OBJS += reftable/basics_test.o\n+\n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \\\n \t$(XDIFF_OBJS) \\\n \t$(FUZZ_OBJS) \\\n+\t$(REFTABLE_OBJS) \\\n+\t$(REFTABLE_TEST_OBJS) \\\n \tcommon-main.o \\\n \tgit.o\n ifndef NO_CURL\n@@ -2534,6 +2546,12 @@ $(LIB_FILE): $(LIB_OBJS)\n $(XDIFF_LIB): $(XDIFF_OBJS)\n \t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n \n+$(REFTABLE_LIB): $(REFTABLE_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n+$(REFTABLE_TEST_LIB): $(REFTABLE_TEST_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n export DEFAULT_EDITOR DEFAULT_PAGER\n \n Documentation/GIT-EXCLUDED-PROGRAMS: FORCE\n@@ -2812,7 +2830,7 @@ perf: all\n \n t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n \n-t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS)\n+t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) $(REFTABLE_TEST_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n \n check-sha1:: t/helper/test-tool$X\n@@ -3142,7 +3160,7 @@ cocciclean:\n clean: profile-clean coverage-clean cocciclean\n \t$(RM) *.res\n \t$(RM) $(OBJECTS)\n-\t$(RM) $(LIB_FILE) $(XDIFF_LIB)\n+\t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) git$X\n \t$(RM) $(TEST_PROGRAMS)\n \t$(RM) $(FUZZ_PROGRAMS)\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex ac3dbc079af8..986bfc04e0dc 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -601,6 +601,12 @@ parse_makefile_for_sources(libxdiff_SOURCES \"XDIFF_OBJS\")\n list(TRANSFORM libxdiff_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n add_library(xdiff STATIC ${libxdiff_SOURCES})\n \n+#reftable\n+parse_makefile_for_sources(reftable_SOURCES \"REFTABLE_OBJS\")\n+\n+list(TRANSFORM reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n+add_library(reftable STATIC ${reftable_SOURCES})\n+\n if(WIN32)\n \tif(NOT MSVC)#use windres when compiling with gcc and clang\n \t\tadd_custom_command(OUTPUT ${CMAKE_BINARY_DIR}/git.res\n@@ -623,7 +629,7 @@ endif()\n #link all required libraries to common-main\n add_library(common-main OBJECT ${CMAKE_SOURCE_DIR}/common-main.c)\n \n-target_link_libraries(common-main libgit xdiff ${ZLIB_LIBRARIES})\n+target_link_libraries(common-main libgit xdiff reftable ${ZLIB_LIBRARIES})\n if(Intl_FOUND)\n \ttarget_link_libraries(common-main ${Intl_LIBRARIES})\n endif()\n@@ -855,11 +861,15 @@ if(BUILD_TESTING)\n add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)\n target_link_libraries(test-fake-ssh common-main)\n \n+#reftable-tests\n+parse_makefile_for_sources(test-reftable_SOURCES \"REFTABLE_TEST_OBJS\")\n+list(TRANSFORM test-reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n+\n #test-tool\n parse_makefile_for_sources(test-tool_SOURCES \"TEST_BUILTINS_OBJS\")\n \n list(TRANSFORM test-tool_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/t/helper/\")\n-add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES})\n+add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES} ${test-reftable_SOURCES})\n target_link_libraries(test-tool common-main)\n \n set_target_properties(test-fake-ssh test-tool\ndiff --git a/reftable/basics.c b/reftable/basics.c\nnew file mode 100644\nindex 000000000000..abd027b98882\n--- /dev/null\n+++ b/reftable/basics.c\n@@ -0,0 +1,128 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+\n+void put_be24(uint8_t *out, uint32_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 16) & 0xff);\n+\tout[1] = (uint8_t)((i >> 8) & 0xff);\n+\tout[2] = (uint8_t)(i & 0xff);\n+}\n+\n+uint32_t get_be24(uint8_t *in)\n+{\n+\treturn (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 |\n+\t (uint32_t)(in[2]);\n+}\n+\n+void put_be16(uint8_t *out, uint16_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 8) & 0xff);\n+\tout[1] = (uint8_t)(i & 0xff);\n+}\n+\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n+{\n+\tsize_t lo = 0;\n+\tsize_t hi = sz;\n+\n+\t/* Invariants:\n+\t *\n+\t * (hi == sz) || f(hi) == true\n+\t * (lo == 0 && f(0) == true) || fi(lo) == false\n+\t */\n+\twhile (hi - lo > 1) {\n+\t\tsize_t mid = lo + (hi - lo) / 2;\n+\n+\t\tif (f(mid, args))\n+\t\t\thi = mid;\n+\t\telse\n+\t\t\tlo = mid;\n+\t}\n+\n+\tif (lo)\n+\t\treturn hi;\n+\n+\treturn f(0, args) ? 0 : 1;\n+}\n+\n+void free_names(char **a)\n+{\n+\tchar **p;\n+\tif (a == NULL) {\n+\t\treturn;\n+\t}\n+\tfor (p = a; *p; p++) {\n+\t\treftable_free(*p);\n+\t}\n+\treftable_free(a);\n+}\n+\n+int names_length(char **names)\n+{\n+\tchar **p = names;\n+\tfor (; *p; p++) {\n+\t\t/* empty */\n+\t}\n+\treturn p - names;\n+}\n+\n+void parse_names(char *buf, int size, char ***namesp)\n+{\n+\tchar **names = NULL;\n+\tsize_t names_cap = 0;\n+\tsize_t names_len = 0;\n+\n+\tchar *p = buf;\n+\tchar *end = buf + size;\n+\twhile (p < end) {\n+\t\tchar *next = strchr(p, '\\n');\n+\t\tif (next && next < end) {\n+\t\t\t*next = 0;\n+\t\t} else {\n+\t\t\tnext = end;\n+\t\t}\n+\t\tif (p < next) {\n+\t\t\tif (names_len == names_cap) {\n+\t\t\t\tnames_cap = 2 * names_cap + 1;\n+\t\t\t\tnames = reftable_realloc(\n+\t\t\t\t\tnames, names_cap * sizeof(*names));\n+\t\t\t}\n+\t\t\tnames[names_len++] = xstrdup(p);\n+\t\t}\n+\t\tp = next + 1;\n+\t}\n+\n+\tnames = reftable_realloc(names, (names_len + 1) * sizeof(*names));\n+\tnames[names_len] = NULL;\n+\t*namesp = names;\n+}\n+\n+int names_equal(char **a, char **b)\n+{\n+\tint i = 0;\n+\tfor (; a[i] && b[i]; i++) {\n+\t\tif (strcmp(a[i], b[i])) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treturn a[i] == b[i];\n+}\n+\n+int common_prefix_size(struct strbuf *a, struct strbuf *b)\n+{\n+\tint p = 0;\n+\tfor (; p < a->len && p < b->len; p++) {\n+\t\tif (a->buf[p] != b->buf[p])\n+\t\t\tbreak;\n+\t}\n+\n+\treturn p;\n+}\ndiff --git a/reftable/basics.h b/reftable/basics.h\nnew file mode 100644\nindex 000000000000..096b36862b9f\n--- /dev/null\n+++ b/reftable/basics.h\n@@ -0,0 +1,60 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BASICS_H\n+#define BASICS_H\n+\n+/*\n+ * miscellaneous utilities that are not provided by Git.\n+ */\n+\n+#include \"system.h\"\n+\n+/* Bigendian en/decoding of integers */\n+\n+void put_be24(uint8_t *out, uint32_t i);\n+uint32_t get_be24(uint8_t *in);\n+void put_be16(uint8_t *out, uint16_t i);\n+\n+/*\n+ * find smallest index i in [0, sz) at which f(i) is true, assuming\n+ * that f is ascending. Return sz if f(i) is false for all indices.\n+ *\n+ * Contrary to bsearch(3), this returns something useful if the argument is not\n+ * found.\n+ */\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args);\n+\n+/*\n+ * Frees a NULL terminated array of malloced strings. The array itself is also\n+ * freed.\n+ */\n+void free_names(char **a);\n+\n+/* parse a newline separated list of names. `size` is the length of the buffer,\n+ * without terminating '\\0'. Empty names are discarded. */\n+void parse_names(char *buf, int size, char ***namesp);\n+\n+/* compares two NULL-terminated arrays of strings. */\n+int names_equal(char **a, char **b);\n+\n+/* returns the array size of a NULL-terminated array of strings. */\n+int names_length(char **names);\n+\n+/* Allocation routines; they invoke the functions set through\n+ * reftable_set_alloc() */\n+void *reftable_malloc(size_t sz);\n+void *reftable_realloc(void *p, size_t sz);\n+void reftable_free(void *p);\n+void *reftable_calloc(size_t sz);\n+\n+/* Find the longest shared prefix size of `a` and `b` */\n+struct strbuf;\n+int common_prefix_size(struct strbuf *a, struct strbuf *b);\n+\n+#endif\ndiff --git a/reftable/basics_test.c b/reftable/basics_test.c\nnew file mode 100644\nindex 000000000000..6d52f0f9d5aa\n--- /dev/null\n+++ b/reftable/basics_test.c\n@@ -0,0 +1,98 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct binsearch_args {\n+\tint key;\n+\tint *arr;\n+};\n+\n+static int binsearch_func(size_t i, void *void_args)\n+{\n+\tstruct binsearch_args *args = (struct binsearch_args *)void_args;\n+\n+\treturn args->key < args->arr[i];\n+}\n+\n+static void test_binsearch(void)\n+{\n+\tint arr[] = { 2, 4, 6, 8, 10 };\n+\tsize_t sz = ARRAY_SIZE(arr);\n+\tstruct binsearch_args args = {\n+\t\t.arr = arr,\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 1; i < 11; i++) {\n+\t\tint res;\n+\t\targs.key = i;\n+\t\tres = binsearch(sz, &binsearch_func, &args);\n+\n+\t\tif (res < sz) {\n+\t\t\tEXPECT(args.key < arr[res]);\n+\t\t\tif (res > 0) {\n+\t\t\t\tEXPECT(args.key >= arr[res - 1]);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tEXPECT(args.key == 10 || args.key == 11);\n+\t\t}\n+\t}\n+}\n+\n+static void test_names_length(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", NULL };\n+\tEXPECT(names_length(a) == 2);\n+}\n+\n+static void test_parse_names_normal(void)\n+{\n+\tchar in[] = \"a\\nb\\n\";\n+\tchar **out = NULL;\n+\tparse_names(in, strlen(in), &out);\n+\tEXPECT(!strcmp(out[0], \"a\"));\n+\tEXPECT(!strcmp(out[1], \"b\"));\n+\tEXPECT(out[2] == NULL);\n+\tfree_names(out);\n+}\n+\n+static void test_parse_names_drop_empty(void)\n+{\n+\tchar in[] = \"a\\n\\n\";\n+\tchar **out = NULL;\n+\tparse_names(in, strlen(in), &out);\n+\tEXPECT(!strcmp(out[0], \"a\"));\n+\tEXPECT(out[1] == NULL);\n+\tfree_names(out);\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct strbuf s1 = STRBUF_INIT;\n+\tstruct strbuf s2 = STRBUF_INIT;\n+\tstrbuf_addstr(&s1, \"abcdef\");\n+\tstrbuf_addstr(&s2, \"abc\");\n+\tEXPECT(common_prefix_size(&s1, &s2) == 3);\n+\tstrbuf_release(&s1);\n+\tstrbuf_release(&s2);\n+}\n+\n+int basics_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_common_prefix);\n+\tRUN_TEST(test_parse_names_normal);\n+\tRUN_TEST(test_parse_names_drop_empty);\n+\tRUN_TEST(test_binsearch);\n+\tRUN_TEST(test_names_length);\n+\treturn 0;\n+}\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\nnew file mode 100644\nindex 000000000000..25639f61af61\n--- /dev/null\n+++ b/reftable/publicbasics.c\n@@ -0,0 +1,58 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-malloc.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+static void *(*reftable_malloc_ptr)(size_t sz) = &malloc;\n+static void *(*reftable_realloc_ptr)(void *, size_t) = &realloc;\n+static void (*reftable_free_ptr)(void *) = &free;\n+\n+void *reftable_malloc(size_t sz)\n+{\n+\treturn (*reftable_malloc_ptr)(sz);\n+}\n+\n+void *reftable_realloc(void *p, size_t sz)\n+{\n+\treturn (*reftable_realloc_ptr)(p, sz);\n+}\n+\n+void reftable_free(void *p)\n+{\n+\treftable_free_ptr(p);\n+}\n+\n+void *reftable_calloc(size_t sz)\n+{\n+\tvoid *p = reftable_malloc(sz);\n+\tmemset(p, 0, sz);\n+\treturn p;\n+}\n+\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n+{\n+\treftable_malloc_ptr = malloc;\n+\treftable_realloc_ptr = realloc;\n+\treftable_free_ptr = free;\n+}\n+\n+int hash_size(uint32_t id)\n+{\n+\tswitch (id) {\n+\tcase 0:\n+\tcase SHA1_ID:\n+\t\treturn SHA1_SIZE;\n+\tcase SHA256_ID:\n+\t\treturn SHA256_SIZE;\n+\t}\n+\tabort();\n+}\ndiff --git a/reftable/reftable-malloc.h b/reftable/reftable-malloc.h\nnew file mode 100644\nindex 000000000000..5f2185f1f343\n--- /dev/null\n+++ b/reftable/reftable-malloc.h\n@@ -0,0 +1,18 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_H\n+#define REFTABLE_H\n+\n+#include \n+\n+/* Overrides the functions to use for memory management. */\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n+\n+#endif\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nnew file mode 100644\nindex 000000000000..5e7698ae654e\n--- /dev/null\n+++ b/reftable/reftable-tests.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_TESTS_H\n+#define REFTABLE_TESTS_H\n+\n+int basics_test_main(int argc, const char **argv);\n+int block_test_main(int argc, const char **argv);\n+int merged_test_main(int argc, const char **argv);\n+int record_test_main(int argc, const char **argv);\n+int refname_test_main(int argc, const char **argv);\n+int reftable_test_main(int argc, const char **argv);\n+int stack_test_main(int argc, const char **argv);\n+int tree_test_main(int argc, const char **argv);\n+int reftable_dump_main(int argc, char *const *argv);\n+\n+#endif\ndiff --git a/reftable/system.h b/reftable/system.h\nnew file mode 100644\nindex 000000000000..07277ca06273\n--- /dev/null\n+++ b/reftable/system.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef SYSTEM_H\n+#define SYSTEM_H\n+\n+#include \"git-compat-util.h\"\n+#include \"strbuf.h\"\n+\n+#include \n+\n+struct strbuf;\n+/* In git, this is declared in dir.h */\n+int remove_dir_recursively(struct strbuf *path, int flags);\n+\n+#define SHA1_ID 0x73686131\n+#define SHA256_ID 0x73323536\n+#define SHA1_SIZE 20\n+#define SHA256_SIZE 32\n+\n+/* This is uncompress2, which is only available in zlib as of 2017.\n+ */\n+int uncompress_return_consumed(Bytef *dest, uLongf *destLen,\n+\t\t\t const Bytef *source, uLong *sourceLen);\n+int hash_size(uint32_t id);\n+\n+#endif\ndiff --git a/reftable/test_framework.c b/reftable/test_framework.c\nnew file mode 100644\nindex 000000000000..a5ff4e2a2d2f\n--- /dev/null\n+++ b/reftable/test_framework.c\n@@ -0,0 +1,23 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"test_framework.h\"\n+\n+#include \"basics.h\"\n+\n+void set_test_hash(uint8_t *p, int i)\n+{\n+\tmemset(p, (uint8_t)i, hash_size(SHA1_ID));\n+}\n+\n+int strbuf_add_void(void *b, const void *data, size_t sz)\n+{\n+\tstrbuf_add((struct strbuf *)b, data, sz);\n+\treturn sz;\n+}\ndiff --git a/reftable/test_framework.h b/reftable/test_framework.h\nnew file mode 100644\nindex 000000000000..5fdc9519a5a5\n--- /dev/null\n+++ b/reftable/test_framework.h\n@@ -0,0 +1,53 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TEST_FRAMEWORK_H\n+#define TEST_FRAMEWORK_H\n+\n+#include \"system.h\"\n+#include \"reftable-error.h\"\n+\n+#define EXPECT_ERR(c) \\\n+\tif (c != 0) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: error == %d (%s), want 0\\n\", \\\n+\t\t\t__FILE__, __LINE__, c, reftable_error_str(c)); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define EXPECT_STREQ(a, b) \\\n+\tif (strcmp(a, b)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s:%d: %s (%s) != %s (%s)\\n\", __FILE__, \\\n+\t\t\t__LINE__, #a, a, #b, b); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define EXPECT(c) \\\n+\tif (!(c)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: failed assertion %s\\n\", __FILE__, \\\n+\t\t\t__LINE__, #c); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define RUN_TEST(f) \\\n+\tfprintf(stderr, \"running %s\\n\", #f); \\\n+\tfflush(stderr); \\\n+\tf();\n+\n+void set_test_hash(uint8_t *p, int i);\n+\n+/* Like strbuf_add, but suitable for passing to reftable_new_writer\n+ */\n+int strbuf_add_void(void *b, const void *data, size_t sz);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nnew file mode 100644\nindex 000000000000..3b58e423e7b1\n--- /dev/null\n+++ b/t/helper/test-reftable.c\n@@ -0,0 +1,9 @@\n+#include \"reftable/reftable-tests.h\"\n+#include \"test-tool.h\"\n+\n+int cmd__reftable(int argc, const char **argv)\n+{\n+\tbasics_test_main(argc, argv);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex f97cd9f48a69..dfa2bee9e8a2 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -49,13 +49,14 @@ static struct test_cmd cmds[] = {\n \t{ \"pcre2-config\", cmd__pcre2_config },\n \t{ \"pkt-line\", cmd__pkt_line },\n \t{ \"prio-queue\", cmd__prio_queue },\n-\t{ \"proc-receive\", cmd__proc_receive},\n+\t{ \"proc-receive\", cmd__proc_receive },\n \t{ \"progress\", cmd__progress },\n \t{ \"reach\", cmd__reach },\n \t{ \"read-cache\", cmd__read_cache },\n \t{ \"read-graph\", cmd__read_graph },\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n+\t{ \"reftable\", cmd__reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 28072c0ad5ab..cad482a78915 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -45,6 +45,7 @@ int cmd__read_cache(int argc, const char **argv);\n int cmd__read_graph(int argc, const char **argv);\n int cmd__read_midx(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n+int cmd__reftable(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__repository(int argc, const char **argv);\n int cmd__revision_walking(int argc, const char **argv);\ndiff --git a/t/t0032-reftable-unittest.sh b/t/t0032-reftable-unittest.sh\nnew file mode 100755\nindex 000000000000..0ed14971a580\n--- /dev/null\n+++ b/t/t0032-reftable-unittest.sh\n@@ -0,0 +1,15 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable unittests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'unittests' '\n+\tTMPDIR=$(pwd) && export TMPDIR &&\n+\ttest-tool reftable\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"418988","messageId":"f2f005afca192b0d9b0fc878ca40f3bbed53a28d.1615580397.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v5.git.git.1615580397.gitgitgadget@gmail.com","subject":"[PATCH v5 09/15] reftable: write reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-12T20:19:50Z","receivedAt":"2021-03-12T20:20:45Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable-writer.h | 147 ++++++++\n reftable/writer.c | 691 +++++++++++++++++++++++++++++++++++++\n reftable/writer.h | 50 +++\n 4 files changed, 889 insertions(+)\n create mode 100644 reftable/reftable-writer.h\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n\ndiff --git a/Makefile b/Makefile\nindex 57a61b359ea7..d37dd79bfab7 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2389,6 +2389,7 @@ REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/tree.o\n+REFTABLE_OBJS += reftable/writer.o\n REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nnew file mode 100644\nindex 000000000000..9d2f8d605558\n--- /dev/null\n+++ b/reftable/reftable-writer.h\n@@ -0,0 +1,147 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_WRITER_H\n+#define REFTABLE_WRITER_H\n+\n+#include \n+\n+#include \"reftable-record.h\"\n+\n+/* Writing single reftables */\n+\n+/* reftable_write_options sets options for writing a single reftable. */\n+struct reftable_write_options {\n+\t/* boolean: do not pad out blocks to block size. */\n+\tunsigned unpadded : 1;\n+\n+\t/* the blocksize. Should be less than 2^24. */\n+\tuint32_t block_size;\n+\n+\t/* boolean: do not generate a SHA1 => ref index. */\n+\tunsigned skip_index_objects : 1;\n+\n+\t/* how often to write complete keys in each block. */\n+\tint restart_interval;\n+\n+\t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n+\t * Defaults to SHA1 if unset\n+\t */\n+\tuint32_t hash_id;\n+\n+\t/* boolean: do not check ref names for validity or dir/file conflicts.\n+\t */\n+\tunsigned skip_name_check : 1;\n+\n+\t/* boolean: copy log messages exactly. If unset, check that the message\n+\t * is a single line, and add '\\n' if missing.\n+\t */\n+\tunsigned exact_log_message : 1;\n+};\n+\n+/* reftable_block_stats holds statistics for a single block type */\n+struct reftable_block_stats {\n+\t/* total number of entries written */\n+\tint entries;\n+\t/* total number of key restarts */\n+\tint restarts;\n+\t/* total number of blocks */\n+\tint blocks;\n+\t/* total number of index blocks */\n+\tint index_blocks;\n+\t/* depth of the index */\n+\tint max_index_level;\n+\n+\t/* offset of the first block for this type */\n+\tuint64_t offset;\n+\t/* offset of the top level index block for this type, or 0 if not\n+\t * present */\n+\tuint64_t index_offset;\n+};\n+\n+/* stats holds overall statistics for a single reftable */\n+struct reftable_stats {\n+\t/* total number of blocks written. */\n+\tint blocks;\n+\t/* stats for ref data */\n+\tstruct reftable_block_stats ref_stats;\n+\t/* stats for the SHA1 to ref map. */\n+\tstruct reftable_block_stats obj_stats;\n+\t/* stats for index blocks */\n+\tstruct reftable_block_stats idx_stats;\n+\t/* stats for log blocks */\n+\tstruct reftable_block_stats log_stats;\n+\n+\t/* disambiguation length of shortened object IDs. */\n+\tint object_id_len;\n+};\n+\n+/* reftable_new_writer creates a new writer */\n+struct reftable_writer *\n+reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts);\n+\n+/* Set the range of update indices for the records we will add. When writing a\n+ table into a stack, the min should be at least\n+ reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned.\n+\n+ For transactional updates to a stack, typically min==max, and the\n+ update_index can be obtained by inspeciting the stack. When converting an\n+ existing ref database into a single reftable, this would be a range of\n+ update-index timestamps.\n+ */\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max);\n+\n+/*\n+ Add a reftable_ref_record. The record should have names that come after\n+ already added records.\n+\n+ The update_index must be within the limits set by\n+ reftable_writer_set_limits(), or REFTABLE_API_ERROR is returned. It is an\n+ REFTABLE_API_ERROR error to write a ref record after a log record.\n+*/\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/*\n+ Convenience function to add multiple reftable_ref_records; the function sorts\n+ the records before adding them, reordering the records array passed in.\n+*/\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n);\n+\n+/*\n+ adds reftable_log_records. Log records are keyed by (refname, decreasing\n+ update_index). The key for the record added must come after the already added\n+ log records.\n+*/\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log);\n+\n+/*\n+ Convenience function to add multiple reftable_log_records; the function sorts\n+ the records before adding them, reordering records array passed in.\n+*/\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n);\n+\n+/* reftable_writer_close finalizes the reftable. The writer is retained so\n+ * statistics can be inspected. */\n+int reftable_writer_close(struct reftable_writer *w);\n+\n+/* writer_stats returns the statistics on the reftable being written.\n+\n+ This struct becomes invalid when the writer is freed.\n+ */\n+const struct reftable_stats *writer_stats(struct reftable_writer *w);\n+\n+/* reftable_writer_free deallocates memory for the writer */\n+void reftable_writer_free(struct reftable_writer *w);\n+\n+#endif\ndiff --git a/reftable/writer.c b/reftable/writer.c\nnew file mode 100644\nindex 000000000000..d42ca8afac11\n--- /dev/null\n+++ b/reftable/writer.c\n@@ -0,0 +1,691 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"writer.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"tree.h\"\n+#include \"reftable-error.h\"\n+\n+/* finishes a block, and writes it to storage */\n+static int writer_flush_block(struct reftable_writer *w);\n+\n+/* deallocates memory related to the index */\n+static void writer_clear_index(struct reftable_writer *w);\n+\n+/* finishes writing a 'r' (refs) or 'g' (reflogs) section */\n+static int writer_finish_public_section(struct reftable_writer *w);\n+\n+static struct reftable_block_stats *\n+writer_reftable_block_stats(struct reftable_writer *w, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase 'r':\n+\t\treturn &w->stats.ref_stats;\n+\tcase 'o':\n+\t\treturn &w->stats.obj_stats;\n+\tcase 'i':\n+\t\treturn &w->stats.idx_stats;\n+\tcase 'g':\n+\t\treturn &w->stats.log_stats;\n+\t}\n+\tabort();\n+\treturn NULL;\n+}\n+\n+/* write data, queuing the padding for the next write. Returns negative for\n+ * error. */\n+static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n+\t\t\tint padding)\n+{\n+\tint n = 0;\n+\tif (w->pending_padding > 0) {\n+\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding);\n+\t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n+\t\tif (n < 0)\n+\t\t\treturn n;\n+\n+\t\tw->pending_padding = 0;\n+\t\treftable_free(zeroed);\n+\t}\n+\n+\tw->pending_padding = padding;\n+\tn = w->write(w->write_arg, data, len);\n+\tif (n < 0)\n+\t\treturn n;\n+\tn += padding;\n+\treturn 0;\n+}\n+\n+static void options_set_defaults(struct reftable_write_options *opts)\n+{\n+\tif (opts->restart_interval == 0) {\n+\t\topts->restart_interval = 16;\n+\t}\n+\n+\tif (opts->hash_id == 0) {\n+\t\topts->hash_id = SHA1_ID;\n+\t}\n+\tif (opts->block_size == 0) {\n+\t\topts->block_size = DEFAULT_BLOCK_SIZE;\n+\t}\n+}\n+\n+static int writer_version(struct reftable_writer *w)\n+{\n+\treturn (w->opts.hash_id == 0 || w->opts.hash_id == SHA1_ID) ? 1 : 2;\n+}\n+\n+static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n+{\n+\tmemcpy((char *)dest, \"REFT\", 4);\n+\n+\tdest[4] = writer_version(w);\n+\n+\tput_be24(dest + 5, w->opts.block_size);\n+\tput_be64(dest + 8, w->min_update_index);\n+\tput_be64(dest + 16, w->max_update_index);\n+\tif (writer_version(w) == 2) {\n+\t\tput_be32(dest + 24, w->opts.hash_id);\n+\t}\n+\treturn header_size(writer_version(w));\n+}\n+\n+static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n+{\n+\tint block_start = 0;\n+\tif (w->next == 0) {\n+\t\tblock_start = header_size(writer_version(w));\n+\t}\n+\n+\tstrbuf_release(&w->last_key);\n+\tblock_writer_init(&w->block_writer_data, typ, w->block,\n+\t\t\t w->opts.block_size, block_start,\n+\t\t\t hash_size(w->opts.hash_id));\n+\tw->block_writer = &w->block_writer_data;\n+\tw->block_writer->restart_interval = w->opts.restart_interval;\n+}\n+\n+static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n+\n+struct reftable_writer *\n+reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts)\n+{\n+\tstruct reftable_writer *wp =\n+\t\treftable_calloc(sizeof(struct reftable_writer));\n+\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n+\toptions_set_defaults(opts);\n+\tif (opts->block_size >= (1 << 24)) {\n+\t\t/* TODO - error return? */\n+\t\tabort();\n+\t}\n+\twp->last_key = reftable_empty_strbuf;\n+\twp->block = reftable_calloc(opts->block_size);\n+\twp->write = writer_func;\n+\twp->write_arg = writer_arg;\n+\twp->opts = *opts;\n+\twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n+\n+\treturn wp;\n+}\n+\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max)\n+{\n+\tw->min_update_index = min;\n+\tw->max_update_index = max;\n+}\n+\n+void reftable_writer_free(struct reftable_writer *w)\n+{\n+\treftable_free(w->block);\n+\treftable_free(w);\n+}\n+\n+struct obj_index_tree_node {\n+\tstruct strbuf hash;\n+\tuint64_t *offsets;\n+\tsize_t offset_len;\n+\tsize_t offset_cap;\n+};\n+\n+#define OBJ_INDEX_TREE_NODE_INIT \\\n+\t{ \\\n+\t\t.hash = STRBUF_INIT \\\n+\t}\n+\n+static int obj_index_tree_node_compare(const void *a, const void *b)\n+{\n+\treturn strbuf_cmp(&((const struct obj_index_tree_node *)a)->hash,\n+\t\t\t &((const struct obj_index_tree_node *)b)->hash);\n+}\n+\n+static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n+{\n+\tuint64_t off = w->next;\n+\n+\tstruct obj_index_tree_node want = { .hash = *hash };\n+\n+\tstruct tree_node *node = tree_search(&want, &w->obj_index_tree,\n+\t\t\t\t\t &obj_index_tree_node_compare, 0);\n+\tstruct obj_index_tree_node *key = NULL;\n+\tif (node == NULL) {\n+\t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n+\t\tkey = reftable_malloc(sizeof(struct obj_index_tree_node));\n+\t\t*key = empty;\n+\n+\t\tstrbuf_reset(&key->hash);\n+\t\tstrbuf_addbuf(&key->hash, hash);\n+\t\ttree_search((void *)key, &w->obj_index_tree,\n+\t\t\t &obj_index_tree_node_compare, 1);\n+\t} else {\n+\t\tkey = node->key;\n+\t}\n+\n+\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off) {\n+\t\treturn;\n+\t}\n+\n+\tif (key->offset_len == key->offset_cap) {\n+\t\tkey->offset_cap = 2 * key->offset_cap + 1;\n+\t\tkey->offsets = reftable_realloc(\n+\t\t\tkey->offsets, sizeof(uint64_t) * key->offset_cap);\n+\t}\n+\n+\tkey->offsets[key->offset_len++] = off;\n+}\n+\n+static int writer_add_record(struct reftable_writer *w,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tint result = -1;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = 0;\n+\treftable_record_key(rec, &key);\n+\tif (strbuf_cmp(&w->last_key, &key) >= 0)\n+\t\tgoto done;\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, &key);\n+\tif (w->block_writer == NULL) {\n+\t\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\t}\n+\n+\tassert(block_writer_type(w->block_writer) == reftable_record_type(rec));\n+\n+\tif (block_writer_add(w->block_writer, rec) == 0) {\n+\t\tresult = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0) {\n+\t\tresult = err;\n+\t\tgoto done;\n+\t}\n+\n+\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\terr = block_writer_add(w->block_writer, rec);\n+\tif (err < 0) {\n+\t\tresult = err;\n+\t\tgoto done;\n+\t}\n+\n+\tresult = 0;\n+done:\n+\tstrbuf_release(&key);\n+\treturn result;\n+}\n+\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_ref_record copy = *ref;\n+\tint err = 0;\n+\n+\tif (ref->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\tif (ref->update_index < w->min_update_index ||\n+\t ref->update_index > w->max_update_index)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\treftable_record_from_ref(&rec, ©);\n+\tcopy.update_index -= w->min_update_index;\n+\n+\terr = writer_add_record(w, &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (!w->opts.skip_index_objects &&\n+\t reftable_ref_record_val1(ref) != NULL) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, (char *)reftable_ref_record_val1(ref),\n+\t\t\t hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\n+\tif (!w->opts.skip_index_objects &&\n+\t reftable_ref_record_val2(ref) != NULL) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, reftable_ref_record_val2(ref),\n+\t\t\t hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(refs, n, reftable_ref_record_compare_name);\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_ref(w, &refs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+static int reftable_writer_add_log_verbatim(struct reftable_writer *w,\n+\t\t\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tif (w->block_writer != NULL &&\n+\t block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {\n+\t\tint err = writer_finish_public_section(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tw->next -= w->pending_padding;\n+\tw->pending_padding = 0;\n+\n+\treftable_record_from_log(&rec, log);\n+\treturn writer_add_record(w, &rec);\n+}\n+\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tchar *input_log_message = NULL;\n+\tstruct strbuf cleaned_message = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tif (log->value_type == REFTABLE_LOG_DELETION)\n+\t\treturn reftable_writer_add_log_verbatim(w, log);\n+\n+\tif (log->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\tinput_log_message = log->update.message;\n+\tif (!w->opts.exact_log_message && log->update.message != NULL) {\n+\t\tstrbuf_addstr(&cleaned_message, log->update.message);\n+\t\twhile (cleaned_message.len &&\n+\t\t cleaned_message.buf[cleaned_message.len - 1] == '\\n')\n+\t\t\tstrbuf_setlen(&cleaned_message,\n+\t\t\t\t cleaned_message.len - 1);\n+\t\tif (strchr(cleaned_message.buf, '\\n')) {\n+\t\t\t// multiple lines not allowed.\n+\t\t\terr = REFTABLE_API_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tstrbuf_addstr(&cleaned_message, \"\\n\");\n+\t\tlog->update.message = cleaned_message.buf;\n+\t}\n+\n+\terr = reftable_writer_add_log_verbatim(w, log);\n+\tlog->update.message = input_log_message;\n+done:\n+\tstrbuf_release(&cleaned_message);\n+\treturn err;\n+}\n+\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(logs, n, reftable_log_record_compare_key);\n+\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_log(w, &logs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+static int writer_finish_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tuint64_t index_start = 0;\n+\tint max_level = 0;\n+\tint threshold = w->opts.unpadded ? 1 : 3;\n+\tint before_blocks = w->stats.idx_stats.blocks;\n+\tint err = writer_flush_block(w);\n+\tint i = 0;\n+\tstruct reftable_block_stats *bstats = NULL;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\twhile (w->index_len > threshold) {\n+\t\tstruct reftable_index_record *idx = NULL;\n+\t\tint idx_len = 0;\n+\n+\t\tmax_level++;\n+\t\tindex_start = w->next;\n+\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\tidx = w->index;\n+\t\tidx_len = w->index_len;\n+\n+\t\tw->index = NULL;\n+\t\tw->index_len = 0;\n+\t\tw->index_cap = 0;\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstruct reftable_record rec = { NULL };\n+\t\t\treftable_record_from_index(&rec, idx + i);\n+\t\t\tif (block_writer_add(w->block_writer, &rec) == 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\terr = writer_flush_block(w);\n+\t\t\tif (err < 0)\n+\t\t\t\treturn err;\n+\n+\t\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\t\terr = block_writer_add(w->block_writer, &rec);\n+\t\t\tif (err != 0) {\n+\t\t\t\t/* write into fresh block should always succeed\n+\t\t\t\t */\n+\t\t\t\tabort();\n+\t\t\t}\n+\t\t}\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstrbuf_release(&idx[i].last_key);\n+\t\t}\n+\t\treftable_free(idx);\n+\t}\n+\n+\twriter_clear_index(w);\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tbstats = writer_reftable_block_stats(w, typ);\n+\tbstats->index_blocks = w->stats.idx_stats.blocks - before_blocks;\n+\tbstats->index_offset = index_start;\n+\tbstats->max_index_level = max_level;\n+\n+\t/* Reinit lastKey, as the next section can start with any key. */\n+\tw->last_key.len = 0;\n+\n+\treturn 0;\n+}\n+\n+struct common_prefix_arg {\n+\tstruct strbuf *last;\n+\tint max;\n+};\n+\n+static void update_common(void *void_arg, void *key)\n+{\n+\tstruct common_prefix_arg *arg = (struct common_prefix_arg *)void_arg;\n+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n+\tif (arg->last != NULL) {\n+\t\tint n = common_prefix_size(&entry->hash, arg->last);\n+\t\tif (n > arg->max) {\n+\t\t\targ->max = n;\n+\t\t}\n+\t}\n+\targ->last = &entry->hash;\n+}\n+\n+struct write_record_arg {\n+\tstruct reftable_writer *w;\n+\tint err;\n+};\n+\n+static void write_object_record(void *void_arg, void *key)\n+{\n+\tstruct write_record_arg *arg = (struct write_record_arg *)void_arg;\n+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n+\tstruct reftable_obj_record obj_rec = {\n+\t\t.hash_prefix = (uint8_t *)entry->hash.buf,\n+\t\t.hash_prefix_len = arg->w->stats.object_id_len,\n+\t\t.offsets = entry->offsets,\n+\t\t.offset_len = entry->offset_len,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\treftable_record_from_obj(&rec, &obj_rec);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\n+\targ->err = writer_flush_block(arg->w);\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\twriter_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\tobj_rec.offset_len = 0;\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\n+\t/* Should be able to write into a fresh block. */\n+\tassert(arg->err == 0);\n+\n+done:;\n+}\n+\n+static void object_record_free(void *void_arg, void *key)\n+{\n+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n+\n+\tFREE_AND_NULL(entry->offsets);\n+\tstrbuf_release(&entry->hash);\n+\treftable_free(entry);\n+}\n+\n+static int writer_dump_object_index(struct reftable_writer *w)\n+{\n+\tstruct write_record_arg closure = { .w = w };\n+\tstruct common_prefix_arg common = { NULL };\n+\tif (w->obj_index_tree != NULL) {\n+\t\tinfix_walk(w->obj_index_tree, &update_common, &common);\n+\t}\n+\tw->stats.object_id_len = common.max + 1;\n+\n+\twriter_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\n+\tif (w->obj_index_tree != NULL) {\n+\t\tinfix_walk(w->obj_index_tree, &write_object_record, &closure);\n+\t}\n+\n+\tif (closure.err < 0)\n+\t\treturn closure.err;\n+\treturn writer_finish_section(w);\n+}\n+\n+static int writer_finish_public_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = 0;\n+\tint err = 0;\n+\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\n+\ttyp = block_writer_type(w->block_writer);\n+\terr = writer_finish_section(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\tif (typ == BLOCK_TYPE_REF && !w->opts.skip_index_objects &&\n+\t w->stats.ref_stats.index_blocks > 0) {\n+\t\terr = writer_dump_object_index(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tif (w->obj_index_tree != NULL) {\n+\t\tinfix_walk(w->obj_index_tree, &object_record_free, NULL);\n+\t\ttree_free(w->obj_index_tree);\n+\t\tw->obj_index_tree = NULL;\n+\t}\n+\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+int reftable_writer_close(struct reftable_writer *w)\n+{\n+\tuint8_t footer[72];\n+\tuint8_t *p = footer;\n+\tint err = writer_finish_public_section(w);\n+\tint empty_table = w->next == 0;\n+\tif (err != 0)\n+\t\tgoto done;\n+\tw->pending_padding = 0;\n+\tif (empty_table) {\n+\t\t/* Empty tables need a header anyway. */\n+\t\tuint8_t header[28];\n+\t\tint n = writer_write_header(w, header);\n+\t\terr = padded_write(w, header, n, 0);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\tp += writer_write_header(w, footer);\n+\tput_be64(p, w->stats.ref_stats.index_offset);\n+\tp += 8;\n+\tput_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n+\tp += 8;\n+\tput_be64(p, w->stats.obj_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be64(p, w->stats.log_stats.offset);\n+\tp += 8;\n+\tput_be64(p, w->stats.log_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be32(p, crc32(0, footer, p - footer));\n+\tp += 4;\n+\n+\terr = padded_write(w, footer, footer_size(writer_version(w)), 0);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (empty_table) {\n+\t\terr = REFTABLE_EMPTY_TABLE_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\t/* free up memory. */\n+\tblock_writer_release(&w->block_writer_data);\n+\twriter_clear_index(w);\n+\tstrbuf_release(&w->last_key);\n+\treturn err;\n+}\n+\n+static void writer_clear_index(struct reftable_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->index_len; i++) {\n+\t\tstrbuf_release(&w->index[i].last_key);\n+\t}\n+\n+\tFREE_AND_NULL(w->index);\n+\tw->index_len = 0;\n+\tw->index_cap = 0;\n+}\n+\n+static const int debug = 0;\n+\n+static int writer_flush_nonempty_block(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tstruct reftable_block_stats *bstats =\n+\t\twriter_reftable_block_stats(w, typ);\n+\tuint64_t block_typ_off = (bstats->blocks == 0) ? w->next : 0;\n+\tint raw_bytes = block_writer_finish(w->block_writer);\n+\tint padding = 0;\n+\tint err = 0;\n+\tstruct reftable_index_record ir = { .last_key = STRBUF_INIT };\n+\tif (raw_bytes < 0)\n+\t\treturn raw_bytes;\n+\n+\tif (!w->opts.unpadded && typ != BLOCK_TYPE_LOG) {\n+\t\tpadding = w->opts.block_size - raw_bytes;\n+\t}\n+\n+\tif (block_typ_off > 0) {\n+\t\tbstats->offset = block_typ_off;\n+\t}\n+\n+\tbstats->entries += w->block_writer->entries;\n+\tbstats->restarts += w->block_writer->restart_len;\n+\tbstats->blocks++;\n+\tw->stats.blocks++;\n+\n+\tif (debug) {\n+\t\tfprintf(stderr, \"block %c off %\" PRIu64 \" sz %d (%d)\\n\", typ,\n+\t\t\tw->next, raw_bytes,\n+\t\t\tget_be24(w->block + w->block_writer->header_off + 1));\n+\t}\n+\n+\tif (w->next == 0) {\n+\t\twriter_write_header(w, w->block);\n+\t}\n+\n+\terr = padded_write(w, w->block, raw_bytes, padding);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (w->index_cap == w->index_len) {\n+\t\tw->index_cap = 2 * w->index_cap + 1;\n+\t\tw->index = reftable_realloc(\n+\t\t\tw->index,\n+\t\t\tsizeof(struct reftable_index_record) * w->index_cap);\n+\t}\n+\n+\tir.offset = w->next;\n+\tstrbuf_reset(&ir.last_key);\n+\tstrbuf_addbuf(&ir.last_key, &w->block_writer->last_key);\n+\tw->index[w->index_len] = ir;\n+\n+\tw->index_len++;\n+\tw->next += padding + raw_bytes;\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+static int writer_flush_block(struct reftable_writer *w)\n+{\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\tif (w->block_writer->entries == 0)\n+\t\treturn 0;\n+\treturn writer_flush_nonempty_block(w);\n+}\n+\n+const struct reftable_stats *writer_stats(struct reftable_writer *w)\n+{\n+\treturn &w->stats;\n+}\ndiff --git a/reftable/writer.h b/reftable/writer.h\nnew file mode 100644\nindex 000000000000..4921c249d06b\n--- /dev/null\n+++ b/reftable/writer.h\n@@ -0,0 +1,50 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef WRITER_H\n+#define WRITER_H\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"tree.h\"\n+#include \"reftable-writer.h\"\n+\n+struct reftable_writer {\n+\tint (*write)(void *, const void *, size_t);\n+\tvoid *write_arg;\n+\tint pending_padding;\n+\tstruct strbuf last_key;\n+\n+\t/* offset of next block to write. */\n+\tuint64_t next;\n+\tuint64_t min_update_index, max_update_index;\n+\tstruct reftable_write_options opts;\n+\n+\t/* memory buffer for writing */\n+\tuint8_t *block;\n+\n+\t/* writer for the current section. NULL or points to\n+\t * block_writer_data */\n+\tstruct block_writer *block_writer;\n+\n+\tstruct block_writer block_writer_data;\n+\n+\t/* pending index records for the current section */\n+\tstruct reftable_index_record *index;\n+\tsize_t index_len;\n+\tsize_t index_cap;\n+\n+\t/*\n+\t * tree for use with tsearch; used to populate the 'o' inverse OID\n+\t * map */\n+\tstruct tree_node *obj_index_tree;\n+\n+\tstruct reftable_stats stats;\n+};\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"418991","messageId":"390eaca4fc8cffe520cbb835582ab6d12de97253.1615580397.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v5.git.git.1615580397.gitgitgadget@gmail.com","subject":"[PATCH v5 06/15] reftable: (de)serialization for the polymorphic record type.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-12T20:19:47Z","receivedAt":"2021-03-12T20:20:45Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is structured as a sequence of blocks, and each block\ncontains a sequence of prefix-compressed key-value records. There are 4 types of\nrecords, and they have similarities in how they must be handled. This is\nachieved by introducing a polymorphic 'record' type that encapsulates ref, log,\nindex and object records.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/constants.h | 21 +\n reftable/record.c | 1203 ++++++++++++++++++++++++++++++++++++\n reftable/record.h | 139 +++++\n reftable/record_test.c | 405 ++++++++++++\n reftable/reftable-record.h | 114 ++++\n t/helper/test-reftable.c | 2 +-\n 7 files changed, 1885 insertions(+), 1 deletion(-)\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/reftable-record.h\n\ndiff --git a/Makefile b/Makefile\nindex 82db2ad5e2fd..e746c42f9e3b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2386,7 +2386,9 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/record.o\n \n+REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n \ndiff --git a/reftable/constants.h b/reftable/constants.h\nnew file mode 100644\nindex 000000000000..5eee72c4c113\n--- /dev/null\n+++ b/reftable/constants.h\n@@ -0,0 +1,21 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef CONSTANTS_H\n+#define CONSTANTS_H\n+\n+#define BLOCK_TYPE_LOG 'g'\n+#define BLOCK_TYPE_INDEX 'i'\n+#define BLOCK_TYPE_REF 'r'\n+#define BLOCK_TYPE_OBJ 'o'\n+#define BLOCK_TYPE_ANY 0\n+\n+#define MAX_RESTARTS ((1 << 16) - 1)\n+#define DEFAULT_BLOCK_SIZE 4096\n+\n+#endif\ndiff --git a/reftable/record.c b/reftable/record.c\nnew file mode 100644\nindex 000000000000..d9e4c7bde411\n--- /dev/null\n+++ b/reftable/record.c\n@@ -0,0 +1,1203 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+/* record.c - methods for different types of records. */\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"constants.h\"\n+#include \"reftable-error.h\"\n+#include \"basics.h\"\n+\n+int get_var_int(uint64_t *dest, struct string_view *in)\n+{\n+\tint ptr = 0;\n+\tuint64_t val;\n+\n+\tif (in->len == 0)\n+\t\treturn -1;\n+\tval = in->buf[ptr] & 0x7f;\n+\n+\twhile (in->buf[ptr] & 0x80) {\n+\t\tptr++;\n+\t\tif (ptr > in->len) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tval = (val + 1) << 7 | (uint64_t)(in->buf[ptr] & 0x7f);\n+\t}\n+\n+\t*dest = val;\n+\treturn ptr + 1;\n+}\n+\n+int put_var_int(struct string_view *dest, uint64_t val)\n+{\n+\tuint8_t buf[10] = { 0 };\n+\tint i = 9;\n+\tint n = 0;\n+\tbuf[i] = (uint8_t)(val & 0x7f);\n+\ti--;\n+\twhile (1) {\n+\t\tval >>= 7;\n+\t\tif (!val) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tval--;\n+\t\tbuf[i] = 0x80 | (uint8_t)(val & 0x7f);\n+\t\ti--;\n+\t}\n+\n+\tn = sizeof(buf) - i - 1;\n+\tif (dest->len < n)\n+\t\treturn -1;\n+\tmemcpy(dest->buf, &buf[i + 1], n);\n+\treturn n;\n+}\n+\n+int reftable_is_block_type(uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\tcase BLOCK_TYPE_LOG:\n+\tcase BLOCK_TYPE_OBJ:\n+\tcase BLOCK_TYPE_INDEX:\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec)\n+{\n+\tswitch (rec->value_type) {\n+\tcase REFTABLE_REF_VAL1:\n+\t\treturn rec->value.val1;\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn rec->value.val2.value;\n+\tdefault:\n+\t\treturn NULL;\n+\t}\n+}\n+\n+uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec)\n+{\n+\tswitch (rec->value_type) {\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn rec->value.val2.target_value;\n+\tdefault:\n+\t\treturn NULL;\n+\t}\n+}\n+\n+static int decode_string(struct strbuf *dest, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t tsize = 0;\n+\tint n = get_var_int(&tsize, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\tif (in.len < tsize)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, in.buf, tsize);\n+\tstring_view_consume(&in, tsize);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static int encode_string(char *str, struct string_view s)\n+{\n+\tstruct string_view start = s;\n+\tint l = strlen(str);\n+\tint n = put_var_int(&s, l);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\tif (s.len < l)\n+\t\treturn -1;\n+\tmemcpy(s.buf, str, l);\n+\tstring_view_consume(&s, l);\n+\n+\treturn start.len - s.len;\n+}\n+\n+int reftable_encode_key(int *restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra)\n+{\n+\tstruct string_view start = dest;\n+\tint prefix_len = common_prefix_size(&prev_key, &key);\n+\tuint64_t suffix_len = key.len - prefix_len;\n+\tint n = put_var_int(&dest, (uint64_t)prefix_len);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\t*restart = (prefix_len == 0);\n+\n+\tn = put_var_int(&dest, suffix_len << 3 | (uint64_t)extra);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\tif (dest.len < suffix_len)\n+\t\treturn -1;\n+\tmemcpy(dest.buf, key.buf + prefix_len, suffix_len);\n+\tstring_view_consume(&dest, suffix_len);\n+\n+\treturn start.len - dest.len;\n+}\n+\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t prefix_len = 0;\n+\tuint64_t suffix_len = 0;\n+\tint n = get_var_int(&prefix_len, &in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tif (prefix_len > last_key.len)\n+\t\treturn -1;\n+\n+\tn = get_var_int(&suffix_len, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\t*extra = (uint8_t)(suffix_len & 0x7);\n+\tsuffix_len >>= 3;\n+\n+\tif (in.len < suffix_len)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(key);\n+\tstrbuf_add(key, last_key.buf, prefix_len);\n+\tstrbuf_add(key, in.buf, suffix_len);\n+\tstring_view_consume(&in, suffix_len);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static void reftable_ref_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_ref_record *rec =\n+\t\t(const struct reftable_ref_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, rec->refname);\n+}\n+\n+static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *ref = (struct reftable_ref_record *)rec;\n+\tstruct reftable_ref_record *src = (struct reftable_ref_record *)src_rec;\n+\tassert(hash_size > 0);\n+\n+\t/* This is simple and correct, but we could probably reuse the hash\n+\t * fields. */\n+\treftable_ref_record_release(ref);\n+\tif (src->refname != NULL) {\n+\t\tref->refname = xstrdup(src->refname);\n+\t}\n+\tref->update_index = src->update_index;\n+\tref->value_type = src->value_type;\n+\tswitch (src->value_type) {\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\tref->value.val1 = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val1, src->value.val1, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\tref->value.val2.value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val2.value, src->value.val2.value, hash_size);\n+\t\tref->value.val2.target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val2.target_value,\n+\t\t src->value.val2.target_value, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tref->value.symref = xstrdup(src->value.symref);\n+\t\tbreak;\n+\t}\n+}\n+\n+static char hexdigit(int c)\n+{\n+\tif (c <= 9)\n+\t\treturn '0' + c;\n+\treturn 'a' + (c - 10);\n+}\n+\n+static void hex_format(char *dest, uint8_t *src, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (src != NULL) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < hash_size; i++) {\n+\t\t\tdest[2 * i] = hexdigit(src[i] >> 4);\n+\t\t\tdest[2 * i + 1] = hexdigit(src[i] & 0xf);\n+\t\t}\n+\t\tdest[2 * hash_size] = 0;\n+\t}\n+}\n+\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[2 * SHA256_SIZE + 1] = { 0 }; /* BUG */\n+\tprintf(\"ref{%s(%\" PRIu64 \") \", ref->refname, ref->update_index);\n+\tswitch (ref->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tprintf(\"=> %s\", ref->value.symref);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\thex_format(hex, ref->value.val2.value, hash_size(hash_id));\n+\t\tprintf(\"val 2 %s\", hex);\n+\t\thex_format(hex, ref->value.val2.target_value,\n+\t\t\t hash_size(hash_id));\n+\t\tprintf(\"(T %s)\", hex);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\thex_format(hex, ref->value.val1, hash_size(hash_id));\n+\t\tprintf(\"val 1 %s\", hex);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tprintf(\"delete\");\n+\t\tbreak;\n+\t}\n+\tprintf(\"}\\n\");\n+}\n+\n+static void reftable_ref_record_release_void(void *rec)\n+{\n+\treftable_ref_record_release((struct reftable_ref_record *)rec);\n+}\n+\n+void reftable_ref_record_release(struct reftable_ref_record *ref)\n+{\n+\tswitch (ref->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\treftable_free(ref->value.symref);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\treftable_free(ref->value.val2.target_value);\n+\t\treftable_free(ref->value.val2.value);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\treftable_free(ref->value.val1);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t}\n+\n+\treftable_free(ref->refname);\n+\tmemset(ref, 0, sizeof(struct reftable_ref_record));\n+}\n+\n+static uint8_t reftable_ref_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\treturn r->value_type;\n+}\n+\n+static int reftable_ref_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\tstruct string_view start = s;\n+\tint n = put_var_int(&s, r->update_index);\n+\tassert(hash_size > 0);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tswitch (r->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tn = encode_string(r->value.symref, s);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\tif (s.len < 2 * hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value.val2.value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tmemcpy(s.buf, r->value.val2.target_value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\tif (s.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value.val1, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t}\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_ref_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *r = (struct reftable_ref_record *)rec;\n+\tstruct string_view start = in;\n+\tuint64_t update_index = 0;\n+\tint n = get_var_int(&update_index, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\treftable_ref_record_release(r);\n+\n+\tassert(hash_size > 0);\n+\n+\tr->refname = reftable_realloc(r->refname, key.len + 1);\n+\tmemcpy(r->refname, key.buf, key.len);\n+\tr->update_index = update_index;\n+\tr->refname[key.len] = 0;\n+\tr->value_type = val_type;\n+\tswitch (val_type) {\n+\tcase REFTABLE_REF_VAL1:\n+\t\tif (in.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tr->value.val1 = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val1, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\n+\tcase REFTABLE_REF_VAL2:\n+\t\tif (in.len < 2 * hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tr->value.val2.value = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val2.value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\n+\t\tr->value.val2.target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val2.target_value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\n+\tcase REFTABLE_REF_SYMREF: {\n+\t\tstruct strbuf dest = STRBUF_INIT;\n+\t\tint n = decode_string(&dest, in);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\t\tr->value.symref = dest.buf;\n+\t} break;\n+\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t\tbreak;\n+\t}\n+\n+\treturn start.len - in.len;\n+}\n+\n+static int reftable_ref_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_ref_record_is_deletion(\n+\t\t(const struct reftable_ref_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_ref_record_vtable = {\n+\t.key = &reftable_ref_record_key,\n+\t.type = BLOCK_TYPE_REF,\n+\t.copy_from = &reftable_ref_record_copy_from,\n+\t.val_type = &reftable_ref_record_val_type,\n+\t.encode = &reftable_ref_record_encode,\n+\t.decode = &reftable_ref_record_decode,\n+\t.release = &reftable_ref_record_release_void,\n+\t.is_deletion = &reftable_ref_record_is_deletion_void,\n+};\n+\n+static void reftable_obj_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_obj_record *rec =\n+\t\t(const struct reftable_obj_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, rec->hash_prefix, rec->hash_prefix_len);\n+}\n+\n+static void reftable_obj_record_release(void *rec)\n+{\n+\tstruct reftable_obj_record *obj = (struct reftable_obj_record *)rec;\n+\tFREE_AND_NULL(obj->hash_prefix);\n+\tFREE_AND_NULL(obj->offsets);\n+\tmemset(obj, 0, sizeof(struct reftable_obj_record));\n+}\n+\n+static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_obj_record *obj = (struct reftable_obj_record *)rec;\n+\tconst struct reftable_obj_record *src =\n+\t\t(const struct reftable_obj_record *)src_rec;\n+\tint olen;\n+\n+\treftable_obj_record_release(obj);\n+\t*obj = *src;\n+\tobj->hash_prefix = reftable_malloc(obj->hash_prefix_len);\n+\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n+\n+\tolen = obj->offset_len * sizeof(uint64_t);\n+\tobj->offsets = reftable_malloc(olen);\n+\tmemcpy(obj->offsets, src->offsets, olen);\n+}\n+\n+static uint8_t reftable_obj_record_val_type(const void *rec)\n+{\n+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n+\tif (r->offset_len > 0 && r->offset_len < 8)\n+\t\treturn r->offset_len;\n+\treturn 0;\n+}\n+\n+static int reftable_obj_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n+\tstruct string_view start = s;\n+\tint i = 0;\n+\tint n = 0;\n+\tuint64_t last = 0;\n+\tif (r->offset_len == 0 || r->offset_len >= 8) {\n+\t\tn = put_var_int(&s, r->offset_len);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t}\n+\tif (r->offset_len == 0)\n+\t\treturn start.len - s.len;\n+\tn = put_var_int(&s, r->offsets[0]);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tlast = r->offsets[0];\n+\tfor (i = 1; i < r->offset_len; i++) {\n+\t\tint n = put_var_int(&s, r->offsets[i] - last);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tlast = r->offsets[i];\n+\t}\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_obj_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n+\tuint64_t count = val_type;\n+\tint n = 0;\n+\tuint64_t last;\n+\tint j;\n+\tr->hash_prefix = reftable_malloc(key.len);\n+\tmemcpy(r->hash_prefix, key.buf, key.len);\n+\tr->hash_prefix_len = key.len;\n+\n+\tif (val_type == 0) {\n+\t\tn = get_var_int(&count, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\n+\t\tstring_view_consume(&in, n);\n+\t}\n+\n+\tr->offsets = NULL;\n+\tr->offset_len = 0;\n+\tif (count == 0)\n+\t\treturn start.len - in.len;\n+\n+\tr->offsets = reftable_malloc(count * sizeof(uint64_t));\n+\tr->offset_len = count;\n+\n+\tn = get_var_int(&r->offsets[0], &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\tlast = r->offsets[0];\n+\tj = 1;\n+\twhile (j < count) {\n+\t\tuint64_t delta = 0;\n+\t\tint n = get_var_int(&delta, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\n+\t\tlast = r->offsets[j] = (delta + last);\n+\t\tj++;\n+\t}\n+\treturn start.len - in.len;\n+}\n+\n+static int not_a_deletion(const void *p)\n+{\n+\treturn 0;\n+}\n+\n+static struct reftable_record_vtable reftable_obj_record_vtable = {\n+\t.key = &reftable_obj_record_key,\n+\t.type = BLOCK_TYPE_OBJ,\n+\t.copy_from = &reftable_obj_record_copy_from,\n+\t.val_type = &reftable_obj_record_val_type,\n+\t.encode = &reftable_obj_record_encode,\n+\t.decode = &reftable_obj_record_decode,\n+\t.release = &reftable_obj_record_release,\n+\t.is_deletion = not_a_deletion,\n+};\n+\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[SHA256_SIZE + 1] = { 0 };\n+\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tprintf(\"log{%s(%\" PRIu64 \") delete\", log->refname,\n+\t\t log->update_index);\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tprintf(\"log{%s(%\" PRIu64 \") %s <%s> %\" PRIu64 \" %04d\\n\",\n+\t\t log->refname, log->update_index, log->update.name,\n+\t\t log->update.email, log->update.time,\n+\t\t log->update.tz_offset);\n+\t\thex_format(hex, log->update.old_hash, hash_size(hash_id));\n+\t\tprintf(\"%s => \", hex);\n+\t\thex_format(hex, log->update.new_hash, hash_size(hash_id));\n+\t\tprintf(\"%s\\n\\n%s\\n}\\n\", hex, log->update.message);\n+\t\tbreak;\n+\t}\n+}\n+\n+static void reftable_log_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_log_record *rec =\n+\t\t(const struct reftable_log_record *)r;\n+\tint len = strlen(rec->refname);\n+\tuint8_t i64[8];\n+\tuint64_t ts = 0;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, (uint8_t *)rec->refname, len + 1);\n+\n+\tts = (~ts) - rec->update_index;\n+\tput_be64(&i64[0], ts);\n+\tstrbuf_add(dest, i64, sizeof(i64));\n+}\n+\n+static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_log_record *dst = (struct reftable_log_record *)rec;\n+\tconst struct reftable_log_record *src =\n+\t\t(const struct reftable_log_record *)src_rec;\n+\n+\treftable_log_record_release(dst);\n+\t*dst = *src;\n+\tif (dst->refname != NULL) {\n+\t\tdst->refname = xstrdup(dst->refname);\n+\t}\n+\tswitch (dst->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tif (dst->update.email != NULL) {\n+\t\t\tdst->update.email = xstrdup(dst->update.email);\n+\t\t}\n+\t\tif (dst->update.name != NULL) {\n+\t\t\tdst->update.name = xstrdup(dst->update.name);\n+\t\t}\n+\t\tif (dst->update.message != NULL) {\n+\t\t\tdst->update.message = xstrdup(dst->update.message);\n+\t\t}\n+\n+\t\tif (dst->update.new_hash != NULL) {\n+\t\t\tdst->update.new_hash = reftable_malloc(hash_size);\n+\t\t\tmemcpy(dst->update.new_hash, src->update.new_hash,\n+\t\t\t hash_size);\n+\t\t}\n+\t\tif (dst->update.old_hash != NULL) {\n+\t\t\tdst->update.old_hash = reftable_malloc(hash_size);\n+\t\t\tmemcpy(dst->update.old_hash, src->update.old_hash,\n+\t\t\t hash_size);\n+\t\t}\n+\t\tbreak;\n+\t}\n+}\n+\n+static void reftable_log_record_release_void(void *rec)\n+{\n+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n+\treftable_log_record_release(r);\n+}\n+\n+void reftable_log_record_release(struct reftable_log_record *r)\n+{\n+\treftable_free(r->refname);\n+\tswitch (r->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\treftable_free(r->update.new_hash);\n+\t\treftable_free(r->update.old_hash);\n+\t\treftable_free(r->update.name);\n+\t\treftable_free(r->update.email);\n+\t\treftable_free(r->update.message);\n+\t\tbreak;\n+\t}\n+\tmemset(r, 0, sizeof(struct reftable_log_record));\n+}\n+\n+static uint8_t reftable_log_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_log_record *log =\n+\t\t(const struct reftable_log_record *)rec;\n+\n+\treturn reftable_log_record_is_deletion(log) ? 0 : 1;\n+}\n+\n+static uint8_t zero[SHA256_SIZE] = { 0 };\n+\n+static int reftable_log_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n+\tstruct string_view start = s;\n+\tint n = 0;\n+\tuint8_t *oldh = NULL;\n+\tuint8_t *newh = NULL;\n+\tif (reftable_log_record_is_deletion(r))\n+\t\treturn 0;\n+\n+\toldh = r->update.old_hash;\n+\tnewh = r->update.new_hash;\n+\tif (oldh == NULL) {\n+\t\toldh = zero;\n+\t}\n+\tif (newh == NULL) {\n+\t\tnewh = zero;\n+\t}\n+\n+\tif (s.len < 2 * hash_size)\n+\t\treturn -1;\n+\n+\tmemcpy(s.buf, oldh, hash_size);\n+\tmemcpy(s.buf + hash_size, newh, hash_size);\n+\tstring_view_consume(&s, 2 * hash_size);\n+\n+\tn = encode_string(r->update.name ? r->update.name : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = encode_string(r->update.email ? r->update.email : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = put_var_int(&s, r->update.time);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tif (s.len < 2)\n+\t\treturn -1;\n+\n+\tput_be16(s.buf, r->update.tz_offset);\n+\tstring_view_consume(&s, 2);\n+\n+\tn = encode_string(r->update.message ? r->update.message : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_log_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n+\tuint64_t max = 0;\n+\tuint64_t ts = 0;\n+\tstruct strbuf dest = STRBUF_INIT;\n+\tint n;\n+\n+\tif (key.len <= 9 || key.buf[key.len - 9] != 0)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->refname = reftable_realloc(r->refname, key.len - 8);\n+\tmemcpy(r->refname, key.buf, key.len - 8);\n+\tts = get_be64(key.buf + key.len - 8);\n+\n+\tr->update_index = (~max) - ts;\n+\n+\tif (val_type != r->value_type) {\n+\t\tswitch (r->value_type) {\n+\t\tcase REFTABLE_LOG_UPDATE:\n+\t\t\tFREE_AND_NULL(r->update.old_hash);\n+\t\t\tFREE_AND_NULL(r->update.new_hash);\n+\t\t\tFREE_AND_NULL(r->update.message);\n+\t\t\tFREE_AND_NULL(r->update.email);\n+\t\t\tFREE_AND_NULL(r->update.name);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_LOG_DELETION:\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tr->value_type = val_type;\n+\tif (val_type == REFTABLE_LOG_DELETION)\n+\t\treturn 0;\n+\n+\tif (in.len < 2 * hash_size)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->update.old_hash = reftable_realloc(r->update.old_hash, hash_size);\n+\tr->update.new_hash = reftable_realloc(r->update.new_hash, hash_size);\n+\n+\tmemcpy(r->update.old_hash, in.buf, hash_size);\n+\tmemcpy(r->update.new_hash, in.buf + hash_size, hash_size);\n+\n+\tstring_view_consume(&in, 2 * hash_size);\n+\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->update.name = reftable_realloc(r->update.name, dest.len + 1);\n+\tmemcpy(r->update.name, dest.buf, dest.len);\n+\tr->update.name[dest.len] = 0;\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->update.email = reftable_realloc(r->update.email, dest.len + 1);\n+\tmemcpy(r->update.email, dest.buf, dest.len);\n+\tr->update.email[dest.len] = 0;\n+\n+\tts = 0;\n+\tn = get_var_int(&ts, &in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\tr->update.time = ts;\n+\tif (in.len < 2)\n+\t\tgoto done;\n+\n+\tr->update.tz_offset = get_be16(in.buf);\n+\tstring_view_consume(&in, 2);\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->update.message = reftable_realloc(r->update.message, dest.len + 1);\n+\tmemcpy(r->update.message, dest.buf, dest.len);\n+\tr->update.message[dest.len] = 0;\n+\n+\tstrbuf_release(&dest);\n+\treturn start.len - in.len;\n+\n+done:\n+\tstrbuf_release(&dest);\n+\treturn REFTABLE_FORMAT_ERROR;\n+}\n+\n+static int null_streq(char *a, char *b)\n+{\n+\tchar *empty = \"\";\n+\tif (a == NULL)\n+\t\ta = empty;\n+\n+\tif (b == NULL)\n+\t\tb = empty;\n+\n+\treturn 0 == strcmp(a, b);\n+}\n+\n+static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)\n+{\n+\tif (a == NULL)\n+\t\ta = zero;\n+\n+\tif (b == NULL)\n+\t\tb = zero;\n+\n+\treturn !memcmp(a, b, sz);\n+}\n+\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size)\n+{\n+\tif (!(null_streq(a->refname, b->refname) &&\n+\t a->update_index == b->update_index &&\n+\t a->value_type == b->value_type))\n+\t\treturn 0;\n+\n+\tswitch (a->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\treturn 1;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\treturn null_streq(a->update.name, b->update.name) &&\n+\t\t a->update.time == b->update.time &&\n+\t\t a->update.tz_offset == b->update.tz_offset &&\n+\t\t null_streq(a->update.email, b->update.email) &&\n+\t\t null_streq(a->update.message, b->update.message) &&\n+\t\t zero_hash_eq(a->update.old_hash, b->update.old_hash,\n+\t\t\t\t hash_size) &&\n+\t\t zero_hash_eq(a->update.new_hash, b->update.new_hash,\n+\t\t\t\t hash_size);\n+\t}\n+\n+\tabort();\n+}\n+\n+static int reftable_log_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_log_record_is_deletion(\n+\t\t(const struct reftable_log_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_log_record_vtable = {\n+\t.key = &reftable_log_record_key,\n+\t.type = BLOCK_TYPE_LOG,\n+\t.copy_from = &reftable_log_record_copy_from,\n+\t.val_type = &reftable_log_record_val_type,\n+\t.encode = &reftable_log_record_encode,\n+\t.decode = &reftable_log_record_decode,\n+\t.release = &reftable_log_record_release_void,\n+\t.is_deletion = &reftable_log_record_is_deletion_void,\n+};\n+\n+struct reftable_record reftable_new_record(uint8_t typ)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF: {\n+\t\tstruct reftable_ref_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_ref_record));\n+\t\treftable_record_from_ref(&rec, r);\n+\t\treturn rec;\n+\t}\n+\n+\tcase BLOCK_TYPE_OBJ: {\n+\t\tstruct reftable_obj_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_obj_record));\n+\t\treftable_record_from_obj(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_LOG: {\n+\t\tstruct reftable_log_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_log_record));\n+\t\treftable_record_from_log(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_INDEX: {\n+\t\tstruct reftable_index_record empty = { .last_key =\n+\t\t\t\t\t\t\t STRBUF_INIT };\n+\t\tstruct reftable_index_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_index_record));\n+\t\t*r = empty;\n+\t\treftable_record_from_index(&rec, r);\n+\t\treturn rec;\n+\t}\n+\t}\n+\tabort();\n+\treturn rec;\n+}\n+\n+/* clear out the record, yielding the reftable_record data that was\n+ * encapsulated. */\n+static void *reftable_record_yield(struct reftable_record *rec)\n+{\n+\tvoid *p = rec->data;\n+\trec->data = NULL;\n+\treturn p;\n+}\n+\n+void reftable_record_destroy(struct reftable_record *rec)\n+{\n+\treftable_record_release(rec);\n+\treftable_free(reftable_record_yield(rec));\n+}\n+\n+static void reftable_index_record_key(const void *r, struct strbuf *dest)\n+{\n+\tstruct reftable_index_record *rec = (struct reftable_index_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addbuf(dest, &rec->last_key);\n+}\n+\n+static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_index_record *dst = (struct reftable_index_record *)rec;\n+\tstruct reftable_index_record *src =\n+\t\t(struct reftable_index_record *)src_rec;\n+\n+\tstrbuf_reset(&dst->last_key);\n+\tstrbuf_addbuf(&dst->last_key, &src->last_key);\n+\tdst->offset = src->offset;\n+}\n+\n+static void reftable_index_record_release(void *rec)\n+{\n+\tstruct reftable_index_record *idx = (struct reftable_index_record *)rec;\n+\tstrbuf_release(&idx->last_key);\n+}\n+\n+static uint8_t reftable_index_record_val_type(const void *rec)\n+{\n+\treturn 0;\n+}\n+\n+static int reftable_index_record_encode(const void *rec, struct string_view out,\n+\t\t\t\t\tint hash_size)\n+{\n+\tconst struct reftable_index_record *r =\n+\t\t(const struct reftable_index_record *)rec;\n+\tstruct string_view start = out;\n+\n+\tint n = put_var_int(&out, r->offset);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&out, n);\n+\n+\treturn start.len - out.len;\n+}\n+\n+static int reftable_index_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t\tuint8_t val_type, struct string_view in,\n+\t\t\t\t\tint hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_index_record *r = (struct reftable_index_record *)rec;\n+\tint n = 0;\n+\n+\tstrbuf_reset(&r->last_key);\n+\tstrbuf_addbuf(&r->last_key, &key);\n+\n+\tn = get_var_int(&r->offset, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&in, n);\n+\treturn start.len - in.len;\n+}\n+\n+static struct reftable_record_vtable reftable_index_record_vtable = {\n+\t.key = &reftable_index_record_key,\n+\t.type = BLOCK_TYPE_INDEX,\n+\t.copy_from = &reftable_index_record_copy_from,\n+\t.val_type = &reftable_index_record_val_type,\n+\t.encode = &reftable_index_record_encode,\n+\t.decode = &reftable_index_record_decode,\n+\t.release = &reftable_index_record_release,\n+\t.is_deletion = ¬_a_deletion,\n+};\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)\n+{\n+\trec->ops->key(rec->data, dest);\n+}\n+\n+uint8_t reftable_record_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->type;\n+}\n+\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size)\n+{\n+\treturn rec->ops->encode(rec->data, dest, hash_size);\n+}\n+\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size)\n+{\n+\tassert(src->ops->type == rec->ops->type);\n+\n+\trec->ops->copy_from(rec->data, src->data, hash_size);\n+}\n+\n+uint8_t reftable_record_val_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->val_type(rec->data);\n+}\n+\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src, int hash_size)\n+{\n+\treturn rec->ops->decode(rec->data, key, extra, src, hash_size);\n+}\n+\n+void reftable_record_release(struct reftable_record *rec)\n+{\n+\trec->ops->release(rec->data);\n+}\n+\n+int reftable_record_is_deletion(struct reftable_record *rec)\n+{\n+\treturn rec->ops->is_deletion(rec->data);\n+}\n+\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *ref_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = ref_rec;\n+\trec->ops = &reftable_ref_record_vtable;\n+}\n+\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *obj_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = obj_rec;\n+\trec->ops = &reftable_obj_record_vtable;\n+}\n+\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *index_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = index_rec;\n+\trec->ops = &reftable_index_record_vtable;\n+}\n+\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *log_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = log_rec;\n+\trec->ops = &reftable_log_record_vtable;\n+}\n+\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_REF);\n+\treturn (struct reftable_ref_record *)rec->data;\n+}\n+\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_LOG);\n+\treturn (struct reftable_log_record *)rec->data;\n+}\n+\n+static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)\n+{\n+\tif (a != NULL && b != NULL)\n+\t\treturn !memcmp(a, b, hash_size);\n+\n+\treturn a == b;\n+}\n+\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (!(0 == strcmp(a->refname, b->refname) &&\n+\t a->update_index == b->update_index &&\n+\t a->value_type == b->value_type))\n+\t\treturn 0;\n+\n+\tswitch (a->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\treturn !strcmp(a->value.symref, b->value.symref);\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn hash_equal(a->value.val2.value, b->value.val2.value,\n+\t\t\t\t hash_size) &&\n+\t\t hash_equal(a->value.val2.target_value,\n+\t\t\t\t b->value.val2.target_value, hash_size);\n+\tcase REFTABLE_REF_VAL1:\n+\t\treturn hash_equal(a->value.val1, b->value.val1, hash_size);\n+\tcase REFTABLE_REF_DELETION:\n+\t\treturn 1;\n+\tdefault:\n+\t\tabort();\n+\t}\n+}\n+\n+int reftable_ref_record_compare_name(const void *a, const void *b)\n+{\n+\treturn strcmp(((struct reftable_ref_record *)a)->refname,\n+\t\t ((struct reftable_ref_record *)b)->refname);\n+}\n+\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref)\n+{\n+\treturn ref->value_type == REFTABLE_REF_DELETION;\n+}\n+\n+int reftable_log_record_compare_key(const void *a, const void *b)\n+{\n+\tstruct reftable_log_record *la = (struct reftable_log_record *)a;\n+\tstruct reftable_log_record *lb = (struct reftable_log_record *)b;\n+\n+\tint cmp = strcmp(la->refname, lb->refname);\n+\tif (cmp)\n+\t\treturn cmp;\n+\tif (la->update_index > lb->update_index)\n+\t\treturn -1;\n+\treturn (la->update_index < lb->update_index) ? 1 : 0;\n+}\n+\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log)\n+{\n+\treturn (log->value_type == REFTABLE_LOG_DELETION);\n+}\n+\n+void string_view_consume(struct string_view *s, int n)\n+{\n+\ts->buf += n;\n+\ts->len -= n;\n+}\ndiff --git a/reftable/record.h b/reftable/record.h\nnew file mode 100644\nindex 000000000000..498e8c50bf4f\n--- /dev/null\n+++ b/reftable/record.h\n@@ -0,0 +1,139 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef RECORD_H\n+#define RECORD_H\n+\n+#include \"system.h\"\n+\n+#include \n+\n+#include \"reftable-record.h\"\n+\n+/*\n+ * A substring of existing string data. This structure takes no responsibility\n+ * for the lifetime of the data it points to.\n+ */\n+struct string_view {\n+\tuint8_t *buf;\n+\tsize_t len;\n+};\n+\n+/* Advance `s.buf` by `n`, and decrease length. */\n+void string_view_consume(struct string_view *s, int n);\n+\n+/* utilities for de/encoding varints */\n+\n+int get_var_int(uint64_t *dest, struct string_view *in);\n+int put_var_int(struct string_view *dest, uint64_t val);\n+\n+/* Methods for records. */\n+struct reftable_record_vtable {\n+\t/* encode the key of to a uint8_t strbuf. */\n+\tvoid (*key)(const void *rec, struct strbuf *dest);\n+\n+\t/* The record type of ('r' for ref). */\n+\tuint8_t type;\n+\n+\tvoid (*copy_from)(void *dest, const void *src, int hash_size);\n+\n+\t/* a value of [0..7], indicating record subvariants (eg. ref vs. symref\n+\t * vs ref deletion) */\n+\tuint8_t (*val_type)(const void *rec);\n+\n+\t/* encodes rec into dest, returning how much space was used. */\n+\tint (*encode)(const void *rec, struct string_view dest, int hash_size);\n+\n+\t/* decode data from `src` into the record. */\n+\tint (*decode)(void *rec, struct strbuf key, uint8_t extra,\n+\t\t struct string_view src, int hash_size);\n+\n+\t/* deallocate and null the record. */\n+\tvoid (*release)(void *rec);\n+\n+\t/* is this a tombstone? */\n+\tint (*is_deletion)(const void *rec);\n+};\n+\n+/* record is a generic wrapper for different types of records. */\n+struct reftable_record {\n+\tvoid *data;\n+\tstruct reftable_record_vtable *ops;\n+};\n+\n+/* returns true for recognized block types. Block start with the block type. */\n+int reftable_is_block_type(uint8_t typ);\n+\n+/* creates a malloced record of the given type. Dispose with record_destroy */\n+struct reftable_record reftable_new_record(uint8_t typ);\n+\n+/* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns\n+ * number of bytes written. */\n+int reftable_encode_key(int *is_restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra);\n+\n+/* Decode into `key` and `extra` from `in` */\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in);\n+\n+/* reftable_index_record are used internally to speed up lookups. */\n+struct reftable_index_record {\n+\tuint64_t offset; /* Offset of block */\n+\tstruct strbuf last_key; /* Last key of the block. */\n+};\n+\n+/* reftable_obj_record stores an object ID => ref mapping. */\n+struct reftable_obj_record {\n+\tuint8_t *hash_prefix; /* leading bytes of the object ID */\n+\tint hash_prefix_len; /* number of leading bytes. Constant\n+\t\t\t * across a single table. */\n+\tuint64_t *offsets; /* a vector of file offsets. */\n+\tint offset_len;\n+};\n+\n+/* see struct record_vtable */\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);\n+uint8_t reftable_record_type(struct reftable_record *rec);\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size);\n+uint8_t reftable_record_val_type(struct reftable_record *rec);\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size);\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src,\n+\t\t\t int hash_size);\n+int reftable_record_is_deletion(struct reftable_record *rec);\n+\n+/* zeroes out the embedded record */\n+void reftable_record_release(struct reftable_record *rec);\n+\n+/* clear and deallocate embedded record, and zero `rec`. */\n+void reftable_record_destroy(struct reftable_record *rec);\n+\n+/* initialize generic records from concrete records. The generic record should\n+ * be zeroed out. */\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *objrec);\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *idxrec);\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *refrec);\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *logrec);\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *ref);\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *ref);\n+\n+/* for qsort. */\n+int reftable_ref_record_compare_name(const void *a, const void *b);\n+\n+/* for qsort. */\n+int reftable_log_record_compare_key(const void *a, const void *b);\n+\n+#endif\ndiff --git a/reftable/record_test.c b/reftable/record_test.c\nnew file mode 100644\nindex 000000000000..da7951bd480b\n--- /dev/null\n+++ b/reftable/record_test.c\n@@ -0,0 +1,405 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_copy(struct reftable_record *rec)\n+{\n+\tstruct reftable_record copy =\n+\t\treftable_new_record(reftable_record_type(rec));\n+\treftable_record_copy_from(©, rec, SHA1_SIZE);\n+\t/* do it twice to catch memory leaks */\n+\treftable_record_copy_from(©, rec, SHA1_SIZE);\n+\tswitch (reftable_record_type(©)) {\n+\tcase BLOCK_TYPE_REF:\n+\t\tEXPECT(reftable_ref_record_equal(reftable_record_as_ref(©),\n+\t\t\t\t\t\t reftable_record_as_ref(rec),\n+\t\t\t\t\t\t SHA1_SIZE));\n+\t\tbreak;\n+\tcase BLOCK_TYPE_LOG:\n+\t\tEXPECT(reftable_log_record_equal(reftable_record_as_log(©),\n+\t\t\t\t\t\t reftable_record_as_log(rec),\n+\t\t\t\t\t\t SHA1_SIZE));\n+\t\tbreak;\n+\t}\n+\treftable_record_destroy(©);\n+}\n+\n+static void test_varint_roundtrip(void)\n+{\n+\tuint64_t inputs[] = { 0,\n+\t\t\t 1,\n+\t\t\t 27,\n+\t\t\t 127,\n+\t\t\t 128,\n+\t\t\t 257,\n+\t\t\t 4096,\n+\t\t\t ((uint64_t)1 << 63),\n+\t\t\t ((uint64_t)1 << 63) + ((uint64_t)1 << 63) - 1 };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(inputs); i++) {\n+\t\tuint8_t dest[10];\n+\n+\t\tstruct string_view out = {\n+\t\t\t.buf = dest,\n+\t\t\t.len = sizeof(dest),\n+\t\t};\n+\t\tuint64_t in = inputs[i];\n+\t\tint n = put_var_int(&out, in);\n+\t\tuint64_t got = 0;\n+\n+\t\tEXPECT(n > 0);\n+\t\tout.len = n;\n+\t\tn = get_var_int(&got, &out);\n+\t\tEXPECT(n > 0);\n+\n+\t\tEXPECT(got == in);\n+\t}\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct {\n+\t\tconst char *a, *b;\n+\t\tint want;\n+\t} cases[] = {\n+\t\t{ \"abc\", \"ab\", 2 },\n+\t\t{ \"\", \"abc\", 0 },\n+\t\t{ \"abc\", \"abd\", 2 },\n+\t\t{ \"abc\", \"pqr\", 0 },\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct strbuf a = STRBUF_INIT;\n+\t\tstruct strbuf b = STRBUF_INIT;\n+\t\tstrbuf_addstr(&a, cases[i].a);\n+\t\tstrbuf_addstr(&b, cases[i].b);\n+\t\tEXPECT(common_prefix_size(&a, &b) == cases[i].want);\n+\n+\t\tstrbuf_release(&a);\n+\t\tstrbuf_release(&b);\n+\t}\n+}\n+\n+static void set_hash(uint8_t *h, int j)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < hash_size(SHA1_ID); i++) {\n+\t\th[i] = (j >> i) & 0xff;\n+\t}\n+}\n+\n+static void test_reftable_ref_record_roundtrip(void)\n+{\n+\tint i = 0;\n+\n+\tfor (i = REFTABLE_REF_DELETION; i < REFTABLE_NR_REF_VALUETYPES; i++) {\n+\t\tstruct reftable_ref_record in = { NULL };\n+\t\tstruct reftable_ref_record out = { NULL };\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\n+\t\tint n, m;\n+\n+\t\tin.value_type = i;\n+\t\tswitch (i) {\n+\t\tcase REFTABLE_REF_DELETION:\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\tin.value.val1 = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_hash(in.value.val1, 1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\tin.value.val2.value = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_hash(in.value.val2.value, 1);\n+\t\t\tin.value.val2.target_value = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_hash(in.value.val2.target_value, 2);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF:\n+\t\t\tin.value.symref = xstrdup(\"target\");\n+\t\t\tbreak;\n+\t\t}\n+\t\tin.refname = xstrdup(\"refs/heads/master\");\n+\n+\t\treftable_record_from_ref(&rec, &in);\n+\t\ttest_copy(&rec);\n+\n+\t\tEXPECT(reftable_record_val_type(&rec) == i);\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\t\tEXPECT(n > 0);\n+\n+\t\t/* decode into a non-zero reftable_record to test for leaks. */\n+\n+\t\treftable_record_from_ref(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, i, dest, SHA1_SIZE);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(reftable_ref_record_equal(&in, &out, SHA1_SIZE));\n+\t\treftable_record_release(&rec_out);\n+\n+\t\tstrbuf_release(&key);\n+\t\treftable_ref_record_release(&in);\n+\t}\n+}\n+\n+static void test_reftable_log_record_equal(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 42,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t}\n+\t};\n+\n+\tEXPECT(!reftable_log_record_equal(&in[0], &in[1], SHA1_SIZE));\n+\tin[1].update_index = in[0].update_index;\n+\tEXPECT(reftable_log_record_equal(&in[0], &in[1], SHA1_SIZE));\n+\treftable_log_record_release(&in[0]);\n+\treftable_log_record_release(&in[1]);\n+}\n+\n+static void test_reftable_log_record_roundtrip(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 42,\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.update = {\n+\t\t\t\t.old_hash = reftable_malloc(SHA1_SIZE),\n+\t\t\t\t.new_hash = reftable_malloc(SHA1_SIZE),\n+\t\t\t\t.name = xstrdup(\"han-wen\"),\n+\t\t\t\t.email = xstrdup(\"hanwen@google.com\"),\n+\t\t\t\t.message = xstrdup(\"test\"),\n+\t\t\t\t.time = 1577123507,\n+\t\t\t\t.tz_offset = 100,\n+\t\t\t}\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t\t.value_type = REFTABLE_LOG_DELETION,\n+\t\t}\n+\t};\n+\tset_test_hash(in[0].update.new_hash, 1);\n+\tset_test_hash(in[0].update.old_hash, 2);\n+\tfor (int i = 0; i < ARRAY_SIZE(in); i++) {\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\t/* populate out, to check for leaks. */\n+\t\tstruct reftable_log_record out = {\n+\t\t\t.refname = xstrdup(\"old name\"),\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.update = {\n+\t\t\t\t.new_hash = reftable_calloc(SHA1_SIZE),\n+\t\t\t\t.old_hash = reftable_calloc(SHA1_SIZE),\n+\t\t\t\t.name = xstrdup(\"old name\"),\n+\t\t\t\t.email = xstrdup(\"old@email\"),\n+\t\t\t\t.message = xstrdup(\"old message\"),\n+\t\t\t},\n+\t\t};\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m, valtype;\n+\n+\t\treftable_record_from_log(&rec, &in[i]);\n+\n+\t\ttest_copy(&rec);\n+\n+\t\treftable_record_key(&rec, &key);\n+\n+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\t\tEXPECT(n >= 0);\n+\t\treftable_record_from_log(&rec_out, &out);\n+\t\tvaltype = reftable_record_val_type(&rec);\n+\t\tm = reftable_record_decode(&rec_out, key, valtype, dest,\n+\t\t\t\t\t SHA1_SIZE);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(reftable_log_record_equal(&in[i], &out, SHA1_SIZE));\n+\t\treftable_log_record_release(&in[i]);\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_release(&rec_out);\n+\t}\n+}\n+\n+static void test_u24_roundtrip(void)\n+{\n+\tuint32_t in = 0x112233;\n+\tuint8_t dest[3];\n+\tuint32_t out;\n+\tput_be24(dest, in);\n+\tout = get_be24(dest);\n+\tEXPECT(in == out);\n+}\n+\n+static void test_key_roundtrip(void)\n+{\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct strbuf roundtrip = STRBUF_INIT;\n+\tint restart;\n+\tuint8_t extra;\n+\tint n, m;\n+\tuint8_t rt_extra;\n+\n+\tstrbuf_addstr(&last_key, \"refs/heads/master\");\n+\tstrbuf_addstr(&key, \"refs/tags/bla\");\n+\textra = 6;\n+\tn = reftable_encode_key(&restart, dest, last_key, key, extra);\n+\tEXPECT(!restart);\n+\tEXPECT(n > 0);\n+\n+\tm = reftable_decode_key(&roundtrip, &rt_extra, last_key, dest);\n+\tEXPECT(n == m);\n+\tEXPECT(0 == strbuf_cmp(&key, &roundtrip));\n+\tEXPECT(rt_extra == extra);\n+\n+\tstrbuf_release(&last_key);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&roundtrip);\n+}\n+\n+static void test_reftable_obj_record_roundtrip(void)\n+{\n+\tuint8_t testHash1[SHA1_SIZE] = { 1, 2, 3, 4, 0 };\n+\tuint64_t till9[] = { 1, 2, 3, 4, 500, 600, 700, 800, 9000 };\n+\tstruct reftable_obj_record recs[3] = { {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 3,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 9,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t } };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(recs); i++) {\n+\t\tstruct reftable_obj_record in = recs[i];\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_obj_record out = { NULL };\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m;\n+\t\tuint8_t extra;\n+\n+\t\treftable_record_from_obj(&rec, &in);\n+\t\ttest_copy(&rec);\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\t\tEXPECT(n > 0);\n+\t\textra = reftable_record_val_type(&rec);\n+\t\treftable_record_from_obj(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, extra, dest,\n+\t\t\t\t\t SHA1_SIZE);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(in.hash_prefix_len == out.hash_prefix_len);\n+\t\tEXPECT(in.offset_len == out.offset_len);\n+\n+\t\tEXPECT(!memcmp(in.hash_prefix, out.hash_prefix,\n+\t\t\t in.hash_prefix_len));\n+\t\tEXPECT(0 == memcmp(in.offsets, out.offsets,\n+\t\t\t\t sizeof(uint64_t) * in.offset_len));\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_release(&rec_out);\n+\t}\n+}\n+\n+static void test_reftable_index_record_roundtrip(void)\n+{\n+\tstruct reftable_index_record in = {\n+\t\t.offset = 42,\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_index_record out = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record out_rec = { NULL };\n+\tint n, m;\n+\tuint8_t extra;\n+\n+\tstrbuf_addstr(&in.last_key, \"refs/heads/master\");\n+\treftable_record_from_index(&rec, &in);\n+\treftable_record_key(&rec, &key);\n+\ttest_copy(&rec);\n+\n+\tEXPECT(0 == strbuf_cmp(&key, &in.last_key));\n+\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\tEXPECT(n > 0);\n+\n+\textra = reftable_record_val_type(&rec);\n+\treftable_record_from_index(&out_rec, &out);\n+\tm = reftable_record_decode(&out_rec, key, extra, dest, SHA1_SIZE);\n+\tEXPECT(m == n);\n+\n+\tEXPECT(in.offset == out.offset);\n+\n+\treftable_record_release(&out_rec);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&in.last_key);\n+}\n+\n+int record_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_reftable_log_record_equal);\n+\tRUN_TEST(test_reftable_log_record_roundtrip);\n+\tRUN_TEST(test_reftable_ref_record_roundtrip);\n+\tRUN_TEST(test_varint_roundtrip);\n+\tRUN_TEST(test_key_roundtrip);\n+\tRUN_TEST(test_common_prefix);\n+\tRUN_TEST(test_reftable_obj_record_roundtrip);\n+\tRUN_TEST(test_reftable_index_record_roundtrip);\n+\tRUN_TEST(test_u24_roundtrip);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-record.h b/reftable/reftable-record.h\nnew file mode 100644\nindex 000000000000..7985b94ae2cc\n--- /dev/null\n+++ b/reftable/reftable-record.h\n@@ -0,0 +1,114 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_RECORD_H\n+#define REFTABLE_RECORD_H\n+\n+#include \n+\n+/*\n+ * Basic data types\n+ *\n+ * Reftables store the state of each ref in struct reftable_ref_record, and they\n+ * store a sequence of reflog updates in struct reftable_log_record.\n+ */\n+\n+/* reftable_ref_record holds a ref database entry target_value */\n+struct reftable_ref_record {\n+\tchar *refname; /* Name of the ref, malloced. */\n+\tuint64_t update_index; /* Logical timestamp at which this value is\n+\t\t\t\t* written */\n+\n+\tenum {\n+\t\t/* tombstone to hide deletions from earlier tables */\n+\t\tREFTABLE_REF_DELETION = 0x0,\n+\n+\t\t/* a simple ref */\n+\t\tREFTABLE_REF_VAL1 = 0x1,\n+\t\t/* a tag, plus its peeled hash */\n+\t\tREFTABLE_REF_VAL2 = 0x2,\n+\n+\t\t/* a symbolic reference */\n+\t\tREFTABLE_REF_SYMREF = 0x3,\n+#define REFTABLE_NR_REF_VALUETYPES 4\n+\t} value_type;\n+\tunion {\n+\t\tuint8_t *val1; /* malloced hash. */\n+\t\tstruct {\n+\t\t\tuint8_t *value; /* first value, malloced hash */\n+\t\t\tuint8_t *target_value; /* second value, malloced hash */\n+\t\t} val2;\n+\t\tchar *symref; /* referent, malloced 0-terminated string */\n+\t} value;\n+};\n+\n+/* Returns the first hash, or NULL if `rec` is not of type\n+ * REFTABLE_REF_VAL1 or REFTABLE_REF_VAL2. */\n+uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec);\n+\n+/* Returns the second hash, or NULL if `rec` is not of type\n+ * REFTABLE_REF_VAL2. */\n+uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec);\n+\n+/* returns whether 'ref' represents a deletion */\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);\n+\n+/* prints a reftable_ref_record onto stdout. Useful for debugging. */\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id);\n+\n+/* frees and nulls all pointer values inside `ref`. */\n+void reftable_ref_record_release(struct reftable_ref_record *ref);\n+\n+/* returns whether two reftable_ref_records are the same. Useful for testing. */\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size);\n+\n+/* reftable_log_record holds a reflog entry */\n+struct reftable_log_record {\n+\tchar *refname;\n+\tuint64_t update_index; /* logical timestamp of a transactional update.\n+\t\t\t\t*/\n+\n+\tenum {\n+\t\t/* tombstone to hide deletions from earlier tables */\n+\t\tREFTABLE_LOG_DELETION = 0x0,\n+\n+\t\t/* a simple update */\n+\t\tREFTABLE_LOG_UPDATE = 0x1,\n+#define REFTABLE_NR_LOG_VALUETYPES 2\n+\t} value_type;\n+\n+\tunion {\n+\t\tstruct {\n+\t\t\tuint8_t *new_hash;\n+\t\t\tuint8_t *old_hash;\n+\t\t\tchar *name;\n+\t\t\tchar *email;\n+\t\t\tuint64_t time;\n+\t\t\tint16_t tz_offset;\n+\t\t\tchar *message;\n+\t\t} update;\n+\t};\n+};\n+\n+/* returns whether 'ref' represents the deletion of a log record. */\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log);\n+\n+/* frees and nulls all pointer values. */\n+void reftable_log_record_release(struct reftable_log_record *log);\n+\n+/* returns whether two records are equal. Useful for testing. */\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size);\n+\n+/* dumps a reftable_log_record on stdout, for debugging/testing. */\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 3b58e423e7b1..09d4b83ef9b8 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,6 +4,6 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n-\n+\trecord_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"418992","messageId":"b108525009d945381bdd5b46862b257766dec79d.1615580397.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v5.git.git.1615580397.gitgitgadget@gmail.com","subject":"[PATCH v5 07/15] reftable: reading/writing blocks","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-12T20:19:48Z","receivedAt":"2021-03-12T20:20:45Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is structured as a sequence of block. Within a block,\nrecords are prefix compressed, with an index of offsets for fully expand keys to\nenable binary search within blocks.\n\nThis commit provides the logic to read and write these blocks.\n\nIncludes a code snippet copied from zlib\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 3 +\n reftable/.gitattributes | 1 +\n reftable/block.c | 440 +++++++++++++++++++++++++++++++++++++++\n reftable/block.h | 127 +++++++++++\n reftable/block_test.c | 121 +++++++++++\n reftable/zlib-compat.c | 92 ++++++++\n t/helper/test-reftable.c | 1 +\n 7 files changed, 785 insertions(+)\n create mode 100644 reftable/.gitattributes\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n create mode 100644 reftable/zlib-compat.c\n\ndiff --git a/Makefile b/Makefile\nindex e746c42f9e3b..3499996aa49e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2384,10 +2384,13 @@ XDIFF_OBJS += xdiff/xutils.o\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/zlib-compat.o\n \n+REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\ndiff --git a/reftable/.gitattributes b/reftable/.gitattributes\nnew file mode 100644\nindex 000000000000..f44451a37954\n--- /dev/null\n+++ b/reftable/.gitattributes\n@@ -0,0 +1 @@\n+/zlib-compat.c\twhitespace=-indent-with-non-tab,-trailing-space\ndiff --git a/reftable/block.c b/reftable/block.c\nnew file mode 100644\nindex 000000000000..5b4e49fb2be2\n--- /dev/null\n+++ b/reftable/block.c\n@@ -0,0 +1,440 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"reftable-error.h\"\n+#include \"system.h\"\n+#include \"zlib.h\"\n+\n+int header_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 24;\n+\tcase 2:\n+\t\treturn 28;\n+\t}\n+\tabort();\n+}\n+\n+int footer_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 68;\n+\tcase 2:\n+\t\treturn 72;\n+\t}\n+\tabort();\n+}\n+\n+static int block_writer_register_restart(struct block_writer *w, int n,\n+\t\t\t\t\t int is_restart, struct strbuf *key)\n+{\n+\tint rlen = w->restart_len;\n+\tif (rlen >= MAX_RESTARTS) {\n+\t\tis_restart = 0;\n+\t}\n+\n+\tif (is_restart) {\n+\t\trlen++;\n+\t}\n+\tif (2 + 3 * rlen + n > w->block_size - w->next)\n+\t\treturn -1;\n+\tif (is_restart) {\n+\t\tif (w->restart_len == w->restart_cap) {\n+\t\t\tw->restart_cap = w->restart_cap * 2 + 1;\n+\t\t\tw->restarts = reftable_realloc(\n+\t\t\t\tw->restarts, sizeof(uint32_t) * w->restart_cap);\n+\t\t}\n+\n+\t\tw->restarts[w->restart_len++] = w->next;\n+\t}\n+\n+\tw->next += n;\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, key);\n+\tw->entries++;\n+\treturn 0;\n+}\n+\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n+{\n+\tbw->buf = buf;\n+\tbw->hash_size = hash_size;\n+\tbw->block_size = block_size;\n+\tbw->header_off = header_off;\n+\tbw->buf[header_off] = typ;\n+\tbw->next = header_off + 4;\n+\tbw->restart_interval = 16;\n+\tbw->entries = 0;\n+\tbw->restart_len = 0;\n+\tbw->last_key.len = 0;\n+}\n+\n+uint8_t block_writer_type(struct block_writer *bw)\n+{\n+\treturn bw->buf[bw->header_off];\n+}\n+\n+/* adds the reftable_record to the block. Returns -1 if it does not fit, 0 on\n+ success */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tstruct strbuf last =\n+\t\tw->entries % w->restart_interval == 0 ? empty : w->last_key;\n+\tstruct string_view out = {\n+\t\t.buf = w->buf + w->next,\n+\t\t.len = w->block_size - w->next,\n+\t};\n+\n+\tstruct string_view start = out;\n+\n+\tint is_restart = 0;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint n = 0;\n+\n+\treftable_record_key(rec, &key);\n+\tn = reftable_encode_key(&is_restart, out, last, key,\n+\t\t\t\treftable_record_val_type(rec));\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tn = reftable_record_encode(rec, out, w->hash_size);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tif (block_writer_register_restart(w, start.len - out.len, is_restart,\n+\t\t\t\t\t &key) < 0)\n+\t\tgoto done;\n+\n+\tstrbuf_release(&key);\n+\treturn 0;\n+\n+done:\n+\tstrbuf_release(&key);\n+\treturn -1;\n+}\n+\n+int block_writer_finish(struct block_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->restart_len; i++) {\n+\t\tput_be24(w->buf + w->next, w->restarts[i]);\n+\t\tw->next += 3;\n+\t}\n+\n+\tput_be16(w->buf + w->next, w->restart_len);\n+\tw->next += 2;\n+\tput_be24(w->buf + 1 + w->header_off, w->next);\n+\n+\tif (block_writer_type(w) == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + w->header_off;\n+\t\tuint8_t *compressed = NULL;\n+\t\tint zresult = 0;\n+\t\tuLongf src_len = w->next - block_header_skip;\n+\t\tsize_t dest_cap = src_len;\n+\n+\t\tcompressed = reftable_malloc(dest_cap);\n+\t\twhile (1) {\n+\t\t\tuLongf out_dest_len = dest_cap;\n+\n+\t\t\tzresult = compress2(compressed, &out_dest_len,\n+\t\t\t\t\t w->buf + block_header_skip, src_len,\n+\t\t\t\t\t 9);\n+\t\t\tif (zresult == Z_BUF_ERROR) {\n+\t\t\t\tdest_cap *= 2;\n+\t\t\t\tcompressed =\n+\t\t\t\t\treftable_realloc(compressed, dest_cap);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (Z_OK != zresult) {\n+\t\t\t\treftable_free(compressed);\n+\t\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t\t}\n+\n+\t\t\tmemcpy(w->buf + block_header_skip, compressed,\n+\t\t\t out_dest_len);\n+\t\t\tw->next = out_dest_len + block_header_skip;\n+\t\t\treftable_free(compressed);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\treturn w->next;\n+}\n+\n+uint8_t block_reader_type(struct block_reader *r)\n+{\n+\treturn r->block.data[r->header_off];\n+}\n+\n+int block_reader_init(struct block_reader *br, struct reftable_block *block,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size)\n+{\n+\tuint32_t full_block_size = table_block_size;\n+\tuint8_t typ = block->data[header_off];\n+\tuint32_t sz = get_be24(block->data + header_off + 1);\n+\n+\tuint16_t restart_count = 0;\n+\tuint32_t restart_start = 0;\n+\tuint8_t *restart_bytes = NULL;\n+\n+\tif (!reftable_is_block_type(typ))\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tif (typ == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + header_off;\n+\t\tuLongf dst_len = sz - block_header_skip; /* total size of dest\n+\t\t\t\t\t\t\t buffer. */\n+\t\tuLongf src_len = block->len - block_header_skip;\n+\t\t/* Log blocks specify the *uncompressed* size in their header.\n+\t\t */\n+\t\tuint8_t *uncompressed = reftable_malloc(sz);\n+\n+\t\t/* Copy over the block header verbatim. It's not compressed. */\n+\t\tmemcpy(uncompressed, block->data, block_header_skip);\n+\n+\t\t/* Uncompress */\n+\t\tif (Z_OK != uncompress_return_consumed(\n+\t\t\t\t uncompressed + block_header_skip, &dst_len,\n+\t\t\t\t block->data + block_header_skip,\n+\t\t\t\t &src_len)) {\n+\t\t\treftable_free(uncompressed);\n+\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t}\n+\n+\t\tif (dst_len + block_header_skip != sz)\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\t\t/* We're done with the input data. */\n+\t\treftable_block_done(block);\n+\t\tblock->data = uncompressed;\n+\t\tblock->len = sz;\n+\t\tblock->source = malloc_block_source();\n+\t\tfull_block_size = src_len + block_header_skip;\n+\t} else if (full_block_size == 0) {\n+\t\tfull_block_size = sz;\n+\t} else if (sz < full_block_size && sz < block->len &&\n+\t\t block->data[sz] != 0) {\n+\t\t/* If the block is smaller than the full block size, it is\n+\t\t padded (data followed by '\\0') or the next block is\n+\t\t unaligned. */\n+\t\tfull_block_size = sz;\n+\t}\n+\n+\trestart_count = get_be16(block->data + sz - 2);\n+\trestart_start = sz - 2 - 3 * restart_count;\n+\trestart_bytes = block->data + restart_start;\n+\n+\t/* transfer ownership. */\n+\tbr->block = *block;\n+\tblock->data = NULL;\n+\tblock->len = 0;\n+\n+\tbr->hash_size = hash_size;\n+\tbr->block_len = restart_start;\n+\tbr->full_block_size = full_block_size;\n+\tbr->header_off = header_off;\n+\tbr->restart_count = restart_count;\n+\tbr->restart_bytes = restart_bytes;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t block_reader_restart_offset(struct block_reader *br, int i)\n+{\n+\treturn get_be24(br->restart_bytes + 3 * i);\n+}\n+\n+void block_reader_start(struct block_reader *br, struct block_iter *it)\n+{\n+\tit->br = br;\n+\tstrbuf_reset(&it->last_key);\n+\tit->next_off = br->header_off + 4;\n+}\n+\n+struct restart_find_args {\n+\tint error;\n+\tstruct strbuf key;\n+\tstruct block_reader *r;\n+};\n+\n+static int restart_key_less(size_t idx, void *args)\n+{\n+\tstruct restart_find_args *a = (struct restart_find_args *)args;\n+\tuint32_t off = block_reader_restart_offset(a->r, idx);\n+\tstruct string_view in = {\n+\t\t.buf = a->r->block.data + off,\n+\t\t.len = a->r->block_len - off,\n+\t};\n+\n+\t/* the restart key is verbatim in the block, so this could avoid the\n+\t alloc for decoding the key */\n+\tstruct strbuf rkey = STRBUF_INIT;\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tuint8_t unused_extra;\n+\tint n = reftable_decode_key(&rkey, &unused_extra, last_key, in);\n+\tint result;\n+\tif (n < 0) {\n+\t\ta->error = 1;\n+\t\treturn -1;\n+\t}\n+\n+\tresult = strbuf_cmp(&a->key, &rkey);\n+\tstrbuf_release(&rkey);\n+\treturn result;\n+}\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src)\n+{\n+\tdest->br = src->br;\n+\tdest->next_off = src->next_off;\n+\tstrbuf_reset(&dest->last_key);\n+\tstrbuf_addbuf(&dest->last_key, &src->last_key);\n+}\n+\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec)\n+{\n+\tstruct string_view in = {\n+\t\t.buf = it->br->block.data + it->next_off,\n+\t\t.len = it->br->block_len - it->next_off,\n+\t};\n+\tstruct string_view start = in;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tuint8_t extra = 0;\n+\tint n = 0;\n+\n+\tif (it->next_off >= it->br->block_len)\n+\t\treturn 1;\n+\n+\tn = reftable_decode_key(&key, &extra, it->last_key, in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\n+\tstring_view_consume(&in, n);\n+\tn = reftable_record_decode(rec, key, extra, in, it->br->hash_size);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tstrbuf_reset(&it->last_key);\n+\tstrbuf_addbuf(&it->last_key, &key);\n+\tit->next_off += start.len - in.len;\n+\tstrbuf_release(&key);\n+\treturn 0;\n+}\n+\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tint off = br->header_off + 4;\n+\tstruct string_view in = {\n+\t\t.buf = br->block.data + off,\n+\t\t.len = br->block_len - off,\n+\t};\n+\n+\tuint8_t extra = 0;\n+\tint n = reftable_decode_key(key, &extra, empty, in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\treturn 0;\n+}\n+\n+int block_iter_seek(struct block_iter *it, struct strbuf *want)\n+{\n+\treturn block_reader_seek(it->br, it, want);\n+}\n+\n+void block_iter_close(struct block_iter *it)\n+{\n+\tstrbuf_release(&it->last_key);\n+}\n+\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want)\n+{\n+\tstruct restart_find_args args = {\n+\t\t.key = *want,\n+\t\t.r = br,\n+\t};\n+\tstruct reftable_record rec = reftable_new_record(block_reader_type(br));\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = 0;\n+\tstruct block_iter next = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\n+\tint i = binsearch(br->restart_count, &restart_key_less, &args);\n+\tif (args.error) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tit->br = br;\n+\tif (i > 0) {\n+\t\ti--;\n+\t\tit->next_off = block_reader_restart_offset(br, i);\n+\t} else {\n+\t\tit->next_off = br->header_off + 4;\n+\t}\n+\n+\t/* We're looking for the last entry less/equal than the wanted key, so\n+\t we have to go one entry too far and then back up.\n+\t*/\n+\twhile (1) {\n+\t\tblock_iter_copy_from(&next, it);\n+\t\terr = block_iter_next(&next, &rec);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tif (err > 0 || strbuf_cmp(&key, want) >= 0) {\n+\t\t\terr = 0;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tblock_iter_copy_from(it, &next);\n+\t}\n+\n+done:\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&next.last_key);\n+\treftable_record_destroy(&rec);\n+\n+\treturn err;\n+}\n+\n+void block_writer_release(struct block_writer *bw)\n+{\n+\tFREE_AND_NULL(bw->restarts);\n+\tstrbuf_release(&bw->last_key);\n+\t/* the block is not owned. */\n+}\n+\n+void reftable_block_done(struct reftable_block *blockp)\n+{\n+\tstruct reftable_block_source source = blockp->source;\n+\tif (blockp != NULL && source.ops != NULL)\n+\t\tsource.ops->return_block(source.arg, blockp);\n+\tblockp->data = NULL;\n+\tblockp->len = 0;\n+\tblockp->source.ops = NULL;\n+\tblockp->source.arg = NULL;\n+}\ndiff --git a/reftable/block.h b/reftable/block.h\nnew file mode 100644\nindex 000000000000..e207706a6448\n--- /dev/null\n+++ b/reftable/block.h\n@@ -0,0 +1,127 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCK_H\n+#define BLOCK_H\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"reftable-blocksource.h\"\n+\n+/*\n+ * Writes reftable blocks. The block_writer is reused across blocks to minimize\n+ * allocation overhead.\n+ */\n+struct block_writer {\n+\tuint8_t *buf;\n+\tuint32_t block_size;\n+\n+\t/* Offset ofof the global header. Nonzero in the first block only. */\n+\tuint32_t header_off;\n+\n+\t/* How often to restart keys. */\n+\tint restart_interval;\n+\tint hash_size;\n+\n+\t/* Offset of next uint8_t to write. */\n+\tuint32_t next;\n+\tuint32_t *restarts;\n+\tuint32_t restart_len;\n+\tuint32_t restart_cap;\n+\n+\tstruct strbuf last_key;\n+\tint entries;\n+};\n+\n+/*\n+ * initializes the blockwriter to write `typ` entries, using `buf` as temporary\n+ * storage. `buf` is not owned by the block_writer. */\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n+\n+/* returns the block type (eg. 'r' for ref records. */\n+uint8_t block_writer_type(struct block_writer *bw);\n+\n+/* appends the record, or -1 if it doesn't fit. */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec);\n+\n+/* appends the key restarts, and compress the block if necessary. */\n+int block_writer_finish(struct block_writer *w);\n+\n+/* clears out internally allocated block_writer members. */\n+void block_writer_release(struct block_writer *bw);\n+\n+/* Read a block. */\n+struct block_reader {\n+\t/* offset of the block header; nonzero for the first block in a\n+\t * reftable. */\n+\tuint32_t header_off;\n+\n+\t/* the memory block */\n+\tstruct reftable_block block;\n+\tint hash_size;\n+\n+\t/* size of the data, excluding restart data. */\n+\tuint32_t block_len;\n+\tuint8_t *restart_bytes;\n+\tuint16_t restart_count;\n+\n+\t/* size of the data in the file. For log blocks, this is the compressed\n+\t * size. */\n+\tuint32_t full_block_size;\n+};\n+\n+/* Iterate over entries in a block */\n+struct block_iter {\n+\t/* offset within the block of the next entry to read. */\n+\tuint32_t next_off;\n+\tstruct block_reader *br;\n+\n+\t/* key for last entry we read. */\n+\tstruct strbuf last_key;\n+};\n+\n+/* initializes a block reader. */\n+int block_reader_init(struct block_reader *br, struct reftable_block *bl,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size);\n+\n+/* Position `it` at start of the block */\n+void block_reader_start(struct block_reader *br, struct block_iter *it);\n+\n+/* Position `it` to the `want` key in the block */\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want);\n+\n+/* Returns the block type (eg. 'r' for refs) */\n+uint8_t block_reader_type(struct block_reader *r);\n+\n+/* Decodes the first key in the block */\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key);\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src);\n+\n+/* return < 0 for error, 0 for OK, > 0 for EOF. */\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec);\n+\n+/* Seek to `want` with in the block pointed to by `it` */\n+int block_iter_seek(struct block_iter *it, struct strbuf *want);\n+\n+/* deallocate memory for `it`. The block reader and its block is left intact. */\n+void block_iter_close(struct block_iter *it);\n+\n+/* size of file header, depending on format version */\n+int header_size(int version);\n+\n+/* size of file footer, depending on format version */\n+int footer_size(int version);\n+\n+/* returns a block to its source. */\n+void reftable_block_done(struct reftable_block *ret);\n+\n+#endif\ndiff --git a/reftable/block_test.c b/reftable/block_test.c\nnew file mode 100644\nindex 000000000000..75fe198e6773\n--- /dev/null\n+++ b/reftable/block_test.c\n@@ -0,0 +1,121 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"system.h\"\n+\n+#include \"blocksource.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_block_read_write(void)\n+{\n+\tconst int header_off = 21; /* random */\n+\tchar *names[30];\n+\tconst int N = ARRAY_SIZE(names);\n+\tconst int block_size = 1024;\n+\tstruct reftable_block block = { NULL };\n+\tstruct block_writer bw = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_record rec = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tstruct block_reader br = { 0 };\n+\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\tint j = 0;\n+\tstruct strbuf want = STRBUF_INIT;\n+\n+\tblock.data = reftable_calloc(block_size);\n+\tblock.len = block_size;\n+\tblock.source = malloc_block_source();\n+\tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n+\t\t\t header_off, hash_size(SHA1_ID));\n+\treftable_record_from_ref(&rec, &ref);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tuint8_t hash[SHA1_SIZE];\n+\t\tsnprintf(name, sizeof(name), \"branch%02d\", i);\n+\t\tmemset(hash, i, sizeof(hash));\n+\n+\t\tref.refname = name;\n+\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\tref.value.val1 = hash;\n+\n+\t\tnames[i] = xstrdup(name);\n+\t\tn = block_writer_add(&bw, &rec);\n+\t\tref.refname = NULL;\n+\t\tref.value_type = REFTABLE_REF_DELETION;\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tn = block_writer_finish(&bw);\n+\tEXPECT(n > 0);\n+\n+\tblock_writer_release(&bw);\n+\n+\tblock_reader_init(&br, &block, header_off, block_size, SHA1_SIZE);\n+\n+\tblock_reader_start(&br, &it);\n+\n+\twhile (1) {\n+\t\tint r = block_iter_next(&it, &rec);\n+\t\tEXPECT(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tEXPECT_STREQ(names[j], ref.refname);\n+\t\tj++;\n+\t}\n+\n+\treftable_record_release(&rec);\n+\tblock_iter_close(&it);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\t\tstrbuf_reset(&want);\n+\t\tstrbuf_addstr(&want, names[i]);\n+\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tEXPECT(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tEXPECT(n == 0);\n+\n+\t\tEXPECT_STREQ(names[i], ref.refname);\n+\n+\t\twant.len--;\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tEXPECT(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tEXPECT(n == 0);\n+\t\tEXPECT_STREQ(names[10 * (i / 10)], ref.refname);\n+\n+\t\tblock_iter_close(&it);\n+\t}\n+\n+\treftable_record_release(&rec);\n+\treftable_block_done(&br.block);\n+\tstrbuf_release(&want);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+}\n+\n+int block_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_block_read_write);\n+\treturn 0;\n+}\ndiff --git a/reftable/zlib-compat.c b/reftable/zlib-compat.c\nnew file mode 100644\nindex 000000000000..3e0b0f24f1c7\n--- /dev/null\n+++ b/reftable/zlib-compat.c\n@@ -0,0 +1,92 @@\n+/* taken from zlib's uncompr.c\n+\n+ commit cacf7f1d4e3d44d871b605da3b647f07d718623f\n+ Author: Mark Adler \n+ Date: Sun Jan 15 09:18:46 2017 -0800\n+\n+ zlib 1.2.11\n+\n+*/\n+\n+/*\n+ * Copyright (C) 1995-2003, 2010, 2014, 2016 Jean-loup Gailly, Mark Adler\n+ * For conditions of distribution and use, see copyright notice in zlib.h\n+ */\n+\n+#include \"system.h\"\n+\n+/* clang-format off */\n+\n+/* ===========================================================================\n+ Decompresses the source buffer into the destination buffer. *sourceLen is\n+ the byte length of the source buffer. Upon entry, *destLen is the total size\n+ of the destination buffer, which must be large enough to hold the entire\n+ uncompressed data. (The size of the uncompressed data must have been saved\n+ previously by the compressor and transmitted to the decompressor by some\n+ mechanism outside the scope of this compression library.) Upon exit,\n+ *destLen is the size of the decompressed data and *sourceLen is the number\n+ of source bytes consumed. Upon return, source + *sourceLen points to the\n+ first unused input byte.\n+\n+ uncompress returns Z_OK if success, Z_MEM_ERROR if there was not enough\n+ memory, Z_BUF_ERROR if there was not enough room in the output buffer, or\n+ Z_DATA_ERROR if the input data was corrupted, including if the input data is\n+ an incomplete zlib stream.\n+*/\n+int ZEXPORT uncompress_return_consumed (\n+ Bytef *dest,\n+ uLongf *destLen,\n+ const Bytef *source,\n+ uLong *sourceLen) {\n+ z_stream stream;\n+ int err;\n+ const uInt max = (uInt)-1;\n+ uLong len, left;\n+ Byte buf[1]; /* for detection of incomplete stream when *destLen == 0 */\n+\n+ len = *sourceLen;\n+ if (*destLen) {\n+ left = *destLen;\n+ *destLen = 0;\n+ }\n+ else {\n+ left = 1;\n+ dest = buf;\n+ }\n+\n+ stream.next_in = (z_const Bytef *)source;\n+ stream.avail_in = 0;\n+ stream.zalloc = (alloc_func)0;\n+ stream.zfree = (free_func)0;\n+ stream.opaque = (voidpf)0;\n+\n+ err = inflateInit(&stream);\n+ if (err != Z_OK) return err;\n+\n+ stream.next_out = dest;\n+ stream.avail_out = 0;\n+\n+ do {\n+ if (stream.avail_out == 0) {\n+ stream.avail_out = left > (uLong)max ? max : (uInt)left;\n+ left -= stream.avail_out;\n+ }\n+ if (stream.avail_in == 0) {\n+ stream.avail_in = len > (uLong)max ? max : (uInt)len;\n+ len -= stream.avail_in;\n+ }\n+ err = inflate(&stream, Z_NO_FLUSH);\n+ } while (err == Z_OK);\n+\n+ *sourceLen -= len + stream.avail_in;\n+ if (dest != buf)\n+ *destLen = stream.total_out;\n+ else if (stream.total_out && err == Z_BUF_ERROR)\n+ left = 1;\n+\n+ inflateEnd(&stream);\n+ return err == Z_STREAM_END ? Z_OK :\n+ err == Z_NEED_DICT ? Z_DATA_ERROR :\n+ err == Z_BUF_ERROR && left + stream.avail_out ? Z_DATA_ERROR :\n+ err;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 09d4b83ef9b8..c9deeaf08c7a 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,6 +4,7 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n+\tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"418993","messageId":"c916629c562ae5b4c0a42a3d261baf48373a3e62.1615580397.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v5.git.git.1615580397.gitgitgadget@gmail.com","subject":"[PATCH v5 11/15] reftable: reftable file level tests","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-12T20:19:52Z","receivedAt":"2021-03-12T20:20:45Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nWith support for reading and writing files in place, we can construct files (in\nmemory) and attempt to read them back.\n\nBecause some sections of the format are optional (eg. indices, log entries), we\nhave to exercise this code using multiple sizes of input data\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable_test.c | 583 +++++++++++++++++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 3 files changed, 585 insertions(+)\n create mode 100644 reftable/reftable_test.c\n\ndiff --git a/Makefile b/Makefile\nindex aba1adc10b75..b0152d9f5f84 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2398,6 +2398,7 @@ REFTABLE_OBJS += reftable/zlib-compat.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n+REFTABLE_TEST_OBJS += reftable/reftable_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/reftable_test.c b/reftable/reftable_test.c\nnew file mode 100644\nindex 000000000000..45c84fa6ecfd\n--- /dev/null\n+++ b/reftable/reftable_test.c\n@@ -0,0 +1,583 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-stack.h\"\n+\n+static const int update_index = 5;\n+\n+static void test_buffer(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_block out = { NULL };\n+\tint n;\n+\tuint8_t in[] = \"hello\";\n+\tstrbuf_add(&buf, in, sizeof(in));\n+\tblock_source_from_strbuf(&source, &buf);\n+\tEXPECT(block_source_size(&source) == 6);\n+\tn = block_source_read_block(&source, &out, 0, sizeof(in));\n+\tEXPECT(n == sizeof(in));\n+\tEXPECT(!memcmp(in, out.data, n));\n+\treftable_block_done(&out);\n+\n+\tn = block_source_read_block(&source, &out, 1, 2);\n+\tEXPECT(n == 2);\n+\tEXPECT(!memcmp(out.data, \"el\", 2));\n+\n+\treftable_block_done(&out);\n+\tblock_source_close(&source);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void write_table(char ***names, struct strbuf *buf, int N,\n+\t\t\tint block_size, uint32_t hash_id)\n+{\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = block_size,\n+\t\t.hash_id = hash_id,\n+\t};\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, buf, &opts);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0, n;\n+\tstruct reftable_log_record log = { NULL };\n+\tconst struct reftable_stats *stats = NULL;\n+\t*names = reftable_calloc(sizeof(char *) * (N + 1));\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[SHA256_SIZE] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tref.refname = name;\n+\t\tref.update_index = update_index;\n+\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\tref.value.val1 = hash;\n+\t\t(*names)[i] = xstrdup(name);\n+\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[SHA256_SIZE] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tlog.refname = name;\n+\t\tlog.update_index = update_index;\n+\t\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog.update.new_hash = hash;\n+\t\tlog.update.message = \"message\";\n+\n+\t\tn = reftable_writer_add_log(w, &log);\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tfor (i = 0; i < stats->ref_stats.blocks; i++) {\n+\t\tint off = i * opts.block_size;\n+\t\tif (off == 0) {\n+\t\t\toff = header_size((hash_id == SHA256_ID) ? 2 : 1);\n+\t\t}\n+\t\tEXPECT(buf->buf[off] == 'r');\n+\t}\n+\n+\tEXPECT(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+}\n+\n+static void test_log_buffer_size(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 4096,\n+\t};\n+\tint err;\n+\tstruct reftable_log_record log = { .refname = \"refs/heads/master\",\n+\t\t\t\t\t .update_index = 0xa,\n+\t\t\t\t\t .value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t\t\t .update = {\n+\t\t\t\t\t\t .name = \"Han-Wen Nienhuys\",\n+\t\t\t\t\t\t .email = \"hanwen@google.com\",\n+\t\t\t\t\t\t .tz_offset = 100,\n+\t\t\t\t\t\t .time = 0x5e430672,\n+\t\t\t\t\t\t .message = \"commit: 9\\n\",\n+\t\t\t\t\t } };\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\t/* This tests buffer extension for log compression. Must use a random\n+\t hash, to ensure that the compressed part is larger than the original.\n+\t*/\n+\tuint8_t hash1[SHA1_SIZE], hash2[SHA1_SIZE];\n+\tfor (int i = 0; i < SHA1_SIZE; i++) {\n+\t\thash1[i] = (uint8_t)(rand() % 256);\n+\t\thash2[i] = (uint8_t)(rand() % 256);\n+\t}\n+\tlog.update.old_hash = hash1;\n+\tlog.update.new_hash = hash2;\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\terr = reftable_writer_add_log(w, &log);\n+\tEXPECT_ERR(err);\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_log_write_read(void)\n+{\n+\tint N = 2;\n+\tchar **names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint err;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\tint n;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tconst struct reftable_stats *stats = NULL;\n+\treftable_writer_set_limits(w, 0, N);\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[256];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tsnprintf(name, sizeof(name), \"b%02d%0*d\", i, 130, 7);\n+\t\tnames[i] = xstrdup(name);\n+\t\tref.refname = name;\n+\t\tref.update_index = i;\n+\n+\t\terr = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash1[SHA1_SIZE], hash2[SHA1_SIZE];\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tset_test_hash(hash1, i);\n+\t\tset_test_hash(hash2, i + 1);\n+\n+\t\tlog.refname = names[i];\n+\t\tlog.update_index = i;\n+\t\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog.update.old_hash = hash1;\n+\t\tlog.update.new_hash = hash2;\n+\n+\t\terr = reftable_writer_add_log(w, &log);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tEXPECT(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[N - 1]);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT_ERR(err);\n+\n+\t/* end of iteration. */\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT(0 < err);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_release(&ref);\n+\n+\terr = reftable_reader_seek_log(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\ti = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT_STREQ(names[i], log.refname);\n+\t\tEXPECT(i == log.update_index);\n+\t\ti++;\n+\t\treftable_log_record_release(&log);\n+\t}\n+\n+\tEXPECT(i == N);\n+\treftable_iterator_destroy(&it);\n+\n+\t/* cleanup. */\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_sequential(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tint err = 0;\n+\tint j = 0;\n+\n+\twrite_table(&names, &buf, N, 256, SHA1_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint r = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tEXPECT(0 == strcmp(names[j], ref.refname));\n+\t\tEXPECT(update_index == ref.update_index);\n+\n+\t\tj++;\n+\t\treftable_ref_record_release(&ref);\n+\t}\n+\tEXPECT(j == N);\n+\treftable_iterator_destroy(&it);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\n+\treader_close(&rd);\n+}\n+\n+static void test_table_write_small_table(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 1;\n+\twrite_table(&names, &buf, N, 4096, SHA1_ID);\n+\tEXPECT(buf.len < 200);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+}\n+\n+static void test_table_read_api(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i;\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, SHA1_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[0]);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_free(names);\n+\treader_close(&rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_table_read_write_seek(int index, int hash_id)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i = 0;\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct strbuf pastLast = STRBUF_INIT;\n+\tstruct reftable_ref_record ref = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, hash_id);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\tEXPECT(hash_id == reftable_reader_hash_id(&rd));\n+\n+\tif (!index) {\n+\t\trd.ref_offsets.index_offset = 0;\n+\t} else {\n+\t\tEXPECT(rd.ref_offsets.index_offset > 0);\n+\t}\n+\n+\tfor (i = 1; i < N; i++) {\n+\t\tint err = reftable_reader_seek_ref(&rd, &it, names[i]);\n+\t\tEXPECT_ERR(err);\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(0 == strcmp(names[i], ref.refname));\n+\t\tEXPECT(REFTABLE_REF_VAL1 == ref.value_type);\n+\t\tEXPECT(i == ref.value.val1[0]);\n+\n+\t\treftable_ref_record_release(&ref);\n+\t\treftable_iterator_destroy(&it);\n+\t}\n+\n+\tstrbuf_addstr(&pastLast, names[N - 1]);\n+\tstrbuf_addstr(&pastLast, \"/\");\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, pastLast.buf);\n+\tif (err == 0) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(err > 0);\n+\t} else {\n+\t\tEXPECT(err > 0);\n+\t}\n+\n+\tstrbuf_release(&pastLast);\n+\treftable_iterator_destroy(&it);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_free(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_seek_linear(void)\n+{\n+\ttest_table_read_write_seek(0, SHA1_ID);\n+}\n+\n+static void test_table_read_write_seek_linear_sha256(void)\n+{\n+\ttest_table_read_write_seek(0, SHA256_ID);\n+}\n+\n+static void test_table_read_write_seek_index(void)\n+{\n+\ttest_table_read_write_seek(1, SHA1_ID);\n+}\n+\n+static void test_table_refs_for(int indexed)\n+{\n+\tint N = 50;\n+\tchar **want_names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint want_names_len = 0;\n+\tuint8_t want_hash[SHA1_SIZE];\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tint err;\n+\tstruct reftable_reader rd;\n+\tstruct reftable_block_source source = { NULL };\n+\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint j;\n+\n+\tset_test_hash(want_hash, 4);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[SHA1_SIZE];\n+\t\tchar fill[51] = { 0 };\n+\t\tchar name[100];\n+\t\tuint8_t hash1[SHA1_SIZE];\n+\t\tuint8_t hash2[SHA1_SIZE];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\n+\t\tmemset(hash, i, sizeof(hash));\n+\t\tmemset(fill, 'x', 50);\n+\t\t/* Put the variable part in the start */\n+\t\tsnprintf(name, sizeof(name), \"br%02d%s\", i, fill);\n+\t\tname[40] = 0;\n+\t\tref.refname = name;\n+\n+\t\tset_test_hash(hash1, i / 4);\n+\t\tset_test_hash(hash2, 3 + i / 4);\n+\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\tref.value.val2.value = hash1;\n+\t\tref.value.val2.target_value = hash2;\n+\n+\t\t/* 80 bytes / entry, so 3 entries per block. Yields 17\n+\t\t */\n+\t\t/* blocks. */\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT(n == 0);\n+\n+\t\tif (!memcmp(hash1, want_hash, SHA1_SIZE) ||\n+\t\t !memcmp(hash2, want_hash, SHA1_SIZE)) {\n+\t\t\twant_names[want_names_len++] = xstrdup(name);\n+\t\t}\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\tif (!indexed) {\n+\t\trd.obj_offsets.is_present = 0;\n+\t}\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_reader_refs_for(&rd, &it, want_hash);\n+\tEXPECT_ERR(err);\n+\n+\tj = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(err >= 0);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tEXPECT(j < want_names_len);\n+\t\tEXPECT(0 == strcmp(ref.refname, want_names[j]));\n+\t\tj++;\n+\t\treftable_ref_record_release(&ref);\n+\t}\n+\tEXPECT(j == want_names_len);\n+\n+\tstrbuf_release(&buf);\n+\tfree_names(want_names);\n+\treftable_iterator_destroy(&it);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_refs_for_no_index(void)\n+{\n+\ttest_table_refs_for(0);\n+}\n+\n+static void test_table_refs_for_obj_index(void)\n+{\n+\ttest_table_refs_for(1);\n+}\n+\n+static void test_table_empty(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_ref_record rec = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT(err == REFTABLE_EMPTY_TABLE_ERROR);\n+\treftable_writer_free(w);\n+\n+\tEXPECT(buf.len == header_size(1) + footer_size(1));\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tEXPECT(err > 0);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+int reftable_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_log_write_read);\n+\tRUN_TEST(test_table_read_write_seek_linear_sha256);\n+\tRUN_TEST(test_log_buffer_size);\n+\tRUN_TEST(test_table_write_small_table);\n+\tRUN_TEST(test_buffer);\n+\tRUN_TEST(test_table_read_api);\n+\tRUN_TEST(test_table_read_write_sequential);\n+\tRUN_TEST(test_table_read_write_seek_linear);\n+\tRUN_TEST(test_table_read_write_seek_index);\n+\tRUN_TEST(test_table_refs_for_no_index);\n+\tRUN_TEST(test_table_refs_for_obj_index);\n+\tRUN_TEST(test_table_empty);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 050551fa6985..fdf925867375 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,6 +6,7 @@ int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\treftable_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"418989","messageId":"59f43252223012d91b2e0e12a4af6a8d2fafe9b6.1615580397.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v5.git.git.1615580397.gitgitgadget@gmail.com","subject":"[PATCH v5 15/15] Add \"test-tool dump-reftable\" command.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-12T20:19:56Z","receivedAt":"2021-03-12T20:20:46Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis command dumps individual tables or a stack of of tables.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/dump.c | 67 ++++++++++++++++++++--------------------\n t/helper/test-reftable.c | 5 +++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 5 files changed, 42 insertions(+), 33 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 90c204501739..224719a6c435 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2402,6 +2402,7 @@ REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/dump.o\n REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/refname_test.o\ndiff --git a/reftable/dump.c b/reftable/dump.c\nindex 00b444e8c9b8..7d620a3cf0fe 100644\n--- a/reftable/dump.c\n+++ b/reftable/dump.c\n@@ -12,18 +12,25 @@ license that can be found in the LICENSE file or at\n #include \n #include \n \n-#include \"reftable.h\"\n+#include \"reftable-blocksource.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-record.h\"\n #include \"reftable-tests.h\"\n+#include \"reftable-writer.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+#include \"reftable-stack.h\"\n \n static uint32_t hash_id;\n \n static int dump_table(const char *tablename)\n {\n-\tstruct reftable_block_source src = { 0 };\n+\tstruct reftable_block_source src = { NULL };\n \tint err = reftable_block_source_from_file(&src, tablename);\n-\tstruct reftable_iterator it = { 0 };\n-\tstruct reftable_ref_record ref = { 0 };\n-\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n \tstruct reftable_reader *r = NULL;\n \n \tif (err < 0)\n@@ -49,7 +56,7 @@ static int dump_table(const char *tablename)\n \t\treftable_ref_record_print(&ref, hash_id);\n \t}\n \treftable_iterator_destroy(&it);\n-\treftable_ref_record_clear(&ref);\n+\treftable_ref_record_release(&ref);\n \n \terr = reftable_reader_seek_log(r, &it, \"\");\n \tif (err < 0) {\n@@ -66,7 +73,7 @@ static int dump_table(const char *tablename)\n \t\treftable_log_record_print(&log, hash_id);\n \t}\n \treftable_iterator_destroy(&it);\n-\treftable_log_record_clear(&log);\n+\treftable_log_record_release(&log);\n \n \treftable_reader_free(r);\n \treturn 0;\n@@ -75,7 +82,7 @@ static int dump_table(const char *tablename)\n static int compact_stack(const char *stackdir)\n {\n \tstruct reftable_stack *stack = NULL;\n-\tstruct reftable_write_options cfg = {};\n+\tstruct reftable_write_options cfg = { 0 };\n \n \tint err = reftable_new_stack(&stack, stackdir, cfg);\n \tif (err < 0)\n@@ -94,10 +101,10 @@ static int compact_stack(const char *stackdir)\n static int dump_stack(const char *stackdir)\n {\n \tstruct reftable_stack *stack = NULL;\n-\tstruct reftable_write_options cfg = {};\n-\tstruct reftable_iterator it = { 0 };\n-\tstruct reftable_ref_record ref = { 0 };\n-\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n \tstruct reftable_merged_table *merged = NULL;\n \n \tint err = reftable_new_stack(&stack, stackdir, cfg);\n@@ -122,7 +129,7 @@ static int dump_stack(const char *stackdir)\n \t\treftable_ref_record_print(&ref, hash_id);\n \t}\n \treftable_iterator_destroy(&it);\n-\treftable_ref_record_clear(&ref);\n+\treftable_ref_record_release(&ref);\n \n \terr = reftable_merged_table_seek_log(merged, &it, \"\");\n \tif (err < 0) {\n@@ -139,7 +146,7 @@ static int dump_stack(const char *stackdir)\n \t\treftable_log_record_print(&log, hash_id);\n \t}\n \treftable_iterator_destroy(&it);\n-\treftable_log_record_clear(&log);\n+\treftable_log_record_release(&log);\n \n \treftable_stack_destroy(stack);\n \treturn 0;\n@@ -160,40 +167,34 @@ static void print_help(void)\n int reftable_dump_main(int argc, char *const *argv)\n {\n \tint err = 0;\n-\tint opt;\n \tint opt_dump_table = 0;\n \tint opt_dump_stack = 0;\n \tint opt_compact = 0;\n-\tconst char *arg = NULL;\n-\twhile ((opt = getopt(argc, argv, \"2chts\")) != -1) {\n-\t\tswitch (opt) {\n-\t\tcase '2':\n-\t\t\thash_id = 0x73323536;\n+\tconst char *arg = NULL, *argv0 = argv[0];\n+\n+\tfor (; argc > 1; argv++, argc--)\n+\t\tif (*argv[1] != '-')\n \t\t\tbreak;\n-\t\tcase 't':\n+\t\telse if (!strcmp(\"-2\", argv[1]))\n+\t\t\thash_id = 0x73323536;\n+\t\telse if (!strcmp(\"-t\", argv[1]))\n \t\t\topt_dump_table = 1;\n-\t\t\tbreak;\n-\t\tcase 's':\n+\t\telse if (!strcmp(\"-s\", argv[1]))\n \t\t\topt_dump_stack = 1;\n-\t\t\tbreak;\n-\t\tcase 'c':\n+\t\telse if (!strcmp(\"-c\", argv[1]))\n \t\t\topt_compact = 1;\n-\t\t\tbreak;\n-\t\tcase '?':\n-\t\tcase 'h':\n+\t\telse if (!strcmp(\"-?\", argv[1]) || !strcmp(\"-h\", argv[1])) {\n \t\t\tprint_help();\n \t\t\treturn 2;\n-\t\t\tbreak;\n \t\t}\n-\t}\n \n-\tif (argv[optind] == NULL) {\n+\tif (argc != 2) {\n \t\tfprintf(stderr, \"need argument\\n\");\n \t\tprint_help();\n \t\treturn 2;\n \t}\n \n-\targ = argv[optind];\n+\targ = argv[1];\n \n \tif (opt_dump_table) {\n \t\terr = dump_table(arg);\n@@ -204,7 +205,7 @@ int reftable_dump_main(int argc, char *const *argv)\n \t}\n \n \tif (err < 0) {\n-\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv[0], arg,\n+\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv0, arg,\n \t\t\treftable_error_str(err));\n \t\treturn 1;\n \t}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 3b702f4855e6..c1ba131c3a48 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -13,3 +13,8 @@ int cmd__reftable(int argc, const char **argv)\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n+\n+int cmd__dump_reftable(int argc, const char **argv)\n+{\n+\treturn reftable_dump_main(argc, (char *const *)argv);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex dfa2bee9e8a2..867446efafa8 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -57,6 +57,7 @@ static struct test_cmd cmds[] = {\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"reftable\", cmd__reftable },\n+\t{ \"dump-reftable\", cmd__dump_reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex cad482a78915..256634eebaeb 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -18,6 +18,7 @@ int cmd__dump_cache_tree(int argc, const char **argv);\n int cmd__dump_fsmonitor(int argc, const char **argv);\n int cmd__dump_split_index(int argc, const char **argv);\n int cmd__dump_untracked_cache(int argc, const char **argv);\n+int cmd__dump_reftable(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n-- \ngitgitgadget\n"},{"id":"418990","messageId":"0d9477941678cc59a4200b30c81c8cf177a2d544.1615580397.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v5.git.git.1615580397.gitgitgadget@gmail.com","subject":"[PATCH v5 14/15] git-prompt: prepare for reftable refs backend","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-12T20:19:55Z","receivedAt":"2021-03-12T20:20:46Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nIn our git-prompt script we strive to use Bash builtins wherever\npossible, because fork()-ing subshells for command substitutions and\nfork()+exec()-ing Git commands are expensive on some platforms. We\neven read and parse '.git/HEAD' using Bash builtins to get the name of\nthe current branch [1]. However, the upcoming reftable refs backend\nwon't use '.git/HEAD' at all, but will write an invalid refname as\nplaceholder for backwards compatibility instead, which will break our\ngit-prompt script.\n\nUpdate the git-prompt script to recognize the placeholder '.git/HEAD'\nwritten by the reftable backend (its content is specified in the\nreftable specs), and then fall back to use 'git symbolic-ref' to get\nthe name of the current branch.\n\n[1] 3a43c4b5bd (bash prompt: use bash builtins to find out current\n branch, 2011-03-31)\n\nSigned-off-by: SZEDER Gábor \n---\n contrib/completion/git-prompt.sh | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/completion/git-prompt.sh b/contrib/completion/git-prompt.sh\nindex 4640a1535d15..2e5a5d80271d 100644\n--- a/contrib/completion/git-prompt.sh\n+++ b/contrib/completion/git-prompt.sh\n@@ -478,10 +478,15 @@ __git_ps1 ()\n \t\t\tif ! __git_eread \"$g/HEAD\" head; then\n \t\t\t\treturn $exit\n \t\t\tfi\n-\t\t\t# is it a symbolic ref?\n \t\t\tb=\"${head#ref: }\"\n \t\t\tif [ \"$head\" = \"$b\" ]; then\n \t\t\t\tdetached=yes\n+\t\t\telif [ \"$b\" = \"refs/heads/.invalid\" ]; then\n+\t\t\t\t# Reftable\n+\t\t\t\tb=\"$(git symbolic-ref HEAD 2>/dev/null)\" ||\n+\t\t\t\tdetached=yes\n+\t\t\tfi\n+\t\t\tif [ \"$detached\" = yes ]; then\n \t\t\t\tb=\"$(\n \t\t\t\tcase \"${GIT_PS1_DESCRIBE_STYLE-}\" in\n \t\t\t\t(contains)\n-- \ngitgitgadget\n\n"},{"id":"418994","messageId":"bdb19af22cc7c4f3383f03f42cb4906c3ec5c5f3.1615580397.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v5.git.git.1615580397.gitgitgadget@gmail.com","subject":"[PATCH v5 13/15] Reftable support for git-core","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-12T20:19:54Z","receivedAt":"2021-03-12T20:20:46Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nFor background, see the previous commit introducing the library.\n\nThis introduces the file refs/reftable-backend.c containing a reftable-powered\nref storage backend.\n\nIt can be activated by passing --ref-storage=reftable to \"git init\", or setting\nGIT_TEST_REFTABLE in the environment.\n\nExample use: see t/t0031-reftable.sh\n\nSigned-off-by: Han-Wen Nienhuys \nSigned-off-by: Johannes Schindelin \nHelped-by: Johannes Schindelin \nHelped-by: Junio Hamano \nCo-authored-by: Jeff King \n---\n Documentation/config/extensions.txt | 9 +\n .../technical/repository-version.txt | 7 +\n Makefile | 4 +\n builtin/clone.c | 5 +-\n builtin/init-db.c | 55 +-\n builtin/worktree.c | 27 +-\n cache.h | 8 +-\n config.mak.uname | 2 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n refs.c | 27 +-\n refs.h | 3 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1446 +++++++++++++++++\n repository.c | 2 +\n repository.h | 3 +\n setup.c | 9 +-\n t/t0031-reftable.sh | 203 +++\n t/t1409-avoid-packing-refs.sh | 6 +\n t/t1450-fsck.sh | 6 +\n t/t3210-pack-refs.sh | 6 +\n t/test-lib.sh | 5 +\n 21 files changed, 1812 insertions(+), 33 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100755 t/t0031-reftable.sh\n\ndiff --git a/Documentation/config/extensions.txt b/Documentation/config/extensions.txt\nindex 4e23d73cdcad..82c5940f1434 100644\n--- a/Documentation/config/extensions.txt\n+++ b/Documentation/config/extensions.txt\n@@ -6,3 +6,12 @@ extensions.objectFormat::\n Note that this setting should only be set by linkgit:git-init[1] or\n linkgit:git-clone[1]. Trying to change it after initialization will not\n work and will produce hard-to-diagnose issues.\n++\n+extensions.refStorage::\n+\tSpecify the ref storage mechanism to use. The acceptable values are `files` and\n+\t`reftable`. If not specified, `files` is assumed. It is an error to specify\n+\tthis key unless `core.repositoryFormatVersion` is 1.\n++\n+Note that this setting should only be set by linkgit:git-init[1] or\n+linkgit:git-clone[1]. Trying to change it after initialization will not\n+work and will produce hard-to-diagnose issues.\ndiff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\nindex 7844ef30ffde..725762358333 100644\n--- a/Documentation/technical/repository-version.txt\n+++ b/Documentation/technical/repository-version.txt\n@@ -100,3 +100,10 @@ If set, by default \"git config\" reads from both \"config\" and\n multiple working directory mode, \"config\" file is shared while\n \"config.worktree\" is per-working directory (i.e., it's in\n GIT_COMMON_DIR/worktrees//config.worktree)\n+\n+==== `refStorage`\n+\n+Specifies the file format for the ref database. Values are `files`\n+(for the traditional packed + loose ref format) and `reftable` for the\n+binary reftable format. See https://github.com/google/reftable for\n+more information.\ndiff --git a/Makefile b/Makefile\nindex a00cc17ec781..90c204501739 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -967,6 +967,7 @@ LIB_OBJS += reflog-walk.o\n LIB_OBJS += refs.o\n LIB_OBJS += refs/debug.o\n LIB_OBJS += refs/files-backend.o\n+LIB_OBJS += refs/reftable-backend.o\n LIB_OBJS += refs/iterator.o\n LIB_OBJS += refs/packed-backend.o\n LIB_OBJS += refs/ref-cache.o\n@@ -2401,8 +2402,11 @@ REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n+REFTABLE_TEST_OBJS += reftable/refname_test.o\n REFTABLE_TEST_OBJS += reftable/reftable_test.o\n+REFTABLE_TEST_OBJS += reftable/stack_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 51e844a2de0a..6eee72d1ffe5 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1139,7 +1139,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t}\n \n \tinit_db(git_dir, real_git_dir, option_template, GIT_HASH_UNKNOWN, NULL,\n-\t\tINIT_DB_QUIET);\n+\t\tdefault_ref_storage(), INIT_DB_QUIET);\n \n \tif (real_git_dir)\n \t\tgit_dir = real_git_dir;\n@@ -1277,7 +1277,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t * Now that we know what algorithm the remote side is using,\n \t\t * let's set ours to the same thing.\n \t\t */\n-\t\tinitialize_repository_version(hash_algo, 1);\n+\t\tinitialize_repository_version(hash_algo, 1,\n+\t\t\t\t\t default_ref_storage());\n \t\trepo_set_hash_algo(the_repository, hash_algo);\n \n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex 7766661a4755..c4fb62056344 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -179,12 +179,14 @@ static int needs_work_tree_config(const char *git_dir, const char *work_tree)\n \treturn 1;\n }\n \n-void initialize_repository_version(int hash_algo, int reinit)\n+void initialize_repository_version(int hash_algo, int reinit,\n+\t\t\t\t const char *ref_storage_format)\n {\n \tchar repo_version_string[10];\n \tint repo_version = GIT_REPO_VERSION;\n \n-\tif (hash_algo != GIT_HASH_SHA1)\n+\tif (hash_algo != GIT_HASH_SHA1 ||\n+\t !strcmp(ref_storage_format, \"reftable\"))\n \t\trepo_version = GIT_REPO_VERSION_READ;\n \n \t/* This forces creation of new config file */\n@@ -238,6 +240,7 @@ static int create_default_files(const char *template_path,\n \tis_bare_repository_cfg = init_is_bare_repository;\n \tif (init_shared_repository != -1)\n \t\tset_shared_repository(init_shared_repository);\n+\tthe_repository->ref_storage_format = xstrdup(fmt->ref_storage);\n \n \t/*\n \t * We would have created the above under user's umask -- under\n@@ -247,6 +250,24 @@ static int create_default_files(const char *template_path,\n \t\tadjust_shared_perm(get_git_dir());\n \t}\n \n+\t/*\n+\t * Check to see if .git/HEAD exists; this must happen before\n+\t * initializing the ref db, because we want to see if there is an\n+\t * existing HEAD.\n+\t */\n+\tpath = git_path_buf(&buf, \"HEAD\");\n+\treinit = (!access(path, R_OK) ||\n+\t\t readlink(path, junk, sizeof(junk) - 1) != -1);\n+\n+\t/*\n+\t * refs/heads is a file when using reftable. We can't reinitialize with\n+\t * a reftable because it will overwrite HEAD\n+\t */\n+\tif (reinit && (!strcmp(fmt->ref_storage, \"reftable\")) ==\n+\t\t\t is_directory(git_path_buf(&buf, \"refs/heads\"))) {\n+\t\tdie(\"cannot switch ref storage format.\");\n+\t}\n+\n \t/*\n \t * We need to create a \"refs\" dir in any case so that older\n \t * versions of git can tell that this is a repository.\n@@ -261,9 +282,6 @@ static int create_default_files(const char *template_path,\n \t * Point the HEAD symref to the initial branch with if HEAD does\n \t * not yet exist.\n \t */\n-\tpath = git_path_buf(&buf, \"HEAD\");\n-\treinit = (!access(path, R_OK)\n-\t\t || readlink(path, junk, sizeof(junk)-1) != -1);\n \tif (!reinit) {\n \t\tchar *ref;\n \n@@ -280,7 +298,7 @@ static int create_default_files(const char *template_path,\n \t\tfree(ref);\n \t}\n \n-\tinitialize_repository_version(fmt->hash_algo, 0);\n+\tinitialize_repository_version(fmt->hash_algo, 0, fmt->ref_storage);\n \n \t/* Check filemode trustability */\n \tpath = git_path_buf(&buf, \"config\");\n@@ -396,7 +414,7 @@ static void validate_hash_algorithm(struct repository_format *repo_fmt, int hash\n \n int init_db(const char *git_dir, const char *real_git_dir,\n \t const char *template_dir, int hash, const char *initial_branch,\n-\t unsigned int flags)\n+\t const char *ref_storage_format, unsigned int flags)\n {\n \tint reinit;\n \tint exist_ok = flags & INIT_DB_EXIST_OK;\n@@ -435,6 +453,7 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \t * is an attempt to reinitialize new repository with an old tool.\n \t */\n \tcheck_repository_format(&repo_fmt);\n+\trepo_fmt.ref_storage = xstrdup(ref_storage_format);\n \n \tvalidate_hash_algorithm(&repo_fmt, hash);\n \n@@ -489,6 +508,9 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \t\tgit_config_set(\"receive.denyNonFastforwards\", \"true\");\n \t}\n \n+\tif (!strcmp(ref_storage_format, \"reftable\"))\n+\t\tgit_config_set(\"extensions.refStorage\", ref_storage_format);\n+\n \tif (!(flags & INIT_DB_QUIET)) {\n \t\tint len = strlen(git_dir);\n \n@@ -562,6 +584,7 @@ static const char *const init_db_usage[] = {\n int cmd_init_db(int argc, const char **argv, const char *prefix)\n {\n \tconst char *git_dir;\n+\tconst char *ref_storage_format = default_ref_storage();\n \tconst char *real_git_dir = NULL;\n \tconst char *work_tree;\n \tconst char *template_dir = NULL;\n@@ -570,15 +593,18 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n \tconst char *initial_branch = NULL;\n \tint hash_algo = GIT_HASH_UNKNOWN;\n \tconst struct option init_db_options[] = {\n-\t\tOPT_STRING(0, \"template\", &template_dir, N_(\"template-directory\"),\n-\t\t\t\tN_(\"directory from which templates will be used\")),\n+\t\tOPT_STRING(0, \"template\", &template_dir,\n+\t\t\t N_(\"template-directory\"),\n+\t\t\t N_(\"directory from which templates will be used\")),\n \t\tOPT_SET_INT(0, \"bare\", &is_bare_repository_cfg,\n-\t\t\t\tN_(\"create a bare repository\"), 1),\n+\t\t\t N_(\"create a bare repository\"), 1),\n \t\t{ OPTION_CALLBACK, 0, \"shared\", &init_shared_repository,\n-\t\t\tN_(\"permissions\"),\n-\t\t\tN_(\"specify that the git repository is to be shared amongst several users\"),\n-\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0},\n+\t\t N_(\"permissions\"),\n+\t\t N_(\"specify that the git repository is to be shared amongst several users\"),\n+\t\t PARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0 },\n \t\tOPT_BIT('q', \"quiet\", &flags, N_(\"be quiet\"), INIT_DB_QUIET),\n+\t\tOPT_STRING(0, \"ref-storage\", &ref_storage_format, N_(\"backend\"),\n+\t\t\t N_(\"the ref storage format to use\")),\n \t\tOPT_STRING(0, \"separate-git-dir\", &real_git_dir, N_(\"gitdir\"),\n \t\t\t N_(\"separate git dir from working tree\")),\n \t\tOPT_STRING('b', \"initial-branch\", &initial_branch, N_(\"name\"),\n@@ -719,10 +745,11 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n \t}\n \n \tUNLEAK(real_git_dir);\n+\tUNLEAK(ref_storage_format);\n \tUNLEAK(git_dir);\n \tUNLEAK(work_tree);\n \n \tflags |= INIT_DB_EXIST_OK;\n \treturn init_db(git_dir, real_git_dir, template_dir, hash_algo,\n-\t\t initial_branch, flags);\n+\t\t initial_branch, ref_storage_format, flags);\n }\ndiff --git a/builtin/worktree.c b/builtin/worktree.c\nindex 1cd5c2016e3f..a5d044e116b1 100644\n--- a/builtin/worktree.c\n+++ b/builtin/worktree.c\n@@ -13,6 +13,7 @@\n #include \"utf8.h\"\n #include \"worktree.h\"\n #include \"quote.h\"\n+#include \"../refs/refs-internal.h\"\n \n static const char * const worktree_usage[] = {\n \tN_(\"git worktree add [] []\"),\n@@ -330,9 +331,29 @@ static int add_worktree(const char *path, const char *refname,\n \t * worktree.\n \t */\n \tstrbuf_reset(&sb);\n-\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n-\twrite_file(sb.buf, \"%s\", oid_to_hex(&null_oid));\n-\tstrbuf_reset(&sb);\n+\tif (get_main_ref_store(the_repository)->be == &refs_be_reftable) {\n+\t\t/* XXX this is cut & paste from reftable_init_db. */\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"%s\", \"ref: refs/heads/.invalid\\n\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs/heads\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\t} else {\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"%s\", oid_to_hex(&null_oid));\n+\t\tstrbuf_reset(&sb);\n+\t}\n+\n \tstrbuf_addf(&sb, \"%s/commondir\", sb_repo.buf);\n \twrite_file(sb.buf, \"../..\");\n \ndiff --git a/cache.h b/cache.h\nindex 6fda8091f11d..be8fa1ddd6fc 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -629,9 +629,10 @@ int path_inside_repo(const char *prefix, const char *path);\n #define INIT_DB_EXIST_OK 0x0002\n \n int init_db(const char *git_dir, const char *real_git_dir,\n-\t const char *template_dir, int hash_algo,\n-\t const char *initial_branch, unsigned int flags);\n-void initialize_repository_version(int hash_algo, int reinit);\n+\t const char *template_dir, int hash_algo, const char *initial_branch,\n+\t const char *ref_storage_format, unsigned int flags);\n+void initialize_repository_version(int hash_algo, int reinit,\n+\t\t\t\t const char *ref_storage_format);\n \n void sanitize_stdfds(void);\n int daemonize(void);\n@@ -1045,6 +1046,7 @@ struct repository_format {\n \tint is_bare;\n \tint hash_algo;\n \tchar *work_tree;\n+\tchar *ref_storage;\n \tstruct string_list unknown_extensions;\n \tstruct string_list v1_only_extensions;\n };\ndiff --git a/config.mak.uname b/config.mak.uname\nindex d204c20a6469..6cba285e8fa3 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -699,7 +699,7 @@ vcxproj:\n \t# Make .vcxproj files and add them\n \tunset QUIET_GEN QUIET_BUILT_IN; \\\n \tperl contrib/buildsystems/generate -g Vcxproj\n-\tgit add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj\n+\tgit add -f git.sln {*,*/lib,*/libreftable,t/helper/*}/*.vcxproj\n \n \t# Generate the LinkOrCopyBuiltins.targets and LinkOrCopyRemoteHttp.targets file\n \t(echo '' && \\\ndiff --git a/contrib/buildsystems/Generators/Vcxproj.pm b/contrib/buildsystems/Generators/Vcxproj.pm\nindex d2584450ba17..1a25789d2851 100644\n--- a/contrib/buildsystems/Generators/Vcxproj.pm\n+++ b/contrib/buildsystems/Generators/Vcxproj.pm\n@@ -77,7 +77,7 @@ sub createProject {\n my $libs_release = \"\\n \";\n my $libs_debug = \"\\n \";\n if (!$static_library) {\n- $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n+ $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib|reftable\\/libreftable\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n $libs_debug = $libs_release;\n $libs_debug =~ s/zlib\\.lib/zlibd\\.lib/g;\n $libs_debug =~ s/libexpat\\.lib/libexpatd\\.lib/g;\n@@ -232,6 +232,7 @@ sub createProject {\n EOM\n if (!$static_library || $target =~ 'vcs-svn' || $target =~ 'xdiff') {\n my $uuid_libgit = $$build_structure{\"LIBS_libgit_GUID\"};\n+ my $uuid_libreftable = $$build_structure{\"LIBS_reftable/libreftable_GUID\"};\n my $uuid_xdiff_lib = $$build_structure{\"LIBS_xdiff/lib_GUID\"};\n \n print F << \"EOM\";\n@@ -241,6 +242,14 @@ sub createProject {\n false\n \n EOM\n+ if (!($name =~ /xdiff|libreftable/)) {\n+ print F << \"EOM\";\n+ \n+ $uuid_libreftable\n+ false\n+ \n+EOM\n+ }\n if (!($name =~ 'xdiff')) {\n print F << \"EOM\";\n \ndiff --git a/refs.c b/refs.c\nindex a665ed5e10ac..bc27a5e1b282 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -19,10 +19,16 @@\n #include \"repository.h\"\n #include \"sigchain.h\"\n \n+const char *default_ref_storage(void)\n+{\n+\tconst char *test = getenv(\"GIT_TEST_REFTABLE\");\n+\treturn test ? \"reftable\" : \"files\";\n+}\n+\n /*\n * List of all available backends\n */\n-static struct ref_storage_be *refs_backends = &refs_be_files;\n+static struct ref_storage_be *refs_backends = &refs_be_reftable;\n \n static struct ref_storage_be *find_ref_storage_backend(const char *name)\n {\n@@ -1875,13 +1881,13 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,\n * Create, record, and return a ref_store instance for the specified\n * gitdir.\n */\n-static struct ref_store *ref_store_init(const char *gitdir,\n+static struct ref_store *ref_store_init(const char *gitdir, const char *be_name,\n \t\t\t\t\tunsigned int flags)\n {\n-\tconst char *be_name = \"files\";\n-\tstruct ref_storage_be *be = find_ref_storage_backend(be_name);\n+\tstruct ref_storage_be *be;\n \tstruct ref_store *refs;\n \n+\tbe = find_ref_storage_backend(be_name);\n \tif (!be)\n \t\tBUG(\"reference backend %s is unknown\", be_name);\n \n@@ -1897,7 +1903,11 @@ struct ref_store *get_main_ref_store(struct repository *r)\n \tif (!r->gitdir)\n \t\tBUG(\"attempting to get main_ref_store outside of repository\");\n \n-\tr->refs_private = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);\n+\tr->refs_private = ref_store_init(r->gitdir,\n+\t\t\t\t\t r->ref_storage_format ?\n+\t\t\t\t\t\t r->ref_storage_format :\n+\t\t\t\t\t\t default_ref_storage(),\n+\t\t\t\t\t REF_STORE_ALL_CAPS);\n \tr->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private);\n \treturn r->refs_private;\n }\n@@ -1953,7 +1963,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \t\tgoto done;\n \n \t/* assume that add_submodule_odb() has been called */\n-\trefs = ref_store_init(submodule_sb.buf,\n+\trefs = ref_store_init(submodule_sb.buf, default_ref_storage(),\n \t\t\t REF_STORE_READ | REF_STORE_ODB);\n \tregister_ref_store_map(&submodule_ref_stores, \"submodule\",\n \t\t\t refs, submodule);\n@@ -1967,6 +1977,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \n struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n {\n+\tconst char *format = default_ref_storage();\n \tstruct ref_store *refs;\n \tconst char *id;\n \n@@ -1980,9 +1991,9 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n \n \tif (wt->id)\n \t\trefs = ref_store_init(git_common_path(\"worktrees/%s\", wt->id),\n-\t\t\t\t REF_STORE_ALL_CAPS);\n+\t\t\t\t format, REF_STORE_ALL_CAPS);\n \telse\n-\t\trefs = ref_store_init(get_git_common_dir(),\n+\t\trefs = ref_store_init(get_git_common_dir(), format,\n \t\t\t\t REF_STORE_ALL_CAPS);\n \n \tif (refs)\ndiff --git a/refs.h b/refs.h\nindex 48970dfc7e0f..5a6d4ca9fa8d 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -11,6 +11,9 @@ struct string_list;\n struct string_list_item;\n struct worktree;\n \n+/* Returns the ref storage backend to use by default. */\n+const char *default_ref_storage(void);\n+\n /*\n * Resolve a reference, recursively following symbolic refererences.\n *\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 467f4b3c936d..28166bf1f82b 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -669,6 +669,7 @@ struct ref_storage_be {\n };\n \n extern struct ref_storage_be refs_be_files;\n+extern struct ref_storage_be refs_be_reftable;\n extern struct ref_storage_be refs_be_packed;\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nnew file mode 100644\nindex 000000000000..f1f3e60a9c7c\n--- /dev/null\n+++ b/refs/reftable-backend.c\n@@ -0,0 +1,1446 @@\n+#include \"../cache.h\"\n+#include \"../chdir-notify.h\"\n+#include \"../config.h\"\n+#include \"../iterator.h\"\n+#include \"../lockfile.h\"\n+#include \"../refs.h\"\n+#include \"../reftable/reftable-stack.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-blocksource.h\"\n+#include \"../reftable/reftable-reader.h\"\n+#include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-merged.h\"\n+#include \"../reftable/reftable-generic.h\"\n+#include \"../worktree.h\"\n+#include \"refs-internal.h\"\n+\n+extern struct ref_storage_be refs_be_reftable;\n+\n+struct git_reftable_ref_store {\n+\tstruct ref_store base;\n+\tunsigned int store_flags;\n+\n+\tint err;\n+\tchar *repo_dir;\n+\n+\tchar *reftable_dir;\n+\tchar *worktree_reftable_dir;\n+\n+\tstruct reftable_stack *main_stack;\n+\tstruct reftable_stack *worktree_stack;\n+};\n+\n+static struct reftable_stack *stack_for(struct git_reftable_ref_store *store,\n+\t\t\t\t\tconst char *refname)\n+{\n+\tif (store->worktree_stack == NULL)\n+\t\treturn store->main_stack;\n+\n+\tswitch (ref_type(refname)) {\n+\tcase REF_TYPE_PER_WORKTREE:\n+\tcase REF_TYPE_PSEUDOREF:\n+\tcase REF_TYPE_OTHER_PSEUDOREF:\n+\t\treturn store->worktree_stack;\n+\tdefault:\n+\tcase REF_TYPE_MAIN_PSEUDOREF:\n+\tcase REF_TYPE_NORMAL:\n+\t\treturn store->main_stack;\n+\t}\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type);\n+\n+static void clear_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tlog->refname = NULL;\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tlog->update.old_hash = NULL;\n+\t\tlog->update.new_hash = NULL;\n+\t\tlog->update.message = NULL;\n+\t\tbreak;\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\t}\n+\treftable_log_record_release(log);\n+}\n+\n+static void fill_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tconst char *info = git_committer_info(0);\n+\tstruct ident_split split = { NULL };\n+\tint result = split_ident_line(&split, info, strlen(info));\n+\tint sign = 1;\n+\tassert(0 == result);\n+\n+\treftable_log_record_release(log);\n+\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\tlog->update.name =\n+\t\txstrndup(split.name_begin, split.name_end - split.name_begin);\n+\tlog->update.email =\n+\t\txstrndup(split.mail_begin, split.mail_end - split.mail_begin);\n+\tlog->update.time = atol(split.date_begin);\n+\tif (*split.tz_begin == '-') {\n+\t\tsign = -1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\tif (*split.tz_begin == '+') {\n+\t\tsign = 1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\n+\tlog->update.tz_offset = sign * atoi(split.tz_begin);\n+}\n+\n+static int has_suffix(struct strbuf *b, const char *suffix)\n+{\n+\tsize_t len = strlen(suffix);\n+\n+\tif (len > b->len) {\n+\t\treturn 0;\n+\t}\n+\n+\treturn 0 == strncmp(b->buf + b->len - len, suffix, len);\n+}\n+\n+/* trims the last path component of b. Returns -1 if it is not\n+ * present, or 0 on success\n+ */\n+static int trim_component(struct strbuf *b)\n+{\n+\tchar *last;\n+\tlast = strrchr(b->buf, '/');\n+\tif (!last)\n+\t\treturn -1;\n+\tstrbuf_setlen(b, last - b->buf);\n+\treturn 0;\n+}\n+\n+/* Returns whether `b` is a worktree path, trimming it to the gitdir\n+ */\n+static int is_worktree(struct strbuf *b)\n+{\n+\tif (trim_component(b) < 0) {\n+\t\treturn 0;\n+\t}\n+\tif (!has_suffix(b, \"/worktrees\")) {\n+\t\treturn 0;\n+\t}\n+\ttrim_component(b);\n+\treturn 1;\n+}\n+\n+static struct ref_store *git_reftable_ref_store_create(const char *path,\n+\t\t\t\t\t\t unsigned int store_flags)\n+{\n+\tstruct git_reftable_ref_store *refs = xcalloc(1, sizeof(*refs));\n+\tstruct ref_store *ref_store = (struct ref_store *)refs;\n+\tstruct reftable_write_options cfg = {\n+\t\t.block_size = 4096,\n+\t\t.hash_id = the_hash_algo->format_id,\n+\t};\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tconst char *gitdir = path;\n+\tstruct strbuf wt_buf = STRBUF_INIT;\n+\tint wt = 0;\n+\n+\tstrbuf_addstr(&wt_buf, path);\n+\n+\t/* this is clumsy, but the official worktree functions (eg.\n+\t * get_worktrees()) function will try to initialize a ref storage\n+\t * backend, leading to infinite recursion. */\n+\twt = is_worktree(&wt_buf);\n+\tif (wt) {\n+\t\tgitdir = wt_buf.buf;\n+\t}\n+\n+\tbase_ref_store_init(ref_store, &refs_be_reftable);\n+\tref_store->gitdir = xstrdup(gitdir);\n+\trefs->store_flags = store_flags;\n+\tstrbuf_addf(&sb, \"%s/reftable\", gitdir);\n+\trefs->reftable_dir = xstrdup(sb.buf);\n+\tstrbuf_reset(&sb);\n+\n+\trefs->err =\n+\t\treftable_new_stack(&refs->main_stack, refs->reftable_dir, cfg);\n+\tassert(refs->err != REFTABLE_API_ERROR);\n+\n+\tif (refs->err == 0 && wt) {\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", path);\n+\t\trefs->worktree_reftable_dir = xstrdup(sb.buf);\n+\n+\t\trefs->err = reftable_new_stack(&refs->worktree_stack,\n+\t\t\t\t\t refs->worktree_reftable_dir,\n+\t\t\t\t\t cfg);\n+\t\tassert(refs->err != REFTABLE_API_ERROR);\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\tstrbuf_release(&wt_buf);\n+\treturn ref_store;\n+}\n+\n+static int git_reftable_init_db(struct ref_store *ref_store, struct strbuf *err)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\tsafe_create_dir(refs->reftable_dir, 1);\n+\tassert(refs->worktree_reftable_dir == NULL);\n+\n+\tstrbuf_addf(&sb, \"%s/HEAD\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"ref: refs/heads/.invalid\");\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs\", refs->base.gitdir);\n+\tsafe_create_dir(sb.buf, 1);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs/heads\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\n+\treturn 0;\n+}\n+\n+struct git_reftable_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_ref_record ref;\n+\tstruct object_id oid;\n+\tstruct ref_store *ref_store;\n+\n+\t/* In case we must iterate over 2 stacks, this is non-null. */\n+\tstruct reftable_merged_table *merged;\n+\tunsigned int flags;\n+\tint err;\n+\tconst char *prefix;\n+};\n+\n+static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\twhile (ri->err == 0) {\n+\t\tri->err = reftable_iterator_next_ref(&ri->iter, &ri->ref);\n+\t\tif (ri->err) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (ref_type(ri->ref.refname) == REF_TYPE_PSEUDOREF) {\n+\t\t\t/*\n+\t\t\t pseudorefs, eg. HEAD, FETCH_HEAD should not be\n+\t\t\t produced, by default.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\t\tri->base.refname = ri->ref.refname;\n+\t\tif (ri->prefix != NULL &&\n+\t\t strncmp(ri->prefix, ri->ref.refname, strlen(ri->prefix))) {\n+\t\t\tri->err = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (ri->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n+\t\t ref_type(ri->base.refname) != REF_TYPE_PER_WORKTREE)\n+\t\t\tcontinue;\n+\n+\t\tri->base.flags = 0;\n+\t\tswitch (ri->ref.value_type) {\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\thashcpy(ri->oid.hash, ri->ref.value.val1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\thashcpy(ri->oid.hash, ri->ref.value.val2.value);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF: {\n+\t\t\tint out_flags = 0;\n+\t\t\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\t\t\tri->ref_store, ri->ref.refname,\n+\t\t\t\tRESOLVE_REF_READING, &ri->oid, &out_flags);\n+\t\t\tri->base.flags = out_flags;\n+\t\t\tif (resolved == NULL &&\n+\t\t\t !(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t\t (ri->base.flags & REF_ISBROKEN)) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\t}\n+\t\tdefault:\n+\t\t\tabort();\n+\t\t}\n+\n+\t\tri->base.oid = &ri->oid;\n+\t\tif (!(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t !ref_resolves_to_object(ri->base.refname, ri->base.oid,\n+\t\t\t\t\t ri->base.flags)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tbreak;\n+\t}\n+\n+\tif (ri->err > 0) {\n+\t\treturn ITER_DONE;\n+\t}\n+\tif (ri->err < 0) {\n+\t\treturn ITER_ERROR;\n+\t}\n+\n+\treturn ITER_OK;\n+}\n+\n+static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\tif (ri->ref.value_type == REFTABLE_REF_VAL2) {\n+\t\thashcpy(peeled->hash, ri->ref.value.val2.target_value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\n+\n+static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\treftable_ref_record_release(&ri->ref);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged) {\n+\t\treftable_merged_table_free(ri->merged);\n+\t}\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n+\treftable_ref_iterator_advance, reftable_ref_iterator_peel,\n+\treftable_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,\n+\t\t\t\tunsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct git_reftable_iterator *ri = xcalloc(1, sizeof(*ri));\n+\n+\tif (refs->err < 0) {\n+\t\tri->err = refs->err;\n+\t} else if (refs->worktree_stack == NULL) {\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tri->err = reftable_merged_table_seek_ref(mt, &ri->iter, prefix);\n+\t} else {\n+\t\tstruct reftable_merged_table *mt1 =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tstruct reftable_merged_table *mt2 =\n+\t\t\treftable_stack_merged_table(refs->worktree_stack);\n+\t\tstruct reftable_table *tabs =\n+\t\t\txcalloc(2, sizeof(struct reftable_table));\n+\t\treftable_table_from_merged_table(&tabs[0], mt1);\n+\t\treftable_table_from_merged_table(&tabs[1], mt2);\n+\t\tri->err = reftable_new_merged_table(&ri->merged, tabs, 2,\n+\t\t\t\t\t\t the_hash_algo->format_id);\n+\t\tif (ri->err == 0)\n+\t\t\tri->err = reftable_merged_table_seek_ref(\n+\t\t\t\tri->merged, &ri->iter, prefix);\n+\t}\n+\n+\tbase_ref_iterator_init(&ri->base, &reftable_ref_iterator_vtable, 1);\n+\tri->prefix = prefix;\n+\tri->base.oid = &ri->oid;\n+\tri->flags = flags;\n+\tri->ref_store = ref_store;\n+\treturn &ri->base;\n+}\n+\n+static int fixup_symrefs(struct ref_store *ref_store,\n+\t\t\t struct ref_transaction *transaction)\n+{\n+\tstruct strbuf referent = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\t\tstruct object_id old_oid;\n+\n+\t\terr = git_reftable_read_raw_ref(ref_store, update->refname,\n+\t\t\t\t\t\t&old_oid, &referent,\n+\t\t\t\t\t\t/* mutate input, like\n+\t\t\t\t\t\t files-backend.c */\n+\t\t\t\t\t\t&update->type);\n+\t\tif (err < 0 && errno == ENOENT &&\n+\t\t is_null_oid(&update->old_oid)) {\n+\t\t\terr = 0;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (!(update->type & REF_ISSYMREF))\n+\t\t\tcontinue;\n+\n+\t\tif (update->flags & REF_NO_DEREF) {\n+\t\t\t/* what should happen here? See files-backend.c\n+\t\t\t * lock_ref_for_update. */\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t If we are updating a symref (eg. HEAD), we should also\n+\t\t\t update the branch that the symref points to.\n+\n+\t\t\t This is generic functionality, and would be better\n+\t\t\t done in refs.c, but the current implementation is\n+\t\t\t intertwined with the locking in files-backend.c.\n+\t\t\t*/\n+\t\t\tint new_flags = update->flags;\n+\t\t\tstruct ref_update *new_update = NULL;\n+\n+\t\t\t/* if this is an update for HEAD, should also record a\n+\t\t\t log entry for HEAD? See files-backend.c,\n+\t\t\t split_head_update()\n+\t\t\t*/\n+\t\t\tnew_update = ref_transaction_add_update(\n+\t\t\t\ttransaction, referent.buf, new_flags,\n+\t\t\t\t&update->new_oid, &update->old_oid,\n+\t\t\t\tupdate->msg);\n+\t\t\tnew_update->parent_update = update;\n+\n+\t\t\t/* files-backend sets REF_LOG_ONLY here. */\n+\t\t\tupdate->flags |= REF_NO_DEREF | REF_LOG_ONLY;\n+\t\t\tupdate->flags &= ~REF_HAVE_OLD;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tstrbuf_release(&referent);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_prepare(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errbuf)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_addition *add = NULL;\n+\tstruct reftable_stack *stack =\n+\t\ttransaction->nr ?\n+\t\t\t stack_for(refs, transaction->updates[0]->refname) :\n+\t\t\t refs->main_stack;\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_new_addition(&add, stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = fixup_symrefs(ref_store, transaction);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\ttransaction->backend_data = add;\n+\ttransaction->state = REF_TRANSACTION_PREPARED;\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tif (err < 0) {\n+\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\t\tstrbuf_addf(errbuf, \"reftable: transaction prepare: %s\",\n+\t\t\t reftable_error_str(err));\n+\t}\n+\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_abort(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\treftable_addition_destroy(add);\n+\ttransaction->backend_data = NULL;\n+\treturn 0;\n+}\n+\n+static int reftable_check_old_oid(struct ref_store *refs, const char *refname,\n+\t\t\t\t struct object_id *want_oid)\n+{\n+\tstruct object_id out_oid;\n+\tint out_flags = 0;\n+\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\trefs, refname, RESOLVE_REF_READING, &out_oid, &out_flags);\n+\tif (is_null_oid(want_oid) != (resolved == NULL)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\tif (resolved != NULL && !oideq(&out_oid, want_oid)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int ref_update_cmp(const void *a, const void *b)\n+{\n+\treturn strcmp((*(struct ref_update **)a)->refname,\n+\t\t (*(struct ref_update **)b)->refname);\n+}\n+\n+static int write_transaction_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct ref_transaction *transaction = (struct ref_transaction *)arg;\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)transaction->ref_store;\n+\tstruct reftable_stack *stack =\n+\t\tstack_for(refs, transaction->updates[0]->refname);\n+\tuint64_t ts = reftable_stack_next_update_index(stack);\n+\tint err = 0;\n+\tint i = 0;\n+\tstruct reftable_log_record *logs =\n+\t\tcalloc(transaction->nr, sizeof(*logs));\n+\tstruct ref_update **sorted =\n+\t\tmalloc(transaction->nr * sizeof(struct ref_update *));\n+\tCOPY_ARRAY(sorted, transaction->updates, transaction->nr);\n+\tQSORT(sorted, transaction->nr, ref_update_cmp);\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = sorted[i];\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tfill_reftable_log_record(log);\n+\t\tlog->update_index = ts;\n+\t\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog->refname = (char *)u->refname;\n+\t\tlog->update.old_hash = u->old_oid.hash;\n+\t\tlog->update.new_hash = u->new_oid.hash;\n+\t\tlog->update.message = u->msg;\n+\n+\t\tif (u->flags & REF_LOG_ONLY) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (u->flags & REF_HAVE_NEW) {\n+\t\t\tstruct reftable_ref_record ref = { NULL };\n+\t\t\tstruct object_id peeled;\n+\n+\t\t\tint peel_error = peel_object(&u->new_oid, &peeled);\n+\t\t\tref.refname = (char *)u->refname;\n+\t\t\tref.update_index = ts;\n+\n+\t\t\tif (!peel_error) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\t\t\tref.value.val2.target_value = peeled.hash;\n+\t\t\t\tref.value.val2.value = u->new_oid.hash;\n+\t\t\t} else if (!is_null_oid(&u->new_oid)) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\t\t\tref.value.val1 = u->new_oid.hash;\n+\t\t\t}\n+\n+\t\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\terr = reftable_writer_add_log(writer, &logs[i]);\n+\t\tclear_reftable_log_record(&logs[i]);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tfree(logs);\n+\tfree(sorted);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_finish(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errmsg)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\tint err = 0;\n+\tint i;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = transaction->updates[i];\n+\t\tif (u->flags & REF_HAVE_OLD) {\n+\t\t\terr = reftable_check_old_oid(transaction->ref_store,\n+\t\t\t\t\t\t u->refname, &u->old_oid);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (transaction->nr) {\n+\t\terr = reftable_addition_add(add, &write_transaction_table,\n+\t\t\t\t\t transaction);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\terr = reftable_addition_commit(add);\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_addition_destroy(add);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\ttransaction->backend_data = NULL;\n+\tif (err) {\n+\t\tstrbuf_addf(errmsg, \"reftable: transaction failure: %s\",\n+\t\t\t reftable_error_str(err));\n+\t\treturn -1;\n+\t}\n+\treturn err;\n+}\n+\n+static int\n+git_reftable_transaction_initial_commit(struct ref_store *ref_store,\n+\t\t\t\t\tstruct ref_transaction *transaction,\n+\t\t\t\t\tstruct strbuf *errmsg)\n+{\n+\tint err = git_reftable_transaction_prepare(ref_store, transaction,\n+\t\t\t\t\t\t errmsg);\n+\tif (err)\n+\t\treturn err;\n+\n+\treturn git_reftable_transaction_finish(ref_store, transaction, errmsg);\n+}\n+\n+struct write_delete_refs_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct string_list *refnames;\n+\tconst char *logmsg;\n+\tunsigned int flags;\n+};\n+\n+static int write_delete_refs_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_delete_refs_arg *arg =\n+\t\t(struct write_delete_refs_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint err = 0;\n+\tint i = 0;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = (char *)arg->refnames->items[i].string,\n+\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_log_record log = {\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\tstruct reftable_ref_record current = { NULL };\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.update_index = ts;\n+\t\tlog.refname = (char *)arg->refnames->items[i].string;\n+\n+\t\tlog.update.message = xstrdup(arg->logmsg);\n+\t\tlog.update.new_hash = NULL;\n+\t\tlog.update.old_hash = NULL;\n+\t\tif (reftable_stack_read_ref(arg->stack, log.refname,\n+\t\t\t\t\t ¤t) == 0) {\n+\t\t\tlog.update.old_hash =\n+\t\t\t\treftable_ref_record_val1(¤t);\n+\t\t}\n+\t\terr = reftable_writer_add_log(writer, &log);\n+\t\tlog.update.old_hash = NULL;\n+\t\treftable_ref_record_release(¤t);\n+\n+\t\tclear_reftable_log_record(&log);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int git_reftable_delete_refs(struct ref_store *ref_store,\n+\t\t\t\t const char *msg,\n+\t\t\t\t struct string_list *refnames,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack =\n+\t\tstack_for(refs, refnames->items[0].string);\n+\tstruct write_delete_refs_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refnames = refnames,\n+\t\t.logmsg = msg,\n+\t\t.flags = flags,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tstring_list_sort(refnames);\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_delete_refs_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_pack_refs(struct ref_store *ref_store,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\terr = reftable_stack_compact_all(refs->main_stack, NULL);\n+\tif (err == 0 && refs->worktree_stack != NULL)\n+\t\terr = reftable_stack_compact_all(refs->worktree_stack, NULL);\n+\treturn err;\n+}\n+\n+struct write_create_symref_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+\tconst char *target;\n+\tconst char *logmsg;\n+};\n+\n+static int write_create_symref_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct write_create_symref_arg *create =\n+\t\t(struct write_create_symref_arg *)arg;\n+\tuint64_t ts = reftable_stack_next_update_index(create->stack);\n+\tint err = 0;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)create->refname,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = (char *)create->target,\n+\t\t.update_index = ts,\n+\t};\n+\treftable_writer_set_limits(writer, ts, ts);\n+\terr = reftable_writer_add_ref(writer, &ref);\n+\tif (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tstruct object_id new_oid;\n+\t\tstruct object_id old_oid;\n+\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.refname = (char *)create->refname;\n+\t\tlog.update_index = ts;\n+\t\tlog.update.message = (char *)create->logmsg;\n+\t\tif (refs_resolve_ref_unsafe(\n+\t\t\t (struct ref_store *)create->refs, create->refname,\n+\t\t\t RESOLVE_REF_READING, &old_oid, NULL) != NULL) {\n+\t\t\tlog.update.old_hash = old_oid.hash;\n+\t\t}\n+\n+\t\tif (refs_resolve_ref_unsafe((struct ref_store *)create->refs,\n+\t\t\t\t\t create->target, RESOLVE_REF_READING,\n+\t\t\t\t\t &new_oid, NULL) != NULL) {\n+\t\t\tlog.update.new_hash = new_oid.hash;\n+\t\t}\n+\n+\t\tif (log.update.old_hash != NULL ||\n+\t\t log.update.new_hash != NULL) {\n+\t\t\terr = reftable_writer_add_log(writer, &log);\n+\t\t}\n+\t\tlog.refname = NULL;\n+\t\tlog.update.message = NULL;\n+\t\tlog.update.old_hash = NULL;\n+\t\tlog.update.new_hash = NULL;\n+\t\tclear_reftable_log_record(&log);\n+\t}\n+\treturn err;\n+}\n+\n+static int git_reftable_create_symref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, const char *target,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_create_symref_arg arg = { .refs = refs,\n+\t\t\t\t\t .stack = stack,\n+\t\t\t\t\t .refname = refname,\n+\t\t\t\t\t .target = target,\n+\t\t\t\t\t .logmsg = logmsg };\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_create_symref_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct write_rename_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *oldname;\n+\tconst char *newname;\n+\tconst char *logmsg;\n+};\n+\n+static int write_rename_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &ref);\n+\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* XXX do ref renames overwrite the target? */\n+\tif (reftable_stack_read_ref(arg->stack, arg->newname, &ref) == 0) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\t{\n+\t\tstruct reftable_ref_record todo[2] = {\n+\t\t\t{\n+\t\t\t\t.refname = (char *)arg->oldname,\n+\t\t\t\t.update_index = ts,\n+\t\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t},\n+\t\t\tref,\n+\t\t};\n+\t\ttodo[1].update_index = ts;\n+\t\tfree(todo[1].refname);\n+\t\ttodo[1].refname = strdup(arg->newname);\n+\n+\t\terr = reftable_writer_add_refs(writer, todo, 2);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (reftable_ref_record_val1(&ref)) {\n+\t\tuint8_t *val1 = reftable_ref_record_val1(&ref);\n+\t\tstruct reftable_log_record todo[2] = { { NULL } };\n+\t\tfill_reftable_log_record(&todo[0]);\n+\t\tfill_reftable_log_record(&todo[1]);\n+\n+\t\ttodo[0].refname = (char *)arg->oldname;\n+\t\ttodo[0].update_index = ts;\n+\t\ttodo[0].update.message = (char *)arg->logmsg;\n+\t\ttodo[0].update.old_hash = val1;\n+\t\ttodo[0].update.new_hash = NULL;\n+\n+\t\ttodo[1].refname = (char *)arg->newname;\n+\t\ttodo[1].update_index = ts;\n+\t\ttodo[1].update.old_hash = NULL;\n+\t\ttodo[1].update.new_hash = val1;\n+\t\ttodo[1].update.message = (char *)arg->logmsg;\n+\n+\t\terr = reftable_writer_add_logs(writer, todo, 2);\n+\n+\t\tclear_reftable_log_record(&todo[0]);\n+\t\tclear_reftable_log_record(&todo[1]);\n+\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t} else {\n+\t\t/* XXX symrefs? */\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+static int git_reftable_rename_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname,\n+\t\t\t\t const char *newrefname, const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n+\tstruct write_rename_arg arg = {\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_rename_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_copy_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname, const char *newrefname,\n+\t\t\t\t const char *logmsg)\n+{\n+\tBUG(\"reftable reference store does not support copying references\");\n+}\n+\n+struct git_reftable_reflog_ref_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_log_record log;\n+\tstruct object_id oid;\n+\n+\t/* Used when iterating over worktree & main */\n+\tstruct reftable_merged_table *merged;\n+\tchar *last_name;\n+};\n+\n+static int\n+git_reftable_reflog_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&ri->iter, &ri->log);\n+\t\tif (err > 0) {\n+\t\t\treturn ITER_DONE;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn ITER_ERROR;\n+\t\t}\n+\n+\t\tri->base.refname = ri->log.refname;\n+\t\tif (ri->last_name != NULL &&\n+\t\t !strcmp(ri->log.refname, ri->last_name)) {\n+\t\t\t/* we want the refnames that we have reflogs for, so we\n+\t\t\t * skip if we've already produced this name. This could\n+\t\t\t * be faster by seeking directly to\n+\t\t\t * reflog@update_index==0.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfree(ri->last_name);\n+\t\tri->last_name = xstrdup(ri->log.refname);\n+\t\thashcpy(ri->oid.hash, ri->log.update.new_hash);\n+\t\treturn ITER_OK;\n+\t}\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tBUG(\"not supported.\");\n+\treturn -1;\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\treftable_log_record_release(&ri->log);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged)\n+\t\treftable_merged_table_free(ri->merged);\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable git_reftable_reflog_ref_iterator_vtable = {\n+\tgit_reftable_reflog_ref_iterator_advance,\n+\tgit_reftable_reflog_ref_iterator_peel,\n+\tgit_reftable_reflog_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_reflog_iterator_begin(struct ref_store *ref_store)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri = xcalloc(1, sizeof(*ri));\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\n+\tif (refs->worktree_stack == NULL) {\n+\t\tstruct reftable_stack *stack = refs->main_stack;\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(stack);\n+\t\tint err = reftable_merged_table_seek_log(mt, &ri->iter, \"\");\n+\t\tif (err < 0) {\n+\t\t\tfree(ri);\n+\t\t\t/* XXX is this allowed? */\n+\t\t\treturn NULL;\n+\t\t}\n+\t} else {\n+\t\tstruct reftable_merged_table *mt1 =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tstruct reftable_merged_table *mt2 =\n+\t\t\treftable_stack_merged_table(refs->worktree_stack);\n+\t\tstruct reftable_table *tabs =\n+\t\t\txcalloc(2, sizeof(struct reftable_table));\n+\t\tint err = 0;\n+\t\treftable_table_from_merged_table(&tabs[0], mt1);\n+\t\treftable_table_from_merged_table(&tabs[1], mt2);\n+\t\terr = reftable_new_merged_table(&ri->merged, tabs, 2,\n+\t\t\t\t\t\tthe_hash_algo->format_id);\n+\t\tif (err < 0) {\n+\t\t\tfree(tabs);\n+\t\t\t/* XXX see above */\n+\t\t\treturn NULL;\n+\t\t}\n+\t\terr = reftable_merged_table_seek_ref(ri->merged, &ri->iter, \"\");\n+\t\tif (err < 0) {\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\tbase_ref_iterator_init(&ri->base,\n+\t\t\t &git_reftable_reflog_ref_iterator_vtable, 1);\n+\tri->base.oid = &ri->oid;\n+\n+\treturn (struct ref_iterator *)ri;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_newest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (err == 0) {\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\thashcpy(old_oid.hash, log.update.old_hash);\n+\t\thashcpy(new_oid.hash, log.update.new_hash);\n+\n+\t\tfull_committer = fmt_ident(log.update.name, log.update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT,\n+\t\t\t\t\t /*date*/ NULL, IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer, log.update.time,\n+\t\t\t log.update.tz_offset, log.update.message, cb_data);\n+\t\tif (err)\n+\t\t\tbreak;\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_oldest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record *logs = NULL;\n+\tint cap = 0;\n+\tint len = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\n+\twhile (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tlogs = realloc(logs, cap * sizeof(*logs));\n+\t\t}\n+\n+\t\tlogs[len++] = log;\n+\t}\n+\n+\tfor (i = len; i--;) {\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\thashcpy(old_oid.hash, log->update.old_hash);\n+\t\thashcpy(new_oid.hash, log->update.new_hash);\n+\n+\t\tfull_committer = fmt_ident(log->update.name, log->update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT, NULL,\n+\t\t\t\t\t IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer, log->update.time,\n+\t\t\t log->update.tz_offset, log->update.message, cb_data);\n+\t\tif (err) {\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tfree(logs);\n+\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_reflog_exists(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record log = { NULL };\n+\tint err = refs->err;\n+\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (strcmp(log.refname, refname)) {\n+\t\terr = 1;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\treturn !err;\n+}\n+\n+static int git_reftable_create_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, int force_create,\n+\t\t\t\t struct strbuf *err)\n+{\n+\treturn 0;\n+}\n+\n+static int git_reftable_delete_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\treturn 0;\n+}\n+\n+struct reflog_expiry_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_log_record *tombstones;\n+\tint len;\n+\tint cap;\n+};\n+\n+static void clear_log_tombstones(struct reflog_expiry_arg *arg)\n+{\n+\tint i = 0;\n+\tfor (; i < arg->len; i++) {\n+\t\treftable_log_record_release(&arg->tombstones[i]);\n+\t}\n+\n+\tFREE_AND_NULL(arg->tombstones);\n+}\n+\n+static void add_log_tombstone(struct reflog_expiry_arg *arg,\n+\t\t\t const char *refname, uint64_t ts)\n+{\n+\tstruct reftable_log_record tombstone = {\n+\t\t.refname = xstrdup(refname),\n+\t\t.update_index = ts,\n+\t};\n+\tif (arg->len == arg->cap) {\n+\t\targ->cap = 2 * arg->cap + 1;\n+\t\targ->tombstones =\n+\t\t\trealloc(arg->tombstones, arg->cap * sizeof(tombstone));\n+\t}\n+\targ->tombstones[arg->len++] = tombstone;\n+}\n+\n+static int write_reflog_expiry_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct reflog_expiry_arg *arg = (struct reflog_expiry_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint i = 0;\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->len; i++) {\n+\t\tint err = reftable_writer_add_log(writer, &arg->tombstones[i]);\n+\t\tif (err) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int\n+git_reftable_reflog_expire(struct ref_store *ref_store, const char *refname,\n+\t\t\t const struct object_id *oid, unsigned int flags,\n+\t\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t void *policy_cb_data)\n+{\n+\t/*\n+\t For log expiry, we write tombstones in place of the expired entries,\n+\t This means that the entries are still retrievable by delving into the\n+\t stack, and expiring entries paradoxically takes extra memory.\n+\n+\t This memory is only reclaimed when some operation issues a\n+\t git_reftable_pack_refs(), which will compact the entire stack and get\n+\t rid of deletion entries.\n+\n+\t It would be better if the refs backend supported an API that sets a\n+\t criterion for all refs, passing the criterion to pack_refs().\n+\t*/\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reflog_expiry_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refs = refs,\n+\t};\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = 0;\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\twhile (1) {\n+\t\tstruct object_id ooid;\n+\t\tstruct object_id noid;\n+\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (err > 0 || strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\t\thashcpy(ooid.hash, log.update.old_hash);\n+\t\thashcpy(noid.hash, log.update.new_hash);\n+\n+\t\tif (should_prune_fn(&ooid, &noid, log.update.email,\n+\t\t\t\t (timestamp_t)log.update.time,\n+\t\t\t\t log.update.tz_offset, log.update.message,\n+\t\t\t\t policy_cb_data)) {\n+\t\t\tadd_log_tombstone(&arg, refname, log.update_index);\n+\t\t}\n+\t}\n+\terr = reftable_stack_add(stack, &write_reflog_expiry_table, &arg);\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\tclear_log_tombstones(&arg);\n+\treturn err;\n+}\n+\n+static int reftable_error_to_errno(int err)\n+{\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn EIO;\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn EFAULT;\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn ENOENT;\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn EBUSY;\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn EINVAL;\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn EDOM;\n+\tdefault:\n+\t\treturn ERANGE;\n+\t}\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\n+\t/* This is usually not needed, but Git doesn't signal to ref backend if\n+\t a subprocess updated the ref DB. So we always check.\n+\t*/\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (err > 0) {\n+\t\terrno = ENOENT;\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0) {\n+\t\terrno = reftable_error_to_errno(err);\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\n+\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (reftable_ref_record_val1(&ref) != NULL) {\n+\t\thashcpy(oid->hash, reftable_ref_record_val1(&ref));\n+\t} else {\n+\t\t*type |= REF_ISBROKEN;\n+\t\terrno = EINVAL;\n+\t\terr = -1;\n+\t}\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+struct ref_storage_be refs_be_reftable = {\n+\t&refs_be_files,\n+\t\"reftable\",\n+\tgit_reftable_ref_store_create,\n+\tgit_reftable_init_db,\n+\tgit_reftable_transaction_prepare,\n+\tgit_reftable_transaction_finish,\n+\tgit_reftable_transaction_abort,\n+\tgit_reftable_transaction_initial_commit,\n+\n+\tgit_reftable_pack_refs,\n+\tgit_reftable_create_symref,\n+\tgit_reftable_delete_refs,\n+\tgit_reftable_rename_ref,\n+\tgit_reftable_copy_ref,\n+\n+\tgit_reftable_ref_iterator_begin,\n+\tgit_reftable_read_raw_ref,\n+\n+\tgit_reftable_reflog_iterator_begin,\n+\tgit_reftable_for_each_reflog_ent_oldest_first,\n+\tgit_reftable_for_each_reflog_ent_newest_first,\n+\tgit_reftable_reflog_exists,\n+\tgit_reftable_create_reflog,\n+\tgit_reftable_delete_reflog,\n+\tgit_reftable_reflog_expire,\n+};\ndiff --git a/repository.c b/repository.c\nindex c98298acd017..4d8f572fc9cd 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -174,6 +174,8 @@ int repo_init(struct repository *repo,\n \tif (worktree)\n \t\trepo_set_worktree(repo, worktree);\n \n+\trepo->ref_storage_format = xstrdup_or_null(format.ref_storage);\n+\n \tclear_repository_format(&format);\n \treturn 0;\n \ndiff --git a/repository.h b/repository.h\nindex b385ca3c94b6..8019a7d0a1fc 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -78,6 +78,9 @@ struct repository {\n \t */\n \tstruct ref_store *refs_private;\n \n+\t/* The format to use for the ref database. */\n+\tchar *ref_storage_format;\n+\n \t/*\n \t * Contains path to often used file names.\n \t */\ndiff --git a/setup.c b/setup.c\nindex c04cd25a30df..c6b57efd0318 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -500,6 +500,9 @@ static enum extension_result handle_extension(const char *var,\n \t\t\treturn error(\"invalid value for 'extensions.objectformat'\");\n \t\tdata->hash_algo = format;\n \t\treturn EXTENSION_OK;\n+\t} else if (!strcmp(ext, \"refstorage\")) {\n+\t\tdata->ref_storage = xstrdup(value);\n+\t\treturn EXTENSION_OK;\n \t}\n \treturn EXTENSION_UNKNOWN;\n }\n@@ -651,6 +654,7 @@ void clear_repository_format(struct repository_format *format)\n \tstring_list_clear(&format->v1_only_extensions, 0);\n \tfree(format->work_tree);\n \tfree(format->partial_clone);\n+\tfree(format->ref_storage);\n \tinit_repository_format(format);\n }\n \n@@ -1308,8 +1312,11 @@ const char *setup_git_directory_gently(int *nongit_ok)\n \t\t\t\tgitdir = DEFAULT_GIT_DIR_ENVIRONMENT;\n \t\t\tsetup_git_env(gitdir);\n \t\t}\n-\t\tif (startup_info->have_repository)\n+\t\tif (startup_info->have_repository) {\n \t\t\trepo_set_hash_algo(the_repository, repo_fmt.hash_algo);\n+\t\t\tthe_repository->ref_storage_format =\n+\t\t\t\txstrdup_or_null(repo_fmt.ref_storage);\n+\t\t}\n \t}\n \n \tstrbuf_release(&dir);\ndiff --git a/t/t0031-reftable.sh b/t/t0031-reftable.sh\nnew file mode 100755\nindex 000000000000..ba0738e51561\n--- /dev/null\n+++ b/t/t0031-reftable.sh\n@@ -0,0 +1,203 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable basics'\n+\n+. ./test-lib.sh\n+\n+INVALID_SHA1=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n+\n+git_init () {\n+\tgit init -b primary \"$@\"\n+}\n+\n+initialize () {\n+\trm -rf .git &&\n+\tgit_init --ref-storage=reftable &&\n+\tmv .git/hooks .git/hooks-disabled\n+}\n+\n+test_expect_success 'SHA256 support, env' '\n+\trm -rf .git &&\n+\tGIT_DEFAULT_HASH=sha256 && export GIT_DEFAULT_HASH &&\n+\tgit_init --ref-storage=reftable &&\n+\tmv .git/hooks .git/hooks-disabled &&\n+\ttest_commit file\n+'\n+\n+test_expect_success 'SHA256 support, option' '\n+\trm -rf .git &&\n+\tgit_init --ref-storage=reftable --object-format=sha256 &&\n+\tmv .git/hooks .git/hooks-disabled &&\n+\ttest_commit file\n+'\n+\n+test_expect_success 'delete ref' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\tSHA=$(git show-ref -s --verify HEAD) &&\n+\ttest_write_lines \"$SHA refs/heads/primary\" \"$SHA refs/tags/file\" >expect &&\n+\tgit show-ref >actual &&\n+\t! git update-ref -d refs/tags/file $INVALID_SHA1 &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d refs/tags/file $SHA &&\n+\ttest_write_lines \"$SHA refs/heads/primary\" >expect &&\n+\tgit show-ref >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+\n+test_expect_success 'clone calls transaction_initial_commit' '\n+\ttest_commit message1 file1 &&\n+\tgit clone . cloned &&\n+\t(test -f cloned/file1 || echo \"Fixme.\")\n+'\n+\n+test_expect_success 'basic operation of reftable storage: commit, show-ref' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_write_lines refs/heads/primary refs/tags/file >expect &&\n+\tgit show-ref &&\n+\tgit show-ref | cut -f2 -d\" \" >actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'reflog, repack' '\n+\tinitialize &&\n+\tfor count in $(test_seq 1 10)\n+\tdo\n+\t\ttest_commit \"number $count\" file.t $count number-$count ||\n+\t\treturn 1\n+\tdone &&\n+\tgit pack-refs &&\n+\tls -1 .git/reftable >table-files &&\n+\ttest_line_count = 2 table-files &&\n+\tgit reflog refs/heads/primary >output &&\n+\ttest_line_count = 10 output &&\n+\tgrep \"commit (initial): number 1\" output &&\n+\tgrep \"commit: number 10\" output &&\n+\tgit gc &&\n+\tgit reflog refs/heads/primary >output &&\n+\ttest_line_count = 0 output\n+'\n+\n+test_expect_success 'branch switch in reflog output' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tgit checkout -b branch1 &&\n+\ttest_commit file2 &&\n+\tgit checkout -b branch2 &&\n+\tgit switch - &&\n+\tgit rev-parse --symbolic-full-name HEAD >actual &&\n+\techo refs/heads/branch1 >expect &&\n+\ttest_cmp actual expect\n+'\n+\n+\n+# This matches show-ref's output\n+print_ref() {\n+\techo \"$(git rev-parse \"$1\") $1\"\n+}\n+\n+test_expect_success 'peeled tags are stored' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\tgit tag -m \"annotated tag\" test_tag HEAD &&\n+\t{\n+\t\tprint_ref \"refs/heads/primary\" &&\n+\t\tprint_ref \"refs/tags/file\" &&\n+\t\tprint_ref \"refs/tags/test_tag\" &&\n+\t\tprint_ref \"refs/tags/test_tag^{}\"\n+\t} >expect &&\n+\tgit show-ref -d >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'show-ref works on fresh repo' '\n+\tinitialize &&\n+\trm -rf .git &&\n+\tgit_init --ref-storage=reftable &&\n+\t>expect &&\n+\t! git show-ref >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'checkout unborn branch' '\n+\tinitialize &&\n+\tgit checkout -b primary\n+'\n+\n+\n+test_expect_success 'dir/file conflict' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\t! git branch primary/forbidden\n+'\n+\n+\n+test_expect_success 'do not clobber existing repo' '\n+\trm -rf .git &&\n+\tgit_init --ref-storage=files &&\n+\tcat .git/HEAD >expect &&\n+\ttest_commit file &&\n+\t(git_init --ref-storage=reftable || true) &&\n+\tcat .git/HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# cherry-pick uses a pseudo ref.\n+test_expect_success 'pseudo refs' '\n+\tinitialize &&\n+\ttest_commit message1 file1 &&\n+\ttest_commit message2 file2 &&\n+\tgit branch source &&\n+\tgit checkout HEAD^ &&\n+\ttest_commit message3 file3 &&\n+\tgit cherry-pick source &&\n+\ttest -f file2\n+'\n+\n+# cherry-pick uses a pseudo ref.\n+test_expect_success 'rebase' '\n+\tinitialize &&\n+\ttest_commit message1 file1 &&\n+\ttest_commit message2 file2 &&\n+\tgit branch source &&\n+\tgit checkout HEAD^ &&\n+\ttest_commit message3 file3 &&\n+\tgit rebase source &&\n+\ttest -f file2\n+'\n+\n+test_expect_success 'worktrees' '\n+\tgit_init --ref-storage=reftable start &&\n+\t(cd start && test_commit file1 && git checkout -b branch1 &&\n+\tgit checkout -b branch2 &&\n+\tgit worktree add ../wt\n+\t) &&\n+\tcd wt &&\n+\tgit checkout branch1 &&\n+\tgit branch\n+'\n+\n+test_expect_success 'worktrees 2' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tmkdir existing_empty &&\n+\tgit worktree add --detach existing_empty primary\n+'\n+\n+test_expect_success 'FETCH_HEAD' '\n+\tinitialize &&\n+\ttest_commit one &&\n+\t(git_init sub && cd sub && test_commit two) &&\n+\tgit --git-dir sub/.git rev-parse HEAD >expect &&\n+\tgit fetch sub &&\n+\tgit checkout FETCH_HEAD &&\n+\tgit rev-parse HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/t1409-avoid-packing-refs.sh b/t/t1409-avoid-packing-refs.sh\nindex be12fb63506e..c6f783255633 100755\n--- a/t/t1409-avoid-packing-refs.sh\n+++ b/t/t1409-avoid-packing-refs.sh\n@@ -4,6 +4,12 @@ test_description='avoid rewriting packed-refs unnecessarily'\n \n . ./test-lib.sh\n \n+if test_have_prereq REFTABLE\n+then\n+ skip_all='skipping pack-refs tests; incompatible with reftable'\n+ test_done\n+fi\n+\n # Add an identifying mark to the packed-refs file header line. This\n # shouldn't upset readers, and it should be omitted if the file is\n # ever rewritten.\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 5071ac63a5b5..16440d58d6bb 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -8,6 +8,12 @@ test_description='git fsck random collection of tests\n \n . ./test-lib.sh\n \n+if test_have_prereq REFTABLE\n+then\n+ skip_all='skipping tests; incompatible with reftable'\n+ test_done\n+fi\n+\n test_expect_success setup '\n \tgit config gc.auto 0 &&\n \tgit config i18n.commitencoding ISO-8859-1 &&\ndiff --git a/t/t3210-pack-refs.sh b/t/t3210-pack-refs.sh\nindex 3b7cdc56ec84..667a23affe36 100755\n--- a/t/t3210-pack-refs.sh\n+++ b/t/t3210-pack-refs.sh\n@@ -14,6 +14,12 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n . ./test-lib.sh\n \n+if test_have_prereq REFTABLE\n+then\n+ skip_all='skipping pack-refs tests; incompatible with reftable'\n+ test_done\n+fi\n+\n test_expect_success 'enable reflogs' '\n \tgit config core.logallrefupdates true\n '\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex d3f6af6a6545..943aa9713961 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1481,6 +1481,11 @@ parisc* | hppa*)\n \t;;\n esac\n \n+if test -n \"$GIT_TEST_REFTABLE\"\n+then\n+ test_set_prereq REFTABLE\n+fi\n+\n ( COLUMNS=1 && test $COLUMNS = 1 ) && test_set_prereq COLUMNS_CAN_BE_1\n test -z \"$NO_PERL\" && test_set_prereq PERL\n test -z \"$NO_PTHREADS\" && test_set_prereq PTHREADS\n-- \ngitgitgadget\n\n"},{"id":"418995","messageId":"1f17d5edab5274e0ed742c4ec7b9e463d4c00538.1615580397.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v5.git.git.1615580397.gitgitgadget@gmail.com","subject":"[PATCH v5 10/15] reftable: read reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-12T20:19:51Z","receivedAt":"2021-03-12T20:20:46Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis supports reading a single reftable file.\n\nThe commit introduces an abstract iterator type, which captures the usecases\nboth of reading individual refs, and iterating over a segment of the ref\nnamespace.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 3 +\n reftable/iter.c | 248 ++++++++++++\n reftable/iter.h | 75 ++++\n reftable/reader.c | 733 +++++++++++++++++++++++++++++++++++\n reftable/reader.h | 75 ++++\n reftable/reftable-iterator.h | 37 ++\n reftable/reftable-reader.h | 89 +++++\n 7 files changed, 1260 insertions(+)\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/reftable-iterator.h\n create mode 100644 reftable/reftable-reader.h\n\ndiff --git a/Makefile b/Makefile\nindex d37dd79bfab7..aba1adc10b75 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2386,8 +2386,11 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n+REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/reftable.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n REFTABLE_OBJS += reftable/zlib-compat.o\ndiff --git a/reftable/iter.c b/reftable/iter.c\nnew file mode 100644\nindex 000000000000..b8c091ffd570\n--- /dev/null\n+++ b/reftable/iter.c\n@@ -0,0 +1,248 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"iter.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"reftable-error.h\"\n+\n+int iterator_is_null(struct reftable_iterator *it)\n+{\n+\treturn it->ops == NULL;\n+}\n+\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n+{\n+\treturn 1;\n+}\n+\n+static void empty_iterator_close(void *arg)\n+{\n+}\n+\n+static struct reftable_iterator_vtable empty_vtable = {\n+\t.next = &empty_iterator_next,\n+\t.close = &empty_iterator_close,\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = NULL;\n+\tit->ops = &empty_vtable;\n+}\n+\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n+{\n+\treturn it->ops->next(it->iter_arg, rec);\n+}\n+\n+void reftable_iterator_destroy(struct reftable_iterator *it)\n+{\n+\tif (it->ops == NULL) {\n+\t\treturn;\n+\t}\n+\tit->ops->close(it->iter_arg);\n+\tit->ops = NULL;\n+\tFREE_AND_NULL(it->iter_arg);\n+}\n+\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, ref);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, log);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+static void filtering_ref_iterator_close(void *iter_arg)\n+{\n+\tstruct filtering_ref_iterator *fri =\n+\t\t(struct filtering_ref_iterator *)iter_arg;\n+\tstrbuf_release(&fri->oid);\n+\treftable_iterator_destroy(&fri->it);\n+}\n+\n+static int filtering_ref_iterator_next(void *iter_arg,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct filtering_ref_iterator *fri =\n+\t\t(struct filtering_ref_iterator *)iter_arg;\n+\tstruct reftable_ref_record *ref =\n+\t\t(struct reftable_ref_record *)rec->data;\n+\tint err = 0;\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&fri->it, ref);\n+\t\tif (err != 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (fri->double_check) {\n+\t\t\tstruct reftable_iterator it = { NULL };\n+\n+\t\t\terr = reftable_table_seek_ref(&fri->tab, &it,\n+\t\t\t\t\t\t ref->refname);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = reftable_iterator_next_ref(&it, ref);\n+\t\t\t}\n+\n+\t\t\treftable_iterator_destroy(&it);\n+\n+\t\t\tif (err < 0) {\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (err > 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (ref->value_type == REFTABLE_REF_VAL2 &&\n+\t\t (!memcmp(fri->oid.buf, ref->value.val2.target_value,\n+\t\t\t fri->oid.len) ||\n+\t\t !memcmp(fri->oid.buf, ref->value.val2.value,\n+\t\t\t fri->oid.len)))\n+\t\t\treturn 0;\n+\n+\t\tif (ref->value_type == REFTABLE_REF_VAL1 &&\n+\t\t !memcmp(fri->oid.buf, ref->value.val1, fri->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treftable_ref_record_release(ref);\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable filtering_ref_iterator_vtable = {\n+\t.next = &filtering_ref_iterator_next,\n+\t.close = &filtering_ref_iterator_close,\n+};\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *it,\n+\t\t\t\t\t struct filtering_ref_iterator *fri)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = fri;\n+\tit->ops = &filtering_ref_iterator_vtable;\n+}\n+\n+static void indexed_table_ref_iter_close(void *p)\n+{\n+\tstruct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p;\n+\tblock_iter_close(&it->cur);\n+\treftable_block_done(&it->block_reader.block);\n+\tstrbuf_release(&it->oid);\n+}\n+\n+static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n+{\n+\tuint64_t off;\n+\tint err = 0;\n+\tif (it->offset_idx == it->offset_len) {\n+\t\tit->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\n+\treftable_block_done(&it->block_reader.block);\n+\n+\toff = it->offsets[it->offset_idx++];\n+\terr = reader_init_block_reader(it->r, &it->block_reader, off,\n+\t\t\t\t BLOCK_TYPE_REF);\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\tif (err > 0) {\n+\t\t/* indexed block does not exist. */\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\t}\n+\tblock_reader_start(&it->block_reader, &it->cur);\n+\treturn 0;\n+}\n+\n+static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\n+{\n+\tstruct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p;\n+\tstruct reftable_ref_record *ref =\n+\t\t(struct reftable_ref_record *)rec->data;\n+\n+\twhile (1) {\n+\t\tint err = block_iter_next(&it->cur, rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\terr = indexed_table_ref_iter_next_block(it);\n+\t\t\tif (err < 0) {\n+\t\t\t\treturn err;\n+\t\t\t}\n+\n+\t\t\tif (it->is_finished) {\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\t\t/* BUG */\n+\t\tif (!memcmp(it->oid.buf, ref->value.val2.target_value,\n+\t\t\t it->oid.len) ||\n+\t\t !memcmp(it->oid.buf, ref->value.val2.value, it->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+}\n+\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len)\n+{\n+\tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n+\tstruct indexed_table_ref_iter *itr =\n+\t\treftable_calloc(sizeof(struct indexed_table_ref_iter));\n+\tint err = 0;\n+\n+\t*itr = empty;\n+\titr->r = r;\n+\tstrbuf_add(&itr->oid, oid, oid_len);\n+\n+\titr->offsets = offsets;\n+\titr->offset_len = offset_len;\n+\n+\terr = indexed_table_ref_iter_next_block(itr);\n+\tif (err < 0) {\n+\t\treftable_free(itr);\n+\t} else {\n+\t\t*dest = itr;\n+\t}\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable indexed_table_ref_iter_vtable = {\n+\t.next = &indexed_table_ref_iter_next,\n+\t.close = &indexed_table_ref_iter_close,\n+};\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = itr;\n+\tit->ops = &indexed_table_ref_iter_vtable;\n+}\ndiff --git a/reftable/iter.h b/reftable/iter.h\nnew file mode 100644\nindex 000000000000..2ba964c29a3f\n--- /dev/null\n+++ b/reftable/iter.h\n@@ -0,0 +1,75 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef ITER_H\n+#define ITER_H\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"record.h\"\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+struct reftable_iterator_vtable {\n+\tint (*next)(void *iter_arg, struct reftable_record *rec);\n+\tvoid (*close)(void *iter_arg);\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it);\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec);\n+\n+/* Returns true for a zeroed out iterator, such as the one returned from\n+ * iterator_destroy. */\n+int iterator_is_null(struct reftable_iterator *it);\n+\n+/* iterator that produces only ref records that point to `oid` */\n+struct filtering_ref_iterator {\n+\tint double_check;\n+\tstruct reftable_table tab;\n+\tstruct strbuf oid;\n+\tstruct reftable_iterator it;\n+};\n+#define FILTERING_REF_ITERATOR_INIT \\\n+\t{ \\\n+\t\t.oid = STRBUF_INIT \\\n+\t}\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *,\n+\t\t\t\t\t struct filtering_ref_iterator *);\n+\n+/* iterator that produces only ref records that point to `oid`,\n+ * but using the object index.\n+ */\n+struct indexed_table_ref_iter {\n+\tstruct reftable_reader *r;\n+\tstruct strbuf oid;\n+\n+\t/* mutable */\n+\tuint64_t *offsets;\n+\n+\t/* Points to the next offset to read. */\n+\tint offset_idx;\n+\tint offset_len;\n+\tstruct block_reader block_reader;\n+\tstruct block_iter cur;\n+\tint is_finished;\n+};\n+\n+#define INDEXED_TABLE_REF_ITER_INIT \\\n+\t{ \\\n+\t\t.cur = { .last_key = STRBUF_INIT }, .oid = STRBUF_INIT, \\\n+\t}\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr);\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len);\n+\n+#endif\ndiff --git a/reftable/reader.c b/reftable/reader.c\nnew file mode 100644\nindex 000000000000..a5b7b1d4fefa\n--- /dev/null\n+++ b/reftable/reader.c\n@@ -0,0 +1,733 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reader.h\"\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"iter.h\"\n+#include \"record.h\"\n+#include \"reftable-error.h\"\n+#include \"tree.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source)\n+{\n+\treturn source->ops->size(source->arg);\n+}\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tint result = source->ops->read_block(source->arg, dest, off, size);\n+\tdest->source = *source;\n+\treturn result;\n+}\n+\n+void block_source_close(struct reftable_block_source *source)\n+{\n+\tif (source->ops == NULL) {\n+\t\treturn;\n+\t}\n+\n+\tsource->ops->close(source->arg);\n+\tsource->ops = NULL;\n+}\n+\n+static struct reftable_reader_offsets *\n+reader_offsets_for(struct reftable_reader *r, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\t\treturn &r->ref_offsets;\n+\tcase BLOCK_TYPE_LOG:\n+\t\treturn &r->log_offsets;\n+\tcase BLOCK_TYPE_OBJ:\n+\t\treturn &r->obj_offsets;\n+\t}\n+\tabort();\n+}\n+\n+static int reader_get_block(struct reftable_reader *r,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t sz)\n+{\n+\tif (off >= r->size)\n+\t\treturn 0;\n+\n+\tif (off + sz > r->size) {\n+\t\tsz = r->size - off;\n+\t}\n+\n+\treturn block_source_read_block(&r->source, dest, off, sz);\n+}\n+\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r)\n+{\n+\treturn r->hash_id;\n+}\n+\n+const char *reader_name(struct reftable_reader *r)\n+{\n+\treturn r->name;\n+}\n+\n+static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n+\t\t\tuint8_t *header)\n+{\n+\tuint8_t *f = footer;\n+\tuint8_t first_block_typ;\n+\tint err = 0;\n+\tuint32_t computed_crc;\n+\tuint32_t file_crc;\n+\n+\tif (memcmp(f, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tf += 4;\n+\n+\tif (memcmp(footer, header, header_size(r->version))) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tf++;\n+\tr->block_size = get_be24(f);\n+\n+\tf += 3;\n+\tr->min_update_index = get_be64(f);\n+\tf += 8;\n+\tr->max_update_index = get_be64(f);\n+\tf += 8;\n+\n+\tif (r->version == 1) {\n+\t\tr->hash_id = SHA1_ID;\n+\t} else {\n+\t\tr->hash_id = get_be32(f);\n+\t\tswitch (r->hash_id) {\n+\t\tcase SHA1_ID:\n+\t\t\tbreak;\n+\t\tcase SHA256_ID:\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tf += 4;\n+\t}\n+\n+\tr->ref_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->obj_offsets.offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1);\n+\tr->obj_offsets.offset >>= 5;\n+\n+\tr->obj_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tcomputed_crc = crc32(0, footer, f - footer);\n+\tfile_crc = get_be32(f);\n+\tf += 4;\n+\tif (computed_crc != file_crc) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tfirst_block_typ = header[header_size(r->version)];\n+\tr->ref_offsets.is_present = (first_block_typ == BLOCK_TYPE_REF);\n+\tr->ref_offsets.offset = 0;\n+\tr->log_offsets.is_present = (first_block_typ == BLOCK_TYPE_LOG ||\n+\t\t\t\t r->log_offsets.offset > 0);\n+\tr->obj_offsets.is_present = r->obj_offsets.offset > 0;\n+\terr = 0;\n+done:\n+\treturn err;\n+}\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name)\n+{\n+\tstruct reftable_block footer = { NULL };\n+\tstruct reftable_block header = { NULL };\n+\tint err = 0;\n+\n+\tmemset(r, 0, sizeof(struct reftable_reader));\n+\n+\t/* Need +1 to read type of first block. */\n+\terr = block_source_read_block(source, &header, 0, header_size(2) + 1);\n+\tif (err != header_size(2) + 1) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tif (memcmp(header.data, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tr->version = header.data[4];\n+\tif (r->version != 1 && r->version != 2) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tr->size = block_source_size(source) - footer_size(r->version);\n+\tr->source = *source;\n+\tr->name = xstrdup(name);\n+\tr->hash_id = 0;\n+\n+\terr = block_source_read_block(source, &footer, r->size,\n+\t\t\t\t footer_size(r->version));\n+\tif (err != footer_size(r->version)) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = parse_footer(r, footer.data, header.data);\n+done:\n+\treftable_block_done(&footer);\n+\treftable_block_done(&header);\n+\treturn err;\n+}\n+\n+struct table_iter {\n+\tstruct reftable_reader *r;\n+\tuint8_t typ;\n+\tuint64_t block_off;\n+\tstruct block_iter bi;\n+\tint is_finished;\n+};\n+#define TABLE_ITER_INIT \\\n+\t{ \\\n+\t\t.bi = {.last_key = STRBUF_INIT } \\\n+\t}\n+\n+static void table_iter_copy_from(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = src->block_off;\n+\tdest->is_finished = src->is_finished;\n+\tblock_iter_copy_from(&dest->bi, &src->bi);\n+}\n+\n+static int table_iter_next_in_block(struct table_iter *ti,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tint res = block_iter_next(&ti->bi, rec);\n+\tif (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {\n+\t\t((struct reftable_ref_record *)rec->data)->update_index +=\n+\t\t\tti->r->min_update_index;\n+\t}\n+\n+\treturn res;\n+}\n+\n+static void table_iter_block_done(struct table_iter *ti)\n+{\n+\tif (ti->bi.br == NULL) {\n+\t\treturn;\n+\t}\n+\treftable_block_done(&ti->bi.br->block);\n+\tFREE_AND_NULL(ti->bi.br);\n+\n+\tti->bi.last_key.len = 0;\n+\tti->bi.next_off = 0;\n+}\n+\n+static int32_t extract_block_size(uint8_t *data, uint8_t *typ, uint64_t off,\n+\t\t\t\t int version)\n+{\n+\tint32_t result = 0;\n+\n+\tif (off == 0) {\n+\t\tdata += header_size(version);\n+\t}\n+\n+\t*typ = data[0];\n+\tif (reftable_is_block_type(*typ)) {\n+\t\tresult = get_be24(data + 1);\n+\t}\n+\treturn result;\n+}\n+\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ)\n+{\n+\tint32_t guess_block_size = r->block_size ? r->block_size :\n+\t\t\t\t\t\t\t DEFAULT_BLOCK_SIZE;\n+\tstruct reftable_block block = { NULL };\n+\tuint8_t block_typ = 0;\n+\tint err = 0;\n+\tuint32_t header_off = next_off ? 0 : header_size(r->version);\n+\tint32_t block_size = 0;\n+\n+\tif (next_off >= r->size)\n+\t\treturn 1;\n+\n+\terr = reader_get_block(r, &block, next_off, guess_block_size);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tblock_size = extract_block_size(block.data, &block_typ, next_off,\n+\t\t\t\t\tr->version);\n+\tif (block_size < 0)\n+\t\treturn block_size;\n+\n+\tif (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {\n+\t\treftable_block_done(&block);\n+\t\treturn 1;\n+\t}\n+\n+\tif (block_size > guess_block_size) {\n+\t\treftable_block_done(&block);\n+\t\terr = reader_get_block(r, &block, next_off, block_size);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\treturn block_reader_init(br, &block, header_off, r->block_size,\n+\t\t\t\t hash_size(r->hash_id));\n+}\n+\n+static int table_iter_next_block(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tuint64_t next_block_off = src->block_off + src->bi.br->full_block_size;\n+\tstruct block_reader br = { 0 };\n+\tint err = 0;\n+\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = next_block_off;\n+\n+\terr = reader_init_block_reader(src->r, &br, next_block_off, src->typ);\n+\tif (err > 0) {\n+\t\tdest->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\tif (err != 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct block_reader *brp =\n+\t\t\treftable_malloc(sizeof(struct block_reader));\n+\t\t*brp = br;\n+\n+\t\tdest->is_finished = 0;\n+\t\tblock_reader_start(brp, &dest->bi);\n+\t}\n+\treturn 0;\n+}\n+\n+static int table_iter_next(struct table_iter *ti, struct reftable_record *rec)\n+{\n+\tif (reftable_record_type(rec) != ti->typ)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\twhile (1) {\n+\t\tstruct table_iter next = TABLE_ITER_INIT;\n+\t\tint err = 0;\n+\t\tif (ti->is_finished) {\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\terr = table_iter_next_in_block(ti, rec);\n+\t\tif (err <= 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err != 0) {\n+\t\t\tti->is_finished = 1;\n+\t\t}\n+\t\ttable_iter_block_done(ti);\n+\t\tif (err != 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\ttable_iter_copy_from(ti, &next);\n+\t\tblock_iter_close(&next.bi);\n+\t}\n+}\n+\n+static int table_iter_next_void(void *ti, struct reftable_record *rec)\n+{\n+\treturn table_iter_next((struct table_iter *)ti, rec);\n+}\n+\n+static void table_iter_close(void *p)\n+{\n+\tstruct table_iter *ti = (struct table_iter *)p;\n+\ttable_iter_block_done(ti);\n+\tblock_iter_close(&ti->bi);\n+}\n+\n+static struct reftable_iterator_vtable table_iter_vtable = {\n+\t.next = &table_iter_next_void,\n+\t.close = &table_iter_close,\n+};\n+\n+static void iterator_from_table_iter(struct reftable_iterator *it,\n+\t\t\t\t struct table_iter *ti)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = ti;\n+\tit->ops = &table_iter_vtable;\n+}\n+\n+static int reader_table_iter_at(struct reftable_reader *r,\n+\t\t\t\tstruct table_iter *ti, uint64_t off,\n+\t\t\t\tuint8_t typ)\n+{\n+\tstruct block_reader br = { 0 };\n+\tstruct block_reader *brp = NULL;\n+\n+\tint err = reader_init_block_reader(r, &br, off, typ);\n+\tif (err != 0)\n+\t\treturn err;\n+\n+\tbrp = reftable_malloc(sizeof(struct block_reader));\n+\t*brp = br;\n+\tti->r = r;\n+\tti->typ = block_reader_type(brp);\n+\tti->block_off = off;\n+\tblock_reader_start(brp, &ti->bi);\n+\treturn 0;\n+}\n+\n+static int reader_start(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\tuint8_t typ, int index)\n+{\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tuint64_t off = offs->offset;\n+\tif (index) {\n+\t\toff = offs->index_offset;\n+\t\tif (off == 0) {\n+\t\t\treturn 1;\n+\t\t}\n+\t\ttyp = BLOCK_TYPE_INDEX;\n+\t}\n+\n+\treturn reader_table_iter_at(r, ti, off, typ);\n+}\n+\n+static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\t struct reftable_record *want)\n+{\n+\tstruct reftable_record rec =\n+\t\treftable_new_record(reftable_record_type(want));\n+\tstruct strbuf want_key = STRBUF_INIT;\n+\tstruct strbuf got_key = STRBUF_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = -1;\n+\n+\treftable_record_key(want, &want_key);\n+\n+\twhile (1) {\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = block_reader_first_key(next.bi.br, &got_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (strbuf_cmp(&got_key, &want_key) > 0) {\n+\t\t\ttable_iter_block_done(&next);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_block_done(ti);\n+\t\ttable_iter_copy_from(ti, &next);\n+\t}\n+\n+\terr = block_iter_seek(&ti->bi, &want_key);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = 0;\n+\n+done:\n+\tblock_iter_close(&next.bi);\n+\treftable_record_destroy(&rec);\n+\tstrbuf_release(&want_key);\n+\tstrbuf_release(&got_key);\n+\treturn err;\n+}\n+\n+static int reader_seek_indexed(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_index_record want_index = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record want_index_rec = { NULL };\n+\tstruct reftable_index_record index_result = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record index_result_rec = { NULL };\n+\tstruct table_iter index_iter = TABLE_ITER_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = 0;\n+\n+\treftable_record_key(rec, &want_index.last_key);\n+\treftable_record_from_index(&want_index_rec, &want_index);\n+\treftable_record_from_index(&index_result_rec, &index_result);\n+\n+\terr = reader_start(r, &index_iter, reftable_record_type(rec), 1);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reader_seek_linear(r, &index_iter, &want_index_rec);\n+\twhile (1) {\n+\t\terr = table_iter_next(&index_iter, &index_result_rec);\n+\t\ttable_iter_block_done(&index_iter);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = reader_table_iter_at(r, &next, index_result.offset, 0);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = block_iter_seek(&next.bi, &want_index.last_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (next.typ == reftable_record_type(rec)) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (next.typ != BLOCK_TYPE_INDEX) {\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_copy_from(&index_iter, &next);\n+\t}\n+\n+\tif (err == 0) {\n+\t\tstruct table_iter empty = TABLE_ITER_INIT;\n+\t\tstruct table_iter *malloced =\n+\t\t\treftable_calloc(sizeof(struct table_iter));\n+\t\t*malloced = empty;\n+\t\ttable_iter_copy_from(malloced, &next);\n+\t\titerator_from_table_iter(it, malloced);\n+\t}\n+done:\n+\tblock_iter_close(&next.bi);\n+\ttable_iter_close(&index_iter);\n+\treftable_record_release(&want_index_rec);\n+\treftable_record_release(&index_result_rec);\n+\treturn err;\n+}\n+\n+static int reader_seek_internal(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it,\n+\t\t\t\tstruct reftable_record *rec)\n+{\n+\tstruct reftable_reader_offsets *offs =\n+\t\treader_offsets_for(r, reftable_record_type(rec));\n+\tuint64_t idx = offs->index_offset;\n+\tstruct table_iter ti = TABLE_ITER_INIT;\n+\tint err = 0;\n+\tif (idx > 0)\n+\t\treturn reader_seek_indexed(r, it, rec);\n+\n+\terr = reader_start(r, &ti, reftable_record_type(rec), 0);\n+\tif (err < 0)\n+\t\treturn err;\n+\terr = reader_seek_linear(r, &ti, rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct table_iter *p =\n+\t\t\treftable_malloc(sizeof(struct table_iter));\n+\t\t*p = ti;\n+\t\titerator_from_table_iter(it, p);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec)\n+{\n+\tuint8_t typ = reftable_record_type(rec);\n+\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tif (!offs->is_present) {\n+\t\titerator_set_empty(it);\n+\t\treturn 0;\n+\t}\n+\n+\treturn reader_seek_internal(r, it, rec);\n+}\n+\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_reader_seek_log_at(r, it, name, max);\n+}\n+\n+void reader_close(struct reftable_reader *r)\n+{\n+\tblock_source_close(&r->source);\n+\tFREE_AND_NULL(r->name);\n+}\n+\n+int reftable_new_reader(struct reftable_reader **p,\n+\t\t\tstruct reftable_block_source *src, char const *name)\n+{\n+\tstruct reftable_reader *rd =\n+\t\treftable_calloc(sizeof(struct reftable_reader));\n+\tint err = init_reader(rd, src, name);\n+\tif (err == 0) {\n+\t\t*p = rd;\n+\t} else {\n+\t\tblock_source_close(src);\n+\t\treftable_free(rd);\n+\t}\n+\treturn err;\n+}\n+\n+void reftable_reader_free(struct reftable_reader *r)\n+{\n+\treader_close(r);\n+\treftable_free(r);\n+}\n+\n+static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct reftable_obj_record want = {\n+\t\t.hash_prefix = oid,\n+\t\t.hash_prefix_len = r->object_id_len,\n+\t};\n+\tstruct reftable_record want_rec = { NULL };\n+\tstruct reftable_iterator oit = { NULL };\n+\tstruct reftable_obj_record got = { NULL };\n+\tstruct reftable_record got_rec = { NULL };\n+\tint err = 0;\n+\tstruct indexed_table_ref_iter *itr = NULL;\n+\n+\t/* Look through the reverse index. */\n+\treftable_record_from_obj(&want_rec, &want);\n+\terr = reader_seek(r, &oit, &want_rec);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\t/* read out the reftable_obj_record */\n+\treftable_record_from_obj(&got_rec, &got);\n+\terr = iterator_next(&oit, &got_rec);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 0 ||\n+\t memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {\n+\t\t/* didn't find it; return empty iterator */\n+\t\titerator_set_empty(it);\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),\n+\t\t\t\t\t got.offsets, got.offset_len);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tgot.offsets = NULL;\n+\titerator_from_indexed_table_ref_iter(it, itr);\n+\n+done:\n+\treftable_iterator_destroy(&oit);\n+\treftable_record_release(&got_rec);\n+\treturn err;\n+}\n+\n+static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct table_iter ti_empty = TABLE_ITER_INIT;\n+\tstruct table_iter *ti = reftable_calloc(sizeof(struct table_iter));\n+\tstruct filtering_ref_iterator *filter = NULL;\n+\tstruct filtering_ref_iterator empty = FILTERING_REF_ITERATOR_INIT;\n+\tint oid_len = hash_size(r->hash_id);\n+\tint err;\n+\n+\t*ti = ti_empty;\n+\terr = reader_start(r, ti, BLOCK_TYPE_REF, 0);\n+\tif (err < 0) {\n+\t\treftable_free(ti);\n+\t\treturn err;\n+\t}\n+\n+\tfilter = reftable_malloc(sizeof(struct filtering_ref_iterator));\n+\t*filter = empty;\n+\n+\tstrbuf_add(&filter->oid, oid, oid_len);\n+\treftable_table_from_reader(&filter->tab, r);\n+\tfilter->double_check = 0;\n+\titerator_from_table_iter(&filter->it, ti);\n+\n+\titerator_from_filtering_ref_iterator(it, filter);\n+\treturn 0;\n+}\n+\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid)\n+{\n+\tif (r->obj_offsets.is_present)\n+\t\treturn reftable_reader_refs_for_indexed(r, it, oid);\n+\treturn reftable_reader_refs_for_unindexed(r, it, oid);\n+}\n+\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r)\n+{\n+\treturn r->max_update_index;\n+}\n+\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r)\n+{\n+\treturn r->min_update_index;\n+}\ndiff --git a/reftable/reader.h b/reftable/reader.h\nnew file mode 100644\nindex 000000000000..6d4927e1c5d6\n--- /dev/null\n+++ b/reftable/reader.h\n@@ -0,0 +1,75 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef READER_H\n+#define READER_H\n+\n+#include \"block.h\"\n+#include \"record.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source);\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size);\n+void block_source_close(struct reftable_block_source *source);\n+\n+/* metadata for a block type */\n+struct reftable_reader_offsets {\n+\tint is_present;\n+\tuint64_t offset;\n+\tuint64_t index_offset;\n+};\n+\n+/* The state for reading a reftable file. */\n+struct reftable_reader {\n+\t/* for convience, associate a name with the instance. */\n+\tchar *name;\n+\tstruct reftable_block_source source;\n+\n+\t/* Size of the file, excluding the footer. */\n+\tuint64_t size;\n+\n+\t/* 'sha1' for SHA1, 's256' for SHA-256 */\n+\tuint32_t hash_id;\n+\n+\tuint32_t block_size;\n+\tuint64_t min_update_index;\n+\tuint64_t max_update_index;\n+\t/* Length of the OID keys in the 'o' section */\n+\tint object_id_len;\n+\tint version;\n+\n+\tstruct reftable_reader_offsets ref_offsets;\n+\tstruct reftable_reader_offsets obj_offsets;\n+\tstruct reftable_reader_offsets log_offsets;\n+};\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name);\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec);\n+void reader_close(struct reftable_reader *r);\n+const char *reader_name(struct reftable_reader *r);\n+\n+/* initialize a block reader to read from `r` */\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ);\n+\n+/* generic interface to reftables */\n+struct reftable_table_vtable {\n+\tint (*seek_record)(void *tab, struct reftable_iterator *it,\n+\t\t\t struct reftable_record *);\n+\tuint32_t (*hash_id)(void *tab);\n+\tuint64_t (*min_update_index)(void *tab);\n+\tuint64_t (*max_update_index)(void *tab);\n+};\n+\n+#endif\ndiff --git a/reftable/reftable-iterator.h b/reftable/reftable-iterator.h\nnew file mode 100644\nindex 000000000000..d8251a45ba19\n--- /dev/null\n+++ b/reftable/reftable-iterator.h\n@@ -0,0 +1,37 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_ITERATOR_H\n+#define REFTABLE_ITERATOR_H\n+\n+#include \"reftable-record.h\"\n+\n+/* iterator is the generic interface for walking over data stored in a\n+ * reftable.\n+ */\n+struct reftable_iterator {\n+\tstruct reftable_iterator_vtable *ops;\n+\tvoid *iter_arg;\n+};\n+\n+/* reads the next reftable_ref_record. Returns < 0 for error, 0 for OK and > 0:\n+ * end of iteration.\n+ */\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* reads the next reftable_log_record. Returns < 0 for error, 0 for OK and > 0:\n+ * end of iteration.\n+ */\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* releases resources associated with an iterator. */\n+void reftable_iterator_destroy(struct reftable_iterator *it);\n+\n+#endif\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nnew file mode 100644\nindex 000000000000..804e28543b35\n--- /dev/null\n+++ b/reftable/reftable-reader.h\n@@ -0,0 +1,89 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_READER_H\n+#define REFTABLE_READER_H\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-blocksource.h\"\n+\n+/*\n+ * Reading single tables\n+ *\n+ * The follow routines are for reading single files. For an application-level\n+ * interface, skip ahead to struct reftable_merged_table and struct\n+ * reftable_stack.\n+ */\n+\n+/* The reader struct is a handle to an open reftable file. */\n+struct reftable_reader;\n+\n+/* reftable_new_reader opens a reftable for reading. If successful, returns 0\n+ * code and sets pp. The name is used for creating a stack. Typically, it is the\n+ * basename of the file. The block source `src` is owned by the reader, and is\n+ * closed on calling reftable_reader_destroy().\n+ */\n+int reftable_new_reader(struct reftable_reader **pp,\n+\t\t\tstruct reftable_block_source *src, const char *name);\n+\n+/* reftable_reader_seek_ref returns an iterator where 'name' would be inserted\n+ in the table. To seek to the start of the table, use name = \"\".\n+\n+ example:\n+\n+ struct reftable_reader *r = NULL;\n+ int err = reftable_new_reader(&r, &src, \"filename\");\n+ if (err < 0) { ... }\n+ struct reftable_iterator it = {0};\n+ err = reftable_reader_seek_ref(r, &it, \"refs/heads/master\");\n+ if (err < 0) { ... }\n+ struct reftable_ref_record ref = {0};\n+ while (1) {\n+ err = reftable_iterator_next_ref(&it, &ref);\n+ if (err > 0) {\n+ break;\n+ }\n+ if (err < 0) {\n+ ..error handling..\n+ }\n+ ..found..\n+ }\n+ reftable_iterator_destroy(&it);\n+ reftable_ref_record_release(&ref);\n+ */\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* returns the hash ID used in this table. */\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r);\n+\n+/* seek to logs for the given name, older than update_index. To seek to the\n+ start of the table, use name = \"\".\n+ */\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index);\n+\n+/* seek to newest log entry for given name. */\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* closes and deallocates a reader. */\n+void reftable_reader_free(struct reftable_reader *);\n+\n+/* return an iterator for the refs pointing to `oid`. */\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid);\n+\n+/* return the max_update_index for a table */\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r);\n+\n+/* return the min_update_index for a table */\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"418996","messageId":"28aa69f7bbcba22b67d7a0300ce62538e3189b95.1615580397.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v5.git.git.1615580397.gitgitgadget@gmail.com","subject":"[PATCH v5 12/15] reftable: rest of library","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-12T20:19:53Z","receivedAt":"2021-03-12T20:20:46Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 4 +\n reftable/VERSION | 1 +\n reftable/dump.c | 212 ++++++\n reftable/merged.c | 366 ++++++++++\n reftable/merged.h | 35 +\n reftable/merged_test.c | 343 ++++++++++\n reftable/pq.c | 115 ++++\n reftable/pq.h | 32 +\n reftable/refname.c | 209 ++++++\n reftable/refname.h | 29 +\n reftable/refname_test.c | 102 +++\n reftable/reftable-generic.h | 48 ++\n reftable/reftable-merged.h | 68 ++\n reftable/reftable-stack.h | 120 ++++\n reftable/reftable.c | 98 +++\n reftable/stack.c | 1260 +++++++++++++++++++++++++++++++++++\n reftable/stack.h | 41 ++\n reftable/stack_test.c | 809 ++++++++++++++++++++++\n t/helper/test-reftable.c | 3 +\n 19 files changed, 3895 insertions(+)\n create mode 100644 reftable/VERSION\n create mode 100644 reftable/dump.c\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n create mode 100644 reftable/reftable-generic.h\n create mode 100644 reftable/reftable-merged.h\n create mode 100644 reftable/reftable-stack.h\n create mode 100644 reftable/reftable.c\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n\ndiff --git a/Makefile b/Makefile\nindex b0152d9f5f84..a00cc17ec781 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2387,10 +2387,14 @@ REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n+REFTABLE_OBJS += reftable/merged.o\n+REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/refname.o\n REFTABLE_OBJS += reftable/reftable.o\n+REFTABLE_OBJS += reftable/stack.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n REFTABLE_OBJS += reftable/zlib-compat.o\ndiff --git a/reftable/VERSION b/reftable/VERSION\nnew file mode 100644\nindex 000000000000..b4d6be9f50f3\n--- /dev/null\n+++ b/reftable/VERSION\n@@ -0,0 +1 @@\n+a337d48d4d42d513d6baa33addc172f0e0e36288 C: use tagged union for reftable_log_record\ndiff --git a/reftable/dump.c b/reftable/dump.c\nnew file mode 100644\nindex 000000000000..00b444e8c9b8\n--- /dev/null\n+++ b/reftable/dump.c\n@@ -0,0 +1,212 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+\n+#include \"reftable.h\"\n+#include \"reftable-tests.h\"\n+\n+static uint32_t hash_id;\n+\n+static int dump_table(const char *tablename)\n+{\n+\tstruct reftable_block_source src = { 0 };\n+\tint err = reftable_block_source_from_file(&src, tablename);\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_ref_record ref = { 0 };\n+\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_reader *r = NULL;\n+\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\terr = reftable_new_reader(&r, &src, tablename);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\terr = reftable_reader_seek_ref(r, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_ref_record_print(&ref, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_clear(&ref);\n+\n+\terr = reftable_reader_seek_log(r, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_log_record_print(&log, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_clear(&log);\n+\n+\treftable_reader_free(r);\n+\treturn 0;\n+}\n+\n+static int compact_stack(const char *stackdir)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = {};\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_stack_compact_all(stack, NULL);\n+\tif (err < 0)\n+\t\tgoto done;\n+done:\n+\tif (stack != NULL) {\n+\t\treftable_stack_destroy(stack);\n+\t}\n+\treturn err;\n+}\n+\n+static int dump_stack(const char *stackdir)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = {};\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_ref_record ref = { 0 };\n+\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_merged_table *merged = NULL;\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tmerged = reftable_stack_merged_table(stack);\n+\n+\terr = reftable_merged_table_seek_ref(merged, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_ref_record_print(&ref, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_clear(&ref);\n+\n+\terr = reftable_merged_table_seek_log(merged, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_log_record_print(&log, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_clear(&log);\n+\n+\treftable_stack_destroy(stack);\n+\treturn 0;\n+}\n+\n+static void print_help(void)\n+{\n+\tprintf(\"usage: dump [-cst] arg\\n\\n\"\n+\t \"options: \\n\"\n+\t \" -c compact\\n\"\n+\t \" -t dump table\\n\"\n+\t \" -s dump stack\\n\"\n+\t \" -h this help\\n\"\n+\t \" -2 use SHA256\\n\"\n+\t \"\\n\");\n+}\n+\n+int reftable_dump_main(int argc, char *const *argv)\n+{\n+\tint err = 0;\n+\tint opt;\n+\tint opt_dump_table = 0;\n+\tint opt_dump_stack = 0;\n+\tint opt_compact = 0;\n+\tconst char *arg = NULL;\n+\twhile ((opt = getopt(argc, argv, \"2chts\")) != -1) {\n+\t\tswitch (opt) {\n+\t\tcase '2':\n+\t\t\thash_id = 0x73323536;\n+\t\t\tbreak;\n+\t\tcase 't':\n+\t\t\topt_dump_table = 1;\n+\t\t\tbreak;\n+\t\tcase 's':\n+\t\t\topt_dump_stack = 1;\n+\t\t\tbreak;\n+\t\tcase 'c':\n+\t\t\topt_compact = 1;\n+\t\t\tbreak;\n+\t\tcase '?':\n+\t\tcase 'h':\n+\t\t\tprint_help();\n+\t\t\treturn 2;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (argv[optind] == NULL) {\n+\t\tfprintf(stderr, \"need argument\\n\");\n+\t\tprint_help();\n+\t\treturn 2;\n+\t}\n+\n+\targ = argv[optind];\n+\n+\tif (opt_dump_table) {\n+\t\terr = dump_table(arg);\n+\t} else if (opt_dump_stack) {\n+\t\terr = dump_stack(arg);\n+\t} else if (opt_compact) {\n+\t\terr = compact_stack(arg);\n+\t}\n+\n+\tif (err < 0) {\n+\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv[0], arg,\n+\t\t\treftable_error_str(err));\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\ndiff --git a/reftable/merged.c b/reftable/merged.c\nnew file mode 100644\nindex 000000000000..ed02625bb548\n--- /dev/null\n+++ b/reftable/merged.c\n@@ -0,0 +1,366 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"constants.h\"\n+#include \"iter.h\"\n+#include \"pq.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-error.h\"\n+#include \"system.h\"\n+\n+static int merged_iter_init(struct merged_iter *mi)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\t\tint err = iterator_next(&mi->stack[i], &rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t\t\treftable_record_destroy(&rec);\n+\t\t} else {\n+\t\t\tstruct pq_entry e = {\n+\t\t\t\t.rec = rec,\n+\t\t\t\t.index = i,\n+\t\t\t};\n+\t\t\tmerged_iter_pqueue_add(&mi->pq, e);\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void merged_iter_close(void *p)\n+{\n+\tstruct merged_iter *mi = (struct merged_iter *)p;\n+\tint i = 0;\n+\tmerged_iter_pqueue_release(&mi->pq);\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t}\n+\treftable_free(mi->stack);\n+}\n+\n+static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,\n+\t\t\t\t\t size_t idx)\n+{\n+\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\tstruct pq_entry e = {\n+\t\t.rec = rec,\n+\t\t.index = idx,\n+\t};\n+\tint err = iterator_next(&mi->stack[idx], &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (err > 0) {\n+\t\treftable_iterator_destroy(&mi->stack[idx]);\n+\t\treftable_record_destroy(&rec);\n+\t\treturn 0;\n+\t}\n+\n+\tmerged_iter_pqueue_add(&mi->pq, e);\n+\treturn 0;\n+}\n+\n+static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)\n+{\n+\tif (iterator_is_null(&mi->stack[idx]))\n+\t\treturn 0;\n+\treturn merged_iter_advance_nonnull_subiter(mi, idx);\n+}\n+\n+static int merged_iter_next_entry(struct merged_iter *mi,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct strbuf entry_key = STRBUF_INIT;\n+\tstruct pq_entry entry = { 0 };\n+\tint err = 0;\n+\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\tentry = merged_iter_pqueue_remove(&mi->pq);\n+\terr = merged_iter_advance_subiter(mi, entry.index);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\t/*\n+\t One can also use reftable as datacenter-local storage, where the ref\n+\t database is maintained in globally consistent database (eg.\n+\t CockroachDB or Spanner). In this scenario, replication delays together\n+\t with compaction may cause newer tables to contain older entries. In\n+\t such a deployment, the loop below must be changed to collect all\n+\t entries for the same key, and return new the newest one.\n+\t*/\n+\treftable_record_key(&entry.rec, &entry_key);\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq)) {\n+\t\tstruct pq_entry top = merged_iter_pqueue_top(mi->pq);\n+\t\tstruct strbuf k = STRBUF_INIT;\n+\t\tint err = 0, cmp = 0;\n+\n+\t\treftable_record_key(&top.rec, &k);\n+\n+\t\tcmp = strbuf_cmp(&k, &entry_key);\n+\t\tstrbuf_release(&k);\n+\n+\t\tif (cmp > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\t\terr = merged_iter_advance_subiter(mi, top.index);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_record_destroy(&top.rec);\n+\t}\n+\n+\treftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));\n+\treftable_record_destroy(&entry.rec);\n+\tstrbuf_release(&entry_key);\n+\treturn 0;\n+}\n+\n+static int merged_iter_next(struct merged_iter *mi, struct reftable_record *rec)\n+{\n+\twhile (1) {\n+\t\tint err = merged_iter_next_entry(mi, rec);\n+\t\tif (err == 0 && mi->suppress_deletions &&\n+\t\t reftable_record_is_deletion(rec)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\treturn err;\n+\t}\n+}\n+\n+static int merged_iter_next_void(void *p, struct reftable_record *rec)\n+{\n+\tstruct merged_iter *mi = (struct merged_iter *)p;\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\treturn merged_iter_next(mi, rec);\n+}\n+\n+static struct reftable_iterator_vtable merged_iter_vtable = {\n+\t.next = &merged_iter_next_void,\n+\t.close = &merged_iter_close,\n+};\n+\n+static void iterator_from_merged_iter(struct reftable_iterator *it,\n+\t\t\t\t struct merged_iter *mi)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = mi;\n+\tit->ops = &merged_iter_vtable;\n+}\n+\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id)\n+{\n+\tstruct reftable_merged_table *m = NULL;\n+\tuint64_t last_max = 0;\n+\tuint64_t first_min = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t min = reftable_table_min_update_index(&stack[i]);\n+\t\tuint64_t max = reftable_table_max_update_index(&stack[i]);\n+\n+\t\tif (reftable_table_hash_id(&stack[i]) != hash_id) {\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\t\t}\n+\t\tif (i == 0 || min < first_min) {\n+\t\t\tfirst_min = min;\n+\t\t}\n+\t\tif (i == 0 || max > last_max) {\n+\t\t\tlast_max = max;\n+\t\t}\n+\t}\n+\n+\tm = (struct reftable_merged_table *)reftable_calloc(\n+\t\tsizeof(struct reftable_merged_table));\n+\tm->stack = stack;\n+\tm->stack_len = n;\n+\tm->min = first_min;\n+\tm->max = last_max;\n+\tm->hash_id = hash_id;\n+\t*dest = m;\n+\treturn 0;\n+}\n+\n+/* clears the list of subtable, without affecting the readers themselves. */\n+void merged_table_release(struct reftable_merged_table *mt)\n+{\n+\tFREE_AND_NULL(mt->stack);\n+\tmt->stack_len = 0;\n+}\n+\n+void reftable_merged_table_free(struct reftable_merged_table *mt)\n+{\n+\tif (mt == NULL) {\n+\t\treturn;\n+\t}\n+\tmerged_table_release(mt);\n+\treftable_free(mt);\n+}\n+\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->max;\n+}\n+\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->min;\n+}\n+\n+static int reftable_table_seek_record(struct reftable_table *tab,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn tab->ops->seek_record(tab->table_arg, it, rec);\n+}\n+\n+static int merged_table_seek_record(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_iterator *iters = reftable_calloc(\n+\t\tsizeof(struct reftable_iterator) * mt->stack_len);\n+\tstruct merged_iter merged = {\n+\t\t.stack = iters,\n+\t\t.typ = reftable_record_type(rec),\n+\t\t.hash_id = mt->hash_id,\n+\t\t.suppress_deletions = mt->suppress_deletions,\n+\t};\n+\tint n = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < mt->stack_len && err == 0; i++) {\n+\t\tint e = reftable_table_seek_record(&mt->stack[i], &iters[n],\n+\t\t\t\t\t\t rec);\n+\t\tif (e < 0) {\n+\t\t\terr = e;\n+\t\t}\n+\t\tif (e == 0) {\n+\t\t\tn++;\n+\t\t}\n+\t}\n+\tif (err < 0) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < n; i++) {\n+\t\t\treftable_iterator_destroy(&iters[i]);\n+\t\t}\n+\t\treftable_free(iters);\n+\t\treturn err;\n+\t}\n+\n+\tmerged.stack_len = n;\n+\terr = merged_iter_init(&merged);\n+\tif (err < 0) {\n+\t\tmerged_iter_close(&merged);\n+\t\treturn err;\n+\t} else {\n+\t\tstruct merged_iter *p =\n+\t\t\treftable_malloc(sizeof(struct merged_iter));\n+\t\t*p = merged;\n+\t\titerator_from_merged_iter(it, p);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_merged_table_seek_log_at(mt, it, name, max);\n+}\n+\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\n+{\n+\treturn mt->hash_id;\n+}\n+\n+static int reftable_merged_table_seek_void(void *tab,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn merged_table_seek_record((struct reftable_merged_table *)tab, it,\n+\t\t\t\t\trec);\n+}\n+\n+static uint32_t reftable_merged_table_hash_id_void(void *tab)\n+{\n+\treturn reftable_merged_table_hash_id(\n+\t\t(struct reftable_merged_table *)tab);\n+}\n+\n+static uint64_t reftable_merged_table_min_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_min_update_index(\n+\t\t(struct reftable_merged_table *)tab);\n+}\n+\n+static uint64_t reftable_merged_table_max_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_max_update_index(\n+\t\t(struct reftable_merged_table *)tab);\n+}\n+\n+static struct reftable_table_vtable merged_table_vtable = {\n+\t.seek_record = reftable_merged_table_seek_void,\n+\t.hash_id = reftable_merged_table_hash_id_void,\n+\t.min_update_index = reftable_merged_table_min_update_index_void,\n+\t.max_update_index = reftable_merged_table_max_update_index_void,\n+};\n+\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *merged)\n+{\n+\tassert(tab->ops == NULL);\n+\ttab->ops = &merged_table_vtable;\n+\ttab->table_arg = merged;\n+}\ndiff --git a/reftable/merged.h b/reftable/merged.h\nnew file mode 100644\nindex 000000000000..8c4d4d58d77a\n--- /dev/null\n+++ b/reftable/merged.h\n@@ -0,0 +1,35 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef MERGED_H\n+#define MERGED_H\n+\n+#include \"pq.h\"\n+\n+struct reftable_merged_table {\n+\tstruct reftable_table *stack;\n+\tsize_t stack_len;\n+\tuint32_t hash_id;\n+\tint suppress_deletions;\n+\n+\tuint64_t min;\n+\tuint64_t max;\n+};\n+\n+struct merged_iter {\n+\tstruct reftable_iterator *stack;\n+\tuint32_t hash_id;\n+\tsize_t stack_len;\n+\tuint8_t typ;\n+\tint suppress_deletions;\n+\tstruct merged_iter_pqueue pq;\n+};\n+\n+void merged_table_release(struct reftable_merged_table *mt);\n+\n+#endif\ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nnew file mode 100644\nindex 000000000000..09a70aa0fda3\n--- /dev/null\n+++ b/reftable/merged_test.c\n@@ -0,0 +1,343 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"pq.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-generic.h\"\n+#include \"reftable-stack.h\"\n+\n+static void test_pq(void)\n+{\n+\tchar *names[54] = { NULL };\n+\tint N = ARRAY_SIZE(names) - 1;\n+\n+\tstruct merged_iter_pqueue pq = { NULL };\n+\tconst char *last = NULL;\n+\n+\tint i = 0;\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tsnprintf(name, sizeof(name), \"%02d\", i);\n+\t\tnames[i] = xstrdup(name);\n+\t}\n+\n+\ti = 1;\n+\tdo {\n+\t\tstruct reftable_record rec =\n+\t\t\treftable_new_record(BLOCK_TYPE_REF);\n+\t\tstruct pq_entry e = { 0 };\n+\n+\t\treftable_record_as_ref(&rec)->refname = names[i];\n+\t\te.rec = rec;\n+\t\tmerged_iter_pqueue_add(&pq, e);\n+\t\tmerged_iter_pqueue_check(pq);\n+\t\ti = (i * 7) % N;\n+\t} while (i != 1);\n+\n+\twhile (!merged_iter_pqueue_is_empty(pq)) {\n+\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct reftable_ref_record *ref =\n+\t\t\treftable_record_as_ref(&e.rec);\n+\n+\t\tmerged_iter_pqueue_check(pq);\n+\n+\t\tif (last != NULL) {\n+\t\t\tassert(strcmp(last, ref->refname) < 0);\n+\t\t}\n+\t\tlast = ref->refname;\n+\t\tref->refname = NULL;\n+\t\treftable_free(ref);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\n+\tmerged_iter_pqueue_release(&pq);\n+}\n+\n+static void write_test_table(struct strbuf *buf,\n+\t\t\t struct reftable_ref_record refs[], int n)\n+{\n+\tint min = 0xffffffff;\n+\tint max = 0;\n+\tint i = 0;\n+\tint err;\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_writer *w = NULL;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t ui = refs[i].update_index;\n+\t\tif (ui > max) {\n+\t\t\tmax = ui;\n+\t\t}\n+\t\tif (ui < min) {\n+\t\t\tmin = ui;\n+\t\t}\n+\t}\n+\n+\tw = reftable_new_writer(&strbuf_add_void, buf, &opts);\n+\treftable_writer_set_limits(w, min, max);\n+\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t before = refs[i].update_index;\n+\t\tint n = reftable_writer_add_ref(w, &refs[i]);\n+\t\tassert(n == 0);\n+\t\tassert(before == refs[i].update_index);\n+\t}\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\n+\treftable_writer_free(w);\n+}\n+\n+static struct reftable_merged_table *\n+merged_table_from_records(struct reftable_ref_record **refs,\n+\t\t\t struct reftable_block_source **source,\n+\t\t\t struct reftable_reader ***readers, int *sizes,\n+\t\t\t struct strbuf *buf, int n)\n+{\n+\tint i = 0;\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err;\n+\tstruct reftable_table *tabs =\n+\t\treftable_calloc(n * sizeof(struct reftable_table));\n+\t*readers = reftable_calloc(n * sizeof(struct reftable_reader *));\n+\t*source = reftable_calloc(n * sizeof(**source));\n+\tfor (i = 0; i < n; i++) {\n+\t\twrite_test_table(&buf[i], refs[i], sizes[i]);\n+\t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n+\n+\t\terr = reftable_new_reader(&(*readers)[i], &(*source)[i],\n+\t\t\t\t\t \"name\");\n+\t\tEXPECT_ERR(err);\n+\t\treftable_table_from_reader(&tabs[i], (*readers)[i]);\n+\t}\n+\n+\terr = reftable_new_merged_table(&mt, tabs, n, SHA1_ID);\n+\tEXPECT_ERR(err);\n+\treturn mt;\n+}\n+\n+static void readers_destroy(struct reftable_reader **readers, size_t n)\n+{\n+\tint i = 0;\n+\tfor (; i < n; i++)\n+\t\treftable_reader_free(readers[i]);\n+\treftable_free(readers);\n+}\n+\n+static void test_merged_between(void)\n+{\n+\tuint8_t hash1[SHA1_SIZE] = { 1, 2, 3, 0 };\n+\n+\tstruct reftable_ref_record r1[] = { {\n+\t\t.refname = \"b\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t.value.val1 = hash1,\n+\t} };\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_DELETION,\n+\t} };\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2 };\n+\tint sizes[] = { 1, 1 };\n+\tstruct strbuf bufs[2] = { STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 2);\n+\tint i;\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT_ERR(err);\n+\tEXPECT(ref.update_index == 2);\n+\treftable_ref_record_release(&ref);\n+\treftable_iterator_destroy(&it);\n+\treaders_destroy(readers, 2);\n+\treftable_merged_table_free(mt);\n+\tfor (i = 0; i < ARRAY_SIZE(bufs); i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treftable_free(bs);\n+}\n+\n+static void test_merged(void)\n+{\n+\tuint8_t hash1[SHA1_SIZE] = { 1 };\n+\tuint8_t hash2[SHA1_SIZE] = { 2 };\n+\tstruct reftable_ref_record r1[] = {\n+\t\t{\n+\t\t\t.refname = \"a\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"b\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t}\n+\t};\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_DELETION,\n+\t} };\n+\tstruct reftable_ref_record r3[] = {\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash2,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"d\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t};\n+\n+\tstruct reftable_ref_record want[] = {\n+\t\tr2[0],\n+\t\tr1[1],\n+\t\tr3[0],\n+\t\tr3[1],\n+\t};\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2, r3 };\n+\tint sizes[3] = { 3, 1, 2 };\n+\tstruct strbuf bufs[3] = { STRBUF_INIT, STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 3);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tstruct reftable_ref_record *out = NULL;\n+\tsize_t len = 0;\n+\tsize_t cap = 0;\n+\tint i = 0;\n+\n+\tEXPECT_ERR(err);\n+\twhile (len < 100) { /* cap loops/recursion. */\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tout = reftable_realloc(\n+\t\t\t\tout, sizeof(struct reftable_ref_record) * cap);\n+\t\t}\n+\t\tout[len++] = ref;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\tassert(ARRAY_SIZE(want) == len);\n+\tfor (i = 0; i < len; i++) {\n+\t\tassert(reftable_ref_record_equal(&want[i], &out[i], SHA1_SIZE));\n+\t}\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_release(&out[i]);\n+\t}\n+\treftable_free(out);\n+\n+\tfor (i = 0; i < 3; i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treaders_destroy(readers, 3);\n+\treftable_merged_table_free(mt);\n+\treftable_free(bs);\n+}\n+\n+static void test_default_write_opts(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"master\",\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_table *tab = reftable_calloc(sizeof(*tab) * 1);\n+\tuint32_t hash_id;\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_merged_table *merged = NULL;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\thash_id = reftable_reader_hash_id(rd);\n+\tassert(hash_id == SHA1_ID);\n+\n+\treftable_table_from_reader(&tab[0], rd);\n+\terr = reftable_new_merged_table(&merged, tab, 1, SHA1_ID);\n+\tEXPECT_ERR(err);\n+\n+\treftable_reader_free(rd);\n+\treftable_merged_table_free(merged);\n+\tstrbuf_release(&buf);\n+}\n+\n+/* XXX test refs_for(oid) */\n+\n+int merged_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_merged_between);\n+\tRUN_TEST(test_pq);\n+\tRUN_TEST(test_merged);\n+\tRUN_TEST(test_default_write_opts);\n+\treturn 0;\n+}\ndiff --git a/reftable/pq.c b/reftable/pq.c\nnew file mode 100644\nindex 000000000000..8918d158e2d4\n--- /dev/null\n+++ b/reftable/pq.c\n@@ -0,0 +1,115 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"pq.h\"\n+\n+#include \"reftable-record.h\"\n+#include \"system.h\"\n+#include \"basics.h\"\n+\n+static int pq_less(struct pq_entry a, struct pq_entry b)\n+{\n+\tstruct strbuf ak = STRBUF_INIT;\n+\tstruct strbuf bk = STRBUF_INIT;\n+\tint cmp = 0;\n+\treftable_record_key(&a.rec, &ak);\n+\treftable_record_key(&b.rec, &bk);\n+\n+\tcmp = strbuf_cmp(&ak, &bk);\n+\n+\tstrbuf_release(&ak);\n+\tstrbuf_release(&bk);\n+\n+\tif (cmp == 0)\n+\t\treturn a.index > b.index;\n+\n+\treturn cmp < 0;\n+}\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.heap[0];\n+}\n+\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.len == 0;\n+}\n+\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq)\n+{\n+\tint i = 0;\n+\tfor (i = 1; i < pq.len; i++) {\n+\t\tint parent = (i - 1) / 2;\n+\n+\t\tassert(pq_less(pq.heap[parent], pq.heap[i]));\n+\t}\n+}\n+\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tstruct pq_entry e = pq->heap[0];\n+\tpq->heap[0] = pq->heap[pq->len - 1];\n+\tpq->len--;\n+\n+\ti = 0;\n+\twhile (i < pq->len) {\n+\t\tint min = i;\n+\t\tint j = 2 * i + 1;\n+\t\tint k = 2 * i + 2;\n+\t\tif (j < pq->len && pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tmin = j;\n+\t\t}\n+\t\tif (k < pq->len && pq_less(pq->heap[k], pq->heap[min])) {\n+\t\t\tmin = k;\n+\t\t}\n+\n+\t\tif (min == i) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[i], pq->heap[min]);\n+\t\ti = min;\n+\t}\n+\n+\treturn e;\n+}\n+\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)\n+{\n+\tint i = 0;\n+\tif (pq->len == pq->cap) {\n+\t\tpq->cap = 2 * pq->cap + 1;\n+\t\tpq->heap = reftable_realloc(pq->heap,\n+\t\t\t\t\t pq->cap * sizeof(struct pq_entry));\n+\t}\n+\n+\tpq->heap[pq->len++] = e;\n+\ti = pq->len - 1;\n+\twhile (i > 0) {\n+\t\tint j = (i - 1) / 2;\n+\t\tif (pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[j], pq->heap[i]);\n+\n+\t\ti = j;\n+\t}\n+}\n+\n+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < pq->len; i++) {\n+\t\treftable_record_destroy(&pq->heap[i].rec);\n+\t}\n+\tFREE_AND_NULL(pq->heap);\n+\tpq->len = pq->cap = 0;\n+}\ndiff --git a/reftable/pq.h b/reftable/pq.h\nnew file mode 100644\nindex 000000000000..385d2fb139a6\n--- /dev/null\n+++ b/reftable/pq.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef PQ_H\n+#define PQ_H\n+\n+#include \"record.h\"\n+\n+struct pq_entry {\n+\tint index;\n+\tstruct reftable_record rec;\n+};\n+\n+struct merged_iter_pqueue {\n+\tstruct pq_entry *heap;\n+\tsize_t len;\n+\tsize_t cap;\n+};\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq);\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq);\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq);\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e);\n+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n+\n+#endif\ndiff --git a/reftable/refname.c b/reftable/refname.c\nnew file mode 100644\nindex 000000000000..0f4eb3b292d3\n--- /dev/null\n+++ b/reftable/refname.c\n@@ -0,0 +1,209 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"reftable-error.h\"\n+#include \"basics.h\"\n+#include \"refname.h\"\n+#include \"reftable-iterator.h\"\n+\n+struct find_arg {\n+\tchar **names;\n+\tconst char *want;\n+};\n+\n+static int find_name(size_t k, void *arg)\n+{\n+\tstruct find_arg *f_arg = (struct find_arg *)arg;\n+\treturn strcmp(f_arg->names[k], f_arg->want) >= 0;\n+}\n+\n+static int modification_has_ref(struct modification *mod, const char *name)\n+{\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len && !strcmp(mod->add[idx], name)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tif (mod->del_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->del,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\tif (idx < mod->del_len && !strcmp(mod->del[idx], name)) {\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\terr = reftable_table_read_ref(&mod->tab, name, &ref);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+static void modification_release(struct modification *mod)\n+{\n+\t/* don't delete the strings themselves; they're owned by ref records.\n+\t */\n+\tFREE_AND_NULL(mod->add);\n+\tFREE_AND_NULL(mod->del);\n+\tmod->add_len = 0;\n+\tmod->del_len = 0;\n+}\n+\n+static int modification_has_ref_with_prefix(struct modification *mod,\n+\t\t\t\t\t const char *prefix)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = prefix,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len &&\n+\t\t !strncmp(prefix, mod->add[idx], strlen(prefix)))\n+\t\t\tgoto done;\n+\t}\n+\terr = reftable_table_seek_ref(&mod->tab, &it, prefix);\n+\tif (err)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\n+\t\tif (mod->del_len > 0) {\n+\t\t\tstruct find_arg arg = {\n+\t\t\t\t.names = mod->del,\n+\t\t\t\t.want = ref.refname,\n+\t\t\t};\n+\t\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\t\tif (idx < mod->del_len &&\n+\t\t\t !strcmp(ref.refname, mod->del[idx])) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (strncmp(ref.refname, prefix, strlen(prefix))) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_ref_record_release(&ref);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int validate_refname(const char *name)\n+{\n+\twhile (1) {\n+\t\tchar *next = strchr(name, '/');\n+\t\tif (!*name) {\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\t}\n+\t\tif (!next) {\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (next - name == 0 || (next - name == 1 && *name == '.') ||\n+\t\t (next - name == 2 && name[0] == '.' && name[1] == '.'))\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\tname = next + 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz)\n+{\n+\tstruct modification mod = {\n+\t\t.tab = tab,\n+\t\t.add = reftable_calloc(sizeof(char *) * sz),\n+\t\t.del = reftable_calloc(sizeof(char *) * sz),\n+\t};\n+\tint i = 0;\n+\tint err = 0;\n+\tfor (; i < sz; i++) {\n+\t\tif (reftable_ref_record_is_deletion(&recs[i])) {\n+\t\t\tmod.del[mod.del_len++] = recs[i].refname;\n+\t\t} else {\n+\t\t\tmod.add[mod.add_len++] = recs[i].refname;\n+\t\t}\n+\t}\n+\n+\terr = modification_validate(&mod);\n+\tmodification_release(&mod);\n+\treturn err;\n+}\n+\n+static void strbuf_trim_component(struct strbuf *sl)\n+{\n+\twhile (sl->len > 0) {\n+\t\tint is_slash = (sl->buf[sl->len - 1] == '/');\n+\t\tstrbuf_setlen(sl, sl->len - 1);\n+\t\tif (is_slash)\n+\t\t\tbreak;\n+\t}\n+}\n+\n+int modification_validate(struct modification *mod)\n+{\n+\tstruct strbuf slashed = STRBUF_INIT;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (; i < mod->add_len; i++) {\n+\t\terr = validate_refname(mod->add[i]);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\tstrbuf_addstr(&slashed, \"/\");\n+\n+\t\terr = modification_has_ref_with_prefix(mod, slashed.buf);\n+\t\tif (err == 0) {\n+\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\twhile (slashed.len) {\n+\t\t\tstrbuf_trim_component(&slashed);\n+\t\t\terr = modification_has_ref(mod, slashed.buf);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\terr = 0;\n+done:\n+\tstrbuf_release(&slashed);\n+\treturn err;\n+}\ndiff --git a/reftable/refname.h b/reftable/refname.h\nnew file mode 100644\nindex 000000000000..a24b40fcb428\n--- /dev/null\n+++ b/reftable/refname.h\n@@ -0,0 +1,29 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+#ifndef REFNAME_H\n+#define REFNAME_H\n+\n+#include \"reftable-record.h\"\n+#include \"reftable-generic.h\"\n+\n+struct modification {\n+\tstruct reftable_table tab;\n+\n+\tchar **add;\n+\tsize_t add_len;\n+\n+\tchar **del;\n+\tsize_t del_len;\n+};\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz);\n+\n+int modification_validate(struct modification *mod);\n+\n+#endif\ndiff --git a/reftable/refname_test.c b/reftable/refname_test.c\nnew file mode 100644\nindex 000000000000..5e005d6af31c\n--- /dev/null\n+++ b/reftable/refname_test.c\n@@ -0,0 +1,102 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"refname.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-writer.h\"\n+#include \"system.h\"\n+\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct testcase {\n+\tchar *add;\n+\tchar *del;\n+\tint error_code;\n+};\n+\n+static void test_conflict(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"a/b\",\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"destination\", /* make sure it's not a symref.\n+\t\t\t\t\t\t*/\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tint i;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct testcase cases[] = {\n+\t\t{ \"a/b/c\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"b\", NULL, 0 },\n+\t\t{ \"a\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"a\", \"a/b\", 0 },\n+\n+\t\t{ \"p/\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p//q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/./q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/../q\", NULL, REFTABLE_REFNAME_ERROR },\n+\n+\t\t{ \"a/b/c\", \"a/b\", 0 },\n+\t\t{ NULL, \"a//b\", 0 },\n+\t};\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\treftable_table_from_reader(&tab, rd);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct modification mod = {\n+\t\t\t.tab = tab,\n+\t\t};\n+\n+\t\tif (cases[i].add != NULL) {\n+\t\t\tmod.add = &cases[i].add;\n+\t\t\tmod.add_len = 1;\n+\t\t}\n+\t\tif (cases[i].del != NULL) {\n+\t\t\tmod.del = &cases[i].del;\n+\t\t\tmod.del_len = 1;\n+\t\t}\n+\n+\t\terr = modification_validate(&mod);\n+\t\tEXPECT(err == cases[i].error_code);\n+\t}\n+\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+int refname_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_conflict);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-generic.h b/reftable/reftable-generic.h\nnew file mode 100644\nindex 000000000000..77eca3b4eb0f\n--- /dev/null\n+++ b/reftable/reftable-generic.h\n@@ -0,0 +1,48 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_GENERIC_H\n+#define REFTABLE_GENERIC_H\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+#include \"reftable-merged.h\"\n+\n+/*\n+ * Provides a unified API for reading tables, either merged tables, or single\n+ * readers. */\n+struct reftable_table {\n+\tstruct reftable_table_vtable *ops;\n+\tvoid *table_arg;\n+};\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader);\n+\n+/* returns the hash ID from a generic reftable_table */\n+uint32_t reftable_table_hash_id(struct reftable_table *tab);\n+\n+/* create a generic table from reftable_merged_table */\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *table);\n+\n+/* returns the max update_index covered by this table. */\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab);\n+\n+/* returns the min update_index covered by this table. */\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0\n+ for success, and 1 if ref not found. */\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+#endif\ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nnew file mode 100644\nindex 000000000000..0e8ebe5a9956\n--- /dev/null\n+++ b/reftable/reftable-merged.h\n@@ -0,0 +1,68 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_MERGED_H\n+#define REFTABLE_MERGED_H\n+\n+#include \"reftable-iterator.h\"\n+\n+/*\n+ * Merged tables\n+ *\n+ * A ref database kept in a sequence of table files. The merged_table presents a\n+ * unified view to reading (seeking, iterating) a sequence of immutable tables.\n+ *\n+ * The merged tables are on purpose kept disconnected from their actual storage\n+ * (eg. files on disk), because it is useful to merge tables aren't files. For\n+ * example, the per-workspace and global ref namespace can be implemented as a\n+ * merged table of two stacks of file-backed reftables.\n+ */\n+\n+/* A merged table is implements seeking/iterating over a stack of tables. */\n+struct reftable_merged_table;\n+\n+/* A generic reftable; see below. */\n+struct reftable_table;\n+\n+/* reftable_new_merged_table creates a new merged table. It takes ownership of\n+ the stack array.\n+*/\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id);\n+\n+/* returns an iterator positioned just before 'name' */\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns an iterator for log entry, at given update_index */\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index);\n+\n+/* like reftable_merged_table_seek_log_at but look for the newest entry. */\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns the max update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt);\n+\n+/* returns the min update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt);\n+\n+/* releases memory for the merged_table */\n+void reftable_merged_table_free(struct reftable_merged_table *m);\n+\n+/* return the hash ID of the merged table. */\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m);\n+\n+#endif\ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nnew file mode 100644\nindex 000000000000..b7060f111e89\n--- /dev/null\n+++ b/reftable/reftable-stack.h\n@@ -0,0 +1,120 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_STACK_H\n+#define REFTABLE_STACK_H\n+\n+#include \"reftable-writer.h\"\n+\n+/*\n+ * The stack presents an interface to a mutable sequence of reftables.\n+\n+ * A stack can be mutated by pushing a table to the top of the stack.\n+\n+ * The reftable_stack automatically compacts files on disk to ensure good\n+ * amortized performance.\n+ *\n+ * For windows and other platforms that cannot have open files as rename\n+ * destinations, concurrent access from multiple processes needs the rand()\n+ * random seed to be randomized.\n+ */\n+struct reftable_stack;\n+\n+/* open a new reftable stack. The tables along with the table list will be\n+ * stored in 'dir'. Typically, this should be .git/reftables.\n+ */\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config);\n+\n+/* returns the update_index at which a next table should be written. */\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\n+\n+/* holds a transaction to add tables at the top of a stack. */\n+struct reftable_addition;\n+\n+/*\n+ * returns a new transaction to add reftables to the given stack. As a side\n+ * effect, the ref database is locked.\n+ */\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st);\n+\n+/* Adds a reftable to transaction. */\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+\n+/* Commits the transaction, releasing the lock. */\n+int reftable_addition_commit(struct reftable_addition *add);\n+\n+/* Release all non-committed data from the transaction, and deallocate the\n+ * transaction. Releases the lock if held. */\n+void reftable_addition_destroy(struct reftable_addition *add);\n+\n+/* add a new table to the stack. The write_table function must call\n+ * reftable_writer_set_limits, add refs and return an error value. */\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *write_arg),\n+\t\t void *write_arg);\n+\n+/* returns the merged_table for seeking. This table is valid until the\n+ * next write or reload, and should not be closed or deleted.\n+ */\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st);\n+\n+/* frees all resources associated with the stack. */\n+void reftable_stack_destroy(struct reftable_stack *st);\n+\n+/* Reloads the stack if necessary. This is very cheap to run if the stack was up\n+ * to date */\n+int reftable_stack_reload(struct reftable_stack *st);\n+\n+/* Policy for expiring reflog entries. */\n+struct reftable_log_expiry_config {\n+\t/* Drop entries older than this timestamp */\n+\tuint64_t time;\n+\n+\t/* Drop older entries */\n+\tuint64_t min_update_index;\n+};\n+\n+/* compacts all reftables into a giant table. Expire reflog entries if config is\n+ * non-NULL */\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config);\n+\n+/* heuristically compact unbalanced table stack. */\n+int reftable_stack_auto_compact(struct reftable_stack *st);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0 for\n+ * success, and 1 if ref not found. */\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* convenience function to read a single log. Returns < 0 for error, 0 for\n+ * success, and 1 if ref not found. */\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* statistics on past compactions. */\n+struct reftable_compaction_stats {\n+\tuint64_t bytes; /* total number of bytes written */\n+\tuint64_t entries_written; /* total number of entries written, including\n+\t\t\t\t failures. */\n+\tint attempts; /* how often we tried to compact */\n+\tint failures; /* failures happen on concurrent updates */\n+};\n+\n+/* return statistics for compaction up till now. */\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st);\n+\n+#endif\ndiff --git a/reftable/reftable.c b/reftable/reftable.c\nnew file mode 100644\nindex 000000000000..dc4fd03d5b25\n--- /dev/null\n+++ b/reftable/reftable.c\n@@ -0,0 +1,98 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"record.h\"\n+#include \"reader.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+static int reftable_reader_seek_void(void *tab, struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn reader_seek((struct reftable_reader *)tab, it, rec);\n+}\n+\n+static uint32_t reftable_reader_hash_id_void(void *tab)\n+{\n+\treturn reftable_reader_hash_id((struct reftable_reader *)tab);\n+}\n+\n+static uint64_t reftable_reader_min_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_min_update_index((struct reftable_reader *)tab);\n+}\n+\n+static uint64_t reftable_reader_max_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_max_update_index((struct reftable_reader *)tab);\n+}\n+\n+static struct reftable_table_vtable reader_vtable = {\n+\t.seek_record = reftable_reader_seek_void,\n+\t.hash_id = reftable_reader_hash_id_void,\n+\t.min_update_index = reftable_reader_min_update_index_void,\n+\t.max_update_index = reftable_reader_max_update_index_void,\n+};\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader)\n+{\n+\tassert(tab->ops == NULL);\n+\ttab->ops = &reader_vtable;\n+\ttab->table_arg = reader;\n+}\n+\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_table_seek_ref(tab, &it, name);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_ref(&it, ref);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref->refname, name) ||\n+\t reftable_ref_record_is_deletion(ref)) {\n+\t\treftable_ref_record_release(ref);\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->max_update_index(tab->table_arg);\n+}\n+\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->min_update_index(tab->table_arg);\n+}\n+\n+uint32_t reftable_table_hash_id(struct reftable_table *tab)\n+{\n+\treturn tab->ops->hash_id(tab->table_arg);\n+}\ndiff --git a/reftable/stack.c b/reftable/stack.c\nnew file mode 100644\nindex 000000000000..cff21c814dff\n--- /dev/null\n+++ b/reftable/stack.c\n@@ -0,0 +1,1260 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+#include \"merged.h\"\n+#include \"reader.h\"\n+#include \"refname.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-record.h\"\n+#include \"writer.h\"\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config);\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name);\n+static void reftable_addition_close(struct reftable_addition *add);\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open);\n+\n+static int reftable_fd_write(void *arg, const void *data, size_t sz)\n+{\n+\tint *fdp = (int *)arg;\n+\treturn write(*fdp, data, sz);\n+}\n+\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config)\n+{\n+\tstruct reftable_stack *p =\n+\t\treftable_calloc(sizeof(struct reftable_stack));\n+\tstruct strbuf list_file_name = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tif (config.hash_id == 0) {\n+\t\tconfig.hash_id = SHA1_ID;\n+\t}\n+\n+\t*dest = NULL;\n+\n+\tstrbuf_reset(&list_file_name);\n+\tstrbuf_addstr(&list_file_name, dir);\n+\tstrbuf_addstr(&list_file_name, \"/tables.list\");\n+\n+\tp->list_file = strbuf_detach(&list_file_name, NULL);\n+\tp->reftable_dir = xstrdup(dir);\n+\tp->config = config;\n+\n+\terr = reftable_stack_reload_maybe_reuse(p, 1);\n+\tif (err < 0) {\n+\t\treftable_stack_destroy(p);\n+\t} else {\n+\t\t*dest = p;\n+\t}\n+\treturn err;\n+}\n+\n+static int fd_read_lines(int fd, char ***namesp)\n+{\n+\toff_t size = lseek(fd, 0, SEEK_END);\n+\tchar *buf = NULL;\n+\tint err = 0;\n+\tif (size < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\terr = lseek(fd, 0, SEEK_SET);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tbuf = reftable_malloc(size + 1);\n+\tif (read(fd, buf, size) != size) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\tbuf[size] = 0;\n+\n+\tparse_names(buf, size, namesp);\n+\n+done:\n+\treftable_free(buf);\n+\treturn err;\n+}\n+\n+int read_lines(const char *filename, char ***namesp)\n+{\n+\tint fd = open(filename, O_RDONLY, 0644);\n+\tint err = 0;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\t*namesp = reftable_calloc(sizeof(char *));\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\treturn REFTABLE_IO_ERROR;\n+\t}\n+\terr = fd_read_lines(fd, namesp);\n+\tclose(fd);\n+\treturn err;\n+}\n+\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st)\n+{\n+\treturn st->merged;\n+}\n+\n+/* Close and free the stack */\n+void reftable_stack_destroy(struct reftable_stack *st)\n+{\n+\tif (st->merged != NULL) {\n+\t\treftable_merged_table_free(st->merged);\n+\t\tst->merged = NULL;\n+\t}\n+\n+\tif (st->readers != NULL) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < st->readers_len; i++) {\n+\t\t\treftable_reader_free(st->readers[i]);\n+\t\t}\n+\t\tst->readers_len = 0;\n+\t\tFREE_AND_NULL(st->readers);\n+\t}\n+\tFREE_AND_NULL(st->list_file);\n+\tFREE_AND_NULL(st->reftable_dir);\n+\treftable_free(st);\n+}\n+\n+static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n+\t\t\t\t\t\t int cur_len)\n+{\n+\tstruct reftable_reader **cur =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * cur_len);\n+\tint i = 0;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tcur[i] = st->readers[i];\n+\t}\n+\treturn cur;\n+}\n+\n+static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n+\t\t\t\t int reuse_open)\n+{\n+\tint cur_len = st->merged == NULL ? 0 : st->merged->stack_len;\n+\tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n+\tint err = 0;\n+\tint names_len = names_length(names);\n+\tstruct reftable_reader **new_readers =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * names_len);\n+\tstruct reftable_table *new_tables =\n+\t\treftable_calloc(sizeof(struct reftable_table) * names_len);\n+\tint new_readers_len = 0;\n+\tstruct reftable_merged_table *new_merged = NULL;\n+\tint i;\n+\n+\twhile (*names) {\n+\t\tstruct reftable_reader *rd = NULL;\n+\t\tchar *name = *names++;\n+\n+\t\t/* this is linear; we assume compaction keeps the number of\n+\t\t tables under control so this is not quadratic. */\n+\t\tint j = 0;\n+\t\tfor (j = 0; reuse_open && j < cur_len; j++) {\n+\t\t\tif (cur[j] != NULL && 0 == strcmp(cur[j]->name, name)) {\n+\t\t\t\trd = cur[j];\n+\t\t\t\tcur[j] = NULL;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (rd == NULL) {\n+\t\t\tstruct reftable_block_source src = { NULL };\n+\t\t\tstruct strbuf table_path = STRBUF_INIT;\n+\t\t\tstrbuf_addstr(&table_path, st->reftable_dir);\n+\t\t\tstrbuf_addstr(&table_path, \"/\");\n+\t\t\tstrbuf_addstr(&table_path, name);\n+\n+\t\t\terr = reftable_block_source_from_file(&src,\n+\t\t\t\t\t\t\t table_path.buf);\n+\t\t\tstrbuf_release(&table_path);\n+\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\n+\t\t\terr = reftable_new_reader(&rd, &src, name);\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\n+\t\tnew_readers[new_readers_len] = rd;\n+\t\treftable_table_from_reader(&new_tables[new_readers_len], rd);\n+\t\tnew_readers_len++;\n+\t}\n+\n+\t/* success! */\n+\terr = reftable_new_merged_table(&new_merged, new_tables,\n+\t\t\t\t\tnew_readers_len, st->config.hash_id);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tnew_tables = NULL;\n+\tst->readers_len = new_readers_len;\n+\tif (st->merged != NULL) {\n+\t\tmerged_table_release(st->merged);\n+\t\treftable_merged_table_free(st->merged);\n+\t}\n+\tif (st->readers != NULL) {\n+\t\treftable_free(st->readers);\n+\t}\n+\tst->readers = new_readers;\n+\tnew_readers = NULL;\n+\tnew_readers_len = 0;\n+\n+\tnew_merged->suppress_deletions = 1;\n+\tst->merged = new_merged;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tif (cur[i] != NULL) {\n+\t\t\treader_close(cur[i]);\n+\t\t\treftable_reader_free(cur[i]);\n+\t\t}\n+\t}\n+\n+done:\n+\tfor (i = 0; i < new_readers_len; i++) {\n+\t\treader_close(new_readers[i]);\n+\t\treftable_reader_free(new_readers[i]);\n+\t}\n+\treftable_free(new_readers);\n+\treftable_free(new_tables);\n+\treftable_free(cur);\n+\treturn err;\n+}\n+\n+/* return negative if a before b. */\n+static int tv_cmp(struct timeval *a, struct timeval *b)\n+{\n+\ttime_t diff = a->tv_sec - b->tv_sec;\n+\tint udiff = a->tv_usec - b->tv_usec;\n+\n+\tif (diff != 0)\n+\t\treturn diff;\n+\n+\treturn udiff;\n+}\n+\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open)\n+{\n+\tstruct timeval deadline = { 0 };\n+\tint err = gettimeofday(&deadline, NULL);\n+\tint64_t delay = 0;\n+\tint tries = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tdeadline.tv_sec += 3;\n+\twhile (1) {\n+\t\tchar **names = NULL;\n+\t\tchar **names_after = NULL;\n+\t\tstruct timeval now = { 0 };\n+\t\tint err = gettimeofday(&now, NULL);\n+\t\tint err2 = 0;\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* Only look at deadlines after the first few times. This\n+\t\t simplifies debugging in GDB */\n+\t\ttries++;\n+\t\tif (tries > 3 && tv_cmp(&now, &deadline) >= 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = read_lines(st->list_file, &names);\n+\t\tif (err < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\t\terr = reftable_stack_reload_once(st, names, reuse_open);\n+\t\tif (err == 0) {\n+\t\t\tfree_names(names);\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err != REFTABLE_NOT_EXIST_ERROR) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* err == REFTABLE_NOT_EXIST_ERROR can be caused by a concurrent\n+\t\t writer. Check if there was one by checking if the name list\n+\t\t changed.\n+\t\t*/\n+\t\terr2 = read_lines(st->list_file, &names_after);\n+\t\tif (err2 < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err2;\n+\t\t}\n+\n+\t\tif (names_equal(names_after, names)) {\n+\t\t\tfree_names(names);\n+\t\t\tfree_names(names_after);\n+\t\t\treturn err;\n+\t\t}\n+\t\tfree_names(names);\n+\t\tfree_names(names_after);\n+\n+\t\tdelay = delay + (delay * rand()) / RAND_MAX + 1;\n+\t\tsleep_millisec(delay);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/* -1 = error\n+ 0 = up to date\n+ 1 = changed. */\n+static int stack_uptodate(struct reftable_stack *st)\n+{\n+\tchar **names = NULL;\n+\tint err = read_lines(st->list_file, &names);\n+\tint i = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tfor (i = 0; i < st->readers_len; i++) {\n+\t\tif (names[i] == NULL) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (strcmp(st->readers[i]->name, names[i])) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (names[st->merged->stack_len] != NULL) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tfree_names(names);\n+\treturn err;\n+}\n+\n+int reftable_stack_reload(struct reftable_stack *st)\n+{\n+\tint err = stack_uptodate(st);\n+\tif (err > 0)\n+\t\treturn reftable_stack_reload_maybe_reuse(st, 1);\n+\treturn err;\n+}\n+\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write)(struct reftable_writer *wr, void *arg),\n+\t\t void *arg)\n+{\n+\tint err = stack_try_add(st, write, arg);\n+\tif (err < 0) {\n+\t\tif (err == REFTABLE_LOCK_ERROR) {\n+\t\t\t/* Ignore error return, we want to propagate\n+\t\t\t REFTABLE_LOCK_ERROR.\n+\t\t\t*/\n+\t\t\treftable_stack_reload(st);\n+\t\t}\n+\t\treturn err;\n+\t}\n+\n+\tif (!st->disable_auto_compact)\n+\t\treturn reftable_stack_auto_compact(st);\n+\n+\treturn 0;\n+}\n+\n+static void format_name(struct strbuf *dest, uint64_t min, uint64_t max)\n+{\n+\tchar buf[100];\n+\tuint32_t rnd = (uint32_t)rand();\n+\tsnprintf(buf, sizeof(buf), \"0x%012\" PRIx64 \"-0x%012\" PRIx64 \"-%08x\",\n+\t\t min, max, rnd);\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, buf);\n+}\n+\n+struct reftable_addition {\n+\tint lock_file_fd;\n+\tstruct strbuf lock_file_name;\n+\tstruct reftable_stack *stack;\n+\n+\tchar **new_tables;\n+\tint new_tables_len;\n+\tuint64_t next_update_index;\n+};\n+\n+#define REFTABLE_ADDITION_INIT \\\n+\t{ \\\n+\t\t.lock_file_name = STRBUF_INIT \\\n+\t}\n+\n+static int reftable_stack_init_addition(struct reftable_addition *add,\n+\t\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tadd->stack = st;\n+\n+\tstrbuf_reset(&add->lock_file_name);\n+\tstrbuf_addstr(&add->lock_file_name, st->list_file);\n+\tstrbuf_addstr(&add->lock_file_name, \".lock\");\n+\n+\tadd->lock_file_fd = open(add->lock_file_name.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (add->lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = REFTABLE_LOCK_ERROR;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\terr = stack_uptodate(st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 1) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->next_update_index = reftable_stack_next_update_index(st);\n+done:\n+\tif (err) {\n+\t\treftable_addition_close(add);\n+\t}\n+\treturn err;\n+}\n+\n+static void reftable_addition_close(struct reftable_addition *add)\n+{\n+\tint i = 0;\n+\tstruct strbuf nm = STRBUF_INIT;\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstrbuf_reset(&nm);\n+\t\tstrbuf_addstr(&nm, add->stack->reftable_dir);\n+\t\tstrbuf_addstr(&nm, \"/\");\n+\t\tstrbuf_addstr(&nm, add->new_tables[i]);\n+\t\tunlink(nm.buf);\n+\t\treftable_free(add->new_tables[i]);\n+\t\tadd->new_tables[i] = NULL;\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\tif (add->lock_file_fd > 0) {\n+\t\tclose(add->lock_file_fd);\n+\t\tadd->lock_file_fd = 0;\n+\t}\n+\tif (add->lock_file_name.len > 0) {\n+\t\tunlink(add->lock_file_name.buf);\n+\t\tstrbuf_release(&add->lock_file_name);\n+\t}\n+\n+\tstrbuf_release(&nm);\n+}\n+\n+void reftable_addition_destroy(struct reftable_addition *add)\n+{\n+\tif (add == NULL) {\n+\t\treturn;\n+\t}\n+\treftable_addition_close(add);\n+\treftable_free(add);\n+}\n+\n+int reftable_addition_commit(struct reftable_addition *add)\n+{\n+\tstruct strbuf table_list = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\tif (add->new_tables_len == 0)\n+\t\tgoto done;\n+\n+\tfor (i = 0; i < add->stack->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->stack->readers[i]->name);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->new_tables[i]);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\n+\terr = write(add->lock_file_fd, table_list.buf, table_list.len);\n+\tstrbuf_release(&table_list);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = close(add->lock_file_fd);\n+\tadd->lock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(add->lock_file_name.buf, add->stack->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\t/* success, no more state to clean up. */\n+\tstrbuf_release(&add->lock_file_name);\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\treftable_free(add->new_tables[i]);\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\terr = reftable_stack_reload(add->stack);\n+done:\n+\treftable_addition_close(add);\n+\treturn err;\n+}\n+\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n+\t*dest = reftable_calloc(sizeof(**dest));\n+\t**dest = empty;\n+\terr = reftable_stack_init_addition(*dest, st);\n+\tif (err) {\n+\t\treftable_free(*dest);\n+\t\t*dest = NULL;\n+\t}\n+\treturn err;\n+}\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct reftable_addition add = REFTABLE_ADDITION_INIT;\n+\tint err = reftable_stack_init_addition(&add, st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tif (err > 0) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_addition_add(&add, write_table, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_addition_commit(&add);\n+done:\n+\treftable_addition_close(&add);\n+\treturn err;\n+}\n+\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf tab_file_name = STRBUF_INIT;\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\tint tab_fd = 0;\n+\n+\tstrbuf_reset(&next_name);\n+\tformat_name(&next_name, add->next_update_index, add->next_update_index);\n+\n+\tstrbuf_addstr(&temp_tab_file_name, add->stack->reftable_dir);\n+\tstrbuf_addstr(&temp_tab_file_name, \"/\");\n+\tstrbuf_addbuf(&temp_tab_file_name, &next_name);\n+\tstrbuf_addstr(&temp_tab_file_name, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab_file_name.buf);\n+\tif (tab_fd < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd,\n+\t\t\t\t &add->stack->config);\n+\terr = write_table(wr, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_writer_close(wr);\n+\tif (err == REFTABLE_EMPTY_TABLE_ERROR) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = stack_check_addition(add->stack, temp_tab_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (wr->min_update_index < add->next_update_index) {\n+\t\terr = REFTABLE_API_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tformat_name(&next_name, wr->min_update_index, wr->max_update_index);\n+\tstrbuf_addstr(&next_name, \".ref\");\n+\n+\tstrbuf_addstr(&tab_file_name, add->stack->reftable_dir);\n+\tstrbuf_addstr(&tab_file_name, \"/\");\n+\tstrbuf_addbuf(&tab_file_name, &next_name);\n+\n+\t/*\n+\t On windows, this relies on rand() picking a unique destination name.\n+\t Maybe we should do retry loop as well?\n+\t */\n+\terr = rename(temp_tab_file_name.buf, tab_file_name.buf);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->new_tables = reftable_realloc(add->new_tables,\n+\t\t\t\t\t sizeof(*add->new_tables) *\n+\t\t\t\t\t\t (add->new_tables_len + 1));\n+\tadd->new_tables[add->new_tables_len] = strbuf_detach(&next_name, NULL);\n+\tadd->new_tables_len++;\n+done:\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (temp_tab_file_name.len > 0) {\n+\t\tunlink(temp_tab_file_name.buf);\n+\t}\n+\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&tab_file_name);\n+\tstrbuf_release(&next_name);\n+\treftable_writer_free(wr);\n+\treturn err;\n+}\n+\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st)\n+{\n+\tint sz = st->merged->stack_len;\n+\tif (sz > 0)\n+\t\treturn reftable_reader_max_update_index(st->readers[sz - 1]) +\n+\t\t 1;\n+\treturn 1;\n+}\n+\n+static int stack_compact_locked(struct reftable_stack *st, int first, int last,\n+\t\t\t\tstruct strbuf *temp_tab,\n+\t\t\t\tstruct reftable_log_expiry_config *config)\n+{\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tint tab_fd = -1;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\n+\tformat_name(&next_name,\n+\t\t reftable_reader_min_update_index(st->readers[first]),\n+\t\t reftable_reader_max_update_index(st->readers[last]));\n+\n+\tstrbuf_reset(temp_tab);\n+\tstrbuf_addstr(temp_tab, st->reftable_dir);\n+\tstrbuf_addstr(temp_tab, \"/\");\n+\tstrbuf_addbuf(temp_tab, &next_name);\n+\tstrbuf_addstr(temp_tab, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab->buf);\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd, &st->config);\n+\n+\terr = stack_write_compact(st, wr, first, last, config);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = reftable_writer_close(wr);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\n+done:\n+\treftable_writer_free(wr);\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (err != 0 && temp_tab->len > 0) {\n+\t\tunlink(temp_tab->buf);\n+\t\tstrbuf_release(temp_tab);\n+\t}\n+\tstrbuf_release(&next_name);\n+\treturn err;\n+}\n+\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint subtabs_len = last - first + 1;\n+\tstruct reftable_table *subtabs = reftable_calloc(\n+\t\tsizeof(struct reftable_table) * (last - first + 1));\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tuint64_t entries = 0;\n+\n+\tint i = 0, j = 0;\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct reftable_reader *t = st->readers[i];\n+\t\treftable_table_from_reader(&subtabs[j++], t);\n+\t\tst->stats.bytes += t->size;\n+\t}\n+\treftable_writer_set_limits(wr, st->readers[first]->min_update_index,\n+\t\t\t\t st->readers[last]->max_update_index);\n+\n+\terr = reftable_new_merged_table(&mt, subtabs, subtabs_len,\n+\t\t\t\t\tst->config.hash_id);\n+\tif (err < 0) {\n+\t\treftable_free(subtabs);\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_merged_table_seek_ref(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (first == 0 && reftable_ref_record_is_deletion(&ref)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_ref(wr, &ref);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_merged_table_seek_log(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (first == 0 && reftable_log_record_is_deletion(&log)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config != NULL && config->min_update_index > 0 &&\n+\t\t log.update_index < config->min_update_index) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config != NULL && config->time > 0 &&\n+\t\t log.update.time < config->time) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_log(wr, &log);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\tif (mt != NULL) {\n+\t\tmerged_table_release(mt);\n+\t\treftable_merged_table_free(mt);\n+\t}\n+\treftable_ref_record_release(&ref);\n+\treftable_log_record_release(&log);\n+\tst->stats.entries_written += entries;\n+\treturn err;\n+}\n+\n+/* < 0: error. 0 == OK, > 0 attempt failed; could retry. */\n+static int stack_compact_range(struct reftable_stack *st, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *expiry)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf new_table_name = STRBUF_INIT;\n+\tstruct strbuf lock_file_name = STRBUF_INIT;\n+\tstruct strbuf ref_list_contents = STRBUF_INIT;\n+\tstruct strbuf new_table_path = STRBUF_INIT;\n+\tint err = 0;\n+\tint have_lock = 0;\n+\tint lock_file_fd = 0;\n+\tint compact_count = last - first + 1;\n+\tchar **listp = NULL;\n+\tchar **delete_on_success =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tchar **subtable_locks =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tint i = 0;\n+\tint j = 0;\n+\tint is_empty_table = 0;\n+\n+\tif (first > last || (expiry == NULL && first == last)) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\tst->stats.attempts++;\n+\n+\tstrbuf_reset(&lock_file_name);\n+\tstrbuf_addstr(&lock_file_name, st->list_file);\n+\tstrbuf_addstr(&lock_file_name, \".lock\");\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\t/* Don't want to write to the lock for now. */\n+\tclose(lock_file_fd);\n+\tlock_file_fd = 0;\n+\n+\thave_lock = 1;\n+\terr = stack_uptodate(st);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct strbuf subtab_file_name = STRBUF_INIT;\n+\t\tstruct strbuf subtab_lock = STRBUF_INIT;\n+\t\tint sublock_file_fd = -1;\n+\n+\t\tstrbuf_addstr(&subtab_file_name, st->reftable_dir);\n+\t\tstrbuf_addstr(&subtab_file_name, \"/\");\n+\t\tstrbuf_addstr(&subtab_file_name, reader_name(st->readers[i]));\n+\n+\t\tstrbuf_reset(&subtab_lock);\n+\t\tstrbuf_addbuf(&subtab_lock, &subtab_file_name);\n+\t\tstrbuf_addstr(&subtab_lock, \".lock\");\n+\n+\t\tsublock_file_fd = open(subtab_lock.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\t\tif (sublock_file_fd > 0) {\n+\t\t\tclose(sublock_file_fd);\n+\t\t} else if (sublock_file_fd < 0) {\n+\t\t\tif (errno == EEXIST) {\n+\t\t\t\terr = 1;\n+\t\t\t} else {\n+\t\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\t}\n+\t\t}\n+\n+\t\tsubtable_locks[j] = subtab_lock.buf;\n+\t\tdelete_on_success[j] = subtab_file_name.buf;\n+\t\tj++;\n+\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\terr = unlink(lock_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\thave_lock = 0;\n+\n+\terr = stack_compact_locked(st, first, last, &temp_tab_file_name,\n+\t\t\t\t expiry);\n+\t/* Compaction + tombstones can create an empty table out of non-empty\n+\t * tables. */\n+\tis_empty_table = (err == REFTABLE_EMPTY_TABLE_ERROR);\n+\tif (is_empty_table) {\n+\t\terr = 0;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\thave_lock = 1;\n+\n+\tformat_name(&new_table_name, st->readers[first]->min_update_index,\n+\t\t st->readers[last]->max_update_index);\n+\tstrbuf_addstr(&new_table_name, \".ref\");\n+\n+\tstrbuf_reset(&new_table_path);\n+\tstrbuf_addstr(&new_table_path, st->reftable_dir);\n+\tstrbuf_addstr(&new_table_path, \"/\");\n+\tstrbuf_addbuf(&new_table_path, &new_table_name);\n+\n+\tif (!is_empty_table) {\n+\t\t/* retry? */\n+\t\terr = rename(temp_tab_file_name.buf, new_table_path.buf);\n+\t\tif (err < 0) {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < first; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tif (!is_empty_table) {\n+\t\tstrbuf_addbuf(&ref_list_contents, &new_table_name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tfor (i = last + 1; i < st->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\n+\terr = write(lock_file_fd, ref_list_contents.buf, ref_list_contents.len);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\terr = close(lock_file_fd);\n+\tlock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(lock_file_name.buf, st->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\thave_lock = 0;\n+\n+\t/* Reload the stack before deleting. On windows, we can only delete the\n+\t files after we closed them.\n+\t*/\n+\terr = reftable_stack_reload_maybe_reuse(st, first < last);\n+\n+\tlistp = delete_on_success;\n+\twhile (*listp) {\n+\t\tif (strcmp(*listp, new_table_path.buf)) {\n+\t\t\tunlink(*listp);\n+\t\t}\n+\t\tlistp++;\n+\t}\n+\n+done:\n+\tfree_names(delete_on_success);\n+\n+\tlistp = subtable_locks;\n+\twhile (*listp) {\n+\t\tunlink(*listp);\n+\t\tlistp++;\n+\t}\n+\tfree_names(subtable_locks);\n+\tif (lock_file_fd > 0) {\n+\t\tclose(lock_file_fd);\n+\t\tlock_file_fd = 0;\n+\t}\n+\tif (have_lock) {\n+\t\tunlink(lock_file_name.buf);\n+\t}\n+\tstrbuf_release(&new_table_name);\n+\tstrbuf_release(&new_table_path);\n+\tstrbuf_release(&ref_list_contents);\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&lock_file_name);\n+\treturn err;\n+}\n+\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\treturn stack_compact_range(st, 0, st->merged->stack_len - 1, config);\n+}\n+\n+static int stack_compact_range_stats(struct reftable_stack *st, int first,\n+\t\t\t\t int last,\n+\t\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint err = stack_compact_range(st, first, last, config);\n+\tif (err > 0) {\n+\t\tst->stats.failures++;\n+\t}\n+\treturn err;\n+}\n+\n+static int segment_size(struct segment *s)\n+{\n+\treturn s->end - s->start;\n+}\n+\n+int fastlog2(uint64_t sz)\n+{\n+\tint l = 0;\n+\tif (sz == 0)\n+\t\treturn 0;\n+\tfor (; sz; sz /= 2) {\n+\t\tl++;\n+\t}\n+\treturn l - 1;\n+}\n+\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n+{\n+\tstruct segment *segs = reftable_calloc(sizeof(struct segment) * n);\n+\tint next = 0;\n+\tstruct segment cur = { 0 };\n+\tint i = 0;\n+\n+\tif (n == 0) {\n+\t\t*seglen = 0;\n+\t\treturn segs;\n+\t}\n+\tfor (i = 0; i < n; i++) {\n+\t\tint log = fastlog2(sizes[i]);\n+\t\tif (cur.log != log && cur.bytes > 0) {\n+\t\t\tstruct segment fresh = {\n+\t\t\t\t.start = i,\n+\t\t\t};\n+\n+\t\t\tsegs[next++] = cur;\n+\t\t\tcur = fresh;\n+\t\t}\n+\n+\t\tcur.log = log;\n+\t\tcur.end = i + 1;\n+\t\tcur.bytes += sizes[i];\n+\t}\n+\tsegs[next++] = cur;\n+\t*seglen = next;\n+\treturn segs;\n+}\n+\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, sizes, n);\n+\tstruct segment min_seg = {\n+\t\t.log = 64,\n+\t};\n+\tint i = 0;\n+\tfor (i = 0; i < seglen; i++) {\n+\t\tif (segment_size(&segs[i]) == 1) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (segs[i].log < min_seg.log) {\n+\t\t\tmin_seg = segs[i];\n+\t\t}\n+\t}\n+\n+\twhile (min_seg.start > 0) {\n+\t\tint prev = min_seg.start - 1;\n+\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmin_seg.start = prev;\n+\t\tmin_seg.bytes += sizes[prev];\n+\t}\n+\n+\treftable_free(segs);\n+\treturn min_seg;\n+}\n+\n+static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes =\n+\t\treftable_calloc(sizeof(uint64_t) * st->merged->stack_len);\n+\tint version = (st->config.hash_id == SHA1_ID) ? 1 : 2;\n+\tint overhead = header_size(version) - 1;\n+\tint i = 0;\n+\tfor (i = 0; i < st->merged->stack_len; i++) {\n+\t\tsizes[i] = st->readers[i]->size - overhead;\n+\t}\n+\treturn sizes;\n+}\n+\n+int reftable_stack_auto_compact(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n+\tstruct segment seg =\n+\t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n+\treftable_free(sizes);\n+\tif (segment_size(&seg) > 0)\n+\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n+\t\t\t\t\t\t NULL);\n+\n+\treturn 0;\n+}\n+\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st)\n+{\n+\treturn &st->stats;\n+}\n+\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_table tab = { NULL };\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\treturn reftable_table_read_ref(&tab, refname, ref);\n+}\n+\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(st);\n+\tint err = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_log(&it, log);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(log->refname, refname) ||\n+\t reftable_log_record_is_deletion(log)) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tif (err) {\n+\t\treftable_log_record_release(log);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name)\n+{\n+\tint err = 0;\n+\tstruct reftable_block_source src = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct reftable_ref_record *refs = NULL;\n+\tstruct reftable_iterator it = { NULL };\n+\tint cap = 0;\n+\tint len = 0;\n+\tint i = 0;\n+\n+\tif (st->config.skip_name_check)\n+\t\treturn 0;\n+\n+\terr = reftable_block_source_from_file(&src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&rd, &src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tif (err > 0) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (len >= cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\trefs = reftable_realloc(refs, cap * sizeof(refs[0]));\n+\t\t}\n+\n+\t\trefs[len++] = ref;\n+\t}\n+\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\n+\terr = validate_ref_record_addition(tab, refs, len);\n+\n+done:\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t}\n+\n+\tfree(refs);\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\treturn err;\n+}\ndiff --git a/reftable/stack.h b/reftable/stack.h\nnew file mode 100644\nindex 000000000000..f57005846e56\n--- /dev/null\n+++ b/reftable/stack.h\n@@ -0,0 +1,41 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef STACK_H\n+#define STACK_H\n+\n+#include \"system.h\"\n+#include \"reftable-writer.h\"\n+#include \"reftable-stack.h\"\n+\n+struct reftable_stack {\n+\tchar *list_file;\n+\tchar *reftable_dir;\n+\tint disable_auto_compact;\n+\n+\tstruct reftable_write_options config;\n+\n+\tstruct reftable_reader **readers;\n+\tsize_t readers_len;\n+\tstruct reftable_merged_table *merged;\n+\tstruct reftable_compaction_stats stats;\n+};\n+\n+int read_lines(const char *filename, char ***lines);\n+\n+struct segment {\n+\tint start, end;\n+\tint log;\n+\tuint64_t bytes;\n+};\n+\n+int fastlog2(uint64_t sz);\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n);\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n);\n+\n+#endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nnew file mode 100644\nindex 000000000000..c215b61e0a16\n--- /dev/null\n+++ b/reftable/stack_test.c\n@@ -0,0 +1,809 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+\n+#include \"merged.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+#include \n+#include \n+\n+static void clear_dir(const char *dirname)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstrbuf_addstr(&path, dirname);\n+\tremove_dir_recursively(&path, 0);\n+\tstrbuf_release(&path);\n+}\n+\n+static char *get_tmp_template(const char *prefix)\n+{\n+\tconst char *tmp = getenv(\"TMPDIR\");\n+\tstatic char template[1024];\n+\tsnprintf(template, sizeof(template) - 1, \"%s/%s.XXXXXX\",\n+\t\t tmp ? tmp : \"/tmp\", prefix);\n+\treturn template;\n+}\n+\n+static void test_read_file(void)\n+{\n+\tchar *fn = get_tmp_template(__FUNCTION__);\n+\tint fd = mkstemp(fn);\n+\tchar out[1024] = \"line1\\n\\nline2\\nline3\";\n+\tint n, err;\n+\tchar **names = NULL;\n+\tchar *want[] = { \"line1\", \"line2\", \"line3\" };\n+\tint i = 0;\n+\n+\tEXPECT(fd > 0);\n+\tn = write(fd, out, strlen(out));\n+\tEXPECT(n == strlen(out));\n+\terr = close(fd);\n+\tEXPECT(err >= 0);\n+\n+\terr = read_lines(fn, &names);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; names[i] != NULL; i++) {\n+\t\tEXPECT(0 == strcmp(want[i], names[i]));\n+\t}\n+\tfree_names(names);\n+\tremove(fn);\n+}\n+\n+static void test_parse_names(void)\n+{\n+\tchar buf[] = \"line\\n\";\n+\tchar **names = NULL;\n+\tparse_names(buf, strlen(buf), &names);\n+\n+\tEXPECT(NULL != names[0]);\n+\tEXPECT(0 == strcmp(names[0], \"line\"));\n+\tEXPECT(NULL == names[1]);\n+\tfree_names(names);\n+}\n+\n+static void test_names_equal(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", \"c\", NULL };\n+\tchar *b[] = { \"a\", \"b\", \"d\", NULL };\n+\tchar *c[] = { \"a\", \"b\", NULL };\n+\n+\tEXPECT(names_equal(a, a));\n+\tEXPECT(!names_equal(a, b));\n+\tEXPECT(!names_equal(a, c));\n+}\n+\n+static int write_test_ref(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct reftable_ref_record *ref = arg;\n+\treftable_writer_set_limits(wr, ref->update_index, ref->update_index);\n+\treturn reftable_writer_add_ref(wr, ref);\n+}\n+\n+struct write_log_arg {\n+\tstruct reftable_log_record *log;\n+\tuint64_t update_index;\n+};\n+\n+static int write_test_log(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct write_log_arg *wla = arg;\n+\n+\treftable_writer_set_limits(wr, wla->update_index, wla->update_index);\n+\treturn reftable_writer_add_log(wr, wla->log);\n+}\n+\n+static void test_reftable_stack_add_one(void)\n+{\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(\"master\", dest.value.symref));\n+\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_uptodate(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL;\n+\tstruct reftable_stack *st2 = NULL;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"branch2\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\t/* simulate multi-process access to the same stack\n+\t by creating two stacks for the same directory.\n+\t */\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st1, &write_test_ref, &ref1);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tEXPECT(err == REFTABLE_LOCK_ERROR);\n+\n+\terr = reftable_stack_reload(st2);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tEXPECT_ERR(err);\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_transaction_api(void)\n+{\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_addition *add = NULL;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_new_addition(&add, st);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_addition_add(add, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_addition_commit(add);\n+\tEXPECT_ERR(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(REFTABLE_REF_SYMREF == dest.value_type);\n+\tEXPECT(0 == strcmp(\"master\", dest.value.symref));\n+\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_validate_refname(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tint i;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"a/b\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tchar *additions[] = { \"a\", \"a/b/c\" };\n+\n+\tEXPECT(mkdtemp(dir));\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(additions); i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = additions[i],\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tEXPECT(err == REFTABLE_NAME_CONFLICT);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static int write_error(struct reftable_writer *wr, void *arg)\n+{\n+\treturn *((int *)arg);\n+}\n+\n+static void test_reftable_stack_update_index_check(void)\n+{\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"name1\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"name2\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref1);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref2);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_lock_failure(void)\n+{\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err, i;\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tfor (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) {\n+\t\terr = reftable_stack_add(st, &write_error, &i);\n+\t\tEXPECT(err == i);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_add(void)\n+{\n+\tint i = 0;\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t.exact_log_message = 1,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tst->disable_auto_compact = 1;\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n+\t\trefs[i].value.val1 = reftable_malloc(SHA1_SIZE);\n+\t\tset_test_hash(refs[i].value.val1, i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = N + i + 1;\n+\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\n+\t\tlogs[i].update.new_hash = reftable_malloc(SHA1_SIZE);\n+\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].update.new_hash, i);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_ref_record dest = { NULL };\n+\n+\t\tint err = reftable_stack_read_ref(st, refs[i].refname, &dest);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(reftable_ref_record_equal(&dest, refs + i, SHA1_SIZE));\n+\t\treftable_ref_record_release(&dest);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_log_record dest = { NULL };\n+\t\tint err = reftable_stack_read_log(st, refs[i].refname, &dest);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(reftable_log_record_equal(&dest, logs + i, SHA1_SIZE));\n+\t\treftable_log_record_release(&dest);\n+\t}\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_log_normalize(void)\n+{\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t0,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\n+\tuint8_t h1[SHA1_SIZE] = { 0x01 }, h2[SHA1_SIZE] = { 0x02 };\n+\n+\tstruct reftable_log_record input = { .refname = \"branch\",\n+\t\t\t\t\t .update_index = 1,\n+\t\t\t\t\t .value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t\t\t .update = {\n+\t\t\t\t\t\t .new_hash = h1,\n+\t\t\t\t\t\t .old_hash = h2,\n+\t\t\t\t\t } };\n+\tstruct reftable_log_record dest = {\n+\t\t.update_index = 0,\n+\t};\n+\tstruct write_log_arg arg = {\n+\t\t.log = &input,\n+\t\t.update_index = 1,\n+\t};\n+\n+\tEXPECT(mkdtemp(dir));\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tinput.update.message = \"one\\ntwo\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\n+\tinput.update.message = \"one\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(dest.update.message, \"one\\n\"));\n+\n+\tinput.update.message = \"two\\n\";\n+\targ.update_index = 2;\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT_ERR(err);\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(dest.update.message, \"two\\n\"));\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\treftable_log_record_release(&dest);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_tombstone(void)\n+{\n+\tint i = 0;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\tstruct reftable_ref_record dest = { NULL };\n+\tstruct reftable_log_record log_dest = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tconst char *buf = \"branch\";\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\tif (i % 2 == 0) {\n+\t\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n+\t\t\trefs[i].value.val1 = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_test_hash(refs[i].value.val1, i);\n+\t\t}\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\t/* update_index is part of the key. */\n+\t\tlogs[i].update_index = 42;\n+\t\tif (i % 2 == 0) {\n+\t\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\t\t\tlogs[i].update.new_hash = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_test_hash(logs[i].update.new_hash, i);\n+\t\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n+\t\t}\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tEXPECT(err == 1);\n+\treftable_ref_record_release(&dest);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tEXPECT(err == 1);\n+\treftable_log_record_release(&log_dest);\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tEXPECT(err == 1);\n+\treftable_ref_record_release(&dest);\n+\treftable_log_record_release(&log_dest);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_hash_id(void)\n+{\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"master\",\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"target\",\n+\t\t.update_index = 1,\n+\t};\n+\tstruct reftable_write_options cfg32 = { .hash_id = SHA256_ID };\n+\tstruct reftable_stack *st32 = NULL;\n+\tstruct reftable_write_options cfg_default = { 0 };\n+\tstruct reftable_stack *st_default = NULL;\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\t/* can't read it with the wrong hash ID. */\n+\terr = reftable_new_stack(&st32, dir, cfg32);\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+\n+\t/* check that we can read it back with default config too. */\n+\terr = reftable_new_stack(&st_default, dir, cfg_default);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n+\tEXPECT_ERR(err);\n+\n+\tEXPECT(reftable_ref_record_equal(&ref, &dest, SHA1_SIZE));\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st_default);\n+\tclear_dir(dir);\n+}\n+\n+static void test_log2(void)\n+{\n+\tEXPECT(1 == fastlog2(3));\n+\tEXPECT(2 == fastlog2(4));\n+\tEXPECT(2 == fastlog2(5));\n+}\n+\n+static void test_sizes_to_segments(void)\n+{\n+\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n+\t/* .................0 1 2 3 4 5 */\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(segs[2].log == 3);\n+\tEXPECT(segs[2].start == 5);\n+\tEXPECT(segs[2].end == 6);\n+\n+\tEXPECT(segs[1].log == 2);\n+\tEXPECT(segs[1].start == 2);\n+\tEXPECT(segs[1].end == 5);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_empty(void)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n+\tEXPECT(seglen == 0);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_all_equal(void)\n+{\n+\tuint64_t sizes[] = { 5, 5 };\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(seglen == 1);\n+\tEXPECT(segs[0].start == 0);\n+\tEXPECT(segs[0].end == 2);\n+\treftable_free(segs);\n+}\n+\n+static void test_suggest_compaction_segment(void)\n+{\n+\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n+\t/* .................0 1 2 3 4 5 6 */\n+\tstruct segment min =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(min.start == 2);\n+\tEXPECT(min.end == 7);\n+}\n+\n+static void test_suggest_compaction_segment_nothing(void)\n+{\n+\tuint64_t sizes[] = { 64, 32, 16, 8, 4, 2 };\n+\tstruct segment result =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(result.start == result.end);\n+}\n+\n+static void test_reflog_expire(void)\n+{\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tstruct reftable_log_record logs[20] = { { NULL } };\n+\tint N = ARRAY_SIZE(logs) - 1;\n+\tint i = 0;\n+\tint err;\n+\tstruct reftable_log_expiry_config expiry = {\n+\t\t.time = 10,\n+\t};\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = i;\n+\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\t\tlogs[i].update.time = i;\n+\t\tlogs[i].update.new_hash = reftable_malloc(SHA1_SIZE);\n+\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].update.new_hash, i);\n+\t}\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, logs[9].refname, &log);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[11].refname, &log);\n+\tEXPECT_ERR(err);\n+\n+\texpiry.min_update_index = 15;\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, logs[14].refname, &log);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[16].refname, &log);\n+\tEXPECT_ERR(err);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i <= N; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+\treftable_log_record_release(&log);\n+}\n+\n+static int write_nothing(struct reftable_writer *wr, void *arg)\n+{\n+\treftable_writer_set_limits(wr, 1, 1);\n+\treturn 0;\n+}\n+\n+static void test_empty_add(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_stack *st2 = NULL;\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_nothing, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tclear_dir(dir);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st2);\n+}\n+\n+static void test_reftable_stack_auto_compaction(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tint err, i;\n+\tint N = 100;\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\n+\t\tEXPECT(i < 3 || st->merged->stack_len < 2 * fastlog2(i));\n+\t}\n+\n+\tEXPECT(reftable_stack_compaction_stats(st)->entries_written <\n+\t (uint64_t)(N * fastlog2(N)));\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+int stack_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_reftable_stack_uptodate);\n+\tRUN_TEST(test_reftable_stack_transaction_api);\n+\tRUN_TEST(test_reftable_stack_hash_id);\n+\tRUN_TEST(test_sizes_to_segments_all_equal);\n+\tRUN_TEST(test_reftable_stack_auto_compaction);\n+\tRUN_TEST(test_reftable_stack_validate_refname);\n+\tRUN_TEST(test_reftable_stack_update_index_check);\n+\tRUN_TEST(test_reftable_stack_lock_failure);\n+\tRUN_TEST(test_reftable_stack_log_normalize);\n+\tRUN_TEST(test_reftable_stack_tombstone);\n+\tRUN_TEST(test_reftable_stack_add_one);\n+\tRUN_TEST(test_empty_add);\n+\tRUN_TEST(test_reflog_expire);\n+\tRUN_TEST(test_suggest_compaction_segment);\n+\tRUN_TEST(test_suggest_compaction_segment_nothing);\n+\tRUN_TEST(test_sizes_to_segments);\n+\tRUN_TEST(test_sizes_to_segments_empty);\n+\tRUN_TEST(test_log2);\n+\tRUN_TEST(test_parse_names);\n+\tRUN_TEST(test_read_file);\n+\tRUN_TEST(test_names_equal);\n+\tRUN_TEST(test_reftable_stack_add);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex fdf925867375..3b702f4855e6 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -5,8 +5,11 @@ int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n+\tmerged_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\trefname_test_main(argc, argv);\n \treftable_test_main(argc, argv);\n+\tstack_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"419999","messageId":"e754900a-1d5d-d4cc-aba6-737b37d66f74@gmail.com","threadId":"54245","inReplyTo":"bdb19af22cc7c4f3383f03f42cb4906c3ec5c5f3.1615580397.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 13/15] Reftable support for git-core","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-03-23T11:40:24Z","receivedAt":"2021-03-23T11:41:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/12/2021 3:19 PM, Han-Wen Nienhuys via GitGitGadget wrote:\n> diff --git a/Documentation/config/extensions.txt b/Documentation/config/extensions.txt\n> index 4e23d73cdcad..82c5940f1434 100644\n> --- a/Documentation/config/extensions.txt\n> +++ b/Documentation/config/extensions.txt\n> @@ -6,3 +6,12 @@ extensions.objectFormat::\n> Note that this setting should only be set by linkgit:git-init[1] or\n> linkgit:git-clone[1]. Trying to change it after initialization will not\n> work and will produce hard-to-diagnose issues.\n> ++\n\nI noticed while resolving conflicts with my series, which also edits this\nfile, that the \"+\" line above should be removed. That likely munges the\nfact that the config entry below should be its own list item, not a\ncontinuation of the previous one.\n\n> +extensions.refStorage::\n> +\tSpecify the ref storage mechanism to use. The acceptable values are `files` and\n> +\t`reftable`. If not specified, `files` is assumed. It is an error to specify\n> +\tthis key unless `core.repositoryFormatVersion` is 1.\n> ++\n> +Note that this setting should only be set by linkgit:git-init[1] or\n> +linkgit:git-clone[1]. Trying to change it after initialization will not\n> +work and will produce hard-to-diagnose issues.\n\nThe only other conflict is that we are trying to write at the end of\nthe file. extensions.sparseIndex should go after this option.\n\nThanks,\n-Stolee\n"},{"id":"420000","messageId":"8735wmqcx5.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"e754900a-1d5d-d4cc-aba6-737b37d66f74@gmail.com","subject":"Re: [PATCH v5 13/15] Reftable support for git-core","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-03-23T12:20:38Z","receivedAt":"2021-03-23T12:21:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 23 2021, Derrick Stolee wrote:\n\n> On 3/12/2021 3:19 PM, Han-Wen Nienhuys via GitGitGadget wrote:\n>> diff --git a/Documentation/config/extensions.txt b/Documentation/config/extensions.txt\n>> index 4e23d73cdcad..82c5940f1434 100644\n>> --- a/Documentation/config/extensions.txt\n>> +++ b/Documentation/config/extensions.txt\n>> @@ -6,3 +6,12 @@ extensions.objectFormat::\n>> Note that this setting should only be set by linkgit:git-init[1] or\n>> linkgit:git-clone[1]. Trying to change it after initialization will not\n>> work and will produce hard-to-diagnose issues.\n>> ++\n>\n> I noticed while resolving conflicts with my series, which also edits this\n> file, that the \"+\" line above should be removed. That likely munges the\n> fact that the config entry below should be its own list item, not a\n> continuation of the previous one.\n\nI haven't tested this patch, but just a plug for the very useful\nDocumentation/doc-diff for discovering any such formatting errors when\nmaking non-trivial doc changes.\n"},{"id":"420059","messageId":"xmqq4kh139z3.fsf@gitster.g","threadId":"54245","inReplyTo":"e754900a-1d5d-d4cc-aba6-737b37d66f74@gmail.com","subject":"Re: [PATCH v5 13/15] Reftable support for git-core","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-03-23T20:12:48Z","receivedAt":"2021-03-23T20:13:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 3/12/2021 3:19 PM, Han-Wen Nienhuys via GitGitGadget wrote:\n>> diff --git a/Documentation/config/extensions.txt b/Documentation/config/extensions.txt\n>> index 4e23d73cdcad..82c5940f1434 100644\n>> --- a/Documentation/config/extensions.txt\n>> +++ b/Documentation/config/extensions.txt\n>> @@ -6,3 +6,12 @@ extensions.objectFormat::\n>> Note that this setting should only be set by linkgit:git-init[1] or\n>> linkgit:git-clone[1]. Trying to change it after initialization will not\n>> work and will produce hard-to-diagnose issues.\n>> ++\n>\n> I noticed while resolving conflicts with my series, which also edits this\n> file, that the \"+\" line above should be removed. That likely munges the\n> fact that the config entry below should be its own list item, not a\n> continuation of the previous one.\n\nYup, I noticed them, too, while merging. If I recall correctly,\neven though using a blank line to separate the list items is\nsemantically more correct, using the plus-alone line produced either\nidentical, or visually indistinguishable, rendering when I tested.\n"},{"id":"420060","messageId":"xmqqzgyt1vb8.fsf@gitster.g","threadId":"54245","inReplyTo":"8735wmqcx5.fsf@evledraar.gmail.com","subject":"Re: [PATCH v5 13/15] Reftable support for git-core","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-03-23T20:14:51Z","receivedAt":"2021-03-23T20:15:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Tue, Mar 23 2021, Derrick Stolee wrote:\n>\n>> On 3/12/2021 3:19 PM, Han-Wen Nienhuys via GitGitGadget wrote:\n>>> diff --git a/Documentation/config/extensions.txt b/Documentation/config/extensions.txt\n>>> index 4e23d73cdcad..82c5940f1434 100644\n>>> --- a/Documentation/config/extensions.txt\n>>> +++ b/Documentation/config/extensions.txt\n>>> @@ -6,3 +6,12 @@ extensions.objectFormat::\n>>> Note that this setting should only be set by linkgit:git-init[1] or\n>>> linkgit:git-clone[1]. Trying to change it after initialization will not\n>>> work and will produce hard-to-diagnose issues.\n>>> ++\n>>\n>> I noticed while resolving conflicts with my series, which also edits this\n>> file, that the \"+\" line above should be removed. That likely munges the\n>> fact that the config entry below should be its own list item, not a\n>> continuation of the previous one.\n>\n> I haven't tested this patch, but just a plug for the very useful\n> Documentation/doc-diff for discovering any such formatting errors when\n> making non-trivial doc changes.\n\nYup, doc-diff is a useful tool to try. I recall however in this\nparticular case there was no difference in the rendered output, even\nthough a blank line that separates two enumerated items is\nsemantically more correct than a plus-alone line that adds another\nparagraph to the single item.\n\n"},{"id":"421753","messageId":"15b8306fbe4b86f8c6b8b26b76f6e65944a4a4dc.1618255552.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 01/20] init-db: set the_repository->hash_algo early on","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:33Z","receivedAt":"2021-04-12T19:26:01Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable backend needs to know the hash algorithm for writing the\ninitialization hash table.\n\nThe initial reftable contains a symref HEAD => \"main\" (or \"master\"), which is\nagnostic to the size of hash value, but this is an exceptional circumstance, and\nthe reftable library does not cater to this exception. It insists that all\ntables in the stack have a consistent format ID for the hash algorithm.\n\nCall set_repo_hash_algo directly after calling validate_hash_algorithm() (which\nreads $GIT_DEFAULT_HASH).\n\nHelped-by: Junio C Hamano \nSigned-off-by: Han-Wen Nienhuys \n---\n builtin/init-db.c | 21 +++++++++++++++++++++\n 1 file changed, 21 insertions(+)\n\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex c19b35f1e691..ff06fe4b9a49 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -424,6 +424,27 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \n \tvalidate_hash_algorithm(&repo_fmt, hash);\n \n+\t/*\n+\t * At this point, the_repository we have in-core does not look\n+\t * anything like one that we would see initialized in an already\n+\t * working repository after calling setup_git_directory().\n+\t *\n+\t * Calling repository.c::initialize_the_repository() may have\n+\t * prepared the .index .objects and .parsed_objects members, but\n+\t * other members like .gitdir, .commondir, etc. have not been\n+\t * initialized.\n+\t *\n+\t * Many API functions assume they are working with the_repository\n+\t * that has sensibly been initialized, but because we haven't\n+\t * really read from an existing repository, we need to hand-craft\n+\t * the necessary members of the structure to get out of this\n+\t * chicken-and-egg situation.\n+\t *\n+\t * For now, we update the hash algorithm member to what the\n+\t * validate_hash_algorithm() call decided for us.\n+\t */\n+\trepo_set_hash_algo(the_repository, repo_fmt.hash_algo);\n+\n \treinit = create_default_files(template_dir, original_git_dir,\n \t\t\t\t initial_branch, &repo_fmt,\n \t\t\t\t flags & INIT_DB_QUIET);\n-- \ngitgitgadget\n\n"},{"id":"421755","messageId":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v5.git.git.1615580397.gitgitgadget@gmail.com","subject":"[PATCH v6 00/20] reftable library","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:32Z","receivedAt":"2021-04-12T19:26:04Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"This splits the giant commit from\nhttps://github.com/gitgitgadget/git/pull/539 into a series of smaller\ncommits, which build and have unittests.\n\nversion 12 Apr 2021:\n\n * handle cleanup for concurrent access on Windows\n * split up commits further\n\nBased on github.com/hanwen/reftable at\nc33a40ad23dd622d8c72139c41a2be1cfa063e5f Add random suffix to table names\n\nHan-Wen Nienhuys (19):\n init-db: set the_repository->hash_algo early on\n reftable: add LICENSE\n reftable: add error related functionality\n reftable: utility functions\n reftable: add blocksource, an abstraction for random access reads\n reftable: (de)serialization for the polymorphic record type.\n reftable: reading/writing blocks\n reftable: a generic binary tree implementation\n reftable: write reftable files\n reftable: generic interface to tables\n reftable: read reftable files\n reftable: reftable file level tests\n reftable: add a heap-based priority queue for reftable records\n reftable: add merged table view\n reftable: implement refname validation\n reftable: implement stack, a mutable database of reftable files.\n reftable: add dump utility\n Reftable support for git-core\n Add \"test-tool dump-reftable\" command.\n\nSZEDER Gábor (1):\n git-prompt: prepare for reftable refs backend\n\n Documentation/config/extensions.txt | 9 +\n .../technical/repository-version.txt | 7 +\n Makefile | 48 +-\n builtin/clone.c | 5 +-\n builtin/init-db.c | 76 +-\n builtin/worktree.c | 27 +-\n cache.h | 8 +-\n config.mak.uname | 2 +-\n contrib/buildsystems/CMakeLists.txt | 14 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n contrib/completion/git-prompt.sh | 7 +-\n refs.c | 27 +-\n refs.h | 3 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1451 +++++++++++++++++\n reftable/.gitattributes | 1 +\n reftable/LICENSE | 31 +\n reftable/basics.c | 128 ++\n reftable/basics.h | 60 +\n reftable/basics_test.c | 98 ++\n reftable/block.c | 440 +++++\n reftable/block.h | 127 ++\n reftable/block_test.c | 121 ++\n reftable/blocksource.c | 148 ++\n reftable/blocksource.h | 22 +\n reftable/constants.h | 21 +\n reftable/dump.c | 213 +++\n reftable/error.c | 41 +\n reftable/generic.h | 32 +\n reftable/iter.c | 197 +++\n reftable/iter.h | 67 +\n reftable/merged.c | 367 +++++\n reftable/merged.h | 35 +\n reftable/merged_test.c | 291 ++++\n reftable/pq.c | 115 ++\n reftable/pq.h | 32 +\n reftable/pq_test.c | 72 +\n reftable/publicbasics.c | 58 +\n reftable/reader.c | 773 +++++++++\n reftable/reader.h | 66 +\n reftable/record.c | 1203 ++++++++++++++\n reftable/record.h | 139 ++\n reftable/record_test.c | 405 +++++\n reftable/refname.c | 209 +++\n reftable/refname.h | 29 +\n reftable/refname_test.c | 102 ++\n reftable/reftable-blocksource.h | 49 +\n reftable/reftable-error.h | 62 +\n reftable/reftable-generic.h | 41 +\n reftable/reftable-iterator.h | 39 +\n reftable/reftable-malloc.h | 18 +\n reftable/reftable-merged.h | 72 +\n reftable/reftable-reader.h | 98 ++\n reftable/reftable-record.h | 114 ++\n reftable/reftable-stack.h | 123 ++\n reftable/reftable-tests.h | 23 +\n reftable/reftable-writer.h | 147 ++\n reftable/reftable.c | 115 ++\n reftable/reftable_test.c | 583 +++++++\n reftable/stack.c | 1376 ++++++++++++++++\n reftable/stack.h | 41 +\n reftable/stack_test.c | 928 +++++++++++\n reftable/system.h | 32 +\n reftable/test_framework.c | 23 +\n reftable/test_framework.h | 53 +\n reftable/tree.c | 63 +\n reftable/tree.h | 34 +\n reftable/tree_test.c | 61 +\n reftable/writer.c | 691 ++++++++\n reftable/writer.h | 50 +\n reftable/zlib-compat.c | 92 ++\n repository.c | 2 +\n repository.h | 3 +\n setup.c | 9 +-\n t/helper/test-reftable.c | 21 +\n t/helper/test-tool.c | 4 +-\n t/helper/test-tool.h | 2 +\n t/t0031-reftable.sh | 203 +++\n t/t0032-reftable-unittest.sh | 15 +\n t/t1409-avoid-packing-refs.sh | 6 +\n t/t1450-fsck.sh | 6 +\n t/t3210-pack-refs.sh | 6 +\n t/test-lib.sh | 5 +\n 83 files changed, 12509 insertions(+), 40 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100644 reftable/.gitattributes\n create mode 100644 reftable/LICENSE\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/basics_test.c\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/dump.c\n create mode 100644 reftable/error.c\n create mode 100644 reftable/generic.h\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/pq_test.c\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n create mode 100644 reftable/reftable-blocksource.h\n create mode 100644 reftable/reftable-error.h\n create mode 100644 reftable/reftable-generic.h\n create mode 100644 reftable/reftable-iterator.h\n create mode 100644 reftable/reftable-malloc.h\n create mode 100644 reftable/reftable-merged.h\n create mode 100644 reftable/reftable-reader.h\n create mode 100644 reftable/reftable-record.h\n create mode 100644 reftable/reftable-stack.h\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/reftable-writer.h\n create mode 100644 reftable/reftable.c\n create mode 100644 reftable/reftable_test.c\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n create mode 100644 reftable/zlib-compat.c\n create mode 100644 t/helper/test-reftable.c\n create mode 100755 t/t0031-reftable.sh\n create mode 100755 t/t0032-reftable-unittest.sh\n\n\nbase-commit: 89b43f80a514aee58b662ad606e6352e03eaeee4\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-847%2Fhanwen%2Flibreftable-v6\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-847/hanwen/libreftable-v6\nPull-Request: https://github.com/git/git/pull/847\n\nRange-diff vs v5:\n\n 1: e1d1d9f49807 = 1: 15b8306fbe4b init-db: set the_repository->hash_algo early on\n 2: 502a66befabe = 2: 02bd1f40fa4e reftable: add LICENSE\n 3: 1bd136f6a6df = 3: bffa33c012ad reftable: add error related functionality\n 4: 5bbbafb428e9 ! 4: df8003cb9a7d reftable: utility functions\n @@ Makefile: THIRD_PARTY_SOURCES += compat/regex/%\n EXTLIBS =\n \n GIT_USER_AGENT = git/$(GIT_VERSION)\n -@@ Makefile: XDIFF_OBJS += xdiff/xpatience.o\n - XDIFF_OBJS += xdiff/xprepare.o\n - XDIFF_OBJS += xdiff/xutils.o\n +@@ Makefile: XDIFF_OBJS += xdiff/xutils.o\n + .PHONY: xdiff-objs\n + xdiff-objs: $(XDIFF_OBJS)\n \n +REFTABLE_OBJS += reftable/basics.o\n +REFTABLE_OBJS += reftable/error.o\n @@ Makefile: XDIFF_OBJS += xdiff/xpatience.o\n +REFTABLE_TEST_OBJS += reftable/basics_test.o\n +\n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n - OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \\\n - \t$(XDIFF_OBJS) \\\n - \t$(FUZZ_OBJS) \\\n -+\t$(REFTABLE_OBJS) \\\n -+\t$(REFTABLE_TEST_OBJS) \\\n - \tcommon-main.o \\\n - \tgit.o\n ++\n + .PHONY: test-objs\n + test-objs: $(TEST_OBJS)\n + \n +@@ Makefile: OBJECTS += $(PROGRAM_OBJS)\n + OBJECTS += $(TEST_OBJS)\n + OBJECTS += $(XDIFF_OBJS)\n + OBJECTS += $(FUZZ_OBJS)\n ++OBJECTS += $(REFTABLE_OBJS) $(REFTABLE_TEST_OBJS)\n ++\n ifndef NO_CURL\n + \tOBJECTS += http.o http-walker.o remote-curl.o\n + endif\n @@ Makefile: $(LIB_FILE): $(LIB_OBJS)\n $(XDIFF_LIB): $(XDIFF_OBJS)\n \t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n 5: e41fa780882d ! 5: 6807af53e9a0 reftable: add blocksource, an abstraction for random access reads\n @@ Commit message\n Signed-off-by: Han-Wen Nienhuys \n \n ## Makefile ##\n -@@ Makefile: XDIFF_OBJS += xdiff/xutils.o\n +@@ Makefile: xdiff-objs: $(XDIFF_OBJS)\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n 6: 390eaca4fc8c = 6: 78a352c96f82 reftable: (de)serialization for the polymorphic record type.\n 7: b108525009d9 ! 7: 9297b9c363f6 reftable: reading/writing blocks\n @@ Commit message\n Signed-off-by: Han-Wen Nienhuys \n \n ## Makefile ##\n -@@ Makefile: XDIFF_OBJS += xdiff/xutils.o\n +@@ Makefile: xdiff-objs: $(XDIFF_OBJS)\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n 8: b6eed7283aac ! 8: ceddefadd48c reftable: a generic binary tree implementation\n @@ Makefile: REFTABLE_OBJS += reftable/block.o\n +REFTABLE_TEST_OBJS += reftable/tree_test.o\n \n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n - OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \\\n + \n \n ## reftable/tree.c (new) ##\n @@\n 9: f2f005afca19 = 9: 1f635e6c86c5 reftable: write reftable files\n -: ------------ > 10: 9c3fd77fbbac reftable: generic interface to tables\n 10: 1f17d5edab52 ! 11: 8ba486f44a72 reftable: read reftable files\n @@ Makefile: REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/publicbasics.o\n +REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n -+REFTABLE_OBJS += reftable/reftable.o\n + REFTABLE_OBJS += reftable/reftable.o\n REFTABLE_OBJS += reftable/tree.o\n - REFTABLE_OBJS += reftable/writer.o\n - REFTABLE_OBJS += reftable/zlib-compat.o\n \n ## reftable/iter.c (new) ##\n @@\n @@ reftable/iter.c (new)\n +#include \"system.h\"\n +\n +#include \"block.h\"\n ++#include \"generic.h\"\n +#include \"constants.h\"\n +#include \"reader.h\"\n +#include \"reftable-error.h\"\n @@ reftable/iter.c (new)\n +\treturn it->ops == NULL;\n +}\n +\n -+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n -+{\n -+\treturn 1;\n -+}\n -+\n -+static void empty_iterator_close(void *arg)\n -+{\n -+}\n -+\n -+static struct reftable_iterator_vtable empty_vtable = {\n -+\t.next = &empty_iterator_next,\n -+\t.close = &empty_iterator_close,\n -+};\n -+\n -+void iterator_set_empty(struct reftable_iterator *it)\n -+{\n -+\tassert(it->ops == NULL);\n -+\tit->iter_arg = NULL;\n -+\tit->ops = &empty_vtable;\n -+}\n -+\n -+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n -+{\n -+\treturn it->ops->next(it->iter_arg, rec);\n -+}\n -+\n -+void reftable_iterator_destroy(struct reftable_iterator *it)\n -+{\n -+\tif (it->ops == NULL) {\n -+\t\treturn;\n -+\t}\n -+\tit->ops->close(it->iter_arg);\n -+\tit->ops = NULL;\n -+\tFREE_AND_NULL(it->iter_arg);\n -+}\n -+\n -+int reftable_iterator_next_ref(struct reftable_iterator *it,\n -+\t\t\t struct reftable_ref_record *ref)\n -+{\n -+\tstruct reftable_record rec = { NULL };\n -+\treftable_record_from_ref(&rec, ref);\n -+\treturn iterator_next(it, &rec);\n -+}\n -+\n -+int reftable_iterator_next_log(struct reftable_iterator *it,\n -+\t\t\t struct reftable_log_record *log)\n -+{\n -+\tstruct reftable_record rec = { NULL };\n -+\treftable_record_from_log(&rec, log);\n -+\treturn iterator_next(it, &rec);\n -+}\n -+\n +static void filtering_ref_iterator_close(void *iter_arg)\n +{\n +\tstruct filtering_ref_iterator *fri =\n @@ reftable/iter.h (new)\n +#include \"reftable-iterator.h\"\n +#include \"reftable-generic.h\"\n +\n -+struct reftable_iterator_vtable {\n -+\tint (*next)(void *iter_arg, struct reftable_record *rec);\n -+\tvoid (*close)(void *iter_arg);\n -+};\n -+\n -+void iterator_set_empty(struct reftable_iterator *it);\n -+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec);\n -+\n +/* Returns true for a zeroed out iterator, such as the one returned from\n + * iterator_destroy. */\n +int iterator_is_null(struct reftable_iterator *it);\n @@ reftable/reader.c (new)\n +#include \"system.h\"\n +#include \"block.h\"\n +#include \"constants.h\"\n ++#include \"generic.h\"\n +#include \"iter.h\"\n +#include \"record.h\"\n +#include \"reftable-error.h\"\n ++#include \"reftable-generic.h\"\n +#include \"tree.h\"\n +\n +uint64_t block_source_size(struct reftable_block_source *source)\n @@ reftable/reader.c (new)\n +uint64_t reftable_reader_min_update_index(struct reftable_reader *r)\n +{\n +\treturn r->min_update_index;\n ++}\n ++\n ++/* generic table interface. */\n ++\n ++static int reftable_reader_seek_void(void *tab, struct reftable_iterator *it,\n ++\t\t\t\t struct reftable_record *rec)\n ++{\n ++\treturn reader_seek((struct reftable_reader *)tab, it, rec);\n ++}\n ++\n ++static uint32_t reftable_reader_hash_id_void(void *tab)\n ++{\n ++\treturn reftable_reader_hash_id((struct reftable_reader *)tab);\n ++}\n ++\n ++static uint64_t reftable_reader_min_update_index_void(void *tab)\n ++{\n ++\treturn reftable_reader_min_update_index((struct reftable_reader *)tab);\n ++}\n ++\n ++static uint64_t reftable_reader_max_update_index_void(void *tab)\n ++{\n ++\treturn reftable_reader_max_update_index((struct reftable_reader *)tab);\n ++}\n ++\n ++static struct reftable_table_vtable reader_vtable = {\n ++\t.seek_record = reftable_reader_seek_void,\n ++\t.hash_id = reftable_reader_hash_id_void,\n ++\t.min_update_index = reftable_reader_min_update_index_void,\n ++\t.max_update_index = reftable_reader_max_update_index_void,\n ++};\n ++\n ++void reftable_table_from_reader(struct reftable_table *tab,\n ++\t\t\t\tstruct reftable_reader *reader)\n ++{\n ++\tassert(tab->ops == NULL);\n ++\ttab->ops = &reader_vtable;\n ++\ttab->table_arg = reader;\n +}\n \n ## reftable/reader.h (new) ##\n @@ reftable/reader.h (new)\n +int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n +\t\t\t uint64_t next_off, uint8_t want_typ);\n +\n -+/* generic interface to reftables */\n -+struct reftable_table_vtable {\n -+\tint (*seek_record)(void *tab, struct reftable_iterator *it,\n -+\t\t\t struct reftable_record *);\n -+\tuint32_t (*hash_id)(void *tab);\n -+\tuint64_t (*min_update_index)(void *tab);\n -+\tuint64_t (*max_update_index)(void *tab);\n -+};\n -+\n -+#endif\n -\n - ## reftable/reftable-iterator.h (new) ##\n -@@\n -+/*\n -+Copyright 2020 Google LLC\n -+\n -+Use of this source code is governed by a BSD-style\n -+license that can be found in the LICENSE file or at\n -+https://developers.google.com/open-source/licenses/bsd\n -+*/\n -+\n -+#ifndef REFTABLE_ITERATOR_H\n -+#define REFTABLE_ITERATOR_H\n -+\n -+#include \"reftable-record.h\"\n -+\n -+/* iterator is the generic interface for walking over data stored in a\n -+ * reftable.\n -+ */\n -+struct reftable_iterator {\n -+\tstruct reftable_iterator_vtable *ops;\n -+\tvoid *iter_arg;\n -+};\n -+\n -+/* reads the next reftable_ref_record. Returns < 0 for error, 0 for OK and > 0:\n -+ * end of iteration.\n -+ */\n -+int reftable_iterator_next_ref(struct reftable_iterator *it,\n -+\t\t\t struct reftable_ref_record *ref);\n -+\n -+/* reads the next reftable_log_record. Returns < 0 for error, 0 for OK and > 0:\n -+ * end of iteration.\n -+ */\n -+int reftable_iterator_next_log(struct reftable_iterator *it,\n -+\t\t\t struct reftable_log_record *log);\n -+\n -+/* releases resources associated with an iterator. */\n -+void reftable_iterator_destroy(struct reftable_iterator *it);\n -+\n +#endif\n \n ## reftable/reftable-reader.h (new) ##\n @@\n +/*\n -+Copyright 2020 Google LLC\n ++ Copyright 2020 Google LLC\n +\n -+Use of this source code is governed by a BSD-style\n -+license that can be found in the LICENSE file or at\n -+https://developers.google.com/open-source/licenses/bsd\n ++ Use of this source code is governed by a BSD-style\n ++ license that can be found in the LICENSE file or at\n ++ https://developers.google.com/open-source/licenses/bsd\n +*/\n +\n +#ifndef REFTABLE_READER_H\n @@ reftable/reftable-reader.h (new)\n +/*\n + * Reading single tables\n + *\n -+ * The follow routines are for reading single files. For an application-level\n -+ * interface, skip ahead to struct reftable_merged_table and struct\n -+ * reftable_stack.\n ++ * The follow routines are for reading single files. For an\n ++ * application-level interface, skip ahead to struct\n ++ * reftable_merged_table and struct reftable_stack.\n + */\n +\n +/* The reader struct is a handle to an open reftable file. */\n +struct reftable_reader;\n +\n -+/* reftable_new_reader opens a reftable for reading. If successful, returns 0\n -+ * code and sets pp. The name is used for creating a stack. Typically, it is the\n -+ * basename of the file. The block source `src` is owned by the reader, and is\n -+ * closed on calling reftable_reader_destroy().\n ++/* Generic table. */\n ++struct reftable_table;\n ++\n ++/* reftable_new_reader opens a reftable for reading. If successful,\n ++ * returns 0 code and sets pp. The name is used for creating a\n ++ * stack. Typically, it is the basename of the file. The block source\n ++ * `src` is owned by the reader, and is closed on calling\n ++ * reftable_reader_destroy(). On error, the block source `src` is\n ++ * closed as well.\n + */\n +int reftable_new_reader(struct reftable_reader **pp,\n +\t\t\tstruct reftable_block_source *src, const char *name);\n @@ reftable/reftable-reader.h (new)\n + if (err < 0) { ... }\n + struct reftable_ref_record ref = {0};\n + while (1) {\n -+ err = reftable_iterator_next_ref(&it, &ref);\n -+ if (err > 0) {\n -+ break;\n -+ }\n -+ if (err < 0) {\n -+ ..error handling..\n -+ }\n -+ ..found..\n ++ err = reftable_iterator_next_ref(&it, &ref);\n ++ if (err > 0) {\n ++ break;\n ++ }\n ++ if (err < 0) {\n ++ ..error handling..\n ++ }\n ++ ..found..\n + }\n + reftable_iterator_destroy(&it);\n + reftable_ref_record_release(&ref);\n -+ */\n ++*/\n +int reftable_reader_seek_ref(struct reftable_reader *r,\n +\t\t\t struct reftable_iterator *it, const char *name);\n +\n @@ reftable/reftable-reader.h (new)\n +\n +/* seek to logs for the given name, older than update_index. To seek to the\n + start of the table, use name = \"\".\n -+ */\n ++*/\n +int reftable_reader_seek_log_at(struct reftable_reader *r,\n +\t\t\t\tstruct reftable_iterator *it, const char *name,\n +\t\t\t\tuint64_t update_index);\n @@ reftable/reftable-reader.h (new)\n +/* return the min_update_index for a table */\n +uint64_t reftable_reader_min_update_index(struct reftable_reader *r);\n +\n ++/* creates a generic table from a file reader. */\n ++void reftable_table_from_reader(struct reftable_table *tab,\n ++\t\t\t\tstruct reftable_reader *reader);\n ++\n +#endif\n 11: c916629c562a ! 12: 9ade9303f08f reftable: reftable file level tests\n @@ reftable/reftable_test.c (new)\n +#include \"record.h\"\n +#include \"test_framework.h\"\n +#include \"reftable-tests.h\"\n -+#include \"reftable-stack.h\"\n ++#include \"reftable-writer.h\"\n +\n +static const int update_index = 5;\n +\n -: ------------ > 13: 0a6119d910a3 reftable: add a heap-based priority queue for reftable records\n -: ------------ > 14: 393fcecdae1e reftable: add merged table view\n -: ------------ > 15: 3186b2b70f73 reftable: implement refname validation\n 12: 28aa69f7bbcb ! 16: b5492d5a13d7 reftable: rest of library\n @@ Metadata\n Author: Han-Wen Nienhuys \n \n ## Commit message ##\n - reftable: rest of library\n + reftable: implement stack, a mutable database of reftable files.\n \n Signed-off-by: Han-Wen Nienhuys \n \n ## Makefile ##\n -@@ Makefile: REFTABLE_OBJS += reftable/error.o\n - REFTABLE_OBJS += reftable/block.o\n - REFTABLE_OBJS += reftable/blocksource.o\n - REFTABLE_OBJS += reftable/iter.o\n -+REFTABLE_OBJS += reftable/merged.o\n -+REFTABLE_OBJS += reftable/pq.o\n - REFTABLE_OBJS += reftable/publicbasics.o\n - REFTABLE_OBJS += reftable/reader.o\n +@@ Makefile: REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n -+REFTABLE_OBJS += reftable/refname.o\n + REFTABLE_OBJS += reftable/refname.o\n REFTABLE_OBJS += reftable/reftable.o\n +REFTABLE_OBJS += reftable/stack.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n REFTABLE_OBJS += reftable/zlib-compat.o\n -\n - ## reftable/VERSION (new) ##\n -@@\n -+a337d48d4d42d513d6baa33addc172f0e0e36288 C: use tagged union for reftable_log_record\n -\n - ## reftable/dump.c (new) ##\n -@@\n -+/*\n -+Copyright 2020 Google LLC\n -+\n -+Use of this source code is governed by a BSD-style\n -+license that can be found in the LICENSE file or at\n -+https://developers.google.com/open-source/licenses/bsd\n -+*/\n -+\n -+#include \n -+#include \n -+#include \n -+#include \n -+#include \n -+\n -+#include \"reftable.h\"\n -+#include \"reftable-tests.h\"\n -+\n -+static uint32_t hash_id;\n -+\n -+static int dump_table(const char *tablename)\n -+{\n -+\tstruct reftable_block_source src = { 0 };\n -+\tint err = reftable_block_source_from_file(&src, tablename);\n -+\tstruct reftable_iterator it = { 0 };\n -+\tstruct reftable_ref_record ref = { 0 };\n -+\tstruct reftable_log_record log = { 0 };\n -+\tstruct reftable_reader *r = NULL;\n -+\n -+\tif (err < 0)\n -+\t\treturn err;\n -+\n -+\terr = reftable_new_reader(&r, &src, tablename);\n -+\tif (err < 0)\n -+\t\treturn err;\n -+\n -+\terr = reftable_reader_seek_ref(r, &it, \"\");\n -+\tif (err < 0) {\n -+\t\treturn err;\n -+\t}\n -+\n -+\twhile (1) {\n -+\t\terr = reftable_iterator_next_ref(&it, &ref);\n -+\t\tif (err > 0) {\n -+\t\t\tbreak;\n -+\t\t}\n -+\t\tif (err < 0) {\n -+\t\t\treturn err;\n -+\t\t}\n -+\t\treftable_ref_record_print(&ref, hash_id);\n -+\t}\n -+\treftable_iterator_destroy(&it);\n -+\treftable_ref_record_clear(&ref);\n -+\n -+\terr = reftable_reader_seek_log(r, &it, \"\");\n -+\tif (err < 0) {\n -+\t\treturn err;\n -+\t}\n -+\twhile (1) {\n -+\t\terr = reftable_iterator_next_log(&it, &log);\n -+\t\tif (err > 0) {\n -+\t\t\tbreak;\n -+\t\t}\n -+\t\tif (err < 0) {\n -+\t\t\treturn err;\n -+\t\t}\n -+\t\treftable_log_record_print(&log, hash_id);\n -+\t}\n -+\treftable_iterator_destroy(&it);\n -+\treftable_log_record_clear(&log);\n -+\n -+\treftable_reader_free(r);\n -+\treturn 0;\n -+}\n -+\n -+static int compact_stack(const char *stackdir)\n -+{\n -+\tstruct reftable_stack *stack = NULL;\n -+\tstruct reftable_write_options cfg = {};\n -+\n -+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n -+\tif (err < 0)\n -+\t\tgoto done;\n -+\n -+\terr = reftable_stack_compact_all(stack, NULL);\n -+\tif (err < 0)\n -+\t\tgoto done;\n -+done:\n -+\tif (stack != NULL) {\n -+\t\treftable_stack_destroy(stack);\n -+\t}\n -+\treturn err;\n -+}\n -+\n -+static int dump_stack(const char *stackdir)\n -+{\n -+\tstruct reftable_stack *stack = NULL;\n -+\tstruct reftable_write_options cfg = {};\n -+\tstruct reftable_iterator it = { 0 };\n -+\tstruct reftable_ref_record ref = { 0 };\n -+\tstruct reftable_log_record log = { 0 };\n -+\tstruct reftable_merged_table *merged = NULL;\n -+\n -+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n -+\tif (err < 0)\n -+\t\treturn err;\n -+\n -+\tmerged = reftable_stack_merged_table(stack);\n -+\n -+\terr = reftable_merged_table_seek_ref(merged, &it, \"\");\n -+\tif (err < 0) {\n -+\t\treturn err;\n -+\t}\n -+\n -+\twhile (1) {\n -+\t\terr = reftable_iterator_next_ref(&it, &ref);\n -+\t\tif (err > 0) {\n -+\t\t\tbreak;\n -+\t\t}\n -+\t\tif (err < 0) {\n -+\t\t\treturn err;\n -+\t\t}\n -+\t\treftable_ref_record_print(&ref, hash_id);\n -+\t}\n -+\treftable_iterator_destroy(&it);\n -+\treftable_ref_record_clear(&ref);\n -+\n -+\terr = reftable_merged_table_seek_log(merged, &it, \"\");\n -+\tif (err < 0) {\n -+\t\treturn err;\n -+\t}\n -+\twhile (1) {\n -+\t\terr = reftable_iterator_next_log(&it, &log);\n -+\t\tif (err > 0) {\n -+\t\t\tbreak;\n -+\t\t}\n -+\t\tif (err < 0) {\n -+\t\t\treturn err;\n -+\t\t}\n -+\t\treftable_log_record_print(&log, hash_id);\n -+\t}\n -+\treftable_iterator_destroy(&it);\n -+\treftable_log_record_clear(&log);\n -+\n -+\treftable_stack_destroy(stack);\n -+\treturn 0;\n -+}\n -+\n -+static void print_help(void)\n -+{\n -+\tprintf(\"usage: dump [-cst] arg\\n\\n\"\n -+\t \"options: \\n\"\n -+\t \" -c compact\\n\"\n -+\t \" -t dump table\\n\"\n -+\t \" -s dump stack\\n\"\n -+\t \" -h this help\\n\"\n -+\t \" -2 use SHA256\\n\"\n -+\t \"\\n\");\n -+}\n -+\n -+int reftable_dump_main(int argc, char *const *argv)\n -+{\n -+\tint err = 0;\n -+\tint opt;\n -+\tint opt_dump_table = 0;\n -+\tint opt_dump_stack = 0;\n -+\tint opt_compact = 0;\n -+\tconst char *arg = NULL;\n -+\twhile ((opt = getopt(argc, argv, \"2chts\")) != -1) {\n -+\t\tswitch (opt) {\n -+\t\tcase '2':\n -+\t\t\thash_id = 0x73323536;\n -+\t\t\tbreak;\n -+\t\tcase 't':\n -+\t\t\topt_dump_table = 1;\n -+\t\t\tbreak;\n -+\t\tcase 's':\n -+\t\t\topt_dump_stack = 1;\n -+\t\t\tbreak;\n -+\t\tcase 'c':\n -+\t\t\topt_compact = 1;\n -+\t\t\tbreak;\n -+\t\tcase '?':\n -+\t\tcase 'h':\n -+\t\t\tprint_help();\n -+\t\t\treturn 2;\n -+\t\t\tbreak;\n -+\t\t}\n -+\t}\n -+\n -+\tif (argv[optind] == NULL) {\n -+\t\tfprintf(stderr, \"need argument\\n\");\n -+\t\tprint_help();\n -+\t\treturn 2;\n -+\t}\n -+\n -+\targ = argv[optind];\n -+\n -+\tif (opt_dump_table) {\n -+\t\terr = dump_table(arg);\n -+\t} else if (opt_dump_stack) {\n -+\t\terr = dump_stack(arg);\n -+\t} else if (opt_compact) {\n -+\t\terr = compact_stack(arg);\n -+\t}\n -+\n -+\tif (err < 0) {\n -+\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv[0], arg,\n -+\t\t\treftable_error_str(err));\n -+\t\treturn 1;\n -+\t}\n -+\treturn 0;\n -+}\n -\n - ## reftable/merged.c (new) ##\n -@@\n -+/*\n -+Copyright 2020 Google LLC\n -+\n -+Use of this source code is governed by a BSD-style\n -+license that can be found in the LICENSE file or at\n -+https://developers.google.com/open-source/licenses/bsd\n -+*/\n -+\n -+#include \"merged.h\"\n -+\n -+#include \"constants.h\"\n -+#include \"iter.h\"\n -+#include \"pq.h\"\n -+#include \"reader.h\"\n -+#include \"record.h\"\n -+#include \"reftable-merged.h\"\n -+#include \"reftable-error.h\"\n -+#include \"system.h\"\n -+\n -+static int merged_iter_init(struct merged_iter *mi)\n -+{\n -+\tint i = 0;\n -+\tfor (i = 0; i < mi->stack_len; i++) {\n -+\t\tstruct reftable_record rec = reftable_new_record(mi->typ);\n -+\t\tint err = iterator_next(&mi->stack[i], &rec);\n -+\t\tif (err < 0) {\n -+\t\t\treturn err;\n -+\t\t}\n -+\n -+\t\tif (err > 0) {\n -+\t\t\treftable_iterator_destroy(&mi->stack[i]);\n -+\t\t\treftable_record_destroy(&rec);\n -+\t\t} else {\n -+\t\t\tstruct pq_entry e = {\n -+\t\t\t\t.rec = rec,\n -+\t\t\t\t.index = i,\n -+\t\t\t};\n -+\t\t\tmerged_iter_pqueue_add(&mi->pq, e);\n -+\t\t}\n -+\t}\n -+\n -+\treturn 0;\n -+}\n -+\n -+static void merged_iter_close(void *p)\n -+{\n -+\tstruct merged_iter *mi = (struct merged_iter *)p;\n -+\tint i = 0;\n -+\tmerged_iter_pqueue_release(&mi->pq);\n -+\tfor (i = 0; i < mi->stack_len; i++) {\n -+\t\treftable_iterator_destroy(&mi->stack[i]);\n -+\t}\n -+\treftable_free(mi->stack);\n -+}\n -+\n -+static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,\n -+\t\t\t\t\t size_t idx)\n -+{\n -+\tstruct reftable_record rec = reftable_new_record(mi->typ);\n -+\tstruct pq_entry e = {\n -+\t\t.rec = rec,\n -+\t\t.index = idx,\n -+\t};\n -+\tint err = iterator_next(&mi->stack[idx], &rec);\n -+\tif (err < 0)\n -+\t\treturn err;\n -+\n -+\tif (err > 0) {\n -+\t\treftable_iterator_destroy(&mi->stack[idx]);\n -+\t\treftable_record_destroy(&rec);\n -+\t\treturn 0;\n -+\t}\n -+\n -+\tmerged_iter_pqueue_add(&mi->pq, e);\n -+\treturn 0;\n -+}\n -+\n -+static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)\n -+{\n -+\tif (iterator_is_null(&mi->stack[idx]))\n -+\t\treturn 0;\n -+\treturn merged_iter_advance_nonnull_subiter(mi, idx);\n -+}\n -+\n -+static int merged_iter_next_entry(struct merged_iter *mi,\n -+\t\t\t\t struct reftable_record *rec)\n -+{\n -+\tstruct strbuf entry_key = STRBUF_INIT;\n -+\tstruct pq_entry entry = { 0 };\n -+\tint err = 0;\n -+\n -+\tif (merged_iter_pqueue_is_empty(mi->pq))\n -+\t\treturn 1;\n -+\n -+\tentry = merged_iter_pqueue_remove(&mi->pq);\n -+\terr = merged_iter_advance_subiter(mi, entry.index);\n -+\tif (err < 0)\n -+\t\treturn err;\n -+\n -+\t/*\n -+\t One can also use reftable as datacenter-local storage, where the ref\n -+\t database is maintained in globally consistent database (eg.\n -+\t CockroachDB or Spanner). In this scenario, replication delays together\n -+\t with compaction may cause newer tables to contain older entries. In\n -+\t such a deployment, the loop below must be changed to collect all\n -+\t entries for the same key, and return new the newest one.\n -+\t*/\n -+\treftable_record_key(&entry.rec, &entry_key);\n -+\twhile (!merged_iter_pqueue_is_empty(mi->pq)) {\n -+\t\tstruct pq_entry top = merged_iter_pqueue_top(mi->pq);\n -+\t\tstruct strbuf k = STRBUF_INIT;\n -+\t\tint err = 0, cmp = 0;\n -+\n -+\t\treftable_record_key(&top.rec, &k);\n -+\n -+\t\tcmp = strbuf_cmp(&k, &entry_key);\n -+\t\tstrbuf_release(&k);\n -+\n -+\t\tif (cmp > 0) {\n -+\t\t\tbreak;\n -+\t\t}\n -+\n -+\t\tmerged_iter_pqueue_remove(&mi->pq);\n -+\t\terr = merged_iter_advance_subiter(mi, top.index);\n -+\t\tif (err < 0) {\n -+\t\t\treturn err;\n -+\t\t}\n -+\t\treftable_record_destroy(&top.rec);\n -+\t}\n -+\n -+\treftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));\n -+\treftable_record_destroy(&entry.rec);\n -+\tstrbuf_release(&entry_key);\n -+\treturn 0;\n -+}\n -+\n -+static int merged_iter_next(struct merged_iter *mi, struct reftable_record *rec)\n -+{\n -+\twhile (1) {\n -+\t\tint err = merged_iter_next_entry(mi, rec);\n -+\t\tif (err == 0 && mi->suppress_deletions &&\n -+\t\t reftable_record_is_deletion(rec)) {\n -+\t\t\tcontinue;\n -+\t\t}\n -+\n -+\t\treturn err;\n -+\t}\n -+}\n -+\n -+static int merged_iter_next_void(void *p, struct reftable_record *rec)\n -+{\n -+\tstruct merged_iter *mi = (struct merged_iter *)p;\n -+\tif (merged_iter_pqueue_is_empty(mi->pq))\n -+\t\treturn 1;\n -+\n -+\treturn merged_iter_next(mi, rec);\n -+}\n -+\n -+static struct reftable_iterator_vtable merged_iter_vtable = {\n -+\t.next = &merged_iter_next_void,\n -+\t.close = &merged_iter_close,\n -+};\n -+\n -+static void iterator_from_merged_iter(struct reftable_iterator *it,\n -+\t\t\t\t struct merged_iter *mi)\n -+{\n -+\tassert(it->ops == NULL);\n -+\tit->iter_arg = mi;\n -+\tit->ops = &merged_iter_vtable;\n -+}\n -+\n -+int reftable_new_merged_table(struct reftable_merged_table **dest,\n -+\t\t\t struct reftable_table *stack, int n,\n -+\t\t\t uint32_t hash_id)\n -+{\n -+\tstruct reftable_merged_table *m = NULL;\n -+\tuint64_t last_max = 0;\n -+\tuint64_t first_min = 0;\n -+\tint i = 0;\n -+\tfor (i = 0; i < n; i++) {\n -+\t\tuint64_t min = reftable_table_min_update_index(&stack[i]);\n -+\t\tuint64_t max = reftable_table_max_update_index(&stack[i]);\n -+\n -+\t\tif (reftable_table_hash_id(&stack[i]) != hash_id) {\n -+\t\t\treturn REFTABLE_FORMAT_ERROR;\n -+\t\t}\n -+\t\tif (i == 0 || min < first_min) {\n -+\t\t\tfirst_min = min;\n -+\t\t}\n -+\t\tif (i == 0 || max > last_max) {\n -+\t\t\tlast_max = max;\n -+\t\t}\n -+\t}\n -+\n -+\tm = (struct reftable_merged_table *)reftable_calloc(\n -+\t\tsizeof(struct reftable_merged_table));\n -+\tm->stack = stack;\n -+\tm->stack_len = n;\n -+\tm->min = first_min;\n -+\tm->max = last_max;\n -+\tm->hash_id = hash_id;\n -+\t*dest = m;\n -+\treturn 0;\n -+}\n -+\n -+/* clears the list of subtable, without affecting the readers themselves. */\n -+void merged_table_release(struct reftable_merged_table *mt)\n -+{\n -+\tFREE_AND_NULL(mt->stack);\n -+\tmt->stack_len = 0;\n -+}\n -+\n -+void reftable_merged_table_free(struct reftable_merged_table *mt)\n -+{\n -+\tif (mt == NULL) {\n -+\t\treturn;\n -+\t}\n -+\tmerged_table_release(mt);\n -+\treftable_free(mt);\n -+}\n -+\n -+uint64_t\n -+reftable_merged_table_max_update_index(struct reftable_merged_table *mt)\n -+{\n -+\treturn mt->max;\n -+}\n -+\n -+uint64_t\n -+reftable_merged_table_min_update_index(struct reftable_merged_table *mt)\n -+{\n -+\treturn mt->min;\n -+}\n -+\n -+static int reftable_table_seek_record(struct reftable_table *tab,\n -+\t\t\t\t struct reftable_iterator *it,\n -+\t\t\t\t struct reftable_record *rec)\n -+{\n -+\treturn tab->ops->seek_record(tab->table_arg, it, rec);\n -+}\n -+\n -+static int merged_table_seek_record(struct reftable_merged_table *mt,\n -+\t\t\t\t struct reftable_iterator *it,\n -+\t\t\t\t struct reftable_record *rec)\n -+{\n -+\tstruct reftable_iterator *iters = reftable_calloc(\n -+\t\tsizeof(struct reftable_iterator) * mt->stack_len);\n -+\tstruct merged_iter merged = {\n -+\t\t.stack = iters,\n -+\t\t.typ = reftable_record_type(rec),\n -+\t\t.hash_id = mt->hash_id,\n -+\t\t.suppress_deletions = mt->suppress_deletions,\n -+\t};\n -+\tint n = 0;\n -+\tint err = 0;\n -+\tint i = 0;\n -+\tfor (i = 0; i < mt->stack_len && err == 0; i++) {\n -+\t\tint e = reftable_table_seek_record(&mt->stack[i], &iters[n],\n -+\t\t\t\t\t\t rec);\n -+\t\tif (e < 0) {\n -+\t\t\terr = e;\n -+\t\t}\n -+\t\tif (e == 0) {\n -+\t\t\tn++;\n -+\t\t}\n -+\t}\n -+\tif (err < 0) {\n -+\t\tint i = 0;\n -+\t\tfor (i = 0; i < n; i++) {\n -+\t\t\treftable_iterator_destroy(&iters[i]);\n -+\t\t}\n -+\t\treftable_free(iters);\n -+\t\treturn err;\n -+\t}\n -+\n -+\tmerged.stack_len = n;\n -+\terr = merged_iter_init(&merged);\n -+\tif (err < 0) {\n -+\t\tmerged_iter_close(&merged);\n -+\t\treturn err;\n -+\t} else {\n -+\t\tstruct merged_iter *p =\n -+\t\t\treftable_malloc(sizeof(struct merged_iter));\n -+\t\t*p = merged;\n -+\t\titerator_from_merged_iter(it, p);\n -+\t}\n -+\treturn 0;\n -+}\n -+\n -+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n -+\t\t\t\t struct reftable_iterator *it,\n -+\t\t\t\t const char *name)\n -+{\n -+\tstruct reftable_ref_record ref = {\n -+\t\t.refname = (char *)name,\n -+\t};\n -+\tstruct reftable_record rec = { NULL };\n -+\treftable_record_from_ref(&rec, &ref);\n -+\treturn merged_table_seek_record(mt, it, &rec);\n -+}\n -+\n -+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n -+\t\t\t\t struct reftable_iterator *it,\n -+\t\t\t\t const char *name, uint64_t update_index)\n -+{\n -+\tstruct reftable_log_record log = {\n -+\t\t.refname = (char *)name,\n -+\t\t.update_index = update_index,\n -+\t};\n -+\tstruct reftable_record rec = { NULL };\n -+\treftable_record_from_log(&rec, &log);\n -+\treturn merged_table_seek_record(mt, it, &rec);\n -+}\n -+\n -+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n -+\t\t\t\t struct reftable_iterator *it,\n -+\t\t\t\t const char *name)\n -+{\n -+\tuint64_t max = ~((uint64_t)0);\n -+\treturn reftable_merged_table_seek_log_at(mt, it, name, max);\n -+}\n -+\n -+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\n -+{\n -+\treturn mt->hash_id;\n -+}\n -+\n -+static int reftable_merged_table_seek_void(void *tab,\n -+\t\t\t\t\t struct reftable_iterator *it,\n -+\t\t\t\t\t struct reftable_record *rec)\n -+{\n -+\treturn merged_table_seek_record((struct reftable_merged_table *)tab, it,\n -+\t\t\t\t\trec);\n -+}\n -+\n -+static uint32_t reftable_merged_table_hash_id_void(void *tab)\n -+{\n -+\treturn reftable_merged_table_hash_id(\n -+\t\t(struct reftable_merged_table *)tab);\n -+}\n -+\n -+static uint64_t reftable_merged_table_min_update_index_void(void *tab)\n -+{\n -+\treturn reftable_merged_table_min_update_index(\n -+\t\t(struct reftable_merged_table *)tab);\n -+}\n -+\n -+static uint64_t reftable_merged_table_max_update_index_void(void *tab)\n -+{\n -+\treturn reftable_merged_table_max_update_index(\n -+\t\t(struct reftable_merged_table *)tab);\n -+}\n -+\n -+static struct reftable_table_vtable merged_table_vtable = {\n -+\t.seek_record = reftable_merged_table_seek_void,\n -+\t.hash_id = reftable_merged_table_hash_id_void,\n -+\t.min_update_index = reftable_merged_table_min_update_index_void,\n -+\t.max_update_index = reftable_merged_table_max_update_index_void,\n -+};\n -+\n -+void reftable_table_from_merged_table(struct reftable_table *tab,\n -+\t\t\t\t struct reftable_merged_table *merged)\n -+{\n -+\tassert(tab->ops == NULL);\n -+\ttab->ops = &merged_table_vtable;\n -+\ttab->table_arg = merged;\n -+}\n -\n - ## reftable/merged.h (new) ##\n -@@\n -+/*\n -+Copyright 2020 Google LLC\n -+\n -+Use of this source code is governed by a BSD-style\n -+license that can be found in the LICENSE file or at\n -+https://developers.google.com/open-source/licenses/bsd\n -+*/\n -+\n -+#ifndef MERGED_H\n -+#define MERGED_H\n -+\n -+#include \"pq.h\"\n -+\n -+struct reftable_merged_table {\n -+\tstruct reftable_table *stack;\n -+\tsize_t stack_len;\n -+\tuint32_t hash_id;\n -+\tint suppress_deletions;\n -+\n -+\tuint64_t min;\n -+\tuint64_t max;\n -+};\n -+\n -+struct merged_iter {\n -+\tstruct reftable_iterator *stack;\n -+\tuint32_t hash_id;\n -+\tsize_t stack_len;\n -+\tuint8_t typ;\n -+\tint suppress_deletions;\n -+\tstruct merged_iter_pqueue pq;\n -+};\n -+\n -+void merged_table_release(struct reftable_merged_table *mt);\n -+\n -+#endif\n -\n - ## reftable/merged_test.c (new) ##\n -@@\n -+/*\n -+Copyright 2020 Google LLC\n -+\n -+Use of this source code is governed by a BSD-style\n -+license that can be found in the LICENSE file or at\n -+https://developers.google.com/open-source/licenses/bsd\n -+*/\n -+\n -+#include \"merged.h\"\n -+\n -+#include \"system.h\"\n -+\n -+#include \"basics.h\"\n -+#include \"blocksource.h\"\n -+#include \"constants.h\"\n -+#include \"pq.h\"\n -+#include \"reader.h\"\n -+#include \"record.h\"\n -+#include \"test_framework.h\"\n -+#include \"reftable-merged.h\"\n -+#include \"reftable-tests.h\"\n -+#include \"reftable-generic.h\"\n -+#include \"reftable-stack.h\"\n -+\n -+static void test_pq(void)\n -+{\n -+\tchar *names[54] = { NULL };\n -+\tint N = ARRAY_SIZE(names) - 1;\n -+\n -+\tstruct merged_iter_pqueue pq = { NULL };\n -+\tconst char *last = NULL;\n -+\n -+\tint i = 0;\n -+\tfor (i = 0; i < N; i++) {\n -+\t\tchar name[100];\n -+\t\tsnprintf(name, sizeof(name), \"%02d\", i);\n -+\t\tnames[i] = xstrdup(name);\n -+\t}\n -+\n -+\ti = 1;\n -+\tdo {\n -+\t\tstruct reftable_record rec =\n -+\t\t\treftable_new_record(BLOCK_TYPE_REF);\n -+\t\tstruct pq_entry e = { 0 };\n -+\n -+\t\treftable_record_as_ref(&rec)->refname = names[i];\n -+\t\te.rec = rec;\n -+\t\tmerged_iter_pqueue_add(&pq, e);\n -+\t\tmerged_iter_pqueue_check(pq);\n -+\t\ti = (i * 7) % N;\n -+\t} while (i != 1);\n -+\n -+\twhile (!merged_iter_pqueue_is_empty(pq)) {\n -+\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n -+\t\tstruct reftable_ref_record *ref =\n -+\t\t\treftable_record_as_ref(&e.rec);\n -+\n -+\t\tmerged_iter_pqueue_check(pq);\n -+\n -+\t\tif (last != NULL) {\n -+\t\t\tassert(strcmp(last, ref->refname) < 0);\n -+\t\t}\n -+\t\tlast = ref->refname;\n -+\t\tref->refname = NULL;\n -+\t\treftable_free(ref);\n -+\t}\n -+\n -+\tfor (i = 0; i < N; i++) {\n -+\t\treftable_free(names[i]);\n -+\t}\n -+\n -+\tmerged_iter_pqueue_release(&pq);\n -+}\n -+\n -+static void write_test_table(struct strbuf *buf,\n -+\t\t\t struct reftable_ref_record refs[], int n)\n -+{\n -+\tint min = 0xffffffff;\n -+\tint max = 0;\n -+\tint i = 0;\n -+\tint err;\n -+\n -+\tstruct reftable_write_options opts = {\n -+\t\t.block_size = 256,\n -+\t};\n -+\tstruct reftable_writer *w = NULL;\n -+\tfor (i = 0; i < n; i++) {\n -+\t\tuint64_t ui = refs[i].update_index;\n -+\t\tif (ui > max) {\n -+\t\t\tmax = ui;\n -+\t\t}\n -+\t\tif (ui < min) {\n -+\t\t\tmin = ui;\n -+\t\t}\n -+\t}\n -+\n -+\tw = reftable_new_writer(&strbuf_add_void, buf, &opts);\n -+\treftable_writer_set_limits(w, min, max);\n -+\n -+\tfor (i = 0; i < n; i++) {\n -+\t\tuint64_t before = refs[i].update_index;\n -+\t\tint n = reftable_writer_add_ref(w, &refs[i]);\n -+\t\tassert(n == 0);\n -+\t\tassert(before == refs[i].update_index);\n -+\t}\n -+\n -+\terr = reftable_writer_close(w);\n -+\tEXPECT_ERR(err);\n -+\n -+\treftable_writer_free(w);\n -+}\n -+\n -+static struct reftable_merged_table *\n -+merged_table_from_records(struct reftable_ref_record **refs,\n -+\t\t\t struct reftable_block_source **source,\n -+\t\t\t struct reftable_reader ***readers, int *sizes,\n -+\t\t\t struct strbuf *buf, int n)\n -+{\n -+\tint i = 0;\n -+\tstruct reftable_merged_table *mt = NULL;\n -+\tint err;\n -+\tstruct reftable_table *tabs =\n -+\t\treftable_calloc(n * sizeof(struct reftable_table));\n -+\t*readers = reftable_calloc(n * sizeof(struct reftable_reader *));\n -+\t*source = reftable_calloc(n * sizeof(**source));\n -+\tfor (i = 0; i < n; i++) {\n -+\t\twrite_test_table(&buf[i], refs[i], sizes[i]);\n -+\t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n -+\n -+\t\terr = reftable_new_reader(&(*readers)[i], &(*source)[i],\n -+\t\t\t\t\t \"name\");\n -+\t\tEXPECT_ERR(err);\n -+\t\treftable_table_from_reader(&tabs[i], (*readers)[i]);\n -+\t}\n -+\n -+\terr = reftable_new_merged_table(&mt, tabs, n, SHA1_ID);\n -+\tEXPECT_ERR(err);\n -+\treturn mt;\n -+}\n -+\n -+static void readers_destroy(struct reftable_reader **readers, size_t n)\n -+{\n -+\tint i = 0;\n -+\tfor (; i < n; i++)\n -+\t\treftable_reader_free(readers[i]);\n -+\treftable_free(readers);\n -+}\n -+\n -+static void test_merged_between(void)\n -+{\n -+\tuint8_t hash1[SHA1_SIZE] = { 1, 2, 3, 0 };\n -+\n -+\tstruct reftable_ref_record r1[] = { {\n -+\t\t.refname = \"b\",\n -+\t\t.update_index = 1,\n -+\t\t.value_type = REFTABLE_REF_VAL1,\n -+\t\t.value.val1 = hash1,\n -+\t} };\n -+\tstruct reftable_ref_record r2[] = { {\n -+\t\t.refname = \"a\",\n -+\t\t.update_index = 2,\n -+\t\t.value_type = REFTABLE_REF_DELETION,\n -+\t} };\n -+\n -+\tstruct reftable_ref_record *refs[] = { r1, r2 };\n -+\tint sizes[] = { 1, 1 };\n -+\tstruct strbuf bufs[2] = { STRBUF_INIT, STRBUF_INIT };\n -+\tstruct reftable_block_source *bs = NULL;\n -+\tstruct reftable_reader **readers = NULL;\n -+\tstruct reftable_merged_table *mt =\n -+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 2);\n -+\tint i;\n -+\tstruct reftable_ref_record ref = { NULL };\n -+\tstruct reftable_iterator it = { NULL };\n -+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n -+\tEXPECT_ERR(err);\n -+\n -+\terr = reftable_iterator_next_ref(&it, &ref);\n -+\tEXPECT_ERR(err);\n -+\tEXPECT(ref.update_index == 2);\n -+\treftable_ref_record_release(&ref);\n -+\treftable_iterator_destroy(&it);\n -+\treaders_destroy(readers, 2);\n -+\treftable_merged_table_free(mt);\n -+\tfor (i = 0; i < ARRAY_SIZE(bufs); i++) {\n -+\t\tstrbuf_release(&bufs[i]);\n -+\t}\n -+\treftable_free(bs);\n -+}\n -+\n -+static void test_merged(void)\n -+{\n -+\tuint8_t hash1[SHA1_SIZE] = { 1 };\n -+\tuint8_t hash2[SHA1_SIZE] = { 2 };\n -+\tstruct reftable_ref_record r1[] = {\n -+\t\t{\n -+\t\t\t.refname = \"a\",\n -+\t\t\t.update_index = 1,\n -+\t\t\t.value_type = REFTABLE_REF_VAL1,\n -+\t\t\t.value.val1 = hash1,\n -+\t\t},\n -+\t\t{\n -+\t\t\t.refname = \"b\",\n -+\t\t\t.update_index = 1,\n -+\t\t\t.value_type = REFTABLE_REF_VAL1,\n -+\t\t\t.value.val1 = hash1,\n -+\t\t},\n -+\t\t{\n -+\t\t\t.refname = \"c\",\n -+\t\t\t.update_index = 1,\n -+\t\t\t.value_type = REFTABLE_REF_VAL1,\n -+\t\t\t.value.val1 = hash1,\n -+\t\t}\n -+\t};\n -+\tstruct reftable_ref_record r2[] = { {\n -+\t\t.refname = \"a\",\n -+\t\t.update_index = 2,\n -+\t\t.value_type = REFTABLE_REF_DELETION,\n -+\t} };\n -+\tstruct reftable_ref_record r3[] = {\n -+\t\t{\n -+\t\t\t.refname = \"c\",\n -+\t\t\t.update_index = 3,\n -+\t\t\t.value_type = REFTABLE_REF_VAL1,\n -+\t\t\t.value.val1 = hash2,\n -+\t\t},\n -+\t\t{\n -+\t\t\t.refname = \"d\",\n -+\t\t\t.update_index = 3,\n -+\t\t\t.value_type = REFTABLE_REF_VAL1,\n -+\t\t\t.value.val1 = hash1,\n -+\t\t},\n -+\t};\n -+\n -+\tstruct reftable_ref_record want[] = {\n -+\t\tr2[0],\n -+\t\tr1[1],\n -+\t\tr3[0],\n -+\t\tr3[1],\n -+\t};\n -+\n -+\tstruct reftable_ref_record *refs[] = { r1, r2, r3 };\n -+\tint sizes[3] = { 3, 1, 2 };\n -+\tstruct strbuf bufs[3] = { STRBUF_INIT, STRBUF_INIT, STRBUF_INIT };\n -+\tstruct reftable_block_source *bs = NULL;\n -+\tstruct reftable_reader **readers = NULL;\n -+\tstruct reftable_merged_table *mt =\n -+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 3);\n -+\n -+\tstruct reftable_iterator it = { NULL };\n -+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n -+\tstruct reftable_ref_record *out = NULL;\n -+\tsize_t len = 0;\n -+\tsize_t cap = 0;\n -+\tint i = 0;\n -+\n -+\tEXPECT_ERR(err);\n -+\twhile (len < 100) { /* cap loops/recursion. */\n -+\t\tstruct reftable_ref_record ref = { NULL };\n -+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n -+\t\tif (err > 0) {\n -+\t\t\tbreak;\n -+\t\t}\n -+\t\tif (len == cap) {\n -+\t\t\tcap = 2 * cap + 1;\n -+\t\t\tout = reftable_realloc(\n -+\t\t\t\tout, sizeof(struct reftable_ref_record) * cap);\n -+\t\t}\n -+\t\tout[len++] = ref;\n -+\t}\n -+\treftable_iterator_destroy(&it);\n -+\n -+\tassert(ARRAY_SIZE(want) == len);\n -+\tfor (i = 0; i < len; i++) {\n -+\t\tassert(reftable_ref_record_equal(&want[i], &out[i], SHA1_SIZE));\n -+\t}\n -+\tfor (i = 0; i < len; i++) {\n -+\t\treftable_ref_record_release(&out[i]);\n -+\t}\n -+\treftable_free(out);\n -+\n -+\tfor (i = 0; i < 3; i++) {\n -+\t\tstrbuf_release(&bufs[i]);\n -+\t}\n -+\treaders_destroy(readers, 3);\n -+\treftable_merged_table_free(mt);\n -+\treftable_free(bs);\n -+}\n -+\n -+static void test_default_write_opts(void)\n -+{\n -+\tstruct reftable_write_options opts = { 0 };\n -+\tstruct strbuf buf = STRBUF_INIT;\n -+\tstruct reftable_writer *w =\n -+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n -+\n -+\tstruct reftable_ref_record rec = {\n -+\t\t.refname = \"master\",\n -+\t\t.update_index = 1,\n -+\t};\n -+\tint err;\n -+\tstruct reftable_block_source source = { NULL };\n -+\tstruct reftable_table *tab = reftable_calloc(sizeof(*tab) * 1);\n -+\tuint32_t hash_id;\n -+\tstruct reftable_reader *rd = NULL;\n -+\tstruct reftable_merged_table *merged = NULL;\n -+\n -+\treftable_writer_set_limits(w, 1, 1);\n -+\n -+\terr = reftable_writer_add_ref(w, &rec);\n -+\tEXPECT_ERR(err);\n -+\n -+\terr = reftable_writer_close(w);\n -+\tEXPECT_ERR(err);\n -+\treftable_writer_free(w);\n -+\n -+\tblock_source_from_strbuf(&source, &buf);\n -+\n -+\terr = reftable_new_reader(&rd, &source, \"filename\");\n -+\tEXPECT_ERR(err);\n -+\n -+\thash_id = reftable_reader_hash_id(rd);\n -+\tassert(hash_id == SHA1_ID);\n -+\n -+\treftable_table_from_reader(&tab[0], rd);\n -+\terr = reftable_new_merged_table(&merged, tab, 1, SHA1_ID);\n -+\tEXPECT_ERR(err);\n -+\n -+\treftable_reader_free(rd);\n -+\treftable_merged_table_free(merged);\n -+\tstrbuf_release(&buf);\n -+}\n -+\n -+/* XXX test refs_for(oid) */\n -+\n -+int merged_test_main(int argc, const char *argv[])\n -+{\n -+\tRUN_TEST(test_merged_between);\n -+\tRUN_TEST(test_pq);\n -+\tRUN_TEST(test_merged);\n -+\tRUN_TEST(test_default_write_opts);\n -+\treturn 0;\n -+}\n -\n - ## reftable/pq.c (new) ##\n -@@\n -+/*\n -+Copyright 2020 Google LLC\n -+\n -+Use of this source code is governed by a BSD-style\n -+license that can be found in the LICENSE file or at\n -+https://developers.google.com/open-source/licenses/bsd\n -+*/\n -+\n -+#include \"pq.h\"\n -+\n -+#include \"reftable-record.h\"\n -+#include \"system.h\"\n -+#include \"basics.h\"\n -+\n -+static int pq_less(struct pq_entry a, struct pq_entry b)\n -+{\n -+\tstruct strbuf ak = STRBUF_INIT;\n -+\tstruct strbuf bk = STRBUF_INIT;\n -+\tint cmp = 0;\n -+\treftable_record_key(&a.rec, &ak);\n -+\treftable_record_key(&b.rec, &bk);\n -+\n -+\tcmp = strbuf_cmp(&ak, &bk);\n -+\n -+\tstrbuf_release(&ak);\n -+\tstrbuf_release(&bk);\n -+\n -+\tif (cmp == 0)\n -+\t\treturn a.index > b.index;\n -+\n -+\treturn cmp < 0;\n -+}\n -+\n -+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq)\n -+{\n -+\treturn pq.heap[0];\n -+}\n -+\n -+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq)\n -+{\n -+\treturn pq.len == 0;\n -+}\n -+\n -+void merged_iter_pqueue_check(struct merged_iter_pqueue pq)\n -+{\n -+\tint i = 0;\n -+\tfor (i = 1; i < pq.len; i++) {\n -+\t\tint parent = (i - 1) / 2;\n -+\n -+\t\tassert(pq_less(pq.heap[parent], pq.heap[i]));\n -+\t}\n -+}\n -+\n -+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n -+{\n -+\tint i = 0;\n -+\tstruct pq_entry e = pq->heap[0];\n -+\tpq->heap[0] = pq->heap[pq->len - 1];\n -+\tpq->len--;\n -+\n -+\ti = 0;\n -+\twhile (i < pq->len) {\n -+\t\tint min = i;\n -+\t\tint j = 2 * i + 1;\n -+\t\tint k = 2 * i + 2;\n -+\t\tif (j < pq->len && pq_less(pq->heap[j], pq->heap[i])) {\n -+\t\t\tmin = j;\n -+\t\t}\n -+\t\tif (k < pq->len && pq_less(pq->heap[k], pq->heap[min])) {\n -+\t\t\tmin = k;\n -+\t\t}\n -+\n -+\t\tif (min == i) {\n -+\t\t\tbreak;\n -+\t\t}\n -+\n -+\t\tSWAP(pq->heap[i], pq->heap[min]);\n -+\t\ti = min;\n -+\t}\n -+\n -+\treturn e;\n -+}\n -+\n -+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)\n -+{\n -+\tint i = 0;\n -+\tif (pq->len == pq->cap) {\n -+\t\tpq->cap = 2 * pq->cap + 1;\n -+\t\tpq->heap = reftable_realloc(pq->heap,\n -+\t\t\t\t\t pq->cap * sizeof(struct pq_entry));\n -+\t}\n -+\n -+\tpq->heap[pq->len++] = e;\n -+\ti = pq->len - 1;\n -+\twhile (i > 0) {\n -+\t\tint j = (i - 1) / 2;\n -+\t\tif (pq_less(pq->heap[j], pq->heap[i])) {\n -+\t\t\tbreak;\n -+\t\t}\n -+\n -+\t\tSWAP(pq->heap[j], pq->heap[i]);\n -+\n -+\t\ti = j;\n -+\t}\n -+}\n -+\n -+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)\n -+{\n -+\tint i = 0;\n -+\tfor (i = 0; i < pq->len; i++) {\n -+\t\treftable_record_destroy(&pq->heap[i].rec);\n -+\t}\n -+\tFREE_AND_NULL(pq->heap);\n -+\tpq->len = pq->cap = 0;\n -+}\n -\n - ## reftable/pq.h (new) ##\n -@@\n -+/*\n -+Copyright 2020 Google LLC\n -+\n -+Use of this source code is governed by a BSD-style\n -+license that can be found in the LICENSE file or at\n -+https://developers.google.com/open-source/licenses/bsd\n -+*/\n -+\n -+#ifndef PQ_H\n -+#define PQ_H\n -+\n -+#include \"record.h\"\n -+\n -+struct pq_entry {\n -+\tint index;\n -+\tstruct reftable_record rec;\n -+};\n -+\n -+struct merged_iter_pqueue {\n -+\tstruct pq_entry *heap;\n -+\tsize_t len;\n -+\tsize_t cap;\n -+};\n -+\n -+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq);\n -+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq);\n -+void merged_iter_pqueue_check(struct merged_iter_pqueue pq);\n -+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n -+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e);\n -+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n -+\n -+#endif\n -\n - ## reftable/refname.c (new) ##\n -@@\n -+/*\n -+ Copyright 2020 Google LLC\n -+\n -+ Use of this source code is governed by a BSD-style\n -+ license that can be found in the LICENSE file or at\n -+ https://developers.google.com/open-source/licenses/bsd\n -+*/\n -+\n -+#include \"system.h\"\n -+#include \"reftable-error.h\"\n -+#include \"basics.h\"\n -+#include \"refname.h\"\n -+#include \"reftable-iterator.h\"\n -+\n -+struct find_arg {\n -+\tchar **names;\n -+\tconst char *want;\n -+};\n -+\n -+static int find_name(size_t k, void *arg)\n -+{\n -+\tstruct find_arg *f_arg = (struct find_arg *)arg;\n -+\treturn strcmp(f_arg->names[k], f_arg->want) >= 0;\n -+}\n -+\n -+static int modification_has_ref(struct modification *mod, const char *name)\n -+{\n -+\tstruct reftable_ref_record ref = { NULL };\n -+\tint err = 0;\n -+\n -+\tif (mod->add_len > 0) {\n -+\t\tstruct find_arg arg = {\n -+\t\t\t.names = mod->add,\n -+\t\t\t.want = name,\n -+\t\t};\n -+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n -+\t\tif (idx < mod->add_len && !strcmp(mod->add[idx], name)) {\n -+\t\t\treturn 0;\n -+\t\t}\n -+\t}\n -+\n -+\tif (mod->del_len > 0) {\n -+\t\tstruct find_arg arg = {\n -+\t\t\t.names = mod->del,\n -+\t\t\t.want = name,\n -+\t\t};\n -+\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n -+\t\tif (idx < mod->del_len && !strcmp(mod->del[idx], name)) {\n -+\t\t\treturn 1;\n -+\t\t}\n -+\t}\n -+\n -+\terr = reftable_table_read_ref(&mod->tab, name, &ref);\n -+\treftable_ref_record_release(&ref);\n -+\treturn err;\n -+}\n -+\n -+static void modification_release(struct modification *mod)\n -+{\n -+\t/* don't delete the strings themselves; they're owned by ref records.\n -+\t */\n -+\tFREE_AND_NULL(mod->add);\n -+\tFREE_AND_NULL(mod->del);\n -+\tmod->add_len = 0;\n -+\tmod->del_len = 0;\n -+}\n -+\n -+static int modification_has_ref_with_prefix(struct modification *mod,\n -+\t\t\t\t\t const char *prefix)\n -+{\n -+\tstruct reftable_iterator it = { NULL };\n -+\tstruct reftable_ref_record ref = { NULL };\n -+\tint err = 0;\n -+\n -+\tif (mod->add_len > 0) {\n -+\t\tstruct find_arg arg = {\n -+\t\t\t.names = mod->add,\n -+\t\t\t.want = prefix,\n -+\t\t};\n -+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n -+\t\tif (idx < mod->add_len &&\n -+\t\t !strncmp(prefix, mod->add[idx], strlen(prefix)))\n -+\t\t\tgoto done;\n -+\t}\n -+\terr = reftable_table_seek_ref(&mod->tab, &it, prefix);\n -+\tif (err)\n -+\t\tgoto done;\n -+\n -+\twhile (1) {\n -+\t\terr = reftable_iterator_next_ref(&it, &ref);\n -+\t\tif (err)\n -+\t\t\tgoto done;\n -+\n -+\t\tif (mod->del_len > 0) {\n -+\t\t\tstruct find_arg arg = {\n -+\t\t\t\t.names = mod->del,\n -+\t\t\t\t.want = ref.refname,\n -+\t\t\t};\n -+\t\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n -+\t\t\tif (idx < mod->del_len &&\n -+\t\t\t !strcmp(ref.refname, mod->del[idx])) {\n -+\t\t\t\tcontinue;\n -+\t\t\t}\n -+\t\t}\n -+\n -+\t\tif (strncmp(ref.refname, prefix, strlen(prefix))) {\n -+\t\t\terr = 1;\n -+\t\t\tgoto done;\n -+\t\t}\n -+\t\terr = 0;\n -+\t\tgoto done;\n -+\t}\n -+\n -+done:\n -+\treftable_ref_record_release(&ref);\n -+\treftable_iterator_destroy(&it);\n -+\treturn err;\n -+}\n -+\n -+static int validate_refname(const char *name)\n -+{\n -+\twhile (1) {\n -+\t\tchar *next = strchr(name, '/');\n -+\t\tif (!*name) {\n -+\t\t\treturn REFTABLE_REFNAME_ERROR;\n -+\t\t}\n -+\t\tif (!next) {\n -+\t\t\treturn 0;\n -+\t\t}\n -+\t\tif (next - name == 0 || (next - name == 1 && *name == '.') ||\n -+\t\t (next - name == 2 && name[0] == '.' && name[1] == '.'))\n -+\t\t\treturn REFTABLE_REFNAME_ERROR;\n -+\t\tname = next + 1;\n -+\t}\n -+\treturn 0;\n -+}\n -+\n -+int validate_ref_record_addition(struct reftable_table tab,\n -+\t\t\t\t struct reftable_ref_record *recs, size_t sz)\n -+{\n -+\tstruct modification mod = {\n -+\t\t.tab = tab,\n -+\t\t.add = reftable_calloc(sizeof(char *) * sz),\n -+\t\t.del = reftable_calloc(sizeof(char *) * sz),\n -+\t};\n -+\tint i = 0;\n -+\tint err = 0;\n -+\tfor (; i < sz; i++) {\n -+\t\tif (reftable_ref_record_is_deletion(&recs[i])) {\n -+\t\t\tmod.del[mod.del_len++] = recs[i].refname;\n -+\t\t} else {\n -+\t\t\tmod.add[mod.add_len++] = recs[i].refname;\n -+\t\t}\n -+\t}\n -+\n -+\terr = modification_validate(&mod);\n -+\tmodification_release(&mod);\n -+\treturn err;\n -+}\n -+\n -+static void strbuf_trim_component(struct strbuf *sl)\n -+{\n -+\twhile (sl->len > 0) {\n -+\t\tint is_slash = (sl->buf[sl->len - 1] == '/');\n -+\t\tstrbuf_setlen(sl, sl->len - 1);\n -+\t\tif (is_slash)\n -+\t\t\tbreak;\n -+\t}\n -+}\n -+\n -+int modification_validate(struct modification *mod)\n -+{\n -+\tstruct strbuf slashed = STRBUF_INIT;\n -+\tint err = 0;\n -+\tint i = 0;\n -+\tfor (; i < mod->add_len; i++) {\n -+\t\terr = validate_refname(mod->add[i]);\n -+\t\tif (err)\n -+\t\t\tgoto done;\n -+\t\tstrbuf_reset(&slashed);\n -+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n -+\t\tstrbuf_addstr(&slashed, \"/\");\n -+\n -+\t\terr = modification_has_ref_with_prefix(mod, slashed.buf);\n -+\t\tif (err == 0) {\n -+\t\t\terr = REFTABLE_NAME_CONFLICT;\n -+\t\t\tgoto done;\n -+\t\t}\n -+\t\tif (err < 0)\n -+\t\t\tgoto done;\n -+\n -+\t\tstrbuf_reset(&slashed);\n -+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n -+\t\twhile (slashed.len) {\n -+\t\t\tstrbuf_trim_component(&slashed);\n -+\t\t\terr = modification_has_ref(mod, slashed.buf);\n -+\t\t\tif (err == 0) {\n -+\t\t\t\terr = REFTABLE_NAME_CONFLICT;\n -+\t\t\t\tgoto done;\n -+\t\t\t}\n -+\t\t\tif (err < 0)\n -+\t\t\t\tgoto done;\n -+\t\t}\n -+\t}\n -+\terr = 0;\n -+done:\n -+\tstrbuf_release(&slashed);\n -+\treturn err;\n -+}\n -\n - ## reftable/refname.h (new) ##\n -@@\n -+/*\n -+ Copyright 2020 Google LLC\n -+\n -+ Use of this source code is governed by a BSD-style\n -+ license that can be found in the LICENSE file or at\n -+ https://developers.google.com/open-source/licenses/bsd\n -+*/\n -+#ifndef REFNAME_H\n -+#define REFNAME_H\n -+\n -+#include \"reftable-record.h\"\n -+#include \"reftable-generic.h\"\n -+\n -+struct modification {\n -+\tstruct reftable_table tab;\n -+\n -+\tchar **add;\n -+\tsize_t add_len;\n -+\n -+\tchar **del;\n -+\tsize_t del_len;\n -+};\n -+\n -+int validate_ref_record_addition(struct reftable_table tab,\n -+\t\t\t\t struct reftable_ref_record *recs, size_t sz);\n -+\n -+int modification_validate(struct modification *mod);\n -+\n -+#endif\n -\n - ## reftable/refname_test.c (new) ##\n -@@\n -+/*\n -+Copyright 2020 Google LLC\n -+\n -+Use of this source code is governed by a BSD-style\n -+license that can be found in the LICENSE file or at\n -+https://developers.google.com/open-source/licenses/bsd\n -+*/\n -+\n -+#include \"basics.h\"\n -+#include \"block.h\"\n -+#include \"blocksource.h\"\n -+#include \"constants.h\"\n -+#include \"reader.h\"\n -+#include \"record.h\"\n -+#include \"refname.h\"\n -+#include \"reftable-error.h\"\n -+#include \"reftable-writer.h\"\n -+#include \"system.h\"\n -+\n -+#include \"test_framework.h\"\n -+#include \"reftable-tests.h\"\n -+\n -+struct testcase {\n -+\tchar *add;\n -+\tchar *del;\n -+\tint error_code;\n -+};\n -+\n -+static void test_conflict(void)\n -+{\n -+\tstruct reftable_write_options opts = { 0 };\n -+\tstruct strbuf buf = STRBUF_INIT;\n -+\tstruct reftable_writer *w =\n -+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n -+\tstruct reftable_ref_record rec = {\n -+\t\t.refname = \"a/b\",\n -+\t\t.value_type = REFTABLE_REF_SYMREF,\n -+\t\t.value.symref = \"destination\", /* make sure it's not a symref.\n -+\t\t\t\t\t\t*/\n -+\t\t.update_index = 1,\n -+\t};\n -+\tint err;\n -+\tint i;\n -+\tstruct reftable_block_source source = { NULL };\n -+\tstruct reftable_reader *rd = NULL;\n -+\tstruct reftable_table tab = { NULL };\n -+\tstruct testcase cases[] = {\n -+\t\t{ \"a/b/c\", NULL, REFTABLE_NAME_CONFLICT },\n -+\t\t{ \"b\", NULL, 0 },\n -+\t\t{ \"a\", NULL, REFTABLE_NAME_CONFLICT },\n -+\t\t{ \"a\", \"a/b\", 0 },\n -+\n -+\t\t{ \"p/\", NULL, REFTABLE_REFNAME_ERROR },\n -+\t\t{ \"p//q\", NULL, REFTABLE_REFNAME_ERROR },\n -+\t\t{ \"p/./q\", NULL, REFTABLE_REFNAME_ERROR },\n -+\t\t{ \"p/../q\", NULL, REFTABLE_REFNAME_ERROR },\n -+\n -+\t\t{ \"a/b/c\", \"a/b\", 0 },\n -+\t\t{ NULL, \"a//b\", 0 },\n -+\t};\n -+\treftable_writer_set_limits(w, 1, 1);\n -+\n -+\terr = reftable_writer_add_ref(w, &rec);\n -+\tEXPECT_ERR(err);\n -+\n -+\terr = reftable_writer_close(w);\n -+\tEXPECT_ERR(err);\n -+\treftable_writer_free(w);\n -+\n -+\tblock_source_from_strbuf(&source, &buf);\n -+\terr = reftable_new_reader(&rd, &source, \"filename\");\n -+\tEXPECT_ERR(err);\n -+\n -+\treftable_table_from_reader(&tab, rd);\n -+\n -+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n -+\t\tstruct modification mod = {\n -+\t\t\t.tab = tab,\n -+\t\t};\n -+\n -+\t\tif (cases[i].add != NULL) {\n -+\t\t\tmod.add = &cases[i].add;\n -+\t\t\tmod.add_len = 1;\n -+\t\t}\n -+\t\tif (cases[i].del != NULL) {\n -+\t\t\tmod.del = &cases[i].del;\n -+\t\t\tmod.del_len = 1;\n -+\t\t}\n -+\n -+\t\terr = modification_validate(&mod);\n -+\t\tEXPECT(err == cases[i].error_code);\n -+\t}\n -+\n -+\treftable_reader_free(rd);\n -+\tstrbuf_release(&buf);\n -+}\n -+\n -+int refname_test_main(int argc, const char *argv[])\n -+{\n -+\tRUN_TEST(test_conflict);\n -+\treturn 0;\n -+}\n -\n - ## reftable/reftable-generic.h (new) ##\n -@@\n -+/*\n -+Copyright 2020 Google LLC\n -+\n -+Use of this source code is governed by a BSD-style\n -+license that can be found in the LICENSE file or at\n -+https://developers.google.com/open-source/licenses/bsd\n -+*/\n -+\n -+#ifndef REFTABLE_GENERIC_H\n -+#define REFTABLE_GENERIC_H\n -+\n -+#include \"reftable-iterator.h\"\n -+#include \"reftable-reader.h\"\n -+#include \"reftable-merged.h\"\n -+\n -+/*\n -+ * Provides a unified API for reading tables, either merged tables, or single\n -+ * readers. */\n -+struct reftable_table {\n -+\tstruct reftable_table_vtable *ops;\n -+\tvoid *table_arg;\n -+};\n -+\n -+int reftable_table_seek_ref(struct reftable_table *tab,\n -+\t\t\t struct reftable_iterator *it, const char *name);\n -+\n -+void reftable_table_from_reader(struct reftable_table *tab,\n -+\t\t\t\tstruct reftable_reader *reader);\n -+\n -+/* returns the hash ID from a generic reftable_table */\n -+uint32_t reftable_table_hash_id(struct reftable_table *tab);\n -+\n -+/* create a generic table from reftable_merged_table */\n -+void reftable_table_from_merged_table(struct reftable_table *tab,\n -+\t\t\t\t struct reftable_merged_table *table);\n -+\n -+/* returns the max update_index covered by this table. */\n -+uint64_t reftable_table_max_update_index(struct reftable_table *tab);\n -+\n -+/* returns the min update_index covered by this table. */\n -+uint64_t reftable_table_min_update_index(struct reftable_table *tab);\n -+\n -+/* convenience function to read a single ref. Returns < 0 for error, 0\n -+ for success, and 1 if ref not found. */\n -+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n -+\t\t\t struct reftable_ref_record *ref);\n -+\n -+#endif\n -\n - ## reftable/reftable-merged.h (new) ##\n -@@\n -+/*\n -+Copyright 2020 Google LLC\n -+\n -+Use of this source code is governed by a BSD-style\n -+license that can be found in the LICENSE file or at\n -+https://developers.google.com/open-source/licenses/bsd\n -+*/\n -+\n -+#ifndef REFTABLE_MERGED_H\n -+#define REFTABLE_MERGED_H\n -+\n -+#include \"reftable-iterator.h\"\n -+\n -+/*\n -+ * Merged tables\n -+ *\n -+ * A ref database kept in a sequence of table files. The merged_table presents a\n -+ * unified view to reading (seeking, iterating) a sequence of immutable tables.\n -+ *\n -+ * The merged tables are on purpose kept disconnected from their actual storage\n -+ * (eg. files on disk), because it is useful to merge tables aren't files. For\n -+ * example, the per-workspace and global ref namespace can be implemented as a\n -+ * merged table of two stacks of file-backed reftables.\n -+ */\n -+\n -+/* A merged table is implements seeking/iterating over a stack of tables. */\n -+struct reftable_merged_table;\n -+\n -+/* A generic reftable; see below. */\n -+struct reftable_table;\n -+\n -+/* reftable_new_merged_table creates a new merged table. It takes ownership of\n -+ the stack array.\n -+*/\n -+int reftable_new_merged_table(struct reftable_merged_table **dest,\n -+\t\t\t struct reftable_table *stack, int n,\n -+\t\t\t uint32_t hash_id);\n -+\n -+/* returns an iterator positioned just before 'name' */\n -+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n -+\t\t\t\t struct reftable_iterator *it,\n -+\t\t\t\t const char *name);\n -+\n -+/* returns an iterator for log entry, at given update_index */\n -+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n -+\t\t\t\t struct reftable_iterator *it,\n -+\t\t\t\t const char *name, uint64_t update_index);\n -+\n -+/* like reftable_merged_table_seek_log_at but look for the newest entry. */\n -+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n -+\t\t\t\t struct reftable_iterator *it,\n -+\t\t\t\t const char *name);\n -+\n -+/* returns the max update_index covered by this merged table. */\n -+uint64_t\n -+reftable_merged_table_max_update_index(struct reftable_merged_table *mt);\n -+\n -+/* returns the min update_index covered by this merged table. */\n -+uint64_t\n -+reftable_merged_table_min_update_index(struct reftable_merged_table *mt);\n -+\n -+/* releases memory for the merged_table */\n -+void reftable_merged_table_free(struct reftable_merged_table *m);\n -+\n -+/* return the hash ID of the merged table. */\n -+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m);\n -+\n -+#endif\n +@@ Makefile: REFTABLE_TEST_OBJS += reftable/pq_test.o\n + REFTABLE_TEST_OBJS += reftable/record_test.o\n + REFTABLE_TEST_OBJS += reftable/refname_test.o\n + REFTABLE_TEST_OBJS += reftable/reftable_test.o\n ++REFTABLE_TEST_OBJS += reftable/stack_test.o\n + REFTABLE_TEST_OBJS += reftable/test_framework.o\n + REFTABLE_TEST_OBJS += reftable/tree_test.o\n + \n \n ## reftable/reftable-stack.h (new) ##\n @@\n @@ reftable/reftable-stack.h (new)\n +/* heuristically compact unbalanced table stack. */\n +int reftable_stack_auto_compact(struct reftable_stack *st);\n +\n ++/* delete stale .ref tables. */\n ++int reftable_stack_clean(struct reftable_stack *st);\n ++\n +/* convenience function to read a single ref. Returns < 0 for error, 0 for\n + * success, and 1 if ref not found. */\n +int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n @@ reftable/reftable-stack.h (new)\n +\n +#endif\n \n - ## reftable/reftable.c (new) ##\n -@@\n -+/*\n -+Copyright 2020 Google LLC\n -+\n -+Use of this source code is governed by a BSD-style\n -+license that can be found in the LICENSE file or at\n -+https://developers.google.com/open-source/licenses/bsd\n -+*/\n -+\n -+#include \"record.h\"\n -+#include \"reader.h\"\n -+#include \"reftable-iterator.h\"\n -+#include \"reftable-generic.h\"\n -+\n -+static int reftable_reader_seek_void(void *tab, struct reftable_iterator *it,\n -+\t\t\t\t struct reftable_record *rec)\n -+{\n -+\treturn reader_seek((struct reftable_reader *)tab, it, rec);\n -+}\n -+\n -+static uint32_t reftable_reader_hash_id_void(void *tab)\n -+{\n -+\treturn reftable_reader_hash_id((struct reftable_reader *)tab);\n -+}\n -+\n -+static uint64_t reftable_reader_min_update_index_void(void *tab)\n -+{\n -+\treturn reftable_reader_min_update_index((struct reftable_reader *)tab);\n -+}\n -+\n -+static uint64_t reftable_reader_max_update_index_void(void *tab)\n -+{\n -+\treturn reftable_reader_max_update_index((struct reftable_reader *)tab);\n -+}\n -+\n -+static struct reftable_table_vtable reader_vtable = {\n -+\t.seek_record = reftable_reader_seek_void,\n -+\t.hash_id = reftable_reader_hash_id_void,\n -+\t.min_update_index = reftable_reader_min_update_index_void,\n -+\t.max_update_index = reftable_reader_max_update_index_void,\n -+};\n -+\n -+int reftable_table_seek_ref(struct reftable_table *tab,\n -+\t\t\t struct reftable_iterator *it, const char *name)\n -+{\n -+\tstruct reftable_ref_record ref = {\n -+\t\t.refname = (char *)name,\n -+\t};\n -+\tstruct reftable_record rec = { NULL };\n -+\treftable_record_from_ref(&rec, &ref);\n -+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n -+}\n -+\n -+void reftable_table_from_reader(struct reftable_table *tab,\n -+\t\t\t\tstruct reftable_reader *reader)\n -+{\n -+\tassert(tab->ops == NULL);\n -+\ttab->ops = &reader_vtable;\n -+\ttab->table_arg = reader;\n -+}\n -+\n -+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n -+\t\t\t struct reftable_ref_record *ref)\n -+{\n -+\tstruct reftable_iterator it = { NULL };\n -+\tint err = reftable_table_seek_ref(tab, &it, name);\n -+\tif (err)\n -+\t\tgoto done;\n -+\n -+\terr = reftable_iterator_next_ref(&it, ref);\n -+\tif (err)\n -+\t\tgoto done;\n -+\n -+\tif (strcmp(ref->refname, name) ||\n -+\t reftable_ref_record_is_deletion(ref)) {\n -+\t\treftable_ref_record_release(ref);\n -+\t\terr = 1;\n -+\t\tgoto done;\n -+\t}\n -+\n -+done:\n -+\treftable_iterator_destroy(&it);\n -+\treturn err;\n -+}\n -+\n -+uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n -+{\n -+\treturn tab->ops->max_update_index(tab->table_arg);\n -+}\n -+\n -+uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n -+{\n -+\treturn tab->ops->min_update_index(tab->table_arg);\n -+}\n -+\n -+uint32_t reftable_table_hash_id(struct reftable_table *tab)\n -+{\n -+\treturn tab->ops->hash_id(tab->table_arg);\n -+}\n -\n ## reftable/stack.c (new) ##\n @@\n +/*\n @@ reftable/stack.c (new)\n +#include \"refname.h\"\n +#include \"reftable-error.h\"\n +#include \"reftable-record.h\"\n ++#include \"reftable-merged.h\"\n +#include \"writer.h\"\n +\n +static int stack_try_add(struct reftable_stack *st,\n @@ reftable/stack.c (new)\n +static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n +\t\t\t\t\t int reuse_open);\n +\n ++static void stack_filename(struct strbuf *dest, struct reftable_stack *st,\n ++\t\t\t const char *name)\n ++{\n ++\tstrbuf_reset(dest);\n ++\tstrbuf_addstr(dest, st->reftable_dir);\n ++\tstrbuf_addstr(dest, \"/\");\n ++\tstrbuf_addstr(dest, name);\n ++}\n ++\n +static int reftable_fd_write(void *arg, const void *data, size_t sz)\n +{\n +\tint *fdp = (int *)arg;\n @@ reftable/stack.c (new)\n +\treturn st->merged;\n +}\n +\n ++static int has_name(char **names, const char *name)\n ++{\n ++\twhile (*names) {\n ++\t\tif (!strcmp(*names, name))\n ++\t\t\treturn 1;\n ++\t\tnames++;\n ++\t}\n ++\treturn 0;\n ++}\n ++\n +/* Close and free the stack */\n +void reftable_stack_destroy(struct reftable_stack *st)\n +{\n ++\tchar **names = NULL;\n ++\tint err = 0;\n +\tif (st->merged != NULL) {\n +\t\treftable_merged_table_free(st->merged);\n +\t\tst->merged = NULL;\n +\t}\n +\n ++\terr = read_lines(st->list_file, &names);\n ++\tif (err < 0) {\n ++\t\tFREE_AND_NULL(names);\n ++\t}\n ++\n +\tif (st->readers != NULL) {\n +\t\tint i = 0;\n ++\t\tstruct strbuf filename = STRBUF_INIT;\n +\t\tfor (i = 0; i < st->readers_len; i++) {\n ++\t\t\tconst char *name = reader_name(st->readers[i]);\n ++\t\t\tstrbuf_reset(&filename);\n ++\t\t\tif (names && !has_name(names, name)) {\n ++\t\t\t\tstack_filename(&filename, st, name);\n ++\t\t\t}\n +\t\t\treftable_reader_free(st->readers[i]);\n ++\n ++\t\t\tif (filename.len) {\n ++\t\t\t\t// On Windows, can only unlink after closing.\n ++\t\t\t\tunlink(filename.buf);\n ++\t\t\t}\n +\t\t}\n ++\t\tstrbuf_release(&filename);\n +\t\tst->readers_len = 0;\n +\t\tFREE_AND_NULL(st->readers);\n +\t}\n +\tFREE_AND_NULL(st->list_file);\n +\tFREE_AND_NULL(st->reftable_dir);\n +\treftable_free(st);\n ++\tfree_names(names);\n +}\n +\n +static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n @@ reftable/stack.c (new)\n +\t\tif (rd == NULL) {\n +\t\t\tstruct reftable_block_source src = { NULL };\n +\t\t\tstruct strbuf table_path = STRBUF_INIT;\n -+\t\t\tstrbuf_addstr(&table_path, st->reftable_dir);\n -+\t\t\tstrbuf_addstr(&table_path, \"/\");\n -+\t\t\tstrbuf_addstr(&table_path, name);\n ++\t\t\tstack_filename(&table_path, st, name);\n +\n +\t\t\terr = reftable_block_source_from_file(&src,\n +\t\t\t\t\t\t\t table_path.buf);\n @@ reftable/stack.c (new)\n +\tst->merged = new_merged;\n +\tfor (i = 0; i < cur_len; i++) {\n +\t\tif (cur[i] != NULL) {\n ++\t\t\tconst char *name = reader_name(cur[i]);\n ++\t\t\tstruct strbuf filename = STRBUF_INIT;\n ++\t\t\tstack_filename(&filename, st, name);\n ++\n +\t\t\treader_close(cur[i]);\n +\t\t\treftable_reader_free(cur[i]);\n ++\n ++\t\t\t// On Windows, can only unlink after closing.\n ++\t\t\tunlink(filename.buf);\n ++\n ++\t\t\tstrbuf_release(&filename);\n +\t\t}\n +\t}\n +\n @@ reftable/stack.c (new)\n +\tint i = 0;\n +\tstruct strbuf nm = STRBUF_INIT;\n +\tfor (i = 0; i < add->new_tables_len; i++) {\n -+\t\tstrbuf_reset(&nm);\n -+\t\tstrbuf_addstr(&nm, add->stack->reftable_dir);\n -+\t\tstrbuf_addstr(&nm, \"/\");\n -+\t\tstrbuf_addstr(&nm, add->new_tables[i]);\n ++\t\tstack_filename(&nm, add->stack, add->new_tables[i]);\n +\t\tunlink(nm.buf);\n +\t\treftable_free(add->new_tables[i]);\n +\t\tadd->new_tables[i] = NULL;\n @@ reftable/stack.c (new)\n +\tstrbuf_reset(&next_name);\n +\tformat_name(&next_name, add->next_update_index, add->next_update_index);\n +\n -+\tstrbuf_addstr(&temp_tab_file_name, add->stack->reftable_dir);\n -+\tstrbuf_addstr(&temp_tab_file_name, \"/\");\n -+\tstrbuf_addbuf(&temp_tab_file_name, &next_name);\n ++\tstack_filename(&temp_tab_file_name, add->stack, next_name.buf);\n +\tstrbuf_addstr(&temp_tab_file_name, \".temp.XXXXXX\");\n +\n +\ttab_fd = mkstemp(temp_tab_file_name.buf);\n @@ reftable/stack.c (new)\n +\tformat_name(&next_name, wr->min_update_index, wr->max_update_index);\n +\tstrbuf_addstr(&next_name, \".ref\");\n +\n -+\tstrbuf_addstr(&tab_file_name, add->stack->reftable_dir);\n -+\tstrbuf_addstr(&tab_file_name, \"/\");\n -+\tstrbuf_addbuf(&tab_file_name, &next_name);\n ++\tstack_filename(&tab_file_name, add->stack, next_name.buf);\n +\n +\t/*\n +\t On windows, this relies on rand() picking a unique destination name.\n @@ reftable/stack.c (new)\n +\t\t reftable_reader_min_update_index(st->readers[first]),\n +\t\t reftable_reader_max_update_index(st->readers[last]));\n +\n -+\tstrbuf_reset(temp_tab);\n -+\tstrbuf_addstr(temp_tab, st->reftable_dir);\n -+\tstrbuf_addstr(temp_tab, \"/\");\n -+\tstrbuf_addbuf(temp_tab, &next_name);\n ++\tstack_filename(temp_tab, st, next_name.buf);\n +\tstrbuf_addstr(temp_tab, \".temp.XXXXXX\");\n +\n +\ttab_fd = mkstemp(temp_tab->buf);\n @@ reftable/stack.c (new)\n +\t\tstruct strbuf subtab_lock = STRBUF_INIT;\n +\t\tint sublock_file_fd = -1;\n +\n -+\t\tstrbuf_addstr(&subtab_file_name, st->reftable_dir);\n -+\t\tstrbuf_addstr(&subtab_file_name, \"/\");\n -+\t\tstrbuf_addstr(&subtab_file_name, reader_name(st->readers[i]));\n ++\t\tstack_filename(&subtab_file_name, st,\n ++\t\t\t reader_name(st->readers[i]));\n +\n +\t\tstrbuf_reset(&subtab_lock);\n +\t\tstrbuf_addbuf(&subtab_lock, &subtab_file_name);\n @@ reftable/stack.c (new)\n +\t\t st->readers[last]->max_update_index);\n +\tstrbuf_addstr(&new_table_name, \".ref\");\n +\n -+\tstrbuf_reset(&new_table_path);\n -+\tstrbuf_addstr(&new_table_path, st->reftable_dir);\n -+\tstrbuf_addstr(&new_table_path, \"/\");\n -+\tstrbuf_addbuf(&new_table_path, &new_table_name);\n ++\tstack_filename(&new_table_path, st, new_table_name.buf);\n +\n +\tif (!is_empty_table) {\n +\t\t/* retry? */\n @@ reftable/stack.c (new)\n +\treftable_iterator_destroy(&it);\n +\treftable_reader_free(rd);\n +\treturn err;\n ++}\n ++\n ++static int is_table_name(const char *s)\n ++{\n ++\tconst char *dot = strrchr(s, '.');\n ++\treturn dot != NULL && !strcmp(dot, \".ref\");\n ++}\n ++\n ++static void remove_maybe_stale_table(struct reftable_stack *st, uint64_t max,\n ++\t\t\t\t const char *name)\n ++{\n ++\tint err = 0;\n ++\tuint64_t update_idx = 0;\n ++\tstruct reftable_block_source src = { NULL };\n ++\tstruct reftable_reader *rd = NULL;\n ++\tstruct strbuf table_path = STRBUF_INIT;\n ++\tstack_filename(&table_path, st, name);\n ++\n ++\terr = reftable_block_source_from_file(&src, table_path.buf);\n ++\tif (err < 0)\n ++\t\tgoto done;\n ++\n ++\terr = reftable_new_reader(&rd, &src, name);\n ++\tif (err < 0)\n ++\t\tgoto done;\n ++\n ++\tupdate_idx = reftable_reader_max_update_index(rd);\n ++\treftable_reader_free(rd);\n ++\n ++\tif (update_idx <= max) {\n ++\t\tunlink(table_path.buf);\n ++\t}\n ++done:\n ++\tstrbuf_release(&table_path);\n ++}\n ++\n ++static int reftable_stack_clean_locked(struct reftable_stack *st)\n ++{\n ++\tuint64_t max = reftable_merged_table_max_update_index(\n ++\t\treftable_stack_merged_table(st));\n ++\tDIR *dir = opendir(st->reftable_dir);\n ++\tstruct dirent *d = NULL;\n ++\tif (dir == NULL) {\n ++\t\treturn REFTABLE_IO_ERROR;\n ++\t}\n ++\n ++\twhile ((d = readdir(dir)) != NULL) {\n ++\t\tint i = 0;\n ++\t\tint found = 0;\n ++\t\tif (!is_table_name(d->d_name))\n ++\t\t\tcontinue;\n ++\n ++\t\tfor (i = 0; !found && i < st->readers_len; i++) {\n ++\t\t\tfound = !strcmp(reader_name(st->readers[i]), d->d_name);\n ++\t\t}\n ++\t\tif (found)\n ++\t\t\tcontinue;\n ++\n ++\t\tremove_maybe_stale_table(st, max, d->d_name);\n ++\t}\n ++\n ++\tclosedir(dir);\n ++\treturn 0;\n ++}\n ++\n ++int reftable_stack_clean(struct reftable_stack *st)\n ++{\n ++\tstruct reftable_addition *add = NULL;\n ++\tint err = reftable_stack_new_addition(&add, st);\n ++\tif (err < 0) {\n ++\t\tgoto done;\n ++\t}\n ++\n ++\terr = reftable_stack_reload(st);\n ++\tif (err < 0) {\n ++\t\tgoto done;\n ++\t}\n ++\n ++\terr = reftable_stack_clean_locked(st);\n ++\n ++done:\n ++\treftable_addition_destroy(add);\n ++\treturn err;\n +}\n \n ## reftable/stack.h (new) ##\n @@ reftable/stack_test.c (new)\n +\n +#include \"system.h\"\n +\n ++#include \"reftable-reader.h\"\n +#include \"merged.h\"\n +#include \"basics.h\"\n +#include \"constants.h\"\n @@ reftable/stack_test.c (new)\n +\tstrbuf_release(&path);\n +}\n +\n ++static int count_dir_entries(const char *dirname)\n ++{\n ++\tDIR *dir = opendir(dirname);\n ++\tint len = 0;\n ++\tstruct dirent *d;\n ++\tif (dir == NULL)\n ++\t\treturn 0;\n ++\n ++\twhile ((d = readdir(dir)) != NULL) {\n ++\t\tif (!strcmp(d->d_name, \"..\") || !strcmp(d->d_name, \".\"))\n ++\t\t\tcontinue;\n ++\t\tlen++;\n ++\t}\n ++\tclosedir(dir);\n ++\treturn len;\n ++}\n ++\n +static char *get_tmp_template(const char *prefix)\n +{\n +\tconst char *tmp = getenv(\"TMPDIR\");\n @@ reftable/stack_test.c (new)\n +\tclear_dir(dir);\n +}\n +\n ++static void test_reftable_stack_compaction_concurrent(void)\n ++{\n ++\tstruct reftable_write_options cfg = { 0 };\n ++\tstruct reftable_stack *st1 = NULL, *st2 = NULL;\n ++\tchar *dir = get_tmp_template(__FUNCTION__);\n ++\tint err, i;\n ++\tint N = 3;\n ++\tEXPECT(mkdtemp(dir));\n ++\n ++\terr = reftable_new_stack(&st1, dir, cfg);\n ++\tEXPECT_ERR(err);\n ++\n ++\tfor (i = 0; i < N; i++) {\n ++\t\tchar name[100];\n ++\t\tstruct reftable_ref_record ref = {\n ++\t\t\t.refname = name,\n ++\t\t\t.update_index = reftable_stack_next_update_index(st1),\n ++\t\t\t.value_type = REFTABLE_REF_SYMREF,\n ++\t\t\t.value.symref = \"master\",\n ++\t\t};\n ++\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n ++\n ++\t\terr = reftable_stack_add(st1, &write_test_ref, &ref);\n ++\t\tEXPECT_ERR(err);\n ++\t}\n ++\n ++\terr = reftable_new_stack(&st2, dir, cfg);\n ++\tEXPECT_ERR(err);\n ++\n ++\terr = reftable_stack_compact_all(st1, NULL);\n ++\tEXPECT_ERR(err);\n ++\n ++\treftable_stack_destroy(st1);\n ++\treftable_stack_destroy(st2);\n ++\n ++\tEXPECT(count_dir_entries(dir) == 2);\n ++\tclear_dir(dir);\n ++}\n ++\n ++static void unclean_stack_close(struct reftable_stack *st)\n ++{\n ++\t// break abstraction boundary to simulate unclean shutdown.\n ++\tint i = 0;\n ++\tfor (; i < st->readers_len; i++) {\n ++\t\treftable_reader_free(st->readers[i]);\n ++\t}\n ++\tst->readers_len = 0;\n ++\tFREE_AND_NULL(st->readers);\n ++}\n ++\n ++static void test_reftable_stack_compaction_concurrent_clean(void)\n ++{\n ++\tstruct reftable_write_options cfg = { 0 };\n ++\tstruct reftable_stack *st1 = NULL, *st2 = NULL, *st3 = NULL;\n ++\tchar *dir = get_tmp_template(__FUNCTION__);\n ++\tint err, i;\n ++\tint N = 3;\n ++\tEXPECT(mkdtemp(dir));\n ++\n ++\terr = reftable_new_stack(&st1, dir, cfg);\n ++\tEXPECT_ERR(err);\n ++\n ++\tfor (i = 0; i < N; i++) {\n ++\t\tchar name[100];\n ++\t\tstruct reftable_ref_record ref = {\n ++\t\t\t.refname = name,\n ++\t\t\t.update_index = reftable_stack_next_update_index(st1),\n ++\t\t\t.value_type = REFTABLE_REF_SYMREF,\n ++\t\t\t.value.symref = \"master\",\n ++\t\t};\n ++\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n ++\n ++\t\terr = reftable_stack_add(st1, &write_test_ref, &ref);\n ++\t\tEXPECT_ERR(err);\n ++\t}\n ++\n ++\terr = reftable_new_stack(&st2, dir, cfg);\n ++\tEXPECT_ERR(err);\n ++\n ++\terr = reftable_stack_compact_all(st1, NULL);\n ++\tEXPECT_ERR(err);\n ++\n ++\tunclean_stack_close(st1);\n ++\tunclean_stack_close(st2);\n ++\n ++\terr = reftable_new_stack(&st3, dir, cfg);\n ++\tEXPECT_ERR(err);\n ++\n ++\terr = reftable_stack_clean(st3);\n ++\tEXPECT_ERR(err);\n ++\tEXPECT(count_dir_entries(dir) == 2);\n ++\n ++\treftable_stack_destroy(st1);\n ++\treftable_stack_destroy(st2);\n ++\treftable_stack_destroy(st3);\n ++\n ++\tclear_dir(dir);\n ++}\n ++\n +int stack_test_main(int argc, const char *argv[])\n +{\n ++\tRUN_TEST(test_reftable_stack_compaction_concurrent_clean);\n ++\tRUN_TEST(test_reftable_stack_compaction_concurrent);\n +\tRUN_TEST(test_reftable_stack_uptodate);\n +\tRUN_TEST(test_reftable_stack_transaction_api);\n +\tRUN_TEST(test_reftable_stack_hash_id);\n @@ reftable/stack_test.c (new)\n \n ## t/helper/test-reftable.c ##\n @@ t/helper/test-reftable.c: int cmd__reftable(int argc, const char **argv)\n - {\n - \tbasics_test_main(argc, argv);\n - \tblock_test_main(argc, argv);\n -+\tmerged_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n -+\trefname_test_main(argc, argv);\n + \trefname_test_main(argc, argv);\n \treftable_test_main(argc, argv);\n +\tstack_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n -: ------------ > 17: 5db7c8ab7f23 reftable: add dump utility\n 13: bdb19af22cc7 ! 18: 2dc73bf2ec96 Reftable support for git-core\n @@ Makefile: LIB_OBJS += reflog-walk.o\n LIB_OBJS += refs/iterator.o\n LIB_OBJS += refs/packed-backend.o\n LIB_OBJS += refs/ref-cache.o\n -@@ Makefile: REFTABLE_OBJS += reftable/zlib-compat.o\n - \n - REFTABLE_TEST_OBJS += reftable/basics_test.o\n - REFTABLE_TEST_OBJS += reftable/block_test.o\n -+REFTABLE_TEST_OBJS += reftable/merged_test.o\n - REFTABLE_TEST_OBJS += reftable/record_test.o\n -+REFTABLE_TEST_OBJS += reftable/refname_test.o\n - REFTABLE_TEST_OBJS += reftable/reftable_test.o\n -+REFTABLE_TEST_OBJS += reftable/stack_test.o\n - REFTABLE_TEST_OBJS += reftable/test_framework.o\n - REFTABLE_TEST_OBJS += reftable/tree_test.o\n - \n \n ## builtin/clone.c ##\n @@ builtin/clone.c: int cmd_clone(int argc, const char **argv, const char *prefix)\n @@ builtin/init-db.c: static int needs_work_tree_config(const char *git_dir, const\n \n \t/* This forces creation of new config file */\n @@ builtin/init-db.c: static int create_default_files(const char *template_path,\n - \tis_bare_repository_cfg = init_is_bare_repository;\n + \tis_bare_repository_cfg = init_is_bare_repository || !work_tree;\n \tif (init_shared_repository != -1)\n \t\tset_shared_repository(init_shared_repository);\n +\tthe_repository->ref_storage_format = xstrdup(fmt->ref_storage);\n @@ refs/reftable-backend.c (new)\n +\terr = reftable_stack_compact_all(refs->main_stack, NULL);\n +\tif (err == 0 && refs->worktree_stack != NULL)\n +\t\terr = reftable_stack_compact_all(refs->worktree_stack, NULL);\n ++\tif (err == 0)\n ++\t\terr = reftable_stack_clean(refs->main_stack);\n ++\tif (err == 0 && refs->worktree_stack != NULL)\n ++\t\terr = reftable_stack_clean(refs->worktree_stack);\n ++\n +\treturn err;\n +}\n +\n 14: 0d9477941678 = 19: b2fa6ea62c16 git-prompt: prepare for reftable refs backend\n 15: 59f432522230 ! 20: 39e644b9436b Add \"test-tool dump-reftable\" command.\n @@ Makefile: REFTABLE_OBJS += reftable/zlib-compat.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n +REFTABLE_TEST_OBJS += reftable/dump.o\n REFTABLE_TEST_OBJS += reftable/merged_test.o\n + REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n - REFTABLE_TEST_OBJS += reftable/refname_test.o\n \n ## reftable/dump.c ##\n @@ reftable/dump.c: license that can be found in the LICENSE file or at\n\n-- \ngitgitgadget\n"},{"id":"421754","messageId":"02bd1f40fa4e281f878d0b9ba4a8b5f8eb21c8ff.1618255552.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 02/20] reftable: add LICENSE","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:34Z","receivedAt":"2021-04-12T19:26:06Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nTODO: relicense?\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/LICENSE | 31 +++++++++++++++++++++++++++++++\n 1 file changed, 31 insertions(+)\n create mode 100644 reftable/LICENSE\n\ndiff --git a/reftable/LICENSE b/reftable/LICENSE\nnew file mode 100644\nindex 000000000000..402e0f9356ba\n--- /dev/null\n+++ b/reftable/LICENSE\n@@ -0,0 +1,31 @@\n+BSD License\n+\n+Copyright (c) 2020, Google LLC\n+All rights reserved.\n+\n+Redistribution and use in source and binary forms, with or without\n+modification, are permitted provided that the following conditions are\n+met:\n+\n+* Redistributions of source code must retain the above copyright notice,\n+this list of conditions and the following disclaimer.\n+\n+* Redistributions in binary form must reproduce the above copyright\n+notice, this list of conditions and the following disclaimer in the\n+documentation and/or other materials provided with the distribution.\n+\n+* Neither the name of Google LLC nor the names of its contributors may\n+be used to endorse or promote products derived from this software\n+without specific prior written permission.\n+\n+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n+\"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n-- \ngitgitgadget\n\n"},{"id":"421756","messageId":"bffa33c012adcf15c8e8e96e8853cfd73f091560.1618255552.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 03/20] reftable: add error related functionality","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:35Z","receivedAt":"2021-04-12T19:26:07Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable/ directory is structured as a library, so it cannot\ncrash on misuse. Instead, it returns an error codes.\n\nIn addition, the error code can be used to signal conditions from lower levels\nof the library to be handled by higher levels of the library. For example, a\ntransaction might legitimately write an empty reftable file, but in that case,\nwe'd want to shortcut the transaction overhead.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/error.c | 41 ++++++++++++++++++++++++++\n reftable/reftable-error.h | 62 +++++++++++++++++++++++++++++++++++++++\n 2 files changed, 103 insertions(+)\n create mode 100644 reftable/error.c\n create mode 100644 reftable/reftable-error.h\n\ndiff --git a/reftable/error.c b/reftable/error.c\nnew file mode 100644\nindex 000000000000..f6f16def9214\n--- /dev/null\n+++ b/reftable/error.c\n@@ -0,0 +1,41 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-error.h\"\n+\n+#include \n+\n+const char *reftable_error_str(int err)\n+{\n+\tstatic char buf[250];\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn \"I/O error\";\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn \"corrupt reftable file\";\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn \"file does not exist\";\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn \"data is outdated\";\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn \"misuse of the reftable API\";\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn \"zlib failure\";\n+\tcase REFTABLE_NAME_CONFLICT:\n+\t\treturn \"file/directory conflict\";\n+\tcase REFTABLE_EMPTY_TABLE_ERROR:\n+\t\treturn \"wrote empty table\";\n+\tcase REFTABLE_REFNAME_ERROR:\n+\t\treturn \"invalid refname\";\n+\tcase -1:\n+\t\treturn \"general error\";\n+\tdefault:\n+\t\tsnprintf(buf, sizeof(buf), \"unknown error code %d\", err);\n+\t\treturn buf;\n+\t}\n+}\ndiff --git a/reftable/reftable-error.h b/reftable/reftable-error.h\nnew file mode 100644\nindex 000000000000..6f89bedf1a58\n--- /dev/null\n+++ b/reftable/reftable-error.h\n@@ -0,0 +1,62 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_ERROR_H\n+#define REFTABLE_ERROR_H\n+\n+/*\n+ * Errors in reftable calls are signaled with negative integer return values. 0\n+ * means success.\n+ */\n+enum reftable_error {\n+\t/* Unexpected file system behavior */\n+\tREFTABLE_IO_ERROR = -2,\n+\n+\t/* Format inconsistency on reading data */\n+\tREFTABLE_FORMAT_ERROR = -3,\n+\n+\t/* File does not exist. Returned from block_source_from_file(), because\n+\t * it needs special handling in stack.\n+\t */\n+\tREFTABLE_NOT_EXIST_ERROR = -4,\n+\n+\t/* Trying to write out-of-date data. */\n+\tREFTABLE_LOCK_ERROR = -5,\n+\n+\t/* Misuse of the API:\n+\t * - on writing a record with NULL refname.\n+\t * - on writing a reftable_ref_record outside the table limits\n+\t * - on writing a ref or log record before the stack's\n+\t * next_update_inde*x\n+\t * - on writing a log record with multiline message with\n+\t * exact_log_message unset\n+\t * - on reading a reftable_ref_record from log iterator, or vice versa.\n+\t *\n+\t * When a call misuses the API, the internal state of the library is\n+\t * kept unchanged.\n+\t */\n+\tREFTABLE_API_ERROR = -6,\n+\n+\t/* Decompression error */\n+\tREFTABLE_ZLIB_ERROR = -7,\n+\n+\t/* Wrote a table without blocks. */\n+\tREFTABLE_EMPTY_TABLE_ERROR = -8,\n+\n+\t/* Dir/file conflict. */\n+\tREFTABLE_NAME_CONFLICT = -9,\n+\n+\t/* Invalid ref name. */\n+\tREFTABLE_REFNAME_ERROR = -10,\n+};\n+\n+/* convert the numeric error code to a string. The string should not be\n+ * deallocated. */\n+const char *reftable_error_str(int err);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"421757","messageId":"df8003cb9a7d9e017d358251a2d22c0e72454e03.1618255552.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 04/20] reftable: utility functions","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:36Z","receivedAt":"2021-04-12T19:26:08Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis commit provides basic utility classes for the reftable library.\n\nSigned-off-by: Han-Wen Nienhuys \nHelped-by: Johannes Schindelin \n---\n Makefile | 25 +++++-\n contrib/buildsystems/CMakeLists.txt | 14 ++-\n reftable/basics.c | 128 ++++++++++++++++++++++++++++\n reftable/basics.h | 60 +++++++++++++\n reftable/basics_test.c | 98 +++++++++++++++++++++\n reftable/publicbasics.c | 58 +++++++++++++\n reftable/reftable-malloc.h | 18 ++++\n reftable/reftable-tests.h | 22 +++++\n reftable/system.h | 32 +++++++\n reftable/test_framework.c | 23 +++++\n reftable/test_framework.h | 53 ++++++++++++\n t/helper/test-reftable.c | 9 ++\n t/helper/test-tool.c | 3 +-\n t/helper/test-tool.h | 1 +\n t/t0032-reftable-unittest.sh | 15 ++++\n 15 files changed, 553 insertions(+), 6 deletions(-)\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/basics_test.c\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-malloc.h\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 t/helper/test-reftable.c\n create mode 100755 t/t0032-reftable-unittest.sh\n\ndiff --git a/Makefile b/Makefile\nindex ffc2ddfd930d..1a6528a40b51 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -735,6 +735,7 @@ TEST_BUILTINS_OBJS += test-read-cache.o\n TEST_BUILTINS_OBJS += test-read-graph.o\n TEST_BUILTINS_OBJS += test-read-midx.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n+TEST_BUILTINS_OBJS += test-reftable.o\n TEST_BUILTINS_OBJS += test-regex.o\n TEST_BUILTINS_OBJS += test-repository.o\n TEST_BUILTINS_OBJS += test-revision-walking.o\n@@ -812,6 +813,8 @@ TEST_SHELL_PATH = $(SHELL_PATH)\n \n LIB_FILE = libgit.a\n XDIFF_LIB = xdiff/lib.a\n+REFTABLE_LIB = reftable/libreftable.a\n+REFTABLE_TEST_LIB = reftable/libreftable_test.a\n \n GENERATED_H += command-list.h\n GENERATED_H += config-list.h\n@@ -1180,7 +1183,7 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n-GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB)\n+GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB)\n EXTLIBS =\n \n GIT_USER_AGENT = git/$(GIT_VERSION)\n@@ -2397,7 +2400,15 @@ XDIFF_OBJS += xdiff/xutils.o\n .PHONY: xdiff-objs\n xdiff-objs: $(XDIFF_OBJS)\n \n+REFTABLE_OBJS += reftable/basics.o\n+REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/publicbasics.o\n+\n+REFTABLE_TEST_OBJS += reftable/test_framework.o\n+REFTABLE_TEST_OBJS += reftable/basics_test.o\n+\n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n+\n .PHONY: test-objs\n test-objs: $(TEST_OBJS)\n \n@@ -2413,6 +2424,8 @@ OBJECTS += $(PROGRAM_OBJS)\n OBJECTS += $(TEST_OBJS)\n OBJECTS += $(XDIFF_OBJS)\n OBJECTS += $(FUZZ_OBJS)\n+OBJECTS += $(REFTABLE_OBJS) $(REFTABLE_TEST_OBJS)\n+\n ifndef NO_CURL\n \tOBJECTS += http.o http-walker.o remote-curl.o\n endif\n@@ -2564,6 +2577,12 @@ $(LIB_FILE): $(LIB_OBJS)\n $(XDIFF_LIB): $(XDIFF_OBJS)\n \t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n \n+$(REFTABLE_LIB): $(REFTABLE_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n+$(REFTABLE_TEST_LIB): $(REFTABLE_TEST_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n export DEFAULT_EDITOR DEFAULT_PAGER\n \n Documentation/GIT-EXCLUDED-PROGRAMS: FORCE\n@@ -2844,7 +2863,7 @@ perf: all\n \n t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n \n-t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS)\n+t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) $(REFTABLE_TEST_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n \n check-sha1:: t/helper/test-tool$X\n@@ -3174,7 +3193,7 @@ cocciclean:\n clean: profile-clean coverage-clean cocciclean\n \t$(RM) *.res\n \t$(RM) $(OBJECTS)\n-\t$(RM) $(LIB_FILE) $(XDIFF_LIB)\n+\t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) git$X\n \t$(RM) $(TEST_PROGRAMS)\n \t$(RM) $(FUZZ_PROGRAMS)\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 75ed198a6a36..f41d3dab9015 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -611,6 +611,12 @@ parse_makefile_for_sources(libxdiff_SOURCES \"XDIFF_OBJS\")\n list(TRANSFORM libxdiff_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n add_library(xdiff STATIC ${libxdiff_SOURCES})\n \n+#reftable\n+parse_makefile_for_sources(reftable_SOURCES \"REFTABLE_OBJS\")\n+\n+list(TRANSFORM reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n+add_library(reftable STATIC ${reftable_SOURCES})\n+\n if(WIN32)\n \tif(NOT MSVC)#use windres when compiling with gcc and clang\n \t\tadd_custom_command(OUTPUT ${CMAKE_BINARY_DIR}/git.res\n@@ -633,7 +639,7 @@ endif()\n #link all required libraries to common-main\n add_library(common-main OBJECT ${CMAKE_SOURCE_DIR}/common-main.c)\n \n-target_link_libraries(common-main libgit xdiff ${ZLIB_LIBRARIES})\n+target_link_libraries(common-main libgit xdiff reftable ${ZLIB_LIBRARIES})\n if(Intl_FOUND)\n \ttarget_link_libraries(common-main ${Intl_LIBRARIES})\n endif()\n@@ -873,11 +879,15 @@ if(BUILD_TESTING)\n add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)\n target_link_libraries(test-fake-ssh common-main)\n \n+#reftable-tests\n+parse_makefile_for_sources(test-reftable_SOURCES \"REFTABLE_TEST_OBJS\")\n+list(TRANSFORM test-reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n+\n #test-tool\n parse_makefile_for_sources(test-tool_SOURCES \"TEST_BUILTINS_OBJS\")\n \n list(TRANSFORM test-tool_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/t/helper/\")\n-add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES})\n+add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES} ${test-reftable_SOURCES})\n target_link_libraries(test-tool common-main)\n \n set_target_properties(test-fake-ssh test-tool\ndiff --git a/reftable/basics.c b/reftable/basics.c\nnew file mode 100644\nindex 000000000000..abd027b98882\n--- /dev/null\n+++ b/reftable/basics.c\n@@ -0,0 +1,128 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+\n+void put_be24(uint8_t *out, uint32_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 16) & 0xff);\n+\tout[1] = (uint8_t)((i >> 8) & 0xff);\n+\tout[2] = (uint8_t)(i & 0xff);\n+}\n+\n+uint32_t get_be24(uint8_t *in)\n+{\n+\treturn (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 |\n+\t (uint32_t)(in[2]);\n+}\n+\n+void put_be16(uint8_t *out, uint16_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 8) & 0xff);\n+\tout[1] = (uint8_t)(i & 0xff);\n+}\n+\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n+{\n+\tsize_t lo = 0;\n+\tsize_t hi = sz;\n+\n+\t/* Invariants:\n+\t *\n+\t * (hi == sz) || f(hi) == true\n+\t * (lo == 0 && f(0) == true) || fi(lo) == false\n+\t */\n+\twhile (hi - lo > 1) {\n+\t\tsize_t mid = lo + (hi - lo) / 2;\n+\n+\t\tif (f(mid, args))\n+\t\t\thi = mid;\n+\t\telse\n+\t\t\tlo = mid;\n+\t}\n+\n+\tif (lo)\n+\t\treturn hi;\n+\n+\treturn f(0, args) ? 0 : 1;\n+}\n+\n+void free_names(char **a)\n+{\n+\tchar **p;\n+\tif (a == NULL) {\n+\t\treturn;\n+\t}\n+\tfor (p = a; *p; p++) {\n+\t\treftable_free(*p);\n+\t}\n+\treftable_free(a);\n+}\n+\n+int names_length(char **names)\n+{\n+\tchar **p = names;\n+\tfor (; *p; p++) {\n+\t\t/* empty */\n+\t}\n+\treturn p - names;\n+}\n+\n+void parse_names(char *buf, int size, char ***namesp)\n+{\n+\tchar **names = NULL;\n+\tsize_t names_cap = 0;\n+\tsize_t names_len = 0;\n+\n+\tchar *p = buf;\n+\tchar *end = buf + size;\n+\twhile (p < end) {\n+\t\tchar *next = strchr(p, '\\n');\n+\t\tif (next && next < end) {\n+\t\t\t*next = 0;\n+\t\t} else {\n+\t\t\tnext = end;\n+\t\t}\n+\t\tif (p < next) {\n+\t\t\tif (names_len == names_cap) {\n+\t\t\t\tnames_cap = 2 * names_cap + 1;\n+\t\t\t\tnames = reftable_realloc(\n+\t\t\t\t\tnames, names_cap * sizeof(*names));\n+\t\t\t}\n+\t\t\tnames[names_len++] = xstrdup(p);\n+\t\t}\n+\t\tp = next + 1;\n+\t}\n+\n+\tnames = reftable_realloc(names, (names_len + 1) * sizeof(*names));\n+\tnames[names_len] = NULL;\n+\t*namesp = names;\n+}\n+\n+int names_equal(char **a, char **b)\n+{\n+\tint i = 0;\n+\tfor (; a[i] && b[i]; i++) {\n+\t\tif (strcmp(a[i], b[i])) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treturn a[i] == b[i];\n+}\n+\n+int common_prefix_size(struct strbuf *a, struct strbuf *b)\n+{\n+\tint p = 0;\n+\tfor (; p < a->len && p < b->len; p++) {\n+\t\tif (a->buf[p] != b->buf[p])\n+\t\t\tbreak;\n+\t}\n+\n+\treturn p;\n+}\ndiff --git a/reftable/basics.h b/reftable/basics.h\nnew file mode 100644\nindex 000000000000..096b36862b9f\n--- /dev/null\n+++ b/reftable/basics.h\n@@ -0,0 +1,60 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BASICS_H\n+#define BASICS_H\n+\n+/*\n+ * miscellaneous utilities that are not provided by Git.\n+ */\n+\n+#include \"system.h\"\n+\n+/* Bigendian en/decoding of integers */\n+\n+void put_be24(uint8_t *out, uint32_t i);\n+uint32_t get_be24(uint8_t *in);\n+void put_be16(uint8_t *out, uint16_t i);\n+\n+/*\n+ * find smallest index i in [0, sz) at which f(i) is true, assuming\n+ * that f is ascending. Return sz if f(i) is false for all indices.\n+ *\n+ * Contrary to bsearch(3), this returns something useful if the argument is not\n+ * found.\n+ */\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args);\n+\n+/*\n+ * Frees a NULL terminated array of malloced strings. The array itself is also\n+ * freed.\n+ */\n+void free_names(char **a);\n+\n+/* parse a newline separated list of names. `size` is the length of the buffer,\n+ * without terminating '\\0'. Empty names are discarded. */\n+void parse_names(char *buf, int size, char ***namesp);\n+\n+/* compares two NULL-terminated arrays of strings. */\n+int names_equal(char **a, char **b);\n+\n+/* returns the array size of a NULL-terminated array of strings. */\n+int names_length(char **names);\n+\n+/* Allocation routines; they invoke the functions set through\n+ * reftable_set_alloc() */\n+void *reftable_malloc(size_t sz);\n+void *reftable_realloc(void *p, size_t sz);\n+void reftable_free(void *p);\n+void *reftable_calloc(size_t sz);\n+\n+/* Find the longest shared prefix size of `a` and `b` */\n+struct strbuf;\n+int common_prefix_size(struct strbuf *a, struct strbuf *b);\n+\n+#endif\ndiff --git a/reftable/basics_test.c b/reftable/basics_test.c\nnew file mode 100644\nindex 000000000000..6d52f0f9d5aa\n--- /dev/null\n+++ b/reftable/basics_test.c\n@@ -0,0 +1,98 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct binsearch_args {\n+\tint key;\n+\tint *arr;\n+};\n+\n+static int binsearch_func(size_t i, void *void_args)\n+{\n+\tstruct binsearch_args *args = (struct binsearch_args *)void_args;\n+\n+\treturn args->key < args->arr[i];\n+}\n+\n+static void test_binsearch(void)\n+{\n+\tint arr[] = { 2, 4, 6, 8, 10 };\n+\tsize_t sz = ARRAY_SIZE(arr);\n+\tstruct binsearch_args args = {\n+\t\t.arr = arr,\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 1; i < 11; i++) {\n+\t\tint res;\n+\t\targs.key = i;\n+\t\tres = binsearch(sz, &binsearch_func, &args);\n+\n+\t\tif (res < sz) {\n+\t\t\tEXPECT(args.key < arr[res]);\n+\t\t\tif (res > 0) {\n+\t\t\t\tEXPECT(args.key >= arr[res - 1]);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tEXPECT(args.key == 10 || args.key == 11);\n+\t\t}\n+\t}\n+}\n+\n+static void test_names_length(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", NULL };\n+\tEXPECT(names_length(a) == 2);\n+}\n+\n+static void test_parse_names_normal(void)\n+{\n+\tchar in[] = \"a\\nb\\n\";\n+\tchar **out = NULL;\n+\tparse_names(in, strlen(in), &out);\n+\tEXPECT(!strcmp(out[0], \"a\"));\n+\tEXPECT(!strcmp(out[1], \"b\"));\n+\tEXPECT(out[2] == NULL);\n+\tfree_names(out);\n+}\n+\n+static void test_parse_names_drop_empty(void)\n+{\n+\tchar in[] = \"a\\n\\n\";\n+\tchar **out = NULL;\n+\tparse_names(in, strlen(in), &out);\n+\tEXPECT(!strcmp(out[0], \"a\"));\n+\tEXPECT(out[1] == NULL);\n+\tfree_names(out);\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct strbuf s1 = STRBUF_INIT;\n+\tstruct strbuf s2 = STRBUF_INIT;\n+\tstrbuf_addstr(&s1, \"abcdef\");\n+\tstrbuf_addstr(&s2, \"abc\");\n+\tEXPECT(common_prefix_size(&s1, &s2) == 3);\n+\tstrbuf_release(&s1);\n+\tstrbuf_release(&s2);\n+}\n+\n+int basics_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_common_prefix);\n+\tRUN_TEST(test_parse_names_normal);\n+\tRUN_TEST(test_parse_names_drop_empty);\n+\tRUN_TEST(test_binsearch);\n+\tRUN_TEST(test_names_length);\n+\treturn 0;\n+}\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\nnew file mode 100644\nindex 000000000000..25639f61af61\n--- /dev/null\n+++ b/reftable/publicbasics.c\n@@ -0,0 +1,58 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-malloc.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+static void *(*reftable_malloc_ptr)(size_t sz) = &malloc;\n+static void *(*reftable_realloc_ptr)(void *, size_t) = &realloc;\n+static void (*reftable_free_ptr)(void *) = &free;\n+\n+void *reftable_malloc(size_t sz)\n+{\n+\treturn (*reftable_malloc_ptr)(sz);\n+}\n+\n+void *reftable_realloc(void *p, size_t sz)\n+{\n+\treturn (*reftable_realloc_ptr)(p, sz);\n+}\n+\n+void reftable_free(void *p)\n+{\n+\treftable_free_ptr(p);\n+}\n+\n+void *reftable_calloc(size_t sz)\n+{\n+\tvoid *p = reftable_malloc(sz);\n+\tmemset(p, 0, sz);\n+\treturn p;\n+}\n+\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n+{\n+\treftable_malloc_ptr = malloc;\n+\treftable_realloc_ptr = realloc;\n+\treftable_free_ptr = free;\n+}\n+\n+int hash_size(uint32_t id)\n+{\n+\tswitch (id) {\n+\tcase 0:\n+\tcase SHA1_ID:\n+\t\treturn SHA1_SIZE;\n+\tcase SHA256_ID:\n+\t\treturn SHA256_SIZE;\n+\t}\n+\tabort();\n+}\ndiff --git a/reftable/reftable-malloc.h b/reftable/reftable-malloc.h\nnew file mode 100644\nindex 000000000000..5f2185f1f343\n--- /dev/null\n+++ b/reftable/reftable-malloc.h\n@@ -0,0 +1,18 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_H\n+#define REFTABLE_H\n+\n+#include \n+\n+/* Overrides the functions to use for memory management. */\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n+\n+#endif\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nnew file mode 100644\nindex 000000000000..5e7698ae654e\n--- /dev/null\n+++ b/reftable/reftable-tests.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_TESTS_H\n+#define REFTABLE_TESTS_H\n+\n+int basics_test_main(int argc, const char **argv);\n+int block_test_main(int argc, const char **argv);\n+int merged_test_main(int argc, const char **argv);\n+int record_test_main(int argc, const char **argv);\n+int refname_test_main(int argc, const char **argv);\n+int reftable_test_main(int argc, const char **argv);\n+int stack_test_main(int argc, const char **argv);\n+int tree_test_main(int argc, const char **argv);\n+int reftable_dump_main(int argc, char *const *argv);\n+\n+#endif\ndiff --git a/reftable/system.h b/reftable/system.h\nnew file mode 100644\nindex 000000000000..07277ca06273\n--- /dev/null\n+++ b/reftable/system.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef SYSTEM_H\n+#define SYSTEM_H\n+\n+#include \"git-compat-util.h\"\n+#include \"strbuf.h\"\n+\n+#include \n+\n+struct strbuf;\n+/* In git, this is declared in dir.h */\n+int remove_dir_recursively(struct strbuf *path, int flags);\n+\n+#define SHA1_ID 0x73686131\n+#define SHA256_ID 0x73323536\n+#define SHA1_SIZE 20\n+#define SHA256_SIZE 32\n+\n+/* This is uncompress2, which is only available in zlib as of 2017.\n+ */\n+int uncompress_return_consumed(Bytef *dest, uLongf *destLen,\n+\t\t\t const Bytef *source, uLong *sourceLen);\n+int hash_size(uint32_t id);\n+\n+#endif\ndiff --git a/reftable/test_framework.c b/reftable/test_framework.c\nnew file mode 100644\nindex 000000000000..a5ff4e2a2d2f\n--- /dev/null\n+++ b/reftable/test_framework.c\n@@ -0,0 +1,23 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"test_framework.h\"\n+\n+#include \"basics.h\"\n+\n+void set_test_hash(uint8_t *p, int i)\n+{\n+\tmemset(p, (uint8_t)i, hash_size(SHA1_ID));\n+}\n+\n+int strbuf_add_void(void *b, const void *data, size_t sz)\n+{\n+\tstrbuf_add((struct strbuf *)b, data, sz);\n+\treturn sz;\n+}\ndiff --git a/reftable/test_framework.h b/reftable/test_framework.h\nnew file mode 100644\nindex 000000000000..5fdc9519a5a5\n--- /dev/null\n+++ b/reftable/test_framework.h\n@@ -0,0 +1,53 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TEST_FRAMEWORK_H\n+#define TEST_FRAMEWORK_H\n+\n+#include \"system.h\"\n+#include \"reftable-error.h\"\n+\n+#define EXPECT_ERR(c) \\\n+\tif (c != 0) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: error == %d (%s), want 0\\n\", \\\n+\t\t\t__FILE__, __LINE__, c, reftable_error_str(c)); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define EXPECT_STREQ(a, b) \\\n+\tif (strcmp(a, b)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s:%d: %s (%s) != %s (%s)\\n\", __FILE__, \\\n+\t\t\t__LINE__, #a, a, #b, b); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define EXPECT(c) \\\n+\tif (!(c)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: failed assertion %s\\n\", __FILE__, \\\n+\t\t\t__LINE__, #c); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define RUN_TEST(f) \\\n+\tfprintf(stderr, \"running %s\\n\", #f); \\\n+\tfflush(stderr); \\\n+\tf();\n+\n+void set_test_hash(uint8_t *p, int i);\n+\n+/* Like strbuf_add, but suitable for passing to reftable_new_writer\n+ */\n+int strbuf_add_void(void *b, const void *data, size_t sz);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nnew file mode 100644\nindex 000000000000..3b58e423e7b1\n--- /dev/null\n+++ b/t/helper/test-reftable.c\n@@ -0,0 +1,9 @@\n+#include \"reftable/reftable-tests.h\"\n+#include \"test-tool.h\"\n+\n+int cmd__reftable(int argc, const char **argv)\n+{\n+\tbasics_test_main(argc, argv);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 287aa6002307..814d2606f8ca 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -49,13 +49,14 @@ static struct test_cmd cmds[] = {\n \t{ \"pcre2-config\", cmd__pcre2_config },\n \t{ \"pkt-line\", cmd__pkt_line },\n \t{ \"prio-queue\", cmd__prio_queue },\n-\t{ \"proc-receive\", cmd__proc_receive},\n+\t{ \"proc-receive\", cmd__proc_receive },\n \t{ \"progress\", cmd__progress },\n \t{ \"reach\", cmd__reach },\n \t{ \"read-cache\", cmd__read_cache },\n \t{ \"read-graph\", cmd__read_graph },\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n+\t{ \"reftable\", cmd__reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 9ea4b31011dd..789c26f302dd 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -45,6 +45,7 @@ int cmd__read_cache(int argc, const char **argv);\n int cmd__read_graph(int argc, const char **argv);\n int cmd__read_midx(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n+int cmd__reftable(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__repository(int argc, const char **argv);\n int cmd__revision_walking(int argc, const char **argv);\ndiff --git a/t/t0032-reftable-unittest.sh b/t/t0032-reftable-unittest.sh\nnew file mode 100755\nindex 000000000000..0ed14971a580\n--- /dev/null\n+++ b/t/t0032-reftable-unittest.sh\n@@ -0,0 +1,15 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable unittests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'unittests' '\n+\tTMPDIR=$(pwd) && export TMPDIR &&\n+\ttest-tool reftable\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"421758","messageId":"6807af53e9a039850364bc7f6b8579322b9a8884.1618255552.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 05/20] reftable: add blocksource, an abstraction for random access reads","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:37Z","receivedAt":"2021-04-12T19:26:15Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is usually used with files for storage. However, we abstract\naway this using the blocksource data structure. This has two advantages:\n\n* log blocks are zlib compressed, and handling them is simplified if we can\n discard byte segments from within the block layer.\n\n* for unittests, it is useful to read and write in-memory. The blocksource\n allows us to abstract the data away from on-disk files.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/blocksource.c | 148 ++++++++++++++++++++++++++++++++\n reftable/blocksource.h | 22 +++++\n reftable/reftable-blocksource.h | 49 +++++++++++\n 4 files changed, 220 insertions(+)\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/reftable-blocksource.h\n\ndiff --git a/Makefile b/Makefile\nindex 1a6528a40b51..bb57e0318cdb 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2402,6 +2402,7 @@ xdiff-objs: $(XDIFF_OBJS)\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n \n REFTABLE_TEST_OBJS += reftable/test_framework.o\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nnew file mode 100644\nindex 000000000000..25d4d95b52b8\n--- /dev/null\n+++ b/reftable/blocksource.c\n@@ -0,0 +1,148 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"reftable-blocksource.h\"\n+#include \"reftable-error.h\"\n+\n+static void strbuf_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void strbuf_close(void *b)\n+{\n+}\n+\n+static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct strbuf *b = (struct strbuf *)v;\n+\tassert(off + size <= b->len);\n+\tdest->data = reftable_calloc(size);\n+\tmemcpy(dest->data, b->buf + off, size);\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static uint64_t strbuf_size(void *b)\n+{\n+\treturn ((struct strbuf *)b)->len;\n+}\n+\n+static struct reftable_block_source_vtable strbuf_vtable = {\n+\t.size = &strbuf_size,\n+\t.read_block = &strbuf_read_block,\n+\t.return_block = &strbuf_return_block,\n+\t.close = &strbuf_close,\n+};\n+\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf)\n+{\n+\tassert(bs->ops == NULL);\n+\tbs->ops = &strbuf_vtable;\n+\tbs->arg = buf;\n+}\n+\n+static void malloc_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static struct reftable_block_source_vtable malloc_vtable = {\n+\t.return_block = &malloc_return_block,\n+};\n+\n+static struct reftable_block_source malloc_block_source_instance = {\n+\t.ops = &malloc_vtable,\n+};\n+\n+struct reftable_block_source malloc_block_source(void)\n+{\n+\treturn malloc_block_source_instance;\n+}\n+\n+struct file_block_source {\n+\tint fd;\n+\tuint64_t size;\n+};\n+\n+static uint64_t file_size(void *b)\n+{\n+\treturn ((struct file_block_source *)b)->size;\n+}\n+\n+static void file_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void file_close(void *b)\n+{\n+\tint fd = ((struct file_block_source *)b)->fd;\n+\tif (fd > 0) {\n+\t\tclose(fd);\n+\t\t((struct file_block_source *)b)->fd = 0;\n+\t}\n+\n+\treftable_free(b);\n+}\n+\n+static int file_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct file_block_source *b = (struct file_block_source *)v;\n+\tassert(off + size <= b->size);\n+\tdest->data = reftable_malloc(size);\n+\tif (pread(b->fd, dest->data, size, off) != size)\n+\t\treturn -1;\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static struct reftable_block_source_vtable file_vtable = {\n+\t.size = &file_size,\n+\t.read_block = &file_read_block,\n+\t.return_block = &file_return_block,\n+\t.close = &file_close,\n+};\n+\n+int reftable_block_source_from_file(struct reftable_block_source *bs,\n+\t\t\t\t const char *name)\n+{\n+\tstruct stat st = { 0 };\n+\tint err = 0;\n+\tint fd = open(name, O_RDONLY);\n+\tstruct file_block_source *p = NULL;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\treturn REFTABLE_NOT_EXIST_ERROR;\n+\t\t}\n+\t\treturn -1;\n+\t}\n+\n+\terr = fstat(fd, &st);\n+\tif (err < 0)\n+\t\treturn -1;\n+\n+\tp = reftable_calloc(sizeof(struct file_block_source));\n+\tp->size = st.st_size;\n+\tp->fd = fd;\n+\n+\tassert(bs->ops == NULL);\n+\tbs->ops = &file_vtable;\n+\tbs->arg = p;\n+\treturn 0;\n+}\ndiff --git a/reftable/blocksource.h b/reftable/blocksource.h\nnew file mode 100644\nindex 000000000000..072e2727ad20\n--- /dev/null\n+++ b/reftable/blocksource.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCKSOURCE_H\n+#define BLOCKSOURCE_H\n+\n+#include \"system.h\"\n+\n+struct reftable_block_source;\n+\n+/* Create an in-memory block source for reading reftables */\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf);\n+\n+struct reftable_block_source malloc_block_source(void);\n+\n+#endif\ndiff --git a/reftable/reftable-blocksource.h b/reftable/reftable-blocksource.h\nnew file mode 100644\nindex 000000000000..5aa3990a5732\n--- /dev/null\n+++ b/reftable/reftable-blocksource.h\n@@ -0,0 +1,49 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_BLOCKSOURCE_H\n+#define REFTABLE_BLOCKSOURCE_H\n+\n+#include \n+\n+/* block_source is a generic wrapper for a seekable readable file.\n+ */\n+struct reftable_block_source {\n+\tstruct reftable_block_source_vtable *ops;\n+\tvoid *arg;\n+};\n+\n+/* a contiguous segment of bytes. It keeps track of its generating block_source\n+ * so it can return itself into the pool. */\n+struct reftable_block {\n+\tuint8_t *data;\n+\tint len;\n+\tstruct reftable_block_source source;\n+};\n+\n+/* block_source_vtable are the operations that make up block_source */\n+struct reftable_block_source_vtable {\n+\t/* returns the size of a block source */\n+\tuint64_t (*size)(void *source);\n+\n+\t/* reads a segment from the block source. It is an error to read\n+\t beyond the end of the block */\n+\tint (*read_block)(void *source, struct reftable_block *dest,\n+\t\t\t uint64_t off, uint32_t size);\n+\t/* mark the block as read; may return the data back to malloc */\n+\tvoid (*return_block)(void *source, struct reftable_block *blockp);\n+\n+\t/* release all resources associated with the block source */\n+\tvoid (*close)(void *source);\n+};\n+\n+/* opens a file on the file system as a block_source */\n+int reftable_block_source_from_file(struct reftable_block_source *block_src,\n+\t\t\t\t const char *name);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"421759","messageId":"78a352c96f82d441d1c82ade032558283e867cbf.1618255553.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 06/20] reftable: (de)serialization for the polymorphic record type.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:38Z","receivedAt":"2021-04-12T19:26:16Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is structured as a sequence of blocks, and each block\ncontains a sequence of prefix-compressed key-value records. There are 4 types of\nrecords, and they have similarities in how they must be handled. This is\nachieved by introducing a polymorphic 'record' type that encapsulates ref, log,\nindex and object records.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/constants.h | 21 +\n reftable/record.c | 1203 ++++++++++++++++++++++++++++++++++++\n reftable/record.h | 139 +++++\n reftable/record_test.c | 405 ++++++++++++\n reftable/reftable-record.h | 114 ++++\n t/helper/test-reftable.c | 2 +-\n 7 files changed, 1885 insertions(+), 1 deletion(-)\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/reftable-record.h\n\ndiff --git a/Makefile b/Makefile\nindex bb57e0318cdb..313421d16012 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2404,7 +2404,9 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/record.o\n \n+REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n \ndiff --git a/reftable/constants.h b/reftable/constants.h\nnew file mode 100644\nindex 000000000000..5eee72c4c113\n--- /dev/null\n+++ b/reftable/constants.h\n@@ -0,0 +1,21 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef CONSTANTS_H\n+#define CONSTANTS_H\n+\n+#define BLOCK_TYPE_LOG 'g'\n+#define BLOCK_TYPE_INDEX 'i'\n+#define BLOCK_TYPE_REF 'r'\n+#define BLOCK_TYPE_OBJ 'o'\n+#define BLOCK_TYPE_ANY 0\n+\n+#define MAX_RESTARTS ((1 << 16) - 1)\n+#define DEFAULT_BLOCK_SIZE 4096\n+\n+#endif\ndiff --git a/reftable/record.c b/reftable/record.c\nnew file mode 100644\nindex 000000000000..d9e4c7bde411\n--- /dev/null\n+++ b/reftable/record.c\n@@ -0,0 +1,1203 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+/* record.c - methods for different types of records. */\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"constants.h\"\n+#include \"reftable-error.h\"\n+#include \"basics.h\"\n+\n+int get_var_int(uint64_t *dest, struct string_view *in)\n+{\n+\tint ptr = 0;\n+\tuint64_t val;\n+\n+\tif (in->len == 0)\n+\t\treturn -1;\n+\tval = in->buf[ptr] & 0x7f;\n+\n+\twhile (in->buf[ptr] & 0x80) {\n+\t\tptr++;\n+\t\tif (ptr > in->len) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tval = (val + 1) << 7 | (uint64_t)(in->buf[ptr] & 0x7f);\n+\t}\n+\n+\t*dest = val;\n+\treturn ptr + 1;\n+}\n+\n+int put_var_int(struct string_view *dest, uint64_t val)\n+{\n+\tuint8_t buf[10] = { 0 };\n+\tint i = 9;\n+\tint n = 0;\n+\tbuf[i] = (uint8_t)(val & 0x7f);\n+\ti--;\n+\twhile (1) {\n+\t\tval >>= 7;\n+\t\tif (!val) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tval--;\n+\t\tbuf[i] = 0x80 | (uint8_t)(val & 0x7f);\n+\t\ti--;\n+\t}\n+\n+\tn = sizeof(buf) - i - 1;\n+\tif (dest->len < n)\n+\t\treturn -1;\n+\tmemcpy(dest->buf, &buf[i + 1], n);\n+\treturn n;\n+}\n+\n+int reftable_is_block_type(uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\tcase BLOCK_TYPE_LOG:\n+\tcase BLOCK_TYPE_OBJ:\n+\tcase BLOCK_TYPE_INDEX:\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec)\n+{\n+\tswitch (rec->value_type) {\n+\tcase REFTABLE_REF_VAL1:\n+\t\treturn rec->value.val1;\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn rec->value.val2.value;\n+\tdefault:\n+\t\treturn NULL;\n+\t}\n+}\n+\n+uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec)\n+{\n+\tswitch (rec->value_type) {\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn rec->value.val2.target_value;\n+\tdefault:\n+\t\treturn NULL;\n+\t}\n+}\n+\n+static int decode_string(struct strbuf *dest, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t tsize = 0;\n+\tint n = get_var_int(&tsize, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\tif (in.len < tsize)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, in.buf, tsize);\n+\tstring_view_consume(&in, tsize);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static int encode_string(char *str, struct string_view s)\n+{\n+\tstruct string_view start = s;\n+\tint l = strlen(str);\n+\tint n = put_var_int(&s, l);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\tif (s.len < l)\n+\t\treturn -1;\n+\tmemcpy(s.buf, str, l);\n+\tstring_view_consume(&s, l);\n+\n+\treturn start.len - s.len;\n+}\n+\n+int reftable_encode_key(int *restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra)\n+{\n+\tstruct string_view start = dest;\n+\tint prefix_len = common_prefix_size(&prev_key, &key);\n+\tuint64_t suffix_len = key.len - prefix_len;\n+\tint n = put_var_int(&dest, (uint64_t)prefix_len);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\t*restart = (prefix_len == 0);\n+\n+\tn = put_var_int(&dest, suffix_len << 3 | (uint64_t)extra);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\tif (dest.len < suffix_len)\n+\t\treturn -1;\n+\tmemcpy(dest.buf, key.buf + prefix_len, suffix_len);\n+\tstring_view_consume(&dest, suffix_len);\n+\n+\treturn start.len - dest.len;\n+}\n+\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t prefix_len = 0;\n+\tuint64_t suffix_len = 0;\n+\tint n = get_var_int(&prefix_len, &in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tif (prefix_len > last_key.len)\n+\t\treturn -1;\n+\n+\tn = get_var_int(&suffix_len, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\t*extra = (uint8_t)(suffix_len & 0x7);\n+\tsuffix_len >>= 3;\n+\n+\tif (in.len < suffix_len)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(key);\n+\tstrbuf_add(key, last_key.buf, prefix_len);\n+\tstrbuf_add(key, in.buf, suffix_len);\n+\tstring_view_consume(&in, suffix_len);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static void reftable_ref_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_ref_record *rec =\n+\t\t(const struct reftable_ref_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, rec->refname);\n+}\n+\n+static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *ref = (struct reftable_ref_record *)rec;\n+\tstruct reftable_ref_record *src = (struct reftable_ref_record *)src_rec;\n+\tassert(hash_size > 0);\n+\n+\t/* This is simple and correct, but we could probably reuse the hash\n+\t * fields. */\n+\treftable_ref_record_release(ref);\n+\tif (src->refname != NULL) {\n+\t\tref->refname = xstrdup(src->refname);\n+\t}\n+\tref->update_index = src->update_index;\n+\tref->value_type = src->value_type;\n+\tswitch (src->value_type) {\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\tref->value.val1 = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val1, src->value.val1, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\tref->value.val2.value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val2.value, src->value.val2.value, hash_size);\n+\t\tref->value.val2.target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val2.target_value,\n+\t\t src->value.val2.target_value, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tref->value.symref = xstrdup(src->value.symref);\n+\t\tbreak;\n+\t}\n+}\n+\n+static char hexdigit(int c)\n+{\n+\tif (c <= 9)\n+\t\treturn '0' + c;\n+\treturn 'a' + (c - 10);\n+}\n+\n+static void hex_format(char *dest, uint8_t *src, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (src != NULL) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < hash_size; i++) {\n+\t\t\tdest[2 * i] = hexdigit(src[i] >> 4);\n+\t\t\tdest[2 * i + 1] = hexdigit(src[i] & 0xf);\n+\t\t}\n+\t\tdest[2 * hash_size] = 0;\n+\t}\n+}\n+\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[2 * SHA256_SIZE + 1] = { 0 }; /* BUG */\n+\tprintf(\"ref{%s(%\" PRIu64 \") \", ref->refname, ref->update_index);\n+\tswitch (ref->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tprintf(\"=> %s\", ref->value.symref);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\thex_format(hex, ref->value.val2.value, hash_size(hash_id));\n+\t\tprintf(\"val 2 %s\", hex);\n+\t\thex_format(hex, ref->value.val2.target_value,\n+\t\t\t hash_size(hash_id));\n+\t\tprintf(\"(T %s)\", hex);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\thex_format(hex, ref->value.val1, hash_size(hash_id));\n+\t\tprintf(\"val 1 %s\", hex);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tprintf(\"delete\");\n+\t\tbreak;\n+\t}\n+\tprintf(\"}\\n\");\n+}\n+\n+static void reftable_ref_record_release_void(void *rec)\n+{\n+\treftable_ref_record_release((struct reftable_ref_record *)rec);\n+}\n+\n+void reftable_ref_record_release(struct reftable_ref_record *ref)\n+{\n+\tswitch (ref->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\treftable_free(ref->value.symref);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\treftable_free(ref->value.val2.target_value);\n+\t\treftable_free(ref->value.val2.value);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\treftable_free(ref->value.val1);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t}\n+\n+\treftable_free(ref->refname);\n+\tmemset(ref, 0, sizeof(struct reftable_ref_record));\n+}\n+\n+static uint8_t reftable_ref_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\treturn r->value_type;\n+}\n+\n+static int reftable_ref_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\tstruct string_view start = s;\n+\tint n = put_var_int(&s, r->update_index);\n+\tassert(hash_size > 0);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tswitch (r->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tn = encode_string(r->value.symref, s);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\tif (s.len < 2 * hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value.val2.value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tmemcpy(s.buf, r->value.val2.target_value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\tif (s.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value.val1, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t}\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_ref_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *r = (struct reftable_ref_record *)rec;\n+\tstruct string_view start = in;\n+\tuint64_t update_index = 0;\n+\tint n = get_var_int(&update_index, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\treftable_ref_record_release(r);\n+\n+\tassert(hash_size > 0);\n+\n+\tr->refname = reftable_realloc(r->refname, key.len + 1);\n+\tmemcpy(r->refname, key.buf, key.len);\n+\tr->update_index = update_index;\n+\tr->refname[key.len] = 0;\n+\tr->value_type = val_type;\n+\tswitch (val_type) {\n+\tcase REFTABLE_REF_VAL1:\n+\t\tif (in.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tr->value.val1 = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val1, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\n+\tcase REFTABLE_REF_VAL2:\n+\t\tif (in.len < 2 * hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tr->value.val2.value = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val2.value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\n+\t\tr->value.val2.target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val2.target_value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\n+\tcase REFTABLE_REF_SYMREF: {\n+\t\tstruct strbuf dest = STRBUF_INIT;\n+\t\tint n = decode_string(&dest, in);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\t\tr->value.symref = dest.buf;\n+\t} break;\n+\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t\tbreak;\n+\t}\n+\n+\treturn start.len - in.len;\n+}\n+\n+static int reftable_ref_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_ref_record_is_deletion(\n+\t\t(const struct reftable_ref_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_ref_record_vtable = {\n+\t.key = &reftable_ref_record_key,\n+\t.type = BLOCK_TYPE_REF,\n+\t.copy_from = &reftable_ref_record_copy_from,\n+\t.val_type = &reftable_ref_record_val_type,\n+\t.encode = &reftable_ref_record_encode,\n+\t.decode = &reftable_ref_record_decode,\n+\t.release = &reftable_ref_record_release_void,\n+\t.is_deletion = &reftable_ref_record_is_deletion_void,\n+};\n+\n+static void reftable_obj_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_obj_record *rec =\n+\t\t(const struct reftable_obj_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, rec->hash_prefix, rec->hash_prefix_len);\n+}\n+\n+static void reftable_obj_record_release(void *rec)\n+{\n+\tstruct reftable_obj_record *obj = (struct reftable_obj_record *)rec;\n+\tFREE_AND_NULL(obj->hash_prefix);\n+\tFREE_AND_NULL(obj->offsets);\n+\tmemset(obj, 0, sizeof(struct reftable_obj_record));\n+}\n+\n+static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_obj_record *obj = (struct reftable_obj_record *)rec;\n+\tconst struct reftable_obj_record *src =\n+\t\t(const struct reftable_obj_record *)src_rec;\n+\tint olen;\n+\n+\treftable_obj_record_release(obj);\n+\t*obj = *src;\n+\tobj->hash_prefix = reftable_malloc(obj->hash_prefix_len);\n+\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n+\n+\tolen = obj->offset_len * sizeof(uint64_t);\n+\tobj->offsets = reftable_malloc(olen);\n+\tmemcpy(obj->offsets, src->offsets, olen);\n+}\n+\n+static uint8_t reftable_obj_record_val_type(const void *rec)\n+{\n+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n+\tif (r->offset_len > 0 && r->offset_len < 8)\n+\t\treturn r->offset_len;\n+\treturn 0;\n+}\n+\n+static int reftable_obj_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n+\tstruct string_view start = s;\n+\tint i = 0;\n+\tint n = 0;\n+\tuint64_t last = 0;\n+\tif (r->offset_len == 0 || r->offset_len >= 8) {\n+\t\tn = put_var_int(&s, r->offset_len);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t}\n+\tif (r->offset_len == 0)\n+\t\treturn start.len - s.len;\n+\tn = put_var_int(&s, r->offsets[0]);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tlast = r->offsets[0];\n+\tfor (i = 1; i < r->offset_len; i++) {\n+\t\tint n = put_var_int(&s, r->offsets[i] - last);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tlast = r->offsets[i];\n+\t}\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_obj_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n+\tuint64_t count = val_type;\n+\tint n = 0;\n+\tuint64_t last;\n+\tint j;\n+\tr->hash_prefix = reftable_malloc(key.len);\n+\tmemcpy(r->hash_prefix, key.buf, key.len);\n+\tr->hash_prefix_len = key.len;\n+\n+\tif (val_type == 0) {\n+\t\tn = get_var_int(&count, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\n+\t\tstring_view_consume(&in, n);\n+\t}\n+\n+\tr->offsets = NULL;\n+\tr->offset_len = 0;\n+\tif (count == 0)\n+\t\treturn start.len - in.len;\n+\n+\tr->offsets = reftable_malloc(count * sizeof(uint64_t));\n+\tr->offset_len = count;\n+\n+\tn = get_var_int(&r->offsets[0], &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\tlast = r->offsets[0];\n+\tj = 1;\n+\twhile (j < count) {\n+\t\tuint64_t delta = 0;\n+\t\tint n = get_var_int(&delta, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\n+\t\tlast = r->offsets[j] = (delta + last);\n+\t\tj++;\n+\t}\n+\treturn start.len - in.len;\n+}\n+\n+static int not_a_deletion(const void *p)\n+{\n+\treturn 0;\n+}\n+\n+static struct reftable_record_vtable reftable_obj_record_vtable = {\n+\t.key = &reftable_obj_record_key,\n+\t.type = BLOCK_TYPE_OBJ,\n+\t.copy_from = &reftable_obj_record_copy_from,\n+\t.val_type = &reftable_obj_record_val_type,\n+\t.encode = &reftable_obj_record_encode,\n+\t.decode = &reftable_obj_record_decode,\n+\t.release = &reftable_obj_record_release,\n+\t.is_deletion = not_a_deletion,\n+};\n+\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[SHA256_SIZE + 1] = { 0 };\n+\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tprintf(\"log{%s(%\" PRIu64 \") delete\", log->refname,\n+\t\t log->update_index);\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tprintf(\"log{%s(%\" PRIu64 \") %s <%s> %\" PRIu64 \" %04d\\n\",\n+\t\t log->refname, log->update_index, log->update.name,\n+\t\t log->update.email, log->update.time,\n+\t\t log->update.tz_offset);\n+\t\thex_format(hex, log->update.old_hash, hash_size(hash_id));\n+\t\tprintf(\"%s => \", hex);\n+\t\thex_format(hex, log->update.new_hash, hash_size(hash_id));\n+\t\tprintf(\"%s\\n\\n%s\\n}\\n\", hex, log->update.message);\n+\t\tbreak;\n+\t}\n+}\n+\n+static void reftable_log_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_log_record *rec =\n+\t\t(const struct reftable_log_record *)r;\n+\tint len = strlen(rec->refname);\n+\tuint8_t i64[8];\n+\tuint64_t ts = 0;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, (uint8_t *)rec->refname, len + 1);\n+\n+\tts = (~ts) - rec->update_index;\n+\tput_be64(&i64[0], ts);\n+\tstrbuf_add(dest, i64, sizeof(i64));\n+}\n+\n+static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_log_record *dst = (struct reftable_log_record *)rec;\n+\tconst struct reftable_log_record *src =\n+\t\t(const struct reftable_log_record *)src_rec;\n+\n+\treftable_log_record_release(dst);\n+\t*dst = *src;\n+\tif (dst->refname != NULL) {\n+\t\tdst->refname = xstrdup(dst->refname);\n+\t}\n+\tswitch (dst->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tif (dst->update.email != NULL) {\n+\t\t\tdst->update.email = xstrdup(dst->update.email);\n+\t\t}\n+\t\tif (dst->update.name != NULL) {\n+\t\t\tdst->update.name = xstrdup(dst->update.name);\n+\t\t}\n+\t\tif (dst->update.message != NULL) {\n+\t\t\tdst->update.message = xstrdup(dst->update.message);\n+\t\t}\n+\n+\t\tif (dst->update.new_hash != NULL) {\n+\t\t\tdst->update.new_hash = reftable_malloc(hash_size);\n+\t\t\tmemcpy(dst->update.new_hash, src->update.new_hash,\n+\t\t\t hash_size);\n+\t\t}\n+\t\tif (dst->update.old_hash != NULL) {\n+\t\t\tdst->update.old_hash = reftable_malloc(hash_size);\n+\t\t\tmemcpy(dst->update.old_hash, src->update.old_hash,\n+\t\t\t hash_size);\n+\t\t}\n+\t\tbreak;\n+\t}\n+}\n+\n+static void reftable_log_record_release_void(void *rec)\n+{\n+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n+\treftable_log_record_release(r);\n+}\n+\n+void reftable_log_record_release(struct reftable_log_record *r)\n+{\n+\treftable_free(r->refname);\n+\tswitch (r->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\treftable_free(r->update.new_hash);\n+\t\treftable_free(r->update.old_hash);\n+\t\treftable_free(r->update.name);\n+\t\treftable_free(r->update.email);\n+\t\treftable_free(r->update.message);\n+\t\tbreak;\n+\t}\n+\tmemset(r, 0, sizeof(struct reftable_log_record));\n+}\n+\n+static uint8_t reftable_log_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_log_record *log =\n+\t\t(const struct reftable_log_record *)rec;\n+\n+\treturn reftable_log_record_is_deletion(log) ? 0 : 1;\n+}\n+\n+static uint8_t zero[SHA256_SIZE] = { 0 };\n+\n+static int reftable_log_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n+\tstruct string_view start = s;\n+\tint n = 0;\n+\tuint8_t *oldh = NULL;\n+\tuint8_t *newh = NULL;\n+\tif (reftable_log_record_is_deletion(r))\n+\t\treturn 0;\n+\n+\toldh = r->update.old_hash;\n+\tnewh = r->update.new_hash;\n+\tif (oldh == NULL) {\n+\t\toldh = zero;\n+\t}\n+\tif (newh == NULL) {\n+\t\tnewh = zero;\n+\t}\n+\n+\tif (s.len < 2 * hash_size)\n+\t\treturn -1;\n+\n+\tmemcpy(s.buf, oldh, hash_size);\n+\tmemcpy(s.buf + hash_size, newh, hash_size);\n+\tstring_view_consume(&s, 2 * hash_size);\n+\n+\tn = encode_string(r->update.name ? r->update.name : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = encode_string(r->update.email ? r->update.email : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = put_var_int(&s, r->update.time);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tif (s.len < 2)\n+\t\treturn -1;\n+\n+\tput_be16(s.buf, r->update.tz_offset);\n+\tstring_view_consume(&s, 2);\n+\n+\tn = encode_string(r->update.message ? r->update.message : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_log_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n+\tuint64_t max = 0;\n+\tuint64_t ts = 0;\n+\tstruct strbuf dest = STRBUF_INIT;\n+\tint n;\n+\n+\tif (key.len <= 9 || key.buf[key.len - 9] != 0)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->refname = reftable_realloc(r->refname, key.len - 8);\n+\tmemcpy(r->refname, key.buf, key.len - 8);\n+\tts = get_be64(key.buf + key.len - 8);\n+\n+\tr->update_index = (~max) - ts;\n+\n+\tif (val_type != r->value_type) {\n+\t\tswitch (r->value_type) {\n+\t\tcase REFTABLE_LOG_UPDATE:\n+\t\t\tFREE_AND_NULL(r->update.old_hash);\n+\t\t\tFREE_AND_NULL(r->update.new_hash);\n+\t\t\tFREE_AND_NULL(r->update.message);\n+\t\t\tFREE_AND_NULL(r->update.email);\n+\t\t\tFREE_AND_NULL(r->update.name);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_LOG_DELETION:\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tr->value_type = val_type;\n+\tif (val_type == REFTABLE_LOG_DELETION)\n+\t\treturn 0;\n+\n+\tif (in.len < 2 * hash_size)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->update.old_hash = reftable_realloc(r->update.old_hash, hash_size);\n+\tr->update.new_hash = reftable_realloc(r->update.new_hash, hash_size);\n+\n+\tmemcpy(r->update.old_hash, in.buf, hash_size);\n+\tmemcpy(r->update.new_hash, in.buf + hash_size, hash_size);\n+\n+\tstring_view_consume(&in, 2 * hash_size);\n+\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->update.name = reftable_realloc(r->update.name, dest.len + 1);\n+\tmemcpy(r->update.name, dest.buf, dest.len);\n+\tr->update.name[dest.len] = 0;\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->update.email = reftable_realloc(r->update.email, dest.len + 1);\n+\tmemcpy(r->update.email, dest.buf, dest.len);\n+\tr->update.email[dest.len] = 0;\n+\n+\tts = 0;\n+\tn = get_var_int(&ts, &in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\tr->update.time = ts;\n+\tif (in.len < 2)\n+\t\tgoto done;\n+\n+\tr->update.tz_offset = get_be16(in.buf);\n+\tstring_view_consume(&in, 2);\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->update.message = reftable_realloc(r->update.message, dest.len + 1);\n+\tmemcpy(r->update.message, dest.buf, dest.len);\n+\tr->update.message[dest.len] = 0;\n+\n+\tstrbuf_release(&dest);\n+\treturn start.len - in.len;\n+\n+done:\n+\tstrbuf_release(&dest);\n+\treturn REFTABLE_FORMAT_ERROR;\n+}\n+\n+static int null_streq(char *a, char *b)\n+{\n+\tchar *empty = \"\";\n+\tif (a == NULL)\n+\t\ta = empty;\n+\n+\tif (b == NULL)\n+\t\tb = empty;\n+\n+\treturn 0 == strcmp(a, b);\n+}\n+\n+static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)\n+{\n+\tif (a == NULL)\n+\t\ta = zero;\n+\n+\tif (b == NULL)\n+\t\tb = zero;\n+\n+\treturn !memcmp(a, b, sz);\n+}\n+\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size)\n+{\n+\tif (!(null_streq(a->refname, b->refname) &&\n+\t a->update_index == b->update_index &&\n+\t a->value_type == b->value_type))\n+\t\treturn 0;\n+\n+\tswitch (a->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\treturn 1;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\treturn null_streq(a->update.name, b->update.name) &&\n+\t\t a->update.time == b->update.time &&\n+\t\t a->update.tz_offset == b->update.tz_offset &&\n+\t\t null_streq(a->update.email, b->update.email) &&\n+\t\t null_streq(a->update.message, b->update.message) &&\n+\t\t zero_hash_eq(a->update.old_hash, b->update.old_hash,\n+\t\t\t\t hash_size) &&\n+\t\t zero_hash_eq(a->update.new_hash, b->update.new_hash,\n+\t\t\t\t hash_size);\n+\t}\n+\n+\tabort();\n+}\n+\n+static int reftable_log_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_log_record_is_deletion(\n+\t\t(const struct reftable_log_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_log_record_vtable = {\n+\t.key = &reftable_log_record_key,\n+\t.type = BLOCK_TYPE_LOG,\n+\t.copy_from = &reftable_log_record_copy_from,\n+\t.val_type = &reftable_log_record_val_type,\n+\t.encode = &reftable_log_record_encode,\n+\t.decode = &reftable_log_record_decode,\n+\t.release = &reftable_log_record_release_void,\n+\t.is_deletion = &reftable_log_record_is_deletion_void,\n+};\n+\n+struct reftable_record reftable_new_record(uint8_t typ)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF: {\n+\t\tstruct reftable_ref_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_ref_record));\n+\t\treftable_record_from_ref(&rec, r);\n+\t\treturn rec;\n+\t}\n+\n+\tcase BLOCK_TYPE_OBJ: {\n+\t\tstruct reftable_obj_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_obj_record));\n+\t\treftable_record_from_obj(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_LOG: {\n+\t\tstruct reftable_log_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_log_record));\n+\t\treftable_record_from_log(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_INDEX: {\n+\t\tstruct reftable_index_record empty = { .last_key =\n+\t\t\t\t\t\t\t STRBUF_INIT };\n+\t\tstruct reftable_index_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_index_record));\n+\t\t*r = empty;\n+\t\treftable_record_from_index(&rec, r);\n+\t\treturn rec;\n+\t}\n+\t}\n+\tabort();\n+\treturn rec;\n+}\n+\n+/* clear out the record, yielding the reftable_record data that was\n+ * encapsulated. */\n+static void *reftable_record_yield(struct reftable_record *rec)\n+{\n+\tvoid *p = rec->data;\n+\trec->data = NULL;\n+\treturn p;\n+}\n+\n+void reftable_record_destroy(struct reftable_record *rec)\n+{\n+\treftable_record_release(rec);\n+\treftable_free(reftable_record_yield(rec));\n+}\n+\n+static void reftable_index_record_key(const void *r, struct strbuf *dest)\n+{\n+\tstruct reftable_index_record *rec = (struct reftable_index_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addbuf(dest, &rec->last_key);\n+}\n+\n+static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_index_record *dst = (struct reftable_index_record *)rec;\n+\tstruct reftable_index_record *src =\n+\t\t(struct reftable_index_record *)src_rec;\n+\n+\tstrbuf_reset(&dst->last_key);\n+\tstrbuf_addbuf(&dst->last_key, &src->last_key);\n+\tdst->offset = src->offset;\n+}\n+\n+static void reftable_index_record_release(void *rec)\n+{\n+\tstruct reftable_index_record *idx = (struct reftable_index_record *)rec;\n+\tstrbuf_release(&idx->last_key);\n+}\n+\n+static uint8_t reftable_index_record_val_type(const void *rec)\n+{\n+\treturn 0;\n+}\n+\n+static int reftable_index_record_encode(const void *rec, struct string_view out,\n+\t\t\t\t\tint hash_size)\n+{\n+\tconst struct reftable_index_record *r =\n+\t\t(const struct reftable_index_record *)rec;\n+\tstruct string_view start = out;\n+\n+\tint n = put_var_int(&out, r->offset);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&out, n);\n+\n+\treturn start.len - out.len;\n+}\n+\n+static int reftable_index_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t\tuint8_t val_type, struct string_view in,\n+\t\t\t\t\tint hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_index_record *r = (struct reftable_index_record *)rec;\n+\tint n = 0;\n+\n+\tstrbuf_reset(&r->last_key);\n+\tstrbuf_addbuf(&r->last_key, &key);\n+\n+\tn = get_var_int(&r->offset, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&in, n);\n+\treturn start.len - in.len;\n+}\n+\n+static struct reftable_record_vtable reftable_index_record_vtable = {\n+\t.key = &reftable_index_record_key,\n+\t.type = BLOCK_TYPE_INDEX,\n+\t.copy_from = &reftable_index_record_copy_from,\n+\t.val_type = &reftable_index_record_val_type,\n+\t.encode = &reftable_index_record_encode,\n+\t.decode = &reftable_index_record_decode,\n+\t.release = &reftable_index_record_release,\n+\t.is_deletion = ¬_a_deletion,\n+};\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)\n+{\n+\trec->ops->key(rec->data, dest);\n+}\n+\n+uint8_t reftable_record_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->type;\n+}\n+\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size)\n+{\n+\treturn rec->ops->encode(rec->data, dest, hash_size);\n+}\n+\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size)\n+{\n+\tassert(src->ops->type == rec->ops->type);\n+\n+\trec->ops->copy_from(rec->data, src->data, hash_size);\n+}\n+\n+uint8_t reftable_record_val_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->val_type(rec->data);\n+}\n+\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src, int hash_size)\n+{\n+\treturn rec->ops->decode(rec->data, key, extra, src, hash_size);\n+}\n+\n+void reftable_record_release(struct reftable_record *rec)\n+{\n+\trec->ops->release(rec->data);\n+}\n+\n+int reftable_record_is_deletion(struct reftable_record *rec)\n+{\n+\treturn rec->ops->is_deletion(rec->data);\n+}\n+\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *ref_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = ref_rec;\n+\trec->ops = &reftable_ref_record_vtable;\n+}\n+\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *obj_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = obj_rec;\n+\trec->ops = &reftable_obj_record_vtable;\n+}\n+\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *index_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = index_rec;\n+\trec->ops = &reftable_index_record_vtable;\n+}\n+\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *log_rec)\n+{\n+\tassert(rec->ops == NULL);\n+\trec->data = log_rec;\n+\trec->ops = &reftable_log_record_vtable;\n+}\n+\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_REF);\n+\treturn (struct reftable_ref_record *)rec->data;\n+}\n+\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_LOG);\n+\treturn (struct reftable_log_record *)rec->data;\n+}\n+\n+static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)\n+{\n+\tif (a != NULL && b != NULL)\n+\t\treturn !memcmp(a, b, hash_size);\n+\n+\treturn a == b;\n+}\n+\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (!(0 == strcmp(a->refname, b->refname) &&\n+\t a->update_index == b->update_index &&\n+\t a->value_type == b->value_type))\n+\t\treturn 0;\n+\n+\tswitch (a->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\treturn !strcmp(a->value.symref, b->value.symref);\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn hash_equal(a->value.val2.value, b->value.val2.value,\n+\t\t\t\t hash_size) &&\n+\t\t hash_equal(a->value.val2.target_value,\n+\t\t\t\t b->value.val2.target_value, hash_size);\n+\tcase REFTABLE_REF_VAL1:\n+\t\treturn hash_equal(a->value.val1, b->value.val1, hash_size);\n+\tcase REFTABLE_REF_DELETION:\n+\t\treturn 1;\n+\tdefault:\n+\t\tabort();\n+\t}\n+}\n+\n+int reftable_ref_record_compare_name(const void *a, const void *b)\n+{\n+\treturn strcmp(((struct reftable_ref_record *)a)->refname,\n+\t\t ((struct reftable_ref_record *)b)->refname);\n+}\n+\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref)\n+{\n+\treturn ref->value_type == REFTABLE_REF_DELETION;\n+}\n+\n+int reftable_log_record_compare_key(const void *a, const void *b)\n+{\n+\tstruct reftable_log_record *la = (struct reftable_log_record *)a;\n+\tstruct reftable_log_record *lb = (struct reftable_log_record *)b;\n+\n+\tint cmp = strcmp(la->refname, lb->refname);\n+\tif (cmp)\n+\t\treturn cmp;\n+\tif (la->update_index > lb->update_index)\n+\t\treturn -1;\n+\treturn (la->update_index < lb->update_index) ? 1 : 0;\n+}\n+\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log)\n+{\n+\treturn (log->value_type == REFTABLE_LOG_DELETION);\n+}\n+\n+void string_view_consume(struct string_view *s, int n)\n+{\n+\ts->buf += n;\n+\ts->len -= n;\n+}\ndiff --git a/reftable/record.h b/reftable/record.h\nnew file mode 100644\nindex 000000000000..498e8c50bf4f\n--- /dev/null\n+++ b/reftable/record.h\n@@ -0,0 +1,139 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef RECORD_H\n+#define RECORD_H\n+\n+#include \"system.h\"\n+\n+#include \n+\n+#include \"reftable-record.h\"\n+\n+/*\n+ * A substring of existing string data. This structure takes no responsibility\n+ * for the lifetime of the data it points to.\n+ */\n+struct string_view {\n+\tuint8_t *buf;\n+\tsize_t len;\n+};\n+\n+/* Advance `s.buf` by `n`, and decrease length. */\n+void string_view_consume(struct string_view *s, int n);\n+\n+/* utilities for de/encoding varints */\n+\n+int get_var_int(uint64_t *dest, struct string_view *in);\n+int put_var_int(struct string_view *dest, uint64_t val);\n+\n+/* Methods for records. */\n+struct reftable_record_vtable {\n+\t/* encode the key of to a uint8_t strbuf. */\n+\tvoid (*key)(const void *rec, struct strbuf *dest);\n+\n+\t/* The record type of ('r' for ref). */\n+\tuint8_t type;\n+\n+\tvoid (*copy_from)(void *dest, const void *src, int hash_size);\n+\n+\t/* a value of [0..7], indicating record subvariants (eg. ref vs. symref\n+\t * vs ref deletion) */\n+\tuint8_t (*val_type)(const void *rec);\n+\n+\t/* encodes rec into dest, returning how much space was used. */\n+\tint (*encode)(const void *rec, struct string_view dest, int hash_size);\n+\n+\t/* decode data from `src` into the record. */\n+\tint (*decode)(void *rec, struct strbuf key, uint8_t extra,\n+\t\t struct string_view src, int hash_size);\n+\n+\t/* deallocate and null the record. */\n+\tvoid (*release)(void *rec);\n+\n+\t/* is this a tombstone? */\n+\tint (*is_deletion)(const void *rec);\n+};\n+\n+/* record is a generic wrapper for different types of records. */\n+struct reftable_record {\n+\tvoid *data;\n+\tstruct reftable_record_vtable *ops;\n+};\n+\n+/* returns true for recognized block types. Block start with the block type. */\n+int reftable_is_block_type(uint8_t typ);\n+\n+/* creates a malloced record of the given type. Dispose with record_destroy */\n+struct reftable_record reftable_new_record(uint8_t typ);\n+\n+/* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns\n+ * number of bytes written. */\n+int reftable_encode_key(int *is_restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra);\n+\n+/* Decode into `key` and `extra` from `in` */\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in);\n+\n+/* reftable_index_record are used internally to speed up lookups. */\n+struct reftable_index_record {\n+\tuint64_t offset; /* Offset of block */\n+\tstruct strbuf last_key; /* Last key of the block. */\n+};\n+\n+/* reftable_obj_record stores an object ID => ref mapping. */\n+struct reftable_obj_record {\n+\tuint8_t *hash_prefix; /* leading bytes of the object ID */\n+\tint hash_prefix_len; /* number of leading bytes. Constant\n+\t\t\t * across a single table. */\n+\tuint64_t *offsets; /* a vector of file offsets. */\n+\tint offset_len;\n+};\n+\n+/* see struct record_vtable */\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);\n+uint8_t reftable_record_type(struct reftable_record *rec);\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size);\n+uint8_t reftable_record_val_type(struct reftable_record *rec);\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size);\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src,\n+\t\t\t int hash_size);\n+int reftable_record_is_deletion(struct reftable_record *rec);\n+\n+/* zeroes out the embedded record */\n+void reftable_record_release(struct reftable_record *rec);\n+\n+/* clear and deallocate embedded record, and zero `rec`. */\n+void reftable_record_destroy(struct reftable_record *rec);\n+\n+/* initialize generic records from concrete records. The generic record should\n+ * be zeroed out. */\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *objrec);\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *idxrec);\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *refrec);\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *logrec);\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *ref);\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *ref);\n+\n+/* for qsort. */\n+int reftable_ref_record_compare_name(const void *a, const void *b);\n+\n+/* for qsort. */\n+int reftable_log_record_compare_key(const void *a, const void *b);\n+\n+#endif\ndiff --git a/reftable/record_test.c b/reftable/record_test.c\nnew file mode 100644\nindex 000000000000..da7951bd480b\n--- /dev/null\n+++ b/reftable/record_test.c\n@@ -0,0 +1,405 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_copy(struct reftable_record *rec)\n+{\n+\tstruct reftable_record copy =\n+\t\treftable_new_record(reftable_record_type(rec));\n+\treftable_record_copy_from(©, rec, SHA1_SIZE);\n+\t/* do it twice to catch memory leaks */\n+\treftable_record_copy_from(©, rec, SHA1_SIZE);\n+\tswitch (reftable_record_type(©)) {\n+\tcase BLOCK_TYPE_REF:\n+\t\tEXPECT(reftable_ref_record_equal(reftable_record_as_ref(©),\n+\t\t\t\t\t\t reftable_record_as_ref(rec),\n+\t\t\t\t\t\t SHA1_SIZE));\n+\t\tbreak;\n+\tcase BLOCK_TYPE_LOG:\n+\t\tEXPECT(reftable_log_record_equal(reftable_record_as_log(©),\n+\t\t\t\t\t\t reftable_record_as_log(rec),\n+\t\t\t\t\t\t SHA1_SIZE));\n+\t\tbreak;\n+\t}\n+\treftable_record_destroy(©);\n+}\n+\n+static void test_varint_roundtrip(void)\n+{\n+\tuint64_t inputs[] = { 0,\n+\t\t\t 1,\n+\t\t\t 27,\n+\t\t\t 127,\n+\t\t\t 128,\n+\t\t\t 257,\n+\t\t\t 4096,\n+\t\t\t ((uint64_t)1 << 63),\n+\t\t\t ((uint64_t)1 << 63) + ((uint64_t)1 << 63) - 1 };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(inputs); i++) {\n+\t\tuint8_t dest[10];\n+\n+\t\tstruct string_view out = {\n+\t\t\t.buf = dest,\n+\t\t\t.len = sizeof(dest),\n+\t\t};\n+\t\tuint64_t in = inputs[i];\n+\t\tint n = put_var_int(&out, in);\n+\t\tuint64_t got = 0;\n+\n+\t\tEXPECT(n > 0);\n+\t\tout.len = n;\n+\t\tn = get_var_int(&got, &out);\n+\t\tEXPECT(n > 0);\n+\n+\t\tEXPECT(got == in);\n+\t}\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct {\n+\t\tconst char *a, *b;\n+\t\tint want;\n+\t} cases[] = {\n+\t\t{ \"abc\", \"ab\", 2 },\n+\t\t{ \"\", \"abc\", 0 },\n+\t\t{ \"abc\", \"abd\", 2 },\n+\t\t{ \"abc\", \"pqr\", 0 },\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct strbuf a = STRBUF_INIT;\n+\t\tstruct strbuf b = STRBUF_INIT;\n+\t\tstrbuf_addstr(&a, cases[i].a);\n+\t\tstrbuf_addstr(&b, cases[i].b);\n+\t\tEXPECT(common_prefix_size(&a, &b) == cases[i].want);\n+\n+\t\tstrbuf_release(&a);\n+\t\tstrbuf_release(&b);\n+\t}\n+}\n+\n+static void set_hash(uint8_t *h, int j)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < hash_size(SHA1_ID); i++) {\n+\t\th[i] = (j >> i) & 0xff;\n+\t}\n+}\n+\n+static void test_reftable_ref_record_roundtrip(void)\n+{\n+\tint i = 0;\n+\n+\tfor (i = REFTABLE_REF_DELETION; i < REFTABLE_NR_REF_VALUETYPES; i++) {\n+\t\tstruct reftable_ref_record in = { NULL };\n+\t\tstruct reftable_ref_record out = { NULL };\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\n+\t\tint n, m;\n+\n+\t\tin.value_type = i;\n+\t\tswitch (i) {\n+\t\tcase REFTABLE_REF_DELETION:\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\tin.value.val1 = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_hash(in.value.val1, 1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\tin.value.val2.value = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_hash(in.value.val2.value, 1);\n+\t\t\tin.value.val2.target_value = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_hash(in.value.val2.target_value, 2);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF:\n+\t\t\tin.value.symref = xstrdup(\"target\");\n+\t\t\tbreak;\n+\t\t}\n+\t\tin.refname = xstrdup(\"refs/heads/master\");\n+\n+\t\treftable_record_from_ref(&rec, &in);\n+\t\ttest_copy(&rec);\n+\n+\t\tEXPECT(reftable_record_val_type(&rec) == i);\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\t\tEXPECT(n > 0);\n+\n+\t\t/* decode into a non-zero reftable_record to test for leaks. */\n+\n+\t\treftable_record_from_ref(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, i, dest, SHA1_SIZE);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(reftable_ref_record_equal(&in, &out, SHA1_SIZE));\n+\t\treftable_record_release(&rec_out);\n+\n+\t\tstrbuf_release(&key);\n+\t\treftable_ref_record_release(&in);\n+\t}\n+}\n+\n+static void test_reftable_log_record_equal(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 42,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t}\n+\t};\n+\n+\tEXPECT(!reftable_log_record_equal(&in[0], &in[1], SHA1_SIZE));\n+\tin[1].update_index = in[0].update_index;\n+\tEXPECT(reftable_log_record_equal(&in[0], &in[1], SHA1_SIZE));\n+\treftable_log_record_release(&in[0]);\n+\treftable_log_record_release(&in[1]);\n+}\n+\n+static void test_reftable_log_record_roundtrip(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 42,\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.update = {\n+\t\t\t\t.old_hash = reftable_malloc(SHA1_SIZE),\n+\t\t\t\t.new_hash = reftable_malloc(SHA1_SIZE),\n+\t\t\t\t.name = xstrdup(\"han-wen\"),\n+\t\t\t\t.email = xstrdup(\"hanwen@google.com\"),\n+\t\t\t\t.message = xstrdup(\"test\"),\n+\t\t\t\t.time = 1577123507,\n+\t\t\t\t.tz_offset = 100,\n+\t\t\t}\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t\t.value_type = REFTABLE_LOG_DELETION,\n+\t\t}\n+\t};\n+\tset_test_hash(in[0].update.new_hash, 1);\n+\tset_test_hash(in[0].update.old_hash, 2);\n+\tfor (int i = 0; i < ARRAY_SIZE(in); i++) {\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\t/* populate out, to check for leaks. */\n+\t\tstruct reftable_log_record out = {\n+\t\t\t.refname = xstrdup(\"old name\"),\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.update = {\n+\t\t\t\t.new_hash = reftable_calloc(SHA1_SIZE),\n+\t\t\t\t.old_hash = reftable_calloc(SHA1_SIZE),\n+\t\t\t\t.name = xstrdup(\"old name\"),\n+\t\t\t\t.email = xstrdup(\"old@email\"),\n+\t\t\t\t.message = xstrdup(\"old message\"),\n+\t\t\t},\n+\t\t};\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m, valtype;\n+\n+\t\treftable_record_from_log(&rec, &in[i]);\n+\n+\t\ttest_copy(&rec);\n+\n+\t\treftable_record_key(&rec, &key);\n+\n+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\t\tEXPECT(n >= 0);\n+\t\treftable_record_from_log(&rec_out, &out);\n+\t\tvaltype = reftable_record_val_type(&rec);\n+\t\tm = reftable_record_decode(&rec_out, key, valtype, dest,\n+\t\t\t\t\t SHA1_SIZE);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(reftable_log_record_equal(&in[i], &out, SHA1_SIZE));\n+\t\treftable_log_record_release(&in[i]);\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_release(&rec_out);\n+\t}\n+}\n+\n+static void test_u24_roundtrip(void)\n+{\n+\tuint32_t in = 0x112233;\n+\tuint8_t dest[3];\n+\tuint32_t out;\n+\tput_be24(dest, in);\n+\tout = get_be24(dest);\n+\tEXPECT(in == out);\n+}\n+\n+static void test_key_roundtrip(void)\n+{\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct strbuf roundtrip = STRBUF_INIT;\n+\tint restart;\n+\tuint8_t extra;\n+\tint n, m;\n+\tuint8_t rt_extra;\n+\n+\tstrbuf_addstr(&last_key, \"refs/heads/master\");\n+\tstrbuf_addstr(&key, \"refs/tags/bla\");\n+\textra = 6;\n+\tn = reftable_encode_key(&restart, dest, last_key, key, extra);\n+\tEXPECT(!restart);\n+\tEXPECT(n > 0);\n+\n+\tm = reftable_decode_key(&roundtrip, &rt_extra, last_key, dest);\n+\tEXPECT(n == m);\n+\tEXPECT(0 == strbuf_cmp(&key, &roundtrip));\n+\tEXPECT(rt_extra == extra);\n+\n+\tstrbuf_release(&last_key);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&roundtrip);\n+}\n+\n+static void test_reftable_obj_record_roundtrip(void)\n+{\n+\tuint8_t testHash1[SHA1_SIZE] = { 1, 2, 3, 4, 0 };\n+\tuint64_t till9[] = { 1, 2, 3, 4, 500, 600, 700, 800, 9000 };\n+\tstruct reftable_obj_record recs[3] = { {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 3,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 9,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t } };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(recs); i++) {\n+\t\tstruct reftable_obj_record in = recs[i];\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_obj_record out = { NULL };\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m;\n+\t\tuint8_t extra;\n+\n+\t\treftable_record_from_obj(&rec, &in);\n+\t\ttest_copy(&rec);\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\t\tEXPECT(n > 0);\n+\t\textra = reftable_record_val_type(&rec);\n+\t\treftable_record_from_obj(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, extra, dest,\n+\t\t\t\t\t SHA1_SIZE);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(in.hash_prefix_len == out.hash_prefix_len);\n+\t\tEXPECT(in.offset_len == out.offset_len);\n+\n+\t\tEXPECT(!memcmp(in.hash_prefix, out.hash_prefix,\n+\t\t\t in.hash_prefix_len));\n+\t\tEXPECT(0 == memcmp(in.offsets, out.offsets,\n+\t\t\t\t sizeof(uint64_t) * in.offset_len));\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_release(&rec_out);\n+\t}\n+}\n+\n+static void test_reftable_index_record_roundtrip(void)\n+{\n+\tstruct reftable_index_record in = {\n+\t\t.offset = 42,\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_index_record out = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record out_rec = { NULL };\n+\tint n, m;\n+\tuint8_t extra;\n+\n+\tstrbuf_addstr(&in.last_key, \"refs/heads/master\");\n+\treftable_record_from_index(&rec, &in);\n+\treftable_record_key(&rec, &key);\n+\ttest_copy(&rec);\n+\n+\tEXPECT(0 == strbuf_cmp(&key, &in.last_key));\n+\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n+\tEXPECT(n > 0);\n+\n+\textra = reftable_record_val_type(&rec);\n+\treftable_record_from_index(&out_rec, &out);\n+\tm = reftable_record_decode(&out_rec, key, extra, dest, SHA1_SIZE);\n+\tEXPECT(m == n);\n+\n+\tEXPECT(in.offset == out.offset);\n+\n+\treftable_record_release(&out_rec);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&in.last_key);\n+}\n+\n+int record_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_reftable_log_record_equal);\n+\tRUN_TEST(test_reftable_log_record_roundtrip);\n+\tRUN_TEST(test_reftable_ref_record_roundtrip);\n+\tRUN_TEST(test_varint_roundtrip);\n+\tRUN_TEST(test_key_roundtrip);\n+\tRUN_TEST(test_common_prefix);\n+\tRUN_TEST(test_reftable_obj_record_roundtrip);\n+\tRUN_TEST(test_reftable_index_record_roundtrip);\n+\tRUN_TEST(test_u24_roundtrip);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-record.h b/reftable/reftable-record.h\nnew file mode 100644\nindex 000000000000..7985b94ae2cc\n--- /dev/null\n+++ b/reftable/reftable-record.h\n@@ -0,0 +1,114 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_RECORD_H\n+#define REFTABLE_RECORD_H\n+\n+#include \n+\n+/*\n+ * Basic data types\n+ *\n+ * Reftables store the state of each ref in struct reftable_ref_record, and they\n+ * store a sequence of reflog updates in struct reftable_log_record.\n+ */\n+\n+/* reftable_ref_record holds a ref database entry target_value */\n+struct reftable_ref_record {\n+\tchar *refname; /* Name of the ref, malloced. */\n+\tuint64_t update_index; /* Logical timestamp at which this value is\n+\t\t\t\t* written */\n+\n+\tenum {\n+\t\t/* tombstone to hide deletions from earlier tables */\n+\t\tREFTABLE_REF_DELETION = 0x0,\n+\n+\t\t/* a simple ref */\n+\t\tREFTABLE_REF_VAL1 = 0x1,\n+\t\t/* a tag, plus its peeled hash */\n+\t\tREFTABLE_REF_VAL2 = 0x2,\n+\n+\t\t/* a symbolic reference */\n+\t\tREFTABLE_REF_SYMREF = 0x3,\n+#define REFTABLE_NR_REF_VALUETYPES 4\n+\t} value_type;\n+\tunion {\n+\t\tuint8_t *val1; /* malloced hash. */\n+\t\tstruct {\n+\t\t\tuint8_t *value; /* first value, malloced hash */\n+\t\t\tuint8_t *target_value; /* second value, malloced hash */\n+\t\t} val2;\n+\t\tchar *symref; /* referent, malloced 0-terminated string */\n+\t} value;\n+};\n+\n+/* Returns the first hash, or NULL if `rec` is not of type\n+ * REFTABLE_REF_VAL1 or REFTABLE_REF_VAL2. */\n+uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec);\n+\n+/* Returns the second hash, or NULL if `rec` is not of type\n+ * REFTABLE_REF_VAL2. */\n+uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec);\n+\n+/* returns whether 'ref' represents a deletion */\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);\n+\n+/* prints a reftable_ref_record onto stdout. Useful for debugging. */\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id);\n+\n+/* frees and nulls all pointer values inside `ref`. */\n+void reftable_ref_record_release(struct reftable_ref_record *ref);\n+\n+/* returns whether two reftable_ref_records are the same. Useful for testing. */\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size);\n+\n+/* reftable_log_record holds a reflog entry */\n+struct reftable_log_record {\n+\tchar *refname;\n+\tuint64_t update_index; /* logical timestamp of a transactional update.\n+\t\t\t\t*/\n+\n+\tenum {\n+\t\t/* tombstone to hide deletions from earlier tables */\n+\t\tREFTABLE_LOG_DELETION = 0x0,\n+\n+\t\t/* a simple update */\n+\t\tREFTABLE_LOG_UPDATE = 0x1,\n+#define REFTABLE_NR_LOG_VALUETYPES 2\n+\t} value_type;\n+\n+\tunion {\n+\t\tstruct {\n+\t\t\tuint8_t *new_hash;\n+\t\t\tuint8_t *old_hash;\n+\t\t\tchar *name;\n+\t\t\tchar *email;\n+\t\t\tuint64_t time;\n+\t\t\tint16_t tz_offset;\n+\t\t\tchar *message;\n+\t\t} update;\n+\t};\n+};\n+\n+/* returns whether 'ref' represents the deletion of a log record. */\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log);\n+\n+/* frees and nulls all pointer values. */\n+void reftable_log_record_release(struct reftable_log_record *log);\n+\n+/* returns whether two records are equal. Useful for testing. */\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size);\n+\n+/* dumps a reftable_log_record on stdout, for debugging/testing. */\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 3b58e423e7b1..09d4b83ef9b8 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,6 +4,6 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n-\n+\trecord_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"421760","messageId":"9297b9c363f6d922443d261375759046e303351d.1618255553.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 07/20] reftable: reading/writing blocks","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:39Z","receivedAt":"2021-04-12T19:26:19Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is structured as a sequence of block. Within a block,\nrecords are prefix compressed, with an index of offsets for fully expand keys to\nenable binary search within blocks.\n\nThis commit provides the logic to read and write these blocks.\n\nIncludes a code snippet copied from zlib\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 3 +\n reftable/.gitattributes | 1 +\n reftable/block.c | 440 +++++++++++++++++++++++++++++++++++++++\n reftable/block.h | 127 +++++++++++\n reftable/block_test.c | 121 +++++++++++\n reftable/zlib-compat.c | 92 ++++++++\n t/helper/test-reftable.c | 1 +\n 7 files changed, 785 insertions(+)\n create mode 100644 reftable/.gitattributes\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n create mode 100644 reftable/zlib-compat.c\n\ndiff --git a/Makefile b/Makefile\nindex 313421d16012..89f998a9a5f2 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2402,10 +2402,13 @@ xdiff-objs: $(XDIFF_OBJS)\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/zlib-compat.o\n \n+REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\ndiff --git a/reftable/.gitattributes b/reftable/.gitattributes\nnew file mode 100644\nindex 000000000000..f44451a37954\n--- /dev/null\n+++ b/reftable/.gitattributes\n@@ -0,0 +1 @@\n+/zlib-compat.c\twhitespace=-indent-with-non-tab,-trailing-space\ndiff --git a/reftable/block.c b/reftable/block.c\nnew file mode 100644\nindex 000000000000..5b4e49fb2be2\n--- /dev/null\n+++ b/reftable/block.c\n@@ -0,0 +1,440 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"reftable-error.h\"\n+#include \"system.h\"\n+#include \"zlib.h\"\n+\n+int header_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 24;\n+\tcase 2:\n+\t\treturn 28;\n+\t}\n+\tabort();\n+}\n+\n+int footer_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 68;\n+\tcase 2:\n+\t\treturn 72;\n+\t}\n+\tabort();\n+}\n+\n+static int block_writer_register_restart(struct block_writer *w, int n,\n+\t\t\t\t\t int is_restart, struct strbuf *key)\n+{\n+\tint rlen = w->restart_len;\n+\tif (rlen >= MAX_RESTARTS) {\n+\t\tis_restart = 0;\n+\t}\n+\n+\tif (is_restart) {\n+\t\trlen++;\n+\t}\n+\tif (2 + 3 * rlen + n > w->block_size - w->next)\n+\t\treturn -1;\n+\tif (is_restart) {\n+\t\tif (w->restart_len == w->restart_cap) {\n+\t\t\tw->restart_cap = w->restart_cap * 2 + 1;\n+\t\t\tw->restarts = reftable_realloc(\n+\t\t\t\tw->restarts, sizeof(uint32_t) * w->restart_cap);\n+\t\t}\n+\n+\t\tw->restarts[w->restart_len++] = w->next;\n+\t}\n+\n+\tw->next += n;\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, key);\n+\tw->entries++;\n+\treturn 0;\n+}\n+\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n+{\n+\tbw->buf = buf;\n+\tbw->hash_size = hash_size;\n+\tbw->block_size = block_size;\n+\tbw->header_off = header_off;\n+\tbw->buf[header_off] = typ;\n+\tbw->next = header_off + 4;\n+\tbw->restart_interval = 16;\n+\tbw->entries = 0;\n+\tbw->restart_len = 0;\n+\tbw->last_key.len = 0;\n+}\n+\n+uint8_t block_writer_type(struct block_writer *bw)\n+{\n+\treturn bw->buf[bw->header_off];\n+}\n+\n+/* adds the reftable_record to the block. Returns -1 if it does not fit, 0 on\n+ success */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tstruct strbuf last =\n+\t\tw->entries % w->restart_interval == 0 ? empty : w->last_key;\n+\tstruct string_view out = {\n+\t\t.buf = w->buf + w->next,\n+\t\t.len = w->block_size - w->next,\n+\t};\n+\n+\tstruct string_view start = out;\n+\n+\tint is_restart = 0;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint n = 0;\n+\n+\treftable_record_key(rec, &key);\n+\tn = reftable_encode_key(&is_restart, out, last, key,\n+\t\t\t\treftable_record_val_type(rec));\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tn = reftable_record_encode(rec, out, w->hash_size);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tif (block_writer_register_restart(w, start.len - out.len, is_restart,\n+\t\t\t\t\t &key) < 0)\n+\t\tgoto done;\n+\n+\tstrbuf_release(&key);\n+\treturn 0;\n+\n+done:\n+\tstrbuf_release(&key);\n+\treturn -1;\n+}\n+\n+int block_writer_finish(struct block_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->restart_len; i++) {\n+\t\tput_be24(w->buf + w->next, w->restarts[i]);\n+\t\tw->next += 3;\n+\t}\n+\n+\tput_be16(w->buf + w->next, w->restart_len);\n+\tw->next += 2;\n+\tput_be24(w->buf + 1 + w->header_off, w->next);\n+\n+\tif (block_writer_type(w) == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + w->header_off;\n+\t\tuint8_t *compressed = NULL;\n+\t\tint zresult = 0;\n+\t\tuLongf src_len = w->next - block_header_skip;\n+\t\tsize_t dest_cap = src_len;\n+\n+\t\tcompressed = reftable_malloc(dest_cap);\n+\t\twhile (1) {\n+\t\t\tuLongf out_dest_len = dest_cap;\n+\n+\t\t\tzresult = compress2(compressed, &out_dest_len,\n+\t\t\t\t\t w->buf + block_header_skip, src_len,\n+\t\t\t\t\t 9);\n+\t\t\tif (zresult == Z_BUF_ERROR) {\n+\t\t\t\tdest_cap *= 2;\n+\t\t\t\tcompressed =\n+\t\t\t\t\treftable_realloc(compressed, dest_cap);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (Z_OK != zresult) {\n+\t\t\t\treftable_free(compressed);\n+\t\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t\t}\n+\n+\t\t\tmemcpy(w->buf + block_header_skip, compressed,\n+\t\t\t out_dest_len);\n+\t\t\tw->next = out_dest_len + block_header_skip;\n+\t\t\treftable_free(compressed);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\treturn w->next;\n+}\n+\n+uint8_t block_reader_type(struct block_reader *r)\n+{\n+\treturn r->block.data[r->header_off];\n+}\n+\n+int block_reader_init(struct block_reader *br, struct reftable_block *block,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size)\n+{\n+\tuint32_t full_block_size = table_block_size;\n+\tuint8_t typ = block->data[header_off];\n+\tuint32_t sz = get_be24(block->data + header_off + 1);\n+\n+\tuint16_t restart_count = 0;\n+\tuint32_t restart_start = 0;\n+\tuint8_t *restart_bytes = NULL;\n+\n+\tif (!reftable_is_block_type(typ))\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tif (typ == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + header_off;\n+\t\tuLongf dst_len = sz - block_header_skip; /* total size of dest\n+\t\t\t\t\t\t\t buffer. */\n+\t\tuLongf src_len = block->len - block_header_skip;\n+\t\t/* Log blocks specify the *uncompressed* size in their header.\n+\t\t */\n+\t\tuint8_t *uncompressed = reftable_malloc(sz);\n+\n+\t\t/* Copy over the block header verbatim. It's not compressed. */\n+\t\tmemcpy(uncompressed, block->data, block_header_skip);\n+\n+\t\t/* Uncompress */\n+\t\tif (Z_OK != uncompress_return_consumed(\n+\t\t\t\t uncompressed + block_header_skip, &dst_len,\n+\t\t\t\t block->data + block_header_skip,\n+\t\t\t\t &src_len)) {\n+\t\t\treftable_free(uncompressed);\n+\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t}\n+\n+\t\tif (dst_len + block_header_skip != sz)\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\t\t/* We're done with the input data. */\n+\t\treftable_block_done(block);\n+\t\tblock->data = uncompressed;\n+\t\tblock->len = sz;\n+\t\tblock->source = malloc_block_source();\n+\t\tfull_block_size = src_len + block_header_skip;\n+\t} else if (full_block_size == 0) {\n+\t\tfull_block_size = sz;\n+\t} else if (sz < full_block_size && sz < block->len &&\n+\t\t block->data[sz] != 0) {\n+\t\t/* If the block is smaller than the full block size, it is\n+\t\t padded (data followed by '\\0') or the next block is\n+\t\t unaligned. */\n+\t\tfull_block_size = sz;\n+\t}\n+\n+\trestart_count = get_be16(block->data + sz - 2);\n+\trestart_start = sz - 2 - 3 * restart_count;\n+\trestart_bytes = block->data + restart_start;\n+\n+\t/* transfer ownership. */\n+\tbr->block = *block;\n+\tblock->data = NULL;\n+\tblock->len = 0;\n+\n+\tbr->hash_size = hash_size;\n+\tbr->block_len = restart_start;\n+\tbr->full_block_size = full_block_size;\n+\tbr->header_off = header_off;\n+\tbr->restart_count = restart_count;\n+\tbr->restart_bytes = restart_bytes;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t block_reader_restart_offset(struct block_reader *br, int i)\n+{\n+\treturn get_be24(br->restart_bytes + 3 * i);\n+}\n+\n+void block_reader_start(struct block_reader *br, struct block_iter *it)\n+{\n+\tit->br = br;\n+\tstrbuf_reset(&it->last_key);\n+\tit->next_off = br->header_off + 4;\n+}\n+\n+struct restart_find_args {\n+\tint error;\n+\tstruct strbuf key;\n+\tstruct block_reader *r;\n+};\n+\n+static int restart_key_less(size_t idx, void *args)\n+{\n+\tstruct restart_find_args *a = (struct restart_find_args *)args;\n+\tuint32_t off = block_reader_restart_offset(a->r, idx);\n+\tstruct string_view in = {\n+\t\t.buf = a->r->block.data + off,\n+\t\t.len = a->r->block_len - off,\n+\t};\n+\n+\t/* the restart key is verbatim in the block, so this could avoid the\n+\t alloc for decoding the key */\n+\tstruct strbuf rkey = STRBUF_INIT;\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tuint8_t unused_extra;\n+\tint n = reftable_decode_key(&rkey, &unused_extra, last_key, in);\n+\tint result;\n+\tif (n < 0) {\n+\t\ta->error = 1;\n+\t\treturn -1;\n+\t}\n+\n+\tresult = strbuf_cmp(&a->key, &rkey);\n+\tstrbuf_release(&rkey);\n+\treturn result;\n+}\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src)\n+{\n+\tdest->br = src->br;\n+\tdest->next_off = src->next_off;\n+\tstrbuf_reset(&dest->last_key);\n+\tstrbuf_addbuf(&dest->last_key, &src->last_key);\n+}\n+\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec)\n+{\n+\tstruct string_view in = {\n+\t\t.buf = it->br->block.data + it->next_off,\n+\t\t.len = it->br->block_len - it->next_off,\n+\t};\n+\tstruct string_view start = in;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tuint8_t extra = 0;\n+\tint n = 0;\n+\n+\tif (it->next_off >= it->br->block_len)\n+\t\treturn 1;\n+\n+\tn = reftable_decode_key(&key, &extra, it->last_key, in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\n+\tstring_view_consume(&in, n);\n+\tn = reftable_record_decode(rec, key, extra, in, it->br->hash_size);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tstrbuf_reset(&it->last_key);\n+\tstrbuf_addbuf(&it->last_key, &key);\n+\tit->next_off += start.len - in.len;\n+\tstrbuf_release(&key);\n+\treturn 0;\n+}\n+\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tint off = br->header_off + 4;\n+\tstruct string_view in = {\n+\t\t.buf = br->block.data + off,\n+\t\t.len = br->block_len - off,\n+\t};\n+\n+\tuint8_t extra = 0;\n+\tint n = reftable_decode_key(key, &extra, empty, in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\treturn 0;\n+}\n+\n+int block_iter_seek(struct block_iter *it, struct strbuf *want)\n+{\n+\treturn block_reader_seek(it->br, it, want);\n+}\n+\n+void block_iter_close(struct block_iter *it)\n+{\n+\tstrbuf_release(&it->last_key);\n+}\n+\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want)\n+{\n+\tstruct restart_find_args args = {\n+\t\t.key = *want,\n+\t\t.r = br,\n+\t};\n+\tstruct reftable_record rec = reftable_new_record(block_reader_type(br));\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = 0;\n+\tstruct block_iter next = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\n+\tint i = binsearch(br->restart_count, &restart_key_less, &args);\n+\tif (args.error) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tit->br = br;\n+\tif (i > 0) {\n+\t\ti--;\n+\t\tit->next_off = block_reader_restart_offset(br, i);\n+\t} else {\n+\t\tit->next_off = br->header_off + 4;\n+\t}\n+\n+\t/* We're looking for the last entry less/equal than the wanted key, so\n+\t we have to go one entry too far and then back up.\n+\t*/\n+\twhile (1) {\n+\t\tblock_iter_copy_from(&next, it);\n+\t\terr = block_iter_next(&next, &rec);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tif (err > 0 || strbuf_cmp(&key, want) >= 0) {\n+\t\t\terr = 0;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tblock_iter_copy_from(it, &next);\n+\t}\n+\n+done:\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&next.last_key);\n+\treftable_record_destroy(&rec);\n+\n+\treturn err;\n+}\n+\n+void block_writer_release(struct block_writer *bw)\n+{\n+\tFREE_AND_NULL(bw->restarts);\n+\tstrbuf_release(&bw->last_key);\n+\t/* the block is not owned. */\n+}\n+\n+void reftable_block_done(struct reftable_block *blockp)\n+{\n+\tstruct reftable_block_source source = blockp->source;\n+\tif (blockp != NULL && source.ops != NULL)\n+\t\tsource.ops->return_block(source.arg, blockp);\n+\tblockp->data = NULL;\n+\tblockp->len = 0;\n+\tblockp->source.ops = NULL;\n+\tblockp->source.arg = NULL;\n+}\ndiff --git a/reftable/block.h b/reftable/block.h\nnew file mode 100644\nindex 000000000000..e207706a6448\n--- /dev/null\n+++ b/reftable/block.h\n@@ -0,0 +1,127 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCK_H\n+#define BLOCK_H\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"reftable-blocksource.h\"\n+\n+/*\n+ * Writes reftable blocks. The block_writer is reused across blocks to minimize\n+ * allocation overhead.\n+ */\n+struct block_writer {\n+\tuint8_t *buf;\n+\tuint32_t block_size;\n+\n+\t/* Offset ofof the global header. Nonzero in the first block only. */\n+\tuint32_t header_off;\n+\n+\t/* How often to restart keys. */\n+\tint restart_interval;\n+\tint hash_size;\n+\n+\t/* Offset of next uint8_t to write. */\n+\tuint32_t next;\n+\tuint32_t *restarts;\n+\tuint32_t restart_len;\n+\tuint32_t restart_cap;\n+\n+\tstruct strbuf last_key;\n+\tint entries;\n+};\n+\n+/*\n+ * initializes the blockwriter to write `typ` entries, using `buf` as temporary\n+ * storage. `buf` is not owned by the block_writer. */\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n+\n+/* returns the block type (eg. 'r' for ref records. */\n+uint8_t block_writer_type(struct block_writer *bw);\n+\n+/* appends the record, or -1 if it doesn't fit. */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec);\n+\n+/* appends the key restarts, and compress the block if necessary. */\n+int block_writer_finish(struct block_writer *w);\n+\n+/* clears out internally allocated block_writer members. */\n+void block_writer_release(struct block_writer *bw);\n+\n+/* Read a block. */\n+struct block_reader {\n+\t/* offset of the block header; nonzero for the first block in a\n+\t * reftable. */\n+\tuint32_t header_off;\n+\n+\t/* the memory block */\n+\tstruct reftable_block block;\n+\tint hash_size;\n+\n+\t/* size of the data, excluding restart data. */\n+\tuint32_t block_len;\n+\tuint8_t *restart_bytes;\n+\tuint16_t restart_count;\n+\n+\t/* size of the data in the file. For log blocks, this is the compressed\n+\t * size. */\n+\tuint32_t full_block_size;\n+};\n+\n+/* Iterate over entries in a block */\n+struct block_iter {\n+\t/* offset within the block of the next entry to read. */\n+\tuint32_t next_off;\n+\tstruct block_reader *br;\n+\n+\t/* key for last entry we read. */\n+\tstruct strbuf last_key;\n+};\n+\n+/* initializes a block reader. */\n+int block_reader_init(struct block_reader *br, struct reftable_block *bl,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size);\n+\n+/* Position `it` at start of the block */\n+void block_reader_start(struct block_reader *br, struct block_iter *it);\n+\n+/* Position `it` to the `want` key in the block */\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want);\n+\n+/* Returns the block type (eg. 'r' for refs) */\n+uint8_t block_reader_type(struct block_reader *r);\n+\n+/* Decodes the first key in the block */\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key);\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src);\n+\n+/* return < 0 for error, 0 for OK, > 0 for EOF. */\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec);\n+\n+/* Seek to `want` with in the block pointed to by `it` */\n+int block_iter_seek(struct block_iter *it, struct strbuf *want);\n+\n+/* deallocate memory for `it`. The block reader and its block is left intact. */\n+void block_iter_close(struct block_iter *it);\n+\n+/* size of file header, depending on format version */\n+int header_size(int version);\n+\n+/* size of file footer, depending on format version */\n+int footer_size(int version);\n+\n+/* returns a block to its source. */\n+void reftable_block_done(struct reftable_block *ret);\n+\n+#endif\ndiff --git a/reftable/block_test.c b/reftable/block_test.c\nnew file mode 100644\nindex 000000000000..75fe198e6773\n--- /dev/null\n+++ b/reftable/block_test.c\n@@ -0,0 +1,121 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"system.h\"\n+\n+#include \"blocksource.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_block_read_write(void)\n+{\n+\tconst int header_off = 21; /* random */\n+\tchar *names[30];\n+\tconst int N = ARRAY_SIZE(names);\n+\tconst int block_size = 1024;\n+\tstruct reftable_block block = { NULL };\n+\tstruct block_writer bw = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_record rec = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tstruct block_reader br = { 0 };\n+\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\tint j = 0;\n+\tstruct strbuf want = STRBUF_INIT;\n+\n+\tblock.data = reftable_calloc(block_size);\n+\tblock.len = block_size;\n+\tblock.source = malloc_block_source();\n+\tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n+\t\t\t header_off, hash_size(SHA1_ID));\n+\treftable_record_from_ref(&rec, &ref);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tuint8_t hash[SHA1_SIZE];\n+\t\tsnprintf(name, sizeof(name), \"branch%02d\", i);\n+\t\tmemset(hash, i, sizeof(hash));\n+\n+\t\tref.refname = name;\n+\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\tref.value.val1 = hash;\n+\n+\t\tnames[i] = xstrdup(name);\n+\t\tn = block_writer_add(&bw, &rec);\n+\t\tref.refname = NULL;\n+\t\tref.value_type = REFTABLE_REF_DELETION;\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tn = block_writer_finish(&bw);\n+\tEXPECT(n > 0);\n+\n+\tblock_writer_release(&bw);\n+\n+\tblock_reader_init(&br, &block, header_off, block_size, SHA1_SIZE);\n+\n+\tblock_reader_start(&br, &it);\n+\n+\twhile (1) {\n+\t\tint r = block_iter_next(&it, &rec);\n+\t\tEXPECT(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tEXPECT_STREQ(names[j], ref.refname);\n+\t\tj++;\n+\t}\n+\n+\treftable_record_release(&rec);\n+\tblock_iter_close(&it);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\t\tstrbuf_reset(&want);\n+\t\tstrbuf_addstr(&want, names[i]);\n+\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tEXPECT(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tEXPECT(n == 0);\n+\n+\t\tEXPECT_STREQ(names[i], ref.refname);\n+\n+\t\twant.len--;\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tEXPECT(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tEXPECT(n == 0);\n+\t\tEXPECT_STREQ(names[10 * (i / 10)], ref.refname);\n+\n+\t\tblock_iter_close(&it);\n+\t}\n+\n+\treftable_record_release(&rec);\n+\treftable_block_done(&br.block);\n+\tstrbuf_release(&want);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+}\n+\n+int block_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_block_read_write);\n+\treturn 0;\n+}\ndiff --git a/reftable/zlib-compat.c b/reftable/zlib-compat.c\nnew file mode 100644\nindex 000000000000..3e0b0f24f1c7\n--- /dev/null\n+++ b/reftable/zlib-compat.c\n@@ -0,0 +1,92 @@\n+/* taken from zlib's uncompr.c\n+\n+ commit cacf7f1d4e3d44d871b605da3b647f07d718623f\n+ Author: Mark Adler \n+ Date: Sun Jan 15 09:18:46 2017 -0800\n+\n+ zlib 1.2.11\n+\n+*/\n+\n+/*\n+ * Copyright (C) 1995-2003, 2010, 2014, 2016 Jean-loup Gailly, Mark Adler\n+ * For conditions of distribution and use, see copyright notice in zlib.h\n+ */\n+\n+#include \"system.h\"\n+\n+/* clang-format off */\n+\n+/* ===========================================================================\n+ Decompresses the source buffer into the destination buffer. *sourceLen is\n+ the byte length of the source buffer. Upon entry, *destLen is the total size\n+ of the destination buffer, which must be large enough to hold the entire\n+ uncompressed data. (The size of the uncompressed data must have been saved\n+ previously by the compressor and transmitted to the decompressor by some\n+ mechanism outside the scope of this compression library.) Upon exit,\n+ *destLen is the size of the decompressed data and *sourceLen is the number\n+ of source bytes consumed. Upon return, source + *sourceLen points to the\n+ first unused input byte.\n+\n+ uncompress returns Z_OK if success, Z_MEM_ERROR if there was not enough\n+ memory, Z_BUF_ERROR if there was not enough room in the output buffer, or\n+ Z_DATA_ERROR if the input data was corrupted, including if the input data is\n+ an incomplete zlib stream.\n+*/\n+int ZEXPORT uncompress_return_consumed (\n+ Bytef *dest,\n+ uLongf *destLen,\n+ const Bytef *source,\n+ uLong *sourceLen) {\n+ z_stream stream;\n+ int err;\n+ const uInt max = (uInt)-1;\n+ uLong len, left;\n+ Byte buf[1]; /* for detection of incomplete stream when *destLen == 0 */\n+\n+ len = *sourceLen;\n+ if (*destLen) {\n+ left = *destLen;\n+ *destLen = 0;\n+ }\n+ else {\n+ left = 1;\n+ dest = buf;\n+ }\n+\n+ stream.next_in = (z_const Bytef *)source;\n+ stream.avail_in = 0;\n+ stream.zalloc = (alloc_func)0;\n+ stream.zfree = (free_func)0;\n+ stream.opaque = (voidpf)0;\n+\n+ err = inflateInit(&stream);\n+ if (err != Z_OK) return err;\n+\n+ stream.next_out = dest;\n+ stream.avail_out = 0;\n+\n+ do {\n+ if (stream.avail_out == 0) {\n+ stream.avail_out = left > (uLong)max ? max : (uInt)left;\n+ left -= stream.avail_out;\n+ }\n+ if (stream.avail_in == 0) {\n+ stream.avail_in = len > (uLong)max ? max : (uInt)len;\n+ len -= stream.avail_in;\n+ }\n+ err = inflate(&stream, Z_NO_FLUSH);\n+ } while (err == Z_OK);\n+\n+ *sourceLen -= len + stream.avail_in;\n+ if (dest != buf)\n+ *destLen = stream.total_out;\n+ else if (stream.total_out && err == Z_BUF_ERROR)\n+ left = 1;\n+\n+ inflateEnd(&stream);\n+ return err == Z_STREAM_END ? Z_OK :\n+ err == Z_NEED_DICT ? Z_DATA_ERROR :\n+ err == Z_BUF_ERROR && left + stream.avail_out ? Z_DATA_ERROR :\n+ err;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 09d4b83ef9b8..c9deeaf08c7a 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,6 +4,7 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n+\tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"421761","messageId":"ceddefadd48c772ea124078d59a76d634d1db1be.1618255553.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 08/20] reftable: a generic binary tree implementation","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:40Z","receivedAt":"2021-04-12T19:26:20Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format includes support for an (OID => ref) map. This map can speed\nup visibility and reachability checks. In particular, various operations along\nthe fetch/push path within Gerrit have ben sped up by using this structure.\n\nThe map is constructed with help of a binary tree. Object IDs are hashes, so\nthey are uniformly distributed. Hence, the tree does not attempt forced\nrebalancing.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 4 ++-\n reftable/tree.c | 63 ++++++++++++++++++++++++++++++++++++++++\n reftable/tree.h | 34 ++++++++++++++++++++++\n reftable/tree_test.c | 61 ++++++++++++++++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 162 insertions(+), 1 deletion(-)\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 89f998a9a5f2..5ac204eb211c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2406,12 +2406,14 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/zlib-compat.o\n \n+REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n-REFTABLE_TEST_OBJS += reftable/basics_test.o\n+REFTABLE_TEST_OBJS += reftable/tree_test.o\n \n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n \ndiff --git a/reftable/tree.c b/reftable/tree.c\nnew file mode 100644\nindex 000000000000..0061d14e306c\n--- /dev/null\n+++ b/reftable/tree.c\n@@ -0,0 +1,63 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert)\n+{\n+\tint res;\n+\tif (*rootp == NULL) {\n+\t\tif (!insert) {\n+\t\t\treturn NULL;\n+\t\t} else {\n+\t\t\tstruct tree_node *n =\n+\t\t\t\treftable_calloc(sizeof(struct tree_node));\n+\t\t\tn->key = key;\n+\t\t\t*rootp = n;\n+\t\t\treturn *rootp;\n+\t\t}\n+\t}\n+\n+\tres = compare(key, (*rootp)->key);\n+\tif (res < 0)\n+\t\treturn tree_search(key, &(*rootp)->left, compare, insert);\n+\telse if (res > 0)\n+\t\treturn tree_search(key, &(*rootp)->right, compare, insert);\n+\treturn *rootp;\n+}\n+\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg)\n+{\n+\tif (t->left != NULL) {\n+\t\tinfix_walk(t->left, action, arg);\n+\t}\n+\taction(arg, t->key);\n+\tif (t->right != NULL) {\n+\t\tinfix_walk(t->right, action, arg);\n+\t}\n+}\n+\n+void tree_free(struct tree_node *t)\n+{\n+\tif (t == NULL) {\n+\t\treturn;\n+\t}\n+\tif (t->left != NULL) {\n+\t\ttree_free(t->left);\n+\t}\n+\tif (t->right != NULL) {\n+\t\ttree_free(t->right);\n+\t}\n+\treftable_free(t);\n+}\ndiff --git a/reftable/tree.h b/reftable/tree.h\nnew file mode 100644\nindex 000000000000..fbdd002e23af\n--- /dev/null\n+++ b/reftable/tree.h\n@@ -0,0 +1,34 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TREE_H\n+#define TREE_H\n+\n+/* tree_node is a generic binary search tree. */\n+struct tree_node {\n+\tvoid *key;\n+\tstruct tree_node *left, *right;\n+};\n+\n+/* looks for `key` in `rootp` using `compare` as comparison function. If insert\n+ * is set, insert the key if it's not found. Else, return NULL.\n+ */\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert);\n+\n+/* performs an infix walk of the tree. */\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg);\n+\n+/*\n+ * deallocates the tree nodes recursively. Keys should be deallocated separately\n+ * by walking over the tree. */\n+void tree_free(struct tree_node *t);\n+\n+#endif\ndiff --git a/reftable/tree_test.c b/reftable/tree_test.c\nnew file mode 100644\nindex 000000000000..26d1e694252e\n--- /dev/null\n+++ b/reftable/tree_test.c\n@@ -0,0 +1,61 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static int test_compare(const void *a, const void *b)\n+{\n+\treturn (char *)a - (char *)b;\n+}\n+\n+struct curry {\n+\tvoid *last;\n+};\n+\n+static void check_increasing(void *arg, void *key)\n+{\n+\tstruct curry *c = (struct curry *)arg;\n+\tif (c->last != NULL) {\n+\t\tassert(test_compare(c->last, key) < 0);\n+\t}\n+\tc->last = key;\n+}\n+\n+static void test_tree(void)\n+{\n+\tstruct tree_node *root = NULL;\n+\n+\tvoid *values[11] = { NULL };\n+\tstruct tree_node *nodes[11] = { NULL };\n+\tint i = 1;\n+\tstruct curry c = { NULL };\n+\tdo {\n+\t\tnodes[i] = tree_search(values + i, &root, &test_compare, 1);\n+\t\ti = (i * 7) % 11;\n+\t} while (i != 1);\n+\n+\tfor (i = 1; i < ARRAY_SIZE(nodes); i++) {\n+\t\tassert(values + i == nodes[i]->key);\n+\t\tassert(nodes[i] ==\n+\t\t tree_search(values + i, &root, &test_compare, 0));\n+\t}\n+\n+\tinfix_walk(root, check_increasing, &c);\n+\ttree_free(root);\n+}\n+\n+int tree_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_tree);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex c9deeaf08c7a..050551fa6985 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,5 +6,6 @@ int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"421762","messageId":"9c3fd77fbbacfea732cae2c6f212ac410c2a6b16.1618255553.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 10/20] reftable: generic interface to tables","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:42Z","receivedAt":"2021-04-12T19:26:21Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/generic.h | 32 ++++++++++\n reftable/reftable-generic.h | 41 +++++++++++++\n reftable/reftable-iterator.h | 39 ++++++++++++\n reftable/reftable.c | 115 +++++++++++++++++++++++++++++++++++\n 5 files changed, 228 insertions(+)\n create mode 100644 reftable/generic.h\n create mode 100644 reftable/reftable-generic.h\n create mode 100644 reftable/reftable-iterator.h\n create mode 100644 reftable/reftable.c\n\ndiff --git a/Makefile b/Makefile\nindex 2653894d0308..1e94c2e0ac2c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2406,6 +2406,7 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/reftable.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n REFTABLE_OBJS += reftable/zlib-compat.o\ndiff --git a/reftable/generic.h b/reftable/generic.h\nnew file mode 100644\nindex 000000000000..98886a06402d\n--- /dev/null\n+++ b/reftable/generic.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef GENERIC_H\n+#define GENERIC_H\n+\n+#include \"record.h\"\n+#include \"reftable-generic.h\"\n+\n+/* generic interface to reftables */\n+struct reftable_table_vtable {\n+\tint (*seek_record)(void *tab, struct reftable_iterator *it,\n+\t\t\t struct reftable_record *);\n+\tuint32_t (*hash_id)(void *tab);\n+\tuint64_t (*min_update_index)(void *tab);\n+\tuint64_t (*max_update_index)(void *tab);\n+};\n+\n+struct reftable_iterator_vtable {\n+\tint (*next)(void *iter_arg, struct reftable_record *rec);\n+\tvoid (*close)(void *iter_arg);\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it);\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec);\n+\n+#endif\ndiff --git a/reftable/reftable-generic.h b/reftable/reftable-generic.h\nnew file mode 100644\nindex 000000000000..d019dbbc19f1\n--- /dev/null\n+++ b/reftable/reftable-generic.h\n@@ -0,0 +1,41 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_GENERIC_H\n+#define REFTABLE_GENERIC_H\n+\n+#include \"reftable-iterator.h\"\n+\n+struct reftable_table_vtable;\n+\n+/*\n+ * Provides a unified API for reading tables, either merged tables, or single\n+ * readers. */\n+struct reftable_table {\n+\tstruct reftable_table_vtable *ops;\n+\tvoid *table_arg;\n+};\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* returns the hash ID from a generic reftable_table */\n+uint32_t reftable_table_hash_id(struct reftable_table *tab);\n+\n+/* returns the max update_index covered by this table. */\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab);\n+\n+/* returns the min update_index covered by this table. */\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0\n+ for success, and 1 if ref not found. */\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+#endif\ndiff --git a/reftable/reftable-iterator.h b/reftable/reftable-iterator.h\nnew file mode 100644\nindex 000000000000..d3eee7af3571\n--- /dev/null\n+++ b/reftable/reftable-iterator.h\n@@ -0,0 +1,39 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_ITERATOR_H\n+#define REFTABLE_ITERATOR_H\n+\n+#include \"reftable-record.h\"\n+\n+struct reftable_iterator_vtable;\n+\n+/* iterator is the generic interface for walking over data stored in a\n+ * reftable.\n+ */\n+struct reftable_iterator {\n+\tstruct reftable_iterator_vtable *ops;\n+\tvoid *iter_arg;\n+};\n+\n+/* reads the next reftable_ref_record. Returns < 0 for error, 0 for OK and > 0:\n+ * end of iteration.\n+ */\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* reads the next reftable_log_record. Returns < 0 for error, 0 for OK and > 0:\n+ * end of iteration.\n+ */\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* releases resources associated with an iterator. */\n+void reftable_iterator_destroy(struct reftable_iterator *it);\n+\n+#endif\ndiff --git a/reftable/reftable.c b/reftable/reftable.c\nnew file mode 100644\nindex 000000000000..85ab8ba9f963\n--- /dev/null\n+++ b/reftable/reftable.c\n@@ -0,0 +1,115 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"generic.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_table_seek_ref(tab, &it, name);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_ref(&it, ref);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref->refname, name) ||\n+\t reftable_ref_record_is_deletion(ref)) {\n+\t\treftable_ref_record_release(ref);\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->max_update_index(tab->table_arg);\n+}\n+\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->min_update_index(tab->table_arg);\n+}\n+\n+uint32_t reftable_table_hash_id(struct reftable_table *tab)\n+{\n+\treturn tab->ops->hash_id(tab->table_arg);\n+}\n+\n+void reftable_iterator_destroy(struct reftable_iterator *it)\n+{\n+\tif (it->ops == NULL) {\n+\t\treturn;\n+\t}\n+\tit->ops->close(it->iter_arg);\n+\tit->ops = NULL;\n+\tFREE_AND_NULL(it->iter_arg);\n+}\n+\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, ref);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, log);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n+{\n+\treturn it->ops->next(it->iter_arg, rec);\n+}\n+\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n+{\n+\treturn 1;\n+}\n+\n+static void empty_iterator_close(void *arg)\n+{\n+}\n+\n+static struct reftable_iterator_vtable empty_vtable = {\n+\t.next = &empty_iterator_next,\n+\t.close = &empty_iterator_close,\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = NULL;\n+\tit->ops = &empty_vtable;\n+}\n-- \ngitgitgadget\n\n"},{"id":"421763","messageId":"1f635e6c86c5ddda465280ef5cf956b688b14aea.1618255553.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 09/20] reftable: write reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:41Z","receivedAt":"2021-04-12T19:26:23Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable-writer.h | 147 ++++++++\n reftable/writer.c | 691 +++++++++++++++++++++++++++++++++++++\n reftable/writer.h | 50 +++\n 4 files changed, 889 insertions(+)\n create mode 100644 reftable/reftable-writer.h\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n\ndiff --git a/Makefile b/Makefile\nindex 5ac204eb211c..2653894d0308 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2407,6 +2407,7 @@ REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/tree.o\n+REFTABLE_OBJS += reftable/writer.o\n REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nnew file mode 100644\nindex 000000000000..9d2f8d605558\n--- /dev/null\n+++ b/reftable/reftable-writer.h\n@@ -0,0 +1,147 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_WRITER_H\n+#define REFTABLE_WRITER_H\n+\n+#include \n+\n+#include \"reftable-record.h\"\n+\n+/* Writing single reftables */\n+\n+/* reftable_write_options sets options for writing a single reftable. */\n+struct reftable_write_options {\n+\t/* boolean: do not pad out blocks to block size. */\n+\tunsigned unpadded : 1;\n+\n+\t/* the blocksize. Should be less than 2^24. */\n+\tuint32_t block_size;\n+\n+\t/* boolean: do not generate a SHA1 => ref index. */\n+\tunsigned skip_index_objects : 1;\n+\n+\t/* how often to write complete keys in each block. */\n+\tint restart_interval;\n+\n+\t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n+\t * Defaults to SHA1 if unset\n+\t */\n+\tuint32_t hash_id;\n+\n+\t/* boolean: do not check ref names for validity or dir/file conflicts.\n+\t */\n+\tunsigned skip_name_check : 1;\n+\n+\t/* boolean: copy log messages exactly. If unset, check that the message\n+\t * is a single line, and add '\\n' if missing.\n+\t */\n+\tunsigned exact_log_message : 1;\n+};\n+\n+/* reftable_block_stats holds statistics for a single block type */\n+struct reftable_block_stats {\n+\t/* total number of entries written */\n+\tint entries;\n+\t/* total number of key restarts */\n+\tint restarts;\n+\t/* total number of blocks */\n+\tint blocks;\n+\t/* total number of index blocks */\n+\tint index_blocks;\n+\t/* depth of the index */\n+\tint max_index_level;\n+\n+\t/* offset of the first block for this type */\n+\tuint64_t offset;\n+\t/* offset of the top level index block for this type, or 0 if not\n+\t * present */\n+\tuint64_t index_offset;\n+};\n+\n+/* stats holds overall statistics for a single reftable */\n+struct reftable_stats {\n+\t/* total number of blocks written. */\n+\tint blocks;\n+\t/* stats for ref data */\n+\tstruct reftable_block_stats ref_stats;\n+\t/* stats for the SHA1 to ref map. */\n+\tstruct reftable_block_stats obj_stats;\n+\t/* stats for index blocks */\n+\tstruct reftable_block_stats idx_stats;\n+\t/* stats for log blocks */\n+\tstruct reftable_block_stats log_stats;\n+\n+\t/* disambiguation length of shortened object IDs. */\n+\tint object_id_len;\n+};\n+\n+/* reftable_new_writer creates a new writer */\n+struct reftable_writer *\n+reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts);\n+\n+/* Set the range of update indices for the records we will add. When writing a\n+ table into a stack, the min should be at least\n+ reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned.\n+\n+ For transactional updates to a stack, typically min==max, and the\n+ update_index can be obtained by inspeciting the stack. When converting an\n+ existing ref database into a single reftable, this would be a range of\n+ update-index timestamps.\n+ */\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max);\n+\n+/*\n+ Add a reftable_ref_record. The record should have names that come after\n+ already added records.\n+\n+ The update_index must be within the limits set by\n+ reftable_writer_set_limits(), or REFTABLE_API_ERROR is returned. It is an\n+ REFTABLE_API_ERROR error to write a ref record after a log record.\n+*/\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/*\n+ Convenience function to add multiple reftable_ref_records; the function sorts\n+ the records before adding them, reordering the records array passed in.\n+*/\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n);\n+\n+/*\n+ adds reftable_log_records. Log records are keyed by (refname, decreasing\n+ update_index). The key for the record added must come after the already added\n+ log records.\n+*/\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log);\n+\n+/*\n+ Convenience function to add multiple reftable_log_records; the function sorts\n+ the records before adding them, reordering records array passed in.\n+*/\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n);\n+\n+/* reftable_writer_close finalizes the reftable. The writer is retained so\n+ * statistics can be inspected. */\n+int reftable_writer_close(struct reftable_writer *w);\n+\n+/* writer_stats returns the statistics on the reftable being written.\n+\n+ This struct becomes invalid when the writer is freed.\n+ */\n+const struct reftable_stats *writer_stats(struct reftable_writer *w);\n+\n+/* reftable_writer_free deallocates memory for the writer */\n+void reftable_writer_free(struct reftable_writer *w);\n+\n+#endif\ndiff --git a/reftable/writer.c b/reftable/writer.c\nnew file mode 100644\nindex 000000000000..d42ca8afac11\n--- /dev/null\n+++ b/reftable/writer.c\n@@ -0,0 +1,691 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"writer.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"tree.h\"\n+#include \"reftable-error.h\"\n+\n+/* finishes a block, and writes it to storage */\n+static int writer_flush_block(struct reftable_writer *w);\n+\n+/* deallocates memory related to the index */\n+static void writer_clear_index(struct reftable_writer *w);\n+\n+/* finishes writing a 'r' (refs) or 'g' (reflogs) section */\n+static int writer_finish_public_section(struct reftable_writer *w);\n+\n+static struct reftable_block_stats *\n+writer_reftable_block_stats(struct reftable_writer *w, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase 'r':\n+\t\treturn &w->stats.ref_stats;\n+\tcase 'o':\n+\t\treturn &w->stats.obj_stats;\n+\tcase 'i':\n+\t\treturn &w->stats.idx_stats;\n+\tcase 'g':\n+\t\treturn &w->stats.log_stats;\n+\t}\n+\tabort();\n+\treturn NULL;\n+}\n+\n+/* write data, queuing the padding for the next write. Returns negative for\n+ * error. */\n+static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n+\t\t\tint padding)\n+{\n+\tint n = 0;\n+\tif (w->pending_padding > 0) {\n+\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding);\n+\t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n+\t\tif (n < 0)\n+\t\t\treturn n;\n+\n+\t\tw->pending_padding = 0;\n+\t\treftable_free(zeroed);\n+\t}\n+\n+\tw->pending_padding = padding;\n+\tn = w->write(w->write_arg, data, len);\n+\tif (n < 0)\n+\t\treturn n;\n+\tn += padding;\n+\treturn 0;\n+}\n+\n+static void options_set_defaults(struct reftable_write_options *opts)\n+{\n+\tif (opts->restart_interval == 0) {\n+\t\topts->restart_interval = 16;\n+\t}\n+\n+\tif (opts->hash_id == 0) {\n+\t\topts->hash_id = SHA1_ID;\n+\t}\n+\tif (opts->block_size == 0) {\n+\t\topts->block_size = DEFAULT_BLOCK_SIZE;\n+\t}\n+}\n+\n+static int writer_version(struct reftable_writer *w)\n+{\n+\treturn (w->opts.hash_id == 0 || w->opts.hash_id == SHA1_ID) ? 1 : 2;\n+}\n+\n+static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n+{\n+\tmemcpy((char *)dest, \"REFT\", 4);\n+\n+\tdest[4] = writer_version(w);\n+\n+\tput_be24(dest + 5, w->opts.block_size);\n+\tput_be64(dest + 8, w->min_update_index);\n+\tput_be64(dest + 16, w->max_update_index);\n+\tif (writer_version(w) == 2) {\n+\t\tput_be32(dest + 24, w->opts.hash_id);\n+\t}\n+\treturn header_size(writer_version(w));\n+}\n+\n+static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n+{\n+\tint block_start = 0;\n+\tif (w->next == 0) {\n+\t\tblock_start = header_size(writer_version(w));\n+\t}\n+\n+\tstrbuf_release(&w->last_key);\n+\tblock_writer_init(&w->block_writer_data, typ, w->block,\n+\t\t\t w->opts.block_size, block_start,\n+\t\t\t hash_size(w->opts.hash_id));\n+\tw->block_writer = &w->block_writer_data;\n+\tw->block_writer->restart_interval = w->opts.restart_interval;\n+}\n+\n+static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n+\n+struct reftable_writer *\n+reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts)\n+{\n+\tstruct reftable_writer *wp =\n+\t\treftable_calloc(sizeof(struct reftable_writer));\n+\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n+\toptions_set_defaults(opts);\n+\tif (opts->block_size >= (1 << 24)) {\n+\t\t/* TODO - error return? */\n+\t\tabort();\n+\t}\n+\twp->last_key = reftable_empty_strbuf;\n+\twp->block = reftable_calloc(opts->block_size);\n+\twp->write = writer_func;\n+\twp->write_arg = writer_arg;\n+\twp->opts = *opts;\n+\twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n+\n+\treturn wp;\n+}\n+\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max)\n+{\n+\tw->min_update_index = min;\n+\tw->max_update_index = max;\n+}\n+\n+void reftable_writer_free(struct reftable_writer *w)\n+{\n+\treftable_free(w->block);\n+\treftable_free(w);\n+}\n+\n+struct obj_index_tree_node {\n+\tstruct strbuf hash;\n+\tuint64_t *offsets;\n+\tsize_t offset_len;\n+\tsize_t offset_cap;\n+};\n+\n+#define OBJ_INDEX_TREE_NODE_INIT \\\n+\t{ \\\n+\t\t.hash = STRBUF_INIT \\\n+\t}\n+\n+static int obj_index_tree_node_compare(const void *a, const void *b)\n+{\n+\treturn strbuf_cmp(&((const struct obj_index_tree_node *)a)->hash,\n+\t\t\t &((const struct obj_index_tree_node *)b)->hash);\n+}\n+\n+static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n+{\n+\tuint64_t off = w->next;\n+\n+\tstruct obj_index_tree_node want = { .hash = *hash };\n+\n+\tstruct tree_node *node = tree_search(&want, &w->obj_index_tree,\n+\t\t\t\t\t &obj_index_tree_node_compare, 0);\n+\tstruct obj_index_tree_node *key = NULL;\n+\tif (node == NULL) {\n+\t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n+\t\tkey = reftable_malloc(sizeof(struct obj_index_tree_node));\n+\t\t*key = empty;\n+\n+\t\tstrbuf_reset(&key->hash);\n+\t\tstrbuf_addbuf(&key->hash, hash);\n+\t\ttree_search((void *)key, &w->obj_index_tree,\n+\t\t\t &obj_index_tree_node_compare, 1);\n+\t} else {\n+\t\tkey = node->key;\n+\t}\n+\n+\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off) {\n+\t\treturn;\n+\t}\n+\n+\tif (key->offset_len == key->offset_cap) {\n+\t\tkey->offset_cap = 2 * key->offset_cap + 1;\n+\t\tkey->offsets = reftable_realloc(\n+\t\t\tkey->offsets, sizeof(uint64_t) * key->offset_cap);\n+\t}\n+\n+\tkey->offsets[key->offset_len++] = off;\n+}\n+\n+static int writer_add_record(struct reftable_writer *w,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tint result = -1;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = 0;\n+\treftable_record_key(rec, &key);\n+\tif (strbuf_cmp(&w->last_key, &key) >= 0)\n+\t\tgoto done;\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, &key);\n+\tif (w->block_writer == NULL) {\n+\t\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\t}\n+\n+\tassert(block_writer_type(w->block_writer) == reftable_record_type(rec));\n+\n+\tif (block_writer_add(w->block_writer, rec) == 0) {\n+\t\tresult = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0) {\n+\t\tresult = err;\n+\t\tgoto done;\n+\t}\n+\n+\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\terr = block_writer_add(w->block_writer, rec);\n+\tif (err < 0) {\n+\t\tresult = err;\n+\t\tgoto done;\n+\t}\n+\n+\tresult = 0;\n+done:\n+\tstrbuf_release(&key);\n+\treturn result;\n+}\n+\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_ref_record copy = *ref;\n+\tint err = 0;\n+\n+\tif (ref->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\tif (ref->update_index < w->min_update_index ||\n+\t ref->update_index > w->max_update_index)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\treftable_record_from_ref(&rec, ©);\n+\tcopy.update_index -= w->min_update_index;\n+\n+\terr = writer_add_record(w, &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (!w->opts.skip_index_objects &&\n+\t reftable_ref_record_val1(ref) != NULL) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, (char *)reftable_ref_record_val1(ref),\n+\t\t\t hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\n+\tif (!w->opts.skip_index_objects &&\n+\t reftable_ref_record_val2(ref) != NULL) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, reftable_ref_record_val2(ref),\n+\t\t\t hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(refs, n, reftable_ref_record_compare_name);\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_ref(w, &refs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+static int reftable_writer_add_log_verbatim(struct reftable_writer *w,\n+\t\t\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tif (w->block_writer != NULL &&\n+\t block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {\n+\t\tint err = writer_finish_public_section(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tw->next -= w->pending_padding;\n+\tw->pending_padding = 0;\n+\n+\treftable_record_from_log(&rec, log);\n+\treturn writer_add_record(w, &rec);\n+}\n+\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tchar *input_log_message = NULL;\n+\tstruct strbuf cleaned_message = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tif (log->value_type == REFTABLE_LOG_DELETION)\n+\t\treturn reftable_writer_add_log_verbatim(w, log);\n+\n+\tif (log->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\tinput_log_message = log->update.message;\n+\tif (!w->opts.exact_log_message && log->update.message != NULL) {\n+\t\tstrbuf_addstr(&cleaned_message, log->update.message);\n+\t\twhile (cleaned_message.len &&\n+\t\t cleaned_message.buf[cleaned_message.len - 1] == '\\n')\n+\t\t\tstrbuf_setlen(&cleaned_message,\n+\t\t\t\t cleaned_message.len - 1);\n+\t\tif (strchr(cleaned_message.buf, '\\n')) {\n+\t\t\t// multiple lines not allowed.\n+\t\t\terr = REFTABLE_API_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tstrbuf_addstr(&cleaned_message, \"\\n\");\n+\t\tlog->update.message = cleaned_message.buf;\n+\t}\n+\n+\terr = reftable_writer_add_log_verbatim(w, log);\n+\tlog->update.message = input_log_message;\n+done:\n+\tstrbuf_release(&cleaned_message);\n+\treturn err;\n+}\n+\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(logs, n, reftable_log_record_compare_key);\n+\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_log(w, &logs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+static int writer_finish_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tuint64_t index_start = 0;\n+\tint max_level = 0;\n+\tint threshold = w->opts.unpadded ? 1 : 3;\n+\tint before_blocks = w->stats.idx_stats.blocks;\n+\tint err = writer_flush_block(w);\n+\tint i = 0;\n+\tstruct reftable_block_stats *bstats = NULL;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\twhile (w->index_len > threshold) {\n+\t\tstruct reftable_index_record *idx = NULL;\n+\t\tint idx_len = 0;\n+\n+\t\tmax_level++;\n+\t\tindex_start = w->next;\n+\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\tidx = w->index;\n+\t\tidx_len = w->index_len;\n+\n+\t\tw->index = NULL;\n+\t\tw->index_len = 0;\n+\t\tw->index_cap = 0;\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstruct reftable_record rec = { NULL };\n+\t\t\treftable_record_from_index(&rec, idx + i);\n+\t\t\tif (block_writer_add(w->block_writer, &rec) == 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\terr = writer_flush_block(w);\n+\t\t\tif (err < 0)\n+\t\t\t\treturn err;\n+\n+\t\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\t\terr = block_writer_add(w->block_writer, &rec);\n+\t\t\tif (err != 0) {\n+\t\t\t\t/* write into fresh block should always succeed\n+\t\t\t\t */\n+\t\t\t\tabort();\n+\t\t\t}\n+\t\t}\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstrbuf_release(&idx[i].last_key);\n+\t\t}\n+\t\treftable_free(idx);\n+\t}\n+\n+\twriter_clear_index(w);\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tbstats = writer_reftable_block_stats(w, typ);\n+\tbstats->index_blocks = w->stats.idx_stats.blocks - before_blocks;\n+\tbstats->index_offset = index_start;\n+\tbstats->max_index_level = max_level;\n+\n+\t/* Reinit lastKey, as the next section can start with any key. */\n+\tw->last_key.len = 0;\n+\n+\treturn 0;\n+}\n+\n+struct common_prefix_arg {\n+\tstruct strbuf *last;\n+\tint max;\n+};\n+\n+static void update_common(void *void_arg, void *key)\n+{\n+\tstruct common_prefix_arg *arg = (struct common_prefix_arg *)void_arg;\n+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n+\tif (arg->last != NULL) {\n+\t\tint n = common_prefix_size(&entry->hash, arg->last);\n+\t\tif (n > arg->max) {\n+\t\t\targ->max = n;\n+\t\t}\n+\t}\n+\targ->last = &entry->hash;\n+}\n+\n+struct write_record_arg {\n+\tstruct reftable_writer *w;\n+\tint err;\n+};\n+\n+static void write_object_record(void *void_arg, void *key)\n+{\n+\tstruct write_record_arg *arg = (struct write_record_arg *)void_arg;\n+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n+\tstruct reftable_obj_record obj_rec = {\n+\t\t.hash_prefix = (uint8_t *)entry->hash.buf,\n+\t\t.hash_prefix_len = arg->w->stats.object_id_len,\n+\t\t.offsets = entry->offsets,\n+\t\t.offset_len = entry->offset_len,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\treftable_record_from_obj(&rec, &obj_rec);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\n+\targ->err = writer_flush_block(arg->w);\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\twriter_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\tobj_rec.offset_len = 0;\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\n+\t/* Should be able to write into a fresh block. */\n+\tassert(arg->err == 0);\n+\n+done:;\n+}\n+\n+static void object_record_free(void *void_arg, void *key)\n+{\n+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n+\n+\tFREE_AND_NULL(entry->offsets);\n+\tstrbuf_release(&entry->hash);\n+\treftable_free(entry);\n+}\n+\n+static int writer_dump_object_index(struct reftable_writer *w)\n+{\n+\tstruct write_record_arg closure = { .w = w };\n+\tstruct common_prefix_arg common = { NULL };\n+\tif (w->obj_index_tree != NULL) {\n+\t\tinfix_walk(w->obj_index_tree, &update_common, &common);\n+\t}\n+\tw->stats.object_id_len = common.max + 1;\n+\n+\twriter_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\n+\tif (w->obj_index_tree != NULL) {\n+\t\tinfix_walk(w->obj_index_tree, &write_object_record, &closure);\n+\t}\n+\n+\tif (closure.err < 0)\n+\t\treturn closure.err;\n+\treturn writer_finish_section(w);\n+}\n+\n+static int writer_finish_public_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = 0;\n+\tint err = 0;\n+\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\n+\ttyp = block_writer_type(w->block_writer);\n+\terr = writer_finish_section(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\tif (typ == BLOCK_TYPE_REF && !w->opts.skip_index_objects &&\n+\t w->stats.ref_stats.index_blocks > 0) {\n+\t\terr = writer_dump_object_index(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tif (w->obj_index_tree != NULL) {\n+\t\tinfix_walk(w->obj_index_tree, &object_record_free, NULL);\n+\t\ttree_free(w->obj_index_tree);\n+\t\tw->obj_index_tree = NULL;\n+\t}\n+\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+int reftable_writer_close(struct reftable_writer *w)\n+{\n+\tuint8_t footer[72];\n+\tuint8_t *p = footer;\n+\tint err = writer_finish_public_section(w);\n+\tint empty_table = w->next == 0;\n+\tif (err != 0)\n+\t\tgoto done;\n+\tw->pending_padding = 0;\n+\tif (empty_table) {\n+\t\t/* Empty tables need a header anyway. */\n+\t\tuint8_t header[28];\n+\t\tint n = writer_write_header(w, header);\n+\t\terr = padded_write(w, header, n, 0);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\tp += writer_write_header(w, footer);\n+\tput_be64(p, w->stats.ref_stats.index_offset);\n+\tp += 8;\n+\tput_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n+\tp += 8;\n+\tput_be64(p, w->stats.obj_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be64(p, w->stats.log_stats.offset);\n+\tp += 8;\n+\tput_be64(p, w->stats.log_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be32(p, crc32(0, footer, p - footer));\n+\tp += 4;\n+\n+\terr = padded_write(w, footer, footer_size(writer_version(w)), 0);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (empty_table) {\n+\t\terr = REFTABLE_EMPTY_TABLE_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\t/* free up memory. */\n+\tblock_writer_release(&w->block_writer_data);\n+\twriter_clear_index(w);\n+\tstrbuf_release(&w->last_key);\n+\treturn err;\n+}\n+\n+static void writer_clear_index(struct reftable_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->index_len; i++) {\n+\t\tstrbuf_release(&w->index[i].last_key);\n+\t}\n+\n+\tFREE_AND_NULL(w->index);\n+\tw->index_len = 0;\n+\tw->index_cap = 0;\n+}\n+\n+static const int debug = 0;\n+\n+static int writer_flush_nonempty_block(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tstruct reftable_block_stats *bstats =\n+\t\twriter_reftable_block_stats(w, typ);\n+\tuint64_t block_typ_off = (bstats->blocks == 0) ? w->next : 0;\n+\tint raw_bytes = block_writer_finish(w->block_writer);\n+\tint padding = 0;\n+\tint err = 0;\n+\tstruct reftable_index_record ir = { .last_key = STRBUF_INIT };\n+\tif (raw_bytes < 0)\n+\t\treturn raw_bytes;\n+\n+\tif (!w->opts.unpadded && typ != BLOCK_TYPE_LOG) {\n+\t\tpadding = w->opts.block_size - raw_bytes;\n+\t}\n+\n+\tif (block_typ_off > 0) {\n+\t\tbstats->offset = block_typ_off;\n+\t}\n+\n+\tbstats->entries += w->block_writer->entries;\n+\tbstats->restarts += w->block_writer->restart_len;\n+\tbstats->blocks++;\n+\tw->stats.blocks++;\n+\n+\tif (debug) {\n+\t\tfprintf(stderr, \"block %c off %\" PRIu64 \" sz %d (%d)\\n\", typ,\n+\t\t\tw->next, raw_bytes,\n+\t\t\tget_be24(w->block + w->block_writer->header_off + 1));\n+\t}\n+\n+\tif (w->next == 0) {\n+\t\twriter_write_header(w, w->block);\n+\t}\n+\n+\terr = padded_write(w, w->block, raw_bytes, padding);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (w->index_cap == w->index_len) {\n+\t\tw->index_cap = 2 * w->index_cap + 1;\n+\t\tw->index = reftable_realloc(\n+\t\t\tw->index,\n+\t\t\tsizeof(struct reftable_index_record) * w->index_cap);\n+\t}\n+\n+\tir.offset = w->next;\n+\tstrbuf_reset(&ir.last_key);\n+\tstrbuf_addbuf(&ir.last_key, &w->block_writer->last_key);\n+\tw->index[w->index_len] = ir;\n+\n+\tw->index_len++;\n+\tw->next += padding + raw_bytes;\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+static int writer_flush_block(struct reftable_writer *w)\n+{\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\tif (w->block_writer->entries == 0)\n+\t\treturn 0;\n+\treturn writer_flush_nonempty_block(w);\n+}\n+\n+const struct reftable_stats *writer_stats(struct reftable_writer *w)\n+{\n+\treturn &w->stats;\n+}\ndiff --git a/reftable/writer.h b/reftable/writer.h\nnew file mode 100644\nindex 000000000000..4921c249d06b\n--- /dev/null\n+++ b/reftable/writer.h\n@@ -0,0 +1,50 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef WRITER_H\n+#define WRITER_H\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"tree.h\"\n+#include \"reftable-writer.h\"\n+\n+struct reftable_writer {\n+\tint (*write)(void *, const void *, size_t);\n+\tvoid *write_arg;\n+\tint pending_padding;\n+\tstruct strbuf last_key;\n+\n+\t/* offset of next block to write. */\n+\tuint64_t next;\n+\tuint64_t min_update_index, max_update_index;\n+\tstruct reftable_write_options opts;\n+\n+\t/* memory buffer for writing */\n+\tuint8_t *block;\n+\n+\t/* writer for the current section. NULL or points to\n+\t * block_writer_data */\n+\tstruct block_writer *block_writer;\n+\n+\tstruct block_writer block_writer_data;\n+\n+\t/* pending index records for the current section */\n+\tstruct reftable_index_record *index;\n+\tsize_t index_len;\n+\tsize_t index_cap;\n+\n+\t/*\n+\t * tree for use with tsearch; used to populate the 'o' inverse OID\n+\t * map */\n+\tstruct tree_node *obj_index_tree;\n+\n+\tstruct reftable_stats stats;\n+};\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"421764","messageId":"8ba486f44a72fa2cb6d3617dfff4a3c2ec9c09a2.1618255553.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 11/20] reftable: read reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:43Z","receivedAt":"2021-04-12T19:26:26Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis supports reading a single reftable file.\n\nThe commit introduces an abstract iterator type, which captures the usecases\nboth of reading individual refs, and iterating over a segment of the ref\nnamespace.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/iter.c | 197 ++++++++++\n reftable/iter.h | 67 ++++\n reftable/reader.c | 773 +++++++++++++++++++++++++++++++++++++\n reftable/reader.h | 66 ++++\n reftable/reftable-reader.h | 98 +++++\n 6 files changed, 1203 insertions(+)\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/reftable-reader.h\n\ndiff --git a/Makefile b/Makefile\nindex 1e94c2e0ac2c..f8167084b80a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2404,7 +2404,9 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n+REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/reftable.o\n REFTABLE_OBJS += reftable/tree.o\ndiff --git a/reftable/iter.c b/reftable/iter.c\nnew file mode 100644\nindex 000000000000..de1b3b898f61\n--- /dev/null\n+++ b/reftable/iter.c\n@@ -0,0 +1,197 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"iter.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"generic.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"reftable-error.h\"\n+\n+int iterator_is_null(struct reftable_iterator *it)\n+{\n+\treturn it->ops == NULL;\n+}\n+\n+static void filtering_ref_iterator_close(void *iter_arg)\n+{\n+\tstruct filtering_ref_iterator *fri =\n+\t\t(struct filtering_ref_iterator *)iter_arg;\n+\tstrbuf_release(&fri->oid);\n+\treftable_iterator_destroy(&fri->it);\n+}\n+\n+static int filtering_ref_iterator_next(void *iter_arg,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct filtering_ref_iterator *fri =\n+\t\t(struct filtering_ref_iterator *)iter_arg;\n+\tstruct reftable_ref_record *ref =\n+\t\t(struct reftable_ref_record *)rec->data;\n+\tint err = 0;\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&fri->it, ref);\n+\t\tif (err != 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (fri->double_check) {\n+\t\t\tstruct reftable_iterator it = { NULL };\n+\n+\t\t\terr = reftable_table_seek_ref(&fri->tab, &it,\n+\t\t\t\t\t\t ref->refname);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = reftable_iterator_next_ref(&it, ref);\n+\t\t\t}\n+\n+\t\t\treftable_iterator_destroy(&it);\n+\n+\t\t\tif (err < 0) {\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (err > 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (ref->value_type == REFTABLE_REF_VAL2 &&\n+\t\t (!memcmp(fri->oid.buf, ref->value.val2.target_value,\n+\t\t\t fri->oid.len) ||\n+\t\t !memcmp(fri->oid.buf, ref->value.val2.value,\n+\t\t\t fri->oid.len)))\n+\t\t\treturn 0;\n+\n+\t\tif (ref->value_type == REFTABLE_REF_VAL1 &&\n+\t\t !memcmp(fri->oid.buf, ref->value.val1, fri->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treftable_ref_record_release(ref);\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable filtering_ref_iterator_vtable = {\n+\t.next = &filtering_ref_iterator_next,\n+\t.close = &filtering_ref_iterator_close,\n+};\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *it,\n+\t\t\t\t\t struct filtering_ref_iterator *fri)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = fri;\n+\tit->ops = &filtering_ref_iterator_vtable;\n+}\n+\n+static void indexed_table_ref_iter_close(void *p)\n+{\n+\tstruct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p;\n+\tblock_iter_close(&it->cur);\n+\treftable_block_done(&it->block_reader.block);\n+\tstrbuf_release(&it->oid);\n+}\n+\n+static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n+{\n+\tuint64_t off;\n+\tint err = 0;\n+\tif (it->offset_idx == it->offset_len) {\n+\t\tit->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\n+\treftable_block_done(&it->block_reader.block);\n+\n+\toff = it->offsets[it->offset_idx++];\n+\terr = reader_init_block_reader(it->r, &it->block_reader, off,\n+\t\t\t\t BLOCK_TYPE_REF);\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\tif (err > 0) {\n+\t\t/* indexed block does not exist. */\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\t}\n+\tblock_reader_start(&it->block_reader, &it->cur);\n+\treturn 0;\n+}\n+\n+static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\n+{\n+\tstruct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p;\n+\tstruct reftable_ref_record *ref =\n+\t\t(struct reftable_ref_record *)rec->data;\n+\n+\twhile (1) {\n+\t\tint err = block_iter_next(&it->cur, rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\terr = indexed_table_ref_iter_next_block(it);\n+\t\t\tif (err < 0) {\n+\t\t\t\treturn err;\n+\t\t\t}\n+\n+\t\t\tif (it->is_finished) {\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\t\t/* BUG */\n+\t\tif (!memcmp(it->oid.buf, ref->value.val2.target_value,\n+\t\t\t it->oid.len) ||\n+\t\t !memcmp(it->oid.buf, ref->value.val2.value, it->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+}\n+\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len)\n+{\n+\tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n+\tstruct indexed_table_ref_iter *itr =\n+\t\treftable_calloc(sizeof(struct indexed_table_ref_iter));\n+\tint err = 0;\n+\n+\t*itr = empty;\n+\titr->r = r;\n+\tstrbuf_add(&itr->oid, oid, oid_len);\n+\n+\titr->offsets = offsets;\n+\titr->offset_len = offset_len;\n+\n+\terr = indexed_table_ref_iter_next_block(itr);\n+\tif (err < 0) {\n+\t\treftable_free(itr);\n+\t} else {\n+\t\t*dest = itr;\n+\t}\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable indexed_table_ref_iter_vtable = {\n+\t.next = &indexed_table_ref_iter_next,\n+\t.close = &indexed_table_ref_iter_close,\n+};\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = itr;\n+\tit->ops = &indexed_table_ref_iter_vtable;\n+}\ndiff --git a/reftable/iter.h b/reftable/iter.h\nnew file mode 100644\nindex 000000000000..656ba3053e9f\n--- /dev/null\n+++ b/reftable/iter.h\n@@ -0,0 +1,67 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef ITER_H\n+#define ITER_H\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"record.h\"\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+/* Returns true for a zeroed out iterator, such as the one returned from\n+ * iterator_destroy. */\n+int iterator_is_null(struct reftable_iterator *it);\n+\n+/* iterator that produces only ref records that point to `oid` */\n+struct filtering_ref_iterator {\n+\tint double_check;\n+\tstruct reftable_table tab;\n+\tstruct strbuf oid;\n+\tstruct reftable_iterator it;\n+};\n+#define FILTERING_REF_ITERATOR_INIT \\\n+\t{ \\\n+\t\t.oid = STRBUF_INIT \\\n+\t}\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *,\n+\t\t\t\t\t struct filtering_ref_iterator *);\n+\n+/* iterator that produces only ref records that point to `oid`,\n+ * but using the object index.\n+ */\n+struct indexed_table_ref_iter {\n+\tstruct reftable_reader *r;\n+\tstruct strbuf oid;\n+\n+\t/* mutable */\n+\tuint64_t *offsets;\n+\n+\t/* Points to the next offset to read. */\n+\tint offset_idx;\n+\tint offset_len;\n+\tstruct block_reader block_reader;\n+\tstruct block_iter cur;\n+\tint is_finished;\n+};\n+\n+#define INDEXED_TABLE_REF_ITER_INIT \\\n+\t{ \\\n+\t\t.cur = { .last_key = STRBUF_INIT }, .oid = STRBUF_INIT, \\\n+\t}\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr);\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len);\n+\n+#endif\ndiff --git a/reftable/reader.c b/reftable/reader.c\nnew file mode 100644\nindex 000000000000..36d7be01a3b3\n--- /dev/null\n+++ b/reftable/reader.c\n@@ -0,0 +1,773 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reader.h\"\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"generic.h\"\n+#include \"iter.h\"\n+#include \"record.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-generic.h\"\n+#include \"tree.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source)\n+{\n+\treturn source->ops->size(source->arg);\n+}\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tint result = source->ops->read_block(source->arg, dest, off, size);\n+\tdest->source = *source;\n+\treturn result;\n+}\n+\n+void block_source_close(struct reftable_block_source *source)\n+{\n+\tif (source->ops == NULL) {\n+\t\treturn;\n+\t}\n+\n+\tsource->ops->close(source->arg);\n+\tsource->ops = NULL;\n+}\n+\n+static struct reftable_reader_offsets *\n+reader_offsets_for(struct reftable_reader *r, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\t\treturn &r->ref_offsets;\n+\tcase BLOCK_TYPE_LOG:\n+\t\treturn &r->log_offsets;\n+\tcase BLOCK_TYPE_OBJ:\n+\t\treturn &r->obj_offsets;\n+\t}\n+\tabort();\n+}\n+\n+static int reader_get_block(struct reftable_reader *r,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t sz)\n+{\n+\tif (off >= r->size)\n+\t\treturn 0;\n+\n+\tif (off + sz > r->size) {\n+\t\tsz = r->size - off;\n+\t}\n+\n+\treturn block_source_read_block(&r->source, dest, off, sz);\n+}\n+\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r)\n+{\n+\treturn r->hash_id;\n+}\n+\n+const char *reader_name(struct reftable_reader *r)\n+{\n+\treturn r->name;\n+}\n+\n+static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n+\t\t\tuint8_t *header)\n+{\n+\tuint8_t *f = footer;\n+\tuint8_t first_block_typ;\n+\tint err = 0;\n+\tuint32_t computed_crc;\n+\tuint32_t file_crc;\n+\n+\tif (memcmp(f, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tf += 4;\n+\n+\tif (memcmp(footer, header, header_size(r->version))) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tf++;\n+\tr->block_size = get_be24(f);\n+\n+\tf += 3;\n+\tr->min_update_index = get_be64(f);\n+\tf += 8;\n+\tr->max_update_index = get_be64(f);\n+\tf += 8;\n+\n+\tif (r->version == 1) {\n+\t\tr->hash_id = SHA1_ID;\n+\t} else {\n+\t\tr->hash_id = get_be32(f);\n+\t\tswitch (r->hash_id) {\n+\t\tcase SHA1_ID:\n+\t\t\tbreak;\n+\t\tcase SHA256_ID:\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tf += 4;\n+\t}\n+\n+\tr->ref_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->obj_offsets.offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1);\n+\tr->obj_offsets.offset >>= 5;\n+\n+\tr->obj_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tcomputed_crc = crc32(0, footer, f - footer);\n+\tfile_crc = get_be32(f);\n+\tf += 4;\n+\tif (computed_crc != file_crc) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tfirst_block_typ = header[header_size(r->version)];\n+\tr->ref_offsets.is_present = (first_block_typ == BLOCK_TYPE_REF);\n+\tr->ref_offsets.offset = 0;\n+\tr->log_offsets.is_present = (first_block_typ == BLOCK_TYPE_LOG ||\n+\t\t\t\t r->log_offsets.offset > 0);\n+\tr->obj_offsets.is_present = r->obj_offsets.offset > 0;\n+\terr = 0;\n+done:\n+\treturn err;\n+}\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name)\n+{\n+\tstruct reftable_block footer = { NULL };\n+\tstruct reftable_block header = { NULL };\n+\tint err = 0;\n+\n+\tmemset(r, 0, sizeof(struct reftable_reader));\n+\n+\t/* Need +1 to read type of first block. */\n+\terr = block_source_read_block(source, &header, 0, header_size(2) + 1);\n+\tif (err != header_size(2) + 1) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tif (memcmp(header.data, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tr->version = header.data[4];\n+\tif (r->version != 1 && r->version != 2) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tr->size = block_source_size(source) - footer_size(r->version);\n+\tr->source = *source;\n+\tr->name = xstrdup(name);\n+\tr->hash_id = 0;\n+\n+\terr = block_source_read_block(source, &footer, r->size,\n+\t\t\t\t footer_size(r->version));\n+\tif (err != footer_size(r->version)) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = parse_footer(r, footer.data, header.data);\n+done:\n+\treftable_block_done(&footer);\n+\treftable_block_done(&header);\n+\treturn err;\n+}\n+\n+struct table_iter {\n+\tstruct reftable_reader *r;\n+\tuint8_t typ;\n+\tuint64_t block_off;\n+\tstruct block_iter bi;\n+\tint is_finished;\n+};\n+#define TABLE_ITER_INIT \\\n+\t{ \\\n+\t\t.bi = {.last_key = STRBUF_INIT } \\\n+\t}\n+\n+static void table_iter_copy_from(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = src->block_off;\n+\tdest->is_finished = src->is_finished;\n+\tblock_iter_copy_from(&dest->bi, &src->bi);\n+}\n+\n+static int table_iter_next_in_block(struct table_iter *ti,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tint res = block_iter_next(&ti->bi, rec);\n+\tif (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {\n+\t\t((struct reftable_ref_record *)rec->data)->update_index +=\n+\t\t\tti->r->min_update_index;\n+\t}\n+\n+\treturn res;\n+}\n+\n+static void table_iter_block_done(struct table_iter *ti)\n+{\n+\tif (ti->bi.br == NULL) {\n+\t\treturn;\n+\t}\n+\treftable_block_done(&ti->bi.br->block);\n+\tFREE_AND_NULL(ti->bi.br);\n+\n+\tti->bi.last_key.len = 0;\n+\tti->bi.next_off = 0;\n+}\n+\n+static int32_t extract_block_size(uint8_t *data, uint8_t *typ, uint64_t off,\n+\t\t\t\t int version)\n+{\n+\tint32_t result = 0;\n+\n+\tif (off == 0) {\n+\t\tdata += header_size(version);\n+\t}\n+\n+\t*typ = data[0];\n+\tif (reftable_is_block_type(*typ)) {\n+\t\tresult = get_be24(data + 1);\n+\t}\n+\treturn result;\n+}\n+\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ)\n+{\n+\tint32_t guess_block_size = r->block_size ? r->block_size :\n+\t\t\t\t\t\t\t DEFAULT_BLOCK_SIZE;\n+\tstruct reftable_block block = { NULL };\n+\tuint8_t block_typ = 0;\n+\tint err = 0;\n+\tuint32_t header_off = next_off ? 0 : header_size(r->version);\n+\tint32_t block_size = 0;\n+\n+\tif (next_off >= r->size)\n+\t\treturn 1;\n+\n+\terr = reader_get_block(r, &block, next_off, guess_block_size);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tblock_size = extract_block_size(block.data, &block_typ, next_off,\n+\t\t\t\t\tr->version);\n+\tif (block_size < 0)\n+\t\treturn block_size;\n+\n+\tif (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {\n+\t\treftable_block_done(&block);\n+\t\treturn 1;\n+\t}\n+\n+\tif (block_size > guess_block_size) {\n+\t\treftable_block_done(&block);\n+\t\terr = reader_get_block(r, &block, next_off, block_size);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\treturn block_reader_init(br, &block, header_off, r->block_size,\n+\t\t\t\t hash_size(r->hash_id));\n+}\n+\n+static int table_iter_next_block(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tuint64_t next_block_off = src->block_off + src->bi.br->full_block_size;\n+\tstruct block_reader br = { 0 };\n+\tint err = 0;\n+\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = next_block_off;\n+\n+\terr = reader_init_block_reader(src->r, &br, next_block_off, src->typ);\n+\tif (err > 0) {\n+\t\tdest->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\tif (err != 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct block_reader *brp =\n+\t\t\treftable_malloc(sizeof(struct block_reader));\n+\t\t*brp = br;\n+\n+\t\tdest->is_finished = 0;\n+\t\tblock_reader_start(brp, &dest->bi);\n+\t}\n+\treturn 0;\n+}\n+\n+static int table_iter_next(struct table_iter *ti, struct reftable_record *rec)\n+{\n+\tif (reftable_record_type(rec) != ti->typ)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\twhile (1) {\n+\t\tstruct table_iter next = TABLE_ITER_INIT;\n+\t\tint err = 0;\n+\t\tif (ti->is_finished) {\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\terr = table_iter_next_in_block(ti, rec);\n+\t\tif (err <= 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err != 0) {\n+\t\t\tti->is_finished = 1;\n+\t\t}\n+\t\ttable_iter_block_done(ti);\n+\t\tif (err != 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\ttable_iter_copy_from(ti, &next);\n+\t\tblock_iter_close(&next.bi);\n+\t}\n+}\n+\n+static int table_iter_next_void(void *ti, struct reftable_record *rec)\n+{\n+\treturn table_iter_next((struct table_iter *)ti, rec);\n+}\n+\n+static void table_iter_close(void *p)\n+{\n+\tstruct table_iter *ti = (struct table_iter *)p;\n+\ttable_iter_block_done(ti);\n+\tblock_iter_close(&ti->bi);\n+}\n+\n+static struct reftable_iterator_vtable table_iter_vtable = {\n+\t.next = &table_iter_next_void,\n+\t.close = &table_iter_close,\n+};\n+\n+static void iterator_from_table_iter(struct reftable_iterator *it,\n+\t\t\t\t struct table_iter *ti)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = ti;\n+\tit->ops = &table_iter_vtable;\n+}\n+\n+static int reader_table_iter_at(struct reftable_reader *r,\n+\t\t\t\tstruct table_iter *ti, uint64_t off,\n+\t\t\t\tuint8_t typ)\n+{\n+\tstruct block_reader br = { 0 };\n+\tstruct block_reader *brp = NULL;\n+\n+\tint err = reader_init_block_reader(r, &br, off, typ);\n+\tif (err != 0)\n+\t\treturn err;\n+\n+\tbrp = reftable_malloc(sizeof(struct block_reader));\n+\t*brp = br;\n+\tti->r = r;\n+\tti->typ = block_reader_type(brp);\n+\tti->block_off = off;\n+\tblock_reader_start(brp, &ti->bi);\n+\treturn 0;\n+}\n+\n+static int reader_start(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\tuint8_t typ, int index)\n+{\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tuint64_t off = offs->offset;\n+\tif (index) {\n+\t\toff = offs->index_offset;\n+\t\tif (off == 0) {\n+\t\t\treturn 1;\n+\t\t}\n+\t\ttyp = BLOCK_TYPE_INDEX;\n+\t}\n+\n+\treturn reader_table_iter_at(r, ti, off, typ);\n+}\n+\n+static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\t struct reftable_record *want)\n+{\n+\tstruct reftable_record rec =\n+\t\treftable_new_record(reftable_record_type(want));\n+\tstruct strbuf want_key = STRBUF_INIT;\n+\tstruct strbuf got_key = STRBUF_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = -1;\n+\n+\treftable_record_key(want, &want_key);\n+\n+\twhile (1) {\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = block_reader_first_key(next.bi.br, &got_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (strbuf_cmp(&got_key, &want_key) > 0) {\n+\t\t\ttable_iter_block_done(&next);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_block_done(ti);\n+\t\ttable_iter_copy_from(ti, &next);\n+\t}\n+\n+\terr = block_iter_seek(&ti->bi, &want_key);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = 0;\n+\n+done:\n+\tblock_iter_close(&next.bi);\n+\treftable_record_destroy(&rec);\n+\tstrbuf_release(&want_key);\n+\tstrbuf_release(&got_key);\n+\treturn err;\n+}\n+\n+static int reader_seek_indexed(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_index_record want_index = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record want_index_rec = { NULL };\n+\tstruct reftable_index_record index_result = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record index_result_rec = { NULL };\n+\tstruct table_iter index_iter = TABLE_ITER_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = 0;\n+\n+\treftable_record_key(rec, &want_index.last_key);\n+\treftable_record_from_index(&want_index_rec, &want_index);\n+\treftable_record_from_index(&index_result_rec, &index_result);\n+\n+\terr = reader_start(r, &index_iter, reftable_record_type(rec), 1);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reader_seek_linear(r, &index_iter, &want_index_rec);\n+\twhile (1) {\n+\t\terr = table_iter_next(&index_iter, &index_result_rec);\n+\t\ttable_iter_block_done(&index_iter);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = reader_table_iter_at(r, &next, index_result.offset, 0);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = block_iter_seek(&next.bi, &want_index.last_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (next.typ == reftable_record_type(rec)) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (next.typ != BLOCK_TYPE_INDEX) {\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_copy_from(&index_iter, &next);\n+\t}\n+\n+\tif (err == 0) {\n+\t\tstruct table_iter empty = TABLE_ITER_INIT;\n+\t\tstruct table_iter *malloced =\n+\t\t\treftable_calloc(sizeof(struct table_iter));\n+\t\t*malloced = empty;\n+\t\ttable_iter_copy_from(malloced, &next);\n+\t\titerator_from_table_iter(it, malloced);\n+\t}\n+done:\n+\tblock_iter_close(&next.bi);\n+\ttable_iter_close(&index_iter);\n+\treftable_record_release(&want_index_rec);\n+\treftable_record_release(&index_result_rec);\n+\treturn err;\n+}\n+\n+static int reader_seek_internal(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it,\n+\t\t\t\tstruct reftable_record *rec)\n+{\n+\tstruct reftable_reader_offsets *offs =\n+\t\treader_offsets_for(r, reftable_record_type(rec));\n+\tuint64_t idx = offs->index_offset;\n+\tstruct table_iter ti = TABLE_ITER_INIT;\n+\tint err = 0;\n+\tif (idx > 0)\n+\t\treturn reader_seek_indexed(r, it, rec);\n+\n+\terr = reader_start(r, &ti, reftable_record_type(rec), 0);\n+\tif (err < 0)\n+\t\treturn err;\n+\terr = reader_seek_linear(r, &ti, rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct table_iter *p =\n+\t\t\treftable_malloc(sizeof(struct table_iter));\n+\t\t*p = ti;\n+\t\titerator_from_table_iter(it, p);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec)\n+{\n+\tuint8_t typ = reftable_record_type(rec);\n+\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tif (!offs->is_present) {\n+\t\titerator_set_empty(it);\n+\t\treturn 0;\n+\t}\n+\n+\treturn reader_seek_internal(r, it, rec);\n+}\n+\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_reader_seek_log_at(r, it, name, max);\n+}\n+\n+void reader_close(struct reftable_reader *r)\n+{\n+\tblock_source_close(&r->source);\n+\tFREE_AND_NULL(r->name);\n+}\n+\n+int reftable_new_reader(struct reftable_reader **p,\n+\t\t\tstruct reftable_block_source *src, char const *name)\n+{\n+\tstruct reftable_reader *rd =\n+\t\treftable_calloc(sizeof(struct reftable_reader));\n+\tint err = init_reader(rd, src, name);\n+\tif (err == 0) {\n+\t\t*p = rd;\n+\t} else {\n+\t\tblock_source_close(src);\n+\t\treftable_free(rd);\n+\t}\n+\treturn err;\n+}\n+\n+void reftable_reader_free(struct reftable_reader *r)\n+{\n+\treader_close(r);\n+\treftable_free(r);\n+}\n+\n+static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct reftable_obj_record want = {\n+\t\t.hash_prefix = oid,\n+\t\t.hash_prefix_len = r->object_id_len,\n+\t};\n+\tstruct reftable_record want_rec = { NULL };\n+\tstruct reftable_iterator oit = { NULL };\n+\tstruct reftable_obj_record got = { NULL };\n+\tstruct reftable_record got_rec = { NULL };\n+\tint err = 0;\n+\tstruct indexed_table_ref_iter *itr = NULL;\n+\n+\t/* Look through the reverse index. */\n+\treftable_record_from_obj(&want_rec, &want);\n+\terr = reader_seek(r, &oit, &want_rec);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\t/* read out the reftable_obj_record */\n+\treftable_record_from_obj(&got_rec, &got);\n+\terr = iterator_next(&oit, &got_rec);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 0 ||\n+\t memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {\n+\t\t/* didn't find it; return empty iterator */\n+\t\titerator_set_empty(it);\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),\n+\t\t\t\t\t got.offsets, got.offset_len);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tgot.offsets = NULL;\n+\titerator_from_indexed_table_ref_iter(it, itr);\n+\n+done:\n+\treftable_iterator_destroy(&oit);\n+\treftable_record_release(&got_rec);\n+\treturn err;\n+}\n+\n+static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct table_iter ti_empty = TABLE_ITER_INIT;\n+\tstruct table_iter *ti = reftable_calloc(sizeof(struct table_iter));\n+\tstruct filtering_ref_iterator *filter = NULL;\n+\tstruct filtering_ref_iterator empty = FILTERING_REF_ITERATOR_INIT;\n+\tint oid_len = hash_size(r->hash_id);\n+\tint err;\n+\n+\t*ti = ti_empty;\n+\terr = reader_start(r, ti, BLOCK_TYPE_REF, 0);\n+\tif (err < 0) {\n+\t\treftable_free(ti);\n+\t\treturn err;\n+\t}\n+\n+\tfilter = reftable_malloc(sizeof(struct filtering_ref_iterator));\n+\t*filter = empty;\n+\n+\tstrbuf_add(&filter->oid, oid, oid_len);\n+\treftable_table_from_reader(&filter->tab, r);\n+\tfilter->double_check = 0;\n+\titerator_from_table_iter(&filter->it, ti);\n+\n+\titerator_from_filtering_ref_iterator(it, filter);\n+\treturn 0;\n+}\n+\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid)\n+{\n+\tif (r->obj_offsets.is_present)\n+\t\treturn reftable_reader_refs_for_indexed(r, it, oid);\n+\treturn reftable_reader_refs_for_unindexed(r, it, oid);\n+}\n+\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r)\n+{\n+\treturn r->max_update_index;\n+}\n+\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r)\n+{\n+\treturn r->min_update_index;\n+}\n+\n+/* generic table interface. */\n+\n+static int reftable_reader_seek_void(void *tab, struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn reader_seek((struct reftable_reader *)tab, it, rec);\n+}\n+\n+static uint32_t reftable_reader_hash_id_void(void *tab)\n+{\n+\treturn reftable_reader_hash_id((struct reftable_reader *)tab);\n+}\n+\n+static uint64_t reftable_reader_min_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_min_update_index((struct reftable_reader *)tab);\n+}\n+\n+static uint64_t reftable_reader_max_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_max_update_index((struct reftable_reader *)tab);\n+}\n+\n+static struct reftable_table_vtable reader_vtable = {\n+\t.seek_record = reftable_reader_seek_void,\n+\t.hash_id = reftable_reader_hash_id_void,\n+\t.min_update_index = reftable_reader_min_update_index_void,\n+\t.max_update_index = reftable_reader_max_update_index_void,\n+};\n+\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader)\n+{\n+\tassert(tab->ops == NULL);\n+\ttab->ops = &reader_vtable;\n+\ttab->table_arg = reader;\n+}\ndiff --git a/reftable/reader.h b/reftable/reader.h\nnew file mode 100644\nindex 000000000000..39583e5dbcd4\n--- /dev/null\n+++ b/reftable/reader.h\n@@ -0,0 +1,66 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef READER_H\n+#define READER_H\n+\n+#include \"block.h\"\n+#include \"record.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source);\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size);\n+void block_source_close(struct reftable_block_source *source);\n+\n+/* metadata for a block type */\n+struct reftable_reader_offsets {\n+\tint is_present;\n+\tuint64_t offset;\n+\tuint64_t index_offset;\n+};\n+\n+/* The state for reading a reftable file. */\n+struct reftable_reader {\n+\t/* for convience, associate a name with the instance. */\n+\tchar *name;\n+\tstruct reftable_block_source source;\n+\n+\t/* Size of the file, excluding the footer. */\n+\tuint64_t size;\n+\n+\t/* 'sha1' for SHA1, 's256' for SHA-256 */\n+\tuint32_t hash_id;\n+\n+\tuint32_t block_size;\n+\tuint64_t min_update_index;\n+\tuint64_t max_update_index;\n+\t/* Length of the OID keys in the 'o' section */\n+\tint object_id_len;\n+\tint version;\n+\n+\tstruct reftable_reader_offsets ref_offsets;\n+\tstruct reftable_reader_offsets obj_offsets;\n+\tstruct reftable_reader_offsets log_offsets;\n+};\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name);\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec);\n+void reader_close(struct reftable_reader *r);\n+const char *reader_name(struct reftable_reader *r);\n+\n+/* initialize a block reader to read from `r` */\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ);\n+\n+#endif\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nnew file mode 100644\nindex 000000000000..201aab3c2af7\n--- /dev/null\n+++ b/reftable/reftable-reader.h\n@@ -0,0 +1,98 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_READER_H\n+#define REFTABLE_READER_H\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-blocksource.h\"\n+\n+/*\n+ * Reading single tables\n+ *\n+ * The follow routines are for reading single files. For an\n+ * application-level interface, skip ahead to struct\n+ * reftable_merged_table and struct reftable_stack.\n+ */\n+\n+/* The reader struct is a handle to an open reftable file. */\n+struct reftable_reader;\n+\n+/* Generic table. */\n+struct reftable_table;\n+\n+/* reftable_new_reader opens a reftable for reading. If successful,\n+ * returns 0 code and sets pp. The name is used for creating a\n+ * stack. Typically, it is the basename of the file. The block source\n+ * `src` is owned by the reader, and is closed on calling\n+ * reftable_reader_destroy(). On error, the block source `src` is\n+ * closed as well.\n+ */\n+int reftable_new_reader(struct reftable_reader **pp,\n+\t\t\tstruct reftable_block_source *src, const char *name);\n+\n+/* reftable_reader_seek_ref returns an iterator where 'name' would be inserted\n+ in the table. To seek to the start of the table, use name = \"\".\n+\n+ example:\n+\n+ struct reftable_reader *r = NULL;\n+ int err = reftable_new_reader(&r, &src, \"filename\");\n+ if (err < 0) { ... }\n+ struct reftable_iterator it = {0};\n+ err = reftable_reader_seek_ref(r, &it, \"refs/heads/master\");\n+ if (err < 0) { ... }\n+ struct reftable_ref_record ref = {0};\n+ while (1) {\n+ err = reftable_iterator_next_ref(&it, &ref);\n+ if (err > 0) {\n+ break;\n+ }\n+ if (err < 0) {\n+ ..error handling..\n+ }\n+ ..found..\n+ }\n+ reftable_iterator_destroy(&it);\n+ reftable_ref_record_release(&ref);\n+*/\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* returns the hash ID used in this table. */\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r);\n+\n+/* seek to logs for the given name, older than update_index. To seek to the\n+ start of the table, use name = \"\".\n+*/\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index);\n+\n+/* seek to newest log entry for given name. */\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* closes and deallocates a reader. */\n+void reftable_reader_free(struct reftable_reader *);\n+\n+/* return an iterator for the refs pointing to `oid`. */\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid);\n+\n+/* return the max_update_index for a table */\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r);\n+\n+/* return the min_update_index for a table */\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r);\n+\n+/* creates a generic table from a file reader. */\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"421765","messageId":"9ade9303f08ffa7e404753ede2c2460c282b94f5.1618255553.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 12/20] reftable: reftable file level tests","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:44Z","receivedAt":"2021-04-12T19:26:27Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nWith support for reading and writing files in place, we can construct files (in\nmemory) and attempt to read them back.\n\nBecause some sections of the format are optional (eg. indices, log entries), we\nhave to exercise this code using multiple sizes of input data\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable_test.c | 583 +++++++++++++++++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 3 files changed, 585 insertions(+)\n create mode 100644 reftable/reftable_test.c\n\ndiff --git a/Makefile b/Makefile\nindex f8167084b80a..b23e4a82063a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2416,6 +2416,7 @@ REFTABLE_OBJS += reftable/zlib-compat.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n+REFTABLE_TEST_OBJS += reftable/reftable_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/reftable_test.c b/reftable/reftable_test.c\nnew file mode 100644\nindex 000000000000..69dbfb09fff9\n--- /dev/null\n+++ b/reftable/reftable_test.c\n@@ -0,0 +1,583 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-writer.h\"\n+\n+static const int update_index = 5;\n+\n+static void test_buffer(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_block out = { NULL };\n+\tint n;\n+\tuint8_t in[] = \"hello\";\n+\tstrbuf_add(&buf, in, sizeof(in));\n+\tblock_source_from_strbuf(&source, &buf);\n+\tEXPECT(block_source_size(&source) == 6);\n+\tn = block_source_read_block(&source, &out, 0, sizeof(in));\n+\tEXPECT(n == sizeof(in));\n+\tEXPECT(!memcmp(in, out.data, n));\n+\treftable_block_done(&out);\n+\n+\tn = block_source_read_block(&source, &out, 1, 2);\n+\tEXPECT(n == 2);\n+\tEXPECT(!memcmp(out.data, \"el\", 2));\n+\n+\treftable_block_done(&out);\n+\tblock_source_close(&source);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void write_table(char ***names, struct strbuf *buf, int N,\n+\t\t\tint block_size, uint32_t hash_id)\n+{\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = block_size,\n+\t\t.hash_id = hash_id,\n+\t};\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, buf, &opts);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0, n;\n+\tstruct reftable_log_record log = { NULL };\n+\tconst struct reftable_stats *stats = NULL;\n+\t*names = reftable_calloc(sizeof(char *) * (N + 1));\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[SHA256_SIZE] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tref.refname = name;\n+\t\tref.update_index = update_index;\n+\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\tref.value.val1 = hash;\n+\t\t(*names)[i] = xstrdup(name);\n+\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[SHA256_SIZE] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tlog.refname = name;\n+\t\tlog.update_index = update_index;\n+\t\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog.update.new_hash = hash;\n+\t\tlog.update.message = \"message\";\n+\n+\t\tn = reftable_writer_add_log(w, &log);\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tfor (i = 0; i < stats->ref_stats.blocks; i++) {\n+\t\tint off = i * opts.block_size;\n+\t\tif (off == 0) {\n+\t\t\toff = header_size((hash_id == SHA256_ID) ? 2 : 1);\n+\t\t}\n+\t\tEXPECT(buf->buf[off] == 'r');\n+\t}\n+\n+\tEXPECT(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+}\n+\n+static void test_log_buffer_size(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 4096,\n+\t};\n+\tint err;\n+\tstruct reftable_log_record log = { .refname = \"refs/heads/master\",\n+\t\t\t\t\t .update_index = 0xa,\n+\t\t\t\t\t .value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t\t\t .update = {\n+\t\t\t\t\t\t .name = \"Han-Wen Nienhuys\",\n+\t\t\t\t\t\t .email = \"hanwen@google.com\",\n+\t\t\t\t\t\t .tz_offset = 100,\n+\t\t\t\t\t\t .time = 0x5e430672,\n+\t\t\t\t\t\t .message = \"commit: 9\\n\",\n+\t\t\t\t\t } };\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\t/* This tests buffer extension for log compression. Must use a random\n+\t hash, to ensure that the compressed part is larger than the original.\n+\t*/\n+\tuint8_t hash1[SHA1_SIZE], hash2[SHA1_SIZE];\n+\tfor (int i = 0; i < SHA1_SIZE; i++) {\n+\t\thash1[i] = (uint8_t)(rand() % 256);\n+\t\thash2[i] = (uint8_t)(rand() % 256);\n+\t}\n+\tlog.update.old_hash = hash1;\n+\tlog.update.new_hash = hash2;\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\terr = reftable_writer_add_log(w, &log);\n+\tEXPECT_ERR(err);\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_log_write_read(void)\n+{\n+\tint N = 2;\n+\tchar **names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint err;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\tint n;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tconst struct reftable_stats *stats = NULL;\n+\treftable_writer_set_limits(w, 0, N);\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[256];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tsnprintf(name, sizeof(name), \"b%02d%0*d\", i, 130, 7);\n+\t\tnames[i] = xstrdup(name);\n+\t\tref.refname = name;\n+\t\tref.update_index = i;\n+\n+\t\terr = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash1[SHA1_SIZE], hash2[SHA1_SIZE];\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tset_test_hash(hash1, i);\n+\t\tset_test_hash(hash2, i + 1);\n+\n+\t\tlog.refname = names[i];\n+\t\tlog.update_index = i;\n+\t\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog.update.old_hash = hash1;\n+\t\tlog.update.new_hash = hash2;\n+\n+\t\terr = reftable_writer_add_log(w, &log);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tEXPECT(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[N - 1]);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT_ERR(err);\n+\n+\t/* end of iteration. */\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT(0 < err);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_release(&ref);\n+\n+\terr = reftable_reader_seek_log(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\ti = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT_STREQ(names[i], log.refname);\n+\t\tEXPECT(i == log.update_index);\n+\t\ti++;\n+\t\treftable_log_record_release(&log);\n+\t}\n+\n+\tEXPECT(i == N);\n+\treftable_iterator_destroy(&it);\n+\n+\t/* cleanup. */\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_sequential(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tint err = 0;\n+\tint j = 0;\n+\n+\twrite_table(&names, &buf, N, 256, SHA1_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint r = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tEXPECT(0 == strcmp(names[j], ref.refname));\n+\t\tEXPECT(update_index == ref.update_index);\n+\n+\t\tj++;\n+\t\treftable_ref_record_release(&ref);\n+\t}\n+\tEXPECT(j == N);\n+\treftable_iterator_destroy(&it);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\n+\treader_close(&rd);\n+}\n+\n+static void test_table_write_small_table(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 1;\n+\twrite_table(&names, &buf, N, 4096, SHA1_ID);\n+\tEXPECT(buf.len < 200);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+}\n+\n+static void test_table_read_api(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i;\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, SHA1_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[0]);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_free(names);\n+\treader_close(&rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_table_read_write_seek(int index, int hash_id)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i = 0;\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct strbuf pastLast = STRBUF_INIT;\n+\tstruct reftable_ref_record ref = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, hash_id);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\tEXPECT(hash_id == reftable_reader_hash_id(&rd));\n+\n+\tif (!index) {\n+\t\trd.ref_offsets.index_offset = 0;\n+\t} else {\n+\t\tEXPECT(rd.ref_offsets.index_offset > 0);\n+\t}\n+\n+\tfor (i = 1; i < N; i++) {\n+\t\tint err = reftable_reader_seek_ref(&rd, &it, names[i]);\n+\t\tEXPECT_ERR(err);\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(0 == strcmp(names[i], ref.refname));\n+\t\tEXPECT(REFTABLE_REF_VAL1 == ref.value_type);\n+\t\tEXPECT(i == ref.value.val1[0]);\n+\n+\t\treftable_ref_record_release(&ref);\n+\t\treftable_iterator_destroy(&it);\n+\t}\n+\n+\tstrbuf_addstr(&pastLast, names[N - 1]);\n+\tstrbuf_addstr(&pastLast, \"/\");\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, pastLast.buf);\n+\tif (err == 0) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(err > 0);\n+\t} else {\n+\t\tEXPECT(err > 0);\n+\t}\n+\n+\tstrbuf_release(&pastLast);\n+\treftable_iterator_destroy(&it);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_free(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_seek_linear(void)\n+{\n+\ttest_table_read_write_seek(0, SHA1_ID);\n+}\n+\n+static void test_table_read_write_seek_linear_sha256(void)\n+{\n+\ttest_table_read_write_seek(0, SHA256_ID);\n+}\n+\n+static void test_table_read_write_seek_index(void)\n+{\n+\ttest_table_read_write_seek(1, SHA1_ID);\n+}\n+\n+static void test_table_refs_for(int indexed)\n+{\n+\tint N = 50;\n+\tchar **want_names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint want_names_len = 0;\n+\tuint8_t want_hash[SHA1_SIZE];\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tint err;\n+\tstruct reftable_reader rd;\n+\tstruct reftable_block_source source = { NULL };\n+\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint j;\n+\n+\tset_test_hash(want_hash, 4);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[SHA1_SIZE];\n+\t\tchar fill[51] = { 0 };\n+\t\tchar name[100];\n+\t\tuint8_t hash1[SHA1_SIZE];\n+\t\tuint8_t hash2[SHA1_SIZE];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\n+\t\tmemset(hash, i, sizeof(hash));\n+\t\tmemset(fill, 'x', 50);\n+\t\t/* Put the variable part in the start */\n+\t\tsnprintf(name, sizeof(name), \"br%02d%s\", i, fill);\n+\t\tname[40] = 0;\n+\t\tref.refname = name;\n+\n+\t\tset_test_hash(hash1, i / 4);\n+\t\tset_test_hash(hash2, 3 + i / 4);\n+\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\tref.value.val2.value = hash1;\n+\t\tref.value.val2.target_value = hash2;\n+\n+\t\t/* 80 bytes / entry, so 3 entries per block. Yields 17\n+\t\t */\n+\t\t/* blocks. */\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT(n == 0);\n+\n+\t\tif (!memcmp(hash1, want_hash, SHA1_SIZE) ||\n+\t\t !memcmp(hash2, want_hash, SHA1_SIZE)) {\n+\t\t\twant_names[want_names_len++] = xstrdup(name);\n+\t\t}\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\tif (!indexed) {\n+\t\trd.obj_offsets.is_present = 0;\n+\t}\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_reader_refs_for(&rd, &it, want_hash);\n+\tEXPECT_ERR(err);\n+\n+\tj = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(err >= 0);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tEXPECT(j < want_names_len);\n+\t\tEXPECT(0 == strcmp(ref.refname, want_names[j]));\n+\t\tj++;\n+\t\treftable_ref_record_release(&ref);\n+\t}\n+\tEXPECT(j == want_names_len);\n+\n+\tstrbuf_release(&buf);\n+\tfree_names(want_names);\n+\treftable_iterator_destroy(&it);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_refs_for_no_index(void)\n+{\n+\ttest_table_refs_for(0);\n+}\n+\n+static void test_table_refs_for_obj_index(void)\n+{\n+\ttest_table_refs_for(1);\n+}\n+\n+static void test_table_empty(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_ref_record rec = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT(err == REFTABLE_EMPTY_TABLE_ERROR);\n+\treftable_writer_free(w);\n+\n+\tEXPECT(buf.len == header_size(1) + footer_size(1));\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tEXPECT(err > 0);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+int reftable_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_log_write_read);\n+\tRUN_TEST(test_table_read_write_seek_linear_sha256);\n+\tRUN_TEST(test_log_buffer_size);\n+\tRUN_TEST(test_table_write_small_table);\n+\tRUN_TEST(test_buffer);\n+\tRUN_TEST(test_table_read_api);\n+\tRUN_TEST(test_table_read_write_sequential);\n+\tRUN_TEST(test_table_read_write_seek_linear);\n+\tRUN_TEST(test_table_read_write_seek_index);\n+\tRUN_TEST(test_table_refs_for_no_index);\n+\tRUN_TEST(test_table_refs_for_obj_index);\n+\tRUN_TEST(test_table_empty);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 050551fa6985..fdf925867375 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,6 +6,7 @@ int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\treftable_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"421766","messageId":"0a6119d910a3f83e6c9c421acd1316b496e6612b.1618255553.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 13/20] reftable: add a heap-based priority queue for reftable records","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:45Z","receivedAt":"2021-04-12T19:26:28Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis is needed to create a merged view multiple reftables\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/pq.c | 115 ++++++++++++++++++++++++++++++++++++++\n reftable/pq.h | 32 +++++++++++\n reftable/pq_test.c | 72 ++++++++++++++++++++++++\n reftable/reftable-tests.h | 1 +\n t/helper/test-reftable.c | 1 +\n 6 files changed, 223 insertions(+)\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/pq_test.c\n\ndiff --git a/Makefile b/Makefile\nindex b23e4a82063a..74560a9605bb 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2406,6 +2406,7 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/reftable.o\n@@ -2415,6 +2416,7 @@ REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/reftable_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\ndiff --git a/reftable/pq.c b/reftable/pq.c\nnew file mode 100644\nindex 000000000000..8918d158e2d4\n--- /dev/null\n+++ b/reftable/pq.c\n@@ -0,0 +1,115 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"pq.h\"\n+\n+#include \"reftable-record.h\"\n+#include \"system.h\"\n+#include \"basics.h\"\n+\n+static int pq_less(struct pq_entry a, struct pq_entry b)\n+{\n+\tstruct strbuf ak = STRBUF_INIT;\n+\tstruct strbuf bk = STRBUF_INIT;\n+\tint cmp = 0;\n+\treftable_record_key(&a.rec, &ak);\n+\treftable_record_key(&b.rec, &bk);\n+\n+\tcmp = strbuf_cmp(&ak, &bk);\n+\n+\tstrbuf_release(&ak);\n+\tstrbuf_release(&bk);\n+\n+\tif (cmp == 0)\n+\t\treturn a.index > b.index;\n+\n+\treturn cmp < 0;\n+}\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.heap[0];\n+}\n+\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.len == 0;\n+}\n+\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq)\n+{\n+\tint i = 0;\n+\tfor (i = 1; i < pq.len; i++) {\n+\t\tint parent = (i - 1) / 2;\n+\n+\t\tassert(pq_less(pq.heap[parent], pq.heap[i]));\n+\t}\n+}\n+\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tstruct pq_entry e = pq->heap[0];\n+\tpq->heap[0] = pq->heap[pq->len - 1];\n+\tpq->len--;\n+\n+\ti = 0;\n+\twhile (i < pq->len) {\n+\t\tint min = i;\n+\t\tint j = 2 * i + 1;\n+\t\tint k = 2 * i + 2;\n+\t\tif (j < pq->len && pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tmin = j;\n+\t\t}\n+\t\tif (k < pq->len && pq_less(pq->heap[k], pq->heap[min])) {\n+\t\t\tmin = k;\n+\t\t}\n+\n+\t\tif (min == i) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[i], pq->heap[min]);\n+\t\ti = min;\n+\t}\n+\n+\treturn e;\n+}\n+\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)\n+{\n+\tint i = 0;\n+\tif (pq->len == pq->cap) {\n+\t\tpq->cap = 2 * pq->cap + 1;\n+\t\tpq->heap = reftable_realloc(pq->heap,\n+\t\t\t\t\t pq->cap * sizeof(struct pq_entry));\n+\t}\n+\n+\tpq->heap[pq->len++] = e;\n+\ti = pq->len - 1;\n+\twhile (i > 0) {\n+\t\tint j = (i - 1) / 2;\n+\t\tif (pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[j], pq->heap[i]);\n+\n+\t\ti = j;\n+\t}\n+}\n+\n+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < pq->len; i++) {\n+\t\treftable_record_destroy(&pq->heap[i].rec);\n+\t}\n+\tFREE_AND_NULL(pq->heap);\n+\tpq->len = pq->cap = 0;\n+}\ndiff --git a/reftable/pq.h b/reftable/pq.h\nnew file mode 100644\nindex 000000000000..385d2fb139a6\n--- /dev/null\n+++ b/reftable/pq.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef PQ_H\n+#define PQ_H\n+\n+#include \"record.h\"\n+\n+struct pq_entry {\n+\tint index;\n+\tstruct reftable_record rec;\n+};\n+\n+struct merged_iter_pqueue {\n+\tstruct pq_entry *heap;\n+\tsize_t len;\n+\tsize_t cap;\n+};\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq);\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq);\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq);\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e);\n+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n+\n+#endif\ndiff --git a/reftable/pq_test.c b/reftable/pq_test.c\nnew file mode 100644\nindex 000000000000..5178980b37ed\n--- /dev/null\n+++ b/reftable/pq_test.c\n@@ -0,0 +1,72 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"pq.h\"\n+#include \"record.h\"\n+#include \"reftable-tests.h\"\n+#include \"test_framework.h\"\n+\n+static void test_pq(void)\n+{\n+\tchar *names[54] = { NULL };\n+\tint N = ARRAY_SIZE(names) - 1;\n+\n+\tstruct merged_iter_pqueue pq = { NULL };\n+\tconst char *last = NULL;\n+\n+\tint i = 0;\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tsnprintf(name, sizeof(name), \"%02d\", i);\n+\t\tnames[i] = xstrdup(name);\n+\t}\n+\n+\ti = 1;\n+\tdo {\n+\t\tstruct reftable_record rec =\n+\t\t\treftable_new_record(BLOCK_TYPE_REF);\n+\t\tstruct pq_entry e = { 0 };\n+\n+\t\treftable_record_as_ref(&rec)->refname = names[i];\n+\t\te.rec = rec;\n+\t\tmerged_iter_pqueue_add(&pq, e);\n+\t\tmerged_iter_pqueue_check(pq);\n+\t\ti = (i * 7) % N;\n+\t} while (i != 1);\n+\n+\twhile (!merged_iter_pqueue_is_empty(pq)) {\n+\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct reftable_ref_record *ref =\n+\t\t\treftable_record_as_ref(&e.rec);\n+\n+\t\tmerged_iter_pqueue_check(pq);\n+\n+\t\tif (last != NULL) {\n+\t\t\tassert(strcmp(last, ref->refname) < 0);\n+\t\t}\n+\t\tlast = ref->refname;\n+\t\tref->refname = NULL;\n+\t\treftable_free(ref);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\n+\tmerged_iter_pqueue_release(&pq);\n+}\n+\n+int pq_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_pq);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nindex 5e7698ae654e..495556572e20 100644\n--- a/reftable/reftable-tests.h\n+++ b/reftable/reftable-tests.h\n@@ -12,6 +12,7 @@ license that can be found in the LICENSE file or at\n int basics_test_main(int argc, const char **argv);\n int block_test_main(int argc, const char **argv);\n int merged_test_main(int argc, const char **argv);\n+int pq_test_main(int argc, const char **argv);\n int record_test_main(int argc, const char **argv);\n int refname_test_main(int argc, const char **argv);\n int reftable_test_main(int argc, const char **argv);\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex fdf925867375..368eb7339eb7 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -5,6 +5,7 @@ int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n+\tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treftable_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n-- \ngitgitgadget\n\n"},{"id":"421767","messageId":"393fcecdae1ec471bbd692d63688a16697ecdf78.1618255553.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 14/20] reftable: add merged table view","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:46Z","receivedAt":"2021-04-12T19:26:29Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis adds an abstract, read-only interface to the ref database.\n\nThis primitive is used to construct the read view of the ref database\n(the read view is constructed by merging several *.ref files). It also\nprovides the mechanism to provide a unified view of the refs in the main\nrepository and the per-worktree refs.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/merged.c | 367 +++++++++++++++++++++++++++++++++++++\n reftable/merged.h | 35 ++++\n reftable/merged_test.c | 291 +++++++++++++++++++++++++++++\n reftable/reftable-merged.h | 72 ++++++++\n t/helper/test-reftable.c | 1 +\n 6 files changed, 768 insertions(+)\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/reftable-merged.h\n\ndiff --git a/Makefile b/Makefile\nindex 74560a9605bb..129f73d118ac 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2406,6 +2406,7 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/merged.o\n REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n@@ -2416,6 +2417,7 @@ REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/reftable_test.o\ndiff --git a/reftable/merged.c b/reftable/merged.c\nnew file mode 100644\nindex 000000000000..02940a79b0a3\n--- /dev/null\n+++ b/reftable/merged.c\n@@ -0,0 +1,367 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"constants.h\"\n+#include \"iter.h\"\n+#include \"pq.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"generic.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-error.h\"\n+#include \"system.h\"\n+\n+static int merged_iter_init(struct merged_iter *mi)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\t\tint err = iterator_next(&mi->stack[i], &rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t\t\treftable_record_destroy(&rec);\n+\t\t} else {\n+\t\t\tstruct pq_entry e = {\n+\t\t\t\t.rec = rec,\n+\t\t\t\t.index = i,\n+\t\t\t};\n+\t\t\tmerged_iter_pqueue_add(&mi->pq, e);\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void merged_iter_close(void *p)\n+{\n+\tstruct merged_iter *mi = (struct merged_iter *)p;\n+\tint i = 0;\n+\tmerged_iter_pqueue_release(&mi->pq);\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t}\n+\treftable_free(mi->stack);\n+}\n+\n+static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,\n+\t\t\t\t\t size_t idx)\n+{\n+\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\tstruct pq_entry e = {\n+\t\t.rec = rec,\n+\t\t.index = idx,\n+\t};\n+\tint err = iterator_next(&mi->stack[idx], &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (err > 0) {\n+\t\treftable_iterator_destroy(&mi->stack[idx]);\n+\t\treftable_record_destroy(&rec);\n+\t\treturn 0;\n+\t}\n+\n+\tmerged_iter_pqueue_add(&mi->pq, e);\n+\treturn 0;\n+}\n+\n+static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)\n+{\n+\tif (iterator_is_null(&mi->stack[idx]))\n+\t\treturn 0;\n+\treturn merged_iter_advance_nonnull_subiter(mi, idx);\n+}\n+\n+static int merged_iter_next_entry(struct merged_iter *mi,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct strbuf entry_key = STRBUF_INIT;\n+\tstruct pq_entry entry = { 0 };\n+\tint err = 0;\n+\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\tentry = merged_iter_pqueue_remove(&mi->pq);\n+\terr = merged_iter_advance_subiter(mi, entry.index);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\t/*\n+\t One can also use reftable as datacenter-local storage, where the ref\n+\t database is maintained in globally consistent database (eg.\n+\t CockroachDB or Spanner). In this scenario, replication delays together\n+\t with compaction may cause newer tables to contain older entries. In\n+\t such a deployment, the loop below must be changed to collect all\n+\t entries for the same key, and return new the newest one.\n+\t*/\n+\treftable_record_key(&entry.rec, &entry_key);\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq)) {\n+\t\tstruct pq_entry top = merged_iter_pqueue_top(mi->pq);\n+\t\tstruct strbuf k = STRBUF_INIT;\n+\t\tint err = 0, cmp = 0;\n+\n+\t\treftable_record_key(&top.rec, &k);\n+\n+\t\tcmp = strbuf_cmp(&k, &entry_key);\n+\t\tstrbuf_release(&k);\n+\n+\t\tif (cmp > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\t\terr = merged_iter_advance_subiter(mi, top.index);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_record_destroy(&top.rec);\n+\t}\n+\n+\treftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));\n+\treftable_record_destroy(&entry.rec);\n+\tstrbuf_release(&entry_key);\n+\treturn 0;\n+}\n+\n+static int merged_iter_next(struct merged_iter *mi, struct reftable_record *rec)\n+{\n+\twhile (1) {\n+\t\tint err = merged_iter_next_entry(mi, rec);\n+\t\tif (err == 0 && mi->suppress_deletions &&\n+\t\t reftable_record_is_deletion(rec)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\treturn err;\n+\t}\n+}\n+\n+static int merged_iter_next_void(void *p, struct reftable_record *rec)\n+{\n+\tstruct merged_iter *mi = (struct merged_iter *)p;\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\treturn merged_iter_next(mi, rec);\n+}\n+\n+static struct reftable_iterator_vtable merged_iter_vtable = {\n+\t.next = &merged_iter_next_void,\n+\t.close = &merged_iter_close,\n+};\n+\n+static void iterator_from_merged_iter(struct reftable_iterator *it,\n+\t\t\t\t struct merged_iter *mi)\n+{\n+\tassert(it->ops == NULL);\n+\tit->iter_arg = mi;\n+\tit->ops = &merged_iter_vtable;\n+}\n+\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id)\n+{\n+\tstruct reftable_merged_table *m = NULL;\n+\tuint64_t last_max = 0;\n+\tuint64_t first_min = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t min = reftable_table_min_update_index(&stack[i]);\n+\t\tuint64_t max = reftable_table_max_update_index(&stack[i]);\n+\n+\t\tif (reftable_table_hash_id(&stack[i]) != hash_id) {\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\t\t}\n+\t\tif (i == 0 || min < first_min) {\n+\t\t\tfirst_min = min;\n+\t\t}\n+\t\tif (i == 0 || max > last_max) {\n+\t\t\tlast_max = max;\n+\t\t}\n+\t}\n+\n+\tm = (struct reftable_merged_table *)reftable_calloc(\n+\t\tsizeof(struct reftable_merged_table));\n+\tm->stack = stack;\n+\tm->stack_len = n;\n+\tm->min = first_min;\n+\tm->max = last_max;\n+\tm->hash_id = hash_id;\n+\t*dest = m;\n+\treturn 0;\n+}\n+\n+/* clears the list of subtable, without affecting the readers themselves. */\n+void merged_table_release(struct reftable_merged_table *mt)\n+{\n+\tFREE_AND_NULL(mt->stack);\n+\tmt->stack_len = 0;\n+}\n+\n+void reftable_merged_table_free(struct reftable_merged_table *mt)\n+{\n+\tif (mt == NULL) {\n+\t\treturn;\n+\t}\n+\tmerged_table_release(mt);\n+\treftable_free(mt);\n+}\n+\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->max;\n+}\n+\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->min;\n+}\n+\n+static int reftable_table_seek_record(struct reftable_table *tab,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn tab->ops->seek_record(tab->table_arg, it, rec);\n+}\n+\n+static int merged_table_seek_record(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_iterator *iters = reftable_calloc(\n+\t\tsizeof(struct reftable_iterator) * mt->stack_len);\n+\tstruct merged_iter merged = {\n+\t\t.stack = iters,\n+\t\t.typ = reftable_record_type(rec),\n+\t\t.hash_id = mt->hash_id,\n+\t\t.suppress_deletions = mt->suppress_deletions,\n+\t};\n+\tint n = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < mt->stack_len && err == 0; i++) {\n+\t\tint e = reftable_table_seek_record(&mt->stack[i], &iters[n],\n+\t\t\t\t\t\t rec);\n+\t\tif (e < 0) {\n+\t\t\terr = e;\n+\t\t}\n+\t\tif (e == 0) {\n+\t\t\tn++;\n+\t\t}\n+\t}\n+\tif (err < 0) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < n; i++) {\n+\t\t\treftable_iterator_destroy(&iters[i]);\n+\t\t}\n+\t\treftable_free(iters);\n+\t\treturn err;\n+\t}\n+\n+\tmerged.stack_len = n;\n+\terr = merged_iter_init(&merged);\n+\tif (err < 0) {\n+\t\tmerged_iter_close(&merged);\n+\t\treturn err;\n+\t} else {\n+\t\tstruct merged_iter *p =\n+\t\t\treftable_malloc(sizeof(struct merged_iter));\n+\t\t*p = merged;\n+\t\titerator_from_merged_iter(it, p);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_merged_table_seek_log_at(mt, it, name, max);\n+}\n+\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\n+{\n+\treturn mt->hash_id;\n+}\n+\n+static int reftable_merged_table_seek_void(void *tab,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn merged_table_seek_record((struct reftable_merged_table *)tab, it,\n+\t\t\t\t\trec);\n+}\n+\n+static uint32_t reftable_merged_table_hash_id_void(void *tab)\n+{\n+\treturn reftable_merged_table_hash_id(\n+\t\t(struct reftable_merged_table *)tab);\n+}\n+\n+static uint64_t reftable_merged_table_min_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_min_update_index(\n+\t\t(struct reftable_merged_table *)tab);\n+}\n+\n+static uint64_t reftable_merged_table_max_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_max_update_index(\n+\t\t(struct reftable_merged_table *)tab);\n+}\n+\n+static struct reftable_table_vtable merged_table_vtable = {\n+\t.seek_record = reftable_merged_table_seek_void,\n+\t.hash_id = reftable_merged_table_hash_id_void,\n+\t.min_update_index = reftable_merged_table_min_update_index_void,\n+\t.max_update_index = reftable_merged_table_max_update_index_void,\n+};\n+\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *merged)\n+{\n+\tassert(tab->ops == NULL);\n+\ttab->ops = &merged_table_vtable;\n+\ttab->table_arg = merged;\n+}\ndiff --git a/reftable/merged.h b/reftable/merged.h\nnew file mode 100644\nindex 000000000000..8c4d4d58d77a\n--- /dev/null\n+++ b/reftable/merged.h\n@@ -0,0 +1,35 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef MERGED_H\n+#define MERGED_H\n+\n+#include \"pq.h\"\n+\n+struct reftable_merged_table {\n+\tstruct reftable_table *stack;\n+\tsize_t stack_len;\n+\tuint32_t hash_id;\n+\tint suppress_deletions;\n+\n+\tuint64_t min;\n+\tuint64_t max;\n+};\n+\n+struct merged_iter {\n+\tstruct reftable_iterator *stack;\n+\tuint32_t hash_id;\n+\tsize_t stack_len;\n+\tuint8_t typ;\n+\tint suppress_deletions;\n+\tstruct merged_iter_pqueue pq;\n+};\n+\n+void merged_table_release(struct reftable_merged_table *mt);\n+\n+#endif\ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nnew file mode 100644\nindex 000000000000..0c301ceccedd\n--- /dev/null\n+++ b/reftable/merged_test.c\n@@ -0,0 +1,291 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-generic.h\"\n+#include \"reftable-writer.h\"\n+\n+static void write_test_table(struct strbuf *buf,\n+\t\t\t struct reftable_ref_record refs[], int n)\n+{\n+\tint min = 0xffffffff;\n+\tint max = 0;\n+\tint i = 0;\n+\tint err;\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_writer *w = NULL;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t ui = refs[i].update_index;\n+\t\tif (ui > max) {\n+\t\t\tmax = ui;\n+\t\t}\n+\t\tif (ui < min) {\n+\t\t\tmin = ui;\n+\t\t}\n+\t}\n+\n+\tw = reftable_new_writer(&strbuf_add_void, buf, &opts);\n+\treftable_writer_set_limits(w, min, max);\n+\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t before = refs[i].update_index;\n+\t\tint n = reftable_writer_add_ref(w, &refs[i]);\n+\t\tassert(n == 0);\n+\t\tassert(before == refs[i].update_index);\n+\t}\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\n+\treftable_writer_free(w);\n+}\n+\n+static struct reftable_merged_table *\n+merged_table_from_records(struct reftable_ref_record **refs,\n+\t\t\t struct reftable_block_source **source,\n+\t\t\t struct reftable_reader ***readers, int *sizes,\n+\t\t\t struct strbuf *buf, int n)\n+{\n+\tint i = 0;\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err;\n+\tstruct reftable_table *tabs =\n+\t\treftable_calloc(n * sizeof(struct reftable_table));\n+\t*readers = reftable_calloc(n * sizeof(struct reftable_reader *));\n+\t*source = reftable_calloc(n * sizeof(**source));\n+\tfor (i = 0; i < n; i++) {\n+\t\twrite_test_table(&buf[i], refs[i], sizes[i]);\n+\t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n+\n+\t\terr = reftable_new_reader(&(*readers)[i], &(*source)[i],\n+\t\t\t\t\t \"name\");\n+\t\tEXPECT_ERR(err);\n+\t\treftable_table_from_reader(&tabs[i], (*readers)[i]);\n+\t}\n+\n+\terr = reftable_new_merged_table(&mt, tabs, n, SHA1_ID);\n+\tEXPECT_ERR(err);\n+\treturn mt;\n+}\n+\n+static void readers_destroy(struct reftable_reader **readers, size_t n)\n+{\n+\tint i = 0;\n+\tfor (; i < n; i++)\n+\t\treftable_reader_free(readers[i]);\n+\treftable_free(readers);\n+}\n+\n+static void test_merged_between(void)\n+{\n+\tuint8_t hash1[SHA1_SIZE] = { 1, 2, 3, 0 };\n+\n+\tstruct reftable_ref_record r1[] = { {\n+\t\t.refname = \"b\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t.value.val1 = hash1,\n+\t} };\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_DELETION,\n+\t} };\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2 };\n+\tint sizes[] = { 1, 1 };\n+\tstruct strbuf bufs[2] = { STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 2);\n+\tint i;\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT_ERR(err);\n+\tEXPECT(ref.update_index == 2);\n+\treftable_ref_record_release(&ref);\n+\treftable_iterator_destroy(&it);\n+\treaders_destroy(readers, 2);\n+\treftable_merged_table_free(mt);\n+\tfor (i = 0; i < ARRAY_SIZE(bufs); i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treftable_free(bs);\n+}\n+\n+static void test_merged(void)\n+{\n+\tuint8_t hash1[SHA1_SIZE] = { 1 };\n+\tuint8_t hash2[SHA1_SIZE] = { 2 };\n+\tstruct reftable_ref_record r1[] = {\n+\t\t{\n+\t\t\t.refname = \"a\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"b\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t}\n+\t};\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_DELETION,\n+\t} };\n+\tstruct reftable_ref_record r3[] = {\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash2,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"d\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t};\n+\n+\tstruct reftable_ref_record want[] = {\n+\t\tr2[0],\n+\t\tr1[1],\n+\t\tr3[0],\n+\t\tr3[1],\n+\t};\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2, r3 };\n+\tint sizes[3] = { 3, 1, 2 };\n+\tstruct strbuf bufs[3] = { STRBUF_INIT, STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 3);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tstruct reftable_ref_record *out = NULL;\n+\tsize_t len = 0;\n+\tsize_t cap = 0;\n+\tint i = 0;\n+\n+\tEXPECT_ERR(err);\n+\twhile (len < 100) { /* cap loops/recursion. */\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tout = reftable_realloc(\n+\t\t\t\tout, sizeof(struct reftable_ref_record) * cap);\n+\t\t}\n+\t\tout[len++] = ref;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\tassert(ARRAY_SIZE(want) == len);\n+\tfor (i = 0; i < len; i++) {\n+\t\tassert(reftable_ref_record_equal(&want[i], &out[i], SHA1_SIZE));\n+\t}\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_release(&out[i]);\n+\t}\n+\treftable_free(out);\n+\n+\tfor (i = 0; i < 3; i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treaders_destroy(readers, 3);\n+\treftable_merged_table_free(mt);\n+\treftable_free(bs);\n+}\n+\n+static void test_default_write_opts(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"master\",\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_table *tab = reftable_calloc(sizeof(*tab) * 1);\n+\tuint32_t hash_id;\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_merged_table *merged = NULL;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\thash_id = reftable_reader_hash_id(rd);\n+\tassert(hash_id == SHA1_ID);\n+\n+\treftable_table_from_reader(&tab[0], rd);\n+\terr = reftable_new_merged_table(&merged, tab, 1, SHA1_ID);\n+\tEXPECT_ERR(err);\n+\n+\treftable_reader_free(rd);\n+\treftable_merged_table_free(merged);\n+\tstrbuf_release(&buf);\n+}\n+\n+/* XXX test refs_for(oid) */\n+\n+int merged_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_merged_between);\n+\tRUN_TEST(test_merged);\n+\tRUN_TEST(test_default_write_opts);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nnew file mode 100644\nindex 000000000000..1a6d16915ab4\n--- /dev/null\n+++ b/reftable/reftable-merged.h\n@@ -0,0 +1,72 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_MERGED_H\n+#define REFTABLE_MERGED_H\n+\n+#include \"reftable-iterator.h\"\n+\n+/*\n+ * Merged tables\n+ *\n+ * A ref database kept in a sequence of table files. The merged_table presents a\n+ * unified view to reading (seeking, iterating) a sequence of immutable tables.\n+ *\n+ * The merged tables are on purpose kept disconnected from their actual storage\n+ * (eg. files on disk), because it is useful to merge tables aren't files. For\n+ * example, the per-workspace and global ref namespace can be implemented as a\n+ * merged table of two stacks of file-backed reftables.\n+ */\n+\n+/* A merged table is implements seeking/iterating over a stack of tables. */\n+struct reftable_merged_table;\n+\n+/* A generic reftable; see below. */\n+struct reftable_table;\n+\n+/* reftable_new_merged_table creates a new merged table. It takes ownership of\n+ the stack array.\n+*/\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id);\n+\n+/* returns an iterator positioned just before 'name' */\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns an iterator for log entry, at given update_index */\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index);\n+\n+/* like reftable_merged_table_seek_log_at but look for the newest entry. */\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns the max update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt);\n+\n+/* returns the min update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt);\n+\n+/* releases memory for the merged_table */\n+void reftable_merged_table_free(struct reftable_merged_table *m);\n+\n+/* return the hash ID of the merged table. */\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m);\n+\n+/* create a generic table from reftable_merged_table */\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *table);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 368eb7339eb7..cb277d911692 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -5,6 +5,7 @@ int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n+\tmerged_test_main(argc, argv);\n \tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treftable_test_main(argc, argv);\n-- \ngitgitgadget\n\n"},{"id":"421768","messageId":"5db7c8ab7f2307d2a1e91e8bd862e1144abbe9cd.1618255553.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 17/20] reftable: add dump utility","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:49Z","receivedAt":"2021-04-12T19:26:30Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nprovide a command-line utility for inspecting individual tables, and\ninspecting a complete ref database\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/dump.c | 212 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 212 insertions(+)\n create mode 100644 reftable/dump.c\n\ndiff --git a/reftable/dump.c b/reftable/dump.c\nnew file mode 100644\nindex 000000000000..00b444e8c9b8\n--- /dev/null\n+++ b/reftable/dump.c\n@@ -0,0 +1,212 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+\n+#include \"reftable.h\"\n+#include \"reftable-tests.h\"\n+\n+static uint32_t hash_id;\n+\n+static int dump_table(const char *tablename)\n+{\n+\tstruct reftable_block_source src = { 0 };\n+\tint err = reftable_block_source_from_file(&src, tablename);\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_ref_record ref = { 0 };\n+\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_reader *r = NULL;\n+\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\terr = reftable_new_reader(&r, &src, tablename);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\terr = reftable_reader_seek_ref(r, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_ref_record_print(&ref, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_clear(&ref);\n+\n+\terr = reftable_reader_seek_log(r, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_log_record_print(&log, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_clear(&log);\n+\n+\treftable_reader_free(r);\n+\treturn 0;\n+}\n+\n+static int compact_stack(const char *stackdir)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = {};\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_stack_compact_all(stack, NULL);\n+\tif (err < 0)\n+\t\tgoto done;\n+done:\n+\tif (stack != NULL) {\n+\t\treftable_stack_destroy(stack);\n+\t}\n+\treturn err;\n+}\n+\n+static int dump_stack(const char *stackdir)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = {};\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_ref_record ref = { 0 };\n+\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_merged_table *merged = NULL;\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tmerged = reftable_stack_merged_table(stack);\n+\n+\terr = reftable_merged_table_seek_ref(merged, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_ref_record_print(&ref, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_clear(&ref);\n+\n+\terr = reftable_merged_table_seek_log(merged, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_log_record_print(&log, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_clear(&log);\n+\n+\treftable_stack_destroy(stack);\n+\treturn 0;\n+}\n+\n+static void print_help(void)\n+{\n+\tprintf(\"usage: dump [-cst] arg\\n\\n\"\n+\t \"options: \\n\"\n+\t \" -c compact\\n\"\n+\t \" -t dump table\\n\"\n+\t \" -s dump stack\\n\"\n+\t \" -h this help\\n\"\n+\t \" -2 use SHA256\\n\"\n+\t \"\\n\");\n+}\n+\n+int reftable_dump_main(int argc, char *const *argv)\n+{\n+\tint err = 0;\n+\tint opt;\n+\tint opt_dump_table = 0;\n+\tint opt_dump_stack = 0;\n+\tint opt_compact = 0;\n+\tconst char *arg = NULL;\n+\twhile ((opt = getopt(argc, argv, \"2chts\")) != -1) {\n+\t\tswitch (opt) {\n+\t\tcase '2':\n+\t\t\thash_id = 0x73323536;\n+\t\t\tbreak;\n+\t\tcase 't':\n+\t\t\topt_dump_table = 1;\n+\t\t\tbreak;\n+\t\tcase 's':\n+\t\t\topt_dump_stack = 1;\n+\t\t\tbreak;\n+\t\tcase 'c':\n+\t\t\topt_compact = 1;\n+\t\t\tbreak;\n+\t\tcase '?':\n+\t\tcase 'h':\n+\t\t\tprint_help();\n+\t\t\treturn 2;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (argv[optind] == NULL) {\n+\t\tfprintf(stderr, \"need argument\\n\");\n+\t\tprint_help();\n+\t\treturn 2;\n+\t}\n+\n+\targ = argv[optind];\n+\n+\tif (opt_dump_table) {\n+\t\terr = dump_table(arg);\n+\t} else if (opt_dump_stack) {\n+\t\terr = dump_stack(arg);\n+\t} else if (opt_compact) {\n+\t\terr = compact_stack(arg);\n+\t}\n+\n+\tif (err < 0) {\n+\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv[0], arg,\n+\t\t\treftable_error_str(err));\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n-- \ngitgitgadget\n\n"},{"id":"421769","messageId":"3186b2b70f736561f2254588ac3dc44458fc3f3f.1618255553.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 15/20] reftable: implement refname validation","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:47Z","receivedAt":"2021-04-12T19:26:32Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe packed/loose format has restrictions on refnames: a and a/b cannot\ncoexist. This limitation does not apply to reftable per se, but must be\nmaintained for interoperability. This code adds validation routines to\nabort transactions that are trying to add invalid names.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/refname.c | 209 +++++++++++++++++++++++++++++++++++++++\n reftable/refname.h | 29 ++++++\n reftable/refname_test.c | 102 +++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 343 insertions(+)\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 129f73d118ac..9b941b69eb4b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2410,6 +2410,7 @@ REFTABLE_OBJS += reftable/merged.o\n REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/refname.o\n REFTABLE_OBJS += reftable/reftable.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n@@ -2420,6 +2421,7 @@ REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n+REFTABLE_TEST_OBJS += reftable/refname_test.o\n REFTABLE_TEST_OBJS += reftable/reftable_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\ndiff --git a/reftable/refname.c b/reftable/refname.c\nnew file mode 100644\nindex 000000000000..0f4eb3b292d3\n--- /dev/null\n+++ b/reftable/refname.c\n@@ -0,0 +1,209 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"reftable-error.h\"\n+#include \"basics.h\"\n+#include \"refname.h\"\n+#include \"reftable-iterator.h\"\n+\n+struct find_arg {\n+\tchar **names;\n+\tconst char *want;\n+};\n+\n+static int find_name(size_t k, void *arg)\n+{\n+\tstruct find_arg *f_arg = (struct find_arg *)arg;\n+\treturn strcmp(f_arg->names[k], f_arg->want) >= 0;\n+}\n+\n+static int modification_has_ref(struct modification *mod, const char *name)\n+{\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len && !strcmp(mod->add[idx], name)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tif (mod->del_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->del,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\tif (idx < mod->del_len && !strcmp(mod->del[idx], name)) {\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\terr = reftable_table_read_ref(&mod->tab, name, &ref);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+static void modification_release(struct modification *mod)\n+{\n+\t/* don't delete the strings themselves; they're owned by ref records.\n+\t */\n+\tFREE_AND_NULL(mod->add);\n+\tFREE_AND_NULL(mod->del);\n+\tmod->add_len = 0;\n+\tmod->del_len = 0;\n+}\n+\n+static int modification_has_ref_with_prefix(struct modification *mod,\n+\t\t\t\t\t const char *prefix)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = prefix,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len &&\n+\t\t !strncmp(prefix, mod->add[idx], strlen(prefix)))\n+\t\t\tgoto done;\n+\t}\n+\terr = reftable_table_seek_ref(&mod->tab, &it, prefix);\n+\tif (err)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\n+\t\tif (mod->del_len > 0) {\n+\t\t\tstruct find_arg arg = {\n+\t\t\t\t.names = mod->del,\n+\t\t\t\t.want = ref.refname,\n+\t\t\t};\n+\t\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\t\tif (idx < mod->del_len &&\n+\t\t\t !strcmp(ref.refname, mod->del[idx])) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (strncmp(ref.refname, prefix, strlen(prefix))) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_ref_record_release(&ref);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int validate_refname(const char *name)\n+{\n+\twhile (1) {\n+\t\tchar *next = strchr(name, '/');\n+\t\tif (!*name) {\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\t}\n+\t\tif (!next) {\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (next - name == 0 || (next - name == 1 && *name == '.') ||\n+\t\t (next - name == 2 && name[0] == '.' && name[1] == '.'))\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\tname = next + 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz)\n+{\n+\tstruct modification mod = {\n+\t\t.tab = tab,\n+\t\t.add = reftable_calloc(sizeof(char *) * sz),\n+\t\t.del = reftable_calloc(sizeof(char *) * sz),\n+\t};\n+\tint i = 0;\n+\tint err = 0;\n+\tfor (; i < sz; i++) {\n+\t\tif (reftable_ref_record_is_deletion(&recs[i])) {\n+\t\t\tmod.del[mod.del_len++] = recs[i].refname;\n+\t\t} else {\n+\t\t\tmod.add[mod.add_len++] = recs[i].refname;\n+\t\t}\n+\t}\n+\n+\terr = modification_validate(&mod);\n+\tmodification_release(&mod);\n+\treturn err;\n+}\n+\n+static void strbuf_trim_component(struct strbuf *sl)\n+{\n+\twhile (sl->len > 0) {\n+\t\tint is_slash = (sl->buf[sl->len - 1] == '/');\n+\t\tstrbuf_setlen(sl, sl->len - 1);\n+\t\tif (is_slash)\n+\t\t\tbreak;\n+\t}\n+}\n+\n+int modification_validate(struct modification *mod)\n+{\n+\tstruct strbuf slashed = STRBUF_INIT;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (; i < mod->add_len; i++) {\n+\t\terr = validate_refname(mod->add[i]);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\tstrbuf_addstr(&slashed, \"/\");\n+\n+\t\terr = modification_has_ref_with_prefix(mod, slashed.buf);\n+\t\tif (err == 0) {\n+\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\twhile (slashed.len) {\n+\t\t\tstrbuf_trim_component(&slashed);\n+\t\t\terr = modification_has_ref(mod, slashed.buf);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\terr = 0;\n+done:\n+\tstrbuf_release(&slashed);\n+\treturn err;\n+}\ndiff --git a/reftable/refname.h b/reftable/refname.h\nnew file mode 100644\nindex 000000000000..a24b40fcb428\n--- /dev/null\n+++ b/reftable/refname.h\n@@ -0,0 +1,29 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+#ifndef REFNAME_H\n+#define REFNAME_H\n+\n+#include \"reftable-record.h\"\n+#include \"reftable-generic.h\"\n+\n+struct modification {\n+\tstruct reftable_table tab;\n+\n+\tchar **add;\n+\tsize_t add_len;\n+\n+\tchar **del;\n+\tsize_t del_len;\n+};\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz);\n+\n+int modification_validate(struct modification *mod);\n+\n+#endif\ndiff --git a/reftable/refname_test.c b/reftable/refname_test.c\nnew file mode 100644\nindex 000000000000..5e005d6af31c\n--- /dev/null\n+++ b/reftable/refname_test.c\n@@ -0,0 +1,102 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"refname.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-writer.h\"\n+#include \"system.h\"\n+\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct testcase {\n+\tchar *add;\n+\tchar *del;\n+\tint error_code;\n+};\n+\n+static void test_conflict(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"a/b\",\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"destination\", /* make sure it's not a symref.\n+\t\t\t\t\t\t*/\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tint i;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct testcase cases[] = {\n+\t\t{ \"a/b/c\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"b\", NULL, 0 },\n+\t\t{ \"a\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"a\", \"a/b\", 0 },\n+\n+\t\t{ \"p/\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p//q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/./q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/../q\", NULL, REFTABLE_REFNAME_ERROR },\n+\n+\t\t{ \"a/b/c\", \"a/b\", 0 },\n+\t\t{ NULL, \"a//b\", 0 },\n+\t};\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\treftable_table_from_reader(&tab, rd);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct modification mod = {\n+\t\t\t.tab = tab,\n+\t\t};\n+\n+\t\tif (cases[i].add != NULL) {\n+\t\t\tmod.add = &cases[i].add;\n+\t\t\tmod.add_len = 1;\n+\t\t}\n+\t\tif (cases[i].del != NULL) {\n+\t\t\tmod.del = &cases[i].del;\n+\t\t\tmod.del_len = 1;\n+\t\t}\n+\n+\t\terr = modification_validate(&mod);\n+\t\tEXPECT(err == cases[i].error_code);\n+\t}\n+\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+int refname_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_conflict);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex cb277d911692..23e050eb5d23 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -8,6 +8,7 @@ int cmd__reftable(int argc, const char **argv)\n \tmerged_test_main(argc, argv);\n \tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\trefname_test_main(argc, argv);\n \treftable_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n-- \ngitgitgadget\n\n"},{"id":"421770","messageId":"b5492d5a13d7e843bb57725d2c8e36dcd35b7f80.1618255553.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 16/20] reftable: implement stack, a mutable database of reftable files.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:48Z","receivedAt":"2021-04-12T19:26:33Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/reftable-stack.h | 123 ++++\n reftable/stack.c | 1376 +++++++++++++++++++++++++++++++++++++\n reftable/stack.h | 41 ++\n reftable/stack_test.c | 928 +++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 6 files changed, 2471 insertions(+)\n create mode 100644 reftable/reftable-stack.h\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 9b941b69eb4b..e5ea8c6627ae 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2412,6 +2412,7 @@ REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/refname.o\n REFTABLE_OBJS += reftable/reftable.o\n+REFTABLE_OBJS += reftable/stack.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n REFTABLE_OBJS += reftable/zlib-compat.o\n@@ -2423,6 +2424,7 @@ REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/refname_test.o\n REFTABLE_TEST_OBJS += reftable/reftable_test.o\n+REFTABLE_TEST_OBJS += reftable/stack_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nnew file mode 100644\nindex 000000000000..2b363c908345\n--- /dev/null\n+++ b/reftable/reftable-stack.h\n@@ -0,0 +1,123 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_STACK_H\n+#define REFTABLE_STACK_H\n+\n+#include \"reftable-writer.h\"\n+\n+/*\n+ * The stack presents an interface to a mutable sequence of reftables.\n+\n+ * A stack can be mutated by pushing a table to the top of the stack.\n+\n+ * The reftable_stack automatically compacts files on disk to ensure good\n+ * amortized performance.\n+ *\n+ * For windows and other platforms that cannot have open files as rename\n+ * destinations, concurrent access from multiple processes needs the rand()\n+ * random seed to be randomized.\n+ */\n+struct reftable_stack;\n+\n+/* open a new reftable stack. The tables along with the table list will be\n+ * stored in 'dir'. Typically, this should be .git/reftables.\n+ */\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config);\n+\n+/* returns the update_index at which a next table should be written. */\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\n+\n+/* holds a transaction to add tables at the top of a stack. */\n+struct reftable_addition;\n+\n+/*\n+ * returns a new transaction to add reftables to the given stack. As a side\n+ * effect, the ref database is locked.\n+ */\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st);\n+\n+/* Adds a reftable to transaction. */\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+\n+/* Commits the transaction, releasing the lock. */\n+int reftable_addition_commit(struct reftable_addition *add);\n+\n+/* Release all non-committed data from the transaction, and deallocate the\n+ * transaction. Releases the lock if held. */\n+void reftable_addition_destroy(struct reftable_addition *add);\n+\n+/* add a new table to the stack. The write_table function must call\n+ * reftable_writer_set_limits, add refs and return an error value. */\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *write_arg),\n+\t\t void *write_arg);\n+\n+/* returns the merged_table for seeking. This table is valid until the\n+ * next write or reload, and should not be closed or deleted.\n+ */\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st);\n+\n+/* frees all resources associated with the stack. */\n+void reftable_stack_destroy(struct reftable_stack *st);\n+\n+/* Reloads the stack if necessary. This is very cheap to run if the stack was up\n+ * to date */\n+int reftable_stack_reload(struct reftable_stack *st);\n+\n+/* Policy for expiring reflog entries. */\n+struct reftable_log_expiry_config {\n+\t/* Drop entries older than this timestamp */\n+\tuint64_t time;\n+\n+\t/* Drop older entries */\n+\tuint64_t min_update_index;\n+};\n+\n+/* compacts all reftables into a giant table. Expire reflog entries if config is\n+ * non-NULL */\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config);\n+\n+/* heuristically compact unbalanced table stack. */\n+int reftable_stack_auto_compact(struct reftable_stack *st);\n+\n+/* delete stale .ref tables. */\n+int reftable_stack_clean(struct reftable_stack *st);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0 for\n+ * success, and 1 if ref not found. */\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* convenience function to read a single log. Returns < 0 for error, 0 for\n+ * success, and 1 if ref not found. */\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* statistics on past compactions. */\n+struct reftable_compaction_stats {\n+\tuint64_t bytes; /* total number of bytes written */\n+\tuint64_t entries_written; /* total number of entries written, including\n+\t\t\t\t failures. */\n+\tint attempts; /* how often we tried to compact */\n+\tint failures; /* failures happen on concurrent updates */\n+};\n+\n+/* return statistics for compaction up till now. */\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st);\n+\n+#endif\ndiff --git a/reftable/stack.c b/reftable/stack.c\nnew file mode 100644\nindex 000000000000..3cdb6e8ed33b\n--- /dev/null\n+++ b/reftable/stack.c\n@@ -0,0 +1,1376 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+#include \"merged.h\"\n+#include \"reader.h\"\n+#include \"refname.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-record.h\"\n+#include \"reftable-merged.h\"\n+#include \"writer.h\"\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config);\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name);\n+static void reftable_addition_close(struct reftable_addition *add);\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open);\n+\n+static void stack_filename(struct strbuf *dest, struct reftable_stack *st,\n+\t\t\t const char *name)\n+{\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, st->reftable_dir);\n+\tstrbuf_addstr(dest, \"/\");\n+\tstrbuf_addstr(dest, name);\n+}\n+\n+static int reftable_fd_write(void *arg, const void *data, size_t sz)\n+{\n+\tint *fdp = (int *)arg;\n+\treturn write(*fdp, data, sz);\n+}\n+\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config)\n+{\n+\tstruct reftable_stack *p =\n+\t\treftable_calloc(sizeof(struct reftable_stack));\n+\tstruct strbuf list_file_name = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tif (config.hash_id == 0) {\n+\t\tconfig.hash_id = SHA1_ID;\n+\t}\n+\n+\t*dest = NULL;\n+\n+\tstrbuf_reset(&list_file_name);\n+\tstrbuf_addstr(&list_file_name, dir);\n+\tstrbuf_addstr(&list_file_name, \"/tables.list\");\n+\n+\tp->list_file = strbuf_detach(&list_file_name, NULL);\n+\tp->reftable_dir = xstrdup(dir);\n+\tp->config = config;\n+\n+\terr = reftable_stack_reload_maybe_reuse(p, 1);\n+\tif (err < 0) {\n+\t\treftable_stack_destroy(p);\n+\t} else {\n+\t\t*dest = p;\n+\t}\n+\treturn err;\n+}\n+\n+static int fd_read_lines(int fd, char ***namesp)\n+{\n+\toff_t size = lseek(fd, 0, SEEK_END);\n+\tchar *buf = NULL;\n+\tint err = 0;\n+\tif (size < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\terr = lseek(fd, 0, SEEK_SET);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tbuf = reftable_malloc(size + 1);\n+\tif (read(fd, buf, size) != size) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\tbuf[size] = 0;\n+\n+\tparse_names(buf, size, namesp);\n+\n+done:\n+\treftable_free(buf);\n+\treturn err;\n+}\n+\n+int read_lines(const char *filename, char ***namesp)\n+{\n+\tint fd = open(filename, O_RDONLY, 0644);\n+\tint err = 0;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\t*namesp = reftable_calloc(sizeof(char *));\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\treturn REFTABLE_IO_ERROR;\n+\t}\n+\terr = fd_read_lines(fd, namesp);\n+\tclose(fd);\n+\treturn err;\n+}\n+\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st)\n+{\n+\treturn st->merged;\n+}\n+\n+static int has_name(char **names, const char *name)\n+{\n+\twhile (*names) {\n+\t\tif (!strcmp(*names, name))\n+\t\t\treturn 1;\n+\t\tnames++;\n+\t}\n+\treturn 0;\n+}\n+\n+/* Close and free the stack */\n+void reftable_stack_destroy(struct reftable_stack *st)\n+{\n+\tchar **names = NULL;\n+\tint err = 0;\n+\tif (st->merged != NULL) {\n+\t\treftable_merged_table_free(st->merged);\n+\t\tst->merged = NULL;\n+\t}\n+\n+\terr = read_lines(st->list_file, &names);\n+\tif (err < 0) {\n+\t\tFREE_AND_NULL(names);\n+\t}\n+\n+\tif (st->readers != NULL) {\n+\t\tint i = 0;\n+\t\tstruct strbuf filename = STRBUF_INIT;\n+\t\tfor (i = 0; i < st->readers_len; i++) {\n+\t\t\tconst char *name = reader_name(st->readers[i]);\n+\t\t\tstrbuf_reset(&filename);\n+\t\t\tif (names && !has_name(names, name)) {\n+\t\t\t\tstack_filename(&filename, st, name);\n+\t\t\t}\n+\t\t\treftable_reader_free(st->readers[i]);\n+\n+\t\t\tif (filename.len) {\n+\t\t\t\t// On Windows, can only unlink after closing.\n+\t\t\t\tunlink(filename.buf);\n+\t\t\t}\n+\t\t}\n+\t\tstrbuf_release(&filename);\n+\t\tst->readers_len = 0;\n+\t\tFREE_AND_NULL(st->readers);\n+\t}\n+\tFREE_AND_NULL(st->list_file);\n+\tFREE_AND_NULL(st->reftable_dir);\n+\treftable_free(st);\n+\tfree_names(names);\n+}\n+\n+static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n+\t\t\t\t\t\t int cur_len)\n+{\n+\tstruct reftable_reader **cur =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * cur_len);\n+\tint i = 0;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tcur[i] = st->readers[i];\n+\t}\n+\treturn cur;\n+}\n+\n+static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n+\t\t\t\t int reuse_open)\n+{\n+\tint cur_len = st->merged == NULL ? 0 : st->merged->stack_len;\n+\tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n+\tint err = 0;\n+\tint names_len = names_length(names);\n+\tstruct reftable_reader **new_readers =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * names_len);\n+\tstruct reftable_table *new_tables =\n+\t\treftable_calloc(sizeof(struct reftable_table) * names_len);\n+\tint new_readers_len = 0;\n+\tstruct reftable_merged_table *new_merged = NULL;\n+\tint i;\n+\n+\twhile (*names) {\n+\t\tstruct reftable_reader *rd = NULL;\n+\t\tchar *name = *names++;\n+\n+\t\t/* this is linear; we assume compaction keeps the number of\n+\t\t tables under control so this is not quadratic. */\n+\t\tint j = 0;\n+\t\tfor (j = 0; reuse_open && j < cur_len; j++) {\n+\t\t\tif (cur[j] != NULL && 0 == strcmp(cur[j]->name, name)) {\n+\t\t\t\trd = cur[j];\n+\t\t\t\tcur[j] = NULL;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (rd == NULL) {\n+\t\t\tstruct reftable_block_source src = { NULL };\n+\t\t\tstruct strbuf table_path = STRBUF_INIT;\n+\t\t\tstack_filename(&table_path, st, name);\n+\n+\t\t\terr = reftable_block_source_from_file(&src,\n+\t\t\t\t\t\t\t table_path.buf);\n+\t\t\tstrbuf_release(&table_path);\n+\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\n+\t\t\terr = reftable_new_reader(&rd, &src, name);\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\n+\t\tnew_readers[new_readers_len] = rd;\n+\t\treftable_table_from_reader(&new_tables[new_readers_len], rd);\n+\t\tnew_readers_len++;\n+\t}\n+\n+\t/* success! */\n+\terr = reftable_new_merged_table(&new_merged, new_tables,\n+\t\t\t\t\tnew_readers_len, st->config.hash_id);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tnew_tables = NULL;\n+\tst->readers_len = new_readers_len;\n+\tif (st->merged != NULL) {\n+\t\tmerged_table_release(st->merged);\n+\t\treftable_merged_table_free(st->merged);\n+\t}\n+\tif (st->readers != NULL) {\n+\t\treftable_free(st->readers);\n+\t}\n+\tst->readers = new_readers;\n+\tnew_readers = NULL;\n+\tnew_readers_len = 0;\n+\n+\tnew_merged->suppress_deletions = 1;\n+\tst->merged = new_merged;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tif (cur[i] != NULL) {\n+\t\t\tconst char *name = reader_name(cur[i]);\n+\t\t\tstruct strbuf filename = STRBUF_INIT;\n+\t\t\tstack_filename(&filename, st, name);\n+\n+\t\t\treader_close(cur[i]);\n+\t\t\treftable_reader_free(cur[i]);\n+\n+\t\t\t// On Windows, can only unlink after closing.\n+\t\t\tunlink(filename.buf);\n+\n+\t\t\tstrbuf_release(&filename);\n+\t\t}\n+\t}\n+\n+done:\n+\tfor (i = 0; i < new_readers_len; i++) {\n+\t\treader_close(new_readers[i]);\n+\t\treftable_reader_free(new_readers[i]);\n+\t}\n+\treftable_free(new_readers);\n+\treftable_free(new_tables);\n+\treftable_free(cur);\n+\treturn err;\n+}\n+\n+/* return negative if a before b. */\n+static int tv_cmp(struct timeval *a, struct timeval *b)\n+{\n+\ttime_t diff = a->tv_sec - b->tv_sec;\n+\tint udiff = a->tv_usec - b->tv_usec;\n+\n+\tif (diff != 0)\n+\t\treturn diff;\n+\n+\treturn udiff;\n+}\n+\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open)\n+{\n+\tstruct timeval deadline = { 0 };\n+\tint err = gettimeofday(&deadline, NULL);\n+\tint64_t delay = 0;\n+\tint tries = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tdeadline.tv_sec += 3;\n+\twhile (1) {\n+\t\tchar **names = NULL;\n+\t\tchar **names_after = NULL;\n+\t\tstruct timeval now = { 0 };\n+\t\tint err = gettimeofday(&now, NULL);\n+\t\tint err2 = 0;\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* Only look at deadlines after the first few times. This\n+\t\t simplifies debugging in GDB */\n+\t\ttries++;\n+\t\tif (tries > 3 && tv_cmp(&now, &deadline) >= 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = read_lines(st->list_file, &names);\n+\t\tif (err < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\t\terr = reftable_stack_reload_once(st, names, reuse_open);\n+\t\tif (err == 0) {\n+\t\t\tfree_names(names);\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err != REFTABLE_NOT_EXIST_ERROR) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* err == REFTABLE_NOT_EXIST_ERROR can be caused by a concurrent\n+\t\t writer. Check if there was one by checking if the name list\n+\t\t changed.\n+\t\t*/\n+\t\terr2 = read_lines(st->list_file, &names_after);\n+\t\tif (err2 < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err2;\n+\t\t}\n+\n+\t\tif (names_equal(names_after, names)) {\n+\t\t\tfree_names(names);\n+\t\t\tfree_names(names_after);\n+\t\t\treturn err;\n+\t\t}\n+\t\tfree_names(names);\n+\t\tfree_names(names_after);\n+\n+\t\tdelay = delay + (delay * rand()) / RAND_MAX + 1;\n+\t\tsleep_millisec(delay);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/* -1 = error\n+ 0 = up to date\n+ 1 = changed. */\n+static int stack_uptodate(struct reftable_stack *st)\n+{\n+\tchar **names = NULL;\n+\tint err = read_lines(st->list_file, &names);\n+\tint i = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tfor (i = 0; i < st->readers_len; i++) {\n+\t\tif (names[i] == NULL) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (strcmp(st->readers[i]->name, names[i])) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (names[st->merged->stack_len] != NULL) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tfree_names(names);\n+\treturn err;\n+}\n+\n+int reftable_stack_reload(struct reftable_stack *st)\n+{\n+\tint err = stack_uptodate(st);\n+\tif (err > 0)\n+\t\treturn reftable_stack_reload_maybe_reuse(st, 1);\n+\treturn err;\n+}\n+\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write)(struct reftable_writer *wr, void *arg),\n+\t\t void *arg)\n+{\n+\tint err = stack_try_add(st, write, arg);\n+\tif (err < 0) {\n+\t\tif (err == REFTABLE_LOCK_ERROR) {\n+\t\t\t/* Ignore error return, we want to propagate\n+\t\t\t REFTABLE_LOCK_ERROR.\n+\t\t\t*/\n+\t\t\treftable_stack_reload(st);\n+\t\t}\n+\t\treturn err;\n+\t}\n+\n+\tif (!st->disable_auto_compact)\n+\t\treturn reftable_stack_auto_compact(st);\n+\n+\treturn 0;\n+}\n+\n+static void format_name(struct strbuf *dest, uint64_t min, uint64_t max)\n+{\n+\tchar buf[100];\n+\tuint32_t rnd = (uint32_t)rand();\n+\tsnprintf(buf, sizeof(buf), \"0x%012\" PRIx64 \"-0x%012\" PRIx64 \"-%08x\",\n+\t\t min, max, rnd);\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, buf);\n+}\n+\n+struct reftable_addition {\n+\tint lock_file_fd;\n+\tstruct strbuf lock_file_name;\n+\tstruct reftable_stack *stack;\n+\n+\tchar **new_tables;\n+\tint new_tables_len;\n+\tuint64_t next_update_index;\n+};\n+\n+#define REFTABLE_ADDITION_INIT \\\n+\t{ \\\n+\t\t.lock_file_name = STRBUF_INIT \\\n+\t}\n+\n+static int reftable_stack_init_addition(struct reftable_addition *add,\n+\t\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tadd->stack = st;\n+\n+\tstrbuf_reset(&add->lock_file_name);\n+\tstrbuf_addstr(&add->lock_file_name, st->list_file);\n+\tstrbuf_addstr(&add->lock_file_name, \".lock\");\n+\n+\tadd->lock_file_fd = open(add->lock_file_name.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (add->lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = REFTABLE_LOCK_ERROR;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\terr = stack_uptodate(st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 1) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->next_update_index = reftable_stack_next_update_index(st);\n+done:\n+\tif (err) {\n+\t\treftable_addition_close(add);\n+\t}\n+\treturn err;\n+}\n+\n+static void reftable_addition_close(struct reftable_addition *add)\n+{\n+\tint i = 0;\n+\tstruct strbuf nm = STRBUF_INIT;\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstack_filename(&nm, add->stack, add->new_tables[i]);\n+\t\tunlink(nm.buf);\n+\t\treftable_free(add->new_tables[i]);\n+\t\tadd->new_tables[i] = NULL;\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\tif (add->lock_file_fd > 0) {\n+\t\tclose(add->lock_file_fd);\n+\t\tadd->lock_file_fd = 0;\n+\t}\n+\tif (add->lock_file_name.len > 0) {\n+\t\tunlink(add->lock_file_name.buf);\n+\t\tstrbuf_release(&add->lock_file_name);\n+\t}\n+\n+\tstrbuf_release(&nm);\n+}\n+\n+void reftable_addition_destroy(struct reftable_addition *add)\n+{\n+\tif (add == NULL) {\n+\t\treturn;\n+\t}\n+\treftable_addition_close(add);\n+\treftable_free(add);\n+}\n+\n+int reftable_addition_commit(struct reftable_addition *add)\n+{\n+\tstruct strbuf table_list = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\tif (add->new_tables_len == 0)\n+\t\tgoto done;\n+\n+\tfor (i = 0; i < add->stack->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->stack->readers[i]->name);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->new_tables[i]);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\n+\terr = write(add->lock_file_fd, table_list.buf, table_list.len);\n+\tstrbuf_release(&table_list);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = close(add->lock_file_fd);\n+\tadd->lock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(add->lock_file_name.buf, add->stack->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\t/* success, no more state to clean up. */\n+\tstrbuf_release(&add->lock_file_name);\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\treftable_free(add->new_tables[i]);\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\terr = reftable_stack_reload(add->stack);\n+done:\n+\treftable_addition_close(add);\n+\treturn err;\n+}\n+\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n+\t*dest = reftable_calloc(sizeof(**dest));\n+\t**dest = empty;\n+\terr = reftable_stack_init_addition(*dest, st);\n+\tif (err) {\n+\t\treftable_free(*dest);\n+\t\t*dest = NULL;\n+\t}\n+\treturn err;\n+}\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct reftable_addition add = REFTABLE_ADDITION_INIT;\n+\tint err = reftable_stack_init_addition(&add, st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tif (err > 0) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_addition_add(&add, write_table, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_addition_commit(&add);\n+done:\n+\treftable_addition_close(&add);\n+\treturn err;\n+}\n+\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf tab_file_name = STRBUF_INIT;\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\tint tab_fd = 0;\n+\n+\tstrbuf_reset(&next_name);\n+\tformat_name(&next_name, add->next_update_index, add->next_update_index);\n+\n+\tstack_filename(&temp_tab_file_name, add->stack, next_name.buf);\n+\tstrbuf_addstr(&temp_tab_file_name, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab_file_name.buf);\n+\tif (tab_fd < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd,\n+\t\t\t\t &add->stack->config);\n+\terr = write_table(wr, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_writer_close(wr);\n+\tif (err == REFTABLE_EMPTY_TABLE_ERROR) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = stack_check_addition(add->stack, temp_tab_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (wr->min_update_index < add->next_update_index) {\n+\t\terr = REFTABLE_API_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tformat_name(&next_name, wr->min_update_index, wr->max_update_index);\n+\tstrbuf_addstr(&next_name, \".ref\");\n+\n+\tstack_filename(&tab_file_name, add->stack, next_name.buf);\n+\n+\t/*\n+\t On windows, this relies on rand() picking a unique destination name.\n+\t Maybe we should do retry loop as well?\n+\t */\n+\terr = rename(temp_tab_file_name.buf, tab_file_name.buf);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->new_tables = reftable_realloc(add->new_tables,\n+\t\t\t\t\t sizeof(*add->new_tables) *\n+\t\t\t\t\t\t (add->new_tables_len + 1));\n+\tadd->new_tables[add->new_tables_len] = strbuf_detach(&next_name, NULL);\n+\tadd->new_tables_len++;\n+done:\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (temp_tab_file_name.len > 0) {\n+\t\tunlink(temp_tab_file_name.buf);\n+\t}\n+\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&tab_file_name);\n+\tstrbuf_release(&next_name);\n+\treftable_writer_free(wr);\n+\treturn err;\n+}\n+\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st)\n+{\n+\tint sz = st->merged->stack_len;\n+\tif (sz > 0)\n+\t\treturn reftable_reader_max_update_index(st->readers[sz - 1]) +\n+\t\t 1;\n+\treturn 1;\n+}\n+\n+static int stack_compact_locked(struct reftable_stack *st, int first, int last,\n+\t\t\t\tstruct strbuf *temp_tab,\n+\t\t\t\tstruct reftable_log_expiry_config *config)\n+{\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tint tab_fd = -1;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\n+\tformat_name(&next_name,\n+\t\t reftable_reader_min_update_index(st->readers[first]),\n+\t\t reftable_reader_max_update_index(st->readers[last]));\n+\n+\tstack_filename(temp_tab, st, next_name.buf);\n+\tstrbuf_addstr(temp_tab, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab->buf);\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd, &st->config);\n+\n+\terr = stack_write_compact(st, wr, first, last, config);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = reftable_writer_close(wr);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\n+done:\n+\treftable_writer_free(wr);\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (err != 0 && temp_tab->len > 0) {\n+\t\tunlink(temp_tab->buf);\n+\t\tstrbuf_release(temp_tab);\n+\t}\n+\tstrbuf_release(&next_name);\n+\treturn err;\n+}\n+\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint subtabs_len = last - first + 1;\n+\tstruct reftable_table *subtabs = reftable_calloc(\n+\t\tsizeof(struct reftable_table) * (last - first + 1));\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tuint64_t entries = 0;\n+\n+\tint i = 0, j = 0;\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct reftable_reader *t = st->readers[i];\n+\t\treftable_table_from_reader(&subtabs[j++], t);\n+\t\tst->stats.bytes += t->size;\n+\t}\n+\treftable_writer_set_limits(wr, st->readers[first]->min_update_index,\n+\t\t\t\t st->readers[last]->max_update_index);\n+\n+\terr = reftable_new_merged_table(&mt, subtabs, subtabs_len,\n+\t\t\t\t\tst->config.hash_id);\n+\tif (err < 0) {\n+\t\treftable_free(subtabs);\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_merged_table_seek_ref(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (first == 0 && reftable_ref_record_is_deletion(&ref)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_ref(wr, &ref);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_merged_table_seek_log(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (first == 0 && reftable_log_record_is_deletion(&log)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config != NULL && config->min_update_index > 0 &&\n+\t\t log.update_index < config->min_update_index) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config != NULL && config->time > 0 &&\n+\t\t log.update.time < config->time) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_log(wr, &log);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\tif (mt != NULL) {\n+\t\tmerged_table_release(mt);\n+\t\treftable_merged_table_free(mt);\n+\t}\n+\treftable_ref_record_release(&ref);\n+\treftable_log_record_release(&log);\n+\tst->stats.entries_written += entries;\n+\treturn err;\n+}\n+\n+/* < 0: error. 0 == OK, > 0 attempt failed; could retry. */\n+static int stack_compact_range(struct reftable_stack *st, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *expiry)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf new_table_name = STRBUF_INIT;\n+\tstruct strbuf lock_file_name = STRBUF_INIT;\n+\tstruct strbuf ref_list_contents = STRBUF_INIT;\n+\tstruct strbuf new_table_path = STRBUF_INIT;\n+\tint err = 0;\n+\tint have_lock = 0;\n+\tint lock_file_fd = 0;\n+\tint compact_count = last - first + 1;\n+\tchar **listp = NULL;\n+\tchar **delete_on_success =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tchar **subtable_locks =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tint i = 0;\n+\tint j = 0;\n+\tint is_empty_table = 0;\n+\n+\tif (first > last || (expiry == NULL && first == last)) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\tst->stats.attempts++;\n+\n+\tstrbuf_reset(&lock_file_name);\n+\tstrbuf_addstr(&lock_file_name, st->list_file);\n+\tstrbuf_addstr(&lock_file_name, \".lock\");\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\t/* Don't want to write to the lock for now. */\n+\tclose(lock_file_fd);\n+\tlock_file_fd = 0;\n+\n+\thave_lock = 1;\n+\terr = stack_uptodate(st);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct strbuf subtab_file_name = STRBUF_INIT;\n+\t\tstruct strbuf subtab_lock = STRBUF_INIT;\n+\t\tint sublock_file_fd = -1;\n+\n+\t\tstack_filename(&subtab_file_name, st,\n+\t\t\t reader_name(st->readers[i]));\n+\n+\t\tstrbuf_reset(&subtab_lock);\n+\t\tstrbuf_addbuf(&subtab_lock, &subtab_file_name);\n+\t\tstrbuf_addstr(&subtab_lock, \".lock\");\n+\n+\t\tsublock_file_fd = open(subtab_lock.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\t\tif (sublock_file_fd > 0) {\n+\t\t\tclose(sublock_file_fd);\n+\t\t} else if (sublock_file_fd < 0) {\n+\t\t\tif (errno == EEXIST) {\n+\t\t\t\terr = 1;\n+\t\t\t} else {\n+\t\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\t}\n+\t\t}\n+\n+\t\tsubtable_locks[j] = subtab_lock.buf;\n+\t\tdelete_on_success[j] = subtab_file_name.buf;\n+\t\tj++;\n+\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\terr = unlink(lock_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\thave_lock = 0;\n+\n+\terr = stack_compact_locked(st, first, last, &temp_tab_file_name,\n+\t\t\t\t expiry);\n+\t/* Compaction + tombstones can create an empty table out of non-empty\n+\t * tables. */\n+\tis_empty_table = (err == REFTABLE_EMPTY_TABLE_ERROR);\n+\tif (is_empty_table) {\n+\t\terr = 0;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\thave_lock = 1;\n+\n+\tformat_name(&new_table_name, st->readers[first]->min_update_index,\n+\t\t st->readers[last]->max_update_index);\n+\tstrbuf_addstr(&new_table_name, \".ref\");\n+\n+\tstack_filename(&new_table_path, st, new_table_name.buf);\n+\n+\tif (!is_empty_table) {\n+\t\t/* retry? */\n+\t\terr = rename(temp_tab_file_name.buf, new_table_path.buf);\n+\t\tif (err < 0) {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < first; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tif (!is_empty_table) {\n+\t\tstrbuf_addbuf(&ref_list_contents, &new_table_name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tfor (i = last + 1; i < st->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\n+\terr = write(lock_file_fd, ref_list_contents.buf, ref_list_contents.len);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\terr = close(lock_file_fd);\n+\tlock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(lock_file_name.buf, st->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\thave_lock = 0;\n+\n+\t/* Reload the stack before deleting. On windows, we can only delete the\n+\t files after we closed them.\n+\t*/\n+\terr = reftable_stack_reload_maybe_reuse(st, first < last);\n+\n+\tlistp = delete_on_success;\n+\twhile (*listp) {\n+\t\tif (strcmp(*listp, new_table_path.buf)) {\n+\t\t\tunlink(*listp);\n+\t\t}\n+\t\tlistp++;\n+\t}\n+\n+done:\n+\tfree_names(delete_on_success);\n+\n+\tlistp = subtable_locks;\n+\twhile (*listp) {\n+\t\tunlink(*listp);\n+\t\tlistp++;\n+\t}\n+\tfree_names(subtable_locks);\n+\tif (lock_file_fd > 0) {\n+\t\tclose(lock_file_fd);\n+\t\tlock_file_fd = 0;\n+\t}\n+\tif (have_lock) {\n+\t\tunlink(lock_file_name.buf);\n+\t}\n+\tstrbuf_release(&new_table_name);\n+\tstrbuf_release(&new_table_path);\n+\tstrbuf_release(&ref_list_contents);\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&lock_file_name);\n+\treturn err;\n+}\n+\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\treturn stack_compact_range(st, 0, st->merged->stack_len - 1, config);\n+}\n+\n+static int stack_compact_range_stats(struct reftable_stack *st, int first,\n+\t\t\t\t int last,\n+\t\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint err = stack_compact_range(st, first, last, config);\n+\tif (err > 0) {\n+\t\tst->stats.failures++;\n+\t}\n+\treturn err;\n+}\n+\n+static int segment_size(struct segment *s)\n+{\n+\treturn s->end - s->start;\n+}\n+\n+int fastlog2(uint64_t sz)\n+{\n+\tint l = 0;\n+\tif (sz == 0)\n+\t\treturn 0;\n+\tfor (; sz; sz /= 2) {\n+\t\tl++;\n+\t}\n+\treturn l - 1;\n+}\n+\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n+{\n+\tstruct segment *segs = reftable_calloc(sizeof(struct segment) * n);\n+\tint next = 0;\n+\tstruct segment cur = { 0 };\n+\tint i = 0;\n+\n+\tif (n == 0) {\n+\t\t*seglen = 0;\n+\t\treturn segs;\n+\t}\n+\tfor (i = 0; i < n; i++) {\n+\t\tint log = fastlog2(sizes[i]);\n+\t\tif (cur.log != log && cur.bytes > 0) {\n+\t\t\tstruct segment fresh = {\n+\t\t\t\t.start = i,\n+\t\t\t};\n+\n+\t\t\tsegs[next++] = cur;\n+\t\t\tcur = fresh;\n+\t\t}\n+\n+\t\tcur.log = log;\n+\t\tcur.end = i + 1;\n+\t\tcur.bytes += sizes[i];\n+\t}\n+\tsegs[next++] = cur;\n+\t*seglen = next;\n+\treturn segs;\n+}\n+\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, sizes, n);\n+\tstruct segment min_seg = {\n+\t\t.log = 64,\n+\t};\n+\tint i = 0;\n+\tfor (i = 0; i < seglen; i++) {\n+\t\tif (segment_size(&segs[i]) == 1) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (segs[i].log < min_seg.log) {\n+\t\t\tmin_seg = segs[i];\n+\t\t}\n+\t}\n+\n+\twhile (min_seg.start > 0) {\n+\t\tint prev = min_seg.start - 1;\n+\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmin_seg.start = prev;\n+\t\tmin_seg.bytes += sizes[prev];\n+\t}\n+\n+\treftable_free(segs);\n+\treturn min_seg;\n+}\n+\n+static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes =\n+\t\treftable_calloc(sizeof(uint64_t) * st->merged->stack_len);\n+\tint version = (st->config.hash_id == SHA1_ID) ? 1 : 2;\n+\tint overhead = header_size(version) - 1;\n+\tint i = 0;\n+\tfor (i = 0; i < st->merged->stack_len; i++) {\n+\t\tsizes[i] = st->readers[i]->size - overhead;\n+\t}\n+\treturn sizes;\n+}\n+\n+int reftable_stack_auto_compact(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n+\tstruct segment seg =\n+\t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n+\treftable_free(sizes);\n+\tif (segment_size(&seg) > 0)\n+\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n+\t\t\t\t\t\t NULL);\n+\n+\treturn 0;\n+}\n+\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st)\n+{\n+\treturn &st->stats;\n+}\n+\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_table tab = { NULL };\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\treturn reftable_table_read_ref(&tab, refname, ref);\n+}\n+\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(st);\n+\tint err = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_log(&it, log);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(log->refname, refname) ||\n+\t reftable_log_record_is_deletion(log)) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tif (err) {\n+\t\treftable_log_record_release(log);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name)\n+{\n+\tint err = 0;\n+\tstruct reftable_block_source src = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct reftable_ref_record *refs = NULL;\n+\tstruct reftable_iterator it = { NULL };\n+\tint cap = 0;\n+\tint len = 0;\n+\tint i = 0;\n+\n+\tif (st->config.skip_name_check)\n+\t\treturn 0;\n+\n+\terr = reftable_block_source_from_file(&src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&rd, &src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tif (err > 0) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (len >= cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\trefs = reftable_realloc(refs, cap * sizeof(refs[0]));\n+\t\t}\n+\n+\t\trefs[len++] = ref;\n+\t}\n+\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\n+\terr = validate_ref_record_addition(tab, refs, len);\n+\n+done:\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t}\n+\n+\tfree(refs);\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\treturn err;\n+}\n+\n+static int is_table_name(const char *s)\n+{\n+\tconst char *dot = strrchr(s, '.');\n+\treturn dot != NULL && !strcmp(dot, \".ref\");\n+}\n+\n+static void remove_maybe_stale_table(struct reftable_stack *st, uint64_t max,\n+\t\t\t\t const char *name)\n+{\n+\tint err = 0;\n+\tuint64_t update_idx = 0;\n+\tstruct reftable_block_source src = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct strbuf table_path = STRBUF_INIT;\n+\tstack_filename(&table_path, st, name);\n+\n+\terr = reftable_block_source_from_file(&src, table_path.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&rd, &src, name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tupdate_idx = reftable_reader_max_update_index(rd);\n+\treftable_reader_free(rd);\n+\n+\tif (update_idx <= max) {\n+\t\tunlink(table_path.buf);\n+\t}\n+done:\n+\tstrbuf_release(&table_path);\n+}\n+\n+static int reftable_stack_clean_locked(struct reftable_stack *st)\n+{\n+\tuint64_t max = reftable_merged_table_max_update_index(\n+\t\treftable_stack_merged_table(st));\n+\tDIR *dir = opendir(st->reftable_dir);\n+\tstruct dirent *d = NULL;\n+\tif (dir == NULL) {\n+\t\treturn REFTABLE_IO_ERROR;\n+\t}\n+\n+\twhile ((d = readdir(dir)) != NULL) {\n+\t\tint i = 0;\n+\t\tint found = 0;\n+\t\tif (!is_table_name(d->d_name))\n+\t\t\tcontinue;\n+\n+\t\tfor (i = 0; !found && i < st->readers_len; i++) {\n+\t\t\tfound = !strcmp(reader_name(st->readers[i]), d->d_name);\n+\t\t}\n+\t\tif (found)\n+\t\t\tcontinue;\n+\n+\t\tremove_maybe_stale_table(st, max, d->d_name);\n+\t}\n+\n+\tclosedir(dir);\n+\treturn 0;\n+}\n+\n+int reftable_stack_clean(struct reftable_stack *st)\n+{\n+\tstruct reftable_addition *add = NULL;\n+\tint err = reftable_stack_new_addition(&add, st);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(st);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_clean_locked(st);\n+\n+done:\n+\treftable_addition_destroy(add);\n+\treturn err;\n+}\ndiff --git a/reftable/stack.h b/reftable/stack.h\nnew file mode 100644\nindex 000000000000..f57005846e56\n--- /dev/null\n+++ b/reftable/stack.h\n@@ -0,0 +1,41 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef STACK_H\n+#define STACK_H\n+\n+#include \"system.h\"\n+#include \"reftable-writer.h\"\n+#include \"reftable-stack.h\"\n+\n+struct reftable_stack {\n+\tchar *list_file;\n+\tchar *reftable_dir;\n+\tint disable_auto_compact;\n+\n+\tstruct reftable_write_options config;\n+\n+\tstruct reftable_reader **readers;\n+\tsize_t readers_len;\n+\tstruct reftable_merged_table *merged;\n+\tstruct reftable_compaction_stats stats;\n+};\n+\n+int read_lines(const char *filename, char ***lines);\n+\n+struct segment {\n+\tint start, end;\n+\tint log;\n+\tuint64_t bytes;\n+};\n+\n+int fastlog2(uint64_t sz);\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n);\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n);\n+\n+#endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nnew file mode 100644\nindex 000000000000..5e1d5f941524\n--- /dev/null\n+++ b/reftable/stack_test.c\n@@ -0,0 +1,928 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+\n+#include \"reftable-reader.h\"\n+#include \"merged.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+#include \n+#include \n+\n+static void clear_dir(const char *dirname)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstrbuf_addstr(&path, dirname);\n+\tremove_dir_recursively(&path, 0);\n+\tstrbuf_release(&path);\n+}\n+\n+static int count_dir_entries(const char *dirname)\n+{\n+\tDIR *dir = opendir(dirname);\n+\tint len = 0;\n+\tstruct dirent *d;\n+\tif (dir == NULL)\n+\t\treturn 0;\n+\n+\twhile ((d = readdir(dir)) != NULL) {\n+\t\tif (!strcmp(d->d_name, \"..\") || !strcmp(d->d_name, \".\"))\n+\t\t\tcontinue;\n+\t\tlen++;\n+\t}\n+\tclosedir(dir);\n+\treturn len;\n+}\n+\n+static char *get_tmp_template(const char *prefix)\n+{\n+\tconst char *tmp = getenv(\"TMPDIR\");\n+\tstatic char template[1024];\n+\tsnprintf(template, sizeof(template) - 1, \"%s/%s.XXXXXX\",\n+\t\t tmp ? tmp : \"/tmp\", prefix);\n+\treturn template;\n+}\n+\n+static void test_read_file(void)\n+{\n+\tchar *fn = get_tmp_template(__FUNCTION__);\n+\tint fd = mkstemp(fn);\n+\tchar out[1024] = \"line1\\n\\nline2\\nline3\";\n+\tint n, err;\n+\tchar **names = NULL;\n+\tchar *want[] = { \"line1\", \"line2\", \"line3\" };\n+\tint i = 0;\n+\n+\tEXPECT(fd > 0);\n+\tn = write(fd, out, strlen(out));\n+\tEXPECT(n == strlen(out));\n+\terr = close(fd);\n+\tEXPECT(err >= 0);\n+\n+\terr = read_lines(fn, &names);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; names[i] != NULL; i++) {\n+\t\tEXPECT(0 == strcmp(want[i], names[i]));\n+\t}\n+\tfree_names(names);\n+\tremove(fn);\n+}\n+\n+static void test_parse_names(void)\n+{\n+\tchar buf[] = \"line\\n\";\n+\tchar **names = NULL;\n+\tparse_names(buf, strlen(buf), &names);\n+\n+\tEXPECT(NULL != names[0]);\n+\tEXPECT(0 == strcmp(names[0], \"line\"));\n+\tEXPECT(NULL == names[1]);\n+\tfree_names(names);\n+}\n+\n+static void test_names_equal(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", \"c\", NULL };\n+\tchar *b[] = { \"a\", \"b\", \"d\", NULL };\n+\tchar *c[] = { \"a\", \"b\", NULL };\n+\n+\tEXPECT(names_equal(a, a));\n+\tEXPECT(!names_equal(a, b));\n+\tEXPECT(!names_equal(a, c));\n+}\n+\n+static int write_test_ref(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct reftable_ref_record *ref = arg;\n+\treftable_writer_set_limits(wr, ref->update_index, ref->update_index);\n+\treturn reftable_writer_add_ref(wr, ref);\n+}\n+\n+struct write_log_arg {\n+\tstruct reftable_log_record *log;\n+\tuint64_t update_index;\n+};\n+\n+static int write_test_log(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct write_log_arg *wla = arg;\n+\n+\treftable_writer_set_limits(wr, wla->update_index, wla->update_index);\n+\treturn reftable_writer_add_log(wr, wla->log);\n+}\n+\n+static void test_reftable_stack_add_one(void)\n+{\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(\"master\", dest.value.symref));\n+\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_uptodate(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL;\n+\tstruct reftable_stack *st2 = NULL;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"branch2\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\t/* simulate multi-process access to the same stack\n+\t by creating two stacks for the same directory.\n+\t */\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st1, &write_test_ref, &ref1);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tEXPECT(err == REFTABLE_LOCK_ERROR);\n+\n+\terr = reftable_stack_reload(st2);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tEXPECT_ERR(err);\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_transaction_api(void)\n+{\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_addition *add = NULL;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_new_addition(&add, st);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_addition_add(add, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_addition_commit(add);\n+\tEXPECT_ERR(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(REFTABLE_REF_SYMREF == dest.value_type);\n+\tEXPECT(0 == strcmp(\"master\", dest.value.symref));\n+\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_validate_refname(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tint i;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"a/b\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tchar *additions[] = { \"a\", \"a/b/c\" };\n+\n+\tEXPECT(mkdtemp(dir));\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(additions); i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = additions[i],\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tEXPECT(err == REFTABLE_NAME_CONFLICT);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static int write_error(struct reftable_writer *wr, void *arg)\n+{\n+\treturn *((int *)arg);\n+}\n+\n+static void test_reftable_stack_update_index_check(void)\n+{\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"name1\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"name2\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref1);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref2);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_lock_failure(void)\n+{\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err, i;\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tfor (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) {\n+\t\terr = reftable_stack_add(st, &write_error, &i);\n+\t\tEXPECT(err == i);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_add(void)\n+{\n+\tint i = 0;\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t.exact_log_message = 1,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tst->disable_auto_compact = 1;\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n+\t\trefs[i].value.val1 = reftable_malloc(SHA1_SIZE);\n+\t\tset_test_hash(refs[i].value.val1, i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = N + i + 1;\n+\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\n+\t\tlogs[i].update.new_hash = reftable_malloc(SHA1_SIZE);\n+\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].update.new_hash, i);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_ref_record dest = { NULL };\n+\n+\t\tint err = reftable_stack_read_ref(st, refs[i].refname, &dest);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(reftable_ref_record_equal(&dest, refs + i, SHA1_SIZE));\n+\t\treftable_ref_record_release(&dest);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_log_record dest = { NULL };\n+\t\tint err = reftable_stack_read_log(st, refs[i].refname, &dest);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(reftable_log_record_equal(&dest, logs + i, SHA1_SIZE));\n+\t\treftable_log_record_release(&dest);\n+\t}\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_log_normalize(void)\n+{\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t0,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\n+\tuint8_t h1[SHA1_SIZE] = { 0x01 }, h2[SHA1_SIZE] = { 0x02 };\n+\n+\tstruct reftable_log_record input = { .refname = \"branch\",\n+\t\t\t\t\t .update_index = 1,\n+\t\t\t\t\t .value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t\t\t .update = {\n+\t\t\t\t\t\t .new_hash = h1,\n+\t\t\t\t\t\t .old_hash = h2,\n+\t\t\t\t\t } };\n+\tstruct reftable_log_record dest = {\n+\t\t.update_index = 0,\n+\t};\n+\tstruct write_log_arg arg = {\n+\t\t.log = &input,\n+\t\t.update_index = 1,\n+\t};\n+\n+\tEXPECT(mkdtemp(dir));\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tinput.update.message = \"one\\ntwo\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\n+\tinput.update.message = \"one\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(dest.update.message, \"one\\n\"));\n+\n+\tinput.update.message = \"two\\n\";\n+\targ.update_index = 2;\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT_ERR(err);\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(dest.update.message, \"two\\n\"));\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\treftable_log_record_release(&dest);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_tombstone(void)\n+{\n+\tint i = 0;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\tstruct reftable_ref_record dest = { NULL };\n+\tstruct reftable_log_record log_dest = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tconst char *buf = \"branch\";\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\tif (i % 2 == 0) {\n+\t\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n+\t\t\trefs[i].value.val1 = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_test_hash(refs[i].value.val1, i);\n+\t\t}\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\t/* update_index is part of the key. */\n+\t\tlogs[i].update_index = 42;\n+\t\tif (i % 2 == 0) {\n+\t\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\t\t\tlogs[i].update.new_hash = reftable_malloc(SHA1_SIZE);\n+\t\t\tset_test_hash(logs[i].update.new_hash, i);\n+\t\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n+\t\t}\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tEXPECT(err == 1);\n+\treftable_ref_record_release(&dest);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tEXPECT(err == 1);\n+\treftable_log_record_release(&log_dest);\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tEXPECT(err == 1);\n+\treftable_ref_record_release(&dest);\n+\treftable_log_record_release(&log_dest);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_hash_id(void)\n+{\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"master\",\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"target\",\n+\t\t.update_index = 1,\n+\t};\n+\tstruct reftable_write_options cfg32 = { .hash_id = SHA256_ID };\n+\tstruct reftable_stack *st32 = NULL;\n+\tstruct reftable_write_options cfg_default = { 0 };\n+\tstruct reftable_stack *st_default = NULL;\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\t/* can't read it with the wrong hash ID. */\n+\terr = reftable_new_stack(&st32, dir, cfg32);\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+\n+\t/* check that we can read it back with default config too. */\n+\terr = reftable_new_stack(&st_default, dir, cfg_default);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n+\tEXPECT_ERR(err);\n+\n+\tEXPECT(reftable_ref_record_equal(&ref, &dest, SHA1_SIZE));\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st_default);\n+\tclear_dir(dir);\n+}\n+\n+static void test_log2(void)\n+{\n+\tEXPECT(1 == fastlog2(3));\n+\tEXPECT(2 == fastlog2(4));\n+\tEXPECT(2 == fastlog2(5));\n+}\n+\n+static void test_sizes_to_segments(void)\n+{\n+\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n+\t/* .................0 1 2 3 4 5 */\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(segs[2].log == 3);\n+\tEXPECT(segs[2].start == 5);\n+\tEXPECT(segs[2].end == 6);\n+\n+\tEXPECT(segs[1].log == 2);\n+\tEXPECT(segs[1].start == 2);\n+\tEXPECT(segs[1].end == 5);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_empty(void)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n+\tEXPECT(seglen == 0);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_all_equal(void)\n+{\n+\tuint64_t sizes[] = { 5, 5 };\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(seglen == 1);\n+\tEXPECT(segs[0].start == 0);\n+\tEXPECT(segs[0].end == 2);\n+\treftable_free(segs);\n+}\n+\n+static void test_suggest_compaction_segment(void)\n+{\n+\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n+\t/* .................0 1 2 3 4 5 6 */\n+\tstruct segment min =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(min.start == 2);\n+\tEXPECT(min.end == 7);\n+}\n+\n+static void test_suggest_compaction_segment_nothing(void)\n+{\n+\tuint64_t sizes[] = { 64, 32, 16, 8, 4, 2 };\n+\tstruct segment result =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(result.start == result.end);\n+}\n+\n+static void test_reflog_expire(void)\n+{\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tstruct reftable_log_record logs[20] = { { NULL } };\n+\tint N = ARRAY_SIZE(logs) - 1;\n+\tint i = 0;\n+\tint err;\n+\tstruct reftable_log_expiry_config expiry = {\n+\t\t.time = 10,\n+\t};\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = i;\n+\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\t\tlogs[i].update.time = i;\n+\t\tlogs[i].update.new_hash = reftable_malloc(SHA1_SIZE);\n+\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].update.new_hash, i);\n+\t}\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, logs[9].refname, &log);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[11].refname, &log);\n+\tEXPECT_ERR(err);\n+\n+\texpiry.min_update_index = 15;\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, logs[14].refname, &log);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[16].refname, &log);\n+\tEXPECT_ERR(err);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i <= N; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+\treftable_log_record_release(&log);\n+}\n+\n+static int write_nothing(struct reftable_writer *wr, void *arg)\n+{\n+\treftable_writer_set_limits(wr, 1, 1);\n+\treturn 0;\n+}\n+\n+static void test_empty_add(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tstruct reftable_stack *st2 = NULL;\n+\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_nothing, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tclear_dir(dir);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st2);\n+}\n+\n+static void test_reftable_stack_auto_compaction(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tint err, i;\n+\tint N = 100;\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\n+\t\tEXPECT(i < 3 || st->merged->stack_len < 2 * fastlog2(i));\n+\t}\n+\n+\tEXPECT(reftable_stack_compaction_stats(st)->entries_written <\n+\t (uint64_t)(N * fastlog2(N)));\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_compaction_concurrent(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL, *st2 = NULL;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tint err, i;\n+\tint N = 3;\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st1),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st1, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st1, NULL);\n+\tEXPECT_ERR(err);\n+\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\n+\tEXPECT(count_dir_entries(dir) == 2);\n+\tclear_dir(dir);\n+}\n+\n+static void unclean_stack_close(struct reftable_stack *st)\n+{\n+\t// break abstraction boundary to simulate unclean shutdown.\n+\tint i = 0;\n+\tfor (; i < st->readers_len; i++) {\n+\t\treftable_reader_free(st->readers[i]);\n+\t}\n+\tst->readers_len = 0;\n+\tFREE_AND_NULL(st->readers);\n+}\n+\n+static void test_reftable_stack_compaction_concurrent_clean(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL, *st2 = NULL, *st3 = NULL;\n+\tchar *dir = get_tmp_template(__FUNCTION__);\n+\tint err, i;\n+\tint N = 3;\n+\tEXPECT(mkdtemp(dir));\n+\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st1),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st1, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st1, NULL);\n+\tEXPECT_ERR(err);\n+\n+\tunclean_stack_close(st1);\n+\tunclean_stack_close(st2);\n+\n+\terr = reftable_new_stack(&st3, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_clean(st3);\n+\tEXPECT_ERR(err);\n+\tEXPECT(count_dir_entries(dir) == 2);\n+\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\treftable_stack_destroy(st3);\n+\n+\tclear_dir(dir);\n+}\n+\n+int stack_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_reftable_stack_compaction_concurrent_clean);\n+\tRUN_TEST(test_reftable_stack_compaction_concurrent);\n+\tRUN_TEST(test_reftable_stack_uptodate);\n+\tRUN_TEST(test_reftable_stack_transaction_api);\n+\tRUN_TEST(test_reftable_stack_hash_id);\n+\tRUN_TEST(test_sizes_to_segments_all_equal);\n+\tRUN_TEST(test_reftable_stack_auto_compaction);\n+\tRUN_TEST(test_reftable_stack_validate_refname);\n+\tRUN_TEST(test_reftable_stack_update_index_check);\n+\tRUN_TEST(test_reftable_stack_lock_failure);\n+\tRUN_TEST(test_reftable_stack_log_normalize);\n+\tRUN_TEST(test_reftable_stack_tombstone);\n+\tRUN_TEST(test_reftable_stack_add_one);\n+\tRUN_TEST(test_empty_add);\n+\tRUN_TEST(test_reflog_expire);\n+\tRUN_TEST(test_suggest_compaction_segment);\n+\tRUN_TEST(test_suggest_compaction_segment_nothing);\n+\tRUN_TEST(test_sizes_to_segments);\n+\tRUN_TEST(test_sizes_to_segments_empty);\n+\tRUN_TEST(test_log2);\n+\tRUN_TEST(test_parse_names);\n+\tRUN_TEST(test_read_file);\n+\tRUN_TEST(test_names_equal);\n+\tRUN_TEST(test_reftable_stack_add);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 23e050eb5d23..2efe48b23bbb 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -10,6 +10,7 @@ int cmd__reftable(int argc, const char **argv)\n \trecord_test_main(argc, argv);\n \trefname_test_main(argc, argv);\n \treftable_test_main(argc, argv);\n+\tstack_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"421772","messageId":"2dc73bf2ec96acae12a17c719083d11401775bc3.1618255553.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 18/20] Reftable support for git-core","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:50Z","receivedAt":"2021-04-12T19:26:38Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nFor background, see the previous commit introducing the library.\n\nThis introduces the file refs/reftable-backend.c containing a reftable-powered\nref storage backend.\n\nIt can be activated by passing --ref-storage=reftable to \"git init\", or setting\nGIT_TEST_REFTABLE in the environment.\n\nExample use: see t/t0031-reftable.sh\n\nSigned-off-by: Han-Wen Nienhuys \nSigned-off-by: Johannes Schindelin \nHelped-by: Johannes Schindelin \nHelped-by: Junio Hamano \nCo-authored-by: Jeff King \n---\n Documentation/config/extensions.txt | 9 +\n .../technical/repository-version.txt | 7 +\n Makefile | 1 +\n builtin/clone.c | 5 +-\n builtin/init-db.c | 55 +-\n builtin/worktree.c | 27 +-\n cache.h | 8 +-\n config.mak.uname | 2 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n refs.c | 27 +-\n refs.h | 3 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1451 +++++++++++++++++\n repository.c | 2 +\n repository.h | 3 +\n setup.c | 9 +-\n t/t0031-reftable.sh | 203 +++\n t/t1409-avoid-packing-refs.sh | 6 +\n t/t1450-fsck.sh | 6 +\n t/t3210-pack-refs.sh | 6 +\n t/test-lib.sh | 5 +\n 21 files changed, 1814 insertions(+), 33 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100755 t/t0031-reftable.sh\n\ndiff --git a/Documentation/config/extensions.txt b/Documentation/config/extensions.txt\nindex 4e23d73cdcad..82c5940f1434 100644\n--- a/Documentation/config/extensions.txt\n+++ b/Documentation/config/extensions.txt\n@@ -6,3 +6,12 @@ extensions.objectFormat::\n Note that this setting should only be set by linkgit:git-init[1] or\n linkgit:git-clone[1]. Trying to change it after initialization will not\n work and will produce hard-to-diagnose issues.\n++\n+extensions.refStorage::\n+\tSpecify the ref storage mechanism to use. The acceptable values are `files` and\n+\t`reftable`. If not specified, `files` is assumed. It is an error to specify\n+\tthis key unless `core.repositoryFormatVersion` is 1.\n++\n+Note that this setting should only be set by linkgit:git-init[1] or\n+linkgit:git-clone[1]. Trying to change it after initialization will not\n+work and will produce hard-to-diagnose issues.\ndiff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\nindex 7844ef30ffde..725762358333 100644\n--- a/Documentation/technical/repository-version.txt\n+++ b/Documentation/technical/repository-version.txt\n@@ -100,3 +100,10 @@ If set, by default \"git config\" reads from both \"config\" and\n multiple working directory mode, \"config\" file is shared while\n \"config.worktree\" is per-working directory (i.e., it's in\n GIT_COMMON_DIR/worktrees//config.worktree)\n+\n+==== `refStorage`\n+\n+Specifies the file format for the ref database. Values are `files`\n+(for the traditional packed + loose ref format) and `reftable` for the\n+binary reftable format. See https://github.com/google/reftable for\n+more information.\ndiff --git a/Makefile b/Makefile\nindex e5ea8c6627ae..5890e7d1cbb2 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -975,6 +975,7 @@ LIB_OBJS += reflog-walk.o\n LIB_OBJS += refs.o\n LIB_OBJS += refs/debug.o\n LIB_OBJS += refs/files-backend.o\n+LIB_OBJS += refs/reftable-backend.o\n LIB_OBJS += refs/iterator.o\n LIB_OBJS += refs/packed-backend.o\n LIB_OBJS += refs/ref-cache.o\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex f6b0c48beda5..bb2509fc1268 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1148,7 +1148,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t}\n \n \tinit_db(git_dir, real_git_dir, option_template, GIT_HASH_UNKNOWN, NULL,\n-\t\tINIT_DB_QUIET);\n+\t\tdefault_ref_storage(), INIT_DB_QUIET);\n \n \tif (real_git_dir)\n \t\tgit_dir = real_git_dir;\n@@ -1299,7 +1299,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t * Now that we know what algorithm the remote side is using,\n \t\t * let's set ours to the same thing.\n \t\t */\n-\t\tinitialize_repository_version(hash_algo, 1);\n+\t\tinitialize_repository_version(hash_algo, 1,\n+\t\t\t\t\t default_ref_storage());\n \t\trepo_set_hash_algo(the_repository, hash_algo);\n \n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex ff06fe4b9a49..31b718259a64 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -167,12 +167,14 @@ static int needs_work_tree_config(const char *git_dir, const char *work_tree)\n \treturn 1;\n }\n \n-void initialize_repository_version(int hash_algo, int reinit)\n+void initialize_repository_version(int hash_algo, int reinit,\n+\t\t\t\t const char *ref_storage_format)\n {\n \tchar repo_version_string[10];\n \tint repo_version = GIT_REPO_VERSION;\n \n-\tif (hash_algo != GIT_HASH_SHA1)\n+\tif (hash_algo != GIT_HASH_SHA1 ||\n+\t !strcmp(ref_storage_format, \"reftable\"))\n \t\trepo_version = GIT_REPO_VERSION_READ;\n \n \t/* This forces creation of new config file */\n@@ -225,6 +227,7 @@ static int create_default_files(const char *template_path,\n \tis_bare_repository_cfg = init_is_bare_repository || !work_tree;\n \tif (init_shared_repository != -1)\n \t\tset_shared_repository(init_shared_repository);\n+\tthe_repository->ref_storage_format = xstrdup(fmt->ref_storage);\n \n \t/*\n \t * We would have created the above under user's umask -- under\n@@ -234,6 +237,24 @@ static int create_default_files(const char *template_path,\n \t\tadjust_shared_perm(get_git_dir());\n \t}\n \n+\t/*\n+\t * Check to see if .git/HEAD exists; this must happen before\n+\t * initializing the ref db, because we want to see if there is an\n+\t * existing HEAD.\n+\t */\n+\tpath = git_path_buf(&buf, \"HEAD\");\n+\treinit = (!access(path, R_OK) ||\n+\t\t readlink(path, junk, sizeof(junk) - 1) != -1);\n+\n+\t/*\n+\t * refs/heads is a file when using reftable. We can't reinitialize with\n+\t * a reftable because it will overwrite HEAD\n+\t */\n+\tif (reinit && (!strcmp(fmt->ref_storage, \"reftable\")) ==\n+\t\t\t is_directory(git_path_buf(&buf, \"refs/heads\"))) {\n+\t\tdie(\"cannot switch ref storage format.\");\n+\t}\n+\n \t/*\n \t * We need to create a \"refs\" dir in any case so that older\n \t * versions of git can tell that this is a repository.\n@@ -248,9 +269,6 @@ static int create_default_files(const char *template_path,\n \t * Point the HEAD symref to the initial branch with if HEAD does\n \t * not yet exist.\n \t */\n-\tpath = git_path_buf(&buf, \"HEAD\");\n-\treinit = (!access(path, R_OK)\n-\t\t || readlink(path, junk, sizeof(junk)-1) != -1);\n \tif (!reinit) {\n \t\tchar *ref;\n \n@@ -267,7 +285,7 @@ static int create_default_files(const char *template_path,\n \t\tfree(ref);\n \t}\n \n-\tinitialize_repository_version(fmt->hash_algo, 0);\n+\tinitialize_repository_version(fmt->hash_algo, 0, fmt->ref_storage);\n \n \t/* Check filemode trustability */\n \tpath = git_path_buf(&buf, \"config\");\n@@ -382,7 +400,7 @@ static void validate_hash_algorithm(struct repository_format *repo_fmt, int hash\n \n int init_db(const char *git_dir, const char *real_git_dir,\n \t const char *template_dir, int hash, const char *initial_branch,\n-\t unsigned int flags)\n+\t const char *ref_storage_format, unsigned int flags)\n {\n \tint reinit;\n \tint exist_ok = flags & INIT_DB_EXIST_OK;\n@@ -421,6 +439,7 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \t * is an attempt to reinitialize new repository with an old tool.\n \t */\n \tcheck_repository_format(&repo_fmt);\n+\trepo_fmt.ref_storage = xstrdup(ref_storage_format);\n \n \tvalidate_hash_algorithm(&repo_fmt, hash);\n \n@@ -475,6 +494,9 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \t\tgit_config_set(\"receive.denyNonFastforwards\", \"true\");\n \t}\n \n+\tif (!strcmp(ref_storage_format, \"reftable\"))\n+\t\tgit_config_set(\"extensions.refStorage\", ref_storage_format);\n+\n \tif (!(flags & INIT_DB_QUIET)) {\n \t\tint len = strlen(git_dir);\n \n@@ -548,6 +570,7 @@ static const char *const init_db_usage[] = {\n int cmd_init_db(int argc, const char **argv, const char *prefix)\n {\n \tconst char *git_dir;\n+\tconst char *ref_storage_format = default_ref_storage();\n \tconst char *real_git_dir = NULL;\n \tconst char *work_tree;\n \tconst char *template_dir = NULL;\n@@ -556,15 +579,18 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n \tconst char *initial_branch = NULL;\n \tint hash_algo = GIT_HASH_UNKNOWN;\n \tconst struct option init_db_options[] = {\n-\t\tOPT_STRING(0, \"template\", &template_dir, N_(\"template-directory\"),\n-\t\t\t\tN_(\"directory from which templates will be used\")),\n+\t\tOPT_STRING(0, \"template\", &template_dir,\n+\t\t\t N_(\"template-directory\"),\n+\t\t\t N_(\"directory from which templates will be used\")),\n \t\tOPT_SET_INT(0, \"bare\", &is_bare_repository_cfg,\n-\t\t\t\tN_(\"create a bare repository\"), 1),\n+\t\t\t N_(\"create a bare repository\"), 1),\n \t\t{ OPTION_CALLBACK, 0, \"shared\", &init_shared_repository,\n-\t\t\tN_(\"permissions\"),\n-\t\t\tN_(\"specify that the git repository is to be shared amongst several users\"),\n-\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0},\n+\t\t N_(\"permissions\"),\n+\t\t N_(\"specify that the git repository is to be shared amongst several users\"),\n+\t\t PARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0 },\n \t\tOPT_BIT('q', \"quiet\", &flags, N_(\"be quiet\"), INIT_DB_QUIET),\n+\t\tOPT_STRING(0, \"ref-storage\", &ref_storage_format, N_(\"backend\"),\n+\t\t\t N_(\"the ref storage format to use\")),\n \t\tOPT_STRING(0, \"separate-git-dir\", &real_git_dir, N_(\"gitdir\"),\n \t\t\t N_(\"separate git dir from working tree\")),\n \t\tOPT_STRING('b', \"initial-branch\", &initial_branch, N_(\"name\"),\n@@ -707,10 +733,11 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n \t}\n \n \tUNLEAK(real_git_dir);\n+\tUNLEAK(ref_storage_format);\n \tUNLEAK(git_dir);\n \tUNLEAK(work_tree);\n \n \tflags |= INIT_DB_EXIST_OK;\n \treturn init_db(git_dir, real_git_dir, template_dir, hash_algo,\n-\t\t initial_branch, flags);\n+\t\t initial_branch, ref_storage_format, flags);\n }\ndiff --git a/builtin/worktree.c b/builtin/worktree.c\nindex 877145349381..fd389c86f6fa 100644\n--- a/builtin/worktree.c\n+++ b/builtin/worktree.c\n@@ -13,6 +13,7 @@\n #include \"utf8.h\"\n #include \"worktree.h\"\n #include \"quote.h\"\n+#include \"../refs/refs-internal.h\"\n \n static const char * const worktree_usage[] = {\n \tN_(\"git worktree add [] []\"),\n@@ -330,9 +331,29 @@ static int add_worktree(const char *path, const char *refname,\n \t * worktree.\n \t */\n \tstrbuf_reset(&sb);\n-\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n-\twrite_file(sb.buf, \"%s\", oid_to_hex(&null_oid));\n-\tstrbuf_reset(&sb);\n+\tif (get_main_ref_store(the_repository)->be == &refs_be_reftable) {\n+\t\t/* XXX this is cut & paste from reftable_init_db. */\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"%s\", \"ref: refs/heads/.invalid\\n\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs/heads\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\t} else {\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"%s\", oid_to_hex(&null_oid));\n+\t\tstrbuf_reset(&sb);\n+\t}\n+\n \tstrbuf_addf(&sb, \"%s/commondir\", sb_repo.buf);\n \twrite_file(sb.buf, \"../..\");\n \ndiff --git a/cache.h b/cache.h\nindex 148d9ab5f185..f299535b0b03 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -629,9 +629,10 @@ int path_inside_repo(const char *prefix, const char *path);\n #define INIT_DB_EXIST_OK 0x0002\n \n int init_db(const char *git_dir, const char *real_git_dir,\n-\t const char *template_dir, int hash_algo,\n-\t const char *initial_branch, unsigned int flags);\n-void initialize_repository_version(int hash_algo, int reinit);\n+\t const char *template_dir, int hash_algo, const char *initial_branch,\n+\t const char *ref_storage_format, unsigned int flags);\n+void initialize_repository_version(int hash_algo, int reinit,\n+\t\t\t\t const char *ref_storage_format);\n \n void sanitize_stdfds(void);\n int daemonize(void);\n@@ -1045,6 +1046,7 @@ struct repository_format {\n \tint is_bare;\n \tint hash_algo;\n \tchar *work_tree;\n+\tchar *ref_storage;\n \tstruct string_list unknown_extensions;\n \tstruct string_list v1_only_extensions;\n };\ndiff --git a/config.mak.uname b/config.mak.uname\nindex cb443b4e023a..96c1268693f1 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -701,7 +701,7 @@ vcxproj:\n \t# Make .vcxproj files and add them\n \tunset QUIET_GEN QUIET_BUILT_IN; \\\n \tperl contrib/buildsystems/generate -g Vcxproj\n-\tgit add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj\n+\tgit add -f git.sln {*,*/lib,*/libreftable,t/helper/*}/*.vcxproj\n \n \t# Generate the LinkOrCopyBuiltins.targets and LinkOrCopyRemoteHttp.targets file\n \t(echo '' && \\\ndiff --git a/contrib/buildsystems/Generators/Vcxproj.pm b/contrib/buildsystems/Generators/Vcxproj.pm\nindex d2584450ba17..1a25789d2851 100644\n--- a/contrib/buildsystems/Generators/Vcxproj.pm\n+++ b/contrib/buildsystems/Generators/Vcxproj.pm\n@@ -77,7 +77,7 @@ sub createProject {\n my $libs_release = \"\\n \";\n my $libs_debug = \"\\n \";\n if (!$static_library) {\n- $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n+ $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib|reftable\\/libreftable\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n $libs_debug = $libs_release;\n $libs_debug =~ s/zlib\\.lib/zlibd\\.lib/g;\n $libs_debug =~ s/libexpat\\.lib/libexpatd\\.lib/g;\n@@ -232,6 +232,7 @@ sub createProject {\n EOM\n if (!$static_library || $target =~ 'vcs-svn' || $target =~ 'xdiff') {\n my $uuid_libgit = $$build_structure{\"LIBS_libgit_GUID\"};\n+ my $uuid_libreftable = $$build_structure{\"LIBS_reftable/libreftable_GUID\"};\n my $uuid_xdiff_lib = $$build_structure{\"LIBS_xdiff/lib_GUID\"};\n \n print F << \"EOM\";\n@@ -241,6 +242,14 @@ sub createProject {\n false\n \n EOM\n+ if (!($name =~ /xdiff|libreftable/)) {\n+ print F << \"EOM\";\n+ \n+ $uuid_libreftable\n+ false\n+ \n+EOM\n+ }\n if (!($name =~ 'xdiff')) {\n print F << \"EOM\";\n \ndiff --git a/refs.c b/refs.c\nindex 261fd82beb98..b7c2decd30fc 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -19,10 +19,16 @@\n #include \"repository.h\"\n #include \"sigchain.h\"\n \n+const char *default_ref_storage(void)\n+{\n+\tconst char *test = getenv(\"GIT_TEST_REFTABLE\");\n+\treturn test ? \"reftable\" : \"files\";\n+}\n+\n /*\n * List of all available backends\n */\n-static struct ref_storage_be *refs_backends = &refs_be_files;\n+static struct ref_storage_be *refs_backends = &refs_be_reftable;\n \n static struct ref_storage_be *find_ref_storage_backend(const char *name)\n {\n@@ -1875,13 +1881,13 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,\n * Create, record, and return a ref_store instance for the specified\n * gitdir.\n */\n-static struct ref_store *ref_store_init(const char *gitdir,\n+static struct ref_store *ref_store_init(const char *gitdir, const char *be_name,\n \t\t\t\t\tunsigned int flags)\n {\n-\tconst char *be_name = \"files\";\n-\tstruct ref_storage_be *be = find_ref_storage_backend(be_name);\n+\tstruct ref_storage_be *be;\n \tstruct ref_store *refs;\n \n+\tbe = find_ref_storage_backend(be_name);\n \tif (!be)\n \t\tBUG(\"reference backend %s is unknown\", be_name);\n \n@@ -1897,7 +1903,11 @@ struct ref_store *get_main_ref_store(struct repository *r)\n \tif (!r->gitdir)\n \t\tBUG(\"attempting to get main_ref_store outside of repository\");\n \n-\tr->refs_private = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);\n+\tr->refs_private = ref_store_init(r->gitdir,\n+\t\t\t\t\t r->ref_storage_format ?\n+\t\t\t\t\t\t r->ref_storage_format :\n+\t\t\t\t\t\t default_ref_storage(),\n+\t\t\t\t\t REF_STORE_ALL_CAPS);\n \tr->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private);\n \treturn r->refs_private;\n }\n@@ -1953,7 +1963,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \t\tgoto done;\n \n \t/* assume that add_submodule_odb() has been called */\n-\trefs = ref_store_init(submodule_sb.buf,\n+\trefs = ref_store_init(submodule_sb.buf, default_ref_storage(),\n \t\t\t REF_STORE_READ | REF_STORE_ODB);\n \tregister_ref_store_map(&submodule_ref_stores, \"submodule\",\n \t\t\t refs, submodule);\n@@ -1967,6 +1977,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \n struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n {\n+\tconst char *format = default_ref_storage();\n \tstruct ref_store *refs;\n \tconst char *id;\n \n@@ -1980,9 +1991,9 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n \n \tif (wt->id)\n \t\trefs = ref_store_init(git_common_path(\"worktrees/%s\", wt->id),\n-\t\t\t\t REF_STORE_ALL_CAPS);\n+\t\t\t\t format, REF_STORE_ALL_CAPS);\n \telse\n-\t\trefs = ref_store_init(get_git_common_dir(),\n+\t\trefs = ref_store_init(get_git_common_dir(), format,\n \t\t\t\t REF_STORE_ALL_CAPS);\n \n \tif (refs)\ndiff --git a/refs.h b/refs.h\nindex 48970dfc7e0f..5a6d4ca9fa8d 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -11,6 +11,9 @@ struct string_list;\n struct string_list_item;\n struct worktree;\n \n+/* Returns the ref storage backend to use by default. */\n+const char *default_ref_storage(void);\n+\n /*\n * Resolve a reference, recursively following symbolic refererences.\n *\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 467f4b3c936d..28166bf1f82b 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -669,6 +669,7 @@ struct ref_storage_be {\n };\n \n extern struct ref_storage_be refs_be_files;\n+extern struct ref_storage_be refs_be_reftable;\n extern struct ref_storage_be refs_be_packed;\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nnew file mode 100644\nindex 000000000000..130fd90e45b8\n--- /dev/null\n+++ b/refs/reftable-backend.c\n@@ -0,0 +1,1451 @@\n+#include \"../cache.h\"\n+#include \"../chdir-notify.h\"\n+#include \"../config.h\"\n+#include \"../iterator.h\"\n+#include \"../lockfile.h\"\n+#include \"../refs.h\"\n+#include \"../reftable/reftable-stack.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-blocksource.h\"\n+#include \"../reftable/reftable-reader.h\"\n+#include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-merged.h\"\n+#include \"../reftable/reftable-generic.h\"\n+#include \"../worktree.h\"\n+#include \"refs-internal.h\"\n+\n+extern struct ref_storage_be refs_be_reftable;\n+\n+struct git_reftable_ref_store {\n+\tstruct ref_store base;\n+\tunsigned int store_flags;\n+\n+\tint err;\n+\tchar *repo_dir;\n+\n+\tchar *reftable_dir;\n+\tchar *worktree_reftable_dir;\n+\n+\tstruct reftable_stack *main_stack;\n+\tstruct reftable_stack *worktree_stack;\n+};\n+\n+static struct reftable_stack *stack_for(struct git_reftable_ref_store *store,\n+\t\t\t\t\tconst char *refname)\n+{\n+\tif (store->worktree_stack == NULL)\n+\t\treturn store->main_stack;\n+\n+\tswitch (ref_type(refname)) {\n+\tcase REF_TYPE_PER_WORKTREE:\n+\tcase REF_TYPE_PSEUDOREF:\n+\tcase REF_TYPE_OTHER_PSEUDOREF:\n+\t\treturn store->worktree_stack;\n+\tdefault:\n+\tcase REF_TYPE_MAIN_PSEUDOREF:\n+\tcase REF_TYPE_NORMAL:\n+\t\treturn store->main_stack;\n+\t}\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type);\n+\n+static void clear_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tlog->refname = NULL;\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tlog->update.old_hash = NULL;\n+\t\tlog->update.new_hash = NULL;\n+\t\tlog->update.message = NULL;\n+\t\tbreak;\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\t}\n+\treftable_log_record_release(log);\n+}\n+\n+static void fill_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tconst char *info = git_committer_info(0);\n+\tstruct ident_split split = { NULL };\n+\tint result = split_ident_line(&split, info, strlen(info));\n+\tint sign = 1;\n+\tassert(0 == result);\n+\n+\treftable_log_record_release(log);\n+\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\tlog->update.name =\n+\t\txstrndup(split.name_begin, split.name_end - split.name_begin);\n+\tlog->update.email =\n+\t\txstrndup(split.mail_begin, split.mail_end - split.mail_begin);\n+\tlog->update.time = atol(split.date_begin);\n+\tif (*split.tz_begin == '-') {\n+\t\tsign = -1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\tif (*split.tz_begin == '+') {\n+\t\tsign = 1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\n+\tlog->update.tz_offset = sign * atoi(split.tz_begin);\n+}\n+\n+static int has_suffix(struct strbuf *b, const char *suffix)\n+{\n+\tsize_t len = strlen(suffix);\n+\n+\tif (len > b->len) {\n+\t\treturn 0;\n+\t}\n+\n+\treturn 0 == strncmp(b->buf + b->len - len, suffix, len);\n+}\n+\n+/* trims the last path component of b. Returns -1 if it is not\n+ * present, or 0 on success\n+ */\n+static int trim_component(struct strbuf *b)\n+{\n+\tchar *last;\n+\tlast = strrchr(b->buf, '/');\n+\tif (!last)\n+\t\treturn -1;\n+\tstrbuf_setlen(b, last - b->buf);\n+\treturn 0;\n+}\n+\n+/* Returns whether `b` is a worktree path, trimming it to the gitdir\n+ */\n+static int is_worktree(struct strbuf *b)\n+{\n+\tif (trim_component(b) < 0) {\n+\t\treturn 0;\n+\t}\n+\tif (!has_suffix(b, \"/worktrees\")) {\n+\t\treturn 0;\n+\t}\n+\ttrim_component(b);\n+\treturn 1;\n+}\n+\n+static struct ref_store *git_reftable_ref_store_create(const char *path,\n+\t\t\t\t\t\t unsigned int store_flags)\n+{\n+\tstruct git_reftable_ref_store *refs = xcalloc(1, sizeof(*refs));\n+\tstruct ref_store *ref_store = (struct ref_store *)refs;\n+\tstruct reftable_write_options cfg = {\n+\t\t.block_size = 4096,\n+\t\t.hash_id = the_hash_algo->format_id,\n+\t};\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tconst char *gitdir = path;\n+\tstruct strbuf wt_buf = STRBUF_INIT;\n+\tint wt = 0;\n+\n+\tstrbuf_addstr(&wt_buf, path);\n+\n+\t/* this is clumsy, but the official worktree functions (eg.\n+\t * get_worktrees()) function will try to initialize a ref storage\n+\t * backend, leading to infinite recursion. */\n+\twt = is_worktree(&wt_buf);\n+\tif (wt) {\n+\t\tgitdir = wt_buf.buf;\n+\t}\n+\n+\tbase_ref_store_init(ref_store, &refs_be_reftable);\n+\tref_store->gitdir = xstrdup(gitdir);\n+\trefs->store_flags = store_flags;\n+\tstrbuf_addf(&sb, \"%s/reftable\", gitdir);\n+\trefs->reftable_dir = xstrdup(sb.buf);\n+\tstrbuf_reset(&sb);\n+\n+\trefs->err =\n+\t\treftable_new_stack(&refs->main_stack, refs->reftable_dir, cfg);\n+\tassert(refs->err != REFTABLE_API_ERROR);\n+\n+\tif (refs->err == 0 && wt) {\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", path);\n+\t\trefs->worktree_reftable_dir = xstrdup(sb.buf);\n+\n+\t\trefs->err = reftable_new_stack(&refs->worktree_stack,\n+\t\t\t\t\t refs->worktree_reftable_dir,\n+\t\t\t\t\t cfg);\n+\t\tassert(refs->err != REFTABLE_API_ERROR);\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\tstrbuf_release(&wt_buf);\n+\treturn ref_store;\n+}\n+\n+static int git_reftable_init_db(struct ref_store *ref_store, struct strbuf *err)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\tsafe_create_dir(refs->reftable_dir, 1);\n+\tassert(refs->worktree_reftable_dir == NULL);\n+\n+\tstrbuf_addf(&sb, \"%s/HEAD\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"ref: refs/heads/.invalid\");\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs\", refs->base.gitdir);\n+\tsafe_create_dir(sb.buf, 1);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs/heads\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\n+\treturn 0;\n+}\n+\n+struct git_reftable_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_ref_record ref;\n+\tstruct object_id oid;\n+\tstruct ref_store *ref_store;\n+\n+\t/* In case we must iterate over 2 stacks, this is non-null. */\n+\tstruct reftable_merged_table *merged;\n+\tunsigned int flags;\n+\tint err;\n+\tconst char *prefix;\n+};\n+\n+static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\twhile (ri->err == 0) {\n+\t\tri->err = reftable_iterator_next_ref(&ri->iter, &ri->ref);\n+\t\tif (ri->err) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (ref_type(ri->ref.refname) == REF_TYPE_PSEUDOREF) {\n+\t\t\t/*\n+\t\t\t pseudorefs, eg. HEAD, FETCH_HEAD should not be\n+\t\t\t produced, by default.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\t\tri->base.refname = ri->ref.refname;\n+\t\tif (ri->prefix != NULL &&\n+\t\t strncmp(ri->prefix, ri->ref.refname, strlen(ri->prefix))) {\n+\t\t\tri->err = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (ri->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n+\t\t ref_type(ri->base.refname) != REF_TYPE_PER_WORKTREE)\n+\t\t\tcontinue;\n+\n+\t\tri->base.flags = 0;\n+\t\tswitch (ri->ref.value_type) {\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\thashcpy(ri->oid.hash, ri->ref.value.val1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\thashcpy(ri->oid.hash, ri->ref.value.val2.value);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF: {\n+\t\t\tint out_flags = 0;\n+\t\t\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\t\t\tri->ref_store, ri->ref.refname,\n+\t\t\t\tRESOLVE_REF_READING, &ri->oid, &out_flags);\n+\t\t\tri->base.flags = out_flags;\n+\t\t\tif (resolved == NULL &&\n+\t\t\t !(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t\t (ri->base.flags & REF_ISBROKEN)) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\t}\n+\t\tdefault:\n+\t\t\tabort();\n+\t\t}\n+\n+\t\tri->base.oid = &ri->oid;\n+\t\tif (!(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t !ref_resolves_to_object(ri->base.refname, ri->base.oid,\n+\t\t\t\t\t ri->base.flags)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tbreak;\n+\t}\n+\n+\tif (ri->err > 0) {\n+\t\treturn ITER_DONE;\n+\t}\n+\tif (ri->err < 0) {\n+\t\treturn ITER_ERROR;\n+\t}\n+\n+\treturn ITER_OK;\n+}\n+\n+static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\tif (ri->ref.value_type == REFTABLE_REF_VAL2) {\n+\t\thashcpy(peeled->hash, ri->ref.value.val2.target_value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\n+\n+static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\treftable_ref_record_release(&ri->ref);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged) {\n+\t\treftable_merged_table_free(ri->merged);\n+\t}\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n+\treftable_ref_iterator_advance, reftable_ref_iterator_peel,\n+\treftable_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,\n+\t\t\t\tunsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct git_reftable_iterator *ri = xcalloc(1, sizeof(*ri));\n+\n+\tif (refs->err < 0) {\n+\t\tri->err = refs->err;\n+\t} else if (refs->worktree_stack == NULL) {\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tri->err = reftable_merged_table_seek_ref(mt, &ri->iter, prefix);\n+\t} else {\n+\t\tstruct reftable_merged_table *mt1 =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tstruct reftable_merged_table *mt2 =\n+\t\t\treftable_stack_merged_table(refs->worktree_stack);\n+\t\tstruct reftable_table *tabs =\n+\t\t\txcalloc(2, sizeof(struct reftable_table));\n+\t\treftable_table_from_merged_table(&tabs[0], mt1);\n+\t\treftable_table_from_merged_table(&tabs[1], mt2);\n+\t\tri->err = reftable_new_merged_table(&ri->merged, tabs, 2,\n+\t\t\t\t\t\t the_hash_algo->format_id);\n+\t\tif (ri->err == 0)\n+\t\t\tri->err = reftable_merged_table_seek_ref(\n+\t\t\t\tri->merged, &ri->iter, prefix);\n+\t}\n+\n+\tbase_ref_iterator_init(&ri->base, &reftable_ref_iterator_vtable, 1);\n+\tri->prefix = prefix;\n+\tri->base.oid = &ri->oid;\n+\tri->flags = flags;\n+\tri->ref_store = ref_store;\n+\treturn &ri->base;\n+}\n+\n+static int fixup_symrefs(struct ref_store *ref_store,\n+\t\t\t struct ref_transaction *transaction)\n+{\n+\tstruct strbuf referent = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\t\tstruct object_id old_oid;\n+\n+\t\terr = git_reftable_read_raw_ref(ref_store, update->refname,\n+\t\t\t\t\t\t&old_oid, &referent,\n+\t\t\t\t\t\t/* mutate input, like\n+\t\t\t\t\t\t files-backend.c */\n+\t\t\t\t\t\t&update->type);\n+\t\tif (err < 0 && errno == ENOENT &&\n+\t\t is_null_oid(&update->old_oid)) {\n+\t\t\terr = 0;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (!(update->type & REF_ISSYMREF))\n+\t\t\tcontinue;\n+\n+\t\tif (update->flags & REF_NO_DEREF) {\n+\t\t\t/* what should happen here? See files-backend.c\n+\t\t\t * lock_ref_for_update. */\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t If we are updating a symref (eg. HEAD), we should also\n+\t\t\t update the branch that the symref points to.\n+\n+\t\t\t This is generic functionality, and would be better\n+\t\t\t done in refs.c, but the current implementation is\n+\t\t\t intertwined with the locking in files-backend.c.\n+\t\t\t*/\n+\t\t\tint new_flags = update->flags;\n+\t\t\tstruct ref_update *new_update = NULL;\n+\n+\t\t\t/* if this is an update for HEAD, should also record a\n+\t\t\t log entry for HEAD? See files-backend.c,\n+\t\t\t split_head_update()\n+\t\t\t*/\n+\t\t\tnew_update = ref_transaction_add_update(\n+\t\t\t\ttransaction, referent.buf, new_flags,\n+\t\t\t\t&update->new_oid, &update->old_oid,\n+\t\t\t\tupdate->msg);\n+\t\t\tnew_update->parent_update = update;\n+\n+\t\t\t/* files-backend sets REF_LOG_ONLY here. */\n+\t\t\tupdate->flags |= REF_NO_DEREF | REF_LOG_ONLY;\n+\t\t\tupdate->flags &= ~REF_HAVE_OLD;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tstrbuf_release(&referent);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_prepare(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errbuf)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_addition *add = NULL;\n+\tstruct reftable_stack *stack =\n+\t\ttransaction->nr ?\n+\t\t\t stack_for(refs, transaction->updates[0]->refname) :\n+\t\t\t refs->main_stack;\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_new_addition(&add, stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = fixup_symrefs(ref_store, transaction);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\ttransaction->backend_data = add;\n+\ttransaction->state = REF_TRANSACTION_PREPARED;\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tif (err < 0) {\n+\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\t\tstrbuf_addf(errbuf, \"reftable: transaction prepare: %s\",\n+\t\t\t reftable_error_str(err));\n+\t}\n+\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_abort(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\treftable_addition_destroy(add);\n+\ttransaction->backend_data = NULL;\n+\treturn 0;\n+}\n+\n+static int reftable_check_old_oid(struct ref_store *refs, const char *refname,\n+\t\t\t\t struct object_id *want_oid)\n+{\n+\tstruct object_id out_oid;\n+\tint out_flags = 0;\n+\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\trefs, refname, RESOLVE_REF_READING, &out_oid, &out_flags);\n+\tif (is_null_oid(want_oid) != (resolved == NULL)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\tif (resolved != NULL && !oideq(&out_oid, want_oid)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int ref_update_cmp(const void *a, const void *b)\n+{\n+\treturn strcmp((*(struct ref_update **)a)->refname,\n+\t\t (*(struct ref_update **)b)->refname);\n+}\n+\n+static int write_transaction_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct ref_transaction *transaction = (struct ref_transaction *)arg;\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)transaction->ref_store;\n+\tstruct reftable_stack *stack =\n+\t\tstack_for(refs, transaction->updates[0]->refname);\n+\tuint64_t ts = reftable_stack_next_update_index(stack);\n+\tint err = 0;\n+\tint i = 0;\n+\tstruct reftable_log_record *logs =\n+\t\tcalloc(transaction->nr, sizeof(*logs));\n+\tstruct ref_update **sorted =\n+\t\tmalloc(transaction->nr * sizeof(struct ref_update *));\n+\tCOPY_ARRAY(sorted, transaction->updates, transaction->nr);\n+\tQSORT(sorted, transaction->nr, ref_update_cmp);\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = sorted[i];\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tfill_reftable_log_record(log);\n+\t\tlog->update_index = ts;\n+\t\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog->refname = (char *)u->refname;\n+\t\tlog->update.old_hash = u->old_oid.hash;\n+\t\tlog->update.new_hash = u->new_oid.hash;\n+\t\tlog->update.message = u->msg;\n+\n+\t\tif (u->flags & REF_LOG_ONLY) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (u->flags & REF_HAVE_NEW) {\n+\t\t\tstruct reftable_ref_record ref = { NULL };\n+\t\t\tstruct object_id peeled;\n+\n+\t\t\tint peel_error = peel_object(&u->new_oid, &peeled);\n+\t\t\tref.refname = (char *)u->refname;\n+\t\t\tref.update_index = ts;\n+\n+\t\t\tif (!peel_error) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\t\t\tref.value.val2.target_value = peeled.hash;\n+\t\t\t\tref.value.val2.value = u->new_oid.hash;\n+\t\t\t} else if (!is_null_oid(&u->new_oid)) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\t\t\tref.value.val1 = u->new_oid.hash;\n+\t\t\t}\n+\n+\t\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\terr = reftable_writer_add_log(writer, &logs[i]);\n+\t\tclear_reftable_log_record(&logs[i]);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tfree(logs);\n+\tfree(sorted);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_finish(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errmsg)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\tint err = 0;\n+\tint i;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = transaction->updates[i];\n+\t\tif (u->flags & REF_HAVE_OLD) {\n+\t\t\terr = reftable_check_old_oid(transaction->ref_store,\n+\t\t\t\t\t\t u->refname, &u->old_oid);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (transaction->nr) {\n+\t\terr = reftable_addition_add(add, &write_transaction_table,\n+\t\t\t\t\t transaction);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\terr = reftable_addition_commit(add);\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_addition_destroy(add);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\ttransaction->backend_data = NULL;\n+\tif (err) {\n+\t\tstrbuf_addf(errmsg, \"reftable: transaction failure: %s\",\n+\t\t\t reftable_error_str(err));\n+\t\treturn -1;\n+\t}\n+\treturn err;\n+}\n+\n+static int\n+git_reftable_transaction_initial_commit(struct ref_store *ref_store,\n+\t\t\t\t\tstruct ref_transaction *transaction,\n+\t\t\t\t\tstruct strbuf *errmsg)\n+{\n+\tint err = git_reftable_transaction_prepare(ref_store, transaction,\n+\t\t\t\t\t\t errmsg);\n+\tif (err)\n+\t\treturn err;\n+\n+\treturn git_reftable_transaction_finish(ref_store, transaction, errmsg);\n+}\n+\n+struct write_delete_refs_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct string_list *refnames;\n+\tconst char *logmsg;\n+\tunsigned int flags;\n+};\n+\n+static int write_delete_refs_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_delete_refs_arg *arg =\n+\t\t(struct write_delete_refs_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint err = 0;\n+\tint i = 0;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = (char *)arg->refnames->items[i].string,\n+\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_log_record log = {\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\tstruct reftable_ref_record current = { NULL };\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.update_index = ts;\n+\t\tlog.refname = (char *)arg->refnames->items[i].string;\n+\n+\t\tlog.update.message = xstrdup(arg->logmsg);\n+\t\tlog.update.new_hash = NULL;\n+\t\tlog.update.old_hash = NULL;\n+\t\tif (reftable_stack_read_ref(arg->stack, log.refname,\n+\t\t\t\t\t ¤t) == 0) {\n+\t\t\tlog.update.old_hash =\n+\t\t\t\treftable_ref_record_val1(¤t);\n+\t\t}\n+\t\terr = reftable_writer_add_log(writer, &log);\n+\t\tlog.update.old_hash = NULL;\n+\t\treftable_ref_record_release(¤t);\n+\n+\t\tclear_reftable_log_record(&log);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int git_reftable_delete_refs(struct ref_store *ref_store,\n+\t\t\t\t const char *msg,\n+\t\t\t\t struct string_list *refnames,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack =\n+\t\tstack_for(refs, refnames->items[0].string);\n+\tstruct write_delete_refs_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refnames = refnames,\n+\t\t.logmsg = msg,\n+\t\t.flags = flags,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tstring_list_sort(refnames);\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_delete_refs_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_pack_refs(struct ref_store *ref_store,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\terr = reftable_stack_compact_all(refs->main_stack, NULL);\n+\tif (err == 0 && refs->worktree_stack != NULL)\n+\t\terr = reftable_stack_compact_all(refs->worktree_stack, NULL);\n+\tif (err == 0)\n+\t\terr = reftable_stack_clean(refs->main_stack);\n+\tif (err == 0 && refs->worktree_stack != NULL)\n+\t\terr = reftable_stack_clean(refs->worktree_stack);\n+\n+\treturn err;\n+}\n+\n+struct write_create_symref_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+\tconst char *target;\n+\tconst char *logmsg;\n+};\n+\n+static int write_create_symref_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct write_create_symref_arg *create =\n+\t\t(struct write_create_symref_arg *)arg;\n+\tuint64_t ts = reftable_stack_next_update_index(create->stack);\n+\tint err = 0;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)create->refname,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = (char *)create->target,\n+\t\t.update_index = ts,\n+\t};\n+\treftable_writer_set_limits(writer, ts, ts);\n+\terr = reftable_writer_add_ref(writer, &ref);\n+\tif (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tstruct object_id new_oid;\n+\t\tstruct object_id old_oid;\n+\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.refname = (char *)create->refname;\n+\t\tlog.update_index = ts;\n+\t\tlog.update.message = (char *)create->logmsg;\n+\t\tif (refs_resolve_ref_unsafe(\n+\t\t\t (struct ref_store *)create->refs, create->refname,\n+\t\t\t RESOLVE_REF_READING, &old_oid, NULL) != NULL) {\n+\t\t\tlog.update.old_hash = old_oid.hash;\n+\t\t}\n+\n+\t\tif (refs_resolve_ref_unsafe((struct ref_store *)create->refs,\n+\t\t\t\t\t create->target, RESOLVE_REF_READING,\n+\t\t\t\t\t &new_oid, NULL) != NULL) {\n+\t\t\tlog.update.new_hash = new_oid.hash;\n+\t\t}\n+\n+\t\tif (log.update.old_hash != NULL ||\n+\t\t log.update.new_hash != NULL) {\n+\t\t\terr = reftable_writer_add_log(writer, &log);\n+\t\t}\n+\t\tlog.refname = NULL;\n+\t\tlog.update.message = NULL;\n+\t\tlog.update.old_hash = NULL;\n+\t\tlog.update.new_hash = NULL;\n+\t\tclear_reftable_log_record(&log);\n+\t}\n+\treturn err;\n+}\n+\n+static int git_reftable_create_symref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, const char *target,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_create_symref_arg arg = { .refs = refs,\n+\t\t\t\t\t .stack = stack,\n+\t\t\t\t\t .refname = refname,\n+\t\t\t\t\t .target = target,\n+\t\t\t\t\t .logmsg = logmsg };\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_create_symref_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct write_rename_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *oldname;\n+\tconst char *newname;\n+\tconst char *logmsg;\n+};\n+\n+static int write_rename_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &ref);\n+\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* XXX do ref renames overwrite the target? */\n+\tif (reftable_stack_read_ref(arg->stack, arg->newname, &ref) == 0) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\t{\n+\t\tstruct reftable_ref_record todo[2] = {\n+\t\t\t{\n+\t\t\t\t.refname = (char *)arg->oldname,\n+\t\t\t\t.update_index = ts,\n+\t\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t},\n+\t\t\tref,\n+\t\t};\n+\t\ttodo[1].update_index = ts;\n+\t\tfree(todo[1].refname);\n+\t\ttodo[1].refname = strdup(arg->newname);\n+\n+\t\terr = reftable_writer_add_refs(writer, todo, 2);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (reftable_ref_record_val1(&ref)) {\n+\t\tuint8_t *val1 = reftable_ref_record_val1(&ref);\n+\t\tstruct reftable_log_record todo[2] = { { NULL } };\n+\t\tfill_reftable_log_record(&todo[0]);\n+\t\tfill_reftable_log_record(&todo[1]);\n+\n+\t\ttodo[0].refname = (char *)arg->oldname;\n+\t\ttodo[0].update_index = ts;\n+\t\ttodo[0].update.message = (char *)arg->logmsg;\n+\t\ttodo[0].update.old_hash = val1;\n+\t\ttodo[0].update.new_hash = NULL;\n+\n+\t\ttodo[1].refname = (char *)arg->newname;\n+\t\ttodo[1].update_index = ts;\n+\t\ttodo[1].update.old_hash = NULL;\n+\t\ttodo[1].update.new_hash = val1;\n+\t\ttodo[1].update.message = (char *)arg->logmsg;\n+\n+\t\terr = reftable_writer_add_logs(writer, todo, 2);\n+\n+\t\tclear_reftable_log_record(&todo[0]);\n+\t\tclear_reftable_log_record(&todo[1]);\n+\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t} else {\n+\t\t/* XXX symrefs? */\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+static int git_reftable_rename_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname,\n+\t\t\t\t const char *newrefname, const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n+\tstruct write_rename_arg arg = {\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_rename_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_copy_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname, const char *newrefname,\n+\t\t\t\t const char *logmsg)\n+{\n+\tBUG(\"reftable reference store does not support copying references\");\n+}\n+\n+struct git_reftable_reflog_ref_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_log_record log;\n+\tstruct object_id oid;\n+\n+\t/* Used when iterating over worktree & main */\n+\tstruct reftable_merged_table *merged;\n+\tchar *last_name;\n+};\n+\n+static int\n+git_reftable_reflog_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&ri->iter, &ri->log);\n+\t\tif (err > 0) {\n+\t\t\treturn ITER_DONE;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn ITER_ERROR;\n+\t\t}\n+\n+\t\tri->base.refname = ri->log.refname;\n+\t\tif (ri->last_name != NULL &&\n+\t\t !strcmp(ri->log.refname, ri->last_name)) {\n+\t\t\t/* we want the refnames that we have reflogs for, so we\n+\t\t\t * skip if we've already produced this name. This could\n+\t\t\t * be faster by seeking directly to\n+\t\t\t * reflog@update_index==0.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfree(ri->last_name);\n+\t\tri->last_name = xstrdup(ri->log.refname);\n+\t\thashcpy(ri->oid.hash, ri->log.update.new_hash);\n+\t\treturn ITER_OK;\n+\t}\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tBUG(\"not supported.\");\n+\treturn -1;\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\treftable_log_record_release(&ri->log);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged)\n+\t\treftable_merged_table_free(ri->merged);\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable git_reftable_reflog_ref_iterator_vtable = {\n+\tgit_reftable_reflog_ref_iterator_advance,\n+\tgit_reftable_reflog_ref_iterator_peel,\n+\tgit_reftable_reflog_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_reflog_iterator_begin(struct ref_store *ref_store)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri = xcalloc(1, sizeof(*ri));\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\n+\tif (refs->worktree_stack == NULL) {\n+\t\tstruct reftable_stack *stack = refs->main_stack;\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(stack);\n+\t\tint err = reftable_merged_table_seek_log(mt, &ri->iter, \"\");\n+\t\tif (err < 0) {\n+\t\t\tfree(ri);\n+\t\t\t/* XXX is this allowed? */\n+\t\t\treturn NULL;\n+\t\t}\n+\t} else {\n+\t\tstruct reftable_merged_table *mt1 =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tstruct reftable_merged_table *mt2 =\n+\t\t\treftable_stack_merged_table(refs->worktree_stack);\n+\t\tstruct reftable_table *tabs =\n+\t\t\txcalloc(2, sizeof(struct reftable_table));\n+\t\tint err = 0;\n+\t\treftable_table_from_merged_table(&tabs[0], mt1);\n+\t\treftable_table_from_merged_table(&tabs[1], mt2);\n+\t\terr = reftable_new_merged_table(&ri->merged, tabs, 2,\n+\t\t\t\t\t\tthe_hash_algo->format_id);\n+\t\tif (err < 0) {\n+\t\t\tfree(tabs);\n+\t\t\t/* XXX see above */\n+\t\t\treturn NULL;\n+\t\t}\n+\t\terr = reftable_merged_table_seek_ref(ri->merged, &ri->iter, \"\");\n+\t\tif (err < 0) {\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\tbase_ref_iterator_init(&ri->base,\n+\t\t\t &git_reftable_reflog_ref_iterator_vtable, 1);\n+\tri->base.oid = &ri->oid;\n+\n+\treturn (struct ref_iterator *)ri;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_newest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (err == 0) {\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\thashcpy(old_oid.hash, log.update.old_hash);\n+\t\thashcpy(new_oid.hash, log.update.new_hash);\n+\n+\t\tfull_committer = fmt_ident(log.update.name, log.update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT,\n+\t\t\t\t\t /*date*/ NULL, IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer, log.update.time,\n+\t\t\t log.update.tz_offset, log.update.message, cb_data);\n+\t\tif (err)\n+\t\t\tbreak;\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_oldest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record *logs = NULL;\n+\tint cap = 0;\n+\tint len = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\n+\twhile (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tlogs = realloc(logs, cap * sizeof(*logs));\n+\t\t}\n+\n+\t\tlogs[len++] = log;\n+\t}\n+\n+\tfor (i = len; i--;) {\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\thashcpy(old_oid.hash, log->update.old_hash);\n+\t\thashcpy(new_oid.hash, log->update.new_hash);\n+\n+\t\tfull_committer = fmt_ident(log->update.name, log->update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT, NULL,\n+\t\t\t\t\t IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer, log->update.time,\n+\t\t\t log->update.tz_offset, log->update.message, cb_data);\n+\t\tif (err) {\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tfree(logs);\n+\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_reflog_exists(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record log = { NULL };\n+\tint err = refs->err;\n+\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (strcmp(log.refname, refname)) {\n+\t\terr = 1;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\treturn !err;\n+}\n+\n+static int git_reftable_create_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, int force_create,\n+\t\t\t\t struct strbuf *err)\n+{\n+\treturn 0;\n+}\n+\n+static int git_reftable_delete_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\treturn 0;\n+}\n+\n+struct reflog_expiry_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_log_record *tombstones;\n+\tint len;\n+\tint cap;\n+};\n+\n+static void clear_log_tombstones(struct reflog_expiry_arg *arg)\n+{\n+\tint i = 0;\n+\tfor (; i < arg->len; i++) {\n+\t\treftable_log_record_release(&arg->tombstones[i]);\n+\t}\n+\n+\tFREE_AND_NULL(arg->tombstones);\n+}\n+\n+static void add_log_tombstone(struct reflog_expiry_arg *arg,\n+\t\t\t const char *refname, uint64_t ts)\n+{\n+\tstruct reftable_log_record tombstone = {\n+\t\t.refname = xstrdup(refname),\n+\t\t.update_index = ts,\n+\t};\n+\tif (arg->len == arg->cap) {\n+\t\targ->cap = 2 * arg->cap + 1;\n+\t\targ->tombstones =\n+\t\t\trealloc(arg->tombstones, arg->cap * sizeof(tombstone));\n+\t}\n+\targ->tombstones[arg->len++] = tombstone;\n+}\n+\n+static int write_reflog_expiry_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct reflog_expiry_arg *arg = (struct reflog_expiry_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint i = 0;\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->len; i++) {\n+\t\tint err = reftable_writer_add_log(writer, &arg->tombstones[i]);\n+\t\tif (err) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int\n+git_reftable_reflog_expire(struct ref_store *ref_store, const char *refname,\n+\t\t\t const struct object_id *oid, unsigned int flags,\n+\t\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t void *policy_cb_data)\n+{\n+\t/*\n+\t For log expiry, we write tombstones in place of the expired entries,\n+\t This means that the entries are still retrievable by delving into the\n+\t stack, and expiring entries paradoxically takes extra memory.\n+\n+\t This memory is only reclaimed when some operation issues a\n+\t git_reftable_pack_refs(), which will compact the entire stack and get\n+\t rid of deletion entries.\n+\n+\t It would be better if the refs backend supported an API that sets a\n+\t criterion for all refs, passing the criterion to pack_refs().\n+\t*/\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reflog_expiry_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refs = refs,\n+\t};\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = 0;\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\twhile (1) {\n+\t\tstruct object_id ooid;\n+\t\tstruct object_id noid;\n+\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (err > 0 || strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\t\thashcpy(ooid.hash, log.update.old_hash);\n+\t\thashcpy(noid.hash, log.update.new_hash);\n+\n+\t\tif (should_prune_fn(&ooid, &noid, log.update.email,\n+\t\t\t\t (timestamp_t)log.update.time,\n+\t\t\t\t log.update.tz_offset, log.update.message,\n+\t\t\t\t policy_cb_data)) {\n+\t\t\tadd_log_tombstone(&arg, refname, log.update_index);\n+\t\t}\n+\t}\n+\terr = reftable_stack_add(stack, &write_reflog_expiry_table, &arg);\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\tclear_log_tombstones(&arg);\n+\treturn err;\n+}\n+\n+static int reftable_error_to_errno(int err)\n+{\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn EIO;\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn EFAULT;\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn ENOENT;\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn EBUSY;\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn EINVAL;\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn EDOM;\n+\tdefault:\n+\t\treturn ERANGE;\n+\t}\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\n+\t/* This is usually not needed, but Git doesn't signal to ref backend if\n+\t a subprocess updated the ref DB. So we always check.\n+\t*/\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (err > 0) {\n+\t\terrno = ENOENT;\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0) {\n+\t\terrno = reftable_error_to_errno(err);\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\n+\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (reftable_ref_record_val1(&ref) != NULL) {\n+\t\thashcpy(oid->hash, reftable_ref_record_val1(&ref));\n+\t} else {\n+\t\t*type |= REF_ISBROKEN;\n+\t\terrno = EINVAL;\n+\t\terr = -1;\n+\t}\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+struct ref_storage_be refs_be_reftable = {\n+\t&refs_be_files,\n+\t\"reftable\",\n+\tgit_reftable_ref_store_create,\n+\tgit_reftable_init_db,\n+\tgit_reftable_transaction_prepare,\n+\tgit_reftable_transaction_finish,\n+\tgit_reftable_transaction_abort,\n+\tgit_reftable_transaction_initial_commit,\n+\n+\tgit_reftable_pack_refs,\n+\tgit_reftable_create_symref,\n+\tgit_reftable_delete_refs,\n+\tgit_reftable_rename_ref,\n+\tgit_reftable_copy_ref,\n+\n+\tgit_reftable_ref_iterator_begin,\n+\tgit_reftable_read_raw_ref,\n+\n+\tgit_reftable_reflog_iterator_begin,\n+\tgit_reftable_for_each_reflog_ent_oldest_first,\n+\tgit_reftable_for_each_reflog_ent_newest_first,\n+\tgit_reftable_reflog_exists,\n+\tgit_reftable_create_reflog,\n+\tgit_reftable_delete_reflog,\n+\tgit_reftable_reflog_expire,\n+};\ndiff --git a/repository.c b/repository.c\nindex 87b355e7a659..e9f78496d338 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -174,6 +174,8 @@ int repo_init(struct repository *repo,\n \tif (worktree)\n \t\trepo_set_worktree(repo, worktree);\n \n+\trepo->ref_storage_format = xstrdup_or_null(format.ref_storage);\n+\n \tclear_repository_format(&format);\n \treturn 0;\n \ndiff --git a/repository.h b/repository.h\nindex b385ca3c94b6..8019a7d0a1fc 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -78,6 +78,9 @@ struct repository {\n \t */\n \tstruct ref_store *refs_private;\n \n+\t/* The format to use for the ref database. */\n+\tchar *ref_storage_format;\n+\n \t/*\n \t * Contains path to often used file names.\n \t */\ndiff --git a/setup.c b/setup.c\nindex c04cd25a30df..c6b57efd0318 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -500,6 +500,9 @@ static enum extension_result handle_extension(const char *var,\n \t\t\treturn error(\"invalid value for 'extensions.objectformat'\");\n \t\tdata->hash_algo = format;\n \t\treturn EXTENSION_OK;\n+\t} else if (!strcmp(ext, \"refstorage\")) {\n+\t\tdata->ref_storage = xstrdup(value);\n+\t\treturn EXTENSION_OK;\n \t}\n \treturn EXTENSION_UNKNOWN;\n }\n@@ -651,6 +654,7 @@ void clear_repository_format(struct repository_format *format)\n \tstring_list_clear(&format->v1_only_extensions, 0);\n \tfree(format->work_tree);\n \tfree(format->partial_clone);\n+\tfree(format->ref_storage);\n \tinit_repository_format(format);\n }\n \n@@ -1308,8 +1312,11 @@ const char *setup_git_directory_gently(int *nongit_ok)\n \t\t\t\tgitdir = DEFAULT_GIT_DIR_ENVIRONMENT;\n \t\t\tsetup_git_env(gitdir);\n \t\t}\n-\t\tif (startup_info->have_repository)\n+\t\tif (startup_info->have_repository) {\n \t\t\trepo_set_hash_algo(the_repository, repo_fmt.hash_algo);\n+\t\t\tthe_repository->ref_storage_format =\n+\t\t\t\txstrdup_or_null(repo_fmt.ref_storage);\n+\t\t}\n \t}\n \n \tstrbuf_release(&dir);\ndiff --git a/t/t0031-reftable.sh b/t/t0031-reftable.sh\nnew file mode 100755\nindex 000000000000..ba0738e51561\n--- /dev/null\n+++ b/t/t0031-reftable.sh\n@@ -0,0 +1,203 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable basics'\n+\n+. ./test-lib.sh\n+\n+INVALID_SHA1=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n+\n+git_init () {\n+\tgit init -b primary \"$@\"\n+}\n+\n+initialize () {\n+\trm -rf .git &&\n+\tgit_init --ref-storage=reftable &&\n+\tmv .git/hooks .git/hooks-disabled\n+}\n+\n+test_expect_success 'SHA256 support, env' '\n+\trm -rf .git &&\n+\tGIT_DEFAULT_HASH=sha256 && export GIT_DEFAULT_HASH &&\n+\tgit_init --ref-storage=reftable &&\n+\tmv .git/hooks .git/hooks-disabled &&\n+\ttest_commit file\n+'\n+\n+test_expect_success 'SHA256 support, option' '\n+\trm -rf .git &&\n+\tgit_init --ref-storage=reftable --object-format=sha256 &&\n+\tmv .git/hooks .git/hooks-disabled &&\n+\ttest_commit file\n+'\n+\n+test_expect_success 'delete ref' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\tSHA=$(git show-ref -s --verify HEAD) &&\n+\ttest_write_lines \"$SHA refs/heads/primary\" \"$SHA refs/tags/file\" >expect &&\n+\tgit show-ref >actual &&\n+\t! git update-ref -d refs/tags/file $INVALID_SHA1 &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d refs/tags/file $SHA &&\n+\ttest_write_lines \"$SHA refs/heads/primary\" >expect &&\n+\tgit show-ref >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+\n+test_expect_success 'clone calls transaction_initial_commit' '\n+\ttest_commit message1 file1 &&\n+\tgit clone . cloned &&\n+\t(test -f cloned/file1 || echo \"Fixme.\")\n+'\n+\n+test_expect_success 'basic operation of reftable storage: commit, show-ref' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_write_lines refs/heads/primary refs/tags/file >expect &&\n+\tgit show-ref &&\n+\tgit show-ref | cut -f2 -d\" \" >actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'reflog, repack' '\n+\tinitialize &&\n+\tfor count in $(test_seq 1 10)\n+\tdo\n+\t\ttest_commit \"number $count\" file.t $count number-$count ||\n+\t\treturn 1\n+\tdone &&\n+\tgit pack-refs &&\n+\tls -1 .git/reftable >table-files &&\n+\ttest_line_count = 2 table-files &&\n+\tgit reflog refs/heads/primary >output &&\n+\ttest_line_count = 10 output &&\n+\tgrep \"commit (initial): number 1\" output &&\n+\tgrep \"commit: number 10\" output &&\n+\tgit gc &&\n+\tgit reflog refs/heads/primary >output &&\n+\ttest_line_count = 0 output\n+'\n+\n+test_expect_success 'branch switch in reflog output' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tgit checkout -b branch1 &&\n+\ttest_commit file2 &&\n+\tgit checkout -b branch2 &&\n+\tgit switch - &&\n+\tgit rev-parse --symbolic-full-name HEAD >actual &&\n+\techo refs/heads/branch1 >expect &&\n+\ttest_cmp actual expect\n+'\n+\n+\n+# This matches show-ref's output\n+print_ref() {\n+\techo \"$(git rev-parse \"$1\") $1\"\n+}\n+\n+test_expect_success 'peeled tags are stored' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\tgit tag -m \"annotated tag\" test_tag HEAD &&\n+\t{\n+\t\tprint_ref \"refs/heads/primary\" &&\n+\t\tprint_ref \"refs/tags/file\" &&\n+\t\tprint_ref \"refs/tags/test_tag\" &&\n+\t\tprint_ref \"refs/tags/test_tag^{}\"\n+\t} >expect &&\n+\tgit show-ref -d >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'show-ref works on fresh repo' '\n+\tinitialize &&\n+\trm -rf .git &&\n+\tgit_init --ref-storage=reftable &&\n+\t>expect &&\n+\t! git show-ref >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'checkout unborn branch' '\n+\tinitialize &&\n+\tgit checkout -b primary\n+'\n+\n+\n+test_expect_success 'dir/file conflict' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\t! git branch primary/forbidden\n+'\n+\n+\n+test_expect_success 'do not clobber existing repo' '\n+\trm -rf .git &&\n+\tgit_init --ref-storage=files &&\n+\tcat .git/HEAD >expect &&\n+\ttest_commit file &&\n+\t(git_init --ref-storage=reftable || true) &&\n+\tcat .git/HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# cherry-pick uses a pseudo ref.\n+test_expect_success 'pseudo refs' '\n+\tinitialize &&\n+\ttest_commit message1 file1 &&\n+\ttest_commit message2 file2 &&\n+\tgit branch source &&\n+\tgit checkout HEAD^ &&\n+\ttest_commit message3 file3 &&\n+\tgit cherry-pick source &&\n+\ttest -f file2\n+'\n+\n+# cherry-pick uses a pseudo ref.\n+test_expect_success 'rebase' '\n+\tinitialize &&\n+\ttest_commit message1 file1 &&\n+\ttest_commit message2 file2 &&\n+\tgit branch source &&\n+\tgit checkout HEAD^ &&\n+\ttest_commit message3 file3 &&\n+\tgit rebase source &&\n+\ttest -f file2\n+'\n+\n+test_expect_success 'worktrees' '\n+\tgit_init --ref-storage=reftable start &&\n+\t(cd start && test_commit file1 && git checkout -b branch1 &&\n+\tgit checkout -b branch2 &&\n+\tgit worktree add ../wt\n+\t) &&\n+\tcd wt &&\n+\tgit checkout branch1 &&\n+\tgit branch\n+'\n+\n+test_expect_success 'worktrees 2' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tmkdir existing_empty &&\n+\tgit worktree add --detach existing_empty primary\n+'\n+\n+test_expect_success 'FETCH_HEAD' '\n+\tinitialize &&\n+\ttest_commit one &&\n+\t(git_init sub && cd sub && test_commit two) &&\n+\tgit --git-dir sub/.git rev-parse HEAD >expect &&\n+\tgit fetch sub &&\n+\tgit checkout FETCH_HEAD &&\n+\tgit rev-parse HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/t1409-avoid-packing-refs.sh b/t/t1409-avoid-packing-refs.sh\nindex be12fb63506e..c6f783255633 100755\n--- a/t/t1409-avoid-packing-refs.sh\n+++ b/t/t1409-avoid-packing-refs.sh\n@@ -4,6 +4,12 @@ test_description='avoid rewriting packed-refs unnecessarily'\n \n . ./test-lib.sh\n \n+if test_have_prereq REFTABLE\n+then\n+ skip_all='skipping pack-refs tests; incompatible with reftable'\n+ test_done\n+fi\n+\n # Add an identifying mark to the packed-refs file header line. This\n # shouldn't upset readers, and it should be omitted if the file is\n # ever rewritten.\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 5071ac63a5b5..16440d58d6bb 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -8,6 +8,12 @@ test_description='git fsck random collection of tests\n \n . ./test-lib.sh\n \n+if test_have_prereq REFTABLE\n+then\n+ skip_all='skipping tests; incompatible with reftable'\n+ test_done\n+fi\n+\n test_expect_success setup '\n \tgit config gc.auto 0 &&\n \tgit config i18n.commitencoding ISO-8859-1 &&\ndiff --git a/t/t3210-pack-refs.sh b/t/t3210-pack-refs.sh\nindex 3b7cdc56ec84..667a23affe36 100755\n--- a/t/t3210-pack-refs.sh\n+++ b/t/t3210-pack-refs.sh\n@@ -14,6 +14,12 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n . ./test-lib.sh\n \n+if test_have_prereq REFTABLE\n+then\n+ skip_all='skipping pack-refs tests; incompatible with reftable'\n+ test_done\n+fi\n+\n test_expect_success 'enable reflogs' '\n \tgit config core.logallrefupdates true\n '\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex d3f6af6a6545..943aa9713961 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1481,6 +1481,11 @@ parisc* | hppa*)\n \t;;\n esac\n \n+if test -n \"$GIT_TEST_REFTABLE\"\n+then\n+ test_set_prereq REFTABLE\n+fi\n+\n ( COLUMNS=1 && test $COLUMNS = 1 ) && test_set_prereq COLUMNS_CAN_BE_1\n test -z \"$NO_PERL\" && test_set_prereq PERL\n test -z \"$NO_PTHREADS\" && test_set_prereq PTHREADS\n-- \ngitgitgadget\n\n"},{"id":"421771","messageId":"b2fa6ea62c16932ec2528f2698a43a9fb3cbf004.1618255553.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 19/20] git-prompt: prepare for reftable refs backend","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:51Z","receivedAt":"2021-04-12T19:26:39Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nIn our git-prompt script we strive to use Bash builtins wherever\npossible, because fork()-ing subshells for command substitutions and\nfork()+exec()-ing Git commands are expensive on some platforms. We\neven read and parse '.git/HEAD' using Bash builtins to get the name of\nthe current branch [1]. However, the upcoming reftable refs backend\nwon't use '.git/HEAD' at all, but will write an invalid refname as\nplaceholder for backwards compatibility instead, which will break our\ngit-prompt script.\n\nUpdate the git-prompt script to recognize the placeholder '.git/HEAD'\nwritten by the reftable backend (its content is specified in the\nreftable specs), and then fall back to use 'git symbolic-ref' to get\nthe name of the current branch.\n\n[1] 3a43c4b5bd (bash prompt: use bash builtins to find out current\n branch, 2011-03-31)\n\nSigned-off-by: SZEDER Gábor \n---\n contrib/completion/git-prompt.sh | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/completion/git-prompt.sh b/contrib/completion/git-prompt.sh\nindex 4640a1535d15..2e5a5d80271d 100644\n--- a/contrib/completion/git-prompt.sh\n+++ b/contrib/completion/git-prompt.sh\n@@ -478,10 +478,15 @@ __git_ps1 ()\n \t\t\tif ! __git_eread \"$g/HEAD\" head; then\n \t\t\t\treturn $exit\n \t\t\tfi\n-\t\t\t# is it a symbolic ref?\n \t\t\tb=\"${head#ref: }\"\n \t\t\tif [ \"$head\" = \"$b\" ]; then\n \t\t\t\tdetached=yes\n+\t\t\telif [ \"$b\" = \"refs/heads/.invalid\" ]; then\n+\t\t\t\t# Reftable\n+\t\t\t\tb=\"$(git symbolic-ref HEAD 2>/dev/null)\" ||\n+\t\t\t\tdetached=yes\n+\t\t\tfi\n+\t\t\tif [ \"$detached\" = yes ]; then\n \t\t\t\tb=\"$(\n \t\t\t\tcase \"${GIT_PS1_DESCRIBE_STYLE-}\" in\n \t\t\t\t(contains)\n-- \ngitgitgadget\n\n"},{"id":"421773","messageId":"39e644b9436b42fd42da463d6ddec2bf64572ef5.1618255553.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v6 20/20] Add \"test-tool dump-reftable\" command.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T19:25:52Z","receivedAt":"2021-04-12T19:26:41Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis command dumps individual tables or a stack of of tables.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/dump.c | 67 ++++++++++++++++++++--------------------\n t/helper/test-reftable.c | 5 +++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 5 files changed, 42 insertions(+), 33 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 5890e7d1cbb2..870155d48b58 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2420,6 +2420,7 @@ REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/dump.o\n REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\ndiff --git a/reftable/dump.c b/reftable/dump.c\nindex 00b444e8c9b8..7d620a3cf0fe 100644\n--- a/reftable/dump.c\n+++ b/reftable/dump.c\n@@ -12,18 +12,25 @@ license that can be found in the LICENSE file or at\n #include \n #include \n \n-#include \"reftable.h\"\n+#include \"reftable-blocksource.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-record.h\"\n #include \"reftable-tests.h\"\n+#include \"reftable-writer.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+#include \"reftable-stack.h\"\n \n static uint32_t hash_id;\n \n static int dump_table(const char *tablename)\n {\n-\tstruct reftable_block_source src = { 0 };\n+\tstruct reftable_block_source src = { NULL };\n \tint err = reftable_block_source_from_file(&src, tablename);\n-\tstruct reftable_iterator it = { 0 };\n-\tstruct reftable_ref_record ref = { 0 };\n-\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n \tstruct reftable_reader *r = NULL;\n \n \tif (err < 0)\n@@ -49,7 +56,7 @@ static int dump_table(const char *tablename)\n \t\treftable_ref_record_print(&ref, hash_id);\n \t}\n \treftable_iterator_destroy(&it);\n-\treftable_ref_record_clear(&ref);\n+\treftable_ref_record_release(&ref);\n \n \terr = reftable_reader_seek_log(r, &it, \"\");\n \tif (err < 0) {\n@@ -66,7 +73,7 @@ static int dump_table(const char *tablename)\n \t\treftable_log_record_print(&log, hash_id);\n \t}\n \treftable_iterator_destroy(&it);\n-\treftable_log_record_clear(&log);\n+\treftable_log_record_release(&log);\n \n \treftable_reader_free(r);\n \treturn 0;\n@@ -75,7 +82,7 @@ static int dump_table(const char *tablename)\n static int compact_stack(const char *stackdir)\n {\n \tstruct reftable_stack *stack = NULL;\n-\tstruct reftable_write_options cfg = {};\n+\tstruct reftable_write_options cfg = { 0 };\n \n \tint err = reftable_new_stack(&stack, stackdir, cfg);\n \tif (err < 0)\n@@ -94,10 +101,10 @@ static int compact_stack(const char *stackdir)\n static int dump_stack(const char *stackdir)\n {\n \tstruct reftable_stack *stack = NULL;\n-\tstruct reftable_write_options cfg = {};\n-\tstruct reftable_iterator it = { 0 };\n-\tstruct reftable_ref_record ref = { 0 };\n-\tstruct reftable_log_record log = { 0 };\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n \tstruct reftable_merged_table *merged = NULL;\n \n \tint err = reftable_new_stack(&stack, stackdir, cfg);\n@@ -122,7 +129,7 @@ static int dump_stack(const char *stackdir)\n \t\treftable_ref_record_print(&ref, hash_id);\n \t}\n \treftable_iterator_destroy(&it);\n-\treftable_ref_record_clear(&ref);\n+\treftable_ref_record_release(&ref);\n \n \terr = reftable_merged_table_seek_log(merged, &it, \"\");\n \tif (err < 0) {\n@@ -139,7 +146,7 @@ static int dump_stack(const char *stackdir)\n \t\treftable_log_record_print(&log, hash_id);\n \t}\n \treftable_iterator_destroy(&it);\n-\treftable_log_record_clear(&log);\n+\treftable_log_record_release(&log);\n \n \treftable_stack_destroy(stack);\n \treturn 0;\n@@ -160,40 +167,34 @@ static void print_help(void)\n int reftable_dump_main(int argc, char *const *argv)\n {\n \tint err = 0;\n-\tint opt;\n \tint opt_dump_table = 0;\n \tint opt_dump_stack = 0;\n \tint opt_compact = 0;\n-\tconst char *arg = NULL;\n-\twhile ((opt = getopt(argc, argv, \"2chts\")) != -1) {\n-\t\tswitch (opt) {\n-\t\tcase '2':\n-\t\t\thash_id = 0x73323536;\n+\tconst char *arg = NULL, *argv0 = argv[0];\n+\n+\tfor (; argc > 1; argv++, argc--)\n+\t\tif (*argv[1] != '-')\n \t\t\tbreak;\n-\t\tcase 't':\n+\t\telse if (!strcmp(\"-2\", argv[1]))\n+\t\t\thash_id = 0x73323536;\n+\t\telse if (!strcmp(\"-t\", argv[1]))\n \t\t\topt_dump_table = 1;\n-\t\t\tbreak;\n-\t\tcase 's':\n+\t\telse if (!strcmp(\"-s\", argv[1]))\n \t\t\topt_dump_stack = 1;\n-\t\t\tbreak;\n-\t\tcase 'c':\n+\t\telse if (!strcmp(\"-c\", argv[1]))\n \t\t\topt_compact = 1;\n-\t\t\tbreak;\n-\t\tcase '?':\n-\t\tcase 'h':\n+\t\telse if (!strcmp(\"-?\", argv[1]) || !strcmp(\"-h\", argv[1])) {\n \t\t\tprint_help();\n \t\t\treturn 2;\n-\t\t\tbreak;\n \t\t}\n-\t}\n \n-\tif (argv[optind] == NULL) {\n+\tif (argc != 2) {\n \t\tfprintf(stderr, \"need argument\\n\");\n \t\tprint_help();\n \t\treturn 2;\n \t}\n \n-\targ = argv[optind];\n+\targ = argv[1];\n \n \tif (opt_dump_table) {\n \t\terr = dump_table(arg);\n@@ -204,7 +205,7 @@ int reftable_dump_main(int argc, char *const *argv)\n \t}\n \n \tif (err < 0) {\n-\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv[0], arg,\n+\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv0, arg,\n \t\t\treftable_error_str(err));\n \t\treturn 1;\n \t}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 2efe48b23bbb..09e727721fe9 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -14,3 +14,8 @@ int cmd__reftable(int argc, const char **argv)\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n+\n+int cmd__dump_reftable(int argc, const char **argv)\n+{\n+\treturn reftable_dump_main(argc, (char *const *)argv);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 814d2606f8ca..bf4958492386 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -57,6 +57,7 @@ static struct test_cmd cmds[] = {\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"reftable\", cmd__reftable },\n+\t{ \"dump-reftable\", cmd__dump_reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 789c26f302dd..f195a7539cd2 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -18,6 +18,7 @@ int cmd__dump_cache_tree(int argc, const char **argv);\n int cmd__dump_fsmonitor(int argc, const char **argv);\n int cmd__dump_split_index(int argc, const char **argv);\n int cmd__dump_untracked_cache(int argc, const char **argv);\n+int cmd__dump_reftable(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n-- \ngitgitgadget\n"},{"id":"421816","messageId":"xmqq8s5nb2ri.fsf@gitster.g","threadId":"54245","inReplyTo":"9297b9c363f6d922443d261375759046e303351d.1618255553.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 07/20] reftable: reading/writing blocks","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-12T21:40:17Z","receivedAt":"2021-04-12T21:40:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"I am seeing this on the receiving end. Thanks.\n\n:832: indent with spaces.\n left = *destLen;\n:833: indent with spaces.\n *destLen = 0;\n:836: indent with spaces.\n left = 1;\n:837: indent with spaces.\n dest = buf;\n:853: indent with spaces.\n if (stream.avail_out == 0) {\nwarning: squelched 13 whitespace errors\nwarning: 18 lines add whitespace errors.\n"},{"id":"421838","messageId":"87im4qejpk.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"2dc73bf2ec96acae12a17c719083d11401775bc3.1618255553.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 18/20] Reftable support for git-core","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T07:18:15Z","receivedAt":"2021-04-13T07:18:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Apr 12 2021, Han-Wen Nienhuys via GitGitGadget wrote:\n\n> It can be activated by passing --ref-storage=reftable to \"git init\", or setting\n> GIT_TEST_REFTABLE in the environment.\n> [...]\n> +const char *default_ref_storage(void)\n> +{\n> +\tconst char *test = getenv(\"GIT_TEST_REFTABLE\");\n> +\treturn test ? \"reftable\" : \"files\";\n> +}\n\nNit: use git_env_bool() here. So GIT_TEST_REFTABLE=true is true, but\nGIT_TEST_REFTABLE= is not.\n\nNot a nit: are these tests supposed to work / do they work for you? For\nme there's a *lot* of failures, e.g. t6120-describe.sh fails for reasons\nyou might expect, manually tweaking things in the FS-refstore, but\ne.g. t5510-fetch.sh segfaults.\n\nt0001-init.sh fails with:\n\n git: refs/reftable-backend.c:194: git_reftable_init_db: Assertion `refs->worktree_reftable_dir == NULL' failed.\n\nThis is on an amd64 Debian w/Linux 4.19, so nothing exotic.\n\n> [...re-arranged a bit...]\n>\n> +if test -n \"$GIT_TEST_REFTABLE\"\n> +then\n> + test_set_prereq REFTABLE\n> +fi\n> +\n> ( COLUMNS=1 && test $COLUMNS = 1 ) && test_set_prereq COLUMNS_CAN_BE_1\n> test -z \"$NO_PERL\" && test_set_prereq PERL\n> test -z \"$NO_PTHREADS\" && test_set_prereq PTHREADS\n\nAh, so here's the (seemingly underused) prereq, but:\n\n> --- a/t/t1450-fsck.sh\n> +++ b/t/t1450-fsck.sh\n> @@ -8,6 +8,12 @@ test_description='git fsck random collection of tests\n> \n> . ./test-lib.sh\n> \n> +if test_have_prereq REFTABLE\n> +then\n> + skip_all='skipping tests; incompatible with reftable'\n> + test_done\n> +fi\n> +\n\nI understand that getting this series to land has been a pain, but this\nsort of thing doesn't inspire confidence.\n\nManually running it under reftable anyway I have 17 out of 81 tests\nfailing. A lot of tests won't run under reftable because they're of the\nform:\n\n\ttest_when_finished \"remove_object $tag\" &&\n\techo $tag >.git/refs/tags/wrong &&\n\ttest_when_finished \"git update-ref -d refs/tags/wrong\" &&\n\nI.e. we have some hack to get around update-ref guarding things for us.\n\nI would think that a better approach here would be to start with some\n(per-se unrelated) series to teach update-ref some mode like\nhash-object's --literally, i.e. \"YOLO this ref update\".\n\nThen adjust our various tests that move, clobber, or intentionally\ncorrupt things in the refstore to use that helper/option.\n\nAt that point we can actually run things like t1450-fsck.sh, maybe we\nsimply have to skip some of them because we think the sort of corruption\nwe're testing/worried about is impossible with reftable, but that in\nitself is *very* interesting.\n\nI.e. do we not have to handle certain edge cases at all in fsck and\nfriends because of inherent properties of the reftable backend (e.g. I'd\nassume, but don't know, that we're not likely to get one corrupt ref\nentry with it), but of course we can have a ref pointing to a loose\nobject that then gets removed, and is thus corrupt.\n\nAnyway. I'm speculating, but that's also my point, that I'm having to\nspeculate. That comes down to us having a new GIT_TEST_* mode that\ndoesn't pass the test suite._\n"},{"id":"421840","messageId":"87fszuej8y.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"02bd1f40fa4e281f878d0b9ba4a8b5f8eb21c8ff.1618255552.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 02/20] reftable: add LICENSE","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T07:28:13Z","receivedAt":"2021-04-13T07:28:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Apr 12 2021, Han-Wen Nienhuys via GitGitGadget wrote:\n\n> From: Han-Wen Nienhuys \n>\n> TODO: relicense?\n\nSo to your[1]:\n\n > However, it is fundamentally blocked on someone willing to spend\n > some time reviewing the series; this seems to be out of my\n > control.\n\nI for one find it hard to follow re-rolls of this when past discussion\nisn't clarified/updated in commit messages/CLs.\n\nSo in particular in v3[2] there was a whole discussion about what what\nthis licence & external hosting of this codebase meant in practical\nterms.\n\nI vaguely recall that that was clarified in some way by you, but didn't\nfind the relevant E-Mail. Something like Google's lawyers said something\nto the effect that this could just be added to git.git, no? Maybe that's\nincorrect, I don't remember.\n\nAnyway, we're now in v6, and we still have nothing but a \"TODO:\nrelicense?\" blurb here.\n\nAs noted in the past discussion I'm not interested in the\nacademic/theoretical side that license discussion, but the very\npractical question of how this library is expected to be\nintegrated/maintained.\n\nI.e. if it's meant to be 100% externally maintained and \"code-dumped\"\ninto git.git like we do with sha1collisiondetection/ that raises one set\nof concerns, but if it's meant to be \"eaten\" by git.git that poses\nanother set of questions for the code review here. I.e. much of what\nyou're doing in later patches in this series is introducing things that\nare redundant/odd if viewed if we're supposing that this code is meant\nto live in git.git.\n\n1. https://lore.kernel.org/git/CAFQ2z_P=MqT81gLjov6A471Z9sd69qQTep8KG8M8=LO9pJtkpQ@mail.gmail.com/\n2. https://lore.kernel.org/git/CAFQ2z_OJQf3+b0TT6BmAV9q9G9c2icbLK0EqrEjpYmpi8g9Fsg@mail.gmail.com/\n"},{"id":"421849","messageId":"87czuyehnv.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"df8003cb9a7d9e017d358251a2d22c0e72454e03.1618255552.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 04/20] reftable: utility functions","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T08:02:28Z","receivedAt":"2021-04-13T08:03:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Apr 12 2021, Han-Wen Nienhuys via GitGitGadget wrote:\n\n> +int strbuf_add_void(void *b, const void *data, size_t sz)\n> +{\n> +\tstrbuf_add((struct strbuf *)b, data, sz);\n> +\treturn sz;\n> +}\n\nIs that cast needed on your compiler? This compiles without warnings for\nme without that.\n\nAlso, maybe this is the sort of thing that makes sense to split into\ngeneral \"APIs needed for reftable\" patches. E.g. something like the\nbelow (just the strbuf.h change):\n\ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nindex 0c301cecced..e49029eed34 100644\n--- a/reftable/merged_test.c\n+++ b/reftable/merged_test.c\n@@ -43,7 +43,7 @@ static void write_test_table(struct strbuf *buf,\n \t\t}\n \t}\n \n-\tw = reftable_new_writer(&strbuf_add_void, buf, &opts);\n+\tw = reftable_new_writer(&strbuf_add_write, buf, &opts);\n \treftable_writer_set_limits(w, min, max);\n \n \tfor (i = 0; i < n; i++) {\n@@ -241,7 +241,7 @@ static void test_default_write_opts(void)\n \tstruct reftable_write_options opts = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_write, &buf, &opts);\n \n \tstruct reftable_ref_record rec = {\n \t\t.refname = \"master\",\ndiff --git a/reftable/refname_test.c b/reftable/refname_test.c\nindex 5e005d6af31..e8ecba1fad9 100644\n--- a/reftable/refname_test.c\n+++ b/reftable/refname_test.c\n@@ -31,7 +31,7 @@ static void test_conflict(void)\n \tstruct reftable_write_options opts = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_write, &buf, &opts);\n \tstruct reftable_ref_record rec = {\n \t\t.refname = \"a/b\",\n \t\t.value_type = REFTABLE_REF_SYMREF,\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 9d2f8d60555..bdef4813f4c 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -83,7 +83,7 @@ struct reftable_stats {\n \n /* reftable_new_writer creates a new writer */\n struct reftable_writer *\n-reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n+reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\t void *writer_arg, struct reftable_write_options *opts);\n \n /* Set the range of update indices for the records we will add. When writing a\ndiff --git a/reftable/reftable_test.c b/reftable/reftable_test.c\nindex 69dbfb09fff..1685b9a07bc 100644\n--- a/reftable/reftable_test.c\n+++ b/reftable/reftable_test.c\n@@ -52,7 +52,7 @@ static void write_table(char ***names, struct strbuf *buf, int N,\n \t\t.hash_id = hash_id,\n \t};\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_write, buf, &opts);\n \tstruct reftable_ref_record ref = { NULL };\n \tint i = 0, n;\n \tstruct reftable_log_record log = { NULL };\n@@ -131,7 +131,7 @@ static void test_log_buffer_size(void)\n \t\t\t\t\t\t .message = \"commit: 9\\n\",\n \t\t\t\t\t } };\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_write, &buf, &opts);\n \n \t/* This tests buffer extension for log compression. Must use a random\n \t hash, to ensure that the compressed part is larger than the original.\n@@ -169,7 +169,7 @@ static void test_log_write_read(void)\n \tstruct reftable_block_source source = { NULL };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_write, &buf, &opts);\n \tconst struct reftable_stats *stats = NULL;\n \treftable_writer_set_limits(w, 0, N);\n \tfor (i = 0; i < N; i++) {\n@@ -437,7 +437,7 @@ static void test_table_refs_for(int indexed)\n \n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_write, &buf, &opts);\n \n \tstruct reftable_iterator it = { NULL };\n \tint j;\n@@ -534,7 +534,7 @@ static void test_table_empty(void)\n \tstruct reftable_write_options opts = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_write, &buf, &opts);\n \tstruct reftable_block_source source = { NULL };\n \tstruct reftable_reader *rd = NULL;\n \tstruct reftable_ref_record rec = { NULL };\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 3cdb6e8ed33..2e3fd8db1dd 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -39,7 +39,7 @@ static void stack_filename(struct strbuf *dest, struct reftable_stack *st,\n \tstrbuf_addstr(dest, name);\n }\n \n-static int reftable_fd_write(void *arg, const void *data, size_t sz)\n+static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz)\n {\n \tint *fdp = (int *)arg;\n \treturn write(*fdp, data, sz);\ndiff --git a/reftable/test_framework.c b/reftable/test_framework.c\nindex a5ff4e2a2d2..e5e39fe8b13 100644\n--- a/reftable/test_framework.c\n+++ b/reftable/test_framework.c\n@@ -15,9 +15,3 @@ void set_test_hash(uint8_t *p, int i)\n {\n \tmemset(p, (uint8_t)i, hash_size(SHA1_ID));\n }\n-\n-int strbuf_add_void(void *b, const void *data, size_t sz)\n-{\n-\tstrbuf_add((struct strbuf *)b, data, sz);\n-\treturn sz;\n-}\ndiff --git a/reftable/test_framework.h b/reftable/test_framework.h\nindex 5fdc9519a5a..c04925ea11d 100644\n--- a/reftable/test_framework.h\n+++ b/reftable/test_framework.h\n@@ -46,8 +46,4 @@ license that can be found in the LICENSE file or at\n \n void set_test_hash(uint8_t *p, int i);\n \n-/* Like strbuf_add, but suitable for passing to reftable_new_writer\n- */\n-int strbuf_add_void(void *b, const void *data, size_t sz);\n-\n #endif\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex d42ca8afac1..ca3b127f83d 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -118,7 +118,7 @@ static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n \n struct reftable_writer *\n-reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n+reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\t void *writer_arg, struct reftable_write_options *opts)\n {\n \tstruct reftable_writer *wp =\ndiff --git a/reftable/writer.h b/reftable/writer.h\nindex 4921c249d06..09b88673d97 100644\n--- a/reftable/writer.h\n+++ b/reftable/writer.h\n@@ -15,7 +15,7 @@ license that can be found in the LICENSE file or at\n #include \"reftable-writer.h\"\n \n struct reftable_writer {\n-\tint (*write)(void *, const void *, size_t);\n+\tssize_t (*write)(void *, const void *, size_t);\n \tvoid *write_arg;\n \tint pending_padding;\n \tstruct strbuf last_key;\ndiff --git a/strbuf.h b/strbuf.h\nindex 223ee2094af..e3ae5b8a9d3 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -290,6 +290,18 @@ void strbuf_add_commented_lines(struct strbuf *out,\n */\n void strbuf_add(struct strbuf *sb, const void *data, size_t len);\n \n+/**\n+ * Like strbuf_add() but emulates write() for APIs that need\n+ * it. Returns the passed-in `len` as-is. The `void *` is really a\n+ * `struct strbuf *`.\n+ */\n+static inline ssize_t strbuf_add_write(void *sb, const void *data,\n+\t\t\t\t size_t len)\n+{\n+\tstrbuf_add(sb, data, len);\n+\treturn len;\n+}\n+\n /**\n * Add a NUL-terminated string to the buffer.\n *\n"},{"id":"421851","messageId":"87a6q2egvy.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"9297b9c363f6d922443d261375759046e303351d.1618255553.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 07/20] reftable: reading/writing blocks","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T08:19:13Z","receivedAt":"2021-04-13T08:19:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Apr 12 2021, Han-Wen Nienhuys via GitGitGadget wrote:\n\n> Includes a code snippet copied from zlib\n> [...]\n> +int ZEXPORT uncompress_return_consumed (\n\nPending the \"how is this integrated?\" question I had in\nhttp://lore.kernel.org/git/87fszuej8y.fsf@evledraar.gmail.com it's a bit\nodd to have a \"compat\" we unconditionally compile.\n\nSince this is for post-2017 zlib doesn't putting it in top-level\ncompat/* and having a flag to enable it make more sense, and then not\nrenaming the function.\n\nThus those who need it will have a symbol conflict when they upgrade\ntheir zlib to not need the compat wrapper, but that's a feature.\n\n> + Bytef *dest,\n> + uLongf *destLen,\n> + const Bytef *source,\n\nI see this is modified from upstream's which would trip our\nDEVELOPER=*-only -Wold-style-definition under -Werror. We have other\ndisabling of flags for compat/* already, seems better to use the\nupstream source file as-is with that.\n"},{"id":"421868","messageId":"CAFQ2z_MQp99uPpdai9Hbh76h-PO3JUHmY-pasSXONjeRp8dTzA@mail.gmail.com","threadId":"54245","inReplyTo":"87fszuej8y.fsf@evledraar.gmail.com","subject":"Re: [PATCH v6 02/20] reftable: add LICENSE","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-04-13T10:50:26Z","receivedAt":"2021-04-13T10:50:41Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Tue, Apr 13, 2021 at 9:28 AM Ævar Arnfjörð Bjarmason\n wrote:\n> > However, it is fundamentally blocked on someone willing to spend\n> > some time reviewing the series; this seems to be out of my\n> > control.\n>\n> I for one find it hard to follow re-rolls of this when past discussion\n> isn't clarified/updated in commit messages/CLs.\n>\n> So in particular in v3[2] there was a whole discussion about what what\n> this licence & external hosting of this codebase meant in practical\n> terms.\n>\n> I vaguely recall that that was clarified in some way by you, but didn't\n> find the relevant E-Mail. Something like Google's lawyers said something\n> to the effect that this could just be added to git.git, no? Maybe that's\n> incorrect, I don't remember.\n\nYou're right, I didn't update the thread. The outcome is that the\nlibrary for now lives in https://github.com/hanwen/reftable while it's\nnot integrated in git.git. In this repo I can accept contributions\n(including fixup changes posted to the git list) without requiring\nCLAs.\n\n> I.e. if it's meant to be 100% externally maintained and \"code-dumped\"\n> into git.git like we do with sha1collisiondetection/ that raises one set\n> of concerns, but if it's meant to be \"eaten\" by git.git that poses\n> another set of questions for the code review here. I.e. much of what\n> you're doing in later patches in this series is introducing things that\n> are redundant/odd if viewed if we're supposing that this code is meant\n> to live in git.git.\n\nThe code is meant to be eaten by git.git, but be amenable to being\nused by libgit2 with minimal modifications. To that end, it tries to\nuse just functionality offered by git-compat-util.\n\nIf you could point out specific oddities, that would be helpful.\n\nAs said earlier, the BSD license here is the most liberal I could\nfind, but I'm open to something else, if that makes integration in\nboth git and libgit2 possible. This was not discussed further, and\nhence the \"TODO: relicense?\" comment.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\nRegistergericht und -nummer: Hamburg, HRB 86891\nSitz der Gesellschaft: Hamburg\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"421870","messageId":"CAFQ2z_P7GjwjerP7S39xkdK20wzeE+z09NRpCnJuNCXFnJ=r8A@mail.gmail.com","threadId":"54245","inReplyTo":"87czuyehnv.fsf@evledraar.gmail.com","subject":"Re: [PATCH v6 04/20] reftable: utility functions","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-04-13T10:58:29Z","receivedAt":"2021-04-13T10:58:46Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Tue, Apr 13, 2021 at 10:02 AM Ævar Arnfjörð Bjarmason\n wrote:\n>\n>\n> On Mon, Apr 12 2021, Han-Wen Nienhuys via GitGitGadget wrote:\n>\n> > +int strbuf_add_void(void *b, const void *data, size_t sz)\n> > +{\n> > + strbuf_add((struct strbuf *)b, data, sz);\n> > + return sz;\n> > +}\n>\n> Is that cast needed on your compiler? This compiles without warnings for\n> me without that.\n\nNo! thanks.\n\n> Also, maybe this is the sort of thing that makes sense to split into\n> general \"APIs needed for reftable\" patches. E.g. something like the\n> below (just the strbuf.h change):\n\nSGTM.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"421888","messageId":"87tuoa2vhz.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"CAFQ2z_P7GjwjerP7S39xkdK20wzeE+z09NRpCnJuNCXFnJ=r8A@mail.gmail.com","subject":"Re: [PATCH v6 04/20] reftable: utility functions","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T12:56:40Z","receivedAt":"2021-04-13T12:56:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Apr 13 2021, Han-Wen Nienhuys wrote:\n\n> On Tue, Apr 13, 2021 at 10:02 AM Ævar Arnfjörð Bjarmason\n> wrote:\n>>\n>>\n>> On Mon, Apr 12 2021, Han-Wen Nienhuys via GitGitGadget wrote:\n>>\n>> > +int strbuf_add_void(void *b, const void *data, size_t sz)\n>> > +{\n>> > + strbuf_add((struct strbuf *)b, data, sz);\n>> > + return sz;\n>> > +}\n>>\n>> Is that cast needed on your compiler? This compiles without warnings for\n>> me without that.\n>\n> No! thanks.\n\nSome more, a combination of redundant casting and casting early to\nanother variable being easier to read. The latter being a matter of\nstyle/opinion. Some more like that in the code:\n\n git grep -W '\\(\\(struct .*\\*'\n\ndiff:\n\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex 25d4d95b52b..91eb0b495f5 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -26,7 +26,7 @@ static void strbuf_close(void *b)\n static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n \t\t\t uint32_t size)\n {\n-\tstruct strbuf *b = (struct strbuf *)v;\n+\tstruct strbuf *b = v;\n \tassert(off + size <= b->len);\n \tdest->data = reftable_calloc(size);\n \tmemcpy(dest->data, b->buf + off, size);\n@@ -36,7 +36,8 @@ static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n \n static uint64_t strbuf_size(void *b)\n {\n-\treturn ((struct strbuf *)b)->len;\n+\tstruct strbuf *buf = b;\n+\treturn buf->len;\n }\n \n static struct reftable_block_source_vtable strbuf_vtable = {\n@@ -91,10 +92,11 @@ static void file_return_block(void *b, struct reftable_block *dest)\n \n static void file_close(void *b)\n {\n-\tint fd = ((struct file_block_source *)b)->fd;\n+\tstruct file_block_source *bs = b;\n+\tint fd = bs->fd;\n \tif (fd > 0) {\n \t\tclose(fd);\n-\t\t((struct file_block_source *)b)->fd = 0;\n+\t\tbs->fd = 0;\n \t}\n \n \treftable_free(b);\n@@ -103,7 +105,7 @@ static void file_close(void *b)\n static int file_read_block(void *v, struct reftable_block *dest, uint64_t off,\n \t\t\t uint32_t size)\n {\n-\tstruct file_block_source *b = (struct file_block_source *)v;\n+\tstruct file_block_source *b = v;\n \tassert(off + size <= b->size);\n \tdest->data = reftable_malloc(size);\n \tif (pread(b->fd, dest->data, size, off) != size)\n"},{"id":"421891","messageId":"87r1je2up2.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"df8003cb9a7d9e017d358251a2d22c0e72454e03.1618255552.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 04/20] reftable: utility functions","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T13:14:01Z","receivedAt":"2021-04-13T13:14:07Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Apr 12 2021, Han-Wen Nienhuys via GitGitGadget wrote:\n\n> diff --git a/reftable/system.h b/reftable/system.h\n> new file mode 100644\n> index 000000000000..07277ca06273\n> --- /dev/null\n> +++ b/reftable/system.h\n> @@ -0,0 +1,32 @@\n> +/*\n> +Copyright 2020 Google LLC\n> +\n> +Use of this source code is governed by a BSD-style\n> +license that can be found in the LICENSE file or at\n> +https://developers.google.com/open-source/licenses/bsd\n> +*/\n> +\n> +#ifndef SYSTEM_H\n> +#define SYSTEM_H\n> +\n> +#include \"git-compat-util.h\"\n> +#include \"strbuf.h\"\n> +\n> +#include \n> +\n> +struct strbuf;\n> +/* In git, this is declared in dir.h */\n> +int remove_dir_recursively(struct strbuf *path, int flags);\n> +\n> +#define SHA1_ID 0x73686131\n> +#define SHA256_ID 0x73323536\n> +#define SHA1_SIZE 20\n> +#define SHA256_SIZE 32\n> +\n> +/* This is uncompress2, which is only available in zlib as of 2017.\n> + */\n> +int uncompress_return_consumed(Bytef *dest, uLongf *destLen,\n> +\t\t\t const Bytef *source, uLong *sourceLen);\n> +int hash_size(uint32_t id);\n\nRelated to the comment in\nhttp://lore.kernel.org/git/87a6q2egvy.fsf@evledraar.gmail.com\n\nI'd think that rather than duplicating magic constants & one thing in\ndir.h we'd be better off having some leading patches splitting off the\nrelevant parts of object-file.c & dir.h, maybe THAT_NAME_minimal.h?\n\nOr: why not simply include dir.h and object.h etc? The compiler/linker\nwill discard the unused functions, and if the worry is overuse of\ngit.git functions creeping in I'd think that would be better done via\nsome test/CI job that checks what objects were used.\n"},{"id":"421892","messageId":"87o8ei2tfo.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"CAFQ2z_MQp99uPpdai9Hbh76h-PO3JUHmY-pasSXONjeRp8dTzA@mail.gmail.com","subject":"Re: [PATCH v6 02/20] reftable: add LICENSE","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T13:41:15Z","receivedAt":"2021-04-13T13:41:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Apr 13 2021, Han-Wen Nienhuys wrote:\n\n> On Tue, Apr 13, 2021 at 9:28 AM Ævar Arnfjörð Bjarmason\n> wrote:\n>> > However, it is fundamentally blocked on someone willing to spend\n>> > some time reviewing the series; this seems to be out of my\n>> > control.\n>>\n>> I for one find it hard to follow re-rolls of this when past discussion\n>> isn't clarified/updated in commit messages/CLs.\n>>\n>> So in particular in v3[2] there was a whole discussion about what what\n>> this licence & external hosting of this codebase meant in practical\n>> terms.\n>>\n>> I vaguely recall that that was clarified in some way by you, but didn't\n>> find the relevant E-Mail. Something like Google's lawyers said something\n>> to the effect that this could just be added to git.git, no? Maybe that's\n>> incorrect, I don't remember.\n>\n> You're right, I didn't update the thread. The outcome is that the\n> library for now lives in https://github.com/hanwen/reftable while it's\n> not integrated in git.git. In this repo I can accept contributions\n> (including fixup changes posted to the git list) without requiring\n> CLAs.\n\nAnd once this lands if there's patches to reftable/* on this ML?\n\nKnowing would be good, a change to Documentation/SubmittingPatches et al\nin the next iteration even better :)\n\n>> I.e. if it's meant to be 100% externally maintained and \"code-dumped\"\n>> into git.git like we do with sha1collisiondetection/ that raises one set\n>> of concerns, but if it's meant to be \"eaten\" by git.git that poses\n>> another set of questions for the code review here. I.e. much of what\n>> you're doing in later patches in this series is introducing things that\n>> are redundant/odd if viewed if we're supposing that this code is meant\n>> to live in git.git.\n>\n> The code is meant to be eaten by git.git, but be amenable to being\n> used by libgit2 with minimal modifications. To that end, it tries to\n> use just functionality offered by git-compat-util.\n>\n> If you could point out specific oddities, that would be helpful.\n\nThings like different coding style, e.g. $(git grep '[!=]= NULL' --\nreftable/) & comment style.\n\nHaving Go-ish (or Google-ish?) *_test.c files instead of that living in\nt/helper/*.\n\nreftable/dump.c not using parse_options().\n\nThis having reftable_free() etc., which at least for in-tree use is\nredundant, see 8d128513429 (grep/pcre2: actually make pcre2 use custom\nallocator, 2021-02-18).\n\nWondering if internal libraries (e.g. in basic.c) are there to avoid\nvarious kitchen-sink things existing in git.git itself, or because no\nexisting thing fit the purpose.\n\nI'm not saying those things aren't OK or the \"sorta-external\" trade-off\nneeds to change, just that reviewing it not knowing what the expected\nstatus is left me hanging.\n\n> As said earlier, the BSD license here is the most liberal I could\n> find, but I'm open to something else, if that makes integration in\n> both git and libgit2 possible. This was not discussed further, and\n> hence the \"TODO: relicense?\" comment.\n\nI don't per-se have issues with the license, just questions about what\nit means for the development flow.\n\nLet's say Junio merges this to master, and someone discovers a bug in\nit. Surely one of these happen:\n\n 1. We just take the patch, if it's a tree-wide patch someone at\n reftable.git would need to either look at GPLv2 patch or clean-room\n duplicate it somehow.\n\n 2. Junio refuses any suggested changes to the reftable/* directory, we\n tell people to submit an upstream PR, once that lands we update. So\n like sha1collisiondetection/ (well, sans the \"Junio refusing\", I\n think I un-forked that at least a couple of times).\n\nOne of those means that e.g. if the change is to a strbuf API reftable/*\nneeds we'd need to version that API or otherwise treat it as stable.\n\nAs an aside, to your [1] again: At least to me reading a series on-list\nthat has TODO comments in commit messages very much sounds like an RFC\nbut not calling itself that, maybe I'm alone in that, but if not\nclearing this sort of thing up / documenting it might go far in getting\nmore reviews.\n\n1. https://lore.kernel.org/git/CAFQ2z_P=MqT81gLjov6A471Z9sd69qQTep8KG8M8=LO9pJtkpQ@mail.gmail.com/\n"},{"id":"421974","messageId":"CAFQ2z_OSbeciLQnoognG+Hh5S1tZTHO6WUviC9h5YMer766k6g@mail.gmail.com","threadId":"54245","inReplyTo":"87im4qejpk.fsf@evledraar.gmail.com","subject":"Re: [PATCH v6 18/20] Reftable support for git-core","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-04-14T16:44:54Z","receivedAt":"2021-04-14T16:45:11Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Tue, Apr 13, 2021 at 9:18 AM Ævar Arnfjörð Bjarmason\n wrote:\n> > It can be activated by passing --ref-storage=reftable to \"git init\", or setting\n> > GIT_TEST_REFTABLE in the environment.\n> > [...]\n> > +const char *default_ref_storage(void)\n> > +{\n> > + const char *test = getenv(\"GIT_TEST_REFTABLE\");\n> > + return test ? \"reftable\" : \"files\";\n> > +}\n>\n> Nit: use git_env_bool() here. So GIT_TEST_REFTABLE=true is true, but\n> GIT_TEST_REFTABLE= is not.\n\nthanks, adapted this.\n\n> Not a nit: are these tests supposed to work / do they work for you? For\n> me there's a *lot* of failures, e.g. t6120-describe.sh fails for reasons\n> you might expect, manually tweaking things in the FS-refstore, but\n> e.g. t5510-fetch.sh segfaults.\n\nI'll have a look; segfaults should not happen, of course.\n\n> > +if test_have_prereq REFTABLE\n> > +then\n> > + skip_all='skipping tests; incompatible with reftable'\n> > + test_done\n> > +fi\n> > +\n>\n> I understand that getting this series to land has been a pain, but this\n> sort of thing doesn't inspire confidence.\n>\n> Manually running it under reftable anyway I have 17 out of 81 tests\n> failing. A lot of tests won't run under reftable because they're of the\n> form:\n>\n> test_when_finished \"remove_object $tag\" &&\n> echo $tag >.git/refs/tags/wrong &&\n> test_when_finished \"git update-ref -d refs/tags/wrong\" &&\n>\n> I.e. we have some hack to get around update-ref guarding things for us.\n>\n> I would think that a better approach here would be to start with some\n> (per-se unrelated) series to teach update-ref some mode like\n> hash-object's --literally, i.e. \"YOLO this ref update\".\n\nI disagree. I think this would be a job better suited to a\ntest-helper. Then we don't put tools into users' hands that\npotentially corrupt the repository. I don't understand why hash-object\n--literally is not a test helper either.\n\n> Then adjust our various tests that move, clobber, or intentionally\n> corrupt things in the refstore to use that helper/option.\n>\n> At that point we can actually run things like t1450-fsck.sh, maybe we\n> simply have to skip some of them because we think the sort of corruption\n> we're testing/worried about is impossible with reftable, but that in\n> itself is *very* interesting.\n\nOne reason for skipping some files wholesale, is that quite a few\ntests are not isolated in their test functions, so skipping an\nindividual test functions affects follow-on tests. I can't remember if\nt1450 is one of them, though.\n\n> I.e. do we not have to handle certain edge cases at all in fsck and\n> friends because of inherent properties of the reftable backend (e.g. I'd\n> assume, but don't know, that we're not likely to get one corrupt ref\n> entry with it), but of course we can have a ref pointing to a loose\n> object that then gets removed, and is thus corrupt.\n>\n> Anyway. I'm speculating, but that's also my point, that I'm having to\n> speculate. That comes down to us having a new GIT_TEST_* mode that\n> doesn't pass the test suite._\n\nThe speculation is warranted, but there are currently about 80 files\nthat have some sort of test failure, and some of these are for quite\ncommon workflows. I would rather focus on those before looking at\ntests like fsck. Functional failures may have far reaching\nimplications on the code structure (example: worktrees considerably\ncomplicated both the reftable library and its git-core glue), while\ncorruption issues if they occur, will be due to localized bugs.\n\n--\nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"422010","messageId":"CAFQ2z_NusDzvVUjm7+pyuBc9wosuLD+rUdDwLO=hQbiRTs08PA@mail.gmail.com","threadId":"54245","inReplyTo":"87a6q2egvy.fsf@evledraar.gmail.com","subject":"Re: [PATCH v6 07/20] reftable: reading/writing blocks","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-04-15T08:57:53Z","receivedAt":"2021-04-15T08:58:07Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Tue, Apr 13, 2021 at 10:19 AM Ævar Arnfjörð Bjarmason\n wrote:\n> > Includes a code snippet copied from zlib\n> > [...]\n> > +int ZEXPORT uncompress_return_consumed (\n>\n> Pending the \"how is this integrated?\" question I had in\n> http://lore.kernel.org/git/87fszuej8y.fsf@evledraar.gmail.com it's a bit\n> odd to have a \"compat\" we unconditionally compile.\n>\n> Since this is for post-2017 zlib doesn't putting it in top-level\n> compat/* and having a flag to enable it make more sense, and then not\n> renaming the function.\n\nThanks. I've moved this into compat/ for the next version of the series\n\n(how should I track this info. Is the email here fine, or do I somehow\nhave to work this into the commit message?)\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"422032","messageId":"CAFQ2z_MaqYTWr2haHVF=oiSbYCct+oFpdwD+VtnpKKMFNgvkJg@mail.gmail.com","threadId":"54245","inReplyTo":"87r1je2up2.fsf@evledraar.gmail.com","subject":"Re: [PATCH v6 04/20] reftable: utility functions","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-04-15T15:00:20Z","receivedAt":"2021-04-15T15:02:40Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Tue, Apr 13, 2021 at 3:14 PM Ævar Arnfjörð Bjarmason\n wrote:\n> I'd think that rather than duplicating magic constants & one thing in\n> dir.h we'd be better off having some leading patches splitting off the\n> relevant parts of object-file.c & dir.h, maybe THAT_NAME_minimal.h?\n>\n> Or: why not simply include dir.h and object.h etc? The compiler/linker\n> will discard the unused functions, and if the worry is overuse of\n> git.git functions creeping in I'd think that would be better done via\n> some test/CI job that checks what objects were used.\n\nIt was useful to see how much (or how little) of Git I needed to make\nit work, but it has served its purpose, so I followed your suggestion\nto use dir.h and object.h.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"422099","messageId":"87wnt2th1v.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"CAFQ2z_OSbeciLQnoognG+Hh5S1tZTHO6WUviC9h5YMer766k6g@mail.gmail.com","subject":"Re: [PATCH v6 18/20] Reftable support for git-core","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-16T14:55:40Z","receivedAt":"2021-04-16T14:55:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Apr 14 2021, Han-Wen Nienhuys wrote:\n\n> On Tue, Apr 13, 2021 at 9:18 AM Ævar Arnfjörð Bjarmason\n> wrote:\n> [...]\n>> I understand that getting this series to land has been a pain, but this\n>> sort of thing doesn't inspire confidence.\n>>\n>> Manually running it under reftable anyway I have 17 out of 81 tests\n>> failing. A lot of tests won't run under reftable because they're of the\n>> form:\n>>\n>> test_when_finished \"remove_object $tag\" &&\n>> echo $tag >.git/refs/tags/wrong &&\n>> test_when_finished \"git update-ref -d refs/tags/wrong\" &&\n>>\n>> I.e. we have some hack to get around update-ref guarding things for us.\n>>\n>> I would think that a better approach here would be to start with some\n>> (per-se unrelated) series to teach update-ref some mode like\n>> hash-object's --literally, i.e. \"YOLO this ref update\".\n>\n> I disagree. I think this would be a job better suited to a\n> test-helper. Then we don't put tools into users' hands that\n> potentially corrupt the repository. I don't understand why hash-object\n> --literally is not a test helper either.\n\nI don't feel too strongly either way, and in any case that's a\ndiscussion for any such mass test update series, and separate from the\nreftable series per-se.\n\nI do lean on the side of shipping a thing like that as part of our\nplumbing, of course with very prominent warnings, just as we do\n--literally.\n\nIt is useful for not only our test suite, but also for others to\nreproduce issues locally with the ref and object store going out of\nsync.\n\nUntil now there hasn't been much of a need in practice, the file backend\nformat is trivially understood and you can just echo a bad value to a\nfile.\n\nBut if I or anyone else encounters some Heisenbug in the wild where we\nsuspect the ref and object store went out of syncy, and we're using\nreftable, it would be very handy to test that with an \"update-ref\n--literally\", instead of having to locally recompile git, or otherwise\nmunge the opaque reftable format.\n\n>> Then adjust our various tests that move, clobber, or intentionally\n>> corrupt things in the refstore to use that helper/option.\n>>\n>> At that point we can actually run things like t1450-fsck.sh, maybe we\n>> simply have to skip some of them because we think the sort of corruption\n>> we're testing/worried about is impossible with reftable, but that in\n>> itself is *very* interesting.\n>\n> One reason for skipping some files wholesale, is that quite a few\n> tests are not isolated in their test functions, so skipping an\n> individual test functions affects follow-on tests. I can't remember if\n> t1450 is one of them, though.\n\n[...]\n\n>> I.e. do we not have to handle certain edge cases at all in fsck and\n>> friends because of inherent properties of the reftable backend (e.g. I'd\n>> assume, but don't know, that we're not likely to get one corrupt ref\n>> entry with it), but of course we can have a ref pointing to a loose\n>> object that then gets removed, and is thus corrupt.\n>>\n>> Anyway. I'm speculating, but that's also my point, that I'm having to\n>> speculate. That comes down to us having a new GIT_TEST_* mode that\n>> doesn't pass the test suite._\n>\n> The speculation is warranted, but there are currently about 80 files\n> that have some sort of test failure, and some of these are for quite\n> common workflows. I would rather focus on those before looking at\n> tests like fsck. Functional failures may have far reaching\n> implications on the code structure (example: worktrees considerably\n> complicated both the reftable library and its git-core glue), while\n> corruption issues if they occur, will be due to localized bugs.\n\nI'm not sure from reading this if/where we disagree, so let's enumerate\nand see where we land point-by-point:\n\n 1. The test suite must pass 100% for any new GIT_TEST_X mode as it\n lands.\n\n 2. It must do so in a meaningful way (i.e. not skipping entire files,\n if some/many of the tests are relevant / stress the codepath\n involved)\n\n 3. #2 should be there as the series lands.\n\nMy comments on that:\n\n 1: I feel strongly that #1 is an absolute prerequisite for integrating\n this or any other major core surgery.\n\n E.g. now we can't do any meaningful cross-platform tests for this to\n see how it breaks on our various supported platforms.\n\n We're also in V6 of this and apparently finding an easily\n discoverable segfault now, if v7 of this had a GIT_TEST_REFTABLE in\n the relevant ci/* code ...\n\n I realize that that's hard, e.g. we had to do large refactoring for\n all of the commit-graph, protocol v2, sha256 etc. modes, but it\n really pays off. E.g. for protocol v2 upload-pack not paying\n attention to hideRefs was discovered that way.\n\n 2-3: I do think we should have this too, but would be willing to be\n convinced otherwise. E.g. we have a \"gcov\" target, once we have #1\n passing do we se that under GIT_TEST_REFTABLE=1 we have the same\n (or trivially explainable) test coverage, or are we simply\n skipping a large part of the fsck etc. code (some/much of which\n will have to do with refs).\n\n Even more than test coverage I think this is interesting to see\n what we *don't* cover, as I noted upthread. I.e. can we skip large\n parts of some sanity checking \"if (reftable)\"? I suspect so, but\n without the coverage we won't be able to tell.\n\nI realize the above would be a lot of work, probably a >50 patch series\nto t/* if previous major GIT_TEST_* refactorings are any indication.\n\nBut if we agree that's a viable way forward that's not all work that\nyou'd have to do.\n\n"},{"id":"422110","messageId":"xmqqo8eem5hr.fsf@gitster.g","threadId":"54245","inReplyTo":"CAFQ2z_OSbeciLQnoognG+Hh5S1tZTHO6WUviC9h5YMer766k6g@mail.gmail.com","subject":"Re: [PATCH v6 18/20] Reftable support for git-core","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-16T18:47:12Z","receivedAt":"2021-04-16T18:47:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Han-Wen Nienhuys writes:\n\n>> I would think that a better approach here would be to start with some\n>> (per-se unrelated) series to teach update-ref some mode like\n>> hash-object's --literally, i.e. \"YOLO this ref update\".\n>\n> I disagree. I think this would be a job better suited to a\n> test-helper. Then we don't put tools into users' hands that\n> potentially corrupt the repository. I don't understand why hash-object\n> --literally is not a test helper either.\n\nAs the person who invented the \"--literally\" option, I'd have to\nagree with this assessment. It does make life a little bit easier\nfor those who hack on Git codebase and reimplementations of it, but\nlittle practical value for those who use Git every day [*].\n\nIf it were a tool to _dump_ the contents of a possibly corrupt\nobject that the \"--literally\" option would have produced to make it\neasier to see by humans, I might be pursuaded to say that such a\nfeature may be better in end-user accessible subcommands. But the\nreason why we invented \"--literally\" was specifically to create\ncorrupt objects in test repository to see end-user accessible tools\nbehave, and in hindsight, such a use case shouldn't have been a good\njustification to add the option.\n\nI wasn't following the discussion of this particular patch closely,\nso I do not know what is being discussed is a tool on the diagnosis\nside (for which I may say it is OK to give to end-usres) or on the\ncurrupting side (for which I would regret to add to end-user tools),\nbut hopefully the above would help guiding the decision between the\ntwo.\n\nThanks.\n\n\n[Footnote]\n\n* ... for any purpose other than creating a corrupt repository,\n asking somebody who is or claims to be a Git expert to figure out\n what is wrong in his or her repository, and either waste expert's\n time or have fun by watching the expert fail and gets embarrased,\n that is ;-)\n"},{"id":"422267","messageId":"8103a80394aefdd4e8b5061dfbb6a6199fe5fcae.1618832276.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 01/28] refs: ref_iterator_peel returns boolean, rather than peel_status","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:29Z","receivedAt":"2021-04-19T11:38:04Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nBefore, the cached ref_iterator would return peel_object() output directly. This\nled to spurious differences in the GIT_TRACE_REFS output, depending on the ref\nstorage backend active.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n refs.c | 2 +-\n refs/ref-cache.c | 2 +-\n refs/refs-internal.h | 3 +++\n 3 files changed, 5 insertions(+), 2 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 261fd82beb98..8873854a44fb 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2010,7 +2010,7 @@ int peel_iterated_oid(const struct object_id *base, struct object_id *peeled)\n \t oideq(current_ref_iter->oid, base)))\n \t\treturn ref_iterator_peel(current_ref_iter, peeled);\n \n-\treturn peel_object(base, peeled);\n+\treturn !!peel_object(base, peeled);\n }\n \n int refs_create_symref(struct ref_store *refs,\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 46f1e5428433..703a12959e1f 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -491,7 +491,7 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n-\treturn peel_object(ref_iterator->oid, peeled);\n+\treturn !!peel_object(ref_iterator->oid, peeled);\n }\n \n static int cache_ref_iterator_abort(struct ref_iterator *ref_iterator)\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 467f4b3c936d..546a6b965dcc 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -453,6 +453,9 @@ void base_ref_iterator_free(struct ref_iterator *iter);\n */\n typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);\n \n+/*\n+ * Peels the current ref, returning 0 for success.\n+ */\n typedef int ref_iterator_peel_fn(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled);\n \n-- \ngitgitgadget\n\n"},{"id":"422268","messageId":"9ae5ddff6aed48184d2a10c569e41441b9199f10.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 03/28] refs/debug: trace into reflog expiry too","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:31Z","receivedAt":"2021-04-19T11:38:07Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n refs/debug.c | 47 ++++++++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 44 insertions(+), 3 deletions(-)\n\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 922e64fa6ad9..3b25e3aeb1ba 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -353,6 +353,40 @@ static int debug_delete_reflog(struct ref_store *ref_store, const char *refname)\n \treturn res;\n }\n \n+struct debug_reflog_expiry_should_prune {\n+\treflog_expiry_prepare_fn *prepare;\n+\treflog_expiry_should_prune_fn *should_prune;\n+\treflog_expiry_cleanup_fn *cleanup;\n+\tvoid *cb_data;\n+};\n+\n+static void debug_reflog_expiry_prepare(const char *refname,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t void *cb_data)\n+{\n+\tstruct debug_reflog_expiry_should_prune *prune = cb_data;\n+\ttrace_printf_key(&trace_refs, \"reflog_expire_prepare: %s\\n\", refname);\n+\tprune->prepare(refname, oid, prune->cb_data);\n+}\n+\n+static int debug_reflog_expiry_should_prune_fn(struct object_id *ooid,\n+\t\t\t\t\t struct object_id *noid,\n+\t\t\t\t\t const char *email,\n+\t\t\t\t\t timestamp_t timestamp, int tz,\n+\t\t\t\t\t const char *message, void *cb_data) {\n+\tstruct debug_reflog_expiry_should_prune *prune = cb_data;\n+\n+\tint result = prune->should_prune(ooid, noid, email, timestamp, tz, message, prune->cb_data);\n+\ttrace_printf_key(&trace_refs, \"reflog_expire_should_prune: %s %ld: %d\\n\", message, (long int) timestamp, result);\n+\treturn result;\n+}\n+\n+static void debug_reflog_expiry_cleanup(void *cb_data)\n+{\n+\tstruct debug_reflog_expiry_should_prune *prune = cb_data;\n+\tprune->cleanup(prune->cb_data);\n+}\n+\n static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n \t\t\t const struct object_id *oid, unsigned int flags,\n \t\t\t reflog_expiry_prepare_fn prepare_fn,\n@@ -361,10 +395,17 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,\n \t\t\t void *policy_cb_data)\n {\n \tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n+\tstruct debug_reflog_expiry_should_prune prune = {\n+\t\t.prepare = prepare_fn,\n+\t\t.cleanup = cleanup_fn,\n+\t\t.should_prune = should_prune_fn,\n+\t\t.cb_data = policy_cb_data,\n+\t};\n \tint res = drefs->refs->be->reflog_expire(drefs->refs, refname, oid,\n-\t\t\t\t\t\t flags, prepare_fn,\n-\t\t\t\t\t\t should_prune_fn, cleanup_fn,\n-\t\t\t\t\t\t policy_cb_data);\n+\t\t\t\t\t\t flags, &debug_reflog_expiry_prepare,\n+\t\t\t\t\t\t &debug_reflog_expiry_should_prune_fn,\n+\t\t\t\t\t\t &debug_reflog_expiry_cleanup,\n+\t\t\t\t\t\t &prune);\n \ttrace_printf_key(&trace_refs, \"reflog_expire: %s: %d\\n\", refname, res);\n \treturn res;\n }\n-- \ngitgitgadget\n\n"},{"id":"422269","messageId":"554bb1ac3aed066eb578a305b7fe76f1660f6153.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 02/28] refs: document reflog_expire_fn's flag argument","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:30Z","receivedAt":"2021-04-19T11:38:09Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n refs/refs-internal.h | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 546a6b965dcc..a31c1f465beb 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -592,6 +592,10 @@ typedef int reflog_exists_fn(struct ref_store *ref_store, const char *refname);\n typedef int create_reflog_fn(struct ref_store *ref_store, const char *refname,\n \t\t\t int force_create, struct strbuf *err);\n typedef int delete_reflog_fn(struct ref_store *ref_store, const char *refname);\n+\n+/*\n+ * `flags` accepts a bitmask of `expire_reflog_flags`.\n+ */\n typedef int reflog_expire_fn(struct ref_store *ref_store,\n \t\t\t const char *refname, const struct object_id *oid,\n \t\t\t unsigned int flags,\n-- \ngitgitgadget\n\n"},{"id":"422270","messageId":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v6.git.git.1618255552.gitgitgadget@gmail.com","subject":"[PATCH v7 00/28] reftable library","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:28Z","receivedAt":"2021-04-19T11:38:14Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"This splits the giant commit from\nhttps://github.com/gitgitgadget/git/pull/539 into a series of smaller\ncommits, which build and have unittests.\n\nChanges relative to last series: version 12 Apr 2021:\n\n * AEvar's comments: zlib into compat/, void* casts, use implicit pointer\n <-> boolean conversions.\n * Plug memory leak in indexed reverse lookup.\n * Take SHA1/SHA256 constants from git headers.\n * Fix branch rename (branch -M)\n * Fix stash (support reflog deletion.)\n * Move debug printing logic out of dump.c\n * Fix reflog --dry-run expire.\n * Implement branch copy (branch -c)\n * Fix deletion of 0 branches\n * Fix transactions without REF_HAVE_OLD\n * Add requirement REFFILES for test cases that are specific to the files\n backend.\n * Fixes for various test files.\n\nThe commits up to \"hash.h: provide constants for the hash IDs\" should be\ngood to merge to 'next'.\n\nThere are several test fixups, but I've put them in another series because\nGGG enforces max 30 commits.\n\nDue to using uncompress2, this build fails on the Linux32 builder; what is\nthe magic incantation to run autoconf? There are other unexplained failures\nin t0031-reftable.sh on CI that I haven't been able to look into. The test\npasses for me, locally.\n\nHan-Wen Nienhuys (27):\n refs: ref_iterator_peel returns boolean, rather than peel_status\n refs: document reflog_expire_fn's flag argument\n refs/debug: trace into reflog expiry too\n hash.h: provide constants for the hash IDs\n init-db: set the_repository->hash_algo early on\n reftable: add LICENSE\n reftable: add error related functionality\n reftable: utility functions\n reftable: add blocksource, an abstraction for random access reads\n reftable: (de)serialization for the polymorphic record type.\n Provide zlib's uncompress2 from compat/zlib-compat.c\n reftable: reading/writing blocks\n reftable: a generic binary tree implementation\n reftable: write reftable files\n reftable: generic interface to tables\n reftable: read reftable files\n reftable: reftable file level tests\n reftable: add a heap-based priority queue for reftable records\n reftable: add merged table view\n reftable: implement refname validation\n reftable: implement stack, a mutable database of reftable files.\n reftable: add dump utility\n Reftable support for git-core\n Add \"test-tool dump-reftable\" command.\n t1301: document what needs to be done for REFTABLE\n t1401,t2011: parameterize HEAD.lock for REFTABLE\n t1404: annotate test cases with REFFILES\n\nSZEDER Gábor (1):\n git-prompt: prepare for reftable refs backend\n\n Documentation/config/extensions.txt | 9 +\n .../technical/repository-version.txt | 7 +\n Makefile | 54 +-\n builtin/clone.c | 5 +-\n builtin/init-db.c | 76 +-\n builtin/stash.c | 8 +-\n builtin/worktree.c | 27 +-\n cache.h | 8 +-\n compat/.gitattributes | 1 +\n compat/zlib-uncompress2.c | 92 +\n config.mak.uname | 2 +-\n configure.ac | 13 +\n contrib/buildsystems/CMakeLists.txt | 14 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n contrib/completion/git-prompt.sh | 7 +-\n git-compat-util.h | 4 +\n hash.h | 6 +\n object-file.c | 6 +-\n refs.c | 28 +-\n refs.h | 3 +\n refs/debug.c | 47 +-\n refs/ref-cache.c | 2 +-\n refs/refs-internal.h | 8 +\n refs/reftable-backend.c | 1616 +++++++++++++++++\n reftable/LICENSE | 31 +\n reftable/basics.c | 128 ++\n reftable/basics.h | 60 +\n reftable/basics_test.c | 98 +\n reftable/block.c | 446 +++++\n reftable/block.h | 127 ++\n reftable/block_test.c | 121 ++\n reftable/blocksource.c | 148 ++\n reftable/blocksource.h | 22 +\n reftable/constants.h | 21 +\n reftable/dump.c | 100 +\n reftable/error.c | 41 +\n reftable/generic.c | 169 ++\n reftable/generic.h | 32 +\n reftable/iter.c | 194 ++\n reftable/iter.h | 69 +\n reftable/merged.c | 362 ++++\n reftable/merged.h | 35 +\n reftable/merged_test.c | 292 +++\n reftable/pq.c | 115 ++\n reftable/pq.h | 32 +\n reftable/pq_test.c | 72 +\n reftable/publicbasics.c | 58 +\n reftable/reader.c | 794 ++++++++\n reftable/reader.h | 66 +\n reftable/readwrite_test.c | 621 +++++++\n reftable/record.c | 1202 ++++++++++++\n reftable/record.h | 139 ++\n reftable/record_test.c | 407 +++++\n reftable/refname.c | 209 +++\n reftable/refname.h | 29 +\n reftable/refname_test.c | 102 ++\n reftable/reftable-blocksource.h | 49 +\n reftable/reftable-error.h | 62 +\n reftable/reftable-generic.h | 47 +\n reftable/reftable-iterator.h | 39 +\n reftable/reftable-malloc.h | 18 +\n reftable/reftable-merged.h | 72 +\n reftable/reftable-reader.h | 101 ++\n reftable/reftable-record.h | 114 ++\n reftable/reftable-stack.h | 128 ++\n reftable/reftable-tests.h | 23 +\n reftable/reftable-writer.h | 147 ++\n reftable/reftable.c | 115 ++\n reftable/stack.c | 1395 ++++++++++++++\n reftable/stack.h | 41 +\n reftable/stack_test.c | 940 ++++++++++\n reftable/system.h | 24 +\n reftable/test_framework.c | 23 +\n reftable/test_framework.h | 53 +\n reftable/tree.c | 63 +\n reftable/tree.h | 34 +\n reftable/tree_test.c | 61 +\n reftable/writer.c | 690 +++++++\n reftable/writer.h | 50 +\n repository.c | 2 +\n repository.h | 3 +\n setup.c | 9 +-\n t/helper/test-reftable.c | 21 +\n t/helper/test-tool.c | 4 +-\n t/helper/test-tool.h | 2 +\n t/t0031-reftable.sh | 271 +++\n t/t0032-reftable-unittest.sh | 15 +\n t/t1301-shared-repo.sh | 8 +-\n t/t1401-symbolic-ref.sh | 11 +-\n t/t1404-update-ref-errors.sh | 86 +-\n t/t1409-avoid-packing-refs.sh | 6 +\n t/t1450-fsck.sh | 6 +\n t/t2011-checkout-invalid-head.sh | 11 +-\n t/t3210-pack-refs.sh | 6 +\n t/test-lib.sh | 5 +\n 95 files changed, 13065 insertions(+), 86 deletions(-)\n create mode 100644 compat/.gitattributes\n create mode 100644 compat/zlib-uncompress2.c\n create mode 100644 refs/reftable-backend.c\n create mode 100644 reftable/LICENSE\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/basics_test.c\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/dump.c\n create mode 100644 reftable/error.c\n create mode 100644 reftable/generic.c\n create mode 100644 reftable/generic.h\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/pq_test.c\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/readwrite_test.c\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n create mode 100644 reftable/reftable-blocksource.h\n create mode 100644 reftable/reftable-error.h\n create mode 100644 reftable/reftable-generic.h\n create mode 100644 reftable/reftable-iterator.h\n create mode 100644 reftable/reftable-malloc.h\n create mode 100644 reftable/reftable-merged.h\n create mode 100644 reftable/reftable-reader.h\n create mode 100644 reftable/reftable-record.h\n create mode 100644 reftable/reftable-stack.h\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/reftable-writer.h\n create mode 100644 reftable/reftable.c\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n create mode 100644 t/helper/test-reftable.c\n create mode 100755 t/t0031-reftable.sh\n create mode 100755 t/t0032-reftable-unittest.sh\n\n\nbase-commit: 54a391711554ed41b4b0792cfef004abc74893bd\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-847%2Fhanwen%2Flibreftable-v7\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-847/hanwen/libreftable-v7\nPull-Request: https://github.com/git/git/pull/847\n\nRange-diff vs v6:\n\n -: ------------ > 1: 8103a80394ae refs: ref_iterator_peel returns boolean, rather than peel_status\n -: ------------ > 2: 554bb1ac3aed refs: document reflog_expire_fn's flag argument\n -: ------------ > 3: 9ae5ddff6aed refs/debug: trace into reflog expiry too\n -: ------------ > 4: f0216ae20b69 hash.h: provide constants for the hash IDs\n 1: 15b8306fbe4b = 5: fa64aa9ce7c1 init-db: set the_repository->hash_algo early on\n 2: 02bd1f40fa4e = 6: 8a960fb77fa9 reftable: add LICENSE\n 3: bffa33c012ad = 7: 5177919a3fd2 reftable: add error related functionality\n 4: df8003cb9a7d ! 8: a24e9e661616 reftable: utility functions\n @@ reftable/basics.c (new)\n +void free_names(char **a)\n +{\n +\tchar **p;\n -+\tif (a == NULL) {\n ++\tif (!a) {\n +\t\treturn;\n +\t}\n +\tfor (p = a; *p; p++) {\n @@ reftable/basics_test.c (new)\n +\n +static int binsearch_func(size_t i, void *void_args)\n +{\n -+\tstruct binsearch_args *args = (struct binsearch_args *)void_args;\n ++\tstruct binsearch_args *args = void_args;\n +\n +\treturn args->key < args->arr[i];\n +}\n @@ reftable/basics_test.c (new)\n +\tparse_names(in, strlen(in), &out);\n +\tEXPECT(!strcmp(out[0], \"a\"));\n +\tEXPECT(!strcmp(out[1], \"b\"));\n -+\tEXPECT(out[2] == NULL);\n ++\tEXPECT(!out[2]);\n +\tfree_names(out);\n +}\n +\n @@ reftable/basics_test.c (new)\n +\tchar **out = NULL;\n +\tparse_names(in, strlen(in), &out);\n +\tEXPECT(!strcmp(out[0], \"a\"));\n -+\tEXPECT(out[1] == NULL);\n ++\tEXPECT(!out[1]);\n +\tfree_names(out);\n +}\n +\n @@ reftable/publicbasics.c (new)\n +{\n +\tswitch (id) {\n +\tcase 0:\n -+\tcase SHA1_ID:\n -+\t\treturn SHA1_SIZE;\n -+\tcase SHA256_ID:\n -+\t\treturn SHA256_SIZE;\n ++\tcase GIT_SHA1_HASH_ID:\n ++\t\treturn GIT_SHA1_RAWSZ;\n ++\tcase GIT_SHA256_HASH_ID:\n ++\t\treturn GIT_SHA256_RAWSZ;\n +\t}\n +\tabort();\n +}\n @@ reftable/system.h (new)\n +#ifndef SYSTEM_H\n +#define SYSTEM_H\n +\n ++// This header glues the reftable library to the rest of Git\n ++\n +#include \"git-compat-util.h\"\n +#include \"strbuf.h\"\n ++#include \"hash.h\" /* hash ID, sizes.*/\n ++#include \"dir.h\" /* remove_dir_recursively, for tests.*/\n +\n +#include \n +\n +struct strbuf;\n -+/* In git, this is declared in dir.h */\n -+int remove_dir_recursively(struct strbuf *path, int flags);\n -+\n -+#define SHA1_ID 0x73686131\n -+#define SHA256_ID 0x73323536\n -+#define SHA1_SIZE 20\n -+#define SHA256_SIZE 32\n -+\n -+/* This is uncompress2, which is only available in zlib as of 2017.\n -+ */\n -+int uncompress_return_consumed(Bytef *dest, uLongf *destLen,\n -+\t\t\t const Bytef *source, uLong *sourceLen);\n +int hash_size(uint32_t id);\n +\n +#endif\n @@ reftable/test_framework.c (new)\n +\n +void set_test_hash(uint8_t *p, int i)\n +{\n -+\tmemset(p, (uint8_t)i, hash_size(SHA1_ID));\n ++\tmemset(p, (uint8_t)i, hash_size(GIT_SHA1_HASH_ID));\n +}\n +\n -+int strbuf_add_void(void *b, const void *data, size_t sz)\n ++ssize_t strbuf_add_void(void *b, const void *data, size_t sz)\n +{\n -+\tstrbuf_add((struct strbuf *)b, data, sz);\n ++\tstrbuf_add(b, data, sz);\n +\treturn sz;\n +}\n \n @@ reftable/test_framework.h (new)\n +\n +/* Like strbuf_add, but suitable for passing to reftable_new_writer\n + */\n -+int strbuf_add_void(void *b, const void *data, size_t sz);\n ++ssize_t strbuf_add_void(void *b, const void *data, size_t sz);\n +\n +#endif\n \n 5: 6807af53e9a0 ! 9: 5d54e6fac99b reftable: add blocksource, an abstraction for random access reads\n @@ reftable/blocksource.c (new)\n +static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n +\t\t\t uint32_t size)\n +{\n -+\tstruct strbuf *b = (struct strbuf *)v;\n ++\tstruct strbuf *b = v;\n +\tassert(off + size <= b->len);\n +\tdest->data = reftable_calloc(size);\n +\tmemcpy(dest->data, b->buf + off, size);\n @@ reftable/blocksource.c (new)\n +void block_source_from_strbuf(struct reftable_block_source *bs,\n +\t\t\t struct strbuf *buf)\n +{\n -+\tassert(bs->ops == NULL);\n ++\tassert(!bs->ops);\n +\tbs->ops = &strbuf_vtable;\n +\tbs->arg = buf;\n +}\n @@ reftable/blocksource.c (new)\n +static int file_read_block(void *v, struct reftable_block *dest, uint64_t off,\n +\t\t\t uint32_t size)\n +{\n -+\tstruct file_block_source *b = (struct file_block_source *)v;\n ++\tstruct file_block_source *b = v;\n +\tassert(off + size <= b->size);\n +\tdest->data = reftable_malloc(size);\n +\tif (pread(b->fd, dest->data, size, off) != size)\n @@ reftable/blocksource.c (new)\n +\tp->size = st.st_size;\n +\tp->fd = fd;\n +\n -+\tassert(bs->ops == NULL);\n ++\tassert(!bs->ops);\n +\tbs->ops = &file_vtable;\n +\tbs->arg = p;\n +\treturn 0;\n 6: 78a352c96f82 ! 10: 028dd13e701b reftable: (de)serialization for the polymorphic record type.\n @@ reftable/record.c (new)\n +static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n +\t\t\t\t\t int hash_size)\n +{\n -+\tstruct reftable_ref_record *ref = (struct reftable_ref_record *)rec;\n -+\tstruct reftable_ref_record *src = (struct reftable_ref_record *)src_rec;\n ++\tstruct reftable_ref_record *ref = rec;\n ++\tconst struct reftable_ref_record *src = src_rec;\n +\tassert(hash_size > 0);\n +\n +\t/* This is simple and correct, but we could probably reuse the hash\n +\t * fields. */\n +\treftable_ref_record_release(ref);\n -+\tif (src->refname != NULL) {\n ++\tif (src->refname) {\n +\t\tref->refname = xstrdup(src->refname);\n +\t}\n +\tref->update_index = src->update_index;\n @@ reftable/record.c (new)\n +static void hex_format(char *dest, uint8_t *src, int hash_size)\n +{\n +\tassert(hash_size > 0);\n -+\tif (src != NULL) {\n ++\tif (src) {\n +\t\tint i = 0;\n +\t\tfor (i = 0; i < hash_size; i++) {\n +\t\t\tdest[2 * i] = hexdigit(src[i] >> 4);\n @@ reftable/record.c (new)\n +void reftable_ref_record_print(struct reftable_ref_record *ref,\n +\t\t\t uint32_t hash_id)\n +{\n -+\tchar hex[2 * SHA256_SIZE + 1] = { 0 }; /* BUG */\n ++\tchar hex[2 * GIT_SHA256_RAWSZ + 1] = { 0 }; /* BUG */\n +\tprintf(\"ref{%s(%\" PRIu64 \") \", ref->refname, ref->update_index);\n +\tswitch (ref->value_type) {\n +\tcase REFTABLE_REF_SYMREF:\n @@ reftable/record.c (new)\n +\n +static void reftable_ref_record_release_void(void *rec)\n +{\n -+\treftable_ref_record_release((struct reftable_ref_record *)rec);\n ++\treftable_ref_record_release(rec);\n +}\n +\n +void reftable_ref_record_release(struct reftable_ref_record *ref)\n @@ reftable/record.c (new)\n +\t\t\t\t uint8_t val_type, struct string_view in,\n +\t\t\t\t int hash_size)\n +{\n -+\tstruct reftable_ref_record *r = (struct reftable_ref_record *)rec;\n ++\tstruct reftable_ref_record *r = rec;\n +\tstruct string_view start = in;\n +\tuint64_t update_index = 0;\n +\tint n = get_var_int(&update_index, &in);\n @@ reftable/record.c (new)\n +\n +static void reftable_obj_record_release(void *rec)\n +{\n -+\tstruct reftable_obj_record *obj = (struct reftable_obj_record *)rec;\n ++\tstruct reftable_obj_record *obj = rec;\n +\tFREE_AND_NULL(obj->hash_prefix);\n +\tFREE_AND_NULL(obj->offsets);\n +\tmemset(obj, 0, sizeof(struct reftable_obj_record));\n @@ reftable/record.c (new)\n +static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n +\t\t\t\t\t int hash_size)\n +{\n -+\tstruct reftable_obj_record *obj = (struct reftable_obj_record *)rec;\n ++\tstruct reftable_obj_record *obj = rec;\n +\tconst struct reftable_obj_record *src =\n +\t\t(const struct reftable_obj_record *)src_rec;\n +\tint olen;\n @@ reftable/record.c (new)\n +\n +static uint8_t reftable_obj_record_val_type(const void *rec)\n +{\n -+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n ++\tconst struct reftable_obj_record *r = rec;\n +\tif (r->offset_len > 0 && r->offset_len < 8)\n +\t\treturn r->offset_len;\n +\treturn 0;\n @@ reftable/record.c (new)\n +static int reftable_obj_record_encode(const void *rec, struct string_view s,\n +\t\t\t\t int hash_size)\n +{\n -+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n ++\tconst struct reftable_obj_record *r = rec;\n +\tstruct string_view start = s;\n +\tint i = 0;\n +\tint n = 0;\n @@ reftable/record.c (new)\n +\t\t\t\t int hash_size)\n +{\n +\tstruct string_view start = in;\n -+\tstruct reftable_obj_record *r = (struct reftable_obj_record *)rec;\n ++\tstruct reftable_obj_record *r = rec;\n +\tuint64_t count = val_type;\n +\tint n = 0;\n +\tuint64_t last;\n @@ reftable/record.c (new)\n +void reftable_log_record_print(struct reftable_log_record *log,\n +\t\t\t uint32_t hash_id)\n +{\n -+\tchar hex[SHA256_SIZE + 1] = { 0 };\n ++\tchar hex[GIT_SHA256_RAWSZ + 1] = { 0 };\n +\n +\tswitch (log->value_type) {\n +\tcase REFTABLE_LOG_DELETION:\n @@ reftable/record.c (new)\n +static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n +\t\t\t\t\t int hash_size)\n +{\n -+\tstruct reftable_log_record *dst = (struct reftable_log_record *)rec;\n ++\tstruct reftable_log_record *dst = rec;\n +\tconst struct reftable_log_record *src =\n +\t\t(const struct reftable_log_record *)src_rec;\n +\n +\treftable_log_record_release(dst);\n +\t*dst = *src;\n -+\tif (dst->refname != NULL) {\n ++\tif (dst->refname) {\n +\t\tdst->refname = xstrdup(dst->refname);\n +\t}\n +\tswitch (dst->value_type) {\n +\tcase REFTABLE_LOG_DELETION:\n +\t\tbreak;\n +\tcase REFTABLE_LOG_UPDATE:\n -+\t\tif (dst->update.email != NULL) {\n ++\t\tif (dst->update.email) {\n +\t\t\tdst->update.email = xstrdup(dst->update.email);\n +\t\t}\n -+\t\tif (dst->update.name != NULL) {\n ++\t\tif (dst->update.name) {\n +\t\t\tdst->update.name = xstrdup(dst->update.name);\n +\t\t}\n -+\t\tif (dst->update.message != NULL) {\n ++\t\tif (dst->update.message) {\n +\t\t\tdst->update.message = xstrdup(dst->update.message);\n +\t\t}\n +\n -+\t\tif (dst->update.new_hash != NULL) {\n ++\t\tif (dst->update.new_hash) {\n +\t\t\tdst->update.new_hash = reftable_malloc(hash_size);\n +\t\t\tmemcpy(dst->update.new_hash, src->update.new_hash,\n +\t\t\t hash_size);\n +\t\t}\n -+\t\tif (dst->update.old_hash != NULL) {\n ++\t\tif (dst->update.old_hash) {\n +\t\t\tdst->update.old_hash = reftable_malloc(hash_size);\n +\t\t\tmemcpy(dst->update.old_hash, src->update.old_hash,\n +\t\t\t hash_size);\n @@ reftable/record.c (new)\n +\n +static void reftable_log_record_release_void(void *rec)\n +{\n -+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n ++\tstruct reftable_log_record *r = rec;\n +\treftable_log_record_release(r);\n +}\n +\n @@ reftable/record.c (new)\n +\treturn reftable_log_record_is_deletion(log) ? 0 : 1;\n +}\n +\n -+static uint8_t zero[SHA256_SIZE] = { 0 };\n ++static uint8_t zero[GIT_SHA256_RAWSZ] = { 0 };\n +\n +static int reftable_log_record_encode(const void *rec, struct string_view s,\n +\t\t\t\t int hash_size)\n +{\n -+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n ++\tconst struct reftable_log_record *r = rec;\n +\tstruct string_view start = s;\n +\tint n = 0;\n +\tuint8_t *oldh = NULL;\n @@ reftable/record.c (new)\n +\n +\toldh = r->update.old_hash;\n +\tnewh = r->update.new_hash;\n -+\tif (oldh == NULL) {\n ++\tif (!oldh) {\n +\t\toldh = zero;\n +\t}\n -+\tif (newh == NULL) {\n ++\tif (!newh) {\n +\t\tnewh = zero;\n +\t}\n +\n @@ reftable/record.c (new)\n +\t\t\t\t int hash_size)\n +{\n +\tstruct string_view start = in;\n -+\tstruct reftable_log_record *r = (struct reftable_log_record *)rec;\n ++\tstruct reftable_log_record *r = rec;\n +\tuint64_t max = 0;\n +\tuint64_t ts = 0;\n +\tstruct strbuf dest = STRBUF_INIT;\n @@ reftable/record.c (new)\n +static int null_streq(char *a, char *b)\n +{\n +\tchar *empty = \"\";\n -+\tif (a == NULL)\n ++\tif (!a)\n +\t\ta = empty;\n +\n -+\tif (b == NULL)\n ++\tif (!b)\n +\t\tb = empty;\n +\n +\treturn 0 == strcmp(a, b);\n @@ reftable/record.c (new)\n +\n +static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)\n +{\n -+\tif (a == NULL)\n ++\tif (!a)\n +\t\ta = zero;\n +\n -+\tif (b == NULL)\n ++\tif (!b)\n +\t\tb = zero;\n +\n +\treturn !memcmp(a, b, sz);\n @@ reftable/record.c (new)\n +\n +static void reftable_index_record_key(const void *r, struct strbuf *dest)\n +{\n -+\tstruct reftable_index_record *rec = (struct reftable_index_record *)r;\n ++\tconst struct reftable_index_record *rec = r;\n +\tstrbuf_reset(dest);\n +\tstrbuf_addbuf(dest, &rec->last_key);\n +}\n @@ reftable/record.c (new)\n +static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n +\t\t\t\t\t int hash_size)\n +{\n -+\tstruct reftable_index_record *dst = (struct reftable_index_record *)rec;\n -+\tstruct reftable_index_record *src =\n -+\t\t(struct reftable_index_record *)src_rec;\n ++\tstruct reftable_index_record *dst = rec;\n ++\tconst struct reftable_index_record *src = src_rec;\n +\n +\tstrbuf_reset(&dst->last_key);\n +\tstrbuf_addbuf(&dst->last_key, &src->last_key);\n @@ reftable/record.c (new)\n +\n +static void reftable_index_record_release(void *rec)\n +{\n -+\tstruct reftable_index_record *idx = (struct reftable_index_record *)rec;\n ++\tstruct reftable_index_record *idx = rec;\n +\tstrbuf_release(&idx->last_key);\n +}\n +\n @@ reftable/record.c (new)\n +\t\t\t\t\tint hash_size)\n +{\n +\tstruct string_view start = in;\n -+\tstruct reftable_index_record *r = (struct reftable_index_record *)rec;\n ++\tstruct reftable_index_record *r = rec;\n +\tint n = 0;\n +\n +\tstrbuf_reset(&r->last_key);\n @@ reftable/record.c (new)\n +void reftable_record_from_ref(struct reftable_record *rec,\n +\t\t\t struct reftable_ref_record *ref_rec)\n +{\n -+\tassert(rec->ops == NULL);\n ++\tassert(!rec->ops);\n +\trec->data = ref_rec;\n +\trec->ops = &reftable_ref_record_vtable;\n +}\n @@ reftable/record.c (new)\n +void reftable_record_from_obj(struct reftable_record *rec,\n +\t\t\t struct reftable_obj_record *obj_rec)\n +{\n -+\tassert(rec->ops == NULL);\n ++\tassert(!rec->ops);\n +\trec->data = obj_rec;\n +\trec->ops = &reftable_obj_record_vtable;\n +}\n @@ reftable/record.c (new)\n +void reftable_record_from_index(struct reftable_record *rec,\n +\t\t\t\tstruct reftable_index_record *index_rec)\n +{\n -+\tassert(rec->ops == NULL);\n ++\tassert(!rec->ops);\n +\trec->data = index_rec;\n +\trec->ops = &reftable_index_record_vtable;\n +}\n @@ reftable/record.c (new)\n +void reftable_record_from_log(struct reftable_record *rec,\n +\t\t\t struct reftable_log_record *log_rec)\n +{\n -+\tassert(rec->ops == NULL);\n ++\tassert(!rec->ops);\n +\trec->data = log_rec;\n +\trec->ops = &reftable_log_record_vtable;\n +}\n @@ reftable/record.c (new)\n +struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)\n +{\n +\tassert(reftable_record_type(rec) == BLOCK_TYPE_REF);\n -+\treturn (struct reftable_ref_record *)rec->data;\n ++\treturn rec->data;\n +}\n +\n +struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)\n +{\n +\tassert(reftable_record_type(rec) == BLOCK_TYPE_LOG);\n -+\treturn (struct reftable_log_record *)rec->data;\n ++\treturn rec->data;\n +}\n +\n +static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)\n +{\n -+\tif (a != NULL && b != NULL)\n ++\tif (a && b)\n +\t\treturn !memcmp(a, b, hash_size);\n +\n +\treturn a == b;\n @@ reftable/record.c (new)\n +\n +int reftable_log_record_compare_key(const void *a, const void *b)\n +{\n -+\tstruct reftable_log_record *la = (struct reftable_log_record *)a;\n -+\tstruct reftable_log_record *lb = (struct reftable_log_record *)b;\n ++\tconst struct reftable_log_record *la = a;\n ++\tconst struct reftable_log_record *lb = b;\n +\n +\tint cmp = strcmp(la->refname, lb->refname);\n +\tif (cmp)\n @@ reftable/record_test.c (new)\n +{\n +\tstruct reftable_record copy =\n +\t\treftable_new_record(reftable_record_type(rec));\n -+\treftable_record_copy_from(©, rec, SHA1_SIZE);\n ++\treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n +\t/* do it twice to catch memory leaks */\n -+\treftable_record_copy_from(©, rec, SHA1_SIZE);\n ++\treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n +\tswitch (reftable_record_type(©)) {\n +\tcase BLOCK_TYPE_REF:\n +\t\tEXPECT(reftable_ref_record_equal(reftable_record_as_ref(©),\n +\t\t\t\t\t\t reftable_record_as_ref(rec),\n -+\t\t\t\t\t\t SHA1_SIZE));\n ++\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n +\t\tbreak;\n +\tcase BLOCK_TYPE_LOG:\n +\t\tEXPECT(reftable_log_record_equal(reftable_record_as_log(©),\n +\t\t\t\t\t\t reftable_record_as_log(rec),\n -+\t\t\t\t\t\t SHA1_SIZE));\n ++\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n +\t\tbreak;\n +\t}\n +\treftable_record_destroy(©);\n @@ reftable/record_test.c (new)\n +static void set_hash(uint8_t *h, int j)\n +{\n +\tint i = 0;\n -+\tfor (i = 0; i < hash_size(SHA1_ID); i++) {\n ++\tfor (i = 0; i < hash_size(GIT_SHA1_HASH_ID); i++) {\n +\t\th[i] = (j >> i) & 0xff;\n +\t}\n +}\n @@ reftable/record_test.c (new)\n +\t\tcase REFTABLE_REF_DELETION:\n +\t\t\tbreak;\n +\t\tcase REFTABLE_REF_VAL1:\n -+\t\t\tin.value.val1 = reftable_malloc(SHA1_SIZE);\n ++\t\t\tin.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);\n +\t\t\tset_hash(in.value.val1, 1);\n +\t\t\tbreak;\n +\t\tcase REFTABLE_REF_VAL2:\n -+\t\t\tin.value.val2.value = reftable_malloc(SHA1_SIZE);\n ++\t\t\tin.value.val2.value = reftable_malloc(GIT_SHA1_RAWSZ);\n +\t\t\tset_hash(in.value.val2.value, 1);\n -+\t\t\tin.value.val2.target_value = reftable_malloc(SHA1_SIZE);\n ++\t\t\tin.value.val2.target_value =\n ++\t\t\t\treftable_malloc(GIT_SHA1_RAWSZ);\n +\t\t\tset_hash(in.value.val2.target_value, 2);\n +\t\t\tbreak;\n +\t\tcase REFTABLE_REF_SYMREF:\n @@ reftable/record_test.c (new)\n +\t\tEXPECT(reftable_record_val_type(&rec) == i);\n +\n +\t\treftable_record_key(&rec, &key);\n -+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n ++\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n +\t\tEXPECT(n > 0);\n +\n +\t\t/* decode into a non-zero reftable_record to test for leaks. */\n +\n +\t\treftable_record_from_ref(&rec_out, &out);\n -+\t\tm = reftable_record_decode(&rec_out, key, i, dest, SHA1_SIZE);\n ++\t\tm = reftable_record_decode(&rec_out, key, i, dest,\n ++\t\t\t\t\t GIT_SHA1_RAWSZ);\n +\t\tEXPECT(n == m);\n +\n -+\t\tEXPECT(reftable_ref_record_equal(&in, &out, SHA1_SIZE));\n ++\t\tEXPECT(reftable_ref_record_equal(&in, &out, GIT_SHA1_RAWSZ));\n +\t\treftable_record_release(&rec_out);\n +\n +\t\tstrbuf_release(&key);\n @@ reftable/record_test.c (new)\n +\t\t}\n +\t};\n +\n -+\tEXPECT(!reftable_log_record_equal(&in[0], &in[1], SHA1_SIZE));\n ++\tEXPECT(!reftable_log_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n +\tin[1].update_index = in[0].update_index;\n -+\tEXPECT(reftable_log_record_equal(&in[0], &in[1], SHA1_SIZE));\n ++\tEXPECT(reftable_log_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n +\treftable_log_record_release(&in[0]);\n +\treftable_log_record_release(&in[1]);\n +}\n @@ reftable/record_test.c (new)\n +\t\t\t.update_index = 42,\n +\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n +\t\t\t.update = {\n -+\t\t\t\t.old_hash = reftable_malloc(SHA1_SIZE),\n -+\t\t\t\t.new_hash = reftable_malloc(SHA1_SIZE),\n ++\t\t\t\t.old_hash = reftable_malloc(GIT_SHA1_RAWSZ),\n ++\t\t\t\t.new_hash = reftable_malloc(GIT_SHA1_RAWSZ),\n +\t\t\t\t.name = xstrdup(\"han-wen\"),\n +\t\t\t\t.email = xstrdup(\"hanwen@google.com\"),\n +\t\t\t\t.message = xstrdup(\"test\"),\n @@ reftable/record_test.c (new)\n +\t\t\t.refname = xstrdup(\"old name\"),\n +\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n +\t\t\t.update = {\n -+\t\t\t\t.new_hash = reftable_calloc(SHA1_SIZE),\n -+\t\t\t\t.old_hash = reftable_calloc(SHA1_SIZE),\n ++\t\t\t\t.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n ++\t\t\t\t.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n +\t\t\t\t.name = xstrdup(\"old name\"),\n +\t\t\t\t.email = xstrdup(\"old@email\"),\n +\t\t\t\t.message = xstrdup(\"old message\"),\n @@ reftable/record_test.c (new)\n +\n +\t\treftable_record_key(&rec, &key);\n +\n -+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n ++\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n +\t\tEXPECT(n >= 0);\n +\t\treftable_record_from_log(&rec_out, &out);\n +\t\tvaltype = reftable_record_val_type(&rec);\n +\t\tm = reftable_record_decode(&rec_out, key, valtype, dest,\n -+\t\t\t\t\t SHA1_SIZE);\n ++\t\t\t\t\t GIT_SHA1_RAWSZ);\n +\t\tEXPECT(n == m);\n +\n -+\t\tEXPECT(reftable_log_record_equal(&in[i], &out, SHA1_SIZE));\n ++\t\tEXPECT(reftable_log_record_equal(&in[i], &out, GIT_SHA1_RAWSZ));\n +\t\treftable_log_record_release(&in[i]);\n +\t\tstrbuf_release(&key);\n +\t\treftable_record_release(&rec_out);\n @@ reftable/record_test.c (new)\n +\n +static void test_reftable_obj_record_roundtrip(void)\n +{\n -+\tuint8_t testHash1[SHA1_SIZE] = { 1, 2, 3, 4, 0 };\n ++\tuint8_t testHash1[GIT_SHA1_RAWSZ] = { 1, 2, 3, 4, 0 };\n +\tuint64_t till9[] = { 1, 2, 3, 4, 500, 600, 700, 800, 9000 };\n +\tstruct reftable_obj_record recs[3] = { {\n +\t\t\t\t\t\t .hash_prefix = testHash1,\n @@ reftable/record_test.c (new)\n +\t\treftable_record_from_obj(&rec, &in);\n +\t\ttest_copy(&rec);\n +\t\treftable_record_key(&rec, &key);\n -+\t\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n ++\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n +\t\tEXPECT(n > 0);\n +\t\textra = reftable_record_val_type(&rec);\n +\t\treftable_record_from_obj(&rec_out, &out);\n +\t\tm = reftable_record_decode(&rec_out, key, extra, dest,\n -+\t\t\t\t\t SHA1_SIZE);\n ++\t\t\t\t\t GIT_SHA1_RAWSZ);\n +\t\tEXPECT(n == m);\n +\n +\t\tEXPECT(in.hash_prefix_len == out.hash_prefix_len);\n @@ reftable/record_test.c (new)\n +\ttest_copy(&rec);\n +\n +\tEXPECT(0 == strbuf_cmp(&key, &in.last_key));\n -+\tn = reftable_record_encode(&rec, dest, SHA1_SIZE);\n ++\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n +\tEXPECT(n > 0);\n +\n +\textra = reftable_record_val_type(&rec);\n +\treftable_record_from_index(&out_rec, &out);\n -+\tm = reftable_record_decode(&out_rec, key, extra, dest, SHA1_SIZE);\n ++\tm = reftable_record_decode(&out_rec, key, extra, dest, GIT_SHA1_RAWSZ);\n +\tEXPECT(m == n);\n +\n +\tEXPECT(in.offset == out.offset);\n -: ------------ > 11: 0c2604d7e7d4 Provide zlib's uncompress2 from compat/zlib-compat.c\n 7: 9297b9c363f6 ! 12: 425cd0185aab reftable: reading/writing blocks\n @@ Commit message\n \n This commit provides the logic to read and write these blocks.\n \n - Includes a code snippet copied from zlib\n -\n Signed-off-by: Han-Wen Nienhuys \n \n ## Makefile ##\n @@ Makefile: xdiff-objs: $(XDIFF_OBJS)\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n -+REFTABLE_OBJS += reftable/zlib-compat.o\n \n +REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n \n - ## reftable/.gitattributes (new) ##\n -@@\n -+/zlib-compat.c\twhitespace=-indent-with-non-tab,-trailing-space\n -\n ## reftable/block.c (new) ##\n @@\n +/*\n @@ reftable/block.c (new)\n +#include \"record.h\"\n +#include \"reftable-error.h\"\n +#include \"system.h\"\n -+#include \"zlib.h\"\n ++#include \n ++\n ++#ifdef NO_UNCOMPRESS2\n ++/* This is uncompress2, which is only available in zlib as of 2017.\n ++ */\n ++int uncompress2(Bytef *dest, uLongf *destLen, const Bytef *source,\n ++\t\tuLong *sourceLen);\n ++#endif\n +\n +int header_size(int version)\n +{\n @@ reftable/block.c (new)\n +\t\tmemcpy(uncompressed, block->data, block_header_skip);\n +\n +\t\t/* Uncompress */\n -+\t\tif (Z_OK != uncompress_return_consumed(\n -+\t\t\t\t uncompressed + block_header_skip, &dst_len,\n -+\t\t\t\t block->data + block_header_skip,\n -+\t\t\t\t &src_len)) {\n ++\t\tif (Z_OK !=\n ++\t\t uncompress2(uncompressed + block_header_skip, &dst_len,\n ++\t\t\t\tblock->data + block_header_skip, &src_len)) {\n +\t\t\treftable_free(uncompressed);\n +\t\t\treturn REFTABLE_ZLIB_ERROR;\n +\t\t}\n @@ reftable/block.c (new)\n +\n +static int restart_key_less(size_t idx, void *args)\n +{\n -+\tstruct restart_find_args *a = (struct restart_find_args *)args;\n ++\tstruct restart_find_args *a = args;\n +\tuint32_t off = block_reader_restart_offset(a->r, idx);\n +\tstruct string_view in = {\n +\t\t.buf = a->r->block.data + off,\n @@ reftable/block.c (new)\n +void reftable_block_done(struct reftable_block *blockp)\n +{\n +\tstruct reftable_block_source source = blockp->source;\n -+\tif (blockp != NULL && source.ops != NULL)\n ++\tif (blockp && source.ops)\n +\t\tsource.ops->return_block(source.arg, blockp);\n +\tblockp->data = NULL;\n +\tblockp->len = 0;\n @@ reftable/block_test.c (new)\n +\tblock.len = block_size;\n +\tblock.source = malloc_block_source();\n +\tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n -+\t\t\t header_off, hash_size(SHA1_ID));\n ++\t\t\t header_off, hash_size(GIT_SHA1_HASH_ID));\n +\treftable_record_from_ref(&rec, &ref);\n +\n +\tfor (i = 0; i < N; i++) {\n +\t\tchar name[100];\n -+\t\tuint8_t hash[SHA1_SIZE];\n ++\t\tuint8_t hash[GIT_SHA1_RAWSZ];\n +\t\tsnprintf(name, sizeof(name), \"branch%02d\", i);\n +\t\tmemset(hash, i, sizeof(hash));\n +\n @@ reftable/block_test.c (new)\n +\n +\tblock_writer_release(&bw);\n +\n -+\tblock_reader_init(&br, &block, header_off, block_size, SHA1_SIZE);\n ++\tblock_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ);\n +\n +\tblock_reader_start(&br, &it);\n +\n @@ reftable/block_test.c (new)\n +{\n +\tRUN_TEST(test_block_read_write);\n +\treturn 0;\n -+}\n -\n - ## reftable/zlib-compat.c (new) ##\n -@@\n -+/* taken from zlib's uncompr.c\n -+\n -+ commit cacf7f1d4e3d44d871b605da3b647f07d718623f\n -+ Author: Mark Adler \n -+ Date: Sun Jan 15 09:18:46 2017 -0800\n -+\n -+ zlib 1.2.11\n -+\n -+*/\n -+\n -+/*\n -+ * Copyright (C) 1995-2003, 2010, 2014, 2016 Jean-loup Gailly, Mark Adler\n -+ * For conditions of distribution and use, see copyright notice in zlib.h\n -+ */\n -+\n -+#include \"system.h\"\n -+\n -+/* clang-format off */\n -+\n -+/* ===========================================================================\n -+ Decompresses the source buffer into the destination buffer. *sourceLen is\n -+ the byte length of the source buffer. Upon entry, *destLen is the total size\n -+ of the destination buffer, which must be large enough to hold the entire\n -+ uncompressed data. (The size of the uncompressed data must have been saved\n -+ previously by the compressor and transmitted to the decompressor by some\n -+ mechanism outside the scope of this compression library.) Upon exit,\n -+ *destLen is the size of the decompressed data and *sourceLen is the number\n -+ of source bytes consumed. Upon return, source + *sourceLen points to the\n -+ first unused input byte.\n -+\n -+ uncompress returns Z_OK if success, Z_MEM_ERROR if there was not enough\n -+ memory, Z_BUF_ERROR if there was not enough room in the output buffer, or\n -+ Z_DATA_ERROR if the input data was corrupted, including if the input data is\n -+ an incomplete zlib stream.\n -+*/\n -+int ZEXPORT uncompress_return_consumed (\n -+ Bytef *dest,\n -+ uLongf *destLen,\n -+ const Bytef *source,\n -+ uLong *sourceLen) {\n -+ z_stream stream;\n -+ int err;\n -+ const uInt max = (uInt)-1;\n -+ uLong len, left;\n -+ Byte buf[1]; /* for detection of incomplete stream when *destLen == 0 */\n -+\n -+ len = *sourceLen;\n -+ if (*destLen) {\n -+ left = *destLen;\n -+ *destLen = 0;\n -+ }\n -+ else {\n -+ left = 1;\n -+ dest = buf;\n -+ }\n -+\n -+ stream.next_in = (z_const Bytef *)source;\n -+ stream.avail_in = 0;\n -+ stream.zalloc = (alloc_func)0;\n -+ stream.zfree = (free_func)0;\n -+ stream.opaque = (voidpf)0;\n -+\n -+ err = inflateInit(&stream);\n -+ if (err != Z_OK) return err;\n -+\n -+ stream.next_out = dest;\n -+ stream.avail_out = 0;\n -+\n -+ do {\n -+ if (stream.avail_out == 0) {\n -+ stream.avail_out = left > (uLong)max ? max : (uInt)left;\n -+ left -= stream.avail_out;\n -+ }\n -+ if (stream.avail_in == 0) {\n -+ stream.avail_in = len > (uLong)max ? max : (uInt)len;\n -+ len -= stream.avail_in;\n -+ }\n -+ err = inflate(&stream, Z_NO_FLUSH);\n -+ } while (err == Z_OK);\n -+\n -+ *sourceLen -= len + stream.avail_in;\n -+ if (dest != buf)\n -+ *destLen = stream.total_out;\n -+ else if (stream.total_out && err == Z_BUF_ERROR)\n -+ left = 1;\n -+\n -+ inflateEnd(&stream);\n -+ return err == Z_STREAM_END ? Z_OK :\n -+ err == Z_NEED_DICT ? Z_DATA_ERROR :\n -+ err == Z_BUF_ERROR && left + stream.avail_out ? Z_DATA_ERROR :\n -+ err;\n +}\n \n ## t/helper/test-reftable.c ##\n 8: ceddefadd48c ! 13: c03957db45c0 reftable: a generic binary tree implementation\n @@ Makefile: REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n +REFTABLE_OBJS += reftable/tree.o\n - REFTABLE_OBJS += reftable/zlib-compat.o\n \n +REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n @@ reftable/tree.c (new)\n +void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n +\t\tvoid *arg)\n +{\n -+\tif (t->left != NULL) {\n ++\tif (t->left) {\n +\t\tinfix_walk(t->left, action, arg);\n +\t}\n +\taction(arg, t->key);\n -+\tif (t->right != NULL) {\n ++\tif (t->right) {\n +\t\tinfix_walk(t->right, action, arg);\n +\t}\n +}\n @@ reftable/tree.c (new)\n +\tif (t == NULL) {\n +\t\treturn;\n +\t}\n -+\tif (t->left != NULL) {\n ++\tif (t->left) {\n +\t\ttree_free(t->left);\n +\t}\n -+\tif (t->right != NULL) {\n ++\tif (t->right) {\n +\t\ttree_free(t->right);\n +\t}\n +\treftable_free(t);\n @@ reftable/tree_test.c (new)\n +\n +static void check_increasing(void *arg, void *key)\n +{\n -+\tstruct curry *c = (struct curry *)arg;\n -+\tif (c->last != NULL) {\n ++\tstruct curry *c = arg;\n ++\tif (c->last) {\n +\t\tassert(test_compare(c->last, key) < 0);\n +\t}\n +\tc->last = key;\n 9: 1f635e6c86c5 ! 14: 922deb272fff reftable: write reftable files\n @@ Makefile: REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/tree.o\n +REFTABLE_OBJS += reftable/writer.o\n - REFTABLE_OBJS += reftable/zlib-compat.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n + REFTABLE_TEST_OBJS += reftable/block_test.o\n \n ## reftable/reftable-writer.h (new) ##\n @@\n @@ reftable/reftable-writer.h (new)\n +\n +/* reftable_new_writer creates a new writer */\n +struct reftable_writer *\n -+reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n ++reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n +\t\t void *writer_arg, struct reftable_write_options *opts);\n +\n +/* Set the range of update indices for the records we will add. When writing a\n @@ reftable/writer.c (new)\n +\t}\n +\n +\tif (opts->hash_id == 0) {\n -+\t\topts->hash_id = SHA1_ID;\n ++\t\topts->hash_id = GIT_SHA1_HASH_ID;\n +\t}\n +\tif (opts->block_size == 0) {\n +\t\topts->block_size = DEFAULT_BLOCK_SIZE;\n @@ reftable/writer.c (new)\n +\n +static int writer_version(struct reftable_writer *w)\n +{\n -+\treturn (w->opts.hash_id == 0 || w->opts.hash_id == SHA1_ID) ? 1 : 2;\n ++\treturn (w->opts.hash_id == 0 || w->opts.hash_id == GIT_SHA1_HASH_ID) ?\n ++\t\t\t 1 :\n ++\t\t\t 2;\n +}\n +\n +static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n @@ reftable/writer.c (new)\n +static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n +\n +struct reftable_writer *\n -+reftable_new_writer(int (*writer_func)(void *, const void *, size_t),\n ++reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n +\t\t void *writer_arg, struct reftable_write_options *opts)\n +{\n +\tstruct reftable_writer *wp =\n @@ reftable/writer.c (new)\n +static int writer_add_record(struct reftable_writer *w,\n +\t\t\t struct reftable_record *rec)\n +{\n -+\tint result = -1;\n +\tstruct strbuf key = STRBUF_INIT;\n -+\tint err = 0;\n ++\tint err = -1;\n +\treftable_record_key(rec, &key);\n -+\tif (strbuf_cmp(&w->last_key, &key) >= 0)\n ++\tif (strbuf_cmp(&w->last_key, &key) >= 0) {\n ++\t\terr = REFTABLE_API_ERROR;\n +\t\tgoto done;\n ++\t}\n +\n +\tstrbuf_reset(&w->last_key);\n +\tstrbuf_addbuf(&w->last_key, &key);\n @@ reftable/writer.c (new)\n +\tassert(block_writer_type(w->block_writer) == reftable_record_type(rec));\n +\n +\tif (block_writer_add(w->block_writer, rec) == 0) {\n -+\t\tresult = 0;\n ++\t\terr = 0;\n +\t\tgoto done;\n +\t}\n +\n +\terr = writer_flush_block(w);\n +\tif (err < 0) {\n -+\t\tresult = err;\n +\t\tgoto done;\n +\t}\n +\n +\twriter_reinit_block_writer(w, reftable_record_type(rec));\n +\terr = block_writer_add(w->block_writer, rec);\n +\tif (err < 0) {\n -+\t\tresult = err;\n +\t\tgoto done;\n +\t}\n +\n -+\tresult = 0;\n ++\terr = 0;\n +done:\n +\tstrbuf_release(&key);\n -+\treturn result;\n ++\treturn err;\n +}\n +\n +int reftable_writer_add_ref(struct reftable_writer *w,\n @@ reftable/writer.c (new)\n +\tif (err < 0)\n +\t\treturn err;\n +\n -+\tif (!w->opts.skip_index_objects &&\n -+\t reftable_ref_record_val1(ref) != NULL) {\n ++\tif (!w->opts.skip_index_objects && reftable_ref_record_val1(ref)) {\n +\t\tstruct strbuf h = STRBUF_INIT;\n +\t\tstrbuf_add(&h, (char *)reftable_ref_record_val1(ref),\n +\t\t\t hash_size(w->opts.hash_id));\n @@ reftable/writer.c (new)\n +\t\tstrbuf_release(&h);\n +\t}\n +\n -+\tif (!w->opts.skip_index_objects &&\n -+\t reftable_ref_record_val2(ref) != NULL) {\n ++\tif (!w->opts.skip_index_objects && reftable_ref_record_val2(ref)) {\n +\t\tstruct strbuf h = STRBUF_INIT;\n +\t\tstrbuf_add(&h, reftable_ref_record_val2(ref),\n +\t\t\t hash_size(w->opts.hash_id));\n @@ reftable/writer.c (new)\n +\t\t\t\t\t struct reftable_log_record *log)\n +{\n +\tstruct reftable_record rec = { NULL };\n -+\tif (w->block_writer != NULL &&\n ++\tif (w->block_writer &&\n +\t block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {\n +\t\tint err = writer_finish_public_section(w);\n +\t\tif (err < 0)\n @@ reftable/writer.c (new)\n +\t\treturn REFTABLE_API_ERROR;\n +\n +\tinput_log_message = log->update.message;\n -+\tif (!w->opts.exact_log_message && log->update.message != NULL) {\n ++\tif (!w->opts.exact_log_message && log->update.message) {\n +\t\tstrbuf_addstr(&cleaned_message, log->update.message);\n +\t\twhile (cleaned_message.len &&\n +\t\t cleaned_message.buf[cleaned_message.len - 1] == '\\n')\n @@ reftable/writer.c (new)\n +\n +static void update_common(void *void_arg, void *key)\n +{\n -+\tstruct common_prefix_arg *arg = (struct common_prefix_arg *)void_arg;\n -+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n -+\tif (arg->last != NULL) {\n ++\tstruct common_prefix_arg *arg = void_arg;\n ++\tstruct obj_index_tree_node *entry = key;\n ++\tif (arg->last) {\n +\t\tint n = common_prefix_size(&entry->hash, arg->last);\n +\t\tif (n > arg->max) {\n +\t\t\targ->max = n;\n @@ reftable/writer.c (new)\n +\n +static void write_object_record(void *void_arg, void *key)\n +{\n -+\tstruct write_record_arg *arg = (struct write_record_arg *)void_arg;\n -+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n ++\tstruct write_record_arg *arg = void_arg;\n ++\tstruct obj_index_tree_node *entry = key;\n +\tstruct reftable_obj_record obj_rec = {\n +\t\t.hash_prefix = (uint8_t *)entry->hash.buf,\n +\t\t.hash_prefix_len = arg->w->stats.object_id_len,\n @@ reftable/writer.c (new)\n +\n +static void object_record_free(void *void_arg, void *key)\n +{\n -+\tstruct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;\n ++\tstruct obj_index_tree_node *entry = key;\n +\n +\tFREE_AND_NULL(entry->offsets);\n +\tstrbuf_release(&entry->hash);\n @@ reftable/writer.c (new)\n +{\n +\tstruct write_record_arg closure = { .w = w };\n +\tstruct common_prefix_arg common = { NULL };\n -+\tif (w->obj_index_tree != NULL) {\n ++\tif (w->obj_index_tree) {\n +\t\tinfix_walk(w->obj_index_tree, &update_common, &common);\n +\t}\n +\tw->stats.object_id_len = common.max + 1;\n +\n +\twriter_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n +\n -+\tif (w->obj_index_tree != NULL) {\n ++\tif (w->obj_index_tree) {\n +\t\tinfix_walk(w->obj_index_tree, &write_object_record, &closure);\n +\t}\n +\n @@ reftable/writer.c (new)\n +\t\t\treturn err;\n +\t}\n +\n -+\tif (w->obj_index_tree != NULL) {\n ++\tif (w->obj_index_tree) {\n +\t\tinfix_walk(w->obj_index_tree, &object_record_free, NULL);\n +\t\ttree_free(w->obj_index_tree);\n +\t\tw->obj_index_tree = NULL;\n @@ reftable/writer.h (new)\n +#include \"reftable-writer.h\"\n +\n +struct reftable_writer {\n -+\tint (*write)(void *, const void *, size_t);\n ++\tssize_t (*write)(void *, const void *, size_t);\n +\tvoid *write_arg;\n +\tint pending_padding;\n +\tstruct strbuf last_key;\n 10: 9c3fd77fbbac ! 15: afca8f8f6a29 reftable: generic interface to tables\n @@ Makefile: REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n -+REFTABLE_OBJS += reftable/reftable.o\n ++REFTABLE_OBJS += reftable/refname.o\n ++REFTABLE_OBJS += reftable/generic.o\n ++REFTABLE_OBJS += reftable/stack.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n - REFTABLE_OBJS += reftable/zlib-compat.o\n + \n +\n + ## reftable/generic.c (new) ##\n +@@\n ++/*\n ++Copyright 2020 Google LLC\n ++\n ++Use of this source code is governed by a BSD-style\n ++license that can be found in the LICENSE file or at\n ++https://developers.google.com/open-source/licenses/bsd\n ++*/\n ++\n ++#include \"basics.h\"\n ++#include \"record.h\"\n ++#include \"generic.h\"\n ++#include \"reftable-iterator.h\"\n ++#include \"reftable-generic.h\"\n ++\n ++int reftable_table_seek_ref(struct reftable_table *tab,\n ++\t\t\t struct reftable_iterator *it, const char *name)\n ++{\n ++\tstruct reftable_ref_record ref = {\n ++\t\t.refname = (char *)name,\n ++\t};\n ++\tstruct reftable_record rec = { NULL };\n ++\treftable_record_from_ref(&rec, &ref);\n ++\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n ++}\n ++\n ++int reftable_table_seek_log(struct reftable_table *tab,\n ++\t\t\t struct reftable_iterator *it, const char *name)\n ++{\n ++\tstruct reftable_log_record log = {\n ++\t\t.refname = (char *)name,\n ++\t\t.update_index = ~((uint64_t)0),\n ++\t};\n ++\tstruct reftable_record rec = { NULL };\n ++\treftable_record_from_log(&rec, &log);\n ++\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n ++}\n ++\n ++int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n ++\t\t\t struct reftable_ref_record *ref)\n ++{\n ++\tstruct reftable_iterator it = { NULL };\n ++\tint err = reftable_table_seek_ref(tab, &it, name);\n ++\tif (err)\n ++\t\tgoto done;\n ++\n ++\terr = reftable_iterator_next_ref(&it, ref);\n ++\tif (err)\n ++\t\tgoto done;\n ++\n ++\tif (strcmp(ref->refname, name) ||\n ++\t reftable_ref_record_is_deletion(ref)) {\n ++\t\treftable_ref_record_release(ref);\n ++\t\terr = 1;\n ++\t\tgoto done;\n ++\t}\n ++\n ++done:\n ++\treftable_iterator_destroy(&it);\n ++\treturn err;\n ++}\n ++\n ++int reftable_table_print(struct reftable_table *tab) {\n ++\tstruct reftable_iterator it = { NULL };\n ++\tstruct reftable_ref_record ref = { NULL };\n ++\tstruct reftable_log_record log = { NULL };\n ++\tuint32_t hash_id = reftable_table_hash_id(tab);\n ++\tint err = reftable_table_seek_ref(tab, &it, \"\");\n ++\tif (err < 0) {\n ++\t\treturn err;\n ++\t}\n ++\n ++\twhile (1) {\n ++\t\terr = reftable_iterator_next_ref(&it, &ref);\n ++\t\tif (err > 0) {\n ++\t\t\tbreak;\n ++\t\t}\n ++\t\tif (err < 0) {\n ++\t\t\treturn err;\n ++\t\t}\n ++\t\treftable_ref_record_print(&ref, hash_id);\n ++\t}\n ++\treftable_iterator_destroy(&it);\n ++\treftable_ref_record_release(&ref);\n ++\n ++\terr = reftable_table_seek_log(tab, &it, \"\");\n ++\tif (err < 0) {\n ++\t\treturn err;\n ++\t}\n ++\twhile (1) {\n ++\t\terr = reftable_iterator_next_log(&it, &log);\n ++\t\tif (err > 0) {\n ++\t\t\tbreak;\n ++\t\t}\n ++\t\tif (err < 0) {\n ++\t\t\treturn err;\n ++\t\t}\n ++\t\treftable_log_record_print(&log, hash_id);\n ++\t}\n ++\treftable_iterator_destroy(&it);\n ++\treftable_log_record_release(&log);\n ++\treturn 0;\n ++}\n ++\n ++uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n ++{\n ++\treturn tab->ops->max_update_index(tab->table_arg);\n ++}\n ++\n ++uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n ++{\n ++\treturn tab->ops->min_update_index(tab->table_arg);\n ++}\n ++\n ++uint32_t reftable_table_hash_id(struct reftable_table *tab)\n ++{\n ++\treturn tab->ops->hash_id(tab->table_arg);\n ++}\n ++\n ++void reftable_iterator_destroy(struct reftable_iterator *it)\n ++{\n ++\tif (!it->ops) {\n ++\t\treturn;\n ++\t}\n ++\tit->ops->close(it->iter_arg);\n ++\tit->ops = NULL;\n ++\tFREE_AND_NULL(it->iter_arg);\n ++}\n ++\n ++int reftable_iterator_next_ref(struct reftable_iterator *it,\n ++\t\t\t struct reftable_ref_record *ref)\n ++{\n ++\tstruct reftable_record rec = { NULL };\n ++\treftable_record_from_ref(&rec, ref);\n ++\treturn iterator_next(it, &rec);\n ++}\n ++\n ++int reftable_iterator_next_log(struct reftable_iterator *it,\n ++\t\t\t struct reftable_log_record *log)\n ++{\n ++\tstruct reftable_record rec = { NULL };\n ++\treftable_record_from_log(&rec, log);\n ++\treturn iterator_next(it, &rec);\n ++}\n ++\n ++int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n ++{\n ++\treturn it->ops->next(it->iter_arg, rec);\n ++}\n ++\n ++static int empty_iterator_next(void *arg, struct reftable_record *rec)\n ++{\n ++\treturn 1;\n ++}\n ++\n ++static void empty_iterator_close(void *arg)\n ++{\n ++}\n ++\n ++static struct reftable_iterator_vtable empty_vtable = {\n ++\t.next = &empty_iterator_next,\n ++\t.close = &empty_iterator_close,\n ++};\n ++\n ++void iterator_set_empty(struct reftable_iterator *it)\n ++{\n ++\tassert(!it->ops);\n ++\tit->iter_arg = NULL;\n ++\tit->ops = &empty_vtable;\n ++}\n \n ## reftable/generic.h (new) ##\n @@\n @@ reftable/reftable-generic.h (new)\n +\tvoid *table_arg;\n +};\n +\n ++int reftable_table_seek_log(struct reftable_table *tab,\n ++\t\t\t struct reftable_iterator *it, const char *name);\n ++\n +int reftable_table_seek_ref(struct reftable_table *tab,\n +\t\t\t struct reftable_iterator *it, const char *name);\n +\n @@ reftable/reftable-generic.h (new)\n +int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n +\t\t\t struct reftable_ref_record *ref);\n +\n ++/* dump table contents onto stdout for debugging */\n ++int reftable_table_print(struct reftable_table *tab);\n ++\n +#endif\n \n ## reftable/reftable-iterator.h (new) ##\n @@ reftable/reftable.c (new)\n +\n +void reftable_iterator_destroy(struct reftable_iterator *it)\n +{\n -+\tif (it->ops == NULL) {\n ++\tif (!it->ops) {\n +\t\treturn;\n +\t}\n +\tit->ops->close(it->iter_arg);\n @@ reftable/reftable.c (new)\n +\n +void iterator_set_empty(struct reftable_iterator *it)\n +{\n -+\tassert(it->ops == NULL);\n ++\tassert(!it->ops);\n +\tit->iter_arg = NULL;\n +\tit->ops = &empty_vtable;\n +}\n 11: 8ba486f44a72 ! 16: 501cf0677ce4 reftable: read reftable files\n @@ Makefile: REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/publicbasics.o\n +REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n - REFTABLE_OBJS += reftable/reftable.o\n - REFTABLE_OBJS += reftable/tree.o\n + REFTABLE_OBJS += reftable/refname.o\n + REFTABLE_OBJS += reftable/generic.o\n \n ## reftable/iter.c (new) ##\n @@\n @@ reftable/iter.c (new)\n +\n +int iterator_is_null(struct reftable_iterator *it)\n +{\n -+\treturn it->ops == NULL;\n ++\treturn !it->ops;\n +}\n +\n +static void filtering_ref_iterator_close(void *iter_arg)\n +{\n -+\tstruct filtering_ref_iterator *fri =\n -+\t\t(struct filtering_ref_iterator *)iter_arg;\n ++\tstruct filtering_ref_iterator *fri = iter_arg;\n +\tstrbuf_release(&fri->oid);\n +\treftable_iterator_destroy(&fri->it);\n +}\n @@ reftable/iter.c (new)\n +static int filtering_ref_iterator_next(void *iter_arg,\n +\t\t\t\t struct reftable_record *rec)\n +{\n -+\tstruct filtering_ref_iterator *fri =\n -+\t\t(struct filtering_ref_iterator *)iter_arg;\n -+\tstruct reftable_ref_record *ref =\n -+\t\t(struct reftable_ref_record *)rec->data;\n ++\tstruct filtering_ref_iterator *fri = iter_arg;\n ++\tstruct reftable_ref_record *ref = rec->data;\n +\tint err = 0;\n +\twhile (1) {\n +\t\terr = reftable_iterator_next_ref(&fri->it, ref);\n @@ reftable/iter.c (new)\n +void iterator_from_filtering_ref_iterator(struct reftable_iterator *it,\n +\t\t\t\t\t struct filtering_ref_iterator *fri)\n +{\n -+\tassert(it->ops == NULL);\n ++\tassert(!it->ops);\n +\tit->iter_arg = fri;\n +\tit->ops = &filtering_ref_iterator_vtable;\n +}\n +\n +static void indexed_table_ref_iter_close(void *p)\n +{\n -+\tstruct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p;\n ++\tstruct indexed_table_ref_iter *it = p;\n +\tblock_iter_close(&it->cur);\n +\treftable_block_done(&it->block_reader.block);\n ++\treftable_free(it->offsets);\n +\tstrbuf_release(&it->oid);\n +}\n +\n @@ reftable/iter.c (new)\n +\n +static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\n +{\n -+\tstruct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p;\n -+\tstruct reftable_ref_record *ref =\n -+\t\t(struct reftable_ref_record *)rec->data;\n ++\tstruct indexed_table_ref_iter *it = p;\n ++\tstruct reftable_ref_record *ref = rec->data;\n +\n +\twhile (1) {\n +\t\tint err = block_iter_next(&it->cur, rec);\n @@ reftable/iter.c (new)\n +void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n +\t\t\t\t\t struct indexed_table_ref_iter *itr)\n +{\n -+\tassert(it->ops == NULL);\n ++\tassert(!it->ops);\n +\tit->iter_arg = itr;\n +\tit->ops = &indexed_table_ref_iter_vtable;\n +}\n @@ reftable/iter.h (new)\n +\n +void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n +\t\t\t\t\t struct indexed_table_ref_iter *itr);\n ++\n ++/* Takes ownership of `offsets` */\n +int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n +\t\t\t struct reftable_reader *r, uint8_t *oid,\n +\t\t\t int oid_len, uint64_t *offsets, int offset_len);\n @@ reftable/reader.c (new)\n +\n +void block_source_close(struct reftable_block_source *source)\n +{\n -+\tif (source->ops == NULL) {\n ++\tif (!source->ops) {\n +\t\treturn;\n +\t}\n +\n @@ reftable/reader.c (new)\n +\tf += 8;\n +\n +\tif (r->version == 1) {\n -+\t\tr->hash_id = SHA1_ID;\n ++\t\tr->hash_id = GIT_SHA1_HASH_ID;\n +\t} else {\n +\t\tr->hash_id = get_be32(f);\n +\t\tswitch (r->hash_id) {\n -+\t\tcase SHA1_ID:\n ++\t\tcase GIT_SHA1_HASH_ID:\n +\t\t\tbreak;\n -+\t\tcase SHA256_ID:\n ++\t\tcase GIT_SHA256_HASH_ID:\n +\t\t\tbreak;\n +\t\tdefault:\n +\t\t\terr = REFTABLE_FORMAT_ERROR;\n @@ reftable/reader.c (new)\n +\n +static void table_iter_block_done(struct table_iter *ti)\n +{\n -+\tif (ti->bi.br == NULL) {\n ++\tif (!ti->bi.br) {\n +\t\treturn;\n +\t}\n +\treftable_block_done(&ti->bi.br->block);\n @@ reftable/reader.c (new)\n +\n +static int table_iter_next_void(void *ti, struct reftable_record *rec)\n +{\n -+\treturn table_iter_next((struct table_iter *)ti, rec);\n ++\treturn table_iter_next(ti, rec);\n +}\n +\n +static void table_iter_close(void *p)\n +{\n -+\tstruct table_iter *ti = (struct table_iter *)p;\n ++\tstruct table_iter *ti = p;\n +\ttable_iter_block_done(ti);\n +\tblock_iter_close(&ti->bi);\n +}\n @@ reftable/reader.c (new)\n +static void iterator_from_table_iter(struct reftable_iterator *it,\n +\t\t\t\t struct table_iter *ti)\n +{\n -+\tassert(it->ops == NULL);\n ++\tassert(!it->ops);\n +\tit->iter_arg = ti;\n +\tit->ops = &table_iter_vtable;\n +}\n @@ reftable/reader.c (new)\n +static int reftable_reader_seek_void(void *tab, struct reftable_iterator *it,\n +\t\t\t\t struct reftable_record *rec)\n +{\n -+\treturn reader_seek((struct reftable_reader *)tab, it, rec);\n ++\treturn reader_seek(tab, it, rec);\n +}\n +\n +static uint32_t reftable_reader_hash_id_void(void *tab)\n +{\n -+\treturn reftable_reader_hash_id((struct reftable_reader *)tab);\n ++\treturn reftable_reader_hash_id(tab);\n +}\n +\n +static uint64_t reftable_reader_min_update_index_void(void *tab)\n +{\n -+\treturn reftable_reader_min_update_index((struct reftable_reader *)tab);\n ++\treturn reftable_reader_min_update_index(tab);\n +}\n +\n +static uint64_t reftable_reader_max_update_index_void(void *tab)\n +{\n -+\treturn reftable_reader_max_update_index((struct reftable_reader *)tab);\n ++\treturn reftable_reader_max_update_index(tab);\n +}\n +\n +static struct reftable_table_vtable reader_vtable = {\n @@ reftable/reader.c (new)\n +void reftable_table_from_reader(struct reftable_table *tab,\n +\t\t\t\tstruct reftable_reader *reader)\n +{\n -+\tassert(tab->ops == NULL);\n ++\tassert(!tab->ops);\n +\ttab->ops = &reader_vtable;\n +\ttab->table_arg = reader;\n ++}\n ++\n ++\n ++int reftable_reader_print_file(const char *tablename)\n ++{\n ++\tstruct reftable_block_source src = { NULL };\n ++\tint err = reftable_block_source_from_file(&src, tablename);\n ++\tstruct reftable_reader *r = NULL;\n ++\tstruct reftable_table tab = { NULL };\n ++\tif (err < 0)\n ++\t\tgoto done;\n ++\n ++\terr = reftable_new_reader(&r, &src, tablename);\n ++\tif (err < 0)\n ++\t\tgoto done;\n ++\n ++\treftable_table_from_reader(&tab, r);\n ++\terr = reftable_table_print(&tab);\n ++done:\n ++\treftable_reader_free(r);\n ++\treturn err;\n +}\n \n ## reftable/reader.h (new) ##\n @@ reftable/reftable-reader.h (new)\n +void reftable_table_from_reader(struct reftable_table *tab,\n +\t\t\t\tstruct reftable_reader *reader);\n +\n ++/* print table onto stdout for debugging. */\n ++int reftable_reader_print_file(const char *tablename);\n ++\n +#endif\n 12: 9ade9303f08f ! 17: dc04ec88cf53 reftable: reftable file level tests\n @@ Commit message\n Signed-off-by: Han-Wen Nienhuys \n \n ## Makefile ##\n -@@ Makefile: REFTABLE_OBJS += reftable/zlib-compat.o\n +@@ Makefile: REFTABLE_OBJS += reftable/writer.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n -+REFTABLE_TEST_OBJS += reftable/reftable_test.o\n ++REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \n \n - ## reftable/reftable_test.c (new) ##\n + ## reftable/readwrite_test.c (new) ##\n @@\n +/*\n +Copyright 2020 Google LLC\n @@ reftable/reftable_test.c (new)\n +\t*names = reftable_calloc(sizeof(char *) * (N + 1));\n +\treftable_writer_set_limits(w, update_index, update_index);\n +\tfor (i = 0; i < N; i++) {\n -+\t\tuint8_t hash[SHA256_SIZE] = { 0 };\n ++\t\tuint8_t hash[GIT_SHA256_RAWSZ] = { 0 };\n +\t\tchar name[100];\n +\t\tint n;\n +\n @@ reftable/reftable_test.c (new)\n +\t}\n +\n +\tfor (i = 0; i < N; i++) {\n -+\t\tuint8_t hash[SHA256_SIZE] = { 0 };\n ++\t\tuint8_t hash[GIT_SHA256_RAWSZ] = { 0 };\n +\t\tchar name[100];\n +\t\tint n;\n +\n @@ reftable/reftable_test.c (new)\n +\tfor (i = 0; i < stats->ref_stats.blocks; i++) {\n +\t\tint off = i * opts.block_size;\n +\t\tif (off == 0) {\n -+\t\t\toff = header_size((hash_id == SHA256_ID) ? 2 : 1);\n ++\t\t\toff = header_size((hash_id == GIT_SHA256_HASH_ID) ? 2 :\n ++\t\t\t\t\t\t\t\t\t\t 1);\n +\t\t}\n +\t\tEXPECT(buf->buf[off] == 'r');\n +\t}\n @@ reftable/reftable_test.c (new)\n +\t/* This tests buffer extension for log compression. Must use a random\n +\t hash, to ensure that the compressed part is larger than the original.\n +\t*/\n -+\tuint8_t hash1[SHA1_SIZE], hash2[SHA1_SIZE];\n -+\tfor (int i = 0; i < SHA1_SIZE; i++) {\n ++\tuint8_t hash1[GIT_SHA1_RAWSZ], hash2[GIT_SHA1_RAWSZ];\n ++\tfor (int i = 0; i < GIT_SHA1_RAWSZ; i++) {\n +\t\thash1[i] = (uint8_t)(rand() % 256);\n +\t\thash2[i] = (uint8_t)(rand() % 256);\n +\t}\n @@ reftable/reftable_test.c (new)\n +\t\tEXPECT_ERR(err);\n +\t}\n +\tfor (i = 0; i < N; i++) {\n -+\t\tuint8_t hash1[SHA1_SIZE], hash2[SHA1_SIZE];\n ++\t\tuint8_t hash1[GIT_SHA1_RAWSZ], hash2[GIT_SHA1_RAWSZ];\n +\t\tstruct reftable_log_record log = { NULL };\n +\t\tset_test_hash(hash1, i);\n +\t\tset_test_hash(hash2, i + 1);\n @@ reftable/reftable_test.c (new)\n +\tint err = 0;\n +\tint j = 0;\n +\n -+\twrite_table(&names, &buf, N, 256, SHA1_ID);\n ++\twrite_table(&names, &buf, N, 256, GIT_SHA1_HASH_ID);\n +\n +\tblock_source_from_strbuf(&source, &buf);\n +\n @@ reftable/reftable_test.c (new)\n +\tchar **names;\n +\tstruct strbuf buf = STRBUF_INIT;\n +\tint N = 1;\n -+\twrite_table(&names, &buf, N, 4096, SHA1_ID);\n ++\twrite_table(&names, &buf, N, 4096, GIT_SHA1_HASH_ID);\n +\tEXPECT(buf.len < 200);\n +\tstrbuf_release(&buf);\n +\tfree_names(names);\n @@ reftable/reftable_test.c (new)\n +\tstruct reftable_log_record log = { NULL };\n +\tstruct reftable_iterator it = { NULL };\n +\n -+\twrite_table(&names, &buf, N, 256, SHA1_ID);\n ++\twrite_table(&names, &buf, N, 256, GIT_SHA1_HASH_ID);\n +\n +\tblock_source_from_strbuf(&source, &buf);\n +\n @@ reftable/reftable_test.c (new)\n +\n +static void test_table_read_write_seek_linear(void)\n +{\n -+\ttest_table_read_write_seek(0, SHA1_ID);\n ++\ttest_table_read_write_seek(0, GIT_SHA1_HASH_ID);\n +}\n +\n +static void test_table_read_write_seek_linear_sha256(void)\n +{\n -+\ttest_table_read_write_seek(0, SHA256_ID);\n ++\ttest_table_read_write_seek(0, GIT_SHA256_HASH_ID);\n +}\n +\n +static void test_table_read_write_seek_index(void)\n +{\n -+\ttest_table_read_write_seek(1, SHA1_ID);\n ++\ttest_table_read_write_seek(1, GIT_SHA1_HASH_ID);\n +}\n +\n +static void test_table_refs_for(int indexed)\n @@ reftable/reftable_test.c (new)\n +\tint N = 50;\n +\tchar **want_names = reftable_calloc(sizeof(char *) * (N + 1));\n +\tint want_names_len = 0;\n -+\tuint8_t want_hash[SHA1_SIZE];\n ++\tuint8_t want_hash[GIT_SHA1_RAWSZ];\n +\n +\tstruct reftable_write_options opts = {\n +\t\t.block_size = 256,\n @@ reftable/reftable_test.c (new)\n +\tset_test_hash(want_hash, 4);\n +\n +\tfor (i = 0; i < N; i++) {\n -+\t\tuint8_t hash[SHA1_SIZE];\n ++\t\tuint8_t hash[GIT_SHA1_RAWSZ];\n +\t\tchar fill[51] = { 0 };\n +\t\tchar name[100];\n -+\t\tuint8_t hash1[SHA1_SIZE];\n -+\t\tuint8_t hash2[SHA1_SIZE];\n ++\t\tuint8_t hash1[GIT_SHA1_RAWSZ];\n ++\t\tuint8_t hash2[GIT_SHA1_RAWSZ];\n +\t\tstruct reftable_ref_record ref = { NULL };\n +\n +\t\tmemset(hash, i, sizeof(hash));\n @@ reftable/reftable_test.c (new)\n +\t\tn = reftable_writer_add_ref(w, &ref);\n +\t\tEXPECT(n == 0);\n +\n -+\t\tif (!memcmp(hash1, want_hash, SHA1_SIZE) ||\n -+\t\t !memcmp(hash2, want_hash, SHA1_SIZE)) {\n ++\t\tif (!memcmp(hash1, want_hash, GIT_SHA1_RAWSZ) ||\n ++\t\t !memcmp(hash2, want_hash, GIT_SHA1_RAWSZ)) {\n +\t\t\twant_names[want_names_len++] = xstrdup(name);\n +\t\t}\n +\t}\n @@ reftable/reftable_test.c (new)\n +\tstrbuf_release(&buf);\n +}\n +\n -+int reftable_test_main(int argc, const char *argv[])\n ++static void test_write_key_order(void)\n ++{\n ++\tstruct reftable_write_options opts = { 0 };\n ++\tstruct strbuf buf = STRBUF_INIT;\n ++\tstruct reftable_writer *w =\n ++\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n ++\tstruct reftable_ref_record refs[2] = {\n ++\t\t{\n ++\t\t\t.refname = \"b\",\n ++\t\t\t.update_index = 1,\n ++\t\t\t.value_type = REFTABLE_REF_SYMREF,\n ++\t\t\t.value = {\n ++\t\t\t\t.symref = \"target\",\n ++\t\t\t},\n ++\t\t}, {\n ++\t\t\t.refname = \"a\",\n ++\t\t\t.update_index = 1,\n ++\t\t\t.value_type = REFTABLE_REF_SYMREF,\n ++\t\t\t.value = {\n ++\t\t\t\t.symref = \"target\",\n ++\t\t\t},\n ++\t\t}\n ++\t};\n ++\tint err;\n ++\n ++\treftable_writer_set_limits(w, 1, 1);\n ++\terr = reftable_writer_add_ref(w, &refs[0]);\n ++\tEXPECT_ERR(err);\n ++\terr = reftable_writer_add_ref(w, &refs[1]);\n ++\tprintf(\"%d\\n\", err);\n ++\tEXPECT(err == REFTABLE_API_ERROR);\n ++\treftable_writer_close(w);\n ++\treftable_writer_free(w);\n ++\tstrbuf_release(&buf);\n ++}\n ++\n ++int readwrite_test_main(int argc, const char *argv[])\n +{\n +\tRUN_TEST(test_log_write_read);\n ++\tRUN_TEST(test_write_key_order);\n +\tRUN_TEST(test_table_read_write_seek_linear_sha256);\n +\tRUN_TEST(test_log_buffer_size);\n +\tRUN_TEST(test_table_write_small_table);\n @@ reftable/reftable_test.c (new)\n +\treturn 0;\n +}\n \n + ## reftable/reftable-tests.h ##\n +@@ reftable/reftable-tests.h: int block_test_main(int argc, const char **argv);\n + int merged_test_main(int argc, const char **argv);\n + int record_test_main(int argc, const char **argv);\n + int refname_test_main(int argc, const char **argv);\n +-int reftable_test_main(int argc, const char **argv);\n ++int readwrite_test_main(int argc, const char **argv);\n + int stack_test_main(int argc, const char **argv);\n + int tree_test_main(int argc, const char **argv);\n + int reftable_dump_main(int argc, char *const *argv);\n +\n ## t/helper/test-reftable.c ##\n @@ t/helper/test-reftable.c: int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n -+\treftable_test_main(argc, argv);\n ++\treadwrite_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n 13: 0a6119d910a3 ! 18: 84578b666a40 reftable: add a heap-based priority queue for reftable records\n @@ Makefile: REFTABLE_OBJS += reftable/block.o\n +REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n - REFTABLE_OBJS += reftable/reftable.o\n -@@ Makefile: REFTABLE_OBJS += reftable/zlib-compat.o\n + REFTABLE_OBJS += reftable/refname.o\n +@@ Makefile: REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n +REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n - REFTABLE_TEST_OBJS += reftable/reftable_test.o\n + REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n \n ## reftable/pq.c (new) ##\n @@ reftable/pq_test.c (new)\n +\n +\t\tmerged_iter_pqueue_check(pq);\n +\n -+\t\tif (last != NULL) {\n ++\t\tif (last) {\n +\t\t\tassert(strcmp(last, ref->refname) < 0);\n +\t\t}\n +\t\tlast = ref->refname;\n @@ reftable/reftable-tests.h: license that can be found in the LICENSE file or at\n +int pq_test_main(int argc, const char **argv);\n int record_test_main(int argc, const char **argv);\n int refname_test_main(int argc, const char **argv);\n - int reftable_test_main(int argc, const char **argv);\n + int readwrite_test_main(int argc, const char **argv);\n \n ## t/helper/test-reftable.c ##\n @@ t/helper/test-reftable.c: int cmd__reftable(int argc, const char **argv)\n @@ t/helper/test-reftable.c: int cmd__reftable(int argc, const char **argv)\n \tblock_test_main(argc, argv);\n +\tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n - \treftable_test_main(argc, argv);\n + \treadwrite_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n 14: 393fcecdae1e ! 19: e3a776f2076c reftable: add merged table view\n @@ Makefile: REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n -@@ Makefile: REFTABLE_OBJS += reftable/zlib-compat.o\n +@@ Makefile: REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n +REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n - REFTABLE_TEST_OBJS += reftable/reftable_test.o\n + REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n \n ## reftable/merged.c (new) ##\n @@\n @@ reftable/merged.c (new)\n +\n +static void merged_iter_close(void *p)\n +{\n -+\tstruct merged_iter *mi = (struct merged_iter *)p;\n ++\tstruct merged_iter *mi = p;\n +\tint i = 0;\n +\tmerged_iter_pqueue_release(&mi->pq);\n +\tfor (i = 0; i < mi->stack_len; i++) {\n @@ reftable/merged.c (new)\n +\n +static int merged_iter_next_void(void *p, struct reftable_record *rec)\n +{\n -+\tstruct merged_iter *mi = (struct merged_iter *)p;\n ++\tstruct merged_iter *mi = p;\n +\tif (merged_iter_pqueue_is_empty(mi->pq))\n +\t\treturn 1;\n +\n @@ reftable/merged.c (new)\n +static void iterator_from_merged_iter(struct reftable_iterator *it,\n +\t\t\t\t struct merged_iter *mi)\n +{\n -+\tassert(it->ops == NULL);\n ++\tassert(!it->ops);\n +\tit->iter_arg = mi;\n +\tit->ops = &merged_iter_vtable;\n +}\n @@ reftable/merged.c (new)\n +\t\t}\n +\t}\n +\n -+\tm = (struct reftable_merged_table *)reftable_calloc(\n -+\t\tsizeof(struct reftable_merged_table));\n ++\tm = reftable_calloc(sizeof(struct reftable_merged_table));\n +\tm->stack = stack;\n +\tm->stack_len = n;\n +\tm->min = first_min;\n @@ reftable/merged.c (new)\n +\n +void reftable_merged_table_free(struct reftable_merged_table *mt)\n +{\n -+\tif (mt == NULL) {\n ++\tif (!mt) {\n +\t\treturn;\n +\t}\n +\tmerged_table_release(mt);\n @@ reftable/merged.c (new)\n +\t\t\t\t\t struct reftable_iterator *it,\n +\t\t\t\t\t struct reftable_record *rec)\n +{\n -+\treturn merged_table_seek_record((struct reftable_merged_table *)tab, it,\n -+\t\t\t\t\trec);\n ++\treturn merged_table_seek_record(tab, it, rec);\n +}\n +\n +static uint32_t reftable_merged_table_hash_id_void(void *tab)\n +{\n -+\treturn reftable_merged_table_hash_id(\n -+\t\t(struct reftable_merged_table *)tab);\n ++\treturn reftable_merged_table_hash_id(tab);\n +}\n +\n +static uint64_t reftable_merged_table_min_update_index_void(void *tab)\n +{\n -+\treturn reftable_merged_table_min_update_index(\n -+\t\t(struct reftable_merged_table *)tab);\n ++\treturn reftable_merged_table_min_update_index(tab);\n +}\n +\n +static uint64_t reftable_merged_table_max_update_index_void(void *tab)\n +{\n -+\treturn reftable_merged_table_max_update_index(\n -+\t\t(struct reftable_merged_table *)tab);\n ++\treturn reftable_merged_table_max_update_index(tab);\n +}\n +\n +static struct reftable_table_vtable merged_table_vtable = {\n @@ reftable/merged.c (new)\n +void reftable_table_from_merged_table(struct reftable_table *tab,\n +\t\t\t\t struct reftable_merged_table *merged)\n +{\n -+\tassert(tab->ops == NULL);\n ++\tassert(!tab->ops);\n +\ttab->ops = &merged_table_vtable;\n +\ttab->table_arg = merged;\n +}\n @@ reftable/merged_test.c (new)\n +\t\treftable_table_from_reader(&tabs[i], (*readers)[i]);\n +\t}\n +\n -+\terr = reftable_new_merged_table(&mt, tabs, n, SHA1_ID);\n ++\terr = reftable_new_merged_table(&mt, tabs, n, GIT_SHA1_HASH_ID);\n +\tEXPECT_ERR(err);\n +\treturn mt;\n +}\n @@ reftable/merged_test.c (new)\n +\n +static void test_merged_between(void)\n +{\n -+\tuint8_t hash1[SHA1_SIZE] = { 1, 2, 3, 0 };\n ++\tuint8_t hash1[GIT_SHA1_RAWSZ] = { 1, 2, 3, 0 };\n +\n +\tstruct reftable_ref_record r1[] = { {\n +\t\t.refname = \"b\",\n @@ reftable/merged_test.c (new)\n +\n +static void test_merged(void)\n +{\n -+\tuint8_t hash1[SHA1_SIZE] = { 1 };\n -+\tuint8_t hash2[SHA1_SIZE] = { 2 };\n ++\tuint8_t hash1[GIT_SHA1_RAWSZ] = { 1 };\n ++\tuint8_t hash2[GIT_SHA1_RAWSZ] = { 2 };\n +\tstruct reftable_ref_record r1[] = {\n +\t\t{\n +\t\t\t.refname = \"a\",\n @@ reftable/merged_test.c (new)\n +\n +\tassert(ARRAY_SIZE(want) == len);\n +\tfor (i = 0; i < len; i++) {\n -+\t\tassert(reftable_ref_record_equal(&want[i], &out[i], SHA1_SIZE));\n ++\t\tassert(reftable_ref_record_equal(&want[i], &out[i],\n ++\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n +\t}\n +\tfor (i = 0; i < len; i++) {\n +\t\treftable_ref_record_release(&out[i]);\n @@ reftable/merged_test.c (new)\n +\tEXPECT_ERR(err);\n +\n +\thash_id = reftable_reader_hash_id(rd);\n -+\tassert(hash_id == SHA1_ID);\n ++\tassert(hash_id == GIT_SHA1_HASH_ID);\n +\n +\treftable_table_from_reader(&tab[0], rd);\n -+\terr = reftable_new_merged_table(&merged, tab, 1, SHA1_ID);\n ++\terr = reftable_new_merged_table(&merged, tab, 1, GIT_SHA1_HASH_ID);\n +\tEXPECT_ERR(err);\n +\n +\treftable_reader_free(rd);\n @@ t/helper/test-reftable.c: int cmd__reftable(int argc, const char **argv)\n +\tmerged_test_main(argc, argv);\n \tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n - \treftable_test_main(argc, argv);\n + \treadwrite_test_main(argc, argv);\n 15: 3186b2b70f73 ! 20: 7913027063a2 reftable: implement refname validation\n @@ Commit message\n Signed-off-by: Han-Wen Nienhuys \n \n ## Makefile ##\n -@@ Makefile: REFTABLE_OBJS += reftable/merged.o\n - REFTABLE_OBJS += reftable/pq.o\n - REFTABLE_OBJS += reftable/reader.o\n - REFTABLE_OBJS += reftable/record.o\n -+REFTABLE_OBJS += reftable/refname.o\n - REFTABLE_OBJS += reftable/reftable.o\n - REFTABLE_OBJS += reftable/tree.o\n - REFTABLE_OBJS += reftable/writer.o\n -@@ Makefile: REFTABLE_TEST_OBJS += reftable/block_test.o\n - REFTABLE_TEST_OBJS += reftable/merged_test.o\n +@@ Makefile: REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n + REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n +REFTABLE_TEST_OBJS += reftable/refname_test.o\n - REFTABLE_TEST_OBJS += reftable/reftable_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n + \n \n ## reftable/refname.c (new) ##\n @@\n @@ reftable/refname.c (new)\n +\n +static int find_name(size_t k, void *arg)\n +{\n -+\tstruct find_arg *f_arg = (struct find_arg *)arg;\n ++\tstruct find_arg *f_arg = arg;\n +\treturn strcmp(f_arg->names[k], f_arg->want) >= 0;\n +}\n +\n @@ reftable/refname_test.c (new)\n +\t\t\t.tab = tab,\n +\t\t};\n +\n -+\t\tif (cases[i].add != NULL) {\n ++\t\tif (cases[i].add) {\n +\t\t\tmod.add = &cases[i].add;\n +\t\t\tmod.add_len = 1;\n +\t\t}\n -+\t\tif (cases[i].del != NULL) {\n ++\t\tif (cases[i].del) {\n +\t\t\tmod.del = &cases[i].del;\n +\t\t\tmod.del_len = 1;\n +\t\t}\n @@ t/helper/test-reftable.c: int cmd__reftable(int argc, const char **argv)\n \tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n +\trefname_test_main(argc, argv);\n - \treftable_test_main(argc, argv);\n + \treadwrite_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n 16: b5492d5a13d7 ! 21: 3f1e5aedf4a8 reftable: implement stack, a mutable database of reftable files.\n @@ Commit message\n Signed-off-by: Han-Wen Nienhuys \n \n ## Makefile ##\n -@@ Makefile: REFTABLE_OBJS += reftable/reader.o\n - REFTABLE_OBJS += reftable/record.o\n - REFTABLE_OBJS += reftable/refname.o\n - REFTABLE_OBJS += reftable/reftable.o\n -+REFTABLE_OBJS += reftable/stack.o\n - REFTABLE_OBJS += reftable/tree.o\n - REFTABLE_OBJS += reftable/writer.o\n - REFTABLE_OBJS += reftable/zlib-compat.o\n @@ Makefile: REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n + REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n REFTABLE_TEST_OBJS += reftable/refname_test.o\n - REFTABLE_TEST_OBJS += reftable/reftable_test.o\n +REFTABLE_TEST_OBJS += reftable/stack_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n @@ reftable/reftable-stack.h (new)\n +\t\t\t\t\t void *arg),\n +\t\t\t void *arg);\n +\n -+/* Commits the transaction, releasing the lock. */\n ++/* Commits the transaction, releasing the lock. After calling this,\n ++ * reftable_addition_destroy should still be called.\n ++ */\n +int reftable_addition_commit(struct reftable_addition *add);\n +\n +/* Release all non-committed data from the transaction, and deallocate the\n @@ reftable/reftable-stack.h (new)\n +struct reftable_compaction_stats *\n +reftable_stack_compaction_stats(struct reftable_stack *st);\n +\n ++/* print the entire stack represented by the directory */\n ++int reftable_stack_print_directory(const char *stackdir);\n ++\n +#endif\n \n ## reftable/stack.c (new) ##\n @@ reftable/stack.c (new)\n +\tstrbuf_addstr(dest, name);\n +}\n +\n -+static int reftable_fd_write(void *arg, const void *data, size_t sz)\n ++static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz)\n +{\n +\tint *fdp = (int *)arg;\n +\treturn write(*fdp, data, sz);\n @@ reftable/stack.c (new)\n +\tint err = 0;\n +\n +\tif (config.hash_id == 0) {\n -+\t\tconfig.hash_id = SHA1_ID;\n ++\t\tconfig.hash_id = GIT_SHA1_HASH_ID;\n +\t}\n +\n +\t*dest = NULL;\n @@ reftable/stack.c (new)\n +{\n +\tchar **names = NULL;\n +\tint err = 0;\n -+\tif (st->merged != NULL) {\n ++\tif (st->merged) {\n +\t\treftable_merged_table_free(st->merged);\n +\t\tst->merged = NULL;\n +\t}\n @@ reftable/stack.c (new)\n +\t\tFREE_AND_NULL(names);\n +\t}\n +\n -+\tif (st->readers != NULL) {\n ++\tif (st->readers) {\n +\t\tint i = 0;\n +\t\tstruct strbuf filename = STRBUF_INIT;\n +\t\tfor (i = 0; i < st->readers_len; i++) {\n @@ reftable/stack.c (new)\n +static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n +\t\t\t\t int reuse_open)\n +{\n -+\tint cur_len = st->merged == NULL ? 0 : st->merged->stack_len;\n ++\tint cur_len = !st->merged ? 0 : st->merged->stack_len;\n +\tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n +\tint err = 0;\n +\tint names_len = names_length(names);\n @@ reftable/stack.c (new)\n +\t\t tables under control so this is not quadratic. */\n +\t\tint j = 0;\n +\t\tfor (j = 0; reuse_open && j < cur_len; j++) {\n -+\t\t\tif (cur[j] != NULL && 0 == strcmp(cur[j]->name, name)) {\n ++\t\t\tif (cur[j] && 0 == strcmp(cur[j]->name, name)) {\n +\t\t\t\trd = cur[j];\n +\t\t\t\tcur[j] = NULL;\n +\t\t\t\tbreak;\n +\t\t\t}\n +\t\t}\n +\n -+\t\tif (rd == NULL) {\n ++\t\tif (!rd) {\n +\t\t\tstruct reftable_block_source src = { NULL };\n +\t\t\tstruct strbuf table_path = STRBUF_INIT;\n +\t\t\tstack_filename(&table_path, st, name);\n @@ reftable/stack.c (new)\n +\n +\tnew_tables = NULL;\n +\tst->readers_len = new_readers_len;\n -+\tif (st->merged != NULL) {\n ++\tif (st->merged) {\n +\t\tmerged_table_release(st->merged);\n +\t\treftable_merged_table_free(st->merged);\n +\t}\n -+\tif (st->readers != NULL) {\n ++\tif (st->readers) {\n +\t\treftable_free(st->readers);\n +\t}\n +\tst->readers = new_readers;\n @@ reftable/stack.c (new)\n +\tnew_merged->suppress_deletions = 1;\n +\tst->merged = new_merged;\n +\tfor (i = 0; i < cur_len; i++) {\n -+\t\tif (cur[i] != NULL) {\n ++\t\tif (cur[i]) {\n +\t\t\tconst char *name = reader_name(cur[i]);\n +\t\t\tstruct strbuf filename = STRBUF_INIT;\n +\t\t\tstack_filename(&filename, st, name);\n @@ reftable/stack.c (new)\n +\t\treturn err;\n +\n +\tfor (i = 0; i < st->readers_len; i++) {\n -+\t\tif (names[i] == NULL) {\n ++\t\tif (!names[i]) {\n +\t\t\terr = 1;\n +\t\t\tgoto done;\n +\t\t}\n @@ reftable/stack.c (new)\n +\t\t}\n +\t}\n +\n -+\tif (names[st->merged->stack_len] != NULL) {\n ++\tif (names[st->merged->stack_len]) {\n +\t\terr = 1;\n +\t\tgoto done;\n +\t}\n @@ reftable/stack.c (new)\n +\n +void reftable_addition_destroy(struct reftable_addition *add)\n +{\n -+\tif (add == NULL) {\n ++\tif (!add) {\n +\t\treturn;\n +\t}\n +\treftable_addition_close(add);\n @@ reftable/stack.c (new)\n +\t\t\tcontinue;\n +\t\t}\n +\n -+\t\tif (config != NULL && config->min_update_index > 0 &&\n ++\t\tif (config && config->min_update_index > 0 &&\n +\t\t log.update_index < config->min_update_index) {\n +\t\t\tcontinue;\n +\t\t}\n +\n -+\t\tif (config != NULL && config->time > 0 &&\n ++\t\tif (config && config->time > 0 &&\n +\t\t log.update.time < config->time) {\n +\t\t\tcontinue;\n +\t\t}\n @@ reftable/stack.c (new)\n +\n +done:\n +\treftable_iterator_destroy(&it);\n -+\tif (mt != NULL) {\n ++\tif (mt) {\n +\t\tmerged_table_release(mt);\n +\t\treftable_merged_table_free(mt);\n +\t}\n @@ reftable/stack.c (new)\n +\tint j = 0;\n +\tint is_empty_table = 0;\n +\n -+\tif (first > last || (expiry == NULL && first == last)) {\n ++\tif (first > last || (!expiry && first == last)) {\n +\t\terr = 0;\n +\t\tgoto done;\n +\t}\n @@ reftable/stack.c (new)\n +{\n +\tuint64_t *sizes =\n +\t\treftable_calloc(sizeof(uint64_t) * st->merged->stack_len);\n -+\tint version = (st->config.hash_id == SHA1_ID) ? 1 : 2;\n ++\tint version = (st->config.hash_id == GIT_SHA1_HASH_ID) ? 1 : 2;\n +\tint overhead = header_size(version) - 1;\n +\tint i = 0;\n +\tfor (i = 0; i < st->merged->stack_len; i++) {\n @@ reftable/stack.c (new)\n +static int is_table_name(const char *s)\n +{\n +\tconst char *dot = strrchr(s, '.');\n -+\treturn dot != NULL && !strcmp(dot, \".ref\");\n ++\treturn dot && !strcmp(dot, \".ref\");\n +}\n +\n +static void remove_maybe_stale_table(struct reftable_stack *st, uint64_t max,\n @@ reftable/stack.c (new)\n +\t\treftable_stack_merged_table(st));\n +\tDIR *dir = opendir(st->reftable_dir);\n +\tstruct dirent *d = NULL;\n -+\tif (dir == NULL) {\n ++\tif (!dir) {\n +\t\treturn REFTABLE_IO_ERROR;\n +\t}\n +\n -+\twhile ((d = readdir(dir)) != NULL) {\n ++\twhile ((d = readdir(dir))) {\n +\t\tint i = 0;\n +\t\tint found = 0;\n +\t\tif (!is_table_name(d->d_name))\n @@ reftable/stack.c (new)\n +done:\n +\treftable_addition_destroy(add);\n +\treturn err;\n ++}\n ++\n ++int reftable_stack_print_directory(const char *stackdir)\n ++{\n ++\tstruct reftable_stack *stack = NULL;\n ++\tstruct reftable_write_options cfg = { 0 };\n ++\tstruct reftable_merged_table *merged = NULL;\n ++\tstruct reftable_table table = { NULL };\n ++\n ++\tint err = reftable_new_stack(&stack, stackdir, cfg);\n ++\tif (err < 0)\n ++\t\tgoto done;\n ++\n ++\tmerged = reftable_stack_merged_table(stack);\n ++\treftable_table_from_merged_table(&table, merged);\n ++\terr = reftable_table_print(&table);\n ++done:\n ++\treftable_stack_destroy(stack);\n ++\treturn err;\n +}\n \n ## reftable/stack.h (new) ##\n @@ reftable/stack_test.c (new)\n +\tif (dir == NULL)\n +\t\treturn 0;\n +\n -+\twhile ((d = readdir(dir)) != NULL) {\n ++\twhile ((d = readdir(dir))) {\n +\t\tif (!strcmp(d->d_name, \"..\") || !strcmp(d->d_name, \".\"))\n +\t\t\tcontinue;\n +\t\tlen++;\n @@ reftable/stack_test.c (new)\n +\treturn template;\n +}\n +\n ++static char *get_tmp_dir(const char *prefix)\n ++{\n ++\tchar *dir = get_tmp_template(prefix);\n ++\tEXPECT(mkdtemp(dir));\n ++\treturn dir;\n ++}\n ++\n +static void test_read_file(void)\n +{\n +\tchar *fn = get_tmp_template(__FUNCTION__);\n @@ reftable/stack_test.c (new)\n +\terr = read_lines(fn, &names);\n +\tEXPECT_ERR(err);\n +\n -+\tfor (i = 0; names[i] != NULL; i++) {\n ++\tfor (i = 0; names[i]; i++) {\n +\t\tEXPECT(0 == strcmp(want[i], names[i]));\n +\t}\n +\tfree_names(names);\n @@ reftable/stack_test.c (new)\n +\n +static void test_reftable_stack_add_one(void)\n +{\n -+\tchar *dir = get_tmp_template(__FUNCTION__);\n ++\tchar *dir = get_tmp_dir(__FUNCTION__);\n ++\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tint err;\n @@ reftable/stack_test.c (new)\n +\t};\n +\tstruct reftable_ref_record dest = { NULL };\n +\n -+\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st, dir, cfg);\n +\tEXPECT_ERR(err);\n @@ reftable/stack_test.c (new)\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st1 = NULL;\n +\tstruct reftable_stack *st2 = NULL;\n -+\tchar *dir = get_tmp_template(__FUNCTION__);\n ++\tchar *dir = get_tmp_dir(__FUNCTION__);\n ++\n +\tint err;\n +\tstruct reftable_ref_record ref1 = {\n +\t\t.refname = \"HEAD\",\n @@ reftable/stack_test.c (new)\n +\t\t.value.symref = \"master\",\n +\t};\n +\n -+\tEXPECT(mkdtemp(dir));\n +\n +\t/* simulate multi-process access to the same stack\n +\t by creating two stacks for the same directory.\n @@ reftable/stack_test.c (new)\n +\n +static void test_reftable_stack_transaction_api(void)\n +{\n -+\tchar *dir = get_tmp_template(__FUNCTION__);\n ++\tchar *dir = get_tmp_dir(__FUNCTION__);\n ++\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tint err;\n @@ reftable/stack_test.c (new)\n +\t};\n +\tstruct reftable_ref_record dest = { NULL };\n +\n -+\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st, dir, cfg);\n +\tEXPECT_ERR(err);\n @@ reftable/stack_test.c (new)\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tint err;\n -+\tchar *dir = get_tmp_template(__FUNCTION__);\n ++\tchar *dir = get_tmp_dir(__FUNCTION__);\n ++\n +\tint i;\n +\tstruct reftable_ref_record ref = {\n +\t\t.refname = \"a/b\",\n @@ reftable/stack_test.c (new)\n +\t};\n +\tchar *additions[] = { \"a\", \"a/b/c\" };\n +\n -+\tEXPECT(mkdtemp(dir));\n +\terr = reftable_new_stack(&st, dir, cfg);\n +\tEXPECT_ERR(err);\n +\n @@ reftable/stack_test.c (new)\n +\n +static void test_reftable_stack_update_index_check(void)\n +{\n -+\tchar *dir = get_tmp_template(__FUNCTION__);\n ++\tchar *dir = get_tmp_dir(__FUNCTION__);\n ++\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tint err;\n @@ reftable/stack_test.c (new)\n +\t\t.value_type = REFTABLE_REF_SYMREF,\n +\t\t.value.symref = \"master\",\n +\t};\n -+\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st, dir, cfg);\n +\tEXPECT_ERR(err);\n @@ reftable/stack_test.c (new)\n +\n +static void test_reftable_stack_lock_failure(void)\n +{\n -+\tchar *dir = get_tmp_template(__FUNCTION__);\n ++\tchar *dir = get_tmp_dir(__FUNCTION__);\n ++\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tint err, i;\n -+\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st, dir, cfg);\n +\tEXPECT_ERR(err);\n @@ reftable/stack_test.c (new)\n +\t\t.exact_log_message = 1,\n +\t};\n +\tstruct reftable_stack *st = NULL;\n -+\tchar *dir = get_tmp_template(__FUNCTION__);\n ++\tchar *dir = get_tmp_dir(__FUNCTION__);\n ++\n +\tstruct reftable_ref_record refs[2] = { { NULL } };\n +\tstruct reftable_log_record logs[2] = { { NULL } };\n +\tint N = ARRAY_SIZE(refs);\n +\n -+\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st, dir, cfg);\n +\tEXPECT_ERR(err);\n @@ reftable/stack_test.c (new)\n +\t\trefs[i].refname = xstrdup(buf);\n +\t\trefs[i].update_index = i + 1;\n +\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n -+\t\trefs[i].value.val1 = reftable_malloc(SHA1_SIZE);\n ++\t\trefs[i].value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);\n +\t\tset_test_hash(refs[i].value.val1, i);\n +\n +\t\tlogs[i].refname = xstrdup(buf);\n +\t\tlogs[i].update_index = N + i + 1;\n +\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n +\n -+\t\tlogs[i].update.new_hash = reftable_malloc(SHA1_SIZE);\n ++\t\tlogs[i].update.new_hash = reftable_malloc(GIT_SHA1_RAWSZ);\n +\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n +\t\tset_test_hash(logs[i].update.new_hash, i);\n +\t}\n @@ reftable/stack_test.c (new)\n +\n +\t\tint err = reftable_stack_read_ref(st, refs[i].refname, &dest);\n +\t\tEXPECT_ERR(err);\n -+\t\tEXPECT(reftable_ref_record_equal(&dest, refs + i, SHA1_SIZE));\n ++\t\tEXPECT(reftable_ref_record_equal(&dest, refs + i,\n ++\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n +\t\treftable_ref_record_release(&dest);\n +\t}\n +\n @@ reftable/stack_test.c (new)\n +\t\tstruct reftable_log_record dest = { NULL };\n +\t\tint err = reftable_stack_read_log(st, refs[i].refname, &dest);\n +\t\tEXPECT_ERR(err);\n -+\t\tEXPECT(reftable_log_record_equal(&dest, logs + i, SHA1_SIZE));\n ++\t\tEXPECT(reftable_log_record_equal(&dest, logs + i,\n ++\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n +\t\treftable_log_record_release(&dest);\n +\t}\n +\n @@ reftable/stack_test.c (new)\n +\t\t0,\n +\t};\n +\tstruct reftable_stack *st = NULL;\n -+\tchar *dir = get_tmp_template(__FUNCTION__);\n ++\tchar *dir = get_tmp_dir(__FUNCTION__);\n ++\n +\n -+\tuint8_t h1[SHA1_SIZE] = { 0x01 }, h2[SHA1_SIZE] = { 0x02 };\n ++\tuint8_t h1[GIT_SHA1_RAWSZ] = { 0x01 }, h2[GIT_SHA1_RAWSZ] = { 0x02 };\n +\n +\tstruct reftable_log_record input = { .refname = \"branch\",\n +\t\t\t\t\t .update_index = 1,\n @@ reftable/stack_test.c (new)\n +\t\t.update_index = 1,\n +\t};\n +\n -+\tEXPECT(mkdtemp(dir));\n +\terr = reftable_new_stack(&st, dir, cfg);\n +\tEXPECT_ERR(err);\n +\n @@ reftable/stack_test.c (new)\n +static void test_reftable_stack_tombstone(void)\n +{\n +\tint i = 0;\n -+\tchar *dir = get_tmp_template(__FUNCTION__);\n ++\tchar *dir = get_tmp_dir(__FUNCTION__);\n ++\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tint err;\n @@ reftable/stack_test.c (new)\n +\tstruct reftable_ref_record dest = { NULL };\n +\tstruct reftable_log_record log_dest = { NULL };\n +\n -+\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st, dir, cfg);\n +\tEXPECT_ERR(err);\n +\n ++\t/* even entries add the refs, odd entries delete them. */\n +\tfor (i = 0; i < N; i++) {\n +\t\tconst char *buf = \"branch\";\n +\t\trefs[i].refname = xstrdup(buf);\n +\t\trefs[i].update_index = i + 1;\n +\t\tif (i % 2 == 0) {\n +\t\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n -+\t\t\trefs[i].value.val1 = reftable_malloc(SHA1_SIZE);\n ++\t\t\trefs[i].value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);\n +\t\t\tset_test_hash(refs[i].value.val1, i);\n +\t\t}\n +\n @@ reftable/stack_test.c (new)\n +\t\tlogs[i].update_index = 42;\n +\t\tif (i % 2 == 0) {\n +\t\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n -+\t\t\tlogs[i].update.new_hash = reftable_malloc(SHA1_SIZE);\n ++\t\t\tlogs[i].update.new_hash =\n ++\t\t\t\treftable_malloc(GIT_SHA1_RAWSZ);\n +\t\t\tset_test_hash(logs[i].update.new_hash, i);\n +\t\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n +\t\t}\n @@ reftable/stack_test.c (new)\n +\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n +\t\tEXPECT_ERR(err);\n +\t}\n ++\n +\tfor (i = 0; i < N; i++) {\n +\t\tstruct write_log_arg arg = {\n +\t\t\t.log = &logs[i],\n @@ reftable/stack_test.c (new)\n +\n +static void test_reftable_stack_hash_id(void)\n +{\n -+\tchar *dir = get_tmp_template(__FUNCTION__);\n ++\tchar *dir = get_tmp_dir(__FUNCTION__);\n ++\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tint err;\n @@ reftable/stack_test.c (new)\n +\t\t.value.symref = \"target\",\n +\t\t.update_index = 1,\n +\t};\n -+\tstruct reftable_write_options cfg32 = { .hash_id = SHA256_ID };\n ++\tstruct reftable_write_options cfg32 = { .hash_id = GIT_SHA256_HASH_ID };\n +\tstruct reftable_stack *st32 = NULL;\n +\tstruct reftable_write_options cfg_default = { 0 };\n +\tstruct reftable_stack *st_default = NULL;\n +\tstruct reftable_ref_record dest = { NULL };\n +\n -+\tEXPECT(mkdtemp(dir));\n +\terr = reftable_new_stack(&st, dir, cfg);\n +\tEXPECT_ERR(err);\n +\n @@ reftable/stack_test.c (new)\n +\terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n +\tEXPECT_ERR(err);\n +\n -+\tEXPECT(reftable_ref_record_equal(&ref, &dest, SHA1_SIZE));\n ++\tEXPECT(reftable_ref_record_equal(&ref, &dest, GIT_SHA1_RAWSZ));\n +\treftable_ref_record_release(&dest);\n +\treftable_stack_destroy(st);\n +\treftable_stack_destroy(st_default);\n @@ reftable/stack_test.c (new)\n +\n +static void test_reflog_expire(void)\n +{\n -+\tchar *dir = get_tmp_template(__FUNCTION__);\n ++\tchar *dir = get_tmp_dir(__FUNCTION__);\n ++\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tstruct reftable_log_record logs[20] = { { NULL } };\n @@ reftable/stack_test.c (new)\n +\t};\n +\tstruct reftable_log_record log = { NULL };\n +\n -+\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st, dir, cfg);\n +\tEXPECT_ERR(err);\n @@ reftable/stack_test.c (new)\n +\t\tlogs[i].update_index = i;\n +\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n +\t\tlogs[i].update.time = i;\n -+\t\tlogs[i].update.new_hash = reftable_malloc(SHA1_SIZE);\n ++\t\tlogs[i].update.new_hash = reftable_malloc(GIT_SHA1_RAWSZ);\n +\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n +\t\tset_test_hash(logs[i].update.new_hash, i);\n +\t}\n @@ reftable/stack_test.c (new)\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n +\tint err;\n -+\tchar *dir = get_tmp_template(__FUNCTION__);\n ++\tchar *dir = get_tmp_dir(__FUNCTION__);\n ++\n +\tstruct reftable_stack *st2 = NULL;\n +\n -+\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st, dir, cfg);\n +\tEXPECT_ERR(err);\n @@ reftable/stack_test.c (new)\n +{\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st = NULL;\n -+\tchar *dir = get_tmp_template(__FUNCTION__);\n ++\tchar *dir = get_tmp_dir(__FUNCTION__);\n ++\n +\tint err, i;\n +\tint N = 100;\n -+\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st, dir, cfg);\n +\tEXPECT_ERR(err);\n @@ reftable/stack_test.c (new)\n +{\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st1 = NULL, *st2 = NULL;\n -+\tchar *dir = get_tmp_template(__FUNCTION__);\n ++\tchar *dir = get_tmp_dir(__FUNCTION__);\n ++\n +\tint err, i;\n +\tint N = 3;\n -+\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st1, dir, cfg);\n +\tEXPECT_ERR(err);\n @@ reftable/stack_test.c (new)\n +{\n +\tstruct reftable_write_options cfg = { 0 };\n +\tstruct reftable_stack *st1 = NULL, *st2 = NULL, *st3 = NULL;\n -+\tchar *dir = get_tmp_template(__FUNCTION__);\n ++\tchar *dir = get_tmp_dir(__FUNCTION__);\n ++\n +\tint err, i;\n +\tint N = 3;\n -+\tEXPECT(mkdtemp(dir));\n +\n +\terr = reftable_new_stack(&st1, dir, cfg);\n +\tEXPECT_ERR(err);\n @@ reftable/stack_test.c (new)\n +\n +int stack_test_main(int argc, const char *argv[])\n +{\n -+\tRUN_TEST(test_reftable_stack_compaction_concurrent_clean);\n ++\tRUN_TEST(test_empty_add);\n ++\tRUN_TEST(test_log2);\n ++\tRUN_TEST(test_names_equal);\n ++\tRUN_TEST(test_parse_names);\n ++\tRUN_TEST(test_read_file);\n ++\tRUN_TEST(test_reflog_expire);\n ++\tRUN_TEST(test_reftable_stack_add);\n ++\tRUN_TEST(test_reftable_stack_add_one);\n ++\tRUN_TEST(test_reftable_stack_auto_compaction);\n +\tRUN_TEST(test_reftable_stack_compaction_concurrent);\n -+\tRUN_TEST(test_reftable_stack_uptodate);\n -+\tRUN_TEST(test_reftable_stack_transaction_api);\n ++\tRUN_TEST(test_reftable_stack_compaction_concurrent_clean);\n +\tRUN_TEST(test_reftable_stack_hash_id);\n -+\tRUN_TEST(test_sizes_to_segments_all_equal);\n -+\tRUN_TEST(test_reftable_stack_auto_compaction);\n -+\tRUN_TEST(test_reftable_stack_validate_refname);\n -+\tRUN_TEST(test_reftable_stack_update_index_check);\n +\tRUN_TEST(test_reftable_stack_lock_failure);\n +\tRUN_TEST(test_reftable_stack_log_normalize);\n +\tRUN_TEST(test_reftable_stack_tombstone);\n -+\tRUN_TEST(test_reftable_stack_add_one);\n -+\tRUN_TEST(test_empty_add);\n -+\tRUN_TEST(test_reflog_expire);\n -+\tRUN_TEST(test_suggest_compaction_segment);\n -+\tRUN_TEST(test_suggest_compaction_segment_nothing);\n ++\tRUN_TEST(test_reftable_stack_transaction_api);\n ++\tRUN_TEST(test_reftable_stack_update_index_check);\n ++\tRUN_TEST(test_reftable_stack_uptodate);\n ++\tRUN_TEST(test_reftable_stack_validate_refname);\n +\tRUN_TEST(test_sizes_to_segments);\n ++\tRUN_TEST(test_sizes_to_segments_all_equal);\n +\tRUN_TEST(test_sizes_to_segments_empty);\n -+\tRUN_TEST(test_log2);\n -+\tRUN_TEST(test_parse_names);\n -+\tRUN_TEST(test_read_file);\n -+\tRUN_TEST(test_names_equal);\n -+\tRUN_TEST(test_reftable_stack_add);\n ++\tRUN_TEST(test_suggest_compaction_segment);\n ++\tRUN_TEST(test_suggest_compaction_segment_nothing);\n +\treturn 0;\n +}\n \n @@ t/helper/test-reftable.c\n @@ t/helper/test-reftable.c: int cmd__reftable(int argc, const char **argv)\n \trecord_test_main(argc, argv);\n \trefname_test_main(argc, argv);\n - \treftable_test_main(argc, argv);\n + \treadwrite_test_main(argc, argv);\n +\tstack_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n 17: 5db7c8ab7f23 ! 22: b2ac95a25af8 reftable: add dump utility\n @@ reftable/dump.c (new)\n +#include \n +#include \n +\n -+#include \"reftable.h\"\n ++#include \"reftable-blocksource.h\"\n ++#include \"reftable-error.h\"\n ++#include \"reftable-merged.h\"\n ++#include \"reftable-record.h\"\n +#include \"reftable-tests.h\"\n -+\n -+static uint32_t hash_id;\n -+\n -+static int dump_table(const char *tablename)\n -+{\n -+\tstruct reftable_block_source src = { 0 };\n -+\tint err = reftable_block_source_from_file(&src, tablename);\n -+\tstruct reftable_iterator it = { 0 };\n -+\tstruct reftable_ref_record ref = { 0 };\n -+\tstruct reftable_log_record log = { 0 };\n -+\tstruct reftable_reader *r = NULL;\n -+\n -+\tif (err < 0)\n -+\t\treturn err;\n -+\n -+\terr = reftable_new_reader(&r, &src, tablename);\n -+\tif (err < 0)\n -+\t\treturn err;\n -+\n -+\terr = reftable_reader_seek_ref(r, &it, \"\");\n -+\tif (err < 0) {\n -+\t\treturn err;\n -+\t}\n -+\n -+\twhile (1) {\n -+\t\terr = reftable_iterator_next_ref(&it, &ref);\n -+\t\tif (err > 0) {\n -+\t\t\tbreak;\n -+\t\t}\n -+\t\tif (err < 0) {\n -+\t\t\treturn err;\n -+\t\t}\n -+\t\treftable_ref_record_print(&ref, hash_id);\n -+\t}\n -+\treftable_iterator_destroy(&it);\n -+\treftable_ref_record_clear(&ref);\n -+\n -+\terr = reftable_reader_seek_log(r, &it, \"\");\n -+\tif (err < 0) {\n -+\t\treturn err;\n -+\t}\n -+\twhile (1) {\n -+\t\terr = reftable_iterator_next_log(&it, &log);\n -+\t\tif (err > 0) {\n -+\t\t\tbreak;\n -+\t\t}\n -+\t\tif (err < 0) {\n -+\t\t\treturn err;\n -+\t\t}\n -+\t\treftable_log_record_print(&log, hash_id);\n -+\t}\n -+\treftable_iterator_destroy(&it);\n -+\treftable_log_record_clear(&log);\n -+\n -+\treftable_reader_free(r);\n -+\treturn 0;\n -+}\n ++#include \"reftable-writer.h\"\n ++#include \"reftable-iterator.h\"\n ++#include \"reftable-reader.h\"\n ++#include \"reftable-stack.h\"\n ++#include \"reftable-generic.h\"\n +\n +static int compact_stack(const char *stackdir)\n +{\n +\tstruct reftable_stack *stack = NULL;\n -+\tstruct reftable_write_options cfg = {};\n ++\tstruct reftable_write_options cfg = { 0 };\n +\n +\tint err = reftable_new_stack(&stack, stackdir, cfg);\n +\tif (err < 0)\n @@ reftable/dump.c (new)\n +\tif (err < 0)\n +\t\tgoto done;\n +done:\n -+\tif (stack != NULL) {\n ++\tif (stack) {\n +\t\treftable_stack_destroy(stack);\n +\t}\n +\treturn err;\n +}\n +\n -+static int dump_stack(const char *stackdir)\n -+{\n -+\tstruct reftable_stack *stack = NULL;\n -+\tstruct reftable_write_options cfg = {};\n -+\tstruct reftable_iterator it = { 0 };\n -+\tstruct reftable_ref_record ref = { 0 };\n -+\tstruct reftable_log_record log = { 0 };\n -+\tstruct reftable_merged_table *merged = NULL;\n -+\n -+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n -+\tif (err < 0)\n -+\t\treturn err;\n -+\n -+\tmerged = reftable_stack_merged_table(stack);\n -+\n -+\terr = reftable_merged_table_seek_ref(merged, &it, \"\");\n -+\tif (err < 0) {\n -+\t\treturn err;\n -+\t}\n -+\n -+\twhile (1) {\n -+\t\terr = reftable_iterator_next_ref(&it, &ref);\n -+\t\tif (err > 0) {\n -+\t\t\tbreak;\n -+\t\t}\n -+\t\tif (err < 0) {\n -+\t\t\treturn err;\n -+\t\t}\n -+\t\treftable_ref_record_print(&ref, hash_id);\n -+\t}\n -+\treftable_iterator_destroy(&it);\n -+\treftable_ref_record_clear(&ref);\n -+\n -+\terr = reftable_merged_table_seek_log(merged, &it, \"\");\n -+\tif (err < 0) {\n -+\t\treturn err;\n -+\t}\n -+\twhile (1) {\n -+\t\terr = reftable_iterator_next_log(&it, &log);\n -+\t\tif (err > 0) {\n -+\t\t\tbreak;\n -+\t\t}\n -+\t\tif (err < 0) {\n -+\t\t\treturn err;\n -+\t\t}\n -+\t\treftable_log_record_print(&log, hash_id);\n -+\t}\n -+\treftable_iterator_destroy(&it);\n -+\treftable_log_record_clear(&log);\n -+\n -+\treftable_stack_destroy(stack);\n -+\treturn 0;\n -+}\n -+\n +static void print_help(void)\n +{\n +\tprintf(\"usage: dump [-cst] arg\\n\\n\"\n @@ reftable/dump.c (new)\n +\t \" -t dump table\\n\"\n +\t \" -s dump stack\\n\"\n +\t \" -h this help\\n\"\n -+\t \" -2 use SHA256\\n\"\n +\t \"\\n\");\n +}\n +\n +int reftable_dump_main(int argc, char *const *argv)\n +{\n +\tint err = 0;\n -+\tint opt;\n +\tint opt_dump_table = 0;\n +\tint opt_dump_stack = 0;\n +\tint opt_compact = 0;\n -+\tconst char *arg = NULL;\n -+\twhile ((opt = getopt(argc, argv, \"2chts\")) != -1) {\n -+\t\tswitch (opt) {\n -+\t\tcase '2':\n -+\t\t\thash_id = 0x73323536;\n ++\tconst char *arg = NULL, *argv0 = argv[0];\n ++\n ++\tfor (; argc > 1; argv++, argc--)\n ++\t\tif (*argv[1] != '-')\n +\t\t\tbreak;\n -+\t\tcase 't':\n ++\t\telse if (!strcmp(\"-t\", argv[1]))\n +\t\t\topt_dump_table = 1;\n -+\t\t\tbreak;\n -+\t\tcase 's':\n ++\t\telse if (!strcmp(\"-s\", argv[1]))\n +\t\t\topt_dump_stack = 1;\n -+\t\t\tbreak;\n -+\t\tcase 'c':\n ++\t\telse if (!strcmp(\"-c\", argv[1]))\n +\t\t\topt_compact = 1;\n -+\t\t\tbreak;\n -+\t\tcase '?':\n -+\t\tcase 'h':\n ++\t\telse if (!strcmp(\"-?\", argv[1]) || !strcmp(\"-h\", argv[1])) {\n +\t\t\tprint_help();\n +\t\t\treturn 2;\n -+\t\t\tbreak;\n +\t\t}\n -+\t}\n +\n -+\tif (argv[optind] == NULL) {\n ++\tif (argc != 2) {\n +\t\tfprintf(stderr, \"need argument\\n\");\n +\t\tprint_help();\n +\t\treturn 2;\n +\t}\n +\n -+\targ = argv[optind];\n ++\targ = argv[1];\n +\n +\tif (opt_dump_table) {\n -+\t\terr = dump_table(arg);\n ++\t\terr = reftable_reader_print_file(arg);\n +\t} else if (opt_dump_stack) {\n -+\t\terr = dump_stack(arg);\n ++\t\terr = reftable_stack_print_directory(arg);\n +\t} else if (opt_compact) {\n +\t\terr = compact_stack(arg);\n +\t}\n +\n +\tif (err < 0) {\n -+\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv[0], arg,\n ++\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv0, arg,\n +\t\t\treftable_error_str(err));\n +\t\treturn 1;\n +\t}\n 18: 2dc73bf2ec96 ! 23: 2fd7cb8c0983 Reftable support for git-core\n @@ Commit message\n Signed-off-by: Johannes Schindelin \n Helped-by: Johannes Schindelin \n Helped-by: Junio Hamano \n + Helped-by: Patrick Steinhardt \n Co-authored-by: Jeff King \n \n ## Documentation/config/extensions.txt ##\n @@ builtin/init-db.c: int cmd_init_db(int argc, const char **argv, const char *pref\n +\t\t initial_branch, ref_storage_format, flags);\n }\n \n + ## builtin/stash.c ##\n +@@ builtin/stash.c: static int get_stash_info(struct stash_info *info, int argc, const char **argv)\n + static int do_clear_stash(void)\n + {\n + \tstruct object_id obj;\n ++\tint result;\n + \tif (get_oid(ref_stash, &obj))\n + \t\treturn 0;\n + \n +-\treturn delete_ref(NULL, ref_stash, &obj, 0);\n ++\tresult = delete_ref(NULL, ref_stash, &obj, 0);\n ++\n ++\t/* Ignore error; this is necessary for reftable, which keeps reflogs\n ++\t * even when refs are deleted. */\n ++\tdelete_reflog(ref_stash);\n ++\treturn result;\n + }\n + \n + static int clear_stash(int argc, const char **argv, const char *prefix)\n +\n ## builtin/worktree.c ##\n @@\n #include \"utf8.h\"\n @@ refs.c\n \n +const char *default_ref_storage(void)\n +{\n -+\tconst char *test = getenv(\"GIT_TEST_REFTABLE\");\n -+\treturn test ? \"reftable\" : \"files\";\n ++\treturn git_env_bool(\"GIT_TEST_REFTABLE\", 0) ? \"reftable\" : \"files\";\n +}\n +\n /*\n @@ refs/reftable-backend.c (new)\n +static struct reftable_stack *stack_for(struct git_reftable_ref_store *store,\n +\t\t\t\t\tconst char *refname)\n +{\n -+\tif (store->worktree_stack == NULL)\n ++\tif (store->worktree_stack == NULL || refname == NULL)\n +\t\treturn store->main_stack;\n +\n +\tswitch (ref_type(refname)) {\n @@ refs/reftable-backend.c (new)\n +\t\tri->base.flags = 0;\n +\t\tswitch (ri->ref.value_type) {\n +\t\tcase REFTABLE_REF_VAL1:\n -+\t\t\thashcpy(ri->oid.hash, ri->ref.value.val1);\n ++\t\t\toidread(&ri->oid, ri->ref.value.val1);\n +\t\t\tbreak;\n +\t\tcase REFTABLE_REF_VAL2:\n -+\t\t\thashcpy(ri->oid.hash, ri->ref.value.val2.value);\n ++\t\t\toidread(&ri->oid, ri->ref.value.val2.value);\n +\t\t\tbreak;\n +\t\tcase REFTABLE_REF_SYMREF: {\n +\t\t\tint out_flags = 0;\n @@ refs/reftable-backend.c (new)\n +\tstruct git_reftable_iterator *ri =\n +\t\t(struct git_reftable_iterator *)ref_iterator;\n +\tif (ri->ref.value_type == REFTABLE_REF_VAL2) {\n -+\t\thashcpy(peeled->hash, ri->ref.value.val2.target_value);\n ++\t\toidread(peeled, ri->ref.value.val2.target_value);\n +\t\treturn 0;\n +\t}\n +\n -+\treturn -1;\n ++\treturn 1;\n +}\n +\n +static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n @@ refs/reftable-backend.c (new)\n +\t\t(struct git_reftable_ref_store *)ref_store;\n +\tstruct reftable_addition *add = NULL;\n +\tstruct reftable_stack *stack =\n -+\t\ttransaction->nr ?\n -+\t\t\t stack_for(refs, transaction->updates[0]->refname) :\n -+\t\t\t refs->main_stack;\n ++\t\tstack_for(refs,\n ++\t\t\t transaction->nr ? transaction->updates[0]->refname : NULL);\n ++\n +\tint err = refs->err;\n +\tif (err < 0) {\n +\t\tgoto done;\n @@ refs/reftable-backend.c (new)\n +\t\tcalloc(transaction->nr, sizeof(*logs));\n +\tstruct ref_update **sorted =\n +\t\tmalloc(transaction->nr * sizeof(struct ref_update *));\n ++\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n ++\tstruct reftable_table tab = {NULL};\n ++\tstruct reftable_ref_record ref = {NULL};\n ++\treftable_table_from_merged_table(&tab, mt);\n +\tCOPY_ARRAY(sorted, transaction->updates, transaction->nr);\n +\tQSORT(sorted, transaction->nr, ref_update_cmp);\n +\treftable_writer_set_limits(writer, ts, ts);\n @@ refs/reftable-backend.c (new)\n +\tfor (i = 0; i < transaction->nr; i++) {\n +\t\tstruct ref_update *u = sorted[i];\n +\t\tstruct reftable_log_record *log = &logs[i];\n ++\t\tstruct object_id old_id;\n +\t\tfill_reftable_log_record(log);\n +\t\tlog->update_index = ts;\n +\t\tlog->value_type = REFTABLE_LOG_UPDATE;\n +\t\tlog->refname = (char *)u->refname;\n -+\t\tlog->update.old_hash = u->old_oid.hash;\n +\t\tlog->update.new_hash = u->new_oid.hash;\n +\t\tlog->update.message = u->msg;\n +\n ++\t\terr = reftable_table_read_ref(&tab, u->refname, &ref);\n ++\t\tif (err < 0)\n ++\t\t\tgoto done;\n ++\t\telse if (err > 0) {\n ++\t\t\told_id = null_oid;\n ++\t\t} else {\n ++\t\t\toidread(&old_id, ref.value.val1);\n ++\t\t}\n ++\n ++\t\t/* XXX fold together with the old_id check below? */\n ++\n ++\t\tlog->update.old_hash = old_id.hash;\n +\t\tif (u->flags & REF_LOG_ONLY) {\n +\t\t\tcontinue;\n +\t\t}\n @@ refs/reftable-backend.c (new)\n +\n +done:\n +\tassert(err != REFTABLE_API_ERROR);\n ++\treftable_ref_record_release(&ref);\n +\tfree(logs);\n +\tfree(sorted);\n +\treturn err;\n @@ refs/reftable-backend.c (new)\n +\tstruct git_reftable_ref_store *refs =\n +\t\t(struct git_reftable_ref_store *)ref_store;\n +\tstruct reftable_stack *stack =\n -+\t\tstack_for(refs, refnames->items[0].string);\n ++\t\tstack_for(refs, refnames->nr ? refnames->items[0].string : NULL);\n +\tstruct write_delete_refs_arg arg = {\n +\t\t.stack = stack,\n +\t\t.refnames = refnames,\n @@ refs/reftable-backend.c (new)\n +{\n +\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n +\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n -+\tstruct reftable_ref_record ref = { NULL };\n -+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &ref);\n ++\tstruct reftable_ref_record old_ref = { NULL };\n ++\tstruct reftable_ref_record new_ref = { NULL };\n ++\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref);\n +\n +\tif (err) {\n +\t\tgoto done;\n +\t}\n +\n -+\t/* XXX do ref renames overwrite the target? */\n -+\tif (reftable_stack_read_ref(arg->stack, arg->newname, &ref) == 0) {\n ++\t/* git-branch supports a --force, but the check is not atomic. */\n ++\tif (reftable_stack_read_ref(arg->stack, arg->newname, &new_ref) == 0) {\n +\t\tgoto done;\n +\t}\n +\n @@ refs/reftable-backend.c (new)\n +\t\t\t\t.update_index = ts,\n +\t\t\t\t.value_type = REFTABLE_REF_DELETION,\n +\t\t\t},\n -+\t\t\tref,\n ++\t\t\told_ref,\n +\t\t};\n +\t\ttodo[1].update_index = ts;\n +\t\tfree(todo[1].refname);\n @@ refs/reftable-backend.c (new)\n +\t\t}\n +\t}\n +\n -+\tif (reftable_ref_record_val1(&ref)) {\n -+\t\tuint8_t *val1 = reftable_ref_record_val1(&ref);\n ++\tif (reftable_ref_record_val1(&old_ref)) {\n ++\t\tuint8_t *val1 = reftable_ref_record_val1(&old_ref);\n +\t\tstruct reftable_log_record todo[2] = { { NULL } };\n +\t\tfill_reftable_log_record(&todo[0]);\n +\t\tfill_reftable_log_record(&todo[1]);\n @@ refs/reftable-backend.c (new)\n +\n +done:\n +\tassert(err != REFTABLE_API_ERROR);\n -+\treftable_ref_record_release(&ref);\n ++\treftable_ref_record_release(&new_ref);\n ++\treftable_ref_record_release(&old_ref);\n ++\treturn err;\n ++}\n ++\n ++static int write_copy_table(struct reftable_writer *writer, void *argv)\n ++{\n ++\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n ++\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n ++\tstruct reftable_ref_record old_ref = { NULL };\n ++\tstruct reftable_ref_record new_ref = { NULL };\n ++\tstruct reftable_log_record log = { NULL };\n ++\tstruct reftable_iterator it = { NULL };\n ++\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref);\n ++\tif (err) {\n ++\t\tgoto done;\n ++\t}\n ++\n ++\t/* git-branch supports a --force, but the check is not atomic. */\n ++\tif (reftable_stack_read_ref(arg->stack, arg->newname, &new_ref) == 0) {\n ++\t\tgoto done;\n ++\t}\n ++\n ++\treftable_writer_set_limits(writer, ts, ts);\n ++\n ++\tFREE_AND_NULL(old_ref.refname);\n ++\told_ref.refname = xstrdup(arg->newname);\n ++\told_ref.update_index = ts;\n ++\terr = reftable_writer_add_ref(writer, &old_ref);\n ++\tif (err < 0) {\n ++\t\tgoto done;\n ++\t}\n ++\n ++\t/* this copies the entire reflog history. Is this the right semantics? */\n ++\t/* XXX should clear out existing reflog entries for oldname? */\n ++\terr = reftable_merged_table_seek_log(reftable_stack_merged_table(arg->stack), &it, arg->oldname);\n ++\tif (err < 0) {\n ++\t\tgoto done;\n ++\t}\n ++\twhile (1) {\n ++\t\tint err = reftable_iterator_next_log(&it, &log);\n ++\t\tif (err < 0) {\n ++\t\t\tgoto done;\n ++\t\t}\n ++\n ++\t\tif (err > 0 || strcmp(log.refname, arg->oldname)) {\n ++\t\t\tbreak;\n ++\t\t}\n ++\t\tFREE_AND_NULL(log.refname);\n ++\t\tlog.refname = xstrdup(arg->newname);\n ++\t\treftable_writer_add_log(writer, &log);\n ++\t\treftable_log_record_release(&log);\n ++\t}\n ++\n ++done:\n ++\tassert(err != REFTABLE_API_ERROR);\n ++\treftable_ref_record_release(&new_ref);\n ++\treftable_ref_record_release(&old_ref);\n ++\treftable_log_record_release(&log);\n ++\treftable_iterator_destroy(&it);\n +\treturn err;\n +}\n +\n @@ refs/reftable-backend.c (new)\n +\t\t\t\t const char *oldrefname, const char *newrefname,\n +\t\t\t\t const char *logmsg)\n +{\n -+\tBUG(\"reftable reference store does not support copying references\");\n ++\tstruct git_reftable_ref_store *refs =\n ++\t\t(struct git_reftable_ref_store *)ref_store;\n ++\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n ++\tstruct write_rename_arg arg = {\n ++\t\t.stack = stack,\n ++\t\t.oldname = oldrefname,\n ++\t\t.newname = newrefname,\n ++\t\t.logmsg = logmsg,\n ++\t};\n ++\tint err = refs->err;\n ++\tif (err < 0) {\n ++\t\tgoto done;\n ++\t}\n ++\terr = reftable_stack_reload(stack);\n ++\tif (err) {\n ++\t\tgoto done;\n ++\t}\n ++\n ++\terr = reftable_stack_add(stack, &write_copy_table, &arg);\n ++done:\n ++\tassert(err != REFTABLE_API_ERROR);\n ++\treturn err;\n +}\n +\n +struct git_reftable_reflog_ref_iterator {\n @@ refs/reftable-backend.c (new)\n +\n +\t\tfree(ri->last_name);\n +\t\tri->last_name = xstrdup(ri->log.refname);\n -+\t\thashcpy(ri->oid.hash, ri->log.update.new_hash);\n ++\t\toidread(&ri->oid, ri->log.update.new_hash);\n +\t\treturn ITER_OK;\n +\t}\n +}\n @@ refs/reftable-backend.c (new)\n +\t\t\tbreak;\n +\t\t}\n +\n -+\t\thashcpy(old_oid.hash, log.update.old_hash);\n -+\t\thashcpy(new_oid.hash, log.update.new_hash);\n ++\t\toidread(&old_oid, log.update.old_hash);\n ++\t\toidread(&new_oid, log.update.new_hash);\n +\n +\t\tfull_committer = fmt_ident(log.update.name, log.update.email,\n +\t\t\t\t\t WANT_COMMITTER_IDENT,\n @@ refs/reftable-backend.c (new)\n +\t\tstruct object_id new_oid;\n +\t\tconst char *full_committer = \"\";\n +\n -+\t\thashcpy(old_oid.hash, log->update.old_hash);\n -+\t\thashcpy(new_oid.hash, log->update.new_hash);\n ++\t\toidread(&old_oid, log->update.old_hash);\n ++\t\toidread(&new_oid, log->update.new_hash);\n +\n +\t\tfull_committer = fmt_ident(log->update.name, log->update.email,\n +\t\t\t\t\t WANT_COMMITTER_IDENT, NULL,\n @@ refs/reftable-backend.c (new)\n +\treturn 0;\n +}\n +\n -+static int git_reftable_delete_reflog(struct ref_store *ref_store,\n -+\t\t\t\t const char *refname)\n -+{\n -+\treturn 0;\n -+}\n -+\n -+struct reflog_expiry_arg {\n -+\tstruct git_reftable_ref_store *refs;\n ++struct write_reflog_delete_arg {\n +\tstruct reftable_stack *stack;\n -+\tstruct reftable_log_record *tombstones;\n -+\tint len;\n -+\tint cap;\n ++\tconst char *refname;\n +};\n +\n -+static void clear_log_tombstones(struct reflog_expiry_arg *arg)\n ++static int write_reflog_delete_table(struct reftable_writer *writer, void *argv)\n +{\n -+\tint i = 0;\n -+\tfor (; i < arg->len; i++) {\n -+\t\treftable_log_record_release(&arg->tombstones[i]);\n ++\tstruct write_reflog_delete_arg *arg = argv;\n ++\tstruct reftable_merged_table *mt = reftable_stack_merged_table(arg->stack);\n ++\tstruct reftable_log_record log = { NULL };\n ++\tstruct reftable_iterator it = { NULL };\n ++\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n ++\tint err = reftable_merged_table_seek_log(mt, &it, arg->refname);\n ++\n ++\treftable_writer_set_limits(writer, ts, ts);\n ++\twhile (err == 0) {\n ++\t\tstruct reftable_log_record tombstone = {\n ++\t\t\t.refname = (char*)arg->refname,\n ++\t\t\t.update_index = REFTABLE_LOG_DELETION,\n ++\t\t};\n ++\t\terr = reftable_iterator_next_log(&it, &log);\n ++\t\tif (err > 0) {\n ++\t\t\terr = 0;\n ++\t\t\tbreak;\n ++\t\t}\n ++\n ++\t\tif (err < 0 || strcmp(log.refname, arg->refname)) {\n ++\t\t\tbreak;\n ++\t\t}\n ++\t\ttombstone.update_index = log.update_index;\n ++\t\terr = reftable_writer_add_log(writer, &tombstone);\n +\t}\n +\n -+\tFREE_AND_NULL(arg->tombstones);\n ++\treturn err;\n +}\n +\n -+static void add_log_tombstone(struct reflog_expiry_arg *arg,\n -+\t\t\t const char *refname, uint64_t ts)\n ++\n ++static int git_reftable_delete_reflog(struct ref_store *ref_store,\n ++\t\t\t\t const char *refname)\n +{\n -+\tstruct reftable_log_record tombstone = {\n -+\t\t.refname = xstrdup(refname),\n -+\t\t.update_index = ts,\n ++\tstruct git_reftable_ref_store *refs =\n ++\t\t(struct git_reftable_ref_store *)ref_store;\n ++\tstruct reftable_stack *stack = stack_for(refs, refname);\n ++\tstruct write_reflog_delete_arg arg = {\n ++\t\t.stack = stack,\n ++\t\t.refname = refname,\n +\t};\n -+\tif (arg->len == arg->cap) {\n -+\t\targ->cap = 2 * arg->cap + 1;\n -+\t\targ->tombstones =\n -+\t\t\trealloc(arg->tombstones, arg->cap * sizeof(tombstone));\n -+\t}\n -+\targ->tombstones[arg->len++] = tombstone;\n ++\tint err = reftable_stack_add(stack, &write_reflog_delete_table, &arg);\n ++\tassert(err != REFTABLE_API_ERROR);\n ++\treturn err;\n +}\n +\n ++struct reflog_expiry_arg {\n ++\tstruct reftable_stack *stack;\n ++\tstruct reftable_log_record *records;\n ++\tint len;\n ++};\n ++\n +static int write_reflog_expiry_table(struct reftable_writer *writer, void *argv)\n +{\n +\tstruct reflog_expiry_arg *arg = (struct reflog_expiry_arg *)argv;\n @@ refs/reftable-backend.c (new)\n +\tint i = 0;\n +\treftable_writer_set_limits(writer, ts, ts);\n +\tfor (i = 0; i < arg->len; i++) {\n -+\t\tint err = reftable_writer_add_log(writer, &arg->tombstones[i]);\n ++\t\tint err = reftable_writer_add_log(writer, &arg->records[i]);\n +\t\tif (err) {\n +\t\t\treturn err;\n +\t\t}\n @@ refs/reftable-backend.c (new)\n +\tstruct reftable_merged_table *mt = NULL;\n +\tstruct reflog_expiry_arg arg = {\n +\t\t.stack = stack,\n -+\t\t.refs = refs,\n +\t};\n -+\tstruct reftable_log_record log = { NULL };\n ++\tstruct reftable_log_record *logs = NULL;\n ++\tstruct reftable_log_record *rewritten = NULL;\n ++\tint logs_len = 0;\n ++\tint logs_cap = 0;\n ++\tint i = 0;\n ++\tuint8_t *last_hash = NULL;\n +\tstruct reftable_iterator it = { NULL };\n ++\tstruct reftable_addition *add = NULL;\n +\tint err = 0;\n +\tif (refs->err < 0) {\n +\t\treturn refs->err;\n @@ refs/reftable-backend.c (new)\n +\t\tgoto done;\n +\t}\n +\n ++\terr = reftable_stack_new_addition(&add, stack);\n ++\tif (err) {\n ++\t\tgoto done;\n ++\t}\n ++\tprepare_fn(refname, oid, policy_cb_data);\n +\twhile (1) {\n -+\t\tstruct object_id ooid;\n -+\t\tstruct object_id noid;\n -+\n ++\t\tstruct reftable_log_record log = {NULL};\n +\t\tint err = reftable_iterator_next_log(&it, &log);\n +\t\tif (err < 0) {\n +\t\t\tgoto done;\n @@ refs/reftable-backend.c (new)\n +\t\tif (err > 0 || strcmp(log.refname, refname)) {\n +\t\t\tbreak;\n +\t\t}\n -+\t\thashcpy(ooid.hash, log.update.old_hash);\n -+\t\thashcpy(noid.hash, log.update.new_hash);\n +\n -+\t\tif (should_prune_fn(&ooid, &noid, log.update.email,\n -+\t\t\t\t (timestamp_t)log.update.time,\n -+\t\t\t\t log.update.tz_offset, log.update.message,\n ++\t\tif (logs_len >= logs_cap) {\n ++\t\t\tint new_cap = logs_cap * 2 + 1;\n ++\t\t\tlogs = realloc(logs, new_cap * sizeof(*logs));\n ++\t\t\tlogs_cap = new_cap;\n ++\t\t}\n ++\t\tlogs[logs_len++] = log;\n ++\t}\n ++\n ++\trewritten = calloc(logs_len, sizeof(*rewritten));\n ++\tfor (i = logs_len-1; i >= 0; i--) {\n ++\t\tstruct object_id ooid;\n ++\t\tstruct object_id noid;\n ++\t\tstruct reftable_log_record *dest = &rewritten[i];\n ++\n ++\t\t*dest = logs[i];\n ++\t\toidread(&ooid, logs[i].update.old_hash);\n ++\t\toidread(&noid, logs[i].update.new_hash);\n ++\n ++\t\tif (should_prune_fn(&ooid, &noid, logs[i].update.email,\n ++\t\t\t\t (timestamp_t)logs[i].update.time,\n ++\t\t\t\t logs[i].update.tz_offset, logs[i].update.message,\n +\t\t\t\t policy_cb_data)) {\n -+\t\t\tadd_log_tombstone(&arg, refname, log.update_index);\n ++\t\t\tdest->value_type = REFTABLE_LOG_DELETION;\n ++\t\t} else {\n ++\t\t\tif ((flags & EXPIRE_REFLOGS_REWRITE) && last_hash != NULL) {\n ++\t\t\t\tdest->update.old_hash = last_hash;\n ++\t\t\t}\n ++\t\t\tlast_hash = logs[i].update.new_hash;\n +\t\t}\n +\t}\n -+\terr = reftable_stack_add(stack, &write_reflog_expiry_table, &arg);\n ++\n ++\targ.records = rewritten;\n ++\targ.len = logs_len;\n ++\terr = reftable_addition_add(add, &write_reflog_expiry_table, &arg);\n ++\tif (err < 0) {\n ++\t\tgoto done;\n ++\t}\n ++\n ++\tif (!(flags & EXPIRE_REFLOGS_DRY_RUN)) {\n ++\t\t/* XXX - skip writing records that were not changed. */\n ++\t\terr = reftable_addition_commit(add);\n ++\t} else {\n ++\t\t/* XXX - print something */\n ++\t}\n +\n +done:\n ++\tif (add) {\n ++\t\tcleanup_fn(policy_cb_data);\n ++\t}\n +\tassert(err != REFTABLE_API_ERROR);\n -+\treftable_log_record_release(&log);\n ++\treftable_addition_destroy(add);\n ++\tfor (i = 0; i < logs_len; i++)\n ++\t\treftable_log_record_release(&logs[i]);\n ++\tfree(logs);\n ++\tfree(rewritten);\n +\treftable_iterator_destroy(&it);\n -+\tclear_log_tombstones(&arg);\n +\treturn err;\n +}\n +\n @@ refs/reftable-backend.c (new)\n +\t\tstrbuf_addstr(referent, ref.value.symref);\n +\t\t*type |= REF_ISSYMREF;\n +\t} else if (reftable_ref_record_val1(&ref) != NULL) {\n -+\t\thashcpy(oid->hash, reftable_ref_record_val1(&ref));\n ++\t\toidread(oid, reftable_ref_record_val1(&ref));\n +\t} else {\n +\t\t*type |= REF_ISBROKEN;\n +\t\terrno = EINVAL;\n @@ setup.c: const char *setup_git_directory_gently(int *nongit_ok)\n +\t\t\t\txstrdup_or_null(repo_fmt.ref_storage);\n +\t\t}\n \t}\n - \n - \tstrbuf_release(&dir);\n + \t/*\n + \t * Since precompose_string_if_needed() needs to look at\n \n ## t/t0031-reftable.sh (new) ##\n @@\n @@ t/t0031-reftable.sh (new)\n +\tmv .git/hooks .git/hooks-disabled\n +}\n +\n ++write_script fake_editor <<\\EOF\n ++echo \"$MSG\" >\"$1\"\n ++echo \"$MSG\" >&2\n ++EOF\n ++GIT_EDITOR=./fake_editor\n ++export GIT_EDITOR\n ++\n ++test_expect_success 'read existing old OID if REF_HAVE_OLD is not set' '\n ++\tinitialize &&\n ++\ttest_commit 1st &&\n ++\ttest_commit 2nd &&\n ++\tMSG=b4 git notes add &&\n ++\tMSG=b3 git notes edit &&\n ++\techo b4 >expect &&\n ++\tgit notes --ref commits@{1} show >actual &&\n ++\ttest_cmp expect actual\n ++'\n ++\n ++test_expect_success 'git reflog delete' '\n ++\tinitialize &&\n ++\ttest_commit file &&\n ++\ttest_commit file2 &&\n ++\ttest_commit file3 &&\n ++\ttest_commit file4 &&\n ++\tgit reflog delete HEAD@{1} &&\n ++\tgit reflog > output &&\n ++\t! grep file3 output\n ++'\n ++\n ++test_expect_success 'branch -D delete nonexistent branch' '\n ++\tinitialize &&\n ++\ttest_commit file &&\n ++\ttest_must_fail git branch -D ../../my-private-file\n ++'\n ++\n ++test_expect_success 'branch copy' '\n ++\tinitialize &&\n ++\ttest_commit file1 &&\n ++\ttest_commit file2 &&\n ++\tgit branch src &&\n ++\tgit reflog src > expect &&\n ++\tgit branch -c src dst &&\n ++\tgit reflog dst | sed \"s/dst/src/g\" > actual &&\n ++\ttest_cmp expect actual\n ++'\n ++\n ++test_expect_success 'git stash' '\n ++\tinitialize &&\n ++\ttest_commit file &&\n ++\ttouch actual expected &&\n ++\tgit -c status.showStash=true status >expected &&\n ++\techo hoi >> file.t &&\n ++\tgit stash push -m stashed &&\n ++\tgit stash clear &&\n ++\tgit -c status.showStash=true status >actual &&\n ++\ttest_cmp expected actual\n ++'\n ++\n ++test_expect_success 'rename branch' '\n ++\tinitialize &&\n ++\tgit symbolic-ref HEAD refs/heads/before &&\n ++\ttest_commit file &&\n ++\tgit show-ref | sed s/before/after/g > expected &&\n ++\tgit branch -M after &&\n ++\tgit show-ref > actual &&\n ++\ttest_cmp expected actual\n ++'\n ++\n +test_expect_success 'SHA256 support, env' '\n +\trm -rf .git &&\n +\tGIT_DEFAULT_HASH=sha256 && export GIT_DEFAULT_HASH &&\n 19: b2fa6ea62c16 = 24: 742d716e4510 git-prompt: prepare for reftable refs backend\n 20: 39e644b9436b ! 25: aca82763e72c Add \"test-tool dump-reftable\" command.\n @@ Commit message\n Signed-off-by: Han-Wen Nienhuys \n \n ## Makefile ##\n -@@ Makefile: REFTABLE_OBJS += reftable/zlib-compat.o\n +@@ Makefile: REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n @@ Makefile: REFTABLE_OBJS += reftable/zlib-compat.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n \n - ## reftable/dump.c ##\n -@@ reftable/dump.c: license that can be found in the LICENSE file or at\n - #include \n - #include \n - \n --#include \"reftable.h\"\n -+#include \"reftable-blocksource.h\"\n -+#include \"reftable-error.h\"\n -+#include \"reftable-merged.h\"\n -+#include \"reftable-record.h\"\n - #include \"reftable-tests.h\"\n -+#include \"reftable-writer.h\"\n -+#include \"reftable-iterator.h\"\n -+#include \"reftable-reader.h\"\n -+#include \"reftable-stack.h\"\n - \n - static uint32_t hash_id;\n - \n - static int dump_table(const char *tablename)\n - {\n --\tstruct reftable_block_source src = { 0 };\n -+\tstruct reftable_block_source src = { NULL };\n - \tint err = reftable_block_source_from_file(&src, tablename);\n --\tstruct reftable_iterator it = { 0 };\n --\tstruct reftable_ref_record ref = { 0 };\n --\tstruct reftable_log_record log = { 0 };\n -+\tstruct reftable_iterator it = { NULL };\n -+\tstruct reftable_ref_record ref = { NULL };\n -+\tstruct reftable_log_record log = { NULL };\n - \tstruct reftable_reader *r = NULL;\n - \n - \tif (err < 0)\n -@@ reftable/dump.c: static int dump_table(const char *tablename)\n - \t\treftable_ref_record_print(&ref, hash_id);\n - \t}\n - \treftable_iterator_destroy(&it);\n --\treftable_ref_record_clear(&ref);\n -+\treftable_ref_record_release(&ref);\n - \n - \terr = reftable_reader_seek_log(r, &it, \"\");\n - \tif (err < 0) {\n -@@ reftable/dump.c: static int dump_table(const char *tablename)\n - \t\treftable_log_record_print(&log, hash_id);\n - \t}\n - \treftable_iterator_destroy(&it);\n --\treftable_log_record_clear(&log);\n -+\treftable_log_record_release(&log);\n - \n - \treftable_reader_free(r);\n - \treturn 0;\n -@@ reftable/dump.c: static int dump_table(const char *tablename)\n - static int compact_stack(const char *stackdir)\n - {\n - \tstruct reftable_stack *stack = NULL;\n --\tstruct reftable_write_options cfg = {};\n -+\tstruct reftable_write_options cfg = { 0 };\n - \n - \tint err = reftable_new_stack(&stack, stackdir, cfg);\n - \tif (err < 0)\n -@@ reftable/dump.c: static int compact_stack(const char *stackdir)\n - static int dump_stack(const char *stackdir)\n - {\n - \tstruct reftable_stack *stack = NULL;\n --\tstruct reftable_write_options cfg = {};\n --\tstruct reftable_iterator it = { 0 };\n --\tstruct reftable_ref_record ref = { 0 };\n --\tstruct reftable_log_record log = { 0 };\n -+\tstruct reftable_write_options cfg = { 0 };\n -+\tstruct reftable_iterator it = { NULL };\n -+\tstruct reftable_ref_record ref = { NULL };\n -+\tstruct reftable_log_record log = { NULL };\n - \tstruct reftable_merged_table *merged = NULL;\n - \n - \tint err = reftable_new_stack(&stack, stackdir, cfg);\n -@@ reftable/dump.c: static int dump_stack(const char *stackdir)\n - \t\treftable_ref_record_print(&ref, hash_id);\n - \t}\n - \treftable_iterator_destroy(&it);\n --\treftable_ref_record_clear(&ref);\n -+\treftable_ref_record_release(&ref);\n - \n - \terr = reftable_merged_table_seek_log(merged, &it, \"\");\n - \tif (err < 0) {\n -@@ reftable/dump.c: static int dump_stack(const char *stackdir)\n - \t\treftable_log_record_print(&log, hash_id);\n - \t}\n - \treftable_iterator_destroy(&it);\n --\treftable_log_record_clear(&log);\n -+\treftable_log_record_release(&log);\n - \n - \treftable_stack_destroy(stack);\n - \treturn 0;\n -@@ reftable/dump.c: static void print_help(void)\n - int reftable_dump_main(int argc, char *const *argv)\n - {\n - \tint err = 0;\n --\tint opt;\n - \tint opt_dump_table = 0;\n - \tint opt_dump_stack = 0;\n - \tint opt_compact = 0;\n --\tconst char *arg = NULL;\n --\twhile ((opt = getopt(argc, argv, \"2chts\")) != -1) {\n --\t\tswitch (opt) {\n --\t\tcase '2':\n --\t\t\thash_id = 0x73323536;\n -+\tconst char *arg = NULL, *argv0 = argv[0];\n -+\n -+\tfor (; argc > 1; argv++, argc--)\n -+\t\tif (*argv[1] != '-')\n - \t\t\tbreak;\n --\t\tcase 't':\n -+\t\telse if (!strcmp(\"-2\", argv[1]))\n -+\t\t\thash_id = 0x73323536;\n -+\t\telse if (!strcmp(\"-t\", argv[1]))\n - \t\t\topt_dump_table = 1;\n --\t\t\tbreak;\n --\t\tcase 's':\n -+\t\telse if (!strcmp(\"-s\", argv[1]))\n - \t\t\topt_dump_stack = 1;\n --\t\t\tbreak;\n --\t\tcase 'c':\n -+\t\telse if (!strcmp(\"-c\", argv[1]))\n - \t\t\topt_compact = 1;\n --\t\t\tbreak;\n --\t\tcase '?':\n --\t\tcase 'h':\n -+\t\telse if (!strcmp(\"-?\", argv[1]) || !strcmp(\"-h\", argv[1])) {\n - \t\t\tprint_help();\n - \t\t\treturn 2;\n --\t\t\tbreak;\n - \t\t}\n --\t}\n - \n --\tif (argv[optind] == NULL) {\n -+\tif (argc != 2) {\n - \t\tfprintf(stderr, \"need argument\\n\");\n - \t\tprint_help();\n - \t\treturn 2;\n - \t}\n - \n --\targ = argv[optind];\n -+\targ = argv[1];\n - \n - \tif (opt_dump_table) {\n - \t\terr = dump_table(arg);\n -@@ reftable/dump.c: int reftable_dump_main(int argc, char *const *argv)\n - \t}\n - \n - \tif (err < 0) {\n --\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv[0], arg,\n -+\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv0, arg,\n - \t\t\treftable_error_str(err));\n - \t\treturn 1;\n - \t}\n -\n ## t/helper/test-reftable.c ##\n @@ t/helper/test-reftable.c: int cmd__reftable(int argc, const char **argv)\n \ttree_test_main(argc, argv);\n -: ------------ > 26: ae88f41de9a9 t1301: document what needs to be done for REFTABLE\n -: ------------ > 27: 0c1eb69900a7 t1401,t2011: parameterize HEAD.lock for REFTABLE\n -: ------------ > 28: dd8ffbf53157 t1404: annotate test cases with REFFILES\n\n-- \ngitgitgadget\n"},{"id":"422271","messageId":"f0216ae20b6988514bdb60d8fff96e18c2ce9f1d.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 04/28] hash.h: provide constants for the hash IDs","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:32Z","receivedAt":"2021-04-19T11:38:31Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis will simplify referencing them from code that is not deeply integrated with\nGit, in particular, the reftable library.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n hash.h | 6 ++++++\n object-file.c | 6 ++----\n 2 files changed, 8 insertions(+), 4 deletions(-)\n\ndiff --git a/hash.h b/hash.h\nindex 3fb0c3d4005a..b17fb2927711 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -161,12 +161,18 @@ static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n \treturn p - hash_algos;\n }\n \n+/* \"sha1\", big-endian */\n+#define GIT_SHA1_HASH_ID 0x73686131\n+\n /* The length in bytes and in hex digits of an object name (SHA-1 value). */\n #define GIT_SHA1_RAWSZ 20\n #define GIT_SHA1_HEXSZ (2 * GIT_SHA1_RAWSZ)\n /* The block size of SHA-1. */\n #define GIT_SHA1_BLKSZ 64\n \n+/* \"s256\", big-endian */\n+#define GIT_SHA256_HASH_ID 0x73323536\n+\n /* The length in bytes and in hex digits of an object name (SHA-256 value). */\n #define GIT_SHA256_RAWSZ 32\n #define GIT_SHA256_HEXSZ (2 * GIT_SHA256_RAWSZ)\ndiff --git a/object-file.c b/object-file.c\nindex 624af408cdcd..5af384a8cfc4 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -146,8 +146,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t},\n \t{\n \t\t\"sha1\",\n-\t\t/* \"sha1\", big-endian */\n-\t\t0x73686131,\n+\t\tGIT_SHA1_HASH_ID,\n \t\tGIT_SHA1_RAWSZ,\n \t\tGIT_SHA1_HEXSZ,\n \t\tGIT_SHA1_BLKSZ,\n@@ -160,8 +159,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t},\n \t{\n \t\t\"sha256\",\n-\t\t/* \"s256\", big-endian */\n-\t\t0x73323536,\n+\t\tGIT_SHA256_HASH_ID,\n \t\tGIT_SHA256_RAWSZ,\n \t\tGIT_SHA256_HEXSZ,\n \t\tGIT_SHA256_BLKSZ,\n-- \ngitgitgadget\n\n"},{"id":"422272","messageId":"fa64aa9ce7c107e9349c5c2e58b276eea053f2ac.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 05/28] init-db: set the_repository->hash_algo early on","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:33Z","receivedAt":"2021-04-19T11:38:45Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable backend needs to know the hash algorithm for writing the\ninitialization hash table.\n\nThe initial reftable contains a symref HEAD => \"main\" (or \"master\"), which is\nagnostic to the size of hash value, but this is an exceptional circumstance, and\nthe reftable library does not cater to this exception. It insists that all\ntables in the stack have a consistent format ID for the hash algorithm.\n\nCall set_repo_hash_algo directly after calling validate_hash_algorithm() (which\nreads $GIT_DEFAULT_HASH).\n\nHelped-by: Junio C Hamano \nSigned-off-by: Han-Wen Nienhuys \n---\n builtin/init-db.c | 21 +++++++++++++++++++++\n 1 file changed, 21 insertions(+)\n\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex c19b35f1e691..ff06fe4b9a49 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -424,6 +424,27 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \n \tvalidate_hash_algorithm(&repo_fmt, hash);\n \n+\t/*\n+\t * At this point, the_repository we have in-core does not look\n+\t * anything like one that we would see initialized in an already\n+\t * working repository after calling setup_git_directory().\n+\t *\n+\t * Calling repository.c::initialize_the_repository() may have\n+\t * prepared the .index .objects and .parsed_objects members, but\n+\t * other members like .gitdir, .commondir, etc. have not been\n+\t * initialized.\n+\t *\n+\t * Many API functions assume they are working with the_repository\n+\t * that has sensibly been initialized, but because we haven't\n+\t * really read from an existing repository, we need to hand-craft\n+\t * the necessary members of the structure to get out of this\n+\t * chicken-and-egg situation.\n+\t *\n+\t * For now, we update the hash algorithm member to what the\n+\t * validate_hash_algorithm() call decided for us.\n+\t */\n+\trepo_set_hash_algo(the_repository, repo_fmt.hash_algo);\n+\n \treinit = create_default_files(template_dir, original_git_dir,\n \t\t\t\t initial_branch, &repo_fmt,\n \t\t\t\t flags & INIT_DB_QUIET);\n-- \ngitgitgadget\n\n"},{"id":"422273","messageId":"8a960fb77fa9117009119ce37aaa9885749bd1a5.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 06/28] reftable: add LICENSE","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:34Z","receivedAt":"2021-04-19T11:38:54Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nTODO: relicense?\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/LICENSE | 31 +++++++++++++++++++++++++++++++\n 1 file changed, 31 insertions(+)\n create mode 100644 reftable/LICENSE\n\ndiff --git a/reftable/LICENSE b/reftable/LICENSE\nnew file mode 100644\nindex 000000000000..402e0f9356ba\n--- /dev/null\n+++ b/reftable/LICENSE\n@@ -0,0 +1,31 @@\n+BSD License\n+\n+Copyright (c) 2020, Google LLC\n+All rights reserved.\n+\n+Redistribution and use in source and binary forms, with or without\n+modification, are permitted provided that the following conditions are\n+met:\n+\n+* Redistributions of source code must retain the above copyright notice,\n+this list of conditions and the following disclaimer.\n+\n+* Redistributions in binary form must reproduce the above copyright\n+notice, this list of conditions and the following disclaimer in the\n+documentation and/or other materials provided with the distribution.\n+\n+* Neither the name of Google LLC nor the names of its contributors may\n+be used to endorse or promote products derived from this software\n+without specific prior written permission.\n+\n+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS\n+\"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT\n+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT\n+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT\n+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,\n+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY\n+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n-- \ngitgitgadget\n\n"},{"id":"422274","messageId":"a24e9e661616e659ed1377a723934df918c8926d.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 08/28] reftable: utility functions","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:36Z","receivedAt":"2021-04-19T11:38:54Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis commit provides basic utility classes for the reftable library.\n\nSigned-off-by: Han-Wen Nienhuys \nHelped-by: Johannes Schindelin \n---\n Makefile | 25 +++++-\n contrib/buildsystems/CMakeLists.txt | 14 ++-\n reftable/basics.c | 128 ++++++++++++++++++++++++++++\n reftable/basics.h | 60 +++++++++++++\n reftable/basics_test.c | 98 +++++++++++++++++++++\n reftable/publicbasics.c | 58 +++++++++++++\n reftable/reftable-malloc.h | 18 ++++\n reftable/reftable-tests.h | 22 +++++\n reftable/system.h | 24 ++++++\n reftable/test_framework.c | 23 +++++\n reftable/test_framework.h | 53 ++++++++++++\n t/helper/test-reftable.c | 9 ++\n t/helper/test-tool.c | 3 +-\n t/helper/test-tool.h | 1 +\n t/t0032-reftable-unittest.sh | 15 ++++\n 15 files changed, 545 insertions(+), 6 deletions(-)\n create mode 100644 reftable/basics.c\n create mode 100644 reftable/basics.h\n create mode 100644 reftable/basics_test.c\n create mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-malloc.h\n create mode 100644 reftable/reftable-tests.h\n create mode 100644 reftable/system.h\n create mode 100644 reftable/test_framework.c\n create mode 100644 reftable/test_framework.h\n create mode 100644 t/helper/test-reftable.c\n create mode 100755 t/t0032-reftable-unittest.sh\n\ndiff --git a/Makefile b/Makefile\nindex 21c0bf16672b..6f6143672333 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -736,6 +736,7 @@ TEST_BUILTINS_OBJS += test-read-cache.o\n TEST_BUILTINS_OBJS += test-read-graph.o\n TEST_BUILTINS_OBJS += test-read-midx.o\n TEST_BUILTINS_OBJS += test-ref-store.o\n+TEST_BUILTINS_OBJS += test-reftable.o\n TEST_BUILTINS_OBJS += test-regex.o\n TEST_BUILTINS_OBJS += test-repository.o\n TEST_BUILTINS_OBJS += test-revision-walking.o\n@@ -813,6 +814,8 @@ TEST_SHELL_PATH = $(SHELL_PATH)\n \n LIB_FILE = libgit.a\n XDIFF_LIB = xdiff/lib.a\n+REFTABLE_LIB = reftable/libreftable.a\n+REFTABLE_TEST_LIB = reftable/libreftable_test.a\n \n GENERATED_H += command-list.h\n GENERATED_H += config-list.h\n@@ -1181,7 +1184,7 @@ THIRD_PARTY_SOURCES += compat/regex/%\n THIRD_PARTY_SOURCES += sha1collisiondetection/%\n THIRD_PARTY_SOURCES += sha1dc/%\n \n-GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB)\n+GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB)\n EXTLIBS =\n \n GIT_USER_AGENT = git/$(GIT_VERSION)\n@@ -2398,7 +2401,15 @@ XDIFF_OBJS += xdiff/xutils.o\n .PHONY: xdiff-objs\n xdiff-objs: $(XDIFF_OBJS)\n \n+REFTABLE_OBJS += reftable/basics.o\n+REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/publicbasics.o\n+\n+REFTABLE_TEST_OBJS += reftable/test_framework.o\n+REFTABLE_TEST_OBJS += reftable/basics_test.o\n+\n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n+\n .PHONY: test-objs\n test-objs: $(TEST_OBJS)\n \n@@ -2414,6 +2425,8 @@ OBJECTS += $(PROGRAM_OBJS)\n OBJECTS += $(TEST_OBJS)\n OBJECTS += $(XDIFF_OBJS)\n OBJECTS += $(FUZZ_OBJS)\n+OBJECTS += $(REFTABLE_OBJS) $(REFTABLE_TEST_OBJS)\n+\n ifndef NO_CURL\n \tOBJECTS += http.o http-walker.o remote-curl.o\n endif\n@@ -2565,6 +2578,12 @@ $(LIB_FILE): $(LIB_OBJS)\n $(XDIFF_LIB): $(XDIFF_OBJS)\n \t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n \n+$(REFTABLE_LIB): $(REFTABLE_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n+$(REFTABLE_TEST_LIB): $(REFTABLE_TEST_OBJS)\n+\t$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^\n+\n export DEFAULT_EDITOR DEFAULT_PAGER\n \n Documentation/GIT-EXCLUDED-PROGRAMS: FORCE\n@@ -2845,7 +2864,7 @@ perf: all\n \n t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n \n-t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS)\n+t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) $(REFTABLE_TEST_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)\n \n check-sha1:: t/helper/test-tool$X\n@@ -3175,7 +3194,7 @@ cocciclean:\n clean: profile-clean coverage-clean cocciclean\n \t$(RM) *.res\n \t$(RM) $(OBJECTS)\n-\t$(RM) $(LIB_FILE) $(XDIFF_LIB)\n+\t$(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB)\n \t$(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) git$X\n \t$(RM) $(TEST_PROGRAMS)\n \t$(RM) $(FUZZ_PROGRAMS)\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 75ed198a6a36..f41d3dab9015 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -611,6 +611,12 @@ parse_makefile_for_sources(libxdiff_SOURCES \"XDIFF_OBJS\")\n list(TRANSFORM libxdiff_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n add_library(xdiff STATIC ${libxdiff_SOURCES})\n \n+#reftable\n+parse_makefile_for_sources(reftable_SOURCES \"REFTABLE_OBJS\")\n+\n+list(TRANSFORM reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n+add_library(reftable STATIC ${reftable_SOURCES})\n+\n if(WIN32)\n \tif(NOT MSVC)#use windres when compiling with gcc and clang\n \t\tadd_custom_command(OUTPUT ${CMAKE_BINARY_DIR}/git.res\n@@ -633,7 +639,7 @@ endif()\n #link all required libraries to common-main\n add_library(common-main OBJECT ${CMAKE_SOURCE_DIR}/common-main.c)\n \n-target_link_libraries(common-main libgit xdiff ${ZLIB_LIBRARIES})\n+target_link_libraries(common-main libgit xdiff reftable ${ZLIB_LIBRARIES})\n if(Intl_FOUND)\n \ttarget_link_libraries(common-main ${Intl_LIBRARIES})\n endif()\n@@ -873,11 +879,15 @@ if(BUILD_TESTING)\n add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)\n target_link_libraries(test-fake-ssh common-main)\n \n+#reftable-tests\n+parse_makefile_for_sources(test-reftable_SOURCES \"REFTABLE_TEST_OBJS\")\n+list(TRANSFORM test-reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n+\n #test-tool\n parse_makefile_for_sources(test-tool_SOURCES \"TEST_BUILTINS_OBJS\")\n \n list(TRANSFORM test-tool_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/t/helper/\")\n-add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES})\n+add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES} ${test-reftable_SOURCES})\n target_link_libraries(test-tool common-main)\n \n set_target_properties(test-fake-ssh test-tool\ndiff --git a/reftable/basics.c b/reftable/basics.c\nnew file mode 100644\nindex 000000000000..f761e48028c4\n--- /dev/null\n+++ b/reftable/basics.c\n@@ -0,0 +1,128 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+\n+void put_be24(uint8_t *out, uint32_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 16) & 0xff);\n+\tout[1] = (uint8_t)((i >> 8) & 0xff);\n+\tout[2] = (uint8_t)(i & 0xff);\n+}\n+\n+uint32_t get_be24(uint8_t *in)\n+{\n+\treturn (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 |\n+\t (uint32_t)(in[2]);\n+}\n+\n+void put_be16(uint8_t *out, uint16_t i)\n+{\n+\tout[0] = (uint8_t)((i >> 8) & 0xff);\n+\tout[1] = (uint8_t)(i & 0xff);\n+}\n+\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n+{\n+\tsize_t lo = 0;\n+\tsize_t hi = sz;\n+\n+\t/* Invariants:\n+\t *\n+\t * (hi == sz) || f(hi) == true\n+\t * (lo == 0 && f(0) == true) || fi(lo) == false\n+\t */\n+\twhile (hi - lo > 1) {\n+\t\tsize_t mid = lo + (hi - lo) / 2;\n+\n+\t\tif (f(mid, args))\n+\t\t\thi = mid;\n+\t\telse\n+\t\t\tlo = mid;\n+\t}\n+\n+\tif (lo)\n+\t\treturn hi;\n+\n+\treturn f(0, args) ? 0 : 1;\n+}\n+\n+void free_names(char **a)\n+{\n+\tchar **p;\n+\tif (!a) {\n+\t\treturn;\n+\t}\n+\tfor (p = a; *p; p++) {\n+\t\treftable_free(*p);\n+\t}\n+\treftable_free(a);\n+}\n+\n+int names_length(char **names)\n+{\n+\tchar **p = names;\n+\tfor (; *p; p++) {\n+\t\t/* empty */\n+\t}\n+\treturn p - names;\n+}\n+\n+void parse_names(char *buf, int size, char ***namesp)\n+{\n+\tchar **names = NULL;\n+\tsize_t names_cap = 0;\n+\tsize_t names_len = 0;\n+\n+\tchar *p = buf;\n+\tchar *end = buf + size;\n+\twhile (p < end) {\n+\t\tchar *next = strchr(p, '\\n');\n+\t\tif (next && next < end) {\n+\t\t\t*next = 0;\n+\t\t} else {\n+\t\t\tnext = end;\n+\t\t}\n+\t\tif (p < next) {\n+\t\t\tif (names_len == names_cap) {\n+\t\t\t\tnames_cap = 2 * names_cap + 1;\n+\t\t\t\tnames = reftable_realloc(\n+\t\t\t\t\tnames, names_cap * sizeof(*names));\n+\t\t\t}\n+\t\t\tnames[names_len++] = xstrdup(p);\n+\t\t}\n+\t\tp = next + 1;\n+\t}\n+\n+\tnames = reftable_realloc(names, (names_len + 1) * sizeof(*names));\n+\tnames[names_len] = NULL;\n+\t*namesp = names;\n+}\n+\n+int names_equal(char **a, char **b)\n+{\n+\tint i = 0;\n+\tfor (; a[i] && b[i]; i++) {\n+\t\tif (strcmp(a[i], b[i])) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treturn a[i] == b[i];\n+}\n+\n+int common_prefix_size(struct strbuf *a, struct strbuf *b)\n+{\n+\tint p = 0;\n+\tfor (; p < a->len && p < b->len; p++) {\n+\t\tif (a->buf[p] != b->buf[p])\n+\t\t\tbreak;\n+\t}\n+\n+\treturn p;\n+}\ndiff --git a/reftable/basics.h b/reftable/basics.h\nnew file mode 100644\nindex 000000000000..096b36862b9f\n--- /dev/null\n+++ b/reftable/basics.h\n@@ -0,0 +1,60 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BASICS_H\n+#define BASICS_H\n+\n+/*\n+ * miscellaneous utilities that are not provided by Git.\n+ */\n+\n+#include \"system.h\"\n+\n+/* Bigendian en/decoding of integers */\n+\n+void put_be24(uint8_t *out, uint32_t i);\n+uint32_t get_be24(uint8_t *in);\n+void put_be16(uint8_t *out, uint16_t i);\n+\n+/*\n+ * find smallest index i in [0, sz) at which f(i) is true, assuming\n+ * that f is ascending. Return sz if f(i) is false for all indices.\n+ *\n+ * Contrary to bsearch(3), this returns something useful if the argument is not\n+ * found.\n+ */\n+int binsearch(size_t sz, int (*f)(size_t k, void *args), void *args);\n+\n+/*\n+ * Frees a NULL terminated array of malloced strings. The array itself is also\n+ * freed.\n+ */\n+void free_names(char **a);\n+\n+/* parse a newline separated list of names. `size` is the length of the buffer,\n+ * without terminating '\\0'. Empty names are discarded. */\n+void parse_names(char *buf, int size, char ***namesp);\n+\n+/* compares two NULL-terminated arrays of strings. */\n+int names_equal(char **a, char **b);\n+\n+/* returns the array size of a NULL-terminated array of strings. */\n+int names_length(char **names);\n+\n+/* Allocation routines; they invoke the functions set through\n+ * reftable_set_alloc() */\n+void *reftable_malloc(size_t sz);\n+void *reftable_realloc(void *p, size_t sz);\n+void reftable_free(void *p);\n+void *reftable_calloc(size_t sz);\n+\n+/* Find the longest shared prefix size of `a` and `b` */\n+struct strbuf;\n+int common_prefix_size(struct strbuf *a, struct strbuf *b);\n+\n+#endif\ndiff --git a/reftable/basics_test.c b/reftable/basics_test.c\nnew file mode 100644\nindex 000000000000..1fcd22972567\n--- /dev/null\n+++ b/reftable/basics_test.c\n@@ -0,0 +1,98 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct binsearch_args {\n+\tint key;\n+\tint *arr;\n+};\n+\n+static int binsearch_func(size_t i, void *void_args)\n+{\n+\tstruct binsearch_args *args = void_args;\n+\n+\treturn args->key < args->arr[i];\n+}\n+\n+static void test_binsearch(void)\n+{\n+\tint arr[] = { 2, 4, 6, 8, 10 };\n+\tsize_t sz = ARRAY_SIZE(arr);\n+\tstruct binsearch_args args = {\n+\t\t.arr = arr,\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 1; i < 11; i++) {\n+\t\tint res;\n+\t\targs.key = i;\n+\t\tres = binsearch(sz, &binsearch_func, &args);\n+\n+\t\tif (res < sz) {\n+\t\t\tEXPECT(args.key < arr[res]);\n+\t\t\tif (res > 0) {\n+\t\t\t\tEXPECT(args.key >= arr[res - 1]);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tEXPECT(args.key == 10 || args.key == 11);\n+\t\t}\n+\t}\n+}\n+\n+static void test_names_length(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", NULL };\n+\tEXPECT(names_length(a) == 2);\n+}\n+\n+static void test_parse_names_normal(void)\n+{\n+\tchar in[] = \"a\\nb\\n\";\n+\tchar **out = NULL;\n+\tparse_names(in, strlen(in), &out);\n+\tEXPECT(!strcmp(out[0], \"a\"));\n+\tEXPECT(!strcmp(out[1], \"b\"));\n+\tEXPECT(!out[2]);\n+\tfree_names(out);\n+}\n+\n+static void test_parse_names_drop_empty(void)\n+{\n+\tchar in[] = \"a\\n\\n\";\n+\tchar **out = NULL;\n+\tparse_names(in, strlen(in), &out);\n+\tEXPECT(!strcmp(out[0], \"a\"));\n+\tEXPECT(!out[1]);\n+\tfree_names(out);\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct strbuf s1 = STRBUF_INIT;\n+\tstruct strbuf s2 = STRBUF_INIT;\n+\tstrbuf_addstr(&s1, \"abcdef\");\n+\tstrbuf_addstr(&s2, \"abc\");\n+\tEXPECT(common_prefix_size(&s1, &s2) == 3);\n+\tstrbuf_release(&s1);\n+\tstrbuf_release(&s2);\n+}\n+\n+int basics_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_common_prefix);\n+\tRUN_TEST(test_parse_names_normal);\n+\tRUN_TEST(test_parse_names_drop_empty);\n+\tRUN_TEST(test_binsearch);\n+\tRUN_TEST(test_names_length);\n+\treturn 0;\n+}\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\nnew file mode 100644\nindex 000000000000..fd418cfaefd1\n--- /dev/null\n+++ b/reftable/publicbasics.c\n@@ -0,0 +1,58 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-malloc.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+static void *(*reftable_malloc_ptr)(size_t sz) = &malloc;\n+static void *(*reftable_realloc_ptr)(void *, size_t) = &realloc;\n+static void (*reftable_free_ptr)(void *) = &free;\n+\n+void *reftable_malloc(size_t sz)\n+{\n+\treturn (*reftable_malloc_ptr)(sz);\n+}\n+\n+void *reftable_realloc(void *p, size_t sz)\n+{\n+\treturn (*reftable_realloc_ptr)(p, sz);\n+}\n+\n+void reftable_free(void *p)\n+{\n+\treftable_free_ptr(p);\n+}\n+\n+void *reftable_calloc(size_t sz)\n+{\n+\tvoid *p = reftable_malloc(sz);\n+\tmemset(p, 0, sz);\n+\treturn p;\n+}\n+\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n+{\n+\treftable_malloc_ptr = malloc;\n+\treftable_realloc_ptr = realloc;\n+\treftable_free_ptr = free;\n+}\n+\n+int hash_size(uint32_t id)\n+{\n+\tswitch (id) {\n+\tcase 0:\n+\tcase GIT_SHA1_HASH_ID:\n+\t\treturn GIT_SHA1_RAWSZ;\n+\tcase GIT_SHA256_HASH_ID:\n+\t\treturn GIT_SHA256_RAWSZ;\n+\t}\n+\tabort();\n+}\ndiff --git a/reftable/reftable-malloc.h b/reftable/reftable-malloc.h\nnew file mode 100644\nindex 000000000000..5f2185f1f343\n--- /dev/null\n+++ b/reftable/reftable-malloc.h\n@@ -0,0 +1,18 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_H\n+#define REFTABLE_H\n+\n+#include \n+\n+/* Overrides the functions to use for memory management. */\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n+\n+#endif\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nnew file mode 100644\nindex 000000000000..5e7698ae654e\n--- /dev/null\n+++ b/reftable/reftable-tests.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_TESTS_H\n+#define REFTABLE_TESTS_H\n+\n+int basics_test_main(int argc, const char **argv);\n+int block_test_main(int argc, const char **argv);\n+int merged_test_main(int argc, const char **argv);\n+int record_test_main(int argc, const char **argv);\n+int refname_test_main(int argc, const char **argv);\n+int reftable_test_main(int argc, const char **argv);\n+int stack_test_main(int argc, const char **argv);\n+int tree_test_main(int argc, const char **argv);\n+int reftable_dump_main(int argc, char *const *argv);\n+\n+#endif\ndiff --git a/reftable/system.h b/reftable/system.h\nnew file mode 100644\nindex 000000000000..bf963ee458e4\n--- /dev/null\n+++ b/reftable/system.h\n@@ -0,0 +1,24 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef SYSTEM_H\n+#define SYSTEM_H\n+\n+// This header glues the reftable library to the rest of Git\n+\n+#include \"git-compat-util.h\"\n+#include \"strbuf.h\"\n+#include \"hash.h\" /* hash ID, sizes.*/\n+#include \"dir.h\" /* remove_dir_recursively, for tests.*/\n+\n+#include \n+\n+struct strbuf;\n+int hash_size(uint32_t id);\n+\n+#endif\ndiff --git a/reftable/test_framework.c b/reftable/test_framework.c\nnew file mode 100644\nindex 000000000000..0932166f2d11\n--- /dev/null\n+++ b/reftable/test_framework.c\n@@ -0,0 +1,23 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"test_framework.h\"\n+\n+#include \"basics.h\"\n+\n+void set_test_hash(uint8_t *p, int i)\n+{\n+\tmemset(p, (uint8_t)i, hash_size(GIT_SHA1_HASH_ID));\n+}\n+\n+ssize_t strbuf_add_void(void *b, const void *data, size_t sz)\n+{\n+\tstrbuf_add(b, data, sz);\n+\treturn sz;\n+}\ndiff --git a/reftable/test_framework.h b/reftable/test_framework.h\nnew file mode 100644\nindex 000000000000..774cb275bf67\n--- /dev/null\n+++ b/reftable/test_framework.h\n@@ -0,0 +1,53 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TEST_FRAMEWORK_H\n+#define TEST_FRAMEWORK_H\n+\n+#include \"system.h\"\n+#include \"reftable-error.h\"\n+\n+#define EXPECT_ERR(c) \\\n+\tif (c != 0) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: error == %d (%s), want 0\\n\", \\\n+\t\t\t__FILE__, __LINE__, c, reftable_error_str(c)); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define EXPECT_STREQ(a, b) \\\n+\tif (strcmp(a, b)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s:%d: %s (%s) != %s (%s)\\n\", __FILE__, \\\n+\t\t\t__LINE__, #a, a, #b, b); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define EXPECT(c) \\\n+\tif (!(c)) { \\\n+\t\tfflush(stderr); \\\n+\t\tfflush(stdout); \\\n+\t\tfprintf(stderr, \"%s: %d: failed assertion %s\\n\", __FILE__, \\\n+\t\t\t__LINE__, #c); \\\n+\t\tabort(); \\\n+\t}\n+\n+#define RUN_TEST(f) \\\n+\tfprintf(stderr, \"running %s\\n\", #f); \\\n+\tfflush(stderr); \\\n+\tf();\n+\n+void set_test_hash(uint8_t *p, int i);\n+\n+/* Like strbuf_add, but suitable for passing to reftable_new_writer\n+ */\n+ssize_t strbuf_add_void(void *b, const void *data, size_t sz);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nnew file mode 100644\nindex 000000000000..3b58e423e7b1\n--- /dev/null\n+++ b/t/helper/test-reftable.c\n@@ -0,0 +1,9 @@\n+#include \"reftable/reftable-tests.h\"\n+#include \"test-tool.h\"\n+\n+int cmd__reftable(int argc, const char **argv)\n+{\n+\tbasics_test_main(argc, argv);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 25c6a37e93e5..bea3285eed9f 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -50,13 +50,14 @@ static struct test_cmd cmds[] = {\n \t{ \"pcre2-config\", cmd__pcre2_config },\n \t{ \"pkt-line\", cmd__pkt_line },\n \t{ \"prio-queue\", cmd__prio_queue },\n-\t{ \"proc-receive\", cmd__proc_receive},\n+\t{ \"proc-receive\", cmd__proc_receive },\n \t{ \"progress\", cmd__progress },\n \t{ \"reach\", cmd__reach },\n \t{ \"read-cache\", cmd__read_cache },\n \t{ \"read-graph\", cmd__read_graph },\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n+\t{ \"reftable\", cmd__reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex f03c5988b20c..90f935310788 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -46,6 +46,7 @@ int cmd__read_cache(int argc, const char **argv);\n int cmd__read_graph(int argc, const char **argv);\n int cmd__read_midx(int argc, const char **argv);\n int cmd__ref_store(int argc, const char **argv);\n+int cmd__reftable(int argc, const char **argv);\n int cmd__regex(int argc, const char **argv);\n int cmd__repository(int argc, const char **argv);\n int cmd__revision_walking(int argc, const char **argv);\ndiff --git a/t/t0032-reftable-unittest.sh b/t/t0032-reftable-unittest.sh\nnew file mode 100755\nindex 000000000000..0ed14971a580\n--- /dev/null\n+++ b/t/t0032-reftable-unittest.sh\n@@ -0,0 +1,15 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable unittests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'unittests' '\n+\tTMPDIR=$(pwd) && export TMPDIR &&\n+\ttest-tool reftable\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"422275","messageId":"5d54e6fac99b7db24d6a65a4ed4ca39ad1d90fe0.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 09/28] reftable: add blocksource, an abstraction for random access reads","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:37Z","receivedAt":"2021-04-19T11:38:55Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is usually used with files for storage. However, we abstract\naway this using the blocksource data structure. This has two advantages:\n\n* log blocks are zlib compressed, and handling them is simplified if we can\n discard byte segments from within the block layer.\n\n* for unittests, it is useful to read and write in-memory. The blocksource\n allows us to abstract the data away from on-disk files.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/blocksource.c | 148 ++++++++++++++++++++++++++++++++\n reftable/blocksource.h | 22 +++++\n reftable/reftable-blocksource.h | 49 +++++++++++\n 4 files changed, 220 insertions(+)\n create mode 100644 reftable/blocksource.c\n create mode 100644 reftable/blocksource.h\n create mode 100644 reftable/reftable-blocksource.h\n\ndiff --git a/Makefile b/Makefile\nindex 6f6143672333..189bf572661b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2403,6 +2403,7 @@ xdiff-objs: $(XDIFF_OBJS)\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n \n REFTABLE_TEST_OBJS += reftable/test_framework.o\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nnew file mode 100644\nindex 000000000000..0044eecd9aa3\n--- /dev/null\n+++ b/reftable/blocksource.c\n@@ -0,0 +1,148 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"reftable-blocksource.h\"\n+#include \"reftable-error.h\"\n+\n+static void strbuf_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void strbuf_close(void *b)\n+{\n+}\n+\n+static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct strbuf *b = v;\n+\tassert(off + size <= b->len);\n+\tdest->data = reftable_calloc(size);\n+\tmemcpy(dest->data, b->buf + off, size);\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static uint64_t strbuf_size(void *b)\n+{\n+\treturn ((struct strbuf *)b)->len;\n+}\n+\n+static struct reftable_block_source_vtable strbuf_vtable = {\n+\t.size = &strbuf_size,\n+\t.read_block = &strbuf_read_block,\n+\t.return_block = &strbuf_return_block,\n+\t.close = &strbuf_close,\n+};\n+\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf)\n+{\n+\tassert(!bs->ops);\n+\tbs->ops = &strbuf_vtable;\n+\tbs->arg = buf;\n+}\n+\n+static void malloc_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static struct reftable_block_source_vtable malloc_vtable = {\n+\t.return_block = &malloc_return_block,\n+};\n+\n+static struct reftable_block_source malloc_block_source_instance = {\n+\t.ops = &malloc_vtable,\n+};\n+\n+struct reftable_block_source malloc_block_source(void)\n+{\n+\treturn malloc_block_source_instance;\n+}\n+\n+struct file_block_source {\n+\tint fd;\n+\tuint64_t size;\n+};\n+\n+static uint64_t file_size(void *b)\n+{\n+\treturn ((struct file_block_source *)b)->size;\n+}\n+\n+static void file_return_block(void *b, struct reftable_block *dest)\n+{\n+\tmemset(dest->data, 0xff, dest->len);\n+\treftable_free(dest->data);\n+}\n+\n+static void file_close(void *b)\n+{\n+\tint fd = ((struct file_block_source *)b)->fd;\n+\tif (fd > 0) {\n+\t\tclose(fd);\n+\t\t((struct file_block_source *)b)->fd = 0;\n+\t}\n+\n+\treftable_free(b);\n+}\n+\n+static int file_read_block(void *v, struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tstruct file_block_source *b = v;\n+\tassert(off + size <= b->size);\n+\tdest->data = reftable_malloc(size);\n+\tif (pread(b->fd, dest->data, size, off) != size)\n+\t\treturn -1;\n+\tdest->len = size;\n+\treturn size;\n+}\n+\n+static struct reftable_block_source_vtable file_vtable = {\n+\t.size = &file_size,\n+\t.read_block = &file_read_block,\n+\t.return_block = &file_return_block,\n+\t.close = &file_close,\n+};\n+\n+int reftable_block_source_from_file(struct reftable_block_source *bs,\n+\t\t\t\t const char *name)\n+{\n+\tstruct stat st = { 0 };\n+\tint err = 0;\n+\tint fd = open(name, O_RDONLY);\n+\tstruct file_block_source *p = NULL;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\treturn REFTABLE_NOT_EXIST_ERROR;\n+\t\t}\n+\t\treturn -1;\n+\t}\n+\n+\terr = fstat(fd, &st);\n+\tif (err < 0)\n+\t\treturn -1;\n+\n+\tp = reftable_calloc(sizeof(struct file_block_source));\n+\tp->size = st.st_size;\n+\tp->fd = fd;\n+\n+\tassert(!bs->ops);\n+\tbs->ops = &file_vtable;\n+\tbs->arg = p;\n+\treturn 0;\n+}\ndiff --git a/reftable/blocksource.h b/reftable/blocksource.h\nnew file mode 100644\nindex 000000000000..072e2727ad20\n--- /dev/null\n+++ b/reftable/blocksource.h\n@@ -0,0 +1,22 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCKSOURCE_H\n+#define BLOCKSOURCE_H\n+\n+#include \"system.h\"\n+\n+struct reftable_block_source;\n+\n+/* Create an in-memory block source for reading reftables */\n+void block_source_from_strbuf(struct reftable_block_source *bs,\n+\t\t\t struct strbuf *buf);\n+\n+struct reftable_block_source malloc_block_source(void);\n+\n+#endif\ndiff --git a/reftable/reftable-blocksource.h b/reftable/reftable-blocksource.h\nnew file mode 100644\nindex 000000000000..5aa3990a5732\n--- /dev/null\n+++ b/reftable/reftable-blocksource.h\n@@ -0,0 +1,49 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_BLOCKSOURCE_H\n+#define REFTABLE_BLOCKSOURCE_H\n+\n+#include \n+\n+/* block_source is a generic wrapper for a seekable readable file.\n+ */\n+struct reftable_block_source {\n+\tstruct reftable_block_source_vtable *ops;\n+\tvoid *arg;\n+};\n+\n+/* a contiguous segment of bytes. It keeps track of its generating block_source\n+ * so it can return itself into the pool. */\n+struct reftable_block {\n+\tuint8_t *data;\n+\tint len;\n+\tstruct reftable_block_source source;\n+};\n+\n+/* block_source_vtable are the operations that make up block_source */\n+struct reftable_block_source_vtable {\n+\t/* returns the size of a block source */\n+\tuint64_t (*size)(void *source);\n+\n+\t/* reads a segment from the block source. It is an error to read\n+\t beyond the end of the block */\n+\tint (*read_block)(void *source, struct reftable_block *dest,\n+\t\t\t uint64_t off, uint32_t size);\n+\t/* mark the block as read; may return the data back to malloc */\n+\tvoid (*return_block)(void *source, struct reftable_block *blockp);\n+\n+\t/* release all resources associated with the block source */\n+\tvoid (*close)(void *source);\n+};\n+\n+/* opens a file on the file system as a block_source */\n+int reftable_block_source_from_file(struct reftable_block_source *block_src,\n+\t\t\t\t const char *name);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"422276","messageId":"0c2604d7e7d453fb86a00c97a139fb593b54eef3.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 11/28] Provide zlib's uncompress2 from compat/zlib-compat.c","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:39Z","receivedAt":"2021-04-19T11:38:55Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis will be needed for reading reflog blocks in reftable.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 7 +++\n compat/.gitattributes | 1 +\n compat/zlib-uncompress2.c | 92 +++++++++++++++++++++++++++++++++++++++\n configure.ac | 13 ++++++\n git-compat-util.h | 4 ++\n 5 files changed, 117 insertions(+)\n create mode 100644 compat/.gitattributes\n create mode 100644 compat/zlib-uncompress2.c\n\ndiff --git a/Makefile b/Makefile\nindex d6bc336c5565..e6df765d3361 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -256,6 +256,8 @@ all::\n #\n # Define NO_DEFLATE_BOUND if your zlib does not have deflateBound.\n #\n+# Define NO_UNCOMPRESS2 if your zlib does not have uncompress2.\n+#\n # Define NO_NORETURN if using buggy versions of gcc 4.6+ and profile feedback,\n # as the compiler can crash (http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49299)\n #\n@@ -1706,6 +1708,11 @@ ifdef NO_DEFLATE_BOUND\n \tBASIC_CFLAGS += -DNO_DEFLATE_BOUND\n endif\n \n+ifdef NO_UNCOMPRESS2\n+\tBASIC_CFLAGS += -DNO_UNCOMPRESS2\n+\tLIB_OBJS += compat/zlib-uncompress2.o\n+endif\n+\n ifdef NO_POSIX_GOODIES\n \tBASIC_CFLAGS += -DNO_POSIX_GOODIES\n endif\ndiff --git a/compat/.gitattributes b/compat/.gitattributes\nnew file mode 100644\nindex 000000000000..40dbfb170dab\n--- /dev/null\n+++ b/compat/.gitattributes\n@@ -0,0 +1 @@\n+/zlib-uncompress2.c\twhitespace=-indent-with-non-tab,-trailing-space\ndiff --git a/compat/zlib-uncompress2.c b/compat/zlib-uncompress2.c\nnew file mode 100644\nindex 000000000000..0adc5ee7eeb6\n--- /dev/null\n+++ b/compat/zlib-uncompress2.c\n@@ -0,0 +1,92 @@\n+/* taken from zlib's uncompr.c\n+\n+ commit cacf7f1d4e3d44d871b605da3b647f07d718623f\n+ Author: Mark Adler \n+ Date: Sun Jan 15 09:18:46 2017 -0800\n+\n+ zlib 1.2.11\n+\n+*/\n+\n+/*\n+ * Copyright (C) 1995-2003, 2010, 2014, 2016 Jean-loup Gailly, Mark Adler\n+ * For conditions of distribution and use, see copyright notice in zlib.h\n+ */\n+\n+#include \"system.h\"\n+\n+/* clang-format off */\n+\n+/* ===========================================================================\n+ Decompresses the source buffer into the destination buffer. *sourceLen is\n+ the byte length of the source buffer. Upon entry, *destLen is the total size\n+ of the destination buffer, which must be large enough to hold the entire\n+ uncompressed data. (The size of the uncompressed data must have been saved\n+ previously by the compressor and transmitted to the decompressor by some\n+ mechanism outside the scope of this compression library.) Upon exit,\n+ *destLen is the size of the decompressed data and *sourceLen is the number\n+ of source bytes consumed. Upon return, source + *sourceLen points to the\n+ first unused input byte.\n+\n+ uncompress returns Z_OK if success, Z_MEM_ERROR if there was not enough\n+ memory, Z_BUF_ERROR if there was not enough room in the output buffer, or\n+ Z_DATA_ERROR if the input data was corrupted, including if the input data is\n+ an incomplete zlib stream.\n+*/\n+int ZEXPORT uncompress2 (\n+ Bytef *dest,\n+ uLongf *destLen,\n+ const Bytef *source,\n+ uLong *sourceLen) {\n+ z_stream stream;\n+ int err;\n+ const uInt max = (uInt)-1;\n+ uLong len, left;\n+ Byte buf[1]; /* for detection of incomplete stream when *destLen == 0 */\n+\n+ len = *sourceLen;\n+ if (*destLen) {\n+ left = *destLen;\n+ *destLen = 0;\n+ }\n+ else {\n+ left = 1;\n+ dest = buf;\n+ }\n+\n+ stream.next_in = (z_const Bytef *)source;\n+ stream.avail_in = 0;\n+ stream.zalloc = (alloc_func)0;\n+ stream.zfree = (free_func)0;\n+ stream.opaque = (voidpf)0;\n+\n+ err = inflateInit(&stream);\n+ if (err != Z_OK) return err;\n+\n+ stream.next_out = dest;\n+ stream.avail_out = 0;\n+\n+ do {\n+ if (stream.avail_out == 0) {\n+ stream.avail_out = left > (uLong)max ? max : (uInt)left;\n+ left -= stream.avail_out;\n+ }\n+ if (stream.avail_in == 0) {\n+ stream.avail_in = len > (uLong)max ? max : (uInt)len;\n+ len -= stream.avail_in;\n+ }\n+ err = inflate(&stream, Z_NO_FLUSH);\n+ } while (err == Z_OK);\n+\n+ *sourceLen -= len + stream.avail_in;\n+ if (dest != buf)\n+ *destLen = stream.total_out;\n+ else if (stream.total_out && err == Z_BUF_ERROR)\n+ left = 1;\n+\n+ inflateEnd(&stream);\n+ return err == Z_STREAM_END ? Z_OK :\n+ err == Z_NEED_DICT ? Z_DATA_ERROR :\n+ err == Z_BUF_ERROR && left + stream.avail_out ? Z_DATA_ERROR :\n+ err;\n+}\ndiff --git a/configure.ac b/configure.ac\nindex 031e8d3fee8f..c3a913103d00 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -672,9 +672,22 @@ AC_LINK_IFELSE([ZLIBTEST_SRC],\n \tNO_DEFLATE_BOUND=yes])\n LIBS=\"$old_LIBS\"\n \n+AC_DEFUN([ZLIBTEST_UNCOMPRESS2_SRC], [\n+AC_LANG_PROGRAM([#include ],\n+ [uncompress2(NULL,NULL,NULL,NULL);])])\n+AC_MSG_CHECKING([for uncompress2 in -lz])\n+old_LIBS=\"$LIBS\"\n+LIBS=\"$LIBS -lz\"\n+AC_LINK_IFELSE([ZLIBTEST_UNCOMPRESS2_SRC],\n+\t[AC_MSG_RESULT([yes])],\n+\t[AC_MSG_RESULT([no])\n+\tNO_UNCOMPRESS2=yes])\n+LIBS=\"$old_LIBS\"\n+\n GIT_UNSTASH_FLAGS($ZLIB_PATH)\n \n GIT_CONF_SUBST([NO_DEFLATE_BOUND])\n+GIT_CONF_SUBST([NO_UNCOMPRESS2])\n \n #\n # Define NEEDS_SOCKET if linking with libc is not enough (SunOS,\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex a508dbe5a351..d405d0bd7a79 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -1382,4 +1382,8 @@ static inline void *container_of_or_null_offset(void *ptr, size_t offset)\n \n void sleep_millisec(int millisec);\n \n+#ifdef NO_UNCOMPRESS2\n+\n+#endif\n+\n #endif\n-- \ngitgitgadget\n\n"},{"id":"422277","messageId":"5177919a3fd2f85cdf28ecf745361ae4ea75a7b8.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 07/28] reftable: add error related functionality","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:35Z","receivedAt":"2021-04-19T11:40:17Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable/ directory is structured as a library, so it cannot\ncrash on misuse. Instead, it returns an error codes.\n\nIn addition, the error code can be used to signal conditions from lower levels\nof the library to be handled by higher levels of the library. For example, a\ntransaction might legitimately write an empty reftable file, but in that case,\nwe'd want to shortcut the transaction overhead.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/error.c | 41 ++++++++++++++++++++++++++\n reftable/reftable-error.h | 62 +++++++++++++++++++++++++++++++++++++++\n 2 files changed, 103 insertions(+)\n create mode 100644 reftable/error.c\n create mode 100644 reftable/reftable-error.h\n\ndiff --git a/reftable/error.c b/reftable/error.c\nnew file mode 100644\nindex 000000000000..f6f16def9214\n--- /dev/null\n+++ b/reftable/error.c\n@@ -0,0 +1,41 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reftable-error.h\"\n+\n+#include \n+\n+const char *reftable_error_str(int err)\n+{\n+\tstatic char buf[250];\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn \"I/O error\";\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn \"corrupt reftable file\";\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn \"file does not exist\";\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn \"data is outdated\";\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn \"misuse of the reftable API\";\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn \"zlib failure\";\n+\tcase REFTABLE_NAME_CONFLICT:\n+\t\treturn \"file/directory conflict\";\n+\tcase REFTABLE_EMPTY_TABLE_ERROR:\n+\t\treturn \"wrote empty table\";\n+\tcase REFTABLE_REFNAME_ERROR:\n+\t\treturn \"invalid refname\";\n+\tcase -1:\n+\t\treturn \"general error\";\n+\tdefault:\n+\t\tsnprintf(buf, sizeof(buf), \"unknown error code %d\", err);\n+\t\treturn buf;\n+\t}\n+}\ndiff --git a/reftable/reftable-error.h b/reftable/reftable-error.h\nnew file mode 100644\nindex 000000000000..6f89bedf1a58\n--- /dev/null\n+++ b/reftable/reftable-error.h\n@@ -0,0 +1,62 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_ERROR_H\n+#define REFTABLE_ERROR_H\n+\n+/*\n+ * Errors in reftable calls are signaled with negative integer return values. 0\n+ * means success.\n+ */\n+enum reftable_error {\n+\t/* Unexpected file system behavior */\n+\tREFTABLE_IO_ERROR = -2,\n+\n+\t/* Format inconsistency on reading data */\n+\tREFTABLE_FORMAT_ERROR = -3,\n+\n+\t/* File does not exist. Returned from block_source_from_file(), because\n+\t * it needs special handling in stack.\n+\t */\n+\tREFTABLE_NOT_EXIST_ERROR = -4,\n+\n+\t/* Trying to write out-of-date data. */\n+\tREFTABLE_LOCK_ERROR = -5,\n+\n+\t/* Misuse of the API:\n+\t * - on writing a record with NULL refname.\n+\t * - on writing a reftable_ref_record outside the table limits\n+\t * - on writing a ref or log record before the stack's\n+\t * next_update_inde*x\n+\t * - on writing a log record with multiline message with\n+\t * exact_log_message unset\n+\t * - on reading a reftable_ref_record from log iterator, or vice versa.\n+\t *\n+\t * When a call misuses the API, the internal state of the library is\n+\t * kept unchanged.\n+\t */\n+\tREFTABLE_API_ERROR = -6,\n+\n+\t/* Decompression error */\n+\tREFTABLE_ZLIB_ERROR = -7,\n+\n+\t/* Wrote a table without blocks. */\n+\tREFTABLE_EMPTY_TABLE_ERROR = -8,\n+\n+\t/* Dir/file conflict. */\n+\tREFTABLE_NAME_CONFLICT = -9,\n+\n+\t/* Invalid ref name. */\n+\tREFTABLE_REFNAME_ERROR = -10,\n+};\n+\n+/* convert the numeric error code to a string. The string should not be\n+ * deallocated. */\n+const char *reftable_error_str(int err);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"422278","messageId":"028dd13e701bf82523cc6da1e68a9dab480c1b27.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 10/28] reftable: (de)serialization for the polymorphic record type.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:38Z","receivedAt":"2021-04-19T11:40:19Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is structured as a sequence of blocks, and each block\ncontains a sequence of prefix-compressed key-value records. There are 4 types of\nrecords, and they have similarities in how they must be handled. This is\nachieved by introducing a polymorphic 'record' type that encapsulates ref, log,\nindex and object records.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/constants.h | 21 +\n reftable/record.c | 1202 ++++++++++++++++++++++++++++++++++++\n reftable/record.h | 139 +++++\n reftable/record_test.c | 407 ++++++++++++\n reftable/reftable-record.h | 114 ++++\n t/helper/test-reftable.c | 2 +-\n 7 files changed, 1886 insertions(+), 1 deletion(-)\n create mode 100644 reftable/constants.h\n create mode 100644 reftable/record.c\n create mode 100644 reftable/record.h\n create mode 100644 reftable/record_test.c\n create mode 100644 reftable/reftable-record.h\n\ndiff --git a/Makefile b/Makefile\nindex 189bf572661b..d6bc336c5565 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2405,7 +2405,9 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/record.o\n \n+REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n \ndiff --git a/reftable/constants.h b/reftable/constants.h\nnew file mode 100644\nindex 000000000000..5eee72c4c113\n--- /dev/null\n+++ b/reftable/constants.h\n@@ -0,0 +1,21 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef CONSTANTS_H\n+#define CONSTANTS_H\n+\n+#define BLOCK_TYPE_LOG 'g'\n+#define BLOCK_TYPE_INDEX 'i'\n+#define BLOCK_TYPE_REF 'r'\n+#define BLOCK_TYPE_OBJ 'o'\n+#define BLOCK_TYPE_ANY 0\n+\n+#define MAX_RESTARTS ((1 << 16) - 1)\n+#define DEFAULT_BLOCK_SIZE 4096\n+\n+#endif\ndiff --git a/reftable/record.c b/reftable/record.c\nnew file mode 100644\nindex 000000000000..b111852b667e\n--- /dev/null\n+++ b/reftable/record.c\n@@ -0,0 +1,1202 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+/* record.c - methods for different types of records. */\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"constants.h\"\n+#include \"reftable-error.h\"\n+#include \"basics.h\"\n+\n+int get_var_int(uint64_t *dest, struct string_view *in)\n+{\n+\tint ptr = 0;\n+\tuint64_t val;\n+\n+\tif (in->len == 0)\n+\t\treturn -1;\n+\tval = in->buf[ptr] & 0x7f;\n+\n+\twhile (in->buf[ptr] & 0x80) {\n+\t\tptr++;\n+\t\tif (ptr > in->len) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tval = (val + 1) << 7 | (uint64_t)(in->buf[ptr] & 0x7f);\n+\t}\n+\n+\t*dest = val;\n+\treturn ptr + 1;\n+}\n+\n+int put_var_int(struct string_view *dest, uint64_t val)\n+{\n+\tuint8_t buf[10] = { 0 };\n+\tint i = 9;\n+\tint n = 0;\n+\tbuf[i] = (uint8_t)(val & 0x7f);\n+\ti--;\n+\twhile (1) {\n+\t\tval >>= 7;\n+\t\tif (!val) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tval--;\n+\t\tbuf[i] = 0x80 | (uint8_t)(val & 0x7f);\n+\t\ti--;\n+\t}\n+\n+\tn = sizeof(buf) - i - 1;\n+\tif (dest->len < n)\n+\t\treturn -1;\n+\tmemcpy(dest->buf, &buf[i + 1], n);\n+\treturn n;\n+}\n+\n+int reftable_is_block_type(uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\tcase BLOCK_TYPE_LOG:\n+\tcase BLOCK_TYPE_OBJ:\n+\tcase BLOCK_TYPE_INDEX:\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec)\n+{\n+\tswitch (rec->value_type) {\n+\tcase REFTABLE_REF_VAL1:\n+\t\treturn rec->value.val1;\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn rec->value.val2.value;\n+\tdefault:\n+\t\treturn NULL;\n+\t}\n+}\n+\n+uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec)\n+{\n+\tswitch (rec->value_type) {\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn rec->value.val2.target_value;\n+\tdefault:\n+\t\treturn NULL;\n+\t}\n+}\n+\n+static int decode_string(struct strbuf *dest, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t tsize = 0;\n+\tint n = get_var_int(&tsize, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\tif (in.len < tsize)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, in.buf, tsize);\n+\tstring_view_consume(&in, tsize);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static int encode_string(char *str, struct string_view s)\n+{\n+\tstruct string_view start = s;\n+\tint l = strlen(str);\n+\tint n = put_var_int(&s, l);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\tif (s.len < l)\n+\t\treturn -1;\n+\tmemcpy(s.buf, str, l);\n+\tstring_view_consume(&s, l);\n+\n+\treturn start.len - s.len;\n+}\n+\n+int reftable_encode_key(int *restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra)\n+{\n+\tstruct string_view start = dest;\n+\tint prefix_len = common_prefix_size(&prev_key, &key);\n+\tuint64_t suffix_len = key.len - prefix_len;\n+\tint n = put_var_int(&dest, (uint64_t)prefix_len);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\t*restart = (prefix_len == 0);\n+\n+\tn = put_var_int(&dest, suffix_len << 3 | (uint64_t)extra);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&dest, n);\n+\n+\tif (dest.len < suffix_len)\n+\t\treturn -1;\n+\tmemcpy(dest.buf, key.buf + prefix_len, suffix_len);\n+\tstring_view_consume(&dest, suffix_len);\n+\n+\treturn start.len - dest.len;\n+}\n+\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in)\n+{\n+\tint start_len = in.len;\n+\tuint64_t prefix_len = 0;\n+\tuint64_t suffix_len = 0;\n+\tint n = get_var_int(&prefix_len, &in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tif (prefix_len > last_key.len)\n+\t\treturn -1;\n+\n+\tn = get_var_int(&suffix_len, &in);\n+\tif (n <= 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\t*extra = (uint8_t)(suffix_len & 0x7);\n+\tsuffix_len >>= 3;\n+\n+\tif (in.len < suffix_len)\n+\t\treturn -1;\n+\n+\tstrbuf_reset(key);\n+\tstrbuf_add(key, last_key.buf, prefix_len);\n+\tstrbuf_add(key, in.buf, suffix_len);\n+\tstring_view_consume(&in, suffix_len);\n+\n+\treturn start_len - in.len;\n+}\n+\n+static void reftable_ref_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_ref_record *rec =\n+\t\t(const struct reftable_ref_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, rec->refname);\n+}\n+\n+static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *ref = rec;\n+\tconst struct reftable_ref_record *src = src_rec;\n+\tassert(hash_size > 0);\n+\n+\t/* This is simple and correct, but we could probably reuse the hash\n+\t * fields. */\n+\treftable_ref_record_release(ref);\n+\tif (src->refname) {\n+\t\tref->refname = xstrdup(src->refname);\n+\t}\n+\tref->update_index = src->update_index;\n+\tref->value_type = src->value_type;\n+\tswitch (src->value_type) {\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\tref->value.val1 = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val1, src->value.val1, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\tref->value.val2.value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val2.value, src->value.val2.value, hash_size);\n+\t\tref->value.val2.target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(ref->value.val2.target_value,\n+\t\t src->value.val2.target_value, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tref->value.symref = xstrdup(src->value.symref);\n+\t\tbreak;\n+\t}\n+}\n+\n+static char hexdigit(int c)\n+{\n+\tif (c <= 9)\n+\t\treturn '0' + c;\n+\treturn 'a' + (c - 10);\n+}\n+\n+static void hex_format(char *dest, uint8_t *src, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (src) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < hash_size; i++) {\n+\t\t\tdest[2 * i] = hexdigit(src[i] >> 4);\n+\t\t\tdest[2 * i + 1] = hexdigit(src[i] & 0xf);\n+\t\t}\n+\t\tdest[2 * hash_size] = 0;\n+\t}\n+}\n+\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[2 * GIT_SHA256_RAWSZ + 1] = { 0 }; /* BUG */\n+\tprintf(\"ref{%s(%\" PRIu64 \") \", ref->refname, ref->update_index);\n+\tswitch (ref->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tprintf(\"=> %s\", ref->value.symref);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\thex_format(hex, ref->value.val2.value, hash_size(hash_id));\n+\t\tprintf(\"val 2 %s\", hex);\n+\t\thex_format(hex, ref->value.val2.target_value,\n+\t\t\t hash_size(hash_id));\n+\t\tprintf(\"(T %s)\", hex);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\thex_format(hex, ref->value.val1, hash_size(hash_id));\n+\t\tprintf(\"val 1 %s\", hex);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tprintf(\"delete\");\n+\t\tbreak;\n+\t}\n+\tprintf(\"}\\n\");\n+}\n+\n+static void reftable_ref_record_release_void(void *rec)\n+{\n+\treftable_ref_record_release(rec);\n+}\n+\n+void reftable_ref_record_release(struct reftable_ref_record *ref)\n+{\n+\tswitch (ref->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\treftable_free(ref->value.symref);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\treftable_free(ref->value.val2.target_value);\n+\t\treftable_free(ref->value.val2.value);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\treftable_free(ref->value.val1);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t}\n+\n+\treftable_free(ref->refname);\n+\tmemset(ref, 0, sizeof(struct reftable_ref_record));\n+}\n+\n+static uint8_t reftable_ref_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\treturn r->value_type;\n+}\n+\n+static int reftable_ref_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_ref_record *r =\n+\t\t(const struct reftable_ref_record *)rec;\n+\tstruct string_view start = s;\n+\tint n = put_var_int(&s, r->update_index);\n+\tassert(hash_size > 0);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tswitch (r->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\tn = encode_string(r->value.symref, s);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL2:\n+\t\tif (s.len < 2 * hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value.val2.value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tmemcpy(s.buf, r->value.val2.target_value, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_VAL1:\n+\t\tif (s.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tmemcpy(s.buf, r->value.val1, hash_size);\n+\t\tstring_view_consume(&s, hash_size);\n+\t\tbreak;\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t}\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_ref_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_ref_record *r = rec;\n+\tstruct string_view start = in;\n+\tuint64_t update_index = 0;\n+\tint n = get_var_int(&update_index, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\treftable_ref_record_release(r);\n+\n+\tassert(hash_size > 0);\n+\n+\tr->refname = reftable_realloc(r->refname, key.len + 1);\n+\tmemcpy(r->refname, key.buf, key.len);\n+\tr->update_index = update_index;\n+\tr->refname[key.len] = 0;\n+\tr->value_type = val_type;\n+\tswitch (val_type) {\n+\tcase REFTABLE_REF_VAL1:\n+\t\tif (in.len < hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tr->value.val1 = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val1, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\n+\tcase REFTABLE_REF_VAL2:\n+\t\tif (in.len < 2 * hash_size) {\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tr->value.val2.value = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val2.value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\n+\t\tr->value.val2.target_value = reftable_malloc(hash_size);\n+\t\tmemcpy(r->value.val2.target_value, in.buf, hash_size);\n+\t\tstring_view_consume(&in, hash_size);\n+\t\tbreak;\n+\n+\tcase REFTABLE_REF_SYMREF: {\n+\t\tstruct strbuf dest = STRBUF_INIT;\n+\t\tint n = decode_string(&dest, in);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\t\tr->value.symref = dest.buf;\n+\t} break;\n+\n+\tcase REFTABLE_REF_DELETION:\n+\t\tbreak;\n+\tdefault:\n+\t\tabort();\n+\t\tbreak;\n+\t}\n+\n+\treturn start.len - in.len;\n+}\n+\n+static int reftable_ref_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_ref_record_is_deletion(\n+\t\t(const struct reftable_ref_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_ref_record_vtable = {\n+\t.key = &reftable_ref_record_key,\n+\t.type = BLOCK_TYPE_REF,\n+\t.copy_from = &reftable_ref_record_copy_from,\n+\t.val_type = &reftable_ref_record_val_type,\n+\t.encode = &reftable_ref_record_encode,\n+\t.decode = &reftable_ref_record_decode,\n+\t.release = &reftable_ref_record_release_void,\n+\t.is_deletion = &reftable_ref_record_is_deletion_void,\n+};\n+\n+static void reftable_obj_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_obj_record *rec =\n+\t\t(const struct reftable_obj_record *)r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, rec->hash_prefix, rec->hash_prefix_len);\n+}\n+\n+static void reftable_obj_record_release(void *rec)\n+{\n+\tstruct reftable_obj_record *obj = rec;\n+\tFREE_AND_NULL(obj->hash_prefix);\n+\tFREE_AND_NULL(obj->offsets);\n+\tmemset(obj, 0, sizeof(struct reftable_obj_record));\n+}\n+\n+static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_obj_record *obj = rec;\n+\tconst struct reftable_obj_record *src =\n+\t\t(const struct reftable_obj_record *)src_rec;\n+\tint olen;\n+\n+\treftable_obj_record_release(obj);\n+\t*obj = *src;\n+\tobj->hash_prefix = reftable_malloc(obj->hash_prefix_len);\n+\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n+\n+\tolen = obj->offset_len * sizeof(uint64_t);\n+\tobj->offsets = reftable_malloc(olen);\n+\tmemcpy(obj->offsets, src->offsets, olen);\n+}\n+\n+static uint8_t reftable_obj_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_obj_record *r = rec;\n+\tif (r->offset_len > 0 && r->offset_len < 8)\n+\t\treturn r->offset_len;\n+\treturn 0;\n+}\n+\n+static int reftable_obj_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_obj_record *r = rec;\n+\tstruct string_view start = s;\n+\tint i = 0;\n+\tint n = 0;\n+\tuint64_t last = 0;\n+\tif (r->offset_len == 0 || r->offset_len >= 8) {\n+\t\tn = put_var_int(&s, r->offset_len);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t}\n+\tif (r->offset_len == 0)\n+\t\treturn start.len - s.len;\n+\tn = put_var_int(&s, r->offsets[0]);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tlast = r->offsets[0];\n+\tfor (i = 1; i < r->offset_len; i++) {\n+\t\tint n = put_var_int(&s, r->offsets[i] - last);\n+\t\tif (n < 0) {\n+\t\t\treturn -1;\n+\t\t}\n+\t\tstring_view_consume(&s, n);\n+\t\tlast = r->offsets[i];\n+\t}\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_obj_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_obj_record *r = rec;\n+\tuint64_t count = val_type;\n+\tint n = 0;\n+\tuint64_t last;\n+\tint j;\n+\tr->hash_prefix = reftable_malloc(key.len);\n+\tmemcpy(r->hash_prefix, key.buf, key.len);\n+\tr->hash_prefix_len = key.len;\n+\n+\tif (val_type == 0) {\n+\t\tn = get_var_int(&count, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\n+\t\tstring_view_consume(&in, n);\n+\t}\n+\n+\tr->offsets = NULL;\n+\tr->offset_len = 0;\n+\tif (count == 0)\n+\t\treturn start.len - in.len;\n+\n+\tr->offsets = reftable_malloc(count * sizeof(uint64_t));\n+\tr->offset_len = count;\n+\n+\tn = get_var_int(&r->offsets[0], &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\tstring_view_consume(&in, n);\n+\n+\tlast = r->offsets[0];\n+\tj = 1;\n+\twhile (j < count) {\n+\t\tuint64_t delta = 0;\n+\t\tint n = get_var_int(&delta, &in);\n+\t\tif (n < 0) {\n+\t\t\treturn n;\n+\t\t}\n+\t\tstring_view_consume(&in, n);\n+\n+\t\tlast = r->offsets[j] = (delta + last);\n+\t\tj++;\n+\t}\n+\treturn start.len - in.len;\n+}\n+\n+static int not_a_deletion(const void *p)\n+{\n+\treturn 0;\n+}\n+\n+static struct reftable_record_vtable reftable_obj_record_vtable = {\n+\t.key = &reftable_obj_record_key,\n+\t.type = BLOCK_TYPE_OBJ,\n+\t.copy_from = &reftable_obj_record_copy_from,\n+\t.val_type = &reftable_obj_record_val_type,\n+\t.encode = &reftable_obj_record_encode,\n+\t.decode = &reftable_obj_record_decode,\n+\t.release = &reftable_obj_record_release,\n+\t.is_deletion = not_a_deletion,\n+};\n+\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id)\n+{\n+\tchar hex[GIT_SHA256_RAWSZ + 1] = { 0 };\n+\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tprintf(\"log{%s(%\" PRIu64 \") delete\", log->refname,\n+\t\t log->update_index);\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tprintf(\"log{%s(%\" PRIu64 \") %s <%s> %\" PRIu64 \" %04d\\n\",\n+\t\t log->refname, log->update_index, log->update.name,\n+\t\t log->update.email, log->update.time,\n+\t\t log->update.tz_offset);\n+\t\thex_format(hex, log->update.old_hash, hash_size(hash_id));\n+\t\tprintf(\"%s => \", hex);\n+\t\thex_format(hex, log->update.new_hash, hash_size(hash_id));\n+\t\tprintf(\"%s\\n\\n%s\\n}\\n\", hex, log->update.message);\n+\t\tbreak;\n+\t}\n+}\n+\n+static void reftable_log_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_log_record *rec =\n+\t\t(const struct reftable_log_record *)r;\n+\tint len = strlen(rec->refname);\n+\tuint8_t i64[8];\n+\tuint64_t ts = 0;\n+\tstrbuf_reset(dest);\n+\tstrbuf_add(dest, (uint8_t *)rec->refname, len + 1);\n+\n+\tts = (~ts) - rec->update_index;\n+\tput_be64(&i64[0], ts);\n+\tstrbuf_add(dest, i64, sizeof(i64));\n+}\n+\n+static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_log_record *dst = rec;\n+\tconst struct reftable_log_record *src =\n+\t\t(const struct reftable_log_record *)src_rec;\n+\n+\treftable_log_record_release(dst);\n+\t*dst = *src;\n+\tif (dst->refname) {\n+\t\tdst->refname = xstrdup(dst->refname);\n+\t}\n+\tswitch (dst->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tif (dst->update.email) {\n+\t\t\tdst->update.email = xstrdup(dst->update.email);\n+\t\t}\n+\t\tif (dst->update.name) {\n+\t\t\tdst->update.name = xstrdup(dst->update.name);\n+\t\t}\n+\t\tif (dst->update.message) {\n+\t\t\tdst->update.message = xstrdup(dst->update.message);\n+\t\t}\n+\n+\t\tif (dst->update.new_hash) {\n+\t\t\tdst->update.new_hash = reftable_malloc(hash_size);\n+\t\t\tmemcpy(dst->update.new_hash, src->update.new_hash,\n+\t\t\t hash_size);\n+\t\t}\n+\t\tif (dst->update.old_hash) {\n+\t\t\tdst->update.old_hash = reftable_malloc(hash_size);\n+\t\t\tmemcpy(dst->update.old_hash, src->update.old_hash,\n+\t\t\t hash_size);\n+\t\t}\n+\t\tbreak;\n+\t}\n+}\n+\n+static void reftable_log_record_release_void(void *rec)\n+{\n+\tstruct reftable_log_record *r = rec;\n+\treftable_log_record_release(r);\n+}\n+\n+void reftable_log_record_release(struct reftable_log_record *r)\n+{\n+\treftable_free(r->refname);\n+\tswitch (r->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\treftable_free(r->update.new_hash);\n+\t\treftable_free(r->update.old_hash);\n+\t\treftable_free(r->update.name);\n+\t\treftable_free(r->update.email);\n+\t\treftable_free(r->update.message);\n+\t\tbreak;\n+\t}\n+\tmemset(r, 0, sizeof(struct reftable_log_record));\n+}\n+\n+static uint8_t reftable_log_record_val_type(const void *rec)\n+{\n+\tconst struct reftable_log_record *log =\n+\t\t(const struct reftable_log_record *)rec;\n+\n+\treturn reftable_log_record_is_deletion(log) ? 0 : 1;\n+}\n+\n+static uint8_t zero[GIT_SHA256_RAWSZ] = { 0 };\n+\n+static int reftable_log_record_encode(const void *rec, struct string_view s,\n+\t\t\t\t int hash_size)\n+{\n+\tconst struct reftable_log_record *r = rec;\n+\tstruct string_view start = s;\n+\tint n = 0;\n+\tuint8_t *oldh = NULL;\n+\tuint8_t *newh = NULL;\n+\tif (reftable_log_record_is_deletion(r))\n+\t\treturn 0;\n+\n+\toldh = r->update.old_hash;\n+\tnewh = r->update.new_hash;\n+\tif (!oldh) {\n+\t\toldh = zero;\n+\t}\n+\tif (!newh) {\n+\t\tnewh = zero;\n+\t}\n+\n+\tif (s.len < 2 * hash_size)\n+\t\treturn -1;\n+\n+\tmemcpy(s.buf, oldh, hash_size);\n+\tmemcpy(s.buf + hash_size, newh, hash_size);\n+\tstring_view_consume(&s, 2 * hash_size);\n+\n+\tn = encode_string(r->update.name ? r->update.name : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = encode_string(r->update.email ? r->update.email : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tn = put_var_int(&s, r->update.time);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\tif (s.len < 2)\n+\t\treturn -1;\n+\n+\tput_be16(s.buf, r->update.tz_offset);\n+\tstring_view_consume(&s, 2);\n+\n+\tn = encode_string(r->update.message ? r->update.message : \"\", s);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&s, n);\n+\n+\treturn start.len - s.len;\n+}\n+\n+static int reftable_log_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t uint8_t val_type, struct string_view in,\n+\t\t\t\t int hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_log_record *r = rec;\n+\tuint64_t max = 0;\n+\tuint64_t ts = 0;\n+\tstruct strbuf dest = STRBUF_INIT;\n+\tint n;\n+\n+\tif (key.len <= 9 || key.buf[key.len - 9] != 0)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->refname = reftable_realloc(r->refname, key.len - 8);\n+\tmemcpy(r->refname, key.buf, key.len - 8);\n+\tts = get_be64(key.buf + key.len - 8);\n+\n+\tr->update_index = (~max) - ts;\n+\n+\tif (val_type != r->value_type) {\n+\t\tswitch (r->value_type) {\n+\t\tcase REFTABLE_LOG_UPDATE:\n+\t\t\tFREE_AND_NULL(r->update.old_hash);\n+\t\t\tFREE_AND_NULL(r->update.new_hash);\n+\t\t\tFREE_AND_NULL(r->update.message);\n+\t\t\tFREE_AND_NULL(r->update.email);\n+\t\t\tFREE_AND_NULL(r->update.name);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_LOG_DELETION:\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tr->value_type = val_type;\n+\tif (val_type == REFTABLE_LOG_DELETION)\n+\t\treturn 0;\n+\n+\tif (in.len < 2 * hash_size)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tr->update.old_hash = reftable_realloc(r->update.old_hash, hash_size);\n+\tr->update.new_hash = reftable_realloc(r->update.new_hash, hash_size);\n+\n+\tmemcpy(r->update.old_hash, in.buf, hash_size);\n+\tmemcpy(r->update.new_hash, in.buf + hash_size, hash_size);\n+\n+\tstring_view_consume(&in, 2 * hash_size);\n+\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->update.name = reftable_realloc(r->update.name, dest.len + 1);\n+\tmemcpy(r->update.name, dest.buf, dest.len);\n+\tr->update.name[dest.len] = 0;\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->update.email = reftable_realloc(r->update.email, dest.len + 1);\n+\tmemcpy(r->update.email, dest.buf, dest.len);\n+\tr->update.email[dest.len] = 0;\n+\n+\tts = 0;\n+\tn = get_var_int(&ts, &in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\tr->update.time = ts;\n+\tif (in.len < 2)\n+\t\tgoto done;\n+\n+\tr->update.tz_offset = get_be16(in.buf);\n+\tstring_view_consume(&in, 2);\n+\n+\tstrbuf_reset(&dest);\n+\tn = decode_string(&dest, in);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&in, n);\n+\n+\tr->update.message = reftable_realloc(r->update.message, dest.len + 1);\n+\tmemcpy(r->update.message, dest.buf, dest.len);\n+\tr->update.message[dest.len] = 0;\n+\n+\tstrbuf_release(&dest);\n+\treturn start.len - in.len;\n+\n+done:\n+\tstrbuf_release(&dest);\n+\treturn REFTABLE_FORMAT_ERROR;\n+}\n+\n+static int null_streq(char *a, char *b)\n+{\n+\tchar *empty = \"\";\n+\tif (!a)\n+\t\ta = empty;\n+\n+\tif (!b)\n+\t\tb = empty;\n+\n+\treturn 0 == strcmp(a, b);\n+}\n+\n+static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)\n+{\n+\tif (!a)\n+\t\ta = zero;\n+\n+\tif (!b)\n+\t\tb = zero;\n+\n+\treturn !memcmp(a, b, sz);\n+}\n+\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size)\n+{\n+\tif (!(null_streq(a->refname, b->refname) &&\n+\t a->update_index == b->update_index &&\n+\t a->value_type == b->value_type))\n+\t\treturn 0;\n+\n+\tswitch (a->value_type) {\n+\tcase REFTABLE_LOG_DELETION:\n+\t\treturn 1;\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\treturn null_streq(a->update.name, b->update.name) &&\n+\t\t a->update.time == b->update.time &&\n+\t\t a->update.tz_offset == b->update.tz_offset &&\n+\t\t null_streq(a->update.email, b->update.email) &&\n+\t\t null_streq(a->update.message, b->update.message) &&\n+\t\t zero_hash_eq(a->update.old_hash, b->update.old_hash,\n+\t\t\t\t hash_size) &&\n+\t\t zero_hash_eq(a->update.new_hash, b->update.new_hash,\n+\t\t\t\t hash_size);\n+\t}\n+\n+\tabort();\n+}\n+\n+static int reftable_log_record_is_deletion_void(const void *p)\n+{\n+\treturn reftable_log_record_is_deletion(\n+\t\t(const struct reftable_log_record *)p);\n+}\n+\n+static struct reftable_record_vtable reftable_log_record_vtable = {\n+\t.key = &reftable_log_record_key,\n+\t.type = BLOCK_TYPE_LOG,\n+\t.copy_from = &reftable_log_record_copy_from,\n+\t.val_type = &reftable_log_record_val_type,\n+\t.encode = &reftable_log_record_encode,\n+\t.decode = &reftable_log_record_decode,\n+\t.release = &reftable_log_record_release_void,\n+\t.is_deletion = &reftable_log_record_is_deletion_void,\n+};\n+\n+struct reftable_record reftable_new_record(uint8_t typ)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF: {\n+\t\tstruct reftable_ref_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_ref_record));\n+\t\treftable_record_from_ref(&rec, r);\n+\t\treturn rec;\n+\t}\n+\n+\tcase BLOCK_TYPE_OBJ: {\n+\t\tstruct reftable_obj_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_obj_record));\n+\t\treftable_record_from_obj(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_LOG: {\n+\t\tstruct reftable_log_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_log_record));\n+\t\treftable_record_from_log(&rec, r);\n+\t\treturn rec;\n+\t}\n+\tcase BLOCK_TYPE_INDEX: {\n+\t\tstruct reftable_index_record empty = { .last_key =\n+\t\t\t\t\t\t\t STRBUF_INIT };\n+\t\tstruct reftable_index_record *r =\n+\t\t\treftable_calloc(sizeof(struct reftable_index_record));\n+\t\t*r = empty;\n+\t\treftable_record_from_index(&rec, r);\n+\t\treturn rec;\n+\t}\n+\t}\n+\tabort();\n+\treturn rec;\n+}\n+\n+/* clear out the record, yielding the reftable_record data that was\n+ * encapsulated. */\n+static void *reftable_record_yield(struct reftable_record *rec)\n+{\n+\tvoid *p = rec->data;\n+\trec->data = NULL;\n+\treturn p;\n+}\n+\n+void reftable_record_destroy(struct reftable_record *rec)\n+{\n+\treftable_record_release(rec);\n+\treftable_free(reftable_record_yield(rec));\n+}\n+\n+static void reftable_index_record_key(const void *r, struct strbuf *dest)\n+{\n+\tconst struct reftable_index_record *rec = r;\n+\tstrbuf_reset(dest);\n+\tstrbuf_addbuf(dest, &rec->last_key);\n+}\n+\n+static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n+{\n+\tstruct reftable_index_record *dst = rec;\n+\tconst struct reftable_index_record *src = src_rec;\n+\n+\tstrbuf_reset(&dst->last_key);\n+\tstrbuf_addbuf(&dst->last_key, &src->last_key);\n+\tdst->offset = src->offset;\n+}\n+\n+static void reftable_index_record_release(void *rec)\n+{\n+\tstruct reftable_index_record *idx = rec;\n+\tstrbuf_release(&idx->last_key);\n+}\n+\n+static uint8_t reftable_index_record_val_type(const void *rec)\n+{\n+\treturn 0;\n+}\n+\n+static int reftable_index_record_encode(const void *rec, struct string_view out,\n+\t\t\t\t\tint hash_size)\n+{\n+\tconst struct reftable_index_record *r =\n+\t\t(const struct reftable_index_record *)rec;\n+\tstruct string_view start = out;\n+\n+\tint n = put_var_int(&out, r->offset);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&out, n);\n+\n+\treturn start.len - out.len;\n+}\n+\n+static int reftable_index_record_decode(void *rec, struct strbuf key,\n+\t\t\t\t\tuint8_t val_type, struct string_view in,\n+\t\t\t\t\tint hash_size)\n+{\n+\tstruct string_view start = in;\n+\tstruct reftable_index_record *r = rec;\n+\tint n = 0;\n+\n+\tstrbuf_reset(&r->last_key);\n+\tstrbuf_addbuf(&r->last_key, &key);\n+\n+\tn = get_var_int(&r->offset, &in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\tstring_view_consume(&in, n);\n+\treturn start.len - in.len;\n+}\n+\n+static struct reftable_record_vtable reftable_index_record_vtable = {\n+\t.key = &reftable_index_record_key,\n+\t.type = BLOCK_TYPE_INDEX,\n+\t.copy_from = &reftable_index_record_copy_from,\n+\t.val_type = &reftable_index_record_val_type,\n+\t.encode = &reftable_index_record_encode,\n+\t.decode = &reftable_index_record_decode,\n+\t.release = &reftable_index_record_release,\n+\t.is_deletion = ¬_a_deletion,\n+};\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)\n+{\n+\trec->ops->key(rec->data, dest);\n+}\n+\n+uint8_t reftable_record_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->type;\n+}\n+\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size)\n+{\n+\treturn rec->ops->encode(rec->data, dest, hash_size);\n+}\n+\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size)\n+{\n+\tassert(src->ops->type == rec->ops->type);\n+\n+\trec->ops->copy_from(rec->data, src->data, hash_size);\n+}\n+\n+uint8_t reftable_record_val_type(struct reftable_record *rec)\n+{\n+\treturn rec->ops->val_type(rec->data);\n+}\n+\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src, int hash_size)\n+{\n+\treturn rec->ops->decode(rec->data, key, extra, src, hash_size);\n+}\n+\n+void reftable_record_release(struct reftable_record *rec)\n+{\n+\trec->ops->release(rec->data);\n+}\n+\n+int reftable_record_is_deletion(struct reftable_record *rec)\n+{\n+\treturn rec->ops->is_deletion(rec->data);\n+}\n+\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *ref_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = ref_rec;\n+\trec->ops = &reftable_ref_record_vtable;\n+}\n+\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *obj_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = obj_rec;\n+\trec->ops = &reftable_obj_record_vtable;\n+}\n+\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *index_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = index_rec;\n+\trec->ops = &reftable_index_record_vtable;\n+}\n+\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *log_rec)\n+{\n+\tassert(!rec->ops);\n+\trec->data = log_rec;\n+\trec->ops = &reftable_log_record_vtable;\n+}\n+\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_REF);\n+\treturn rec->data;\n+}\n+\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)\n+{\n+\tassert(reftable_record_type(rec) == BLOCK_TYPE_LOG);\n+\treturn rec->data;\n+}\n+\n+static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)\n+{\n+\tif (a && b)\n+\t\treturn !memcmp(a, b, hash_size);\n+\n+\treturn a == b;\n+}\n+\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size)\n+{\n+\tassert(hash_size > 0);\n+\tif (!(0 == strcmp(a->refname, b->refname) &&\n+\t a->update_index == b->update_index &&\n+\t a->value_type == b->value_type))\n+\t\treturn 0;\n+\n+\tswitch (a->value_type) {\n+\tcase REFTABLE_REF_SYMREF:\n+\t\treturn !strcmp(a->value.symref, b->value.symref);\n+\tcase REFTABLE_REF_VAL2:\n+\t\treturn hash_equal(a->value.val2.value, b->value.val2.value,\n+\t\t\t\t hash_size) &&\n+\t\t hash_equal(a->value.val2.target_value,\n+\t\t\t\t b->value.val2.target_value, hash_size);\n+\tcase REFTABLE_REF_VAL1:\n+\t\treturn hash_equal(a->value.val1, b->value.val1, hash_size);\n+\tcase REFTABLE_REF_DELETION:\n+\t\treturn 1;\n+\tdefault:\n+\t\tabort();\n+\t}\n+}\n+\n+int reftable_ref_record_compare_name(const void *a, const void *b)\n+{\n+\treturn strcmp(((struct reftable_ref_record *)a)->refname,\n+\t\t ((struct reftable_ref_record *)b)->refname);\n+}\n+\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref)\n+{\n+\treturn ref->value_type == REFTABLE_REF_DELETION;\n+}\n+\n+int reftable_log_record_compare_key(const void *a, const void *b)\n+{\n+\tconst struct reftable_log_record *la = a;\n+\tconst struct reftable_log_record *lb = b;\n+\n+\tint cmp = strcmp(la->refname, lb->refname);\n+\tif (cmp)\n+\t\treturn cmp;\n+\tif (la->update_index > lb->update_index)\n+\t\treturn -1;\n+\treturn (la->update_index < lb->update_index) ? 1 : 0;\n+}\n+\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log)\n+{\n+\treturn (log->value_type == REFTABLE_LOG_DELETION);\n+}\n+\n+void string_view_consume(struct string_view *s, int n)\n+{\n+\ts->buf += n;\n+\ts->len -= n;\n+}\ndiff --git a/reftable/record.h b/reftable/record.h\nnew file mode 100644\nindex 000000000000..498e8c50bf4f\n--- /dev/null\n+++ b/reftable/record.h\n@@ -0,0 +1,139 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef RECORD_H\n+#define RECORD_H\n+\n+#include \"system.h\"\n+\n+#include \n+\n+#include \"reftable-record.h\"\n+\n+/*\n+ * A substring of existing string data. This structure takes no responsibility\n+ * for the lifetime of the data it points to.\n+ */\n+struct string_view {\n+\tuint8_t *buf;\n+\tsize_t len;\n+};\n+\n+/* Advance `s.buf` by `n`, and decrease length. */\n+void string_view_consume(struct string_view *s, int n);\n+\n+/* utilities for de/encoding varints */\n+\n+int get_var_int(uint64_t *dest, struct string_view *in);\n+int put_var_int(struct string_view *dest, uint64_t val);\n+\n+/* Methods for records. */\n+struct reftable_record_vtable {\n+\t/* encode the key of to a uint8_t strbuf. */\n+\tvoid (*key)(const void *rec, struct strbuf *dest);\n+\n+\t/* The record type of ('r' for ref). */\n+\tuint8_t type;\n+\n+\tvoid (*copy_from)(void *dest, const void *src, int hash_size);\n+\n+\t/* a value of [0..7], indicating record subvariants (eg. ref vs. symref\n+\t * vs ref deletion) */\n+\tuint8_t (*val_type)(const void *rec);\n+\n+\t/* encodes rec into dest, returning how much space was used. */\n+\tint (*encode)(const void *rec, struct string_view dest, int hash_size);\n+\n+\t/* decode data from `src` into the record. */\n+\tint (*decode)(void *rec, struct strbuf key, uint8_t extra,\n+\t\t struct string_view src, int hash_size);\n+\n+\t/* deallocate and null the record. */\n+\tvoid (*release)(void *rec);\n+\n+\t/* is this a tombstone? */\n+\tint (*is_deletion)(const void *rec);\n+};\n+\n+/* record is a generic wrapper for different types of records. */\n+struct reftable_record {\n+\tvoid *data;\n+\tstruct reftable_record_vtable *ops;\n+};\n+\n+/* returns true for recognized block types. Block start with the block type. */\n+int reftable_is_block_type(uint8_t typ);\n+\n+/* creates a malloced record of the given type. Dispose with record_destroy */\n+struct reftable_record reftable_new_record(uint8_t typ);\n+\n+/* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns\n+ * number of bytes written. */\n+int reftable_encode_key(int *is_restart, struct string_view dest,\n+\t\t\tstruct strbuf prev_key, struct strbuf key,\n+\t\t\tuint8_t extra);\n+\n+/* Decode into `key` and `extra` from `in` */\n+int reftable_decode_key(struct strbuf *key, uint8_t *extra,\n+\t\t\tstruct strbuf last_key, struct string_view in);\n+\n+/* reftable_index_record are used internally to speed up lookups. */\n+struct reftable_index_record {\n+\tuint64_t offset; /* Offset of block */\n+\tstruct strbuf last_key; /* Last key of the block. */\n+};\n+\n+/* reftable_obj_record stores an object ID => ref mapping. */\n+struct reftable_obj_record {\n+\tuint8_t *hash_prefix; /* leading bytes of the object ID */\n+\tint hash_prefix_len; /* number of leading bytes. Constant\n+\t\t\t * across a single table. */\n+\tuint64_t *offsets; /* a vector of file offsets. */\n+\tint offset_len;\n+};\n+\n+/* see struct record_vtable */\n+\n+void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);\n+uint8_t reftable_record_type(struct reftable_record *rec);\n+void reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size);\n+uint8_t reftable_record_val_type(struct reftable_record *rec);\n+int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n+\t\t\t int hash_size);\n+int reftable_record_decode(struct reftable_record *rec, struct strbuf key,\n+\t\t\t uint8_t extra, struct string_view src,\n+\t\t\t int hash_size);\n+int reftable_record_is_deletion(struct reftable_record *rec);\n+\n+/* zeroes out the embedded record */\n+void reftable_record_release(struct reftable_record *rec);\n+\n+/* clear and deallocate embedded record, and zero `rec`. */\n+void reftable_record_destroy(struct reftable_record *rec);\n+\n+/* initialize generic records from concrete records. The generic record should\n+ * be zeroed out. */\n+void reftable_record_from_obj(struct reftable_record *rec,\n+\t\t\t struct reftable_obj_record *objrec);\n+void reftable_record_from_index(struct reftable_record *rec,\n+\t\t\t\tstruct reftable_index_record *idxrec);\n+void reftable_record_from_ref(struct reftable_record *rec,\n+\t\t\t struct reftable_ref_record *refrec);\n+void reftable_record_from_log(struct reftable_record *rec,\n+\t\t\t struct reftable_log_record *logrec);\n+struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *ref);\n+struct reftable_log_record *reftable_record_as_log(struct reftable_record *ref);\n+\n+/* for qsort. */\n+int reftable_ref_record_compare_name(const void *a, const void *b);\n+\n+/* for qsort. */\n+int reftable_log_record_compare_key(const void *a, const void *b);\n+\n+#endif\ndiff --git a/reftable/record_test.c b/reftable/record_test.c\nnew file mode 100644\nindex 000000000000..0a1901897641\n--- /dev/null\n+++ b/reftable/record_test.c\n@@ -0,0 +1,407 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"record.h\"\n+\n+#include \"system.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_copy(struct reftable_record *rec)\n+{\n+\tstruct reftable_record copy =\n+\t\treftable_new_record(reftable_record_type(rec));\n+\treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n+\t/* do it twice to catch memory leaks */\n+\treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n+\tswitch (reftable_record_type(©)) {\n+\tcase BLOCK_TYPE_REF:\n+\t\tEXPECT(reftable_ref_record_equal(reftable_record_as_ref(©),\n+\t\t\t\t\t\t reftable_record_as_ref(rec),\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\tbreak;\n+\tcase BLOCK_TYPE_LOG:\n+\t\tEXPECT(reftable_log_record_equal(reftable_record_as_log(©),\n+\t\t\t\t\t\t reftable_record_as_log(rec),\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\tbreak;\n+\t}\n+\treftable_record_destroy(©);\n+}\n+\n+static void test_varint_roundtrip(void)\n+{\n+\tuint64_t inputs[] = { 0,\n+\t\t\t 1,\n+\t\t\t 27,\n+\t\t\t 127,\n+\t\t\t 128,\n+\t\t\t 257,\n+\t\t\t 4096,\n+\t\t\t ((uint64_t)1 << 63),\n+\t\t\t ((uint64_t)1 << 63) + ((uint64_t)1 << 63) - 1 };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(inputs); i++) {\n+\t\tuint8_t dest[10];\n+\n+\t\tstruct string_view out = {\n+\t\t\t.buf = dest,\n+\t\t\t.len = sizeof(dest),\n+\t\t};\n+\t\tuint64_t in = inputs[i];\n+\t\tint n = put_var_int(&out, in);\n+\t\tuint64_t got = 0;\n+\n+\t\tEXPECT(n > 0);\n+\t\tout.len = n;\n+\t\tn = get_var_int(&got, &out);\n+\t\tEXPECT(n > 0);\n+\n+\t\tEXPECT(got == in);\n+\t}\n+}\n+\n+static void test_common_prefix(void)\n+{\n+\tstruct {\n+\t\tconst char *a, *b;\n+\t\tint want;\n+\t} cases[] = {\n+\t\t{ \"abc\", \"ab\", 2 },\n+\t\t{ \"\", \"abc\", 0 },\n+\t\t{ \"abc\", \"abd\", 2 },\n+\t\t{ \"abc\", \"pqr\", 0 },\n+\t};\n+\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct strbuf a = STRBUF_INIT;\n+\t\tstruct strbuf b = STRBUF_INIT;\n+\t\tstrbuf_addstr(&a, cases[i].a);\n+\t\tstrbuf_addstr(&b, cases[i].b);\n+\t\tEXPECT(common_prefix_size(&a, &b) == cases[i].want);\n+\n+\t\tstrbuf_release(&a);\n+\t\tstrbuf_release(&b);\n+\t}\n+}\n+\n+static void set_hash(uint8_t *h, int j)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < hash_size(GIT_SHA1_HASH_ID); i++) {\n+\t\th[i] = (j >> i) & 0xff;\n+\t}\n+}\n+\n+static void test_reftable_ref_record_roundtrip(void)\n+{\n+\tint i = 0;\n+\n+\tfor (i = REFTABLE_REF_DELETION; i < REFTABLE_NR_REF_VALUETYPES; i++) {\n+\t\tstruct reftable_ref_record in = { NULL };\n+\t\tstruct reftable_ref_record out = { NULL };\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\n+\t\tint n, m;\n+\n+\t\tin.value_type = i;\n+\t\tswitch (i) {\n+\t\tcase REFTABLE_REF_DELETION:\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\tin.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_hash(in.value.val1, 1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\tin.value.val2.value = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_hash(in.value.val2.value, 1);\n+\t\t\tin.value.val2.target_value =\n+\t\t\t\treftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_hash(in.value.val2.target_value, 2);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF:\n+\t\t\tin.value.symref = xstrdup(\"target\");\n+\t\t\tbreak;\n+\t\t}\n+\t\tin.refname = xstrdup(\"refs/heads/master\");\n+\n+\t\treftable_record_from_ref(&rec, &in);\n+\t\ttest_copy(&rec);\n+\n+\t\tEXPECT(reftable_record_val_type(&rec) == i);\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n > 0);\n+\n+\t\t/* decode into a non-zero reftable_record to test for leaks. */\n+\n+\t\treftable_record_from_ref(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, i, dest,\n+\t\t\t\t\t GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(reftable_ref_record_equal(&in, &out, GIT_SHA1_RAWSZ));\n+\t\treftable_record_release(&rec_out);\n+\n+\t\tstrbuf_release(&key);\n+\t\treftable_ref_record_release(&in);\n+\t}\n+}\n+\n+static void test_reftable_log_record_equal(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 42,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t}\n+\t};\n+\n+\tEXPECT(!reftable_log_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tin[1].update_index = in[0].update_index;\n+\tEXPECT(reftable_log_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\treftable_log_record_release(&in[0]);\n+\treftable_log_record_release(&in[1]);\n+}\n+\n+static void test_reftable_log_record_roundtrip(void)\n+{\n+\tstruct reftable_log_record in[2] = {\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 42,\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.update = {\n+\t\t\t\t.old_hash = reftable_malloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t.new_hash = reftable_malloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t.name = xstrdup(\"han-wen\"),\n+\t\t\t\t.email = xstrdup(\"hanwen@google.com\"),\n+\t\t\t\t.message = xstrdup(\"test\"),\n+\t\t\t\t.time = 1577123507,\n+\t\t\t\t.tz_offset = 100,\n+\t\t\t}\n+\t\t},\n+\t\t{\n+\t\t\t.refname = xstrdup(\"refs/heads/master\"),\n+\t\t\t.update_index = 22,\n+\t\t\t.value_type = REFTABLE_LOG_DELETION,\n+\t\t}\n+\t};\n+\tset_test_hash(in[0].update.new_hash, 1);\n+\tset_test_hash(in[0].update.old_hash, 2);\n+\tfor (int i = 0; i < ARRAY_SIZE(in); i++) {\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\t/* populate out, to check for leaks. */\n+\t\tstruct reftable_log_record out = {\n+\t\t\t.refname = xstrdup(\"old name\"),\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.update = {\n+\t\t\t\t.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t.name = xstrdup(\"old name\"),\n+\t\t\t\t.email = xstrdup(\"old@email\"),\n+\t\t\t\t.message = xstrdup(\"old message\"),\n+\t\t\t},\n+\t\t};\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m, valtype;\n+\n+\t\treftable_record_from_log(&rec, &in[i]);\n+\n+\t\ttest_copy(&rec);\n+\n+\t\treftable_record_key(&rec, &key);\n+\n+\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n >= 0);\n+\t\treftable_record_from_log(&rec_out, &out);\n+\t\tvaltype = reftable_record_val_type(&rec);\n+\t\tm = reftable_record_decode(&rec_out, key, valtype, dest,\n+\t\t\t\t\t GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(reftable_log_record_equal(&in[i], &out, GIT_SHA1_RAWSZ));\n+\t\treftable_log_record_release(&in[i]);\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_release(&rec_out);\n+\t}\n+}\n+\n+static void test_u24_roundtrip(void)\n+{\n+\tuint32_t in = 0x112233;\n+\tuint8_t dest[3];\n+\tuint32_t out;\n+\tput_be24(dest, in);\n+\tout = get_be24(dest);\n+\tEXPECT(in == out);\n+}\n+\n+static void test_key_roundtrip(void)\n+{\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct strbuf roundtrip = STRBUF_INIT;\n+\tint restart;\n+\tuint8_t extra;\n+\tint n, m;\n+\tuint8_t rt_extra;\n+\n+\tstrbuf_addstr(&last_key, \"refs/heads/master\");\n+\tstrbuf_addstr(&key, \"refs/tags/bla\");\n+\textra = 6;\n+\tn = reftable_encode_key(&restart, dest, last_key, key, extra);\n+\tEXPECT(!restart);\n+\tEXPECT(n > 0);\n+\n+\tm = reftable_decode_key(&roundtrip, &rt_extra, last_key, dest);\n+\tEXPECT(n == m);\n+\tEXPECT(0 == strbuf_cmp(&key, &roundtrip));\n+\tEXPECT(rt_extra == extra);\n+\n+\tstrbuf_release(&last_key);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&roundtrip);\n+}\n+\n+static void test_reftable_obj_record_roundtrip(void)\n+{\n+\tuint8_t testHash1[GIT_SHA1_RAWSZ] = { 1, 2, 3, 4, 0 };\n+\tuint64_t till9[] = { 1, 2, 3, 4, 500, 600, 700, 800, 9000 };\n+\tstruct reftable_obj_record recs[3] = { {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 3,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t\t .offsets = till9,\n+\t\t\t\t\t\t .offset_len = 9,\n+\t\t\t\t\t },\n+\t\t\t\t\t {\n+\t\t\t\t\t\t .hash_prefix = testHash1,\n+\t\t\t\t\t\t .hash_prefix_len = 5,\n+\t\t\t\t\t } };\n+\tint i = 0;\n+\tfor (i = 0; i < ARRAY_SIZE(recs); i++) {\n+\t\tstruct reftable_obj_record in = recs[i];\n+\t\tuint8_t buffer[1024] = { 0 };\n+\t\tstruct string_view dest = {\n+\t\t\t.buf = buffer,\n+\t\t\t.len = sizeof(buffer),\n+\t\t};\n+\t\tstruct reftable_record rec = { NULL };\n+\t\tstruct strbuf key = STRBUF_INIT;\n+\t\tstruct reftable_obj_record out = { NULL };\n+\t\tstruct reftable_record rec_out = { NULL };\n+\t\tint n, m;\n+\t\tuint8_t extra;\n+\n+\t\treftable_record_from_obj(&rec, &in);\n+\t\ttest_copy(&rec);\n+\t\treftable_record_key(&rec, &key);\n+\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n > 0);\n+\t\textra = reftable_record_val_type(&rec);\n+\t\treftable_record_from_obj(&rec_out, &out);\n+\t\tm = reftable_record_decode(&rec_out, key, extra, dest,\n+\t\t\t\t\t GIT_SHA1_RAWSZ);\n+\t\tEXPECT(n == m);\n+\n+\t\tEXPECT(in.hash_prefix_len == out.hash_prefix_len);\n+\t\tEXPECT(in.offset_len == out.offset_len);\n+\n+\t\tEXPECT(!memcmp(in.hash_prefix, out.hash_prefix,\n+\t\t\t in.hash_prefix_len));\n+\t\tEXPECT(0 == memcmp(in.offsets, out.offsets,\n+\t\t\t\t sizeof(uint64_t) * in.offset_len));\n+\t\tstrbuf_release(&key);\n+\t\treftable_record_release(&rec_out);\n+\t}\n+}\n+\n+static void test_reftable_index_record_roundtrip(void)\n+{\n+\tstruct reftable_index_record in = {\n+\t\t.offset = 42,\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tuint8_t buffer[1024] = { 0 };\n+\tstruct string_view dest = {\n+\t\t.buf = buffer,\n+\t\t.len = sizeof(buffer),\n+\t};\n+\tstruct strbuf key = STRBUF_INIT;\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_index_record out = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record out_rec = { NULL };\n+\tint n, m;\n+\tuint8_t extra;\n+\n+\tstrbuf_addstr(&in.last_key, \"refs/heads/master\");\n+\treftable_record_from_index(&rec, &in);\n+\treftable_record_key(&rec, &key);\n+\ttest_copy(&rec);\n+\n+\tEXPECT(0 == strbuf_cmp(&key, &in.last_key));\n+\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\tEXPECT(n > 0);\n+\n+\textra = reftable_record_val_type(&rec);\n+\treftable_record_from_index(&out_rec, &out);\n+\tm = reftable_record_decode(&out_rec, key, extra, dest, GIT_SHA1_RAWSZ);\n+\tEXPECT(m == n);\n+\n+\tEXPECT(in.offset == out.offset);\n+\n+\treftable_record_release(&out_rec);\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&in.last_key);\n+}\n+\n+int record_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_reftable_log_record_equal);\n+\tRUN_TEST(test_reftable_log_record_roundtrip);\n+\tRUN_TEST(test_reftable_ref_record_roundtrip);\n+\tRUN_TEST(test_varint_roundtrip);\n+\tRUN_TEST(test_key_roundtrip);\n+\tRUN_TEST(test_common_prefix);\n+\tRUN_TEST(test_reftable_obj_record_roundtrip);\n+\tRUN_TEST(test_reftable_index_record_roundtrip);\n+\tRUN_TEST(test_u24_roundtrip);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-record.h b/reftable/reftable-record.h\nnew file mode 100644\nindex 000000000000..7985b94ae2cc\n--- /dev/null\n+++ b/reftable/reftable-record.h\n@@ -0,0 +1,114 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_RECORD_H\n+#define REFTABLE_RECORD_H\n+\n+#include \n+\n+/*\n+ * Basic data types\n+ *\n+ * Reftables store the state of each ref in struct reftable_ref_record, and they\n+ * store a sequence of reflog updates in struct reftable_log_record.\n+ */\n+\n+/* reftable_ref_record holds a ref database entry target_value */\n+struct reftable_ref_record {\n+\tchar *refname; /* Name of the ref, malloced. */\n+\tuint64_t update_index; /* Logical timestamp at which this value is\n+\t\t\t\t* written */\n+\n+\tenum {\n+\t\t/* tombstone to hide deletions from earlier tables */\n+\t\tREFTABLE_REF_DELETION = 0x0,\n+\n+\t\t/* a simple ref */\n+\t\tREFTABLE_REF_VAL1 = 0x1,\n+\t\t/* a tag, plus its peeled hash */\n+\t\tREFTABLE_REF_VAL2 = 0x2,\n+\n+\t\t/* a symbolic reference */\n+\t\tREFTABLE_REF_SYMREF = 0x3,\n+#define REFTABLE_NR_REF_VALUETYPES 4\n+\t} value_type;\n+\tunion {\n+\t\tuint8_t *val1; /* malloced hash. */\n+\t\tstruct {\n+\t\t\tuint8_t *value; /* first value, malloced hash */\n+\t\t\tuint8_t *target_value; /* second value, malloced hash */\n+\t\t} val2;\n+\t\tchar *symref; /* referent, malloced 0-terminated string */\n+\t} value;\n+};\n+\n+/* Returns the first hash, or NULL if `rec` is not of type\n+ * REFTABLE_REF_VAL1 or REFTABLE_REF_VAL2. */\n+uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec);\n+\n+/* Returns the second hash, or NULL if `rec` is not of type\n+ * REFTABLE_REF_VAL2. */\n+uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec);\n+\n+/* returns whether 'ref' represents a deletion */\n+int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);\n+\n+/* prints a reftable_ref_record onto stdout. Useful for debugging. */\n+void reftable_ref_record_print(struct reftable_ref_record *ref,\n+\t\t\t uint32_t hash_id);\n+\n+/* frees and nulls all pointer values inside `ref`. */\n+void reftable_ref_record_release(struct reftable_ref_record *ref);\n+\n+/* returns whether two reftable_ref_records are the same. Useful for testing. */\n+int reftable_ref_record_equal(struct reftable_ref_record *a,\n+\t\t\t struct reftable_ref_record *b, int hash_size);\n+\n+/* reftable_log_record holds a reflog entry */\n+struct reftable_log_record {\n+\tchar *refname;\n+\tuint64_t update_index; /* logical timestamp of a transactional update.\n+\t\t\t\t*/\n+\n+\tenum {\n+\t\t/* tombstone to hide deletions from earlier tables */\n+\t\tREFTABLE_LOG_DELETION = 0x0,\n+\n+\t\t/* a simple update */\n+\t\tREFTABLE_LOG_UPDATE = 0x1,\n+#define REFTABLE_NR_LOG_VALUETYPES 2\n+\t} value_type;\n+\n+\tunion {\n+\t\tstruct {\n+\t\t\tuint8_t *new_hash;\n+\t\t\tuint8_t *old_hash;\n+\t\t\tchar *name;\n+\t\t\tchar *email;\n+\t\t\tuint64_t time;\n+\t\t\tint16_t tz_offset;\n+\t\t\tchar *message;\n+\t\t} update;\n+\t};\n+};\n+\n+/* returns whether 'ref' represents the deletion of a log record. */\n+int reftable_log_record_is_deletion(const struct reftable_log_record *log);\n+\n+/* frees and nulls all pointer values. */\n+void reftable_log_record_release(struct reftable_log_record *log);\n+\n+/* returns whether two records are equal. Useful for testing. */\n+int reftable_log_record_equal(struct reftable_log_record *a,\n+\t\t\t struct reftable_log_record *b, int hash_size);\n+\n+/* dumps a reftable_log_record on stdout, for debugging/testing. */\n+void reftable_log_record_print(struct reftable_log_record *log,\n+\t\t\t uint32_t hash_id);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 3b58e423e7b1..09d4b83ef9b8 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,6 +4,6 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n-\n+\trecord_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"422279","messageId":"425cd0185aabf454a69d9b9d6cb51b025e105107.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 12/28] reftable: reading/writing blocks","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:40Z","receivedAt":"2021-04-19T11:40:22Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format is structured as a sequence of block. Within a block,\nrecords are prefix compressed, with an index of offsets for fully expand keys to\nenable binary search within blocks.\n\nThis commit provides the logic to read and write these blocks.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/block.c | 446 +++++++++++++++++++++++++++++++++++++++\n reftable/block.h | 127 +++++++++++\n reftable/block_test.c | 121 +++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 697 insertions(+)\n create mode 100644 reftable/block.c\n create mode 100644 reftable/block.h\n create mode 100644 reftable/block_test.c\n\ndiff --git a/Makefile b/Makefile\nindex e6df765d3361..cd8a9de2519e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2410,10 +2410,12 @@ xdiff-objs: $(XDIFF_OBJS)\n \n REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n+REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n \n+REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\ndiff --git a/reftable/block.c b/reftable/block.c\nnew file mode 100644\nindex 000000000000..92f8e5abfad7\n--- /dev/null\n+++ b/reftable/block.c\n@@ -0,0 +1,446 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"reftable-error.h\"\n+#include \"system.h\"\n+#include \n+\n+#ifdef NO_UNCOMPRESS2\n+/* This is uncompress2, which is only available in zlib as of 2017.\n+ */\n+int uncompress2(Bytef *dest, uLongf *destLen, const Bytef *source,\n+\t\tuLong *sourceLen);\n+#endif\n+\n+int header_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 24;\n+\tcase 2:\n+\t\treturn 28;\n+\t}\n+\tabort();\n+}\n+\n+int footer_size(int version)\n+{\n+\tswitch (version) {\n+\tcase 1:\n+\t\treturn 68;\n+\tcase 2:\n+\t\treturn 72;\n+\t}\n+\tabort();\n+}\n+\n+static int block_writer_register_restart(struct block_writer *w, int n,\n+\t\t\t\t\t int is_restart, struct strbuf *key)\n+{\n+\tint rlen = w->restart_len;\n+\tif (rlen >= MAX_RESTARTS) {\n+\t\tis_restart = 0;\n+\t}\n+\n+\tif (is_restart) {\n+\t\trlen++;\n+\t}\n+\tif (2 + 3 * rlen + n > w->block_size - w->next)\n+\t\treturn -1;\n+\tif (is_restart) {\n+\t\tif (w->restart_len == w->restart_cap) {\n+\t\t\tw->restart_cap = w->restart_cap * 2 + 1;\n+\t\t\tw->restarts = reftable_realloc(\n+\t\t\t\tw->restarts, sizeof(uint32_t) * w->restart_cap);\n+\t\t}\n+\n+\t\tw->restarts[w->restart_len++] = w->next;\n+\t}\n+\n+\tw->next += n;\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, key);\n+\tw->entries++;\n+\treturn 0;\n+}\n+\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n+{\n+\tbw->buf = buf;\n+\tbw->hash_size = hash_size;\n+\tbw->block_size = block_size;\n+\tbw->header_off = header_off;\n+\tbw->buf[header_off] = typ;\n+\tbw->next = header_off + 4;\n+\tbw->restart_interval = 16;\n+\tbw->entries = 0;\n+\tbw->restart_len = 0;\n+\tbw->last_key.len = 0;\n+}\n+\n+uint8_t block_writer_type(struct block_writer *bw)\n+{\n+\treturn bw->buf[bw->header_off];\n+}\n+\n+/* adds the reftable_record to the block. Returns -1 if it does not fit, 0 on\n+ success */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tstruct strbuf last =\n+\t\tw->entries % w->restart_interval == 0 ? empty : w->last_key;\n+\tstruct string_view out = {\n+\t\t.buf = w->buf + w->next,\n+\t\t.len = w->block_size - w->next,\n+\t};\n+\n+\tstruct string_view start = out;\n+\n+\tint is_restart = 0;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint n = 0;\n+\n+\treftable_record_key(rec, &key);\n+\tn = reftable_encode_key(&is_restart, out, last, key,\n+\t\t\t\treftable_record_val_type(rec));\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tn = reftable_record_encode(rec, out, w->hash_size);\n+\tif (n < 0)\n+\t\tgoto done;\n+\tstring_view_consume(&out, n);\n+\n+\tif (block_writer_register_restart(w, start.len - out.len, is_restart,\n+\t\t\t\t\t &key) < 0)\n+\t\tgoto done;\n+\n+\tstrbuf_release(&key);\n+\treturn 0;\n+\n+done:\n+\tstrbuf_release(&key);\n+\treturn -1;\n+}\n+\n+int block_writer_finish(struct block_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->restart_len; i++) {\n+\t\tput_be24(w->buf + w->next, w->restarts[i]);\n+\t\tw->next += 3;\n+\t}\n+\n+\tput_be16(w->buf + w->next, w->restart_len);\n+\tw->next += 2;\n+\tput_be24(w->buf + 1 + w->header_off, w->next);\n+\n+\tif (block_writer_type(w) == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + w->header_off;\n+\t\tuint8_t *compressed = NULL;\n+\t\tint zresult = 0;\n+\t\tuLongf src_len = w->next - block_header_skip;\n+\t\tsize_t dest_cap = src_len;\n+\n+\t\tcompressed = reftable_malloc(dest_cap);\n+\t\twhile (1) {\n+\t\t\tuLongf out_dest_len = dest_cap;\n+\n+\t\t\tzresult = compress2(compressed, &out_dest_len,\n+\t\t\t\t\t w->buf + block_header_skip, src_len,\n+\t\t\t\t\t 9);\n+\t\t\tif (zresult == Z_BUF_ERROR) {\n+\t\t\t\tdest_cap *= 2;\n+\t\t\t\tcompressed =\n+\t\t\t\t\treftable_realloc(compressed, dest_cap);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (Z_OK != zresult) {\n+\t\t\t\treftable_free(compressed);\n+\t\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t\t}\n+\n+\t\t\tmemcpy(w->buf + block_header_skip, compressed,\n+\t\t\t out_dest_len);\n+\t\t\tw->next = out_dest_len + block_header_skip;\n+\t\t\treftable_free(compressed);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\treturn w->next;\n+}\n+\n+uint8_t block_reader_type(struct block_reader *r)\n+{\n+\treturn r->block.data[r->header_off];\n+}\n+\n+int block_reader_init(struct block_reader *br, struct reftable_block *block,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size)\n+{\n+\tuint32_t full_block_size = table_block_size;\n+\tuint8_t typ = block->data[header_off];\n+\tuint32_t sz = get_be24(block->data + header_off + 1);\n+\n+\tuint16_t restart_count = 0;\n+\tuint32_t restart_start = 0;\n+\tuint8_t *restart_bytes = NULL;\n+\n+\tif (!reftable_is_block_type(typ))\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\tif (typ == BLOCK_TYPE_LOG) {\n+\t\tint block_header_skip = 4 + header_off;\n+\t\tuLongf dst_len = sz - block_header_skip; /* total size of dest\n+\t\t\t\t\t\t\t buffer. */\n+\t\tuLongf src_len = block->len - block_header_skip;\n+\t\t/* Log blocks specify the *uncompressed* size in their header.\n+\t\t */\n+\t\tuint8_t *uncompressed = reftable_malloc(sz);\n+\n+\t\t/* Copy over the block header verbatim. It's not compressed. */\n+\t\tmemcpy(uncompressed, block->data, block_header_skip);\n+\n+\t\t/* Uncompress */\n+\t\tif (Z_OK !=\n+\t\t uncompress2(uncompressed + block_header_skip, &dst_len,\n+\t\t\t\tblock->data + block_header_skip, &src_len)) {\n+\t\t\treftable_free(uncompressed);\n+\t\t\treturn REFTABLE_ZLIB_ERROR;\n+\t\t}\n+\n+\t\tif (dst_len + block_header_skip != sz)\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\n+\t\t/* We're done with the input data. */\n+\t\treftable_block_done(block);\n+\t\tblock->data = uncompressed;\n+\t\tblock->len = sz;\n+\t\tblock->source = malloc_block_source();\n+\t\tfull_block_size = src_len + block_header_skip;\n+\t} else if (full_block_size == 0) {\n+\t\tfull_block_size = sz;\n+\t} else if (sz < full_block_size && sz < block->len &&\n+\t\t block->data[sz] != 0) {\n+\t\t/* If the block is smaller than the full block size, it is\n+\t\t padded (data followed by '\\0') or the next block is\n+\t\t unaligned. */\n+\t\tfull_block_size = sz;\n+\t}\n+\n+\trestart_count = get_be16(block->data + sz - 2);\n+\trestart_start = sz - 2 - 3 * restart_count;\n+\trestart_bytes = block->data + restart_start;\n+\n+\t/* transfer ownership. */\n+\tbr->block = *block;\n+\tblock->data = NULL;\n+\tblock->len = 0;\n+\n+\tbr->hash_size = hash_size;\n+\tbr->block_len = restart_start;\n+\tbr->full_block_size = full_block_size;\n+\tbr->header_off = header_off;\n+\tbr->restart_count = restart_count;\n+\tbr->restart_bytes = restart_bytes;\n+\n+\treturn 0;\n+}\n+\n+static uint32_t block_reader_restart_offset(struct block_reader *br, int i)\n+{\n+\treturn get_be24(br->restart_bytes + 3 * i);\n+}\n+\n+void block_reader_start(struct block_reader *br, struct block_iter *it)\n+{\n+\tit->br = br;\n+\tstrbuf_reset(&it->last_key);\n+\tit->next_off = br->header_off + 4;\n+}\n+\n+struct restart_find_args {\n+\tint error;\n+\tstruct strbuf key;\n+\tstruct block_reader *r;\n+};\n+\n+static int restart_key_less(size_t idx, void *args)\n+{\n+\tstruct restart_find_args *a = args;\n+\tuint32_t off = block_reader_restart_offset(a->r, idx);\n+\tstruct string_view in = {\n+\t\t.buf = a->r->block.data + off,\n+\t\t.len = a->r->block_len - off,\n+\t};\n+\n+\t/* the restart key is verbatim in the block, so this could avoid the\n+\t alloc for decoding the key */\n+\tstruct strbuf rkey = STRBUF_INIT;\n+\tstruct strbuf last_key = STRBUF_INIT;\n+\tuint8_t unused_extra;\n+\tint n = reftable_decode_key(&rkey, &unused_extra, last_key, in);\n+\tint result;\n+\tif (n < 0) {\n+\t\ta->error = 1;\n+\t\treturn -1;\n+\t}\n+\n+\tresult = strbuf_cmp(&a->key, &rkey);\n+\tstrbuf_release(&rkey);\n+\treturn result;\n+}\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src)\n+{\n+\tdest->br = src->br;\n+\tdest->next_off = src->next_off;\n+\tstrbuf_reset(&dest->last_key);\n+\tstrbuf_addbuf(&dest->last_key, &src->last_key);\n+}\n+\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec)\n+{\n+\tstruct string_view in = {\n+\t\t.buf = it->br->block.data + it->next_off,\n+\t\t.len = it->br->block_len - it->next_off,\n+\t};\n+\tstruct string_view start = in;\n+\tstruct strbuf key = STRBUF_INIT;\n+\tuint8_t extra = 0;\n+\tint n = 0;\n+\n+\tif (it->next_off >= it->br->block_len)\n+\t\treturn 1;\n+\n+\tn = reftable_decode_key(&key, &extra, it->last_key, in);\n+\tif (n < 0)\n+\t\treturn -1;\n+\n+\tstring_view_consume(&in, n);\n+\tn = reftable_record_decode(rec, key, extra, in, it->br->hash_size);\n+\tif (n < 0)\n+\t\treturn -1;\n+\tstring_view_consume(&in, n);\n+\n+\tstrbuf_reset(&it->last_key);\n+\tstrbuf_addbuf(&it->last_key, &key);\n+\tit->next_off += start.len - in.len;\n+\tstrbuf_release(&key);\n+\treturn 0;\n+}\n+\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key)\n+{\n+\tstruct strbuf empty = STRBUF_INIT;\n+\tint off = br->header_off + 4;\n+\tstruct string_view in = {\n+\t\t.buf = br->block.data + off,\n+\t\t.len = br->block_len - off,\n+\t};\n+\n+\tuint8_t extra = 0;\n+\tint n = reftable_decode_key(key, &extra, empty, in);\n+\tif (n < 0)\n+\t\treturn n;\n+\n+\treturn 0;\n+}\n+\n+int block_iter_seek(struct block_iter *it, struct strbuf *want)\n+{\n+\treturn block_reader_seek(it->br, it, want);\n+}\n+\n+void block_iter_close(struct block_iter *it)\n+{\n+\tstrbuf_release(&it->last_key);\n+}\n+\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want)\n+{\n+\tstruct restart_find_args args = {\n+\t\t.key = *want,\n+\t\t.r = br,\n+\t};\n+\tstruct reftable_record rec = reftable_new_record(block_reader_type(br));\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = 0;\n+\tstruct block_iter next = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\n+\tint i = binsearch(br->restart_count, &restart_key_less, &args);\n+\tif (args.error) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tit->br = br;\n+\tif (i > 0) {\n+\t\ti--;\n+\t\tit->next_off = block_reader_restart_offset(br, i);\n+\t} else {\n+\t\tit->next_off = br->header_off + 4;\n+\t}\n+\n+\t/* We're looking for the last entry less/equal than the wanted key, so\n+\t we have to go one entry too far and then back up.\n+\t*/\n+\twhile (1) {\n+\t\tblock_iter_copy_from(&next, it);\n+\t\terr = block_iter_next(&next, &rec);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\treftable_record_key(&rec, &key);\n+\t\tif (err > 0 || strbuf_cmp(&key, want) >= 0) {\n+\t\t\terr = 0;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tblock_iter_copy_from(it, &next);\n+\t}\n+\n+done:\n+\tstrbuf_release(&key);\n+\tstrbuf_release(&next.last_key);\n+\treftable_record_destroy(&rec);\n+\n+\treturn err;\n+}\n+\n+void block_writer_release(struct block_writer *bw)\n+{\n+\tFREE_AND_NULL(bw->restarts);\n+\tstrbuf_release(&bw->last_key);\n+\t/* the block is not owned. */\n+}\n+\n+void reftable_block_done(struct reftable_block *blockp)\n+{\n+\tstruct reftable_block_source source = blockp->source;\n+\tif (blockp && source.ops)\n+\t\tsource.ops->return_block(source.arg, blockp);\n+\tblockp->data = NULL;\n+\tblockp->len = 0;\n+\tblockp->source.ops = NULL;\n+\tblockp->source.arg = NULL;\n+}\ndiff --git a/reftable/block.h b/reftable/block.h\nnew file mode 100644\nindex 000000000000..e207706a6448\n--- /dev/null\n+++ b/reftable/block.h\n@@ -0,0 +1,127 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef BLOCK_H\n+#define BLOCK_H\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"reftable-blocksource.h\"\n+\n+/*\n+ * Writes reftable blocks. The block_writer is reused across blocks to minimize\n+ * allocation overhead.\n+ */\n+struct block_writer {\n+\tuint8_t *buf;\n+\tuint32_t block_size;\n+\n+\t/* Offset ofof the global header. Nonzero in the first block only. */\n+\tuint32_t header_off;\n+\n+\t/* How often to restart keys. */\n+\tint restart_interval;\n+\tint hash_size;\n+\n+\t/* Offset of next uint8_t to write. */\n+\tuint32_t next;\n+\tuint32_t *restarts;\n+\tuint32_t restart_len;\n+\tuint32_t restart_cap;\n+\n+\tstruct strbuf last_key;\n+\tint entries;\n+};\n+\n+/*\n+ * initializes the blockwriter to write `typ` entries, using `buf` as temporary\n+ * storage. `buf` is not owned by the block_writer. */\n+void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n+\n+/* returns the block type (eg. 'r' for ref records. */\n+uint8_t block_writer_type(struct block_writer *bw);\n+\n+/* appends the record, or -1 if it doesn't fit. */\n+int block_writer_add(struct block_writer *w, struct reftable_record *rec);\n+\n+/* appends the key restarts, and compress the block if necessary. */\n+int block_writer_finish(struct block_writer *w);\n+\n+/* clears out internally allocated block_writer members. */\n+void block_writer_release(struct block_writer *bw);\n+\n+/* Read a block. */\n+struct block_reader {\n+\t/* offset of the block header; nonzero for the first block in a\n+\t * reftable. */\n+\tuint32_t header_off;\n+\n+\t/* the memory block */\n+\tstruct reftable_block block;\n+\tint hash_size;\n+\n+\t/* size of the data, excluding restart data. */\n+\tuint32_t block_len;\n+\tuint8_t *restart_bytes;\n+\tuint16_t restart_count;\n+\n+\t/* size of the data in the file. For log blocks, this is the compressed\n+\t * size. */\n+\tuint32_t full_block_size;\n+};\n+\n+/* Iterate over entries in a block */\n+struct block_iter {\n+\t/* offset within the block of the next entry to read. */\n+\tuint32_t next_off;\n+\tstruct block_reader *br;\n+\n+\t/* key for last entry we read. */\n+\tstruct strbuf last_key;\n+};\n+\n+/* initializes a block reader. */\n+int block_reader_init(struct block_reader *br, struct reftable_block *bl,\n+\t\t uint32_t header_off, uint32_t table_block_size,\n+\t\t int hash_size);\n+\n+/* Position `it` at start of the block */\n+void block_reader_start(struct block_reader *br, struct block_iter *it);\n+\n+/* Position `it` to the `want` key in the block */\n+int block_reader_seek(struct block_reader *br, struct block_iter *it,\n+\t\t struct strbuf *want);\n+\n+/* Returns the block type (eg. 'r' for refs) */\n+uint8_t block_reader_type(struct block_reader *r);\n+\n+/* Decodes the first key in the block */\n+int block_reader_first_key(struct block_reader *br, struct strbuf *key);\n+\n+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src);\n+\n+/* return < 0 for error, 0 for OK, > 0 for EOF. */\n+int block_iter_next(struct block_iter *it, struct reftable_record *rec);\n+\n+/* Seek to `want` with in the block pointed to by `it` */\n+int block_iter_seek(struct block_iter *it, struct strbuf *want);\n+\n+/* deallocate memory for `it`. The block reader and its block is left intact. */\n+void block_iter_close(struct block_iter *it);\n+\n+/* size of file header, depending on format version */\n+int header_size(int version);\n+\n+/* size of file footer, depending on format version */\n+int footer_size(int version);\n+\n+/* returns a block to its source. */\n+void reftable_block_done(struct reftable_block *ret);\n+\n+#endif\ndiff --git a/reftable/block_test.c b/reftable/block_test.c\nnew file mode 100644\nindex 000000000000..85e53186ef74\n--- /dev/null\n+++ b/reftable/block_test.c\n@@ -0,0 +1,121 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"block.h\"\n+\n+#include \"system.h\"\n+\n+#include \"blocksource.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static void test_block_read_write(void)\n+{\n+\tconst int header_off = 21; /* random */\n+\tchar *names[30];\n+\tconst int N = ARRAY_SIZE(names);\n+\tconst int block_size = 1024;\n+\tstruct reftable_block block = { NULL };\n+\tstruct block_writer bw = {\n+\t\t.last_key = STRBUF_INIT,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_record rec = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tstruct block_reader br = { 0 };\n+\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\tint j = 0;\n+\tstruct strbuf want = STRBUF_INIT;\n+\n+\tblock.data = reftable_calloc(block_size);\n+\tblock.len = block_size;\n+\tblock.source = malloc_block_source();\n+\tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n+\t\t\t header_off, hash_size(GIT_SHA1_HASH_ID));\n+\treftable_record_from_ref(&rec, &ref);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tuint8_t hash[GIT_SHA1_RAWSZ];\n+\t\tsnprintf(name, sizeof(name), \"branch%02d\", i);\n+\t\tmemset(hash, i, sizeof(hash));\n+\n+\t\tref.refname = name;\n+\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\tref.value.val1 = hash;\n+\n+\t\tnames[i] = xstrdup(name);\n+\t\tn = block_writer_add(&bw, &rec);\n+\t\tref.refname = NULL;\n+\t\tref.value_type = REFTABLE_REF_DELETION;\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tn = block_writer_finish(&bw);\n+\tEXPECT(n > 0);\n+\n+\tblock_writer_release(&bw);\n+\n+\tblock_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ);\n+\n+\tblock_reader_start(&br, &it);\n+\n+\twhile (1) {\n+\t\tint r = block_iter_next(&it, &rec);\n+\t\tEXPECT(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tEXPECT_STREQ(names[j], ref.refname);\n+\t\tj++;\n+\t}\n+\n+\treftable_record_release(&rec);\n+\tblock_iter_close(&it);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct block_iter it = { .last_key = STRBUF_INIT };\n+\t\tstrbuf_reset(&want);\n+\t\tstrbuf_addstr(&want, names[i]);\n+\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tEXPECT(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tEXPECT(n == 0);\n+\n+\t\tEXPECT_STREQ(names[i], ref.refname);\n+\n+\t\twant.len--;\n+\t\tn = block_reader_seek(&br, &it, &want);\n+\t\tEXPECT(n == 0);\n+\n+\t\tn = block_iter_next(&it, &rec);\n+\t\tEXPECT(n == 0);\n+\t\tEXPECT_STREQ(names[10 * (i / 10)], ref.refname);\n+\n+\t\tblock_iter_close(&it);\n+\t}\n+\n+\treftable_record_release(&rec);\n+\treftable_block_done(&br.block);\n+\tstrbuf_release(&want);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+}\n+\n+int block_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_block_read_write);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 09d4b83ef9b8..c9deeaf08c7a 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -4,6 +4,7 @@\n int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n+\tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"422280","messageId":"dc04ec88cf5388fc6eb7aa643f4462f57dc909c9.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 17/28] reftable: reftable file level tests","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:45Z","receivedAt":"2021-04-19T11:40:24Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nWith support for reading and writing files in place, we can construct files (in\nmemory) and attempt to read them back.\n\nBecause some sections of the format are optional (eg. indices, log entries), we\nhave to exercise this code using multiple sizes of input data\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/readwrite_test.c | 621 ++++++++++++++++++++++++++++++++++++++\n reftable/reftable-tests.h | 2 +-\n t/helper/test-reftable.c | 1 +\n 4 files changed, 624 insertions(+), 1 deletion(-)\n create mode 100644 reftable/readwrite_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 6593e632ac08..26a805afb5ad 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2425,6 +2425,7 @@ REFTABLE_OBJS += reftable/writer.o\n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n+REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c\nnew file mode 100644\nindex 000000000000..6520447a346d\n--- /dev/null\n+++ b/reftable/readwrite_test.c\n@@ -0,0 +1,621 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-writer.h\"\n+\n+static const int update_index = 5;\n+\n+static void test_buffer(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_block out = { NULL };\n+\tint n;\n+\tuint8_t in[] = \"hello\";\n+\tstrbuf_add(&buf, in, sizeof(in));\n+\tblock_source_from_strbuf(&source, &buf);\n+\tEXPECT(block_source_size(&source) == 6);\n+\tn = block_source_read_block(&source, &out, 0, sizeof(in));\n+\tEXPECT(n == sizeof(in));\n+\tEXPECT(!memcmp(in, out.data, n));\n+\treftable_block_done(&out);\n+\n+\tn = block_source_read_block(&source, &out, 1, 2);\n+\tEXPECT(n == 2);\n+\tEXPECT(!memcmp(out.data, \"el\", 2));\n+\n+\treftable_block_done(&out);\n+\tblock_source_close(&source);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void write_table(char ***names, struct strbuf *buf, int N,\n+\t\t\tint block_size, uint32_t hash_id)\n+{\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = block_size,\n+\t\t.hash_id = hash_id,\n+\t};\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, buf, &opts);\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0, n;\n+\tstruct reftable_log_record log = { NULL };\n+\tconst struct reftable_stats *stats = NULL;\n+\t*names = reftable_calloc(sizeof(char *) * (N + 1));\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[GIT_SHA256_RAWSZ] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tref.refname = name;\n+\t\tref.update_index = update_index;\n+\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\tref.value.val1 = hash;\n+\t\t(*names)[i] = xstrdup(name);\n+\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[GIT_SHA256_RAWSZ] = { 0 };\n+\t\tchar name[100];\n+\t\tint n;\n+\n+\t\tset_test_hash(hash, i);\n+\n+\t\tsnprintf(name, sizeof(name), \"refs/heads/branch%02d\", i);\n+\n+\t\tlog.refname = name;\n+\t\tlog.update_index = update_index;\n+\t\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog.update.new_hash = hash;\n+\t\tlog.update.message = \"message\";\n+\n+\t\tn = reftable_writer_add_log(w, &log);\n+\t\tEXPECT(n == 0);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tfor (i = 0; i < stats->ref_stats.blocks; i++) {\n+\t\tint off = i * opts.block_size;\n+\t\tif (off == 0) {\n+\t\t\toff = header_size((hash_id == GIT_SHA256_HASH_ID) ? 2 :\n+\t\t\t\t\t\t\t\t\t\t 1);\n+\t\t}\n+\t\tEXPECT(buf->buf[off] == 'r');\n+\t}\n+\n+\tEXPECT(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+}\n+\n+static void test_log_buffer_size(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 4096,\n+\t};\n+\tint err;\n+\tstruct reftable_log_record log = { .refname = \"refs/heads/master\",\n+\t\t\t\t\t .update_index = 0xa,\n+\t\t\t\t\t .value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t\t\t .update = {\n+\t\t\t\t\t\t .name = \"Han-Wen Nienhuys\",\n+\t\t\t\t\t\t .email = \"hanwen@google.com\",\n+\t\t\t\t\t\t .tz_offset = 100,\n+\t\t\t\t\t\t .time = 0x5e430672,\n+\t\t\t\t\t\t .message = \"commit: 9\\n\",\n+\t\t\t\t\t } };\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\t/* This tests buffer extension for log compression. Must use a random\n+\t hash, to ensure that the compressed part is larger than the original.\n+\t*/\n+\tuint8_t hash1[GIT_SHA1_RAWSZ], hash2[GIT_SHA1_RAWSZ];\n+\tfor (int i = 0; i < GIT_SHA1_RAWSZ; i++) {\n+\t\thash1[i] = (uint8_t)(rand() % 256);\n+\t\thash2[i] = (uint8_t)(rand() % 256);\n+\t}\n+\tlog.update.old_hash = hash1;\n+\tlog.update.new_hash = hash2;\n+\treftable_writer_set_limits(w, update_index, update_index);\n+\terr = reftable_writer_add_log(w, &log);\n+\tEXPECT_ERR(err);\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_log_write_read(void)\n+{\n+\tint N = 2;\n+\tchar **names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint err;\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\tint n;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tconst struct reftable_stats *stats = NULL;\n+\treftable_writer_set_limits(w, 0, N);\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[256];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tsnprintf(name, sizeof(name), \"b%02d%0*d\", i, 130, 7);\n+\t\tnames[i] = xstrdup(name);\n+\t\tref.refname = name;\n+\t\tref.update_index = i;\n+\n+\t\terr = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash1[GIT_SHA1_RAWSZ], hash2[GIT_SHA1_RAWSZ];\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tset_test_hash(hash1, i);\n+\t\tset_test_hash(hash2, i + 1);\n+\n+\t\tlog.refname = names[i];\n+\t\tlog.update_index = i;\n+\t\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog.update.old_hash = hash1;\n+\t\tlog.update.new_hash = hash2;\n+\n+\t\terr = reftable_writer_add_log(w, &log);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\tstats = writer_stats(w);\n+\tEXPECT(stats->log_stats.blocks > 0);\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.log\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[N - 1]);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT_ERR(err);\n+\n+\t/* end of iteration. */\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT(0 < err);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_release(&ref);\n+\n+\terr = reftable_reader_seek_log(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\ti = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT_STREQ(names[i], log.refname);\n+\t\tEXPECT(i == log.update_index);\n+\t\ti++;\n+\t\treftable_log_record_release(&log);\n+\t}\n+\n+\tEXPECT(i == N);\n+\treftable_iterator_destroy(&it);\n+\n+\t/* cleanup. */\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_sequential(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader rd = { NULL };\n+\tint err = 0;\n+\tint j = 0;\n+\n+\twrite_table(&names, &buf, N, 256, GIT_SHA1_HASH_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint r = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(r >= 0);\n+\t\tif (r > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tEXPECT(0 == strcmp(names[j], ref.refname));\n+\t\tEXPECT(update_index == ref.update_index);\n+\n+\t\tj++;\n+\t\treftable_ref_record_release(&ref);\n+\t}\n+\tEXPECT(j == N);\n+\treftable_iterator_destroy(&it);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+\n+\treader_close(&rd);\n+}\n+\n+static void test_table_write_small_table(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 1;\n+\twrite_table(&names, &buf, N, 4096, GIT_SHA1_HASH_ID);\n+\tEXPECT(buf.len < 200);\n+\tstrbuf_release(&buf);\n+\tfree_names(names);\n+}\n+\n+static void test_table_read_api(void)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i;\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, GIT_SHA1_HASH_ID);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, names[0]);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_free(names);\n+\treader_close(&rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_table_read_write_seek(int index, int hash_id)\n+{\n+\tchar **names;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint N = 50;\n+\tstruct reftable_reader rd = { NULL };\n+\tstruct reftable_block_source source = { NULL };\n+\tint err;\n+\tint i = 0;\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct strbuf pastLast = STRBUF_INIT;\n+\tstruct reftable_ref_record ref = { NULL };\n+\n+\twrite_table(&names, &buf, N, 256, hash_id);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\tEXPECT(hash_id == reftable_reader_hash_id(&rd));\n+\n+\tif (!index) {\n+\t\trd.ref_offsets.index_offset = 0;\n+\t} else {\n+\t\tEXPECT(rd.ref_offsets.index_offset > 0);\n+\t}\n+\n+\tfor (i = 1; i < N; i++) {\n+\t\tint err = reftable_reader_seek_ref(&rd, &it, names[i]);\n+\t\tEXPECT_ERR(err);\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(0 == strcmp(names[i], ref.refname));\n+\t\tEXPECT(REFTABLE_REF_VAL1 == ref.value_type);\n+\t\tEXPECT(i == ref.value.val1[0]);\n+\n+\t\treftable_ref_record_release(&ref);\n+\t\treftable_iterator_destroy(&it);\n+\t}\n+\n+\tstrbuf_addstr(&pastLast, names[N - 1]);\n+\tstrbuf_addstr(&pastLast, \"/\");\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, pastLast.buf);\n+\tif (err == 0) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(err > 0);\n+\t} else {\n+\t\tEXPECT(err > 0);\n+\t}\n+\n+\tstrbuf_release(&pastLast);\n+\treftable_iterator_destroy(&it);\n+\n+\tstrbuf_release(&buf);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\treftable_free(names);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_read_write_seek_linear(void)\n+{\n+\ttest_table_read_write_seek(0, GIT_SHA1_HASH_ID);\n+}\n+\n+static void test_table_read_write_seek_linear_sha256(void)\n+{\n+\ttest_table_read_write_seek(0, GIT_SHA256_HASH_ID);\n+}\n+\n+static void test_table_read_write_seek_index(void)\n+{\n+\ttest_table_read_write_seek(1, GIT_SHA1_HASH_ID);\n+}\n+\n+static void test_table_refs_for(int indexed)\n+{\n+\tint N = 50;\n+\tchar **want_names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tint want_names_len = 0;\n+\tuint8_t want_hash[GIT_SHA1_RAWSZ];\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint i = 0;\n+\tint n;\n+\tint err;\n+\tstruct reftable_reader rd;\n+\tstruct reftable_block_source source = { NULL };\n+\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint j;\n+\n+\tset_test_hash(want_hash, 4);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tuint8_t hash[GIT_SHA1_RAWSZ];\n+\t\tchar fill[51] = { 0 };\n+\t\tchar name[100];\n+\t\tuint8_t hash1[GIT_SHA1_RAWSZ];\n+\t\tuint8_t hash2[GIT_SHA1_RAWSZ];\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\n+\t\tmemset(hash, i, sizeof(hash));\n+\t\tmemset(fill, 'x', 50);\n+\t\t/* Put the variable part in the start */\n+\t\tsnprintf(name, sizeof(name), \"br%02d%s\", i, fill);\n+\t\tname[40] = 0;\n+\t\tref.refname = name;\n+\n+\t\tset_test_hash(hash1, i / 4);\n+\t\tset_test_hash(hash2, 3 + i / 4);\n+\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\tref.value.val2.value = hash1;\n+\t\tref.value.val2.target_value = hash2;\n+\n+\t\t/* 80 bytes / entry, so 3 entries per block. Yields 17\n+\t\t */\n+\t\t/* blocks. */\n+\t\tn = reftable_writer_add_ref(w, &ref);\n+\t\tEXPECT(n == 0);\n+\n+\t\tif (!memcmp(hash1, want_hash, GIT_SHA1_RAWSZ) ||\n+\t\t !memcmp(hash2, want_hash, GIT_SHA1_RAWSZ)) {\n+\t\t\twant_names[want_names_len++] = xstrdup(name);\n+\t\t}\n+\t}\n+\n+\tn = reftable_writer_close(w);\n+\tEXPECT(n == 0);\n+\n+\treftable_writer_free(w);\n+\tw = NULL;\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = init_reader(&rd, &source, \"file.ref\");\n+\tEXPECT_ERR(err);\n+\tif (!indexed) {\n+\t\trd.obj_offsets.is_present = 0;\n+\t}\n+\n+\terr = reftable_reader_seek_ref(&rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_reader_refs_for(&rd, &it, want_hash);\n+\tEXPECT_ERR(err);\n+\n+\tj = 0;\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tEXPECT(err >= 0);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tEXPECT(j < want_names_len);\n+\t\tEXPECT(0 == strcmp(ref.refname, want_names[j]));\n+\t\tj++;\n+\t\treftable_ref_record_release(&ref);\n+\t}\n+\tEXPECT(j == want_names_len);\n+\n+\tstrbuf_release(&buf);\n+\tfree_names(want_names);\n+\treftable_iterator_destroy(&it);\n+\treader_close(&rd);\n+}\n+\n+static void test_table_refs_for_no_index(void)\n+{\n+\ttest_table_refs_for(0);\n+}\n+\n+static void test_table_refs_for_obj_index(void)\n+{\n+\ttest_table_refs_for(1);\n+}\n+\n+static void test_table_empty(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_ref_record rec = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT(err == REFTABLE_EMPTY_TABLE_ERROR);\n+\treftable_writer_free(w);\n+\n+\tEXPECT(buf.len == header_size(1) + footer_size(1));\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tEXPECT(err > 0);\n+\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void test_write_key_order(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_ref_record refs[2] = {\n+\t\t{\n+\t\t\t.refname = \"b\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value = {\n+\t\t\t\t.symref = \"target\",\n+\t\t\t},\n+\t\t}, {\n+\t\t\t.refname = \"a\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value = {\n+\t\t\t\t.symref = \"target\",\n+\t\t\t},\n+\t\t}\n+\t};\n+\tint err;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\terr = reftable_writer_add_ref(w, &refs[0]);\n+\tEXPECT_ERR(err);\n+\terr = reftable_writer_add_ref(w, &refs[1]);\n+\tprintf(\"%d\\n\", err);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\treftable_writer_close(w);\n+\treftable_writer_free(w);\n+\tstrbuf_release(&buf);\n+}\n+\n+int readwrite_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_log_write_read);\n+\tRUN_TEST(test_write_key_order);\n+\tRUN_TEST(test_table_read_write_seek_linear_sha256);\n+\tRUN_TEST(test_log_buffer_size);\n+\tRUN_TEST(test_table_write_small_table);\n+\tRUN_TEST(test_buffer);\n+\tRUN_TEST(test_table_read_api);\n+\tRUN_TEST(test_table_read_write_sequential);\n+\tRUN_TEST(test_table_read_write_seek_linear);\n+\tRUN_TEST(test_table_read_write_seek_index);\n+\tRUN_TEST(test_table_refs_for_no_index);\n+\tRUN_TEST(test_table_refs_for_obj_index);\n+\tRUN_TEST(test_table_empty);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nindex 5e7698ae654e..3d541fa5c0ce 100644\n--- a/reftable/reftable-tests.h\n+++ b/reftable/reftable-tests.h\n@@ -14,7 +14,7 @@ int block_test_main(int argc, const char **argv);\n int merged_test_main(int argc, const char **argv);\n int record_test_main(int argc, const char **argv);\n int refname_test_main(int argc, const char **argv);\n-int reftable_test_main(int argc, const char **argv);\n+int readwrite_test_main(int argc, const char **argv);\n int stack_test_main(int argc, const char **argv);\n int tree_test_main(int argc, const char **argv);\n int reftable_dump_main(int argc, char *const *argv);\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 050551fa6985..898aba836fd1 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,6 +6,7 @@ int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\treadwrite_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"422281","messageId":"501cf0677ce47f5e0f2f5b5e32b84bd180f98002.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 16/28] reftable: read reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:44Z","receivedAt":"2021-04-19T11:40:26Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis supports reading a single reftable file.\n\nThe commit introduces an abstract iterator type, which captures the usecases\nboth of reading individual refs, and iterating over a segment of the ref\nnamespace.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/iter.c | 194 +++++++++\n reftable/iter.h | 69 ++++\n reftable/reader.c | 794 +++++++++++++++++++++++++++++++++++++\n reftable/reader.h | 66 +++\n reftable/reftable-reader.h | 101 +++++\n 6 files changed, 1226 insertions(+)\n create mode 100644 reftable/iter.c\n create mode 100644 reftable/iter.h\n create mode 100644 reftable/reader.c\n create mode 100644 reftable/reader.h\n create mode 100644 reftable/reftable-reader.h\n\ndiff --git a/Makefile b/Makefile\nindex 97b69ce8f080..6593e632ac08 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2412,7 +2412,9 @@ REFTABLE_OBJS += reftable/basics.o\n REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n+REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/refname.o\n REFTABLE_OBJS += reftable/generic.o\ndiff --git a/reftable/iter.c b/reftable/iter.c\nnew file mode 100644\nindex 000000000000..93d04f735b85\n--- /dev/null\n+++ b/reftable/iter.c\n@@ -0,0 +1,194 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"iter.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"generic.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"reftable-error.h\"\n+\n+int iterator_is_null(struct reftable_iterator *it)\n+{\n+\treturn !it->ops;\n+}\n+\n+static void filtering_ref_iterator_close(void *iter_arg)\n+{\n+\tstruct filtering_ref_iterator *fri = iter_arg;\n+\tstrbuf_release(&fri->oid);\n+\treftable_iterator_destroy(&fri->it);\n+}\n+\n+static int filtering_ref_iterator_next(void *iter_arg,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct filtering_ref_iterator *fri = iter_arg;\n+\tstruct reftable_ref_record *ref = rec->data;\n+\tint err = 0;\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&fri->it, ref);\n+\t\tif (err != 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (fri->double_check) {\n+\t\t\tstruct reftable_iterator it = { NULL };\n+\n+\t\t\terr = reftable_table_seek_ref(&fri->tab, &it,\n+\t\t\t\t\t\t ref->refname);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = reftable_iterator_next_ref(&it, ref);\n+\t\t\t}\n+\n+\t\t\treftable_iterator_destroy(&it);\n+\n+\t\t\tif (err < 0) {\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (err > 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (ref->value_type == REFTABLE_REF_VAL2 &&\n+\t\t (!memcmp(fri->oid.buf, ref->value.val2.target_value,\n+\t\t\t fri->oid.len) ||\n+\t\t !memcmp(fri->oid.buf, ref->value.val2.value,\n+\t\t\t fri->oid.len)))\n+\t\t\treturn 0;\n+\n+\t\tif (ref->value_type == REFTABLE_REF_VAL1 &&\n+\t\t !memcmp(fri->oid.buf, ref->value.val1, fri->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\treftable_ref_record_release(ref);\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable filtering_ref_iterator_vtable = {\n+\t.next = &filtering_ref_iterator_next,\n+\t.close = &filtering_ref_iterator_close,\n+};\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *it,\n+\t\t\t\t\t struct filtering_ref_iterator *fri)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = fri;\n+\tit->ops = &filtering_ref_iterator_vtable;\n+}\n+\n+static void indexed_table_ref_iter_close(void *p)\n+{\n+\tstruct indexed_table_ref_iter *it = p;\n+\tblock_iter_close(&it->cur);\n+\treftable_block_done(&it->block_reader.block);\n+\treftable_free(it->offsets);\n+\tstrbuf_release(&it->oid);\n+}\n+\n+static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n+{\n+\tuint64_t off;\n+\tint err = 0;\n+\tif (it->offset_idx == it->offset_len) {\n+\t\tit->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\n+\treftable_block_done(&it->block_reader.block);\n+\n+\toff = it->offsets[it->offset_idx++];\n+\terr = reader_init_block_reader(it->r, &it->block_reader, off,\n+\t\t\t\t BLOCK_TYPE_REF);\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\tif (err > 0) {\n+\t\t/* indexed block does not exist. */\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\t}\n+\tblock_reader_start(&it->block_reader, &it->cur);\n+\treturn 0;\n+}\n+\n+static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\n+{\n+\tstruct indexed_table_ref_iter *it = p;\n+\tstruct reftable_ref_record *ref = rec->data;\n+\n+\twhile (1) {\n+\t\tint err = block_iter_next(&it->cur, rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\terr = indexed_table_ref_iter_next_block(it);\n+\t\t\tif (err < 0) {\n+\t\t\t\treturn err;\n+\t\t\t}\n+\n+\t\t\tif (it->is_finished) {\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\t\t/* BUG */\n+\t\tif (!memcmp(it->oid.buf, ref->value.val2.target_value,\n+\t\t\t it->oid.len) ||\n+\t\t !memcmp(it->oid.buf, ref->value.val2.value, it->oid.len)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+}\n+\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len)\n+{\n+\tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n+\tstruct indexed_table_ref_iter *itr =\n+\t\treftable_calloc(sizeof(struct indexed_table_ref_iter));\n+\tint err = 0;\n+\n+\t*itr = empty;\n+\titr->r = r;\n+\tstrbuf_add(&itr->oid, oid, oid_len);\n+\n+\titr->offsets = offsets;\n+\titr->offset_len = offset_len;\n+\n+\terr = indexed_table_ref_iter_next_block(itr);\n+\tif (err < 0) {\n+\t\treftable_free(itr);\n+\t} else {\n+\t\t*dest = itr;\n+\t}\n+\treturn err;\n+}\n+\n+static struct reftable_iterator_vtable indexed_table_ref_iter_vtable = {\n+\t.next = &indexed_table_ref_iter_next,\n+\t.close = &indexed_table_ref_iter_close,\n+};\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = itr;\n+\tit->ops = &indexed_table_ref_iter_vtable;\n+}\ndiff --git a/reftable/iter.h b/reftable/iter.h\nnew file mode 100644\nindex 000000000000..09eb0cbfa599\n--- /dev/null\n+++ b/reftable/iter.h\n@@ -0,0 +1,69 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef ITER_H\n+#define ITER_H\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"record.h\"\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+/* Returns true for a zeroed out iterator, such as the one returned from\n+ * iterator_destroy. */\n+int iterator_is_null(struct reftable_iterator *it);\n+\n+/* iterator that produces only ref records that point to `oid` */\n+struct filtering_ref_iterator {\n+\tint double_check;\n+\tstruct reftable_table tab;\n+\tstruct strbuf oid;\n+\tstruct reftable_iterator it;\n+};\n+#define FILTERING_REF_ITERATOR_INIT \\\n+\t{ \\\n+\t\t.oid = STRBUF_INIT \\\n+\t}\n+\n+void iterator_from_filtering_ref_iterator(struct reftable_iterator *,\n+\t\t\t\t\t struct filtering_ref_iterator *);\n+\n+/* iterator that produces only ref records that point to `oid`,\n+ * but using the object index.\n+ */\n+struct indexed_table_ref_iter {\n+\tstruct reftable_reader *r;\n+\tstruct strbuf oid;\n+\n+\t/* mutable */\n+\tuint64_t *offsets;\n+\n+\t/* Points to the next offset to read. */\n+\tint offset_idx;\n+\tint offset_len;\n+\tstruct block_reader block_reader;\n+\tstruct block_iter cur;\n+\tint is_finished;\n+};\n+\n+#define INDEXED_TABLE_REF_ITER_INIT \\\n+\t{ \\\n+\t\t.cur = { .last_key = STRBUF_INIT }, .oid = STRBUF_INIT, \\\n+\t}\n+\n+void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n+\t\t\t\t\t struct indexed_table_ref_iter *itr);\n+\n+/* Takes ownership of `offsets` */\n+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+\t\t\t struct reftable_reader *r, uint8_t *oid,\n+\t\t\t int oid_len, uint64_t *offsets, int offset_len);\n+\n+#endif\ndiff --git a/reftable/reader.c b/reftable/reader.c\nnew file mode 100644\nindex 000000000000..5c9e1d6b1da7\n--- /dev/null\n+++ b/reftable/reader.c\n@@ -0,0 +1,794 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"reader.h\"\n+\n+#include \"system.h\"\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"generic.h\"\n+#include \"iter.h\"\n+#include \"record.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-generic.h\"\n+#include \"tree.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source)\n+{\n+\treturn source->ops->size(source->arg);\n+}\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size)\n+{\n+\tint result = source->ops->read_block(source->arg, dest, off, size);\n+\tdest->source = *source;\n+\treturn result;\n+}\n+\n+void block_source_close(struct reftable_block_source *source)\n+{\n+\tif (!source->ops) {\n+\t\treturn;\n+\t}\n+\n+\tsource->ops->close(source->arg);\n+\tsource->ops = NULL;\n+}\n+\n+static struct reftable_reader_offsets *\n+reader_offsets_for(struct reftable_reader *r, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase BLOCK_TYPE_REF:\n+\t\treturn &r->ref_offsets;\n+\tcase BLOCK_TYPE_LOG:\n+\t\treturn &r->log_offsets;\n+\tcase BLOCK_TYPE_OBJ:\n+\t\treturn &r->obj_offsets;\n+\t}\n+\tabort();\n+}\n+\n+static int reader_get_block(struct reftable_reader *r,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t sz)\n+{\n+\tif (off >= r->size)\n+\t\treturn 0;\n+\n+\tif (off + sz > r->size) {\n+\t\tsz = r->size - off;\n+\t}\n+\n+\treturn block_source_read_block(&r->source, dest, off, sz);\n+}\n+\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r)\n+{\n+\treturn r->hash_id;\n+}\n+\n+const char *reader_name(struct reftable_reader *r)\n+{\n+\treturn r->name;\n+}\n+\n+static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n+\t\t\tuint8_t *header)\n+{\n+\tuint8_t *f = footer;\n+\tuint8_t first_block_typ;\n+\tint err = 0;\n+\tuint32_t computed_crc;\n+\tuint32_t file_crc;\n+\n+\tif (memcmp(f, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tf += 4;\n+\n+\tif (memcmp(footer, header, header_size(r->version))) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tf++;\n+\tr->block_size = get_be24(f);\n+\n+\tf += 3;\n+\tr->min_update_index = get_be64(f);\n+\tf += 8;\n+\tr->max_update_index = get_be64(f);\n+\tf += 8;\n+\n+\tif (r->version == 1) {\n+\t\tr->hash_id = GIT_SHA1_HASH_ID;\n+\t} else {\n+\t\tr->hash_id = get_be32(f);\n+\t\tswitch (r->hash_id) {\n+\t\tcase GIT_SHA1_HASH_ID:\n+\t\t\tbreak;\n+\t\tcase GIT_SHA256_HASH_ID:\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tf += 4;\n+\t}\n+\n+\tr->ref_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->obj_offsets.offset = get_be64(f);\n+\tf += 8;\n+\n+\tr->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1);\n+\tr->obj_offsets.offset >>= 5;\n+\n+\tr->obj_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.offset = get_be64(f);\n+\tf += 8;\n+\tr->log_offsets.index_offset = get_be64(f);\n+\tf += 8;\n+\n+\tcomputed_crc = crc32(0, footer, f - footer);\n+\tfile_crc = get_be32(f);\n+\tf += 4;\n+\tif (computed_crc != file_crc) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tfirst_block_typ = header[header_size(r->version)];\n+\tr->ref_offsets.is_present = (first_block_typ == BLOCK_TYPE_REF);\n+\tr->ref_offsets.offset = 0;\n+\tr->log_offsets.is_present = (first_block_typ == BLOCK_TYPE_LOG ||\n+\t\t\t\t r->log_offsets.offset > 0);\n+\tr->obj_offsets.is_present = r->obj_offsets.offset > 0;\n+\terr = 0;\n+done:\n+\treturn err;\n+}\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name)\n+{\n+\tstruct reftable_block footer = { NULL };\n+\tstruct reftable_block header = { NULL };\n+\tint err = 0;\n+\n+\tmemset(r, 0, sizeof(struct reftable_reader));\n+\n+\t/* Need +1 to read type of first block. */\n+\terr = block_source_read_block(source, &header, 0, header_size(2) + 1);\n+\tif (err != header_size(2) + 1) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tif (memcmp(header.data, \"REFT\", 4)) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\tr->version = header.data[4];\n+\tif (r->version != 1 && r->version != 2) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tr->size = block_source_size(source) - footer_size(r->version);\n+\tr->source = *source;\n+\tr->name = xstrdup(name);\n+\tr->hash_id = 0;\n+\n+\terr = block_source_read_block(source, &footer, r->size,\n+\t\t\t\t footer_size(r->version));\n+\tif (err != footer_size(r->version)) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = parse_footer(r, footer.data, header.data);\n+done:\n+\treftable_block_done(&footer);\n+\treftable_block_done(&header);\n+\treturn err;\n+}\n+\n+struct table_iter {\n+\tstruct reftable_reader *r;\n+\tuint8_t typ;\n+\tuint64_t block_off;\n+\tstruct block_iter bi;\n+\tint is_finished;\n+};\n+#define TABLE_ITER_INIT \\\n+\t{ \\\n+\t\t.bi = {.last_key = STRBUF_INIT } \\\n+\t}\n+\n+static void table_iter_copy_from(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = src->block_off;\n+\tdest->is_finished = src->is_finished;\n+\tblock_iter_copy_from(&dest->bi, &src->bi);\n+}\n+\n+static int table_iter_next_in_block(struct table_iter *ti,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tint res = block_iter_next(&ti->bi, rec);\n+\tif (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {\n+\t\t((struct reftable_ref_record *)rec->data)->update_index +=\n+\t\t\tti->r->min_update_index;\n+\t}\n+\n+\treturn res;\n+}\n+\n+static void table_iter_block_done(struct table_iter *ti)\n+{\n+\tif (!ti->bi.br) {\n+\t\treturn;\n+\t}\n+\treftable_block_done(&ti->bi.br->block);\n+\tFREE_AND_NULL(ti->bi.br);\n+\n+\tti->bi.last_key.len = 0;\n+\tti->bi.next_off = 0;\n+}\n+\n+static int32_t extract_block_size(uint8_t *data, uint8_t *typ, uint64_t off,\n+\t\t\t\t int version)\n+{\n+\tint32_t result = 0;\n+\n+\tif (off == 0) {\n+\t\tdata += header_size(version);\n+\t}\n+\n+\t*typ = data[0];\n+\tif (reftable_is_block_type(*typ)) {\n+\t\tresult = get_be24(data + 1);\n+\t}\n+\treturn result;\n+}\n+\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ)\n+{\n+\tint32_t guess_block_size = r->block_size ? r->block_size :\n+\t\t\t\t\t\t\t DEFAULT_BLOCK_SIZE;\n+\tstruct reftable_block block = { NULL };\n+\tuint8_t block_typ = 0;\n+\tint err = 0;\n+\tuint32_t header_off = next_off ? 0 : header_size(r->version);\n+\tint32_t block_size = 0;\n+\n+\tif (next_off >= r->size)\n+\t\treturn 1;\n+\n+\terr = reader_get_block(r, &block, next_off, guess_block_size);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tblock_size = extract_block_size(block.data, &block_typ, next_off,\n+\t\t\t\t\tr->version);\n+\tif (block_size < 0)\n+\t\treturn block_size;\n+\n+\tif (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {\n+\t\treftable_block_done(&block);\n+\t\treturn 1;\n+\t}\n+\n+\tif (block_size > guess_block_size) {\n+\t\treftable_block_done(&block);\n+\t\terr = reader_get_block(r, &block, next_off, block_size);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\treturn block_reader_init(br, &block, header_off, r->block_size,\n+\t\t\t\t hash_size(r->hash_id));\n+}\n+\n+static int table_iter_next_block(struct table_iter *dest,\n+\t\t\t\t struct table_iter *src)\n+{\n+\tuint64_t next_block_off = src->block_off + src->bi.br->full_block_size;\n+\tstruct block_reader br = { 0 };\n+\tint err = 0;\n+\n+\tdest->r = src->r;\n+\tdest->typ = src->typ;\n+\tdest->block_off = next_block_off;\n+\n+\terr = reader_init_block_reader(src->r, &br, next_block_off, src->typ);\n+\tif (err > 0) {\n+\t\tdest->is_finished = 1;\n+\t\treturn 1;\n+\t}\n+\tif (err != 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct block_reader *brp =\n+\t\t\treftable_malloc(sizeof(struct block_reader));\n+\t\t*brp = br;\n+\n+\t\tdest->is_finished = 0;\n+\t\tblock_reader_start(brp, &dest->bi);\n+\t}\n+\treturn 0;\n+}\n+\n+static int table_iter_next(struct table_iter *ti, struct reftable_record *rec)\n+{\n+\tif (reftable_record_type(rec) != ti->typ)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\twhile (1) {\n+\t\tstruct table_iter next = TABLE_ITER_INIT;\n+\t\tint err = 0;\n+\t\tif (ti->is_finished) {\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\terr = table_iter_next_in_block(ti, rec);\n+\t\tif (err <= 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err != 0) {\n+\t\t\tti->is_finished = 1;\n+\t\t}\n+\t\ttable_iter_block_done(ti);\n+\t\tif (err != 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\ttable_iter_copy_from(ti, &next);\n+\t\tblock_iter_close(&next.bi);\n+\t}\n+}\n+\n+static int table_iter_next_void(void *ti, struct reftable_record *rec)\n+{\n+\treturn table_iter_next(ti, rec);\n+}\n+\n+static void table_iter_close(void *p)\n+{\n+\tstruct table_iter *ti = p;\n+\ttable_iter_block_done(ti);\n+\tblock_iter_close(&ti->bi);\n+}\n+\n+static struct reftable_iterator_vtable table_iter_vtable = {\n+\t.next = &table_iter_next_void,\n+\t.close = &table_iter_close,\n+};\n+\n+static void iterator_from_table_iter(struct reftable_iterator *it,\n+\t\t\t\t struct table_iter *ti)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = ti;\n+\tit->ops = &table_iter_vtable;\n+}\n+\n+static int reader_table_iter_at(struct reftable_reader *r,\n+\t\t\t\tstruct table_iter *ti, uint64_t off,\n+\t\t\t\tuint8_t typ)\n+{\n+\tstruct block_reader br = { 0 };\n+\tstruct block_reader *brp = NULL;\n+\n+\tint err = reader_init_block_reader(r, &br, off, typ);\n+\tif (err != 0)\n+\t\treturn err;\n+\n+\tbrp = reftable_malloc(sizeof(struct block_reader));\n+\t*brp = br;\n+\tti->r = r;\n+\tti->typ = block_reader_type(brp);\n+\tti->block_off = off;\n+\tblock_reader_start(brp, &ti->bi);\n+\treturn 0;\n+}\n+\n+static int reader_start(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\tuint8_t typ, int index)\n+{\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tuint64_t off = offs->offset;\n+\tif (index) {\n+\t\toff = offs->index_offset;\n+\t\tif (off == 0) {\n+\t\t\treturn 1;\n+\t\t}\n+\t\ttyp = BLOCK_TYPE_INDEX;\n+\t}\n+\n+\treturn reader_table_iter_at(r, ti, off, typ);\n+}\n+\n+static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,\n+\t\t\t struct reftable_record *want)\n+{\n+\tstruct reftable_record rec =\n+\t\treftable_new_record(reftable_record_type(want));\n+\tstruct strbuf want_key = STRBUF_INIT;\n+\tstruct strbuf got_key = STRBUF_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = -1;\n+\n+\treftable_record_key(want, &want_key);\n+\n+\twhile (1) {\n+\t\terr = table_iter_next_block(&next, ti);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = block_reader_first_key(next.bi.br, &got_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (strbuf_cmp(&got_key, &want_key) > 0) {\n+\t\t\ttable_iter_block_done(&next);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_block_done(ti);\n+\t\ttable_iter_copy_from(ti, &next);\n+\t}\n+\n+\terr = block_iter_seek(&ti->bi, &want_key);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = 0;\n+\n+done:\n+\tblock_iter_close(&next.bi);\n+\treftable_record_destroy(&rec);\n+\tstrbuf_release(&want_key);\n+\tstrbuf_release(&got_key);\n+\treturn err;\n+}\n+\n+static int reader_seek_indexed(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_index_record want_index = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record want_index_rec = { NULL };\n+\tstruct reftable_index_record index_result = { .last_key = STRBUF_INIT };\n+\tstruct reftable_record index_result_rec = { NULL };\n+\tstruct table_iter index_iter = TABLE_ITER_INIT;\n+\tstruct table_iter next = TABLE_ITER_INIT;\n+\tint err = 0;\n+\n+\treftable_record_key(rec, &want_index.last_key);\n+\treftable_record_from_index(&want_index_rec, &want_index);\n+\treftable_record_from_index(&index_result_rec, &index_result);\n+\n+\terr = reader_start(r, &index_iter, reftable_record_type(rec), 1);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reader_seek_linear(r, &index_iter, &want_index_rec);\n+\twhile (1) {\n+\t\terr = table_iter_next(&index_iter, &index_result_rec);\n+\t\ttable_iter_block_done(&index_iter);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = reader_table_iter_at(r, &next, index_result.offset, 0);\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\n+\t\terr = block_iter_seek(&next.bi, &want_index.last_key);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (next.typ == reftable_record_type(rec)) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (next.typ != BLOCK_TYPE_INDEX) {\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttable_iter_copy_from(&index_iter, &next);\n+\t}\n+\n+\tif (err == 0) {\n+\t\tstruct table_iter empty = TABLE_ITER_INIT;\n+\t\tstruct table_iter *malloced =\n+\t\t\treftable_calloc(sizeof(struct table_iter));\n+\t\t*malloced = empty;\n+\t\ttable_iter_copy_from(malloced, &next);\n+\t\titerator_from_table_iter(it, malloced);\n+\t}\n+done:\n+\tblock_iter_close(&next.bi);\n+\ttable_iter_close(&index_iter);\n+\treftable_record_release(&want_index_rec);\n+\treftable_record_release(&index_result_rec);\n+\treturn err;\n+}\n+\n+static int reader_seek_internal(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it,\n+\t\t\t\tstruct reftable_record *rec)\n+{\n+\tstruct reftable_reader_offsets *offs =\n+\t\treader_offsets_for(r, reftable_record_type(rec));\n+\tuint64_t idx = offs->index_offset;\n+\tstruct table_iter ti = TABLE_ITER_INIT;\n+\tint err = 0;\n+\tif (idx > 0)\n+\t\treturn reader_seek_indexed(r, it, rec);\n+\n+\terr = reader_start(r, &ti, reftable_record_type(rec), 0);\n+\tif (err < 0)\n+\t\treturn err;\n+\terr = reader_seek_linear(r, &ti, rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\telse {\n+\t\tstruct table_iter *p =\n+\t\t\treftable_malloc(sizeof(struct table_iter));\n+\t\t*p = ti;\n+\t\titerator_from_table_iter(it, p);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec)\n+{\n+\tuint8_t typ = reftable_record_type(rec);\n+\n+\tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n+\tif (!offs->is_present) {\n+\t\titerator_set_empty(it);\n+\t\treturn 0;\n+\t}\n+\n+\treturn reader_seek_internal(r, it, rec);\n+}\n+\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn reader_seek(r, it, &rec);\n+}\n+\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_reader_seek_log_at(r, it, name, max);\n+}\n+\n+void reader_close(struct reftable_reader *r)\n+{\n+\tblock_source_close(&r->source);\n+\tFREE_AND_NULL(r->name);\n+}\n+\n+int reftable_new_reader(struct reftable_reader **p,\n+\t\t\tstruct reftable_block_source *src, char const *name)\n+{\n+\tstruct reftable_reader *rd =\n+\t\treftable_calloc(sizeof(struct reftable_reader));\n+\tint err = init_reader(rd, src, name);\n+\tif (err == 0) {\n+\t\t*p = rd;\n+\t} else {\n+\t\tblock_source_close(src);\n+\t\treftable_free(rd);\n+\t}\n+\treturn err;\n+}\n+\n+void reftable_reader_free(struct reftable_reader *r)\n+{\n+\treader_close(r);\n+\treftable_free(r);\n+}\n+\n+static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct reftable_obj_record want = {\n+\t\t.hash_prefix = oid,\n+\t\t.hash_prefix_len = r->object_id_len,\n+\t};\n+\tstruct reftable_record want_rec = { NULL };\n+\tstruct reftable_iterator oit = { NULL };\n+\tstruct reftable_obj_record got = { NULL };\n+\tstruct reftable_record got_rec = { NULL };\n+\tint err = 0;\n+\tstruct indexed_table_ref_iter *itr = NULL;\n+\n+\t/* Look through the reverse index. */\n+\treftable_record_from_obj(&want_rec, &want);\n+\terr = reader_seek(r, &oit, &want_rec);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\t/* read out the reftable_obj_record */\n+\treftable_record_from_obj(&got_rec, &got);\n+\terr = iterator_next(&oit, &got_rec);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 0 ||\n+\t memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {\n+\t\t/* didn't find it; return empty iterator */\n+\t\titerator_set_empty(it);\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),\n+\t\t\t\t\t got.offsets, got.offset_len);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tgot.offsets = NULL;\n+\titerator_from_indexed_table_ref_iter(it, itr);\n+\n+done:\n+\treftable_iterator_destroy(&oit);\n+\treftable_record_release(&got_rec);\n+\treturn err;\n+}\n+\n+static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t uint8_t *oid)\n+{\n+\tstruct table_iter ti_empty = TABLE_ITER_INIT;\n+\tstruct table_iter *ti = reftable_calloc(sizeof(struct table_iter));\n+\tstruct filtering_ref_iterator *filter = NULL;\n+\tstruct filtering_ref_iterator empty = FILTERING_REF_ITERATOR_INIT;\n+\tint oid_len = hash_size(r->hash_id);\n+\tint err;\n+\n+\t*ti = ti_empty;\n+\terr = reader_start(r, ti, BLOCK_TYPE_REF, 0);\n+\tif (err < 0) {\n+\t\treftable_free(ti);\n+\t\treturn err;\n+\t}\n+\n+\tfilter = reftable_malloc(sizeof(struct filtering_ref_iterator));\n+\t*filter = empty;\n+\n+\tstrbuf_add(&filter->oid, oid, oid_len);\n+\treftable_table_from_reader(&filter->tab, r);\n+\tfilter->double_check = 0;\n+\titerator_from_table_iter(&filter->it, ti);\n+\n+\titerator_from_filtering_ref_iterator(it, filter);\n+\treturn 0;\n+}\n+\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid)\n+{\n+\tif (r->obj_offsets.is_present)\n+\t\treturn reftable_reader_refs_for_indexed(r, it, oid);\n+\treturn reftable_reader_refs_for_unindexed(r, it, oid);\n+}\n+\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r)\n+{\n+\treturn r->max_update_index;\n+}\n+\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r)\n+{\n+\treturn r->min_update_index;\n+}\n+\n+/* generic table interface. */\n+\n+static int reftable_reader_seek_void(void *tab, struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn reader_seek(tab, it, rec);\n+}\n+\n+static uint32_t reftable_reader_hash_id_void(void *tab)\n+{\n+\treturn reftable_reader_hash_id(tab);\n+}\n+\n+static uint64_t reftable_reader_min_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_min_update_index(tab);\n+}\n+\n+static uint64_t reftable_reader_max_update_index_void(void *tab)\n+{\n+\treturn reftable_reader_max_update_index(tab);\n+}\n+\n+static struct reftable_table_vtable reader_vtable = {\n+\t.seek_record = reftable_reader_seek_void,\n+\t.hash_id = reftable_reader_hash_id_void,\n+\t.min_update_index = reftable_reader_min_update_index_void,\n+\t.max_update_index = reftable_reader_max_update_index_void,\n+};\n+\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader)\n+{\n+\tassert(!tab->ops);\n+\ttab->ops = &reader_vtable;\n+\ttab->table_arg = reader;\n+}\n+\n+\n+int reftable_reader_print_file(const char *tablename)\n+{\n+\tstruct reftable_block_source src = { NULL };\n+\tint err = reftable_block_source_from_file(&src, tablename);\n+\tstruct reftable_reader *r = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&r, &src, tablename);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\treftable_table_from_reader(&tab, r);\n+\terr = reftable_table_print(&tab);\n+done:\n+\treftable_reader_free(r);\n+\treturn err;\n+}\ndiff --git a/reftable/reader.h b/reftable/reader.h\nnew file mode 100644\nindex 000000000000..39583e5dbcd4\n--- /dev/null\n+++ b/reftable/reader.h\n@@ -0,0 +1,66 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef READER_H\n+#define READER_H\n+\n+#include \"block.h\"\n+#include \"record.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+\n+uint64_t block_source_size(struct reftable_block_source *source);\n+\n+int block_source_read_block(struct reftable_block_source *source,\n+\t\t\t struct reftable_block *dest, uint64_t off,\n+\t\t\t uint32_t size);\n+void block_source_close(struct reftable_block_source *source);\n+\n+/* metadata for a block type */\n+struct reftable_reader_offsets {\n+\tint is_present;\n+\tuint64_t offset;\n+\tuint64_t index_offset;\n+};\n+\n+/* The state for reading a reftable file. */\n+struct reftable_reader {\n+\t/* for convience, associate a name with the instance. */\n+\tchar *name;\n+\tstruct reftable_block_source source;\n+\n+\t/* Size of the file, excluding the footer. */\n+\tuint64_t size;\n+\n+\t/* 'sha1' for SHA1, 's256' for SHA-256 */\n+\tuint32_t hash_id;\n+\n+\tuint32_t block_size;\n+\tuint64_t min_update_index;\n+\tuint64_t max_update_index;\n+\t/* Length of the OID keys in the 'o' section */\n+\tint object_id_len;\n+\tint version;\n+\n+\tstruct reftable_reader_offsets ref_offsets;\n+\tstruct reftable_reader_offsets obj_offsets;\n+\tstruct reftable_reader_offsets log_offsets;\n+};\n+\n+int init_reader(struct reftable_reader *r, struct reftable_block_source *source,\n+\t\tconst char *name);\n+int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,\n+\t\tstruct reftable_record *rec);\n+void reader_close(struct reftable_reader *r);\n+const char *reader_name(struct reftable_reader *r);\n+\n+/* initialize a block reader to read from `r` */\n+int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\n+\t\t\t uint64_t next_off, uint8_t want_typ);\n+\n+#endif\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nnew file mode 100644\nindex 000000000000..4a4bc2fdf85a\n--- /dev/null\n+++ b/reftable/reftable-reader.h\n@@ -0,0 +1,101 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_READER_H\n+#define REFTABLE_READER_H\n+\n+#include \"reftable-iterator.h\"\n+#include \"reftable-blocksource.h\"\n+\n+/*\n+ * Reading single tables\n+ *\n+ * The follow routines are for reading single files. For an\n+ * application-level interface, skip ahead to struct\n+ * reftable_merged_table and struct reftable_stack.\n+ */\n+\n+/* The reader struct is a handle to an open reftable file. */\n+struct reftable_reader;\n+\n+/* Generic table. */\n+struct reftable_table;\n+\n+/* reftable_new_reader opens a reftable for reading. If successful,\n+ * returns 0 code and sets pp. The name is used for creating a\n+ * stack. Typically, it is the basename of the file. The block source\n+ * `src` is owned by the reader, and is closed on calling\n+ * reftable_reader_destroy(). On error, the block source `src` is\n+ * closed as well.\n+ */\n+int reftable_new_reader(struct reftable_reader **pp,\n+\t\t\tstruct reftable_block_source *src, const char *name);\n+\n+/* reftable_reader_seek_ref returns an iterator where 'name' would be inserted\n+ in the table. To seek to the start of the table, use name = \"\".\n+\n+ example:\n+\n+ struct reftable_reader *r = NULL;\n+ int err = reftable_new_reader(&r, &src, \"filename\");\n+ if (err < 0) { ... }\n+ struct reftable_iterator it = {0};\n+ err = reftable_reader_seek_ref(r, &it, \"refs/heads/master\");\n+ if (err < 0) { ... }\n+ struct reftable_ref_record ref = {0};\n+ while (1) {\n+ err = reftable_iterator_next_ref(&it, &ref);\n+ if (err > 0) {\n+ break;\n+ }\n+ if (err < 0) {\n+ ..error handling..\n+ }\n+ ..found..\n+ }\n+ reftable_iterator_destroy(&it);\n+ reftable_ref_record_release(&ref);\n+*/\n+int reftable_reader_seek_ref(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* returns the hash ID used in this table. */\n+uint32_t reftable_reader_hash_id(struct reftable_reader *r);\n+\n+/* seek to logs for the given name, older than update_index. To seek to the\n+ start of the table, use name = \"\".\n+*/\n+int reftable_reader_seek_log_at(struct reftable_reader *r,\n+\t\t\t\tstruct reftable_iterator *it, const char *name,\n+\t\t\t\tuint64_t update_index);\n+\n+/* seek to newest log entry for given name. */\n+int reftable_reader_seek_log(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* closes and deallocates a reader. */\n+void reftable_reader_free(struct reftable_reader *);\n+\n+/* return an iterator for the refs pointing to `oid`. */\n+int reftable_reader_refs_for(struct reftable_reader *r,\n+\t\t\t struct reftable_iterator *it, uint8_t *oid);\n+\n+/* return the max_update_index for a table */\n+uint64_t reftable_reader_max_update_index(struct reftable_reader *r);\n+\n+/* return the min_update_index for a table */\n+uint64_t reftable_reader_min_update_index(struct reftable_reader *r);\n+\n+/* creates a generic table from a file reader. */\n+void reftable_table_from_reader(struct reftable_table *tab,\n+\t\t\t\tstruct reftable_reader *reader);\n+\n+/* print table onto stdout for debugging. */\n+int reftable_reader_print_file(const char *tablename);\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"422282","messageId":"afca8f8f6a2905c4864c9b8b6baf6a546de310e8.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 15/28] reftable: generic interface to tables","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:43Z","receivedAt":"2021-04-19T11:40:31Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 3 +\n reftable/generic.c | 169 +++++++++++++++++++++++++++++++++++\n reftable/generic.h | 32 +++++++\n reftable/reftable-generic.h | 47 ++++++++++\n reftable/reftable-iterator.h | 39 ++++++++\n reftable/reftable.c | 115 ++++++++++++++++++++++++\n 6 files changed, 405 insertions(+)\n create mode 100644 reftable/generic.c\n create mode 100644 reftable/generic.h\n create mode 100644 reftable/reftable-generic.h\n create mode 100644 reftable/reftable-iterator.h\n create mode 100644 reftable/reftable.c\n\ndiff --git a/Makefile b/Makefile\nindex f7954b362aeb..97b69ce8f080 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2414,6 +2414,9 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/refname.o\n+REFTABLE_OBJS += reftable/generic.o\n+REFTABLE_OBJS += reftable/stack.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n \ndiff --git a/reftable/generic.c b/reftable/generic.c\nnew file mode 100644\nindex 000000000000..7a8a738d8609\n--- /dev/null\n+++ b/reftable/generic.c\n@@ -0,0 +1,169 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"generic.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+int reftable_table_seek_log(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = ~((uint64_t)0),\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_table_seek_ref(tab, &it, name);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_ref(&it, ref);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref->refname, name) ||\n+\t reftable_ref_record_is_deletion(ref)) {\n+\t\treftable_ref_record_release(ref);\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+int reftable_table_print(struct reftable_table *tab) {\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\tuint32_t hash_id = reftable_table_hash_id(tab);\n+\tint err = reftable_table_seek_ref(tab, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_ref_record_print(&ref, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_ref_record_release(&ref);\n+\n+\terr = reftable_table_seek_log(tab, &it, \"\");\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_log_record_print(&log, hash_id);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\treturn 0;\n+}\n+\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->max_update_index(tab->table_arg);\n+}\n+\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->min_update_index(tab->table_arg);\n+}\n+\n+uint32_t reftable_table_hash_id(struct reftable_table *tab)\n+{\n+\treturn tab->ops->hash_id(tab->table_arg);\n+}\n+\n+void reftable_iterator_destroy(struct reftable_iterator *it)\n+{\n+\tif (!it->ops) {\n+\t\treturn;\n+\t}\n+\tit->ops->close(it->iter_arg);\n+\tit->ops = NULL;\n+\tFREE_AND_NULL(it->iter_arg);\n+}\n+\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, ref);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, log);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n+{\n+\treturn it->ops->next(it->iter_arg, rec);\n+}\n+\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n+{\n+\treturn 1;\n+}\n+\n+static void empty_iterator_close(void *arg)\n+{\n+}\n+\n+static struct reftable_iterator_vtable empty_vtable = {\n+\t.next = &empty_iterator_next,\n+\t.close = &empty_iterator_close,\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = NULL;\n+\tit->ops = &empty_vtable;\n+}\ndiff --git a/reftable/generic.h b/reftable/generic.h\nnew file mode 100644\nindex 000000000000..98886a06402d\n--- /dev/null\n+++ b/reftable/generic.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef GENERIC_H\n+#define GENERIC_H\n+\n+#include \"record.h\"\n+#include \"reftable-generic.h\"\n+\n+/* generic interface to reftables */\n+struct reftable_table_vtable {\n+\tint (*seek_record)(void *tab, struct reftable_iterator *it,\n+\t\t\t struct reftable_record *);\n+\tuint32_t (*hash_id)(void *tab);\n+\tuint64_t (*min_update_index)(void *tab);\n+\tuint64_t (*max_update_index)(void *tab);\n+};\n+\n+struct reftable_iterator_vtable {\n+\tint (*next)(void *iter_arg, struct reftable_record *rec);\n+\tvoid (*close)(void *iter_arg);\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it);\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec);\n+\n+#endif\ndiff --git a/reftable/reftable-generic.h b/reftable/reftable-generic.h\nnew file mode 100644\nindex 000000000000..d239751a7783\n--- /dev/null\n+++ b/reftable/reftable-generic.h\n@@ -0,0 +1,47 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_GENERIC_H\n+#define REFTABLE_GENERIC_H\n+\n+#include \"reftable-iterator.h\"\n+\n+struct reftable_table_vtable;\n+\n+/*\n+ * Provides a unified API for reading tables, either merged tables, or single\n+ * readers. */\n+struct reftable_table {\n+\tstruct reftable_table_vtable *ops;\n+\tvoid *table_arg;\n+};\n+\n+int reftable_table_seek_log(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name);\n+\n+/* returns the hash ID from a generic reftable_table */\n+uint32_t reftable_table_hash_id(struct reftable_table *tab);\n+\n+/* returns the max update_index covered by this table. */\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab);\n+\n+/* returns the min update_index covered by this table. */\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0\n+ for success, and 1 if ref not found. */\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* dump table contents onto stdout for debugging */\n+int reftable_table_print(struct reftable_table *tab);\n+\n+#endif\ndiff --git a/reftable/reftable-iterator.h b/reftable/reftable-iterator.h\nnew file mode 100644\nindex 000000000000..d3eee7af3571\n--- /dev/null\n+++ b/reftable/reftable-iterator.h\n@@ -0,0 +1,39 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_ITERATOR_H\n+#define REFTABLE_ITERATOR_H\n+\n+#include \"reftable-record.h\"\n+\n+struct reftable_iterator_vtable;\n+\n+/* iterator is the generic interface for walking over data stored in a\n+ * reftable.\n+ */\n+struct reftable_iterator {\n+\tstruct reftable_iterator_vtable *ops;\n+\tvoid *iter_arg;\n+};\n+\n+/* reads the next reftable_ref_record. Returns < 0 for error, 0 for OK and > 0:\n+ * end of iteration.\n+ */\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* reads the next reftable_log_record. Returns < 0 for error, 0 for OK and > 0:\n+ * end of iteration.\n+ */\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* releases resources associated with an iterator. */\n+void reftable_iterator_destroy(struct reftable_iterator *it);\n+\n+#endif\ndiff --git a/reftable/reftable.c b/reftable/reftable.c\nnew file mode 100644\nindex 000000000000..0e4607a7cd6c\n--- /dev/null\n+++ b/reftable/reftable.c\n@@ -0,0 +1,115 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"generic.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-generic.h\"\n+\n+int reftable_table_seek_ref(struct reftable_table *tab,\n+\t\t\t struct reftable_iterator *it, const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn tab->ops->seek_record(tab->table_arg, it, &rec);\n+}\n+\n+int reftable_table_read_ref(struct reftable_table *tab, const char *name,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_table_seek_ref(tab, &it, name);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_ref(&it, ref);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref->refname, name) ||\n+\t reftable_ref_record_is_deletion(ref)) {\n+\t\treftable_ref_record_release(ref);\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+uint64_t reftable_table_max_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->max_update_index(tab->table_arg);\n+}\n+\n+uint64_t reftable_table_min_update_index(struct reftable_table *tab)\n+{\n+\treturn tab->ops->min_update_index(tab->table_arg);\n+}\n+\n+uint32_t reftable_table_hash_id(struct reftable_table *tab)\n+{\n+\treturn tab->ops->hash_id(tab->table_arg);\n+}\n+\n+void reftable_iterator_destroy(struct reftable_iterator *it)\n+{\n+\tif (!it->ops) {\n+\t\treturn;\n+\t}\n+\tit->ops->close(it->iter_arg);\n+\tit->ops = NULL;\n+\tFREE_AND_NULL(it->iter_arg);\n+}\n+\n+int reftable_iterator_next_ref(struct reftable_iterator *it,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, ref);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int reftable_iterator_next_log(struct reftable_iterator *it,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, log);\n+\treturn iterator_next(it, &rec);\n+}\n+\n+int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n+{\n+\treturn it->ops->next(it->iter_arg, rec);\n+}\n+\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n+{\n+\treturn 1;\n+}\n+\n+static void empty_iterator_close(void *arg)\n+{\n+}\n+\n+static struct reftable_iterator_vtable empty_vtable = {\n+\t.next = &empty_iterator_next,\n+\t.close = &empty_iterator_close,\n+};\n+\n+void iterator_set_empty(struct reftable_iterator *it)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = NULL;\n+\tit->ops = &empty_vtable;\n+}\n-- \ngitgitgadget\n\n"},{"id":"422283","messageId":"922deb272fff198401f2d2962069e559019b6d12.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 14/28] reftable: write reftable files","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:42Z","receivedAt":"2021-04-19T11:40:33Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable-writer.h | 147 ++++++++\n reftable/writer.c | 690 +++++++++++++++++++++++++++++++++++++\n reftable/writer.h | 50 +++\n 4 files changed, 888 insertions(+)\n create mode 100644 reftable/reftable-writer.h\n create mode 100644 reftable/writer.c\n create mode 100644 reftable/writer.h\n\ndiff --git a/Makefile b/Makefile\nindex 01916cefda0f..f7954b362aeb 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2415,6 +2415,7 @@ REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/tree.o\n+REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nnew file mode 100644\nindex 000000000000..bdef4813f4c7\n--- /dev/null\n+++ b/reftable/reftable-writer.h\n@@ -0,0 +1,147 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_WRITER_H\n+#define REFTABLE_WRITER_H\n+\n+#include \n+\n+#include \"reftable-record.h\"\n+\n+/* Writing single reftables */\n+\n+/* reftable_write_options sets options for writing a single reftable. */\n+struct reftable_write_options {\n+\t/* boolean: do not pad out blocks to block size. */\n+\tunsigned unpadded : 1;\n+\n+\t/* the blocksize. Should be less than 2^24. */\n+\tuint32_t block_size;\n+\n+\t/* boolean: do not generate a SHA1 => ref index. */\n+\tunsigned skip_index_objects : 1;\n+\n+\t/* how often to write complete keys in each block. */\n+\tint restart_interval;\n+\n+\t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n+\t * Defaults to SHA1 if unset\n+\t */\n+\tuint32_t hash_id;\n+\n+\t/* boolean: do not check ref names for validity or dir/file conflicts.\n+\t */\n+\tunsigned skip_name_check : 1;\n+\n+\t/* boolean: copy log messages exactly. If unset, check that the message\n+\t * is a single line, and add '\\n' if missing.\n+\t */\n+\tunsigned exact_log_message : 1;\n+};\n+\n+/* reftable_block_stats holds statistics for a single block type */\n+struct reftable_block_stats {\n+\t/* total number of entries written */\n+\tint entries;\n+\t/* total number of key restarts */\n+\tint restarts;\n+\t/* total number of blocks */\n+\tint blocks;\n+\t/* total number of index blocks */\n+\tint index_blocks;\n+\t/* depth of the index */\n+\tint max_index_level;\n+\n+\t/* offset of the first block for this type */\n+\tuint64_t offset;\n+\t/* offset of the top level index block for this type, or 0 if not\n+\t * present */\n+\tuint64_t index_offset;\n+};\n+\n+/* stats holds overall statistics for a single reftable */\n+struct reftable_stats {\n+\t/* total number of blocks written. */\n+\tint blocks;\n+\t/* stats for ref data */\n+\tstruct reftable_block_stats ref_stats;\n+\t/* stats for the SHA1 to ref map. */\n+\tstruct reftable_block_stats obj_stats;\n+\t/* stats for index blocks */\n+\tstruct reftable_block_stats idx_stats;\n+\t/* stats for log blocks */\n+\tstruct reftable_block_stats log_stats;\n+\n+\t/* disambiguation length of shortened object IDs. */\n+\tint object_id_len;\n+};\n+\n+/* reftable_new_writer creates a new writer */\n+struct reftable_writer *\n+reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts);\n+\n+/* Set the range of update indices for the records we will add. When writing a\n+ table into a stack, the min should be at least\n+ reftable_stack_next_update_index(), or REFTABLE_API_ERROR is returned.\n+\n+ For transactional updates to a stack, typically min==max, and the\n+ update_index can be obtained by inspeciting the stack. When converting an\n+ existing ref database into a single reftable, this would be a range of\n+ update-index timestamps.\n+ */\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max);\n+\n+/*\n+ Add a reftable_ref_record. The record should have names that come after\n+ already added records.\n+\n+ The update_index must be within the limits set by\n+ reftable_writer_set_limits(), or REFTABLE_API_ERROR is returned. It is an\n+ REFTABLE_API_ERROR error to write a ref record after a log record.\n+*/\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/*\n+ Convenience function to add multiple reftable_ref_records; the function sorts\n+ the records before adding them, reordering the records array passed in.\n+*/\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n);\n+\n+/*\n+ adds reftable_log_records. Log records are keyed by (refname, decreasing\n+ update_index). The key for the record added must come after the already added\n+ log records.\n+*/\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log);\n+\n+/*\n+ Convenience function to add multiple reftable_log_records; the function sorts\n+ the records before adding them, reordering records array passed in.\n+*/\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n);\n+\n+/* reftable_writer_close finalizes the reftable. The writer is retained so\n+ * statistics can be inspected. */\n+int reftable_writer_close(struct reftable_writer *w);\n+\n+/* writer_stats returns the statistics on the reftable being written.\n+\n+ This struct becomes invalid when the writer is freed.\n+ */\n+const struct reftable_stats *writer_stats(struct reftable_writer *w);\n+\n+/* reftable_writer_free deallocates memory for the writer */\n+void reftable_writer_free(struct reftable_writer *w);\n+\n+#endif\ndiff --git a/reftable/writer.c b/reftable/writer.c\nnew file mode 100644\nindex 000000000000..81e9515c5358\n--- /dev/null\n+++ b/reftable/writer.c\n@@ -0,0 +1,690 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"writer.h\"\n+\n+#include \"system.h\"\n+\n+#include \"block.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"tree.h\"\n+#include \"reftable-error.h\"\n+\n+/* finishes a block, and writes it to storage */\n+static int writer_flush_block(struct reftable_writer *w);\n+\n+/* deallocates memory related to the index */\n+static void writer_clear_index(struct reftable_writer *w);\n+\n+/* finishes writing a 'r' (refs) or 'g' (reflogs) section */\n+static int writer_finish_public_section(struct reftable_writer *w);\n+\n+static struct reftable_block_stats *\n+writer_reftable_block_stats(struct reftable_writer *w, uint8_t typ)\n+{\n+\tswitch (typ) {\n+\tcase 'r':\n+\t\treturn &w->stats.ref_stats;\n+\tcase 'o':\n+\t\treturn &w->stats.obj_stats;\n+\tcase 'i':\n+\t\treturn &w->stats.idx_stats;\n+\tcase 'g':\n+\t\treturn &w->stats.log_stats;\n+\t}\n+\tabort();\n+\treturn NULL;\n+}\n+\n+/* write data, queuing the padding for the next write. Returns negative for\n+ * error. */\n+static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n+\t\t\tint padding)\n+{\n+\tint n = 0;\n+\tif (w->pending_padding > 0) {\n+\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding);\n+\t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n+\t\tif (n < 0)\n+\t\t\treturn n;\n+\n+\t\tw->pending_padding = 0;\n+\t\treftable_free(zeroed);\n+\t}\n+\n+\tw->pending_padding = padding;\n+\tn = w->write(w->write_arg, data, len);\n+\tif (n < 0)\n+\t\treturn n;\n+\tn += padding;\n+\treturn 0;\n+}\n+\n+static void options_set_defaults(struct reftable_write_options *opts)\n+{\n+\tif (opts->restart_interval == 0) {\n+\t\topts->restart_interval = 16;\n+\t}\n+\n+\tif (opts->hash_id == 0) {\n+\t\topts->hash_id = GIT_SHA1_HASH_ID;\n+\t}\n+\tif (opts->block_size == 0) {\n+\t\topts->block_size = DEFAULT_BLOCK_SIZE;\n+\t}\n+}\n+\n+static int writer_version(struct reftable_writer *w)\n+{\n+\treturn (w->opts.hash_id == 0 || w->opts.hash_id == GIT_SHA1_HASH_ID) ?\n+\t\t\t 1 :\n+\t\t\t 2;\n+}\n+\n+static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n+{\n+\tmemcpy((char *)dest, \"REFT\", 4);\n+\n+\tdest[4] = writer_version(w);\n+\n+\tput_be24(dest + 5, w->opts.block_size);\n+\tput_be64(dest + 8, w->min_update_index);\n+\tput_be64(dest + 16, w->max_update_index);\n+\tif (writer_version(w) == 2) {\n+\t\tput_be32(dest + 24, w->opts.hash_id);\n+\t}\n+\treturn header_size(writer_version(w));\n+}\n+\n+static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n+{\n+\tint block_start = 0;\n+\tif (w->next == 0) {\n+\t\tblock_start = header_size(writer_version(w));\n+\t}\n+\n+\tstrbuf_release(&w->last_key);\n+\tblock_writer_init(&w->block_writer_data, typ, w->block,\n+\t\t\t w->opts.block_size, block_start,\n+\t\t\t hash_size(w->opts.hash_id));\n+\tw->block_writer = &w->block_writer_data;\n+\tw->block_writer->restart_interval = w->opts.restart_interval;\n+}\n+\n+static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n+\n+struct reftable_writer *\n+reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t void *writer_arg, struct reftable_write_options *opts)\n+{\n+\tstruct reftable_writer *wp =\n+\t\treftable_calloc(sizeof(struct reftable_writer));\n+\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n+\toptions_set_defaults(opts);\n+\tif (opts->block_size >= (1 << 24)) {\n+\t\t/* TODO - error return? */\n+\t\tabort();\n+\t}\n+\twp->last_key = reftable_empty_strbuf;\n+\twp->block = reftable_calloc(opts->block_size);\n+\twp->write = writer_func;\n+\twp->write_arg = writer_arg;\n+\twp->opts = *opts;\n+\twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n+\n+\treturn wp;\n+}\n+\n+void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\n+\t\t\t\tuint64_t max)\n+{\n+\tw->min_update_index = min;\n+\tw->max_update_index = max;\n+}\n+\n+void reftable_writer_free(struct reftable_writer *w)\n+{\n+\treftable_free(w->block);\n+\treftable_free(w);\n+}\n+\n+struct obj_index_tree_node {\n+\tstruct strbuf hash;\n+\tuint64_t *offsets;\n+\tsize_t offset_len;\n+\tsize_t offset_cap;\n+};\n+\n+#define OBJ_INDEX_TREE_NODE_INIT \\\n+\t{ \\\n+\t\t.hash = STRBUF_INIT \\\n+\t}\n+\n+static int obj_index_tree_node_compare(const void *a, const void *b)\n+{\n+\treturn strbuf_cmp(&((const struct obj_index_tree_node *)a)->hash,\n+\t\t\t &((const struct obj_index_tree_node *)b)->hash);\n+}\n+\n+static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n+{\n+\tuint64_t off = w->next;\n+\n+\tstruct obj_index_tree_node want = { .hash = *hash };\n+\n+\tstruct tree_node *node = tree_search(&want, &w->obj_index_tree,\n+\t\t\t\t\t &obj_index_tree_node_compare, 0);\n+\tstruct obj_index_tree_node *key = NULL;\n+\tif (node == NULL) {\n+\t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n+\t\tkey = reftable_malloc(sizeof(struct obj_index_tree_node));\n+\t\t*key = empty;\n+\n+\t\tstrbuf_reset(&key->hash);\n+\t\tstrbuf_addbuf(&key->hash, hash);\n+\t\ttree_search((void *)key, &w->obj_index_tree,\n+\t\t\t &obj_index_tree_node_compare, 1);\n+\t} else {\n+\t\tkey = node->key;\n+\t}\n+\n+\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off) {\n+\t\treturn;\n+\t}\n+\n+\tif (key->offset_len == key->offset_cap) {\n+\t\tkey->offset_cap = 2 * key->offset_cap + 1;\n+\t\tkey->offsets = reftable_realloc(\n+\t\t\tkey->offsets, sizeof(uint64_t) * key->offset_cap);\n+\t}\n+\n+\tkey->offsets[key->offset_len++] = off;\n+}\n+\n+static int writer_add_record(struct reftable_writer *w,\n+\t\t\t struct reftable_record *rec)\n+{\n+\tstruct strbuf key = STRBUF_INIT;\n+\tint err = -1;\n+\treftable_record_key(rec, &key);\n+\tif (strbuf_cmp(&w->last_key, &key) >= 0) {\n+\t\terr = REFTABLE_API_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tstrbuf_reset(&w->last_key);\n+\tstrbuf_addbuf(&w->last_key, &key);\n+\tif (w->block_writer == NULL) {\n+\t\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\t}\n+\n+\tassert(block_writer_type(w->block_writer) == reftable_record_type(rec));\n+\n+\tif (block_writer_add(w->block_writer, rec) == 0) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\terr = block_writer_add(w->block_writer, rec);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = 0;\n+done:\n+\tstrbuf_release(&key);\n+\treturn err;\n+}\n+\n+int reftable_writer_add_ref(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tstruct reftable_ref_record copy = *ref;\n+\tint err = 0;\n+\n+\tif (ref->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\tif (ref->update_index < w->min_update_index ||\n+\t ref->update_index > w->max_update_index)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\treftable_record_from_ref(&rec, ©);\n+\tcopy.update_index -= w->min_update_index;\n+\n+\terr = writer_add_record(w, &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (!w->opts.skip_index_objects && reftable_ref_record_val1(ref)) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, (char *)reftable_ref_record_val1(ref),\n+\t\t\t hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\n+\tif (!w->opts.skip_index_objects && reftable_ref_record_val2(ref)) {\n+\t\tstruct strbuf h = STRBUF_INIT;\n+\t\tstrbuf_add(&h, reftable_ref_record_val2(ref),\n+\t\t\t hash_size(w->opts.hash_id));\n+\t\twriter_index_hash(w, &h);\n+\t\tstrbuf_release(&h);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_writer_add_refs(struct reftable_writer *w,\n+\t\t\t struct reftable_ref_record *refs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(refs, n, reftable_ref_record_compare_name);\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_ref(w, &refs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+static int reftable_writer_add_log_verbatim(struct reftable_writer *w,\n+\t\t\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_record rec = { NULL };\n+\tif (w->block_writer &&\n+\t block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {\n+\t\tint err = writer_finish_public_section(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tw->next -= w->pending_padding;\n+\tw->pending_padding = 0;\n+\n+\treftable_record_from_log(&rec, log);\n+\treturn writer_add_record(w, &rec);\n+}\n+\n+int reftable_writer_add_log(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tchar *input_log_message = NULL;\n+\tstruct strbuf cleaned_message = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tif (log->value_type == REFTABLE_LOG_DELETION)\n+\t\treturn reftable_writer_add_log_verbatim(w, log);\n+\n+\tif (log->refname == NULL)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\tinput_log_message = log->update.message;\n+\tif (!w->opts.exact_log_message && log->update.message) {\n+\t\tstrbuf_addstr(&cleaned_message, log->update.message);\n+\t\twhile (cleaned_message.len &&\n+\t\t cleaned_message.buf[cleaned_message.len - 1] == '\\n')\n+\t\t\tstrbuf_setlen(&cleaned_message,\n+\t\t\t\t cleaned_message.len - 1);\n+\t\tif (strchr(cleaned_message.buf, '\\n')) {\n+\t\t\t// multiple lines not allowed.\n+\t\t\terr = REFTABLE_API_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tstrbuf_addstr(&cleaned_message, \"\\n\");\n+\t\tlog->update.message = cleaned_message.buf;\n+\t}\n+\n+\terr = reftable_writer_add_log_verbatim(w, log);\n+\tlog->update.message = input_log_message;\n+done:\n+\tstrbuf_release(&cleaned_message);\n+\treturn err;\n+}\n+\n+int reftable_writer_add_logs(struct reftable_writer *w,\n+\t\t\t struct reftable_log_record *logs, int n)\n+{\n+\tint err = 0;\n+\tint i = 0;\n+\tQSORT(logs, n, reftable_log_record_compare_key);\n+\n+\tfor (i = 0; err == 0 && i < n; i++) {\n+\t\terr = reftable_writer_add_log(w, &logs[i]);\n+\t}\n+\treturn err;\n+}\n+\n+static int writer_finish_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tuint64_t index_start = 0;\n+\tint max_level = 0;\n+\tint threshold = w->opts.unpadded ? 1 : 3;\n+\tint before_blocks = w->stats.idx_stats.blocks;\n+\tint err = writer_flush_block(w);\n+\tint i = 0;\n+\tstruct reftable_block_stats *bstats = NULL;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\twhile (w->index_len > threshold) {\n+\t\tstruct reftable_index_record *idx = NULL;\n+\t\tint idx_len = 0;\n+\n+\t\tmax_level++;\n+\t\tindex_start = w->next;\n+\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\tidx = w->index;\n+\t\tidx_len = w->index_len;\n+\n+\t\tw->index = NULL;\n+\t\tw->index_len = 0;\n+\t\tw->index_cap = 0;\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstruct reftable_record rec = { NULL };\n+\t\t\treftable_record_from_index(&rec, idx + i);\n+\t\t\tif (block_writer_add(w->block_writer, &rec) == 0) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\terr = writer_flush_block(w);\n+\t\t\tif (err < 0)\n+\t\t\t\treturn err;\n+\n+\t\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\n+\t\t\terr = block_writer_add(w->block_writer, &rec);\n+\t\t\tif (err != 0) {\n+\t\t\t\t/* write into fresh block should always succeed\n+\t\t\t\t */\n+\t\t\t\tabort();\n+\t\t\t}\n+\t\t}\n+\t\tfor (i = 0; i < idx_len; i++) {\n+\t\t\tstrbuf_release(&idx[i].last_key);\n+\t\t}\n+\t\treftable_free(idx);\n+\t}\n+\n+\twriter_clear_index(w);\n+\n+\terr = writer_flush_block(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tbstats = writer_reftable_block_stats(w, typ);\n+\tbstats->index_blocks = w->stats.idx_stats.blocks - before_blocks;\n+\tbstats->index_offset = index_start;\n+\tbstats->max_index_level = max_level;\n+\n+\t/* Reinit lastKey, as the next section can start with any key. */\n+\tw->last_key.len = 0;\n+\n+\treturn 0;\n+}\n+\n+struct common_prefix_arg {\n+\tstruct strbuf *last;\n+\tint max;\n+};\n+\n+static void update_common(void *void_arg, void *key)\n+{\n+\tstruct common_prefix_arg *arg = void_arg;\n+\tstruct obj_index_tree_node *entry = key;\n+\tif (arg->last) {\n+\t\tint n = common_prefix_size(&entry->hash, arg->last);\n+\t\tif (n > arg->max) {\n+\t\t\targ->max = n;\n+\t\t}\n+\t}\n+\targ->last = &entry->hash;\n+}\n+\n+struct write_record_arg {\n+\tstruct reftable_writer *w;\n+\tint err;\n+};\n+\n+static void write_object_record(void *void_arg, void *key)\n+{\n+\tstruct write_record_arg *arg = void_arg;\n+\tstruct obj_index_tree_node *entry = key;\n+\tstruct reftable_obj_record obj_rec = {\n+\t\t.hash_prefix = (uint8_t *)entry->hash.buf,\n+\t\t.hash_prefix_len = arg->w->stats.object_id_len,\n+\t\t.offsets = entry->offsets,\n+\t\t.offset_len = entry->offset_len,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\treftable_record_from_obj(&rec, &obj_rec);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\n+\targ->err = writer_flush_block(arg->w);\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n+\twriter_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\tif (arg->err == 0)\n+\t\tgoto done;\n+\tobj_rec.offset_len = 0;\n+\targ->err = block_writer_add(arg->w->block_writer, &rec);\n+\n+\t/* Should be able to write into a fresh block. */\n+\tassert(arg->err == 0);\n+\n+done:;\n+}\n+\n+static void object_record_free(void *void_arg, void *key)\n+{\n+\tstruct obj_index_tree_node *entry = key;\n+\n+\tFREE_AND_NULL(entry->offsets);\n+\tstrbuf_release(&entry->hash);\n+\treftable_free(entry);\n+}\n+\n+static int writer_dump_object_index(struct reftable_writer *w)\n+{\n+\tstruct write_record_arg closure = { .w = w };\n+\tstruct common_prefix_arg common = { NULL };\n+\tif (w->obj_index_tree) {\n+\t\tinfix_walk(w->obj_index_tree, &update_common, &common);\n+\t}\n+\tw->stats.object_id_len = common.max + 1;\n+\n+\twriter_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\n+\tif (w->obj_index_tree) {\n+\t\tinfix_walk(w->obj_index_tree, &write_object_record, &closure);\n+\t}\n+\n+\tif (closure.err < 0)\n+\t\treturn closure.err;\n+\treturn writer_finish_section(w);\n+}\n+\n+static int writer_finish_public_section(struct reftable_writer *w)\n+{\n+\tuint8_t typ = 0;\n+\tint err = 0;\n+\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\n+\ttyp = block_writer_type(w->block_writer);\n+\terr = writer_finish_section(w);\n+\tif (err < 0)\n+\t\treturn err;\n+\tif (typ == BLOCK_TYPE_REF && !w->opts.skip_index_objects &&\n+\t w->stats.ref_stats.index_blocks > 0) {\n+\t\terr = writer_dump_object_index(w);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n+\n+\tif (w->obj_index_tree) {\n+\t\tinfix_walk(w->obj_index_tree, &object_record_free, NULL);\n+\t\ttree_free(w->obj_index_tree);\n+\t\tw->obj_index_tree = NULL;\n+\t}\n+\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+int reftable_writer_close(struct reftable_writer *w)\n+{\n+\tuint8_t footer[72];\n+\tuint8_t *p = footer;\n+\tint err = writer_finish_public_section(w);\n+\tint empty_table = w->next == 0;\n+\tif (err != 0)\n+\t\tgoto done;\n+\tw->pending_padding = 0;\n+\tif (empty_table) {\n+\t\t/* Empty tables need a header anyway. */\n+\t\tuint8_t header[28];\n+\t\tint n = writer_write_header(w, header);\n+\t\terr = padded_write(w, header, n, 0);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\tp += writer_write_header(w, footer);\n+\tput_be64(p, w->stats.ref_stats.index_offset);\n+\tp += 8;\n+\tput_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n+\tp += 8;\n+\tput_be64(p, w->stats.obj_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be64(p, w->stats.log_stats.offset);\n+\tp += 8;\n+\tput_be64(p, w->stats.log_stats.index_offset);\n+\tp += 8;\n+\n+\tput_be32(p, crc32(0, footer, p - footer));\n+\tp += 4;\n+\n+\terr = padded_write(w, footer, footer_size(writer_version(w)), 0);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (empty_table) {\n+\t\terr = REFTABLE_EMPTY_TABLE_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\t/* free up memory. */\n+\tblock_writer_release(&w->block_writer_data);\n+\twriter_clear_index(w);\n+\tstrbuf_release(&w->last_key);\n+\treturn err;\n+}\n+\n+static void writer_clear_index(struct reftable_writer *w)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < w->index_len; i++) {\n+\t\tstrbuf_release(&w->index[i].last_key);\n+\t}\n+\n+\tFREE_AND_NULL(w->index);\n+\tw->index_len = 0;\n+\tw->index_cap = 0;\n+}\n+\n+static const int debug = 0;\n+\n+static int writer_flush_nonempty_block(struct reftable_writer *w)\n+{\n+\tuint8_t typ = block_writer_type(w->block_writer);\n+\tstruct reftable_block_stats *bstats =\n+\t\twriter_reftable_block_stats(w, typ);\n+\tuint64_t block_typ_off = (bstats->blocks == 0) ? w->next : 0;\n+\tint raw_bytes = block_writer_finish(w->block_writer);\n+\tint padding = 0;\n+\tint err = 0;\n+\tstruct reftable_index_record ir = { .last_key = STRBUF_INIT };\n+\tif (raw_bytes < 0)\n+\t\treturn raw_bytes;\n+\n+\tif (!w->opts.unpadded && typ != BLOCK_TYPE_LOG) {\n+\t\tpadding = w->opts.block_size - raw_bytes;\n+\t}\n+\n+\tif (block_typ_off > 0) {\n+\t\tbstats->offset = block_typ_off;\n+\t}\n+\n+\tbstats->entries += w->block_writer->entries;\n+\tbstats->restarts += w->block_writer->restart_len;\n+\tbstats->blocks++;\n+\tw->stats.blocks++;\n+\n+\tif (debug) {\n+\t\tfprintf(stderr, \"block %c off %\" PRIu64 \" sz %d (%d)\\n\", typ,\n+\t\t\tw->next, raw_bytes,\n+\t\t\tget_be24(w->block + w->block_writer->header_off + 1));\n+\t}\n+\n+\tif (w->next == 0) {\n+\t\twriter_write_header(w, w->block);\n+\t}\n+\n+\terr = padded_write(w, w->block, raw_bytes, padding);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (w->index_cap == w->index_len) {\n+\t\tw->index_cap = 2 * w->index_cap + 1;\n+\t\tw->index = reftable_realloc(\n+\t\t\tw->index,\n+\t\t\tsizeof(struct reftable_index_record) * w->index_cap);\n+\t}\n+\n+\tir.offset = w->next;\n+\tstrbuf_reset(&ir.last_key);\n+\tstrbuf_addbuf(&ir.last_key, &w->block_writer->last_key);\n+\tw->index[w->index_len] = ir;\n+\n+\tw->index_len++;\n+\tw->next += padding + raw_bytes;\n+\tw->block_writer = NULL;\n+\treturn 0;\n+}\n+\n+static int writer_flush_block(struct reftable_writer *w)\n+{\n+\tif (w->block_writer == NULL)\n+\t\treturn 0;\n+\tif (w->block_writer->entries == 0)\n+\t\treturn 0;\n+\treturn writer_flush_nonempty_block(w);\n+}\n+\n+const struct reftable_stats *writer_stats(struct reftable_writer *w)\n+{\n+\treturn &w->stats;\n+}\ndiff --git a/reftable/writer.h b/reftable/writer.h\nnew file mode 100644\nindex 000000000000..09b88673d975\n--- /dev/null\n+++ b/reftable/writer.h\n@@ -0,0 +1,50 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef WRITER_H\n+#define WRITER_H\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"tree.h\"\n+#include \"reftable-writer.h\"\n+\n+struct reftable_writer {\n+\tssize_t (*write)(void *, const void *, size_t);\n+\tvoid *write_arg;\n+\tint pending_padding;\n+\tstruct strbuf last_key;\n+\n+\t/* offset of next block to write. */\n+\tuint64_t next;\n+\tuint64_t min_update_index, max_update_index;\n+\tstruct reftable_write_options opts;\n+\n+\t/* memory buffer for writing */\n+\tuint8_t *block;\n+\n+\t/* writer for the current section. NULL or points to\n+\t * block_writer_data */\n+\tstruct block_writer *block_writer;\n+\n+\tstruct block_writer block_writer_data;\n+\n+\t/* pending index records for the current section */\n+\tstruct reftable_index_record *index;\n+\tsize_t index_len;\n+\tsize_t index_cap;\n+\n+\t/*\n+\t * tree for use with tsearch; used to populate the 'o' inverse OID\n+\t * map */\n+\tstruct tree_node *obj_index_tree;\n+\n+\tstruct reftable_stats stats;\n+};\n+\n+#endif\n-- \ngitgitgadget\n\n"},{"id":"422284","messageId":"c03957db45c076ccb4400ddfa621159c816a86e1.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 13/28] reftable: a generic binary tree implementation","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:41Z","receivedAt":"2021-04-19T11:40:40Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe reftable format includes support for an (OID => ref) map. This map can speed\nup visibility and reachability checks. In particular, various operations along\nthe fetch/push path within Gerrit have ben sped up by using this structure.\n\nThe map is constructed with help of a binary tree. Object IDs are hashes, so\nthey are uniformly distributed. Hence, the tree does not attempt forced\nrebalancing.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 4 ++-\n reftable/tree.c | 63 ++++++++++++++++++++++++++++++++++++++++\n reftable/tree.h | 34 ++++++++++++++++++++++\n reftable/tree_test.c | 61 ++++++++++++++++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 162 insertions(+), 1 deletion(-)\n create mode 100644 reftable/tree.c\n create mode 100644 reftable/tree.h\n create mode 100644 reftable/tree_test.c\n\ndiff --git a/Makefile b/Makefile\nindex cd8a9de2519e..01916cefda0f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2414,11 +2414,13 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/record.o\n+REFTABLE_OBJS += reftable/tree.o\n \n+REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n-REFTABLE_TEST_OBJS += reftable/basics_test.o\n+REFTABLE_TEST_OBJS += reftable/tree_test.o\n \n TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))\n \ndiff --git a/reftable/tree.c b/reftable/tree.c\nnew file mode 100644\nindex 000000000000..82db7995dd68\n--- /dev/null\n+++ b/reftable/tree.c\n@@ -0,0 +1,63 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"system.h\"\n+\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert)\n+{\n+\tint res;\n+\tif (*rootp == NULL) {\n+\t\tif (!insert) {\n+\t\t\treturn NULL;\n+\t\t} else {\n+\t\t\tstruct tree_node *n =\n+\t\t\t\treftable_calloc(sizeof(struct tree_node));\n+\t\t\tn->key = key;\n+\t\t\t*rootp = n;\n+\t\t\treturn *rootp;\n+\t\t}\n+\t}\n+\n+\tres = compare(key, (*rootp)->key);\n+\tif (res < 0)\n+\t\treturn tree_search(key, &(*rootp)->left, compare, insert);\n+\telse if (res > 0)\n+\t\treturn tree_search(key, &(*rootp)->right, compare, insert);\n+\treturn *rootp;\n+}\n+\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg)\n+{\n+\tif (t->left) {\n+\t\tinfix_walk(t->left, action, arg);\n+\t}\n+\taction(arg, t->key);\n+\tif (t->right) {\n+\t\tinfix_walk(t->right, action, arg);\n+\t}\n+}\n+\n+void tree_free(struct tree_node *t)\n+{\n+\tif (t == NULL) {\n+\t\treturn;\n+\t}\n+\tif (t->left) {\n+\t\ttree_free(t->left);\n+\t}\n+\tif (t->right) {\n+\t\ttree_free(t->right);\n+\t}\n+\treftable_free(t);\n+}\ndiff --git a/reftable/tree.h b/reftable/tree.h\nnew file mode 100644\nindex 000000000000..fbdd002e23af\n--- /dev/null\n+++ b/reftable/tree.h\n@@ -0,0 +1,34 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef TREE_H\n+#define TREE_H\n+\n+/* tree_node is a generic binary search tree. */\n+struct tree_node {\n+\tvoid *key;\n+\tstruct tree_node *left, *right;\n+};\n+\n+/* looks for `key` in `rootp` using `compare` as comparison function. If insert\n+ * is set, insert the key if it's not found. Else, return NULL.\n+ */\n+struct tree_node *tree_search(void *key, struct tree_node **rootp,\n+\t\t\t int (*compare)(const void *, const void *),\n+\t\t\t int insert);\n+\n+/* performs an infix walk of the tree. */\n+void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\n+\t\tvoid *arg);\n+\n+/*\n+ * deallocates the tree nodes recursively. Keys should be deallocated separately\n+ * by walking over the tree. */\n+void tree_free(struct tree_node *t);\n+\n+#endif\ndiff --git a/reftable/tree_test.c b/reftable/tree_test.c\nnew file mode 100644\nindex 000000000000..09a970e17b9c\n--- /dev/null\n+++ b/reftable/tree_test.c\n@@ -0,0 +1,61 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"tree.h\"\n+\n+#include \"basics.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+static int test_compare(const void *a, const void *b)\n+{\n+\treturn (char *)a - (char *)b;\n+}\n+\n+struct curry {\n+\tvoid *last;\n+};\n+\n+static void check_increasing(void *arg, void *key)\n+{\n+\tstruct curry *c = arg;\n+\tif (c->last) {\n+\t\tassert(test_compare(c->last, key) < 0);\n+\t}\n+\tc->last = key;\n+}\n+\n+static void test_tree(void)\n+{\n+\tstruct tree_node *root = NULL;\n+\n+\tvoid *values[11] = { NULL };\n+\tstruct tree_node *nodes[11] = { NULL };\n+\tint i = 1;\n+\tstruct curry c = { NULL };\n+\tdo {\n+\t\tnodes[i] = tree_search(values + i, &root, &test_compare, 1);\n+\t\ti = (i * 7) % 11;\n+\t} while (i != 1);\n+\n+\tfor (i = 1; i < ARRAY_SIZE(nodes); i++) {\n+\t\tassert(values + i == nodes[i]->key);\n+\t\tassert(nodes[i] ==\n+\t\t tree_search(values + i, &root, &test_compare, 0));\n+\t}\n+\n+\tinfix_walk(root, check_increasing, &c);\n+\ttree_free(root);\n+}\n+\n+int tree_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_tree);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex c9deeaf08c7a..050551fa6985 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -6,5 +6,6 @@ int cmd__reftable(int argc, const char **argv)\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"422286","messageId":"3f1e5aedf4a8c692032e9ff014473e3f36136e6f.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 21/28] reftable: implement stack, a mutable database of reftable files.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:49Z","receivedAt":"2021-04-19T11:40:44Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/reftable-stack.h | 128 ++++\n reftable/stack.c | 1395 +++++++++++++++++++++++++++++++++++++\n reftable/stack.h | 41 ++\n reftable/stack_test.c | 940 +++++++++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 6 files changed, 2506 insertions(+)\n create mode 100644 reftable/reftable-stack.h\n create mode 100644 reftable/stack.c\n create mode 100644 reftable/stack.h\n create mode 100644 reftable/stack_test.c\n\ndiff --git a/Makefile b/Makefile\nindex b3973d8f5a8f..f78d1cbc2d3c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2431,6 +2431,7 @@ REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n REFTABLE_TEST_OBJS += reftable/refname_test.o\n+REFTABLE_TEST_OBJS += reftable/stack_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nnew file mode 100644\nindex 000000000000..b64daf120a3c\n--- /dev/null\n+++ b/reftable/reftable-stack.h\n@@ -0,0 +1,128 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_STACK_H\n+#define REFTABLE_STACK_H\n+\n+#include \"reftable-writer.h\"\n+\n+/*\n+ * The stack presents an interface to a mutable sequence of reftables.\n+\n+ * A stack can be mutated by pushing a table to the top of the stack.\n+\n+ * The reftable_stack automatically compacts files on disk to ensure good\n+ * amortized performance.\n+ *\n+ * For windows and other platforms that cannot have open files as rename\n+ * destinations, concurrent access from multiple processes needs the rand()\n+ * random seed to be randomized.\n+ */\n+struct reftable_stack;\n+\n+/* open a new reftable stack. The tables along with the table list will be\n+ * stored in 'dir'. Typically, this should be .git/reftables.\n+ */\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config);\n+\n+/* returns the update_index at which a next table should be written. */\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\n+\n+/* holds a transaction to add tables at the top of a stack. */\n+struct reftable_addition;\n+\n+/*\n+ * returns a new transaction to add reftables to the given stack. As a side\n+ * effect, the ref database is locked.\n+ */\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st);\n+\n+/* Adds a reftable to transaction. */\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+\n+/* Commits the transaction, releasing the lock. After calling this,\n+ * reftable_addition_destroy should still be called.\n+ */\n+int reftable_addition_commit(struct reftable_addition *add);\n+\n+/* Release all non-committed data from the transaction, and deallocate the\n+ * transaction. Releases the lock if held. */\n+void reftable_addition_destroy(struct reftable_addition *add);\n+\n+/* add a new table to the stack. The write_table function must call\n+ * reftable_writer_set_limits, add refs and return an error value. */\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *write_arg),\n+\t\t void *write_arg);\n+\n+/* returns the merged_table for seeking. This table is valid until the\n+ * next write or reload, and should not be closed or deleted.\n+ */\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st);\n+\n+/* frees all resources associated with the stack. */\n+void reftable_stack_destroy(struct reftable_stack *st);\n+\n+/* Reloads the stack if necessary. This is very cheap to run if the stack was up\n+ * to date */\n+int reftable_stack_reload(struct reftable_stack *st);\n+\n+/* Policy for expiring reflog entries. */\n+struct reftable_log_expiry_config {\n+\t/* Drop entries older than this timestamp */\n+\tuint64_t time;\n+\n+\t/* Drop older entries */\n+\tuint64_t min_update_index;\n+};\n+\n+/* compacts all reftables into a giant table. Expire reflog entries if config is\n+ * non-NULL */\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config);\n+\n+/* heuristically compact unbalanced table stack. */\n+int reftable_stack_auto_compact(struct reftable_stack *st);\n+\n+/* delete stale .ref tables. */\n+int reftable_stack_clean(struct reftable_stack *st);\n+\n+/* convenience function to read a single ref. Returns < 0 for error, 0 for\n+ * success, and 1 if ref not found. */\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref);\n+\n+/* convenience function to read a single log. Returns < 0 for error, 0 for\n+ * success, and 1 if ref not found. */\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log);\n+\n+/* statistics on past compactions. */\n+struct reftable_compaction_stats {\n+\tuint64_t bytes; /* total number of bytes written */\n+\tuint64_t entries_written; /* total number of entries written, including\n+\t\t\t\t failures. */\n+\tint attempts; /* how often we tried to compact */\n+\tint failures; /* failures happen on concurrent updates */\n+};\n+\n+/* return statistics for compaction up till now. */\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st);\n+\n+/* print the entire stack represented by the directory */\n+int reftable_stack_print_directory(const char *stackdir);\n+\n+#endif\ndiff --git a/reftable/stack.c b/reftable/stack.c\nnew file mode 100644\nindex 000000000000..e3e94e320664\n--- /dev/null\n+++ b/reftable/stack.c\n@@ -0,0 +1,1395 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+#include \"merged.h\"\n+#include \"reader.h\"\n+#include \"refname.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-record.h\"\n+#include \"reftable-merged.h\"\n+#include \"writer.h\"\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg);\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config);\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name);\n+static void reftable_addition_close(struct reftable_addition *add);\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open);\n+\n+static void stack_filename(struct strbuf *dest, struct reftable_stack *st,\n+\t\t\t const char *name)\n+{\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, st->reftable_dir);\n+\tstrbuf_addstr(dest, \"/\");\n+\tstrbuf_addstr(dest, name);\n+}\n+\n+static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz)\n+{\n+\tint *fdp = (int *)arg;\n+\treturn write(*fdp, data, sz);\n+}\n+\n+int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n+\t\t struct reftable_write_options config)\n+{\n+\tstruct reftable_stack *p =\n+\t\treftable_calloc(sizeof(struct reftable_stack));\n+\tstruct strbuf list_file_name = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tif (config.hash_id == 0) {\n+\t\tconfig.hash_id = GIT_SHA1_HASH_ID;\n+\t}\n+\n+\t*dest = NULL;\n+\n+\tstrbuf_reset(&list_file_name);\n+\tstrbuf_addstr(&list_file_name, dir);\n+\tstrbuf_addstr(&list_file_name, \"/tables.list\");\n+\n+\tp->list_file = strbuf_detach(&list_file_name, NULL);\n+\tp->reftable_dir = xstrdup(dir);\n+\tp->config = config;\n+\n+\terr = reftable_stack_reload_maybe_reuse(p, 1);\n+\tif (err < 0) {\n+\t\treftable_stack_destroy(p);\n+\t} else {\n+\t\t*dest = p;\n+\t}\n+\treturn err;\n+}\n+\n+static int fd_read_lines(int fd, char ***namesp)\n+{\n+\toff_t size = lseek(fd, 0, SEEK_END);\n+\tchar *buf = NULL;\n+\tint err = 0;\n+\tif (size < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\terr = lseek(fd, 0, SEEK_SET);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tbuf = reftable_malloc(size + 1);\n+\tif (read(fd, buf, size) != size) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\tbuf[size] = 0;\n+\n+\tparse_names(buf, size, namesp);\n+\n+done:\n+\treftable_free(buf);\n+\treturn err;\n+}\n+\n+int read_lines(const char *filename, char ***namesp)\n+{\n+\tint fd = open(filename, O_RDONLY, 0644);\n+\tint err = 0;\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\t*namesp = reftable_calloc(sizeof(char *));\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\treturn REFTABLE_IO_ERROR;\n+\t}\n+\terr = fd_read_lines(fd, namesp);\n+\tclose(fd);\n+\treturn err;\n+}\n+\n+struct reftable_merged_table *\n+reftable_stack_merged_table(struct reftable_stack *st)\n+{\n+\treturn st->merged;\n+}\n+\n+static int has_name(char **names, const char *name)\n+{\n+\twhile (*names) {\n+\t\tif (!strcmp(*names, name))\n+\t\t\treturn 1;\n+\t\tnames++;\n+\t}\n+\treturn 0;\n+}\n+\n+/* Close and free the stack */\n+void reftable_stack_destroy(struct reftable_stack *st)\n+{\n+\tchar **names = NULL;\n+\tint err = 0;\n+\tif (st->merged) {\n+\t\treftable_merged_table_free(st->merged);\n+\t\tst->merged = NULL;\n+\t}\n+\n+\terr = read_lines(st->list_file, &names);\n+\tif (err < 0) {\n+\t\tFREE_AND_NULL(names);\n+\t}\n+\n+\tif (st->readers) {\n+\t\tint i = 0;\n+\t\tstruct strbuf filename = STRBUF_INIT;\n+\t\tfor (i = 0; i < st->readers_len; i++) {\n+\t\t\tconst char *name = reader_name(st->readers[i]);\n+\t\t\tstrbuf_reset(&filename);\n+\t\t\tif (names && !has_name(names, name)) {\n+\t\t\t\tstack_filename(&filename, st, name);\n+\t\t\t}\n+\t\t\treftable_reader_free(st->readers[i]);\n+\n+\t\t\tif (filename.len) {\n+\t\t\t\t// On Windows, can only unlink after closing.\n+\t\t\t\tunlink(filename.buf);\n+\t\t\t}\n+\t\t}\n+\t\tstrbuf_release(&filename);\n+\t\tst->readers_len = 0;\n+\t\tFREE_AND_NULL(st->readers);\n+\t}\n+\tFREE_AND_NULL(st->list_file);\n+\tFREE_AND_NULL(st->reftable_dir);\n+\treftable_free(st);\n+\tfree_names(names);\n+}\n+\n+static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n+\t\t\t\t\t\t int cur_len)\n+{\n+\tstruct reftable_reader **cur =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * cur_len);\n+\tint i = 0;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tcur[i] = st->readers[i];\n+\t}\n+\treturn cur;\n+}\n+\n+static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n+\t\t\t\t int reuse_open)\n+{\n+\tint cur_len = !st->merged ? 0 : st->merged->stack_len;\n+\tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n+\tint err = 0;\n+\tint names_len = names_length(names);\n+\tstruct reftable_reader **new_readers =\n+\t\treftable_calloc(sizeof(struct reftable_reader *) * names_len);\n+\tstruct reftable_table *new_tables =\n+\t\treftable_calloc(sizeof(struct reftable_table) * names_len);\n+\tint new_readers_len = 0;\n+\tstruct reftable_merged_table *new_merged = NULL;\n+\tint i;\n+\n+\twhile (*names) {\n+\t\tstruct reftable_reader *rd = NULL;\n+\t\tchar *name = *names++;\n+\n+\t\t/* this is linear; we assume compaction keeps the number of\n+\t\t tables under control so this is not quadratic. */\n+\t\tint j = 0;\n+\t\tfor (j = 0; reuse_open && j < cur_len; j++) {\n+\t\t\tif (cur[j] && 0 == strcmp(cur[j]->name, name)) {\n+\t\t\t\trd = cur[j];\n+\t\t\t\tcur[j] = NULL;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (!rd) {\n+\t\t\tstruct reftable_block_source src = { NULL };\n+\t\t\tstruct strbuf table_path = STRBUF_INIT;\n+\t\t\tstack_filename(&table_path, st, name);\n+\n+\t\t\terr = reftable_block_source_from_file(&src,\n+\t\t\t\t\t\t\t table_path.buf);\n+\t\t\tstrbuf_release(&table_path);\n+\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\n+\t\t\terr = reftable_new_reader(&rd, &src, name);\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\n+\t\tnew_readers[new_readers_len] = rd;\n+\t\treftable_table_from_reader(&new_tables[new_readers_len], rd);\n+\t\tnew_readers_len++;\n+\t}\n+\n+\t/* success! */\n+\terr = reftable_new_merged_table(&new_merged, new_tables,\n+\t\t\t\t\tnew_readers_len, st->config.hash_id);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tnew_tables = NULL;\n+\tst->readers_len = new_readers_len;\n+\tif (st->merged) {\n+\t\tmerged_table_release(st->merged);\n+\t\treftable_merged_table_free(st->merged);\n+\t}\n+\tif (st->readers) {\n+\t\treftable_free(st->readers);\n+\t}\n+\tst->readers = new_readers;\n+\tnew_readers = NULL;\n+\tnew_readers_len = 0;\n+\n+\tnew_merged->suppress_deletions = 1;\n+\tst->merged = new_merged;\n+\tfor (i = 0; i < cur_len; i++) {\n+\t\tif (cur[i]) {\n+\t\t\tconst char *name = reader_name(cur[i]);\n+\t\t\tstruct strbuf filename = STRBUF_INIT;\n+\t\t\tstack_filename(&filename, st, name);\n+\n+\t\t\treader_close(cur[i]);\n+\t\t\treftable_reader_free(cur[i]);\n+\n+\t\t\t// On Windows, can only unlink after closing.\n+\t\t\tunlink(filename.buf);\n+\n+\t\t\tstrbuf_release(&filename);\n+\t\t}\n+\t}\n+\n+done:\n+\tfor (i = 0; i < new_readers_len; i++) {\n+\t\treader_close(new_readers[i]);\n+\t\treftable_reader_free(new_readers[i]);\n+\t}\n+\treftable_free(new_readers);\n+\treftable_free(new_tables);\n+\treftable_free(cur);\n+\treturn err;\n+}\n+\n+/* return negative if a before b. */\n+static int tv_cmp(struct timeval *a, struct timeval *b)\n+{\n+\ttime_t diff = a->tv_sec - b->tv_sec;\n+\tint udiff = a->tv_usec - b->tv_usec;\n+\n+\tif (diff != 0)\n+\t\treturn diff;\n+\n+\treturn udiff;\n+}\n+\n+static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n+\t\t\t\t\t int reuse_open)\n+{\n+\tstruct timeval deadline = { 0 };\n+\tint err = gettimeofday(&deadline, NULL);\n+\tint64_t delay = 0;\n+\tint tries = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tdeadline.tv_sec += 3;\n+\twhile (1) {\n+\t\tchar **names = NULL;\n+\t\tchar **names_after = NULL;\n+\t\tstruct timeval now = { 0 };\n+\t\tint err = gettimeofday(&now, NULL);\n+\t\tint err2 = 0;\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* Only look at deadlines after the first few times. This\n+\t\t simplifies debugging in GDB */\n+\t\ttries++;\n+\t\tif (tries > 3 && tv_cmp(&now, &deadline) >= 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = read_lines(st->list_file, &names);\n+\t\tif (err < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\t\terr = reftable_stack_reload_once(st, names, reuse_open);\n+\t\tif (err == 0) {\n+\t\t\tfree_names(names);\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err != REFTABLE_NOT_EXIST_ERROR) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\t/* err == REFTABLE_NOT_EXIST_ERROR can be caused by a concurrent\n+\t\t writer. Check if there was one by checking if the name list\n+\t\t changed.\n+\t\t*/\n+\t\terr2 = read_lines(st->list_file, &names_after);\n+\t\tif (err2 < 0) {\n+\t\t\tfree_names(names);\n+\t\t\treturn err2;\n+\t\t}\n+\n+\t\tif (names_equal(names_after, names)) {\n+\t\t\tfree_names(names);\n+\t\t\tfree_names(names_after);\n+\t\t\treturn err;\n+\t\t}\n+\t\tfree_names(names);\n+\t\tfree_names(names_after);\n+\n+\t\tdelay = delay + (delay * rand()) / RAND_MAX + 1;\n+\t\tsleep_millisec(delay);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/* -1 = error\n+ 0 = up to date\n+ 1 = changed. */\n+static int stack_uptodate(struct reftable_stack *st)\n+{\n+\tchar **names = NULL;\n+\tint err = read_lines(st->list_file, &names);\n+\tint i = 0;\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tfor (i = 0; i < st->readers_len; i++) {\n+\t\tif (!names[i]) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (strcmp(st->readers[i]->name, names[i])) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (names[st->merged->stack_len]) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tfree_names(names);\n+\treturn err;\n+}\n+\n+int reftable_stack_reload(struct reftable_stack *st)\n+{\n+\tint err = stack_uptodate(st);\n+\tif (err > 0)\n+\t\treturn reftable_stack_reload_maybe_reuse(st, 1);\n+\treturn err;\n+}\n+\n+int reftable_stack_add(struct reftable_stack *st,\n+\t\t int (*write)(struct reftable_writer *wr, void *arg),\n+\t\t void *arg)\n+{\n+\tint err = stack_try_add(st, write, arg);\n+\tif (err < 0) {\n+\t\tif (err == REFTABLE_LOCK_ERROR) {\n+\t\t\t/* Ignore error return, we want to propagate\n+\t\t\t REFTABLE_LOCK_ERROR.\n+\t\t\t*/\n+\t\t\treftable_stack_reload(st);\n+\t\t}\n+\t\treturn err;\n+\t}\n+\n+\tif (!st->disable_auto_compact)\n+\t\treturn reftable_stack_auto_compact(st);\n+\n+\treturn 0;\n+}\n+\n+static void format_name(struct strbuf *dest, uint64_t min, uint64_t max)\n+{\n+\tchar buf[100];\n+\tuint32_t rnd = (uint32_t)rand();\n+\tsnprintf(buf, sizeof(buf), \"0x%012\" PRIx64 \"-0x%012\" PRIx64 \"-%08x\",\n+\t\t min, max, rnd);\n+\tstrbuf_reset(dest);\n+\tstrbuf_addstr(dest, buf);\n+}\n+\n+struct reftable_addition {\n+\tint lock_file_fd;\n+\tstruct strbuf lock_file_name;\n+\tstruct reftable_stack *stack;\n+\n+\tchar **new_tables;\n+\tint new_tables_len;\n+\tuint64_t next_update_index;\n+};\n+\n+#define REFTABLE_ADDITION_INIT \\\n+\t{ \\\n+\t\t.lock_file_name = STRBUF_INIT \\\n+\t}\n+\n+static int reftable_stack_init_addition(struct reftable_addition *add,\n+\t\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tadd->stack = st;\n+\n+\tstrbuf_reset(&add->lock_file_name);\n+\tstrbuf_addstr(&add->lock_file_name, st->list_file);\n+\tstrbuf_addstr(&add->lock_file_name, \".lock\");\n+\n+\tadd->lock_file_fd = open(add->lock_file_name.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (add->lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = REFTABLE_LOCK_ERROR;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\terr = stack_uptodate(st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (err > 1) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->next_update_index = reftable_stack_next_update_index(st);\n+done:\n+\tif (err) {\n+\t\treftable_addition_close(add);\n+\t}\n+\treturn err;\n+}\n+\n+static void reftable_addition_close(struct reftable_addition *add)\n+{\n+\tint i = 0;\n+\tstruct strbuf nm = STRBUF_INIT;\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstack_filename(&nm, add->stack, add->new_tables[i]);\n+\t\tunlink(nm.buf);\n+\t\treftable_free(add->new_tables[i]);\n+\t\tadd->new_tables[i] = NULL;\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\tif (add->lock_file_fd > 0) {\n+\t\tclose(add->lock_file_fd);\n+\t\tadd->lock_file_fd = 0;\n+\t}\n+\tif (add->lock_file_name.len > 0) {\n+\t\tunlink(add->lock_file_name.buf);\n+\t\tstrbuf_release(&add->lock_file_name);\n+\t}\n+\n+\tstrbuf_release(&nm);\n+}\n+\n+void reftable_addition_destroy(struct reftable_addition *add)\n+{\n+\tif (!add) {\n+\t\treturn;\n+\t}\n+\treftable_addition_close(add);\n+\treftable_free(add);\n+}\n+\n+int reftable_addition_commit(struct reftable_addition *add)\n+{\n+\tstruct strbuf table_list = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\tif (add->new_tables_len == 0)\n+\t\tgoto done;\n+\n+\tfor (i = 0; i < add->stack->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->stack->readers[i]->name);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\tstrbuf_addstr(&table_list, add->new_tables[i]);\n+\t\tstrbuf_addstr(&table_list, \"\\n\");\n+\t}\n+\n+\terr = write(add->lock_file_fd, table_list.buf, table_list.len);\n+\tstrbuf_release(&table_list);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = close(add->lock_file_fd);\n+\tadd->lock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(add->lock_file_name.buf, add->stack->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\t/* success, no more state to clean up. */\n+\tstrbuf_release(&add->lock_file_name);\n+\tfor (i = 0; i < add->new_tables_len; i++) {\n+\t\treftable_free(add->new_tables[i]);\n+\t}\n+\treftable_free(add->new_tables);\n+\tadd->new_tables = NULL;\n+\tadd->new_tables_len = 0;\n+\n+\terr = reftable_stack_reload(add->stack);\n+done:\n+\treftable_addition_close(add);\n+\treturn err;\n+}\n+\n+int reftable_stack_new_addition(struct reftable_addition **dest,\n+\t\t\t\tstruct reftable_stack *st)\n+{\n+\tint err = 0;\n+\tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n+\t*dest = reftable_calloc(sizeof(**dest));\n+\t**dest = empty;\n+\terr = reftable_stack_init_addition(*dest, st);\n+\tif (err) {\n+\t\treftable_free(*dest);\n+\t\t*dest = NULL;\n+\t}\n+\treturn err;\n+}\n+\n+static int stack_try_add(struct reftable_stack *st,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct reftable_addition add = REFTABLE_ADDITION_INIT;\n+\tint err = reftable_stack_init_addition(&add, st);\n+\tif (err < 0)\n+\t\tgoto done;\n+\tif (err > 0) {\n+\t\terr = REFTABLE_LOCK_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_addition_add(&add, write_table, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_addition_commit(&add);\n+done:\n+\treftable_addition_close(&add);\n+\treturn err;\n+}\n+\n+int reftable_addition_add(struct reftable_addition *add,\n+\t\t\t int (*write_table)(struct reftable_writer *wr,\n+\t\t\t\t\t void *arg),\n+\t\t\t void *arg)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf tab_file_name = STRBUF_INIT;\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\tint tab_fd = 0;\n+\n+\tstrbuf_reset(&next_name);\n+\tformat_name(&next_name, add->next_update_index, add->next_update_index);\n+\n+\tstack_filename(&temp_tab_file_name, add->stack, next_name.buf);\n+\tstrbuf_addstr(&temp_tab_file_name, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab_file_name.buf);\n+\tif (tab_fd < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd,\n+\t\t\t\t &add->stack->config);\n+\terr = write_table(wr, arg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_writer_close(wr);\n+\tif (err == REFTABLE_EMPTY_TABLE_ERROR) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\terr = stack_check_addition(add->stack, temp_tab_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tif (wr->min_update_index < add->next_update_index) {\n+\t\terr = REFTABLE_API_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tformat_name(&next_name, wr->min_update_index, wr->max_update_index);\n+\tstrbuf_addstr(&next_name, \".ref\");\n+\n+\tstack_filename(&tab_file_name, add->stack, next_name.buf);\n+\n+\t/*\n+\t On windows, this relies on rand() picking a unique destination name.\n+\t Maybe we should do retry loop as well?\n+\t */\n+\terr = rename(temp_tab_file_name.buf, tab_file_name.buf);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tadd->new_tables = reftable_realloc(add->new_tables,\n+\t\t\t\t\t sizeof(*add->new_tables) *\n+\t\t\t\t\t\t (add->new_tables_len + 1));\n+\tadd->new_tables[add->new_tables_len] = strbuf_detach(&next_name, NULL);\n+\tadd->new_tables_len++;\n+done:\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (temp_tab_file_name.len > 0) {\n+\t\tunlink(temp_tab_file_name.buf);\n+\t}\n+\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&tab_file_name);\n+\tstrbuf_release(&next_name);\n+\treftable_writer_free(wr);\n+\treturn err;\n+}\n+\n+uint64_t reftable_stack_next_update_index(struct reftable_stack *st)\n+{\n+\tint sz = st->merged->stack_len;\n+\tif (sz > 0)\n+\t\treturn reftable_reader_max_update_index(st->readers[sz - 1]) +\n+\t\t 1;\n+\treturn 1;\n+}\n+\n+static int stack_compact_locked(struct reftable_stack *st, int first, int last,\n+\t\t\t\tstruct strbuf *temp_tab,\n+\t\t\t\tstruct reftable_log_expiry_config *config)\n+{\n+\tstruct strbuf next_name = STRBUF_INIT;\n+\tint tab_fd = -1;\n+\tstruct reftable_writer *wr = NULL;\n+\tint err = 0;\n+\n+\tformat_name(&next_name,\n+\t\t reftable_reader_min_update_index(st->readers[first]),\n+\t\t reftable_reader_max_update_index(st->readers[last]));\n+\n+\tstack_filename(temp_tab, st, next_name.buf);\n+\tstrbuf_addstr(temp_tab, \".temp.XXXXXX\");\n+\n+\ttab_fd = mkstemp(temp_tab->buf);\n+\twr = reftable_new_writer(reftable_fd_write, &tab_fd, &st->config);\n+\n+\terr = stack_write_compact(st, wr, first, last, config);\n+\tif (err < 0)\n+\t\tgoto done;\n+\terr = reftable_writer_close(wr);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = close(tab_fd);\n+\ttab_fd = 0;\n+\n+done:\n+\treftable_writer_free(wr);\n+\tif (tab_fd > 0) {\n+\t\tclose(tab_fd);\n+\t\ttab_fd = 0;\n+\t}\n+\tif (err != 0 && temp_tab->len > 0) {\n+\t\tunlink(temp_tab->buf);\n+\t\tstrbuf_release(temp_tab);\n+\t}\n+\tstrbuf_release(&next_name);\n+\treturn err;\n+}\n+\n+static int stack_write_compact(struct reftable_stack *st,\n+\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint subtabs_len = last - first + 1;\n+\tstruct reftable_table *subtabs = reftable_calloc(\n+\t\tsizeof(struct reftable_table) * (last - first + 1));\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tuint64_t entries = 0;\n+\n+\tint i = 0, j = 0;\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct reftable_reader *t = st->readers[i];\n+\t\treftable_table_from_reader(&subtabs[j++], t);\n+\t\tst->stats.bytes += t->size;\n+\t}\n+\treftable_writer_set_limits(wr, st->readers[first]->min_update_index,\n+\t\t\t\t st->readers[last]->max_update_index);\n+\n+\terr = reftable_new_merged_table(&mt, subtabs, subtabs_len,\n+\t\t\t\t\tst->config.hash_id);\n+\tif (err < 0) {\n+\t\treftable_free(subtabs);\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_merged_table_seek_ref(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (first == 0 && reftable_ref_record_is_deletion(&ref)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_ref(wr, &ref);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\terr = reftable_merged_table_seek_log(mt, &it, \"\");\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (first == 0 && reftable_log_record_is_deletion(&log)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config && config->min_update_index > 0 &&\n+\t\t log.update_index < config->min_update_index) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (config && config->time > 0 &&\n+\t\t log.update.time < config->time) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\terr = reftable_writer_add_log(wr, &log);\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tentries++;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\tif (mt) {\n+\t\tmerged_table_release(mt);\n+\t\treftable_merged_table_free(mt);\n+\t}\n+\treftable_ref_record_release(&ref);\n+\treftable_log_record_release(&log);\n+\tst->stats.entries_written += entries;\n+\treturn err;\n+}\n+\n+/* < 0: error. 0 == OK, > 0 attempt failed; could retry. */\n+static int stack_compact_range(struct reftable_stack *st, int first, int last,\n+\t\t\t struct reftable_log_expiry_config *expiry)\n+{\n+\tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n+\tstruct strbuf new_table_name = STRBUF_INIT;\n+\tstruct strbuf lock_file_name = STRBUF_INIT;\n+\tstruct strbuf ref_list_contents = STRBUF_INIT;\n+\tstruct strbuf new_table_path = STRBUF_INIT;\n+\tint err = 0;\n+\tint have_lock = 0;\n+\tint lock_file_fd = 0;\n+\tint compact_count = last - first + 1;\n+\tchar **listp = NULL;\n+\tchar **delete_on_success =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tchar **subtable_locks =\n+\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\tint i = 0;\n+\tint j = 0;\n+\tint is_empty_table = 0;\n+\n+\tif (first > last || (!expiry && first == last)) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+\tst->stats.attempts++;\n+\n+\tstrbuf_reset(&lock_file_name);\n+\tstrbuf_addstr(&lock_file_name, st->list_file);\n+\tstrbuf_addstr(&lock_file_name, \".lock\");\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\t/* Don't want to write to the lock for now. */\n+\tclose(lock_file_fd);\n+\tlock_file_fd = 0;\n+\n+\thave_lock = 1;\n+\terr = stack_uptodate(st);\n+\tif (err != 0)\n+\t\tgoto done;\n+\n+\tfor (i = first, j = 0; i <= last; i++) {\n+\t\tstruct strbuf subtab_file_name = STRBUF_INIT;\n+\t\tstruct strbuf subtab_lock = STRBUF_INIT;\n+\t\tint sublock_file_fd = -1;\n+\n+\t\tstack_filename(&subtab_file_name, st,\n+\t\t\t reader_name(st->readers[i]));\n+\n+\t\tstrbuf_reset(&subtab_lock);\n+\t\tstrbuf_addbuf(&subtab_lock, &subtab_file_name);\n+\t\tstrbuf_addstr(&subtab_lock, \".lock\");\n+\n+\t\tsublock_file_fd = open(subtab_lock.buf,\n+\t\t\t\t O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\t\tif (sublock_file_fd > 0) {\n+\t\t\tclose(sublock_file_fd);\n+\t\t} else if (sublock_file_fd < 0) {\n+\t\t\tif (errno == EEXIST) {\n+\t\t\t\terr = 1;\n+\t\t\t} else {\n+\t\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\t}\n+\t\t}\n+\n+\t\tsubtable_locks[j] = subtab_lock.buf;\n+\t\tdelete_on_success[j] = subtab_file_name.buf;\n+\t\tj++;\n+\n+\t\tif (err != 0)\n+\t\t\tgoto done;\n+\t}\n+\n+\terr = unlink(lock_file_name.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\thave_lock = 0;\n+\n+\terr = stack_compact_locked(st, first, last, &temp_tab_file_name,\n+\t\t\t\t expiry);\n+\t/* Compaction + tombstones can create an empty table out of non-empty\n+\t * tables. */\n+\tis_empty_table = (err == REFTABLE_EMPTY_TABLE_ERROR);\n+\tif (is_empty_table) {\n+\t\terr = 0;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tlock_file_fd =\n+\t\topen(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0644);\n+\tif (lock_file_fd < 0) {\n+\t\tif (errno == EEXIST) {\n+\t\t\terr = 1;\n+\t\t} else {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\thave_lock = 1;\n+\n+\tformat_name(&new_table_name, st->readers[first]->min_update_index,\n+\t\t st->readers[last]->max_update_index);\n+\tstrbuf_addstr(&new_table_name, \".ref\");\n+\n+\tstack_filename(&new_table_path, st, new_table_name.buf);\n+\n+\tif (!is_empty_table) {\n+\t\t/* retry? */\n+\t\terr = rename(temp_tab_file_name.buf, new_table_path.buf);\n+\t\tif (err < 0) {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < first; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tif (!is_empty_table) {\n+\t\tstrbuf_addbuf(&ref_list_contents, &new_table_name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\tfor (i = last + 1; i < st->merged->stack_len; i++) {\n+\t\tstrbuf_addstr(&ref_list_contents, st->readers[i]->name);\n+\t\tstrbuf_addstr(&ref_list_contents, \"\\n\");\n+\t}\n+\n+\terr = write(lock_file_fd, ref_list_contents.buf, ref_list_contents.len);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\terr = close(lock_file_fd);\n+\tlock_file_fd = 0;\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\n+\terr = rename(lock_file_name.buf, st->list_file);\n+\tif (err < 0) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tunlink(new_table_path.buf);\n+\t\tgoto done;\n+\t}\n+\thave_lock = 0;\n+\n+\t/* Reload the stack before deleting. On windows, we can only delete the\n+\t files after we closed them.\n+\t*/\n+\terr = reftable_stack_reload_maybe_reuse(st, first < last);\n+\n+\tlistp = delete_on_success;\n+\twhile (*listp) {\n+\t\tif (strcmp(*listp, new_table_path.buf)) {\n+\t\t\tunlink(*listp);\n+\t\t}\n+\t\tlistp++;\n+\t}\n+\n+done:\n+\tfree_names(delete_on_success);\n+\n+\tlistp = subtable_locks;\n+\twhile (*listp) {\n+\t\tunlink(*listp);\n+\t\tlistp++;\n+\t}\n+\tfree_names(subtable_locks);\n+\tif (lock_file_fd > 0) {\n+\t\tclose(lock_file_fd);\n+\t\tlock_file_fd = 0;\n+\t}\n+\tif (have_lock) {\n+\t\tunlink(lock_file_name.buf);\n+\t}\n+\tstrbuf_release(&new_table_name);\n+\tstrbuf_release(&new_table_path);\n+\tstrbuf_release(&ref_list_contents);\n+\tstrbuf_release(&temp_tab_file_name);\n+\tstrbuf_release(&lock_file_name);\n+\treturn err;\n+}\n+\n+int reftable_stack_compact_all(struct reftable_stack *st,\n+\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\treturn stack_compact_range(st, 0, st->merged->stack_len - 1, config);\n+}\n+\n+static int stack_compact_range_stats(struct reftable_stack *st, int first,\n+\t\t\t\t int last,\n+\t\t\t\t struct reftable_log_expiry_config *config)\n+{\n+\tint err = stack_compact_range(st, first, last, config);\n+\tif (err > 0) {\n+\t\tst->stats.failures++;\n+\t}\n+\treturn err;\n+}\n+\n+static int segment_size(struct segment *s)\n+{\n+\treturn s->end - s->start;\n+}\n+\n+int fastlog2(uint64_t sz)\n+{\n+\tint l = 0;\n+\tif (sz == 0)\n+\t\treturn 0;\n+\tfor (; sz; sz /= 2) {\n+\t\tl++;\n+\t}\n+\treturn l - 1;\n+}\n+\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n+{\n+\tstruct segment *segs = reftable_calloc(sizeof(struct segment) * n);\n+\tint next = 0;\n+\tstruct segment cur = { 0 };\n+\tint i = 0;\n+\n+\tif (n == 0) {\n+\t\t*seglen = 0;\n+\t\treturn segs;\n+\t}\n+\tfor (i = 0; i < n; i++) {\n+\t\tint log = fastlog2(sizes[i]);\n+\t\tif (cur.log != log && cur.bytes > 0) {\n+\t\t\tstruct segment fresh = {\n+\t\t\t\t.start = i,\n+\t\t\t};\n+\n+\t\t\tsegs[next++] = cur;\n+\t\t\tcur = fresh;\n+\t\t}\n+\n+\t\tcur.log = log;\n+\t\tcur.end = i + 1;\n+\t\tcur.bytes += sizes[i];\n+\t}\n+\tsegs[next++] = cur;\n+\t*seglen = next;\n+\treturn segs;\n+}\n+\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, sizes, n);\n+\tstruct segment min_seg = {\n+\t\t.log = 64,\n+\t};\n+\tint i = 0;\n+\tfor (i = 0; i < seglen; i++) {\n+\t\tif (segment_size(&segs[i]) == 1) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (segs[i].log < min_seg.log) {\n+\t\t\tmin_seg = segs[i];\n+\t\t}\n+\t}\n+\n+\twhile (min_seg.start > 0) {\n+\t\tint prev = min_seg.start - 1;\n+\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmin_seg.start = prev;\n+\t\tmin_seg.bytes += sizes[prev];\n+\t}\n+\n+\treftable_free(segs);\n+\treturn min_seg;\n+}\n+\n+static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes =\n+\t\treftable_calloc(sizeof(uint64_t) * st->merged->stack_len);\n+\tint version = (st->config.hash_id == GIT_SHA1_HASH_ID) ? 1 : 2;\n+\tint overhead = header_size(version) - 1;\n+\tint i = 0;\n+\tfor (i = 0; i < st->merged->stack_len; i++) {\n+\t\tsizes[i] = st->readers[i]->size - overhead;\n+\t}\n+\treturn sizes;\n+}\n+\n+int reftable_stack_auto_compact(struct reftable_stack *st)\n+{\n+\tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n+\tstruct segment seg =\n+\t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n+\treftable_free(sizes);\n+\tif (segment_size(&seg) > 0)\n+\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n+\t\t\t\t\t\t NULL);\n+\n+\treturn 0;\n+}\n+\n+struct reftable_compaction_stats *\n+reftable_stack_compaction_stats(struct reftable_stack *st)\n+{\n+\treturn &st->stats;\n+}\n+\n+int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_ref_record *ref)\n+{\n+\tstruct reftable_table tab = { NULL };\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\treturn reftable_table_read_ref(&tab, refname, ref);\n+}\n+\n+int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n+\t\t\t struct reftable_log_record *log)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(st);\n+\tint err = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = reftable_iterator_next_log(&it, log);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tif (strcmp(log->refname, refname) ||\n+\t reftable_log_record_is_deletion(log)) {\n+\t\terr = 1;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\tif (err) {\n+\t\treftable_log_record_release(log);\n+\t}\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int stack_check_addition(struct reftable_stack *st,\n+\t\t\t\tconst char *new_tab_name)\n+{\n+\tint err = 0;\n+\tstruct reftable_block_source src = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct reftable_ref_record *refs = NULL;\n+\tstruct reftable_iterator it = { NULL };\n+\tint cap = 0;\n+\tint len = 0;\n+\tint i = 0;\n+\n+\tif (st->config.skip_name_check)\n+\t\treturn 0;\n+\n+\terr = reftable_block_source_from_file(&src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&rd, &src, new_tab_name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_reader_seek_ref(rd, &it, \"\");\n+\tif (err > 0) {\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (len >= cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\trefs = reftable_realloc(refs, cap * sizeof(refs[0]));\n+\t\t}\n+\n+\t\trefs[len++] = ref;\n+\t}\n+\n+\treftable_table_from_merged_table(&tab, reftable_stack_merged_table(st));\n+\n+\terr = validate_ref_record_addition(tab, refs, len);\n+\n+done:\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t}\n+\n+\tfree(refs);\n+\treftable_iterator_destroy(&it);\n+\treftable_reader_free(rd);\n+\treturn err;\n+}\n+\n+static int is_table_name(const char *s)\n+{\n+\tconst char *dot = strrchr(s, '.');\n+\treturn dot && !strcmp(dot, \".ref\");\n+}\n+\n+static void remove_maybe_stale_table(struct reftable_stack *st, uint64_t max,\n+\t\t\t\t const char *name)\n+{\n+\tint err = 0;\n+\tuint64_t update_idx = 0;\n+\tstruct reftable_block_source src = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct strbuf table_path = STRBUF_INIT;\n+\tstack_filename(&table_path, st, name);\n+\n+\terr = reftable_block_source_from_file(&src, table_path.buf);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&rd, &src, name);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tupdate_idx = reftable_reader_max_update_index(rd);\n+\treftable_reader_free(rd);\n+\n+\tif (update_idx <= max) {\n+\t\tunlink(table_path.buf);\n+\t}\n+done:\n+\tstrbuf_release(&table_path);\n+}\n+\n+static int reftable_stack_clean_locked(struct reftable_stack *st)\n+{\n+\tuint64_t max = reftable_merged_table_max_update_index(\n+\t\treftable_stack_merged_table(st));\n+\tDIR *dir = opendir(st->reftable_dir);\n+\tstruct dirent *d = NULL;\n+\tif (!dir) {\n+\t\treturn REFTABLE_IO_ERROR;\n+\t}\n+\n+\twhile ((d = readdir(dir))) {\n+\t\tint i = 0;\n+\t\tint found = 0;\n+\t\tif (!is_table_name(d->d_name))\n+\t\t\tcontinue;\n+\n+\t\tfor (i = 0; !found && i < st->readers_len; i++) {\n+\t\t\tfound = !strcmp(reader_name(st->readers[i]), d->d_name);\n+\t\t}\n+\t\tif (found)\n+\t\t\tcontinue;\n+\n+\t\tremove_maybe_stale_table(st, max, d->d_name);\n+\t}\n+\n+\tclosedir(dir);\n+\treturn 0;\n+}\n+\n+int reftable_stack_clean(struct reftable_stack *st)\n+{\n+\tstruct reftable_addition *add = NULL;\n+\tint err = reftable_stack_new_addition(&add, st);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(st);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_clean_locked(st);\n+\n+done:\n+\treftable_addition_destroy(add);\n+\treturn err;\n+}\n+\n+int reftable_stack_print_directory(const char *stackdir)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_merged_table *merged = NULL;\n+\tstruct reftable_table table = { NULL };\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tmerged = reftable_stack_merged_table(stack);\n+\treftable_table_from_merged_table(&table, merged);\n+\terr = reftable_table_print(&table);\n+done:\n+\treftable_stack_destroy(stack);\n+\treturn err;\n+}\ndiff --git a/reftable/stack.h b/reftable/stack.h\nnew file mode 100644\nindex 000000000000..f57005846e56\n--- /dev/null\n+++ b/reftable/stack.h\n@@ -0,0 +1,41 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef STACK_H\n+#define STACK_H\n+\n+#include \"system.h\"\n+#include \"reftable-writer.h\"\n+#include \"reftable-stack.h\"\n+\n+struct reftable_stack {\n+\tchar *list_file;\n+\tchar *reftable_dir;\n+\tint disable_auto_compact;\n+\n+\tstruct reftable_write_options config;\n+\n+\tstruct reftable_reader **readers;\n+\tsize_t readers_len;\n+\tstruct reftable_merged_table *merged;\n+\tstruct reftable_compaction_stats stats;\n+};\n+\n+int read_lines(const char *filename, char ***lines);\n+\n+struct segment {\n+\tint start, end;\n+\tint log;\n+\tuint64_t bytes;\n+};\n+\n+int fastlog2(uint64_t sz);\n+struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n);\n+struct segment suggest_compaction_segment(uint64_t *sizes, int n);\n+\n+#endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nnew file mode 100644\nindex 000000000000..3bd78d2a2bc5\n--- /dev/null\n+++ b/reftable/stack_test.c\n@@ -0,0 +1,940 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"stack.h\"\n+\n+#include \"system.h\"\n+\n+#include \"reftable-reader.h\"\n+#include \"merged.h\"\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+#include \n+#include \n+\n+static void clear_dir(const char *dirname)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstrbuf_addstr(&path, dirname);\n+\tremove_dir_recursively(&path, 0);\n+\tstrbuf_release(&path);\n+}\n+\n+static int count_dir_entries(const char *dirname)\n+{\n+\tDIR *dir = opendir(dirname);\n+\tint len = 0;\n+\tstruct dirent *d;\n+\tif (dir == NULL)\n+\t\treturn 0;\n+\n+\twhile ((d = readdir(dir))) {\n+\t\tif (!strcmp(d->d_name, \"..\") || !strcmp(d->d_name, \".\"))\n+\t\t\tcontinue;\n+\t\tlen++;\n+\t}\n+\tclosedir(dir);\n+\treturn len;\n+}\n+\n+static char *get_tmp_template(const char *prefix)\n+{\n+\tconst char *tmp = getenv(\"TMPDIR\");\n+\tstatic char template[1024];\n+\tsnprintf(template, sizeof(template) - 1, \"%s/%s.XXXXXX\",\n+\t\t tmp ? tmp : \"/tmp\", prefix);\n+\treturn template;\n+}\n+\n+static char *get_tmp_dir(const char *prefix)\n+{\n+\tchar *dir = get_tmp_template(prefix);\n+\tEXPECT(mkdtemp(dir));\n+\treturn dir;\n+}\n+\n+static void test_read_file(void)\n+{\n+\tchar *fn = get_tmp_template(__FUNCTION__);\n+\tint fd = mkstemp(fn);\n+\tchar out[1024] = \"line1\\n\\nline2\\nline3\";\n+\tint n, err;\n+\tchar **names = NULL;\n+\tchar *want[] = { \"line1\", \"line2\", \"line3\" };\n+\tint i = 0;\n+\n+\tEXPECT(fd > 0);\n+\tn = write(fd, out, strlen(out));\n+\tEXPECT(n == strlen(out));\n+\terr = close(fd);\n+\tEXPECT(err >= 0);\n+\n+\terr = read_lines(fn, &names);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; names[i]; i++) {\n+\t\tEXPECT(0 == strcmp(want[i], names[i]));\n+\t}\n+\tfree_names(names);\n+\tremove(fn);\n+}\n+\n+static void test_parse_names(void)\n+{\n+\tchar buf[] = \"line\\n\";\n+\tchar **names = NULL;\n+\tparse_names(buf, strlen(buf), &names);\n+\n+\tEXPECT(NULL != names[0]);\n+\tEXPECT(0 == strcmp(names[0], \"line\"));\n+\tEXPECT(NULL == names[1]);\n+\tfree_names(names);\n+}\n+\n+static void test_names_equal(void)\n+{\n+\tchar *a[] = { \"a\", \"b\", \"c\", NULL };\n+\tchar *b[] = { \"a\", \"b\", \"d\", NULL };\n+\tchar *c[] = { \"a\", \"b\", NULL };\n+\n+\tEXPECT(names_equal(a, a));\n+\tEXPECT(!names_equal(a, b));\n+\tEXPECT(!names_equal(a, c));\n+}\n+\n+static int write_test_ref(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct reftable_ref_record *ref = arg;\n+\treftable_writer_set_limits(wr, ref->update_index, ref->update_index);\n+\treturn reftable_writer_add_ref(wr, ref);\n+}\n+\n+struct write_log_arg {\n+\tstruct reftable_log_record *log;\n+\tuint64_t update_index;\n+};\n+\n+static int write_test_log(struct reftable_writer *wr, void *arg)\n+{\n+\tstruct write_log_arg *wla = arg;\n+\n+\treftable_writer_set_limits(wr, wla->update_index, wla->update_index);\n+\treturn reftable_writer_add_log(wr, wla->log);\n+}\n+\n+static void test_reftable_stack_add_one(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(\"master\", dest.value.symref));\n+\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_uptodate(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL;\n+\tstruct reftable_stack *st2 = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"branch2\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\n+\n+\t/* simulate multi-process access to the same stack\n+\t by creating two stacks for the same directory.\n+\t */\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st1, &write_test_ref, &ref1);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tEXPECT(err == REFTABLE_LOCK_ERROR);\n+\n+\terr = reftable_stack_reload(st2);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st2, &write_test_ref, &ref2);\n+\tEXPECT_ERR(err);\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_transaction_api(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_addition *add = NULL;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_new_addition(&add, st);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_addition_add(add, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_addition_commit(add);\n+\tEXPECT_ERR(err);\n+\n+\treftable_addition_destroy(add);\n+\n+\terr = reftable_stack_read_ref(st, ref.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(REFTABLE_REF_SYMREF == dest.value_type);\n+\tEXPECT(0 == strcmp(\"master\", dest.value.symref));\n+\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_validate_refname(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint i;\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"a/b\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tchar *additions[] = { \"a\", \"a/b/c\" };\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(additions); i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = additions[i],\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tEXPECT(err == REFTABLE_NAME_CONFLICT);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static int write_error(struct reftable_writer *wr, void *arg)\n+{\n+\treturn *((int *)arg);\n+}\n+\n+static void test_reftable_stack_update_index_check(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record ref1 = {\n+\t\t.refname = \"name1\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\tstruct reftable_ref_record ref2 = {\n+\t\t.refname = \"name2\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"master\",\n+\t};\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref1);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref2);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_lock_failure(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err, i;\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tfor (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) {\n+\t\terr = reftable_stack_add(st, &write_error, &i);\n+\t\tEXPECT(err == i);\n+\t}\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_add(void)\n+{\n+\tint i = 0;\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t.exact_log_message = 1,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tst->disable_auto_compact = 1;\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n+\t\trefs[i].value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\tset_test_hash(refs[i].value.val1, i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = N + i + 1;\n+\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\n+\t\tlogs[i].update.new_hash = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].update.new_hash, i);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_ref_record dest = { NULL };\n+\n+\t\tint err = reftable_stack_read_ref(st, refs[i].refname, &dest);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(reftable_ref_record_equal(&dest, refs + i,\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\treftable_ref_record_release(&dest);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct reftable_log_record dest = { NULL };\n+\t\tint err = reftable_stack_read_log(st, refs[i].refname, &dest);\n+\t\tEXPECT_ERR(err);\n+\t\tEXPECT(reftable_log_record_equal(&dest, logs + i,\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\treftable_log_record_release(&dest);\n+\t}\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_log_normalize(void)\n+{\n+\tint err = 0;\n+\tstruct reftable_write_options cfg = {\n+\t\t0,\n+\t};\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\n+\tuint8_t h1[GIT_SHA1_RAWSZ] = { 0x01 }, h2[GIT_SHA1_RAWSZ] = { 0x02 };\n+\n+\tstruct reftable_log_record input = { .refname = \"branch\",\n+\t\t\t\t\t .update_index = 1,\n+\t\t\t\t\t .value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t\t\t .update = {\n+\t\t\t\t\t\t .new_hash = h1,\n+\t\t\t\t\t\t .old_hash = h2,\n+\t\t\t\t\t } };\n+\tstruct reftable_log_record dest = {\n+\t\t.update_index = 0,\n+\t};\n+\tstruct write_log_arg arg = {\n+\t\t.log = &input,\n+\t\t.update_index = 1,\n+\t};\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tinput.update.message = \"one\\ntwo\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT(err == REFTABLE_API_ERROR);\n+\n+\tinput.update.message = \"one\";\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(dest.update.message, \"one\\n\"));\n+\n+\tinput.update.message = \"two\\n\";\n+\targ.update_index = 2;\n+\terr = reftable_stack_add(st, &write_test_log, &arg);\n+\tEXPECT_ERR(err);\n+\terr = reftable_stack_read_log(st, input.refname, &dest);\n+\tEXPECT_ERR(err);\n+\tEXPECT(0 == strcmp(dest.update.message, \"two\\n\"));\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\treftable_log_record_release(&dest);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_tombstone(void)\n+{\n+\tint i = 0;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tstruct reftable_ref_record refs[2] = { { NULL } };\n+\tstruct reftable_log_record logs[2] = { { NULL } };\n+\tint N = ARRAY_SIZE(refs);\n+\tstruct reftable_ref_record dest = { NULL };\n+\tstruct reftable_log_record log_dest = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\t/* even entries add the refs, odd entries delete them. */\n+\tfor (i = 0; i < N; i++) {\n+\t\tconst char *buf = \"branch\";\n+\t\trefs[i].refname = xstrdup(buf);\n+\t\trefs[i].update_index = i + 1;\n+\t\tif (i % 2 == 0) {\n+\t\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n+\t\t\trefs[i].value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_test_hash(refs[i].value.val1, i);\n+\t\t}\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\t/* update_index is part of the key. */\n+\t\tlogs[i].update_index = 42;\n+\t\tif (i % 2 == 0) {\n+\t\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\t\t\tlogs[i].update.new_hash =\n+\t\t\t\treftable_malloc(GIT_SHA1_RAWSZ);\n+\t\t\tset_test_hash(logs[i].update.new_hash, i);\n+\t\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n+\t\t}\n+\t}\n+\tfor (i = 0; i < N; i++) {\n+\t\tint err = reftable_stack_add(st, &write_test_ref, &refs[i]);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tEXPECT(err == 1);\n+\treftable_ref_record_release(&dest);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tEXPECT(err == 1);\n+\treftable_log_record_release(&log_dest);\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st, \"branch\", &dest);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, \"branch\", &log_dest);\n+\tEXPECT(err == 1);\n+\treftable_ref_record_release(&dest);\n+\treftable_log_record_release(&log_dest);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_ref_record_release(&refs[i]);\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_hash_id(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = \"master\",\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"target\",\n+\t\t.update_index = 1,\n+\t};\n+\tstruct reftable_write_options cfg32 = { .hash_id = GIT_SHA256_HASH_ID };\n+\tstruct reftable_stack *st32 = NULL;\n+\tstruct reftable_write_options cfg_default = { 0 };\n+\tstruct reftable_stack *st_default = NULL;\n+\tstruct reftable_ref_record dest = { NULL };\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\tEXPECT_ERR(err);\n+\n+\t/* can't read it with the wrong hash ID. */\n+\terr = reftable_new_stack(&st32, dir, cfg32);\n+\tEXPECT(err == REFTABLE_FORMAT_ERROR);\n+\n+\t/* check that we can read it back with default config too. */\n+\terr = reftable_new_stack(&st_default, dir, cfg_default);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n+\tEXPECT_ERR(err);\n+\n+\tEXPECT(reftable_ref_record_equal(&ref, &dest, GIT_SHA1_RAWSZ));\n+\treftable_ref_record_release(&dest);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st_default);\n+\tclear_dir(dir);\n+}\n+\n+static void test_log2(void)\n+{\n+\tEXPECT(1 == fastlog2(3));\n+\tEXPECT(2 == fastlog2(4));\n+\tEXPECT(2 == fastlog2(5));\n+}\n+\n+static void test_sizes_to_segments(void)\n+{\n+\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n+\t/* .................0 1 2 3 4 5 */\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(segs[2].log == 3);\n+\tEXPECT(segs[2].start == 5);\n+\tEXPECT(segs[2].end == 6);\n+\n+\tEXPECT(segs[1].log == 2);\n+\tEXPECT(segs[1].start == 2);\n+\tEXPECT(segs[1].end == 5);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_empty(void)\n+{\n+\tint seglen = 0;\n+\tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n+\tEXPECT(seglen == 0);\n+\treftable_free(segs);\n+}\n+\n+static void test_sizes_to_segments_all_equal(void)\n+{\n+\tuint64_t sizes[] = { 5, 5 };\n+\n+\tint seglen = 0;\n+\tstruct segment *segs =\n+\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(seglen == 1);\n+\tEXPECT(segs[0].start == 0);\n+\tEXPECT(segs[0].end == 2);\n+\treftable_free(segs);\n+}\n+\n+static void test_suggest_compaction_segment(void)\n+{\n+\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n+\t/* .................0 1 2 3 4 5 6 */\n+\tstruct segment min =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(min.start == 2);\n+\tEXPECT(min.end == 7);\n+}\n+\n+static void test_suggest_compaction_segment_nothing(void)\n+{\n+\tuint64_t sizes[] = { 64, 32, 16, 8, 4, 2 };\n+\tstruct segment result =\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\tEXPECT(result.start == result.end);\n+}\n+\n+static void test_reflog_expire(void)\n+{\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tstruct reftable_log_record logs[20] = { { NULL } };\n+\tint N = ARRAY_SIZE(logs) - 1;\n+\tint i = 0;\n+\tint err;\n+\tstruct reftable_log_expiry_config expiry = {\n+\t\t.time = 10,\n+\t};\n+\tstruct reftable_log_record log = { NULL };\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tchar buf[256];\n+\t\tsnprintf(buf, sizeof(buf), \"branch%02d\", i);\n+\n+\t\tlogs[i].refname = xstrdup(buf);\n+\t\tlogs[i].update_index = i;\n+\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n+\t\tlogs[i].update.time = i;\n+\t\tlogs[i].update.new_hash = reftable_malloc(GIT_SHA1_RAWSZ);\n+\t\tlogs[i].update.email = xstrdup(\"identity@invalid\");\n+\t\tset_test_hash(logs[i].update.new_hash, i);\n+\t}\n+\n+\tfor (i = 1; i <= N; i++) {\n+\t\tstruct write_log_arg arg = {\n+\t\t\t.log = &logs[i],\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t};\n+\t\tint err = reftable_stack_add(st, &write_test_log, &arg);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_stack_compact_all(st, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, logs[9].refname, &log);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[11].refname, &log);\n+\tEXPECT_ERR(err);\n+\n+\texpiry.min_update_index = 15;\n+\terr = reftable_stack_compact_all(st, &expiry);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_read_log(st, logs[14].refname, &log);\n+\tEXPECT(err == 1);\n+\n+\terr = reftable_stack_read_log(st, logs[16].refname, &log);\n+\tEXPECT_ERR(err);\n+\n+\t/* cleanup */\n+\treftable_stack_destroy(st);\n+\tfor (i = 0; i <= N; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tclear_dir(dir);\n+\treftable_log_record_release(&log);\n+}\n+\n+static int write_nothing(struct reftable_writer *wr, void *arg)\n+{\n+\treftable_writer_set_limits(wr, 1, 1);\n+\treturn 0;\n+}\n+\n+static void test_empty_add(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tint err;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tstruct reftable_stack *st2 = NULL;\n+\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_add(st, &write_nothing, NULL);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\tclear_dir(dir);\n+\treftable_stack_destroy(st);\n+\treftable_stack_destroy(st2);\n+}\n+\n+static void test_reftable_stack_auto_compaction(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err, i;\n+\tint N = 100;\n+\n+\terr = reftable_new_stack(&st, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\n+\t\tEXPECT(i < 3 || st->merged->stack_len < 2 * fastlog2(i));\n+\t}\n+\n+\tEXPECT(reftable_stack_compaction_stats(st)->entries_written <\n+\t (uint64_t)(N * fastlog2(N)));\n+\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n+\n+static void test_reftable_stack_compaction_concurrent(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL, *st2 = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err, i;\n+\tint N = 3;\n+\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st1),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st1, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st1, NULL);\n+\tEXPECT_ERR(err);\n+\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\n+\tEXPECT(count_dir_entries(dir) == 2);\n+\tclear_dir(dir);\n+}\n+\n+static void unclean_stack_close(struct reftable_stack *st)\n+{\n+\t// break abstraction boundary to simulate unclean shutdown.\n+\tint i = 0;\n+\tfor (; i < st->readers_len; i++) {\n+\t\treftable_reader_free(st->readers[i]);\n+\t}\n+\tst->readers_len = 0;\n+\tFREE_AND_NULL(st->readers);\n+}\n+\n+static void test_reftable_stack_compaction_concurrent_clean(void)\n+{\n+\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_stack *st1 = NULL, *st2 = NULL, *st3 = NULL;\n+\tchar *dir = get_tmp_dir(__FUNCTION__);\n+\n+\tint err, i;\n+\tint N = 3;\n+\n+\terr = reftable_new_stack(&st1, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = name,\n+\t\t\t.update_index = reftable_stack_next_update_index(st1),\n+\t\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t\t.value.symref = \"master\",\n+\t\t};\n+\t\tsnprintf(name, sizeof(name), \"branch%04d\", i);\n+\n+\t\terr = reftable_stack_add(st1, &write_test_ref, &ref);\n+\t\tEXPECT_ERR(err);\n+\t}\n+\n+\terr = reftable_new_stack(&st2, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_compact_all(st1, NULL);\n+\tEXPECT_ERR(err);\n+\n+\tunclean_stack_close(st1);\n+\tunclean_stack_close(st2);\n+\n+\terr = reftable_new_stack(&st3, dir, cfg);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_stack_clean(st3);\n+\tEXPECT_ERR(err);\n+\tEXPECT(count_dir_entries(dir) == 2);\n+\n+\treftable_stack_destroy(st1);\n+\treftable_stack_destroy(st2);\n+\treftable_stack_destroy(st3);\n+\n+\tclear_dir(dir);\n+}\n+\n+int stack_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_empty_add);\n+\tRUN_TEST(test_log2);\n+\tRUN_TEST(test_names_equal);\n+\tRUN_TEST(test_parse_names);\n+\tRUN_TEST(test_read_file);\n+\tRUN_TEST(test_reflog_expire);\n+\tRUN_TEST(test_reftable_stack_add);\n+\tRUN_TEST(test_reftable_stack_add_one);\n+\tRUN_TEST(test_reftable_stack_auto_compaction);\n+\tRUN_TEST(test_reftable_stack_compaction_concurrent);\n+\tRUN_TEST(test_reftable_stack_compaction_concurrent_clean);\n+\tRUN_TEST(test_reftable_stack_hash_id);\n+\tRUN_TEST(test_reftable_stack_lock_failure);\n+\tRUN_TEST(test_reftable_stack_log_normalize);\n+\tRUN_TEST(test_reftable_stack_tombstone);\n+\tRUN_TEST(test_reftable_stack_transaction_api);\n+\tRUN_TEST(test_reftable_stack_update_index_check);\n+\tRUN_TEST(test_reftable_stack_uptodate);\n+\tRUN_TEST(test_reftable_stack_validate_refname);\n+\tRUN_TEST(test_sizes_to_segments);\n+\tRUN_TEST(test_sizes_to_segments_all_equal);\n+\tRUN_TEST(test_sizes_to_segments_empty);\n+\tRUN_TEST(test_suggest_compaction_segment);\n+\tRUN_TEST(test_suggest_compaction_segment_nothing);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex c8db6852c353..996da85f7b5c 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -10,6 +10,7 @@ int cmd__reftable(int argc, const char **argv)\n \trecord_test_main(argc, argv);\n \trefname_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n+\tstack_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"422285","messageId":"7913027063a269c524acffb9aafe7f6754094dd1.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 20/28] reftable: implement refname validation","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:48Z","receivedAt":"2021-04-19T11:40:46Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThe packed/loose format has restrictions on refnames: a and a/b cannot\ncoexist. This limitation does not apply to reftable per se, but must be\nmaintained for interoperability. This code adds validation routines to\nabort transactions that are trying to add invalid names.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n reftable/refname.c | 209 +++++++++++++++++++++++++++++++++++++++\n reftable/refname.h | 29 ++++++\n reftable/refname_test.c | 102 +++++++++++++++++++\n t/helper/test-reftable.c | 1 +\n 5 files changed, 342 insertions(+)\n create mode 100644 reftable/refname.c\n create mode 100644 reftable/refname.h\n create mode 100644 reftable/refname_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 6139f02faabe..b3973d8f5a8f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2430,6 +2430,7 @@ REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n+REFTABLE_TEST_OBJS += reftable/refname_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\n REFTABLE_TEST_OBJS += reftable/tree_test.o\n \ndiff --git a/reftable/refname.c b/reftable/refname.c\nnew file mode 100644\nindex 000000000000..957349693247\n--- /dev/null\n+++ b/reftable/refname.c\n@@ -0,0 +1,209 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+#include \"reftable-error.h\"\n+#include \"basics.h\"\n+#include \"refname.h\"\n+#include \"reftable-iterator.h\"\n+\n+struct find_arg {\n+\tchar **names;\n+\tconst char *want;\n+};\n+\n+static int find_name(size_t k, void *arg)\n+{\n+\tstruct find_arg *f_arg = arg;\n+\treturn strcmp(f_arg->names[k], f_arg->want) >= 0;\n+}\n+\n+static int modification_has_ref(struct modification *mod, const char *name)\n+{\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len && !strcmp(mod->add[idx], name)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tif (mod->del_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->del,\n+\t\t\t.want = name,\n+\t\t};\n+\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\tif (idx < mod->del_len && !strcmp(mod->del[idx], name)) {\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\terr = reftable_table_read_ref(&mod->tab, name, &ref);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+static void modification_release(struct modification *mod)\n+{\n+\t/* don't delete the strings themselves; they're owned by ref records.\n+\t */\n+\tFREE_AND_NULL(mod->add);\n+\tFREE_AND_NULL(mod->del);\n+\tmod->add_len = 0;\n+\tmod->del_len = 0;\n+}\n+\n+static int modification_has_ref_with_prefix(struct modification *mod,\n+\t\t\t\t\t const char *prefix)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\n+\tif (mod->add_len > 0) {\n+\t\tstruct find_arg arg = {\n+\t\t\t.names = mod->add,\n+\t\t\t.want = prefix,\n+\t\t};\n+\t\tint idx = binsearch(mod->add_len, find_name, &arg);\n+\t\tif (idx < mod->add_len &&\n+\t\t !strncmp(prefix, mod->add[idx], strlen(prefix)))\n+\t\t\tgoto done;\n+\t}\n+\terr = reftable_table_seek_ref(&mod->tab, &it, prefix);\n+\tif (err)\n+\t\tgoto done;\n+\n+\twhile (1) {\n+\t\terr = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\n+\t\tif (mod->del_len > 0) {\n+\t\t\tstruct find_arg arg = {\n+\t\t\t\t.names = mod->del,\n+\t\t\t\t.want = ref.refname,\n+\t\t\t};\n+\t\t\tint idx = binsearch(mod->del_len, find_name, &arg);\n+\t\t\tif (idx < mod->del_len &&\n+\t\t\t !strcmp(ref.refname, mod->del[idx])) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (strncmp(ref.refname, prefix, strlen(prefix))) {\n+\t\t\terr = 1;\n+\t\t\tgoto done;\n+\t\t}\n+\t\terr = 0;\n+\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_ref_record_release(&ref);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int validate_refname(const char *name)\n+{\n+\twhile (1) {\n+\t\tchar *next = strchr(name, '/');\n+\t\tif (!*name) {\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\t}\n+\t\tif (!next) {\n+\t\t\treturn 0;\n+\t\t}\n+\t\tif (next - name == 0 || (next - name == 1 && *name == '.') ||\n+\t\t (next - name == 2 && name[0] == '.' && name[1] == '.'))\n+\t\t\treturn REFTABLE_REFNAME_ERROR;\n+\t\tname = next + 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz)\n+{\n+\tstruct modification mod = {\n+\t\t.tab = tab,\n+\t\t.add = reftable_calloc(sizeof(char *) * sz),\n+\t\t.del = reftable_calloc(sizeof(char *) * sz),\n+\t};\n+\tint i = 0;\n+\tint err = 0;\n+\tfor (; i < sz; i++) {\n+\t\tif (reftable_ref_record_is_deletion(&recs[i])) {\n+\t\t\tmod.del[mod.del_len++] = recs[i].refname;\n+\t\t} else {\n+\t\t\tmod.add[mod.add_len++] = recs[i].refname;\n+\t\t}\n+\t}\n+\n+\terr = modification_validate(&mod);\n+\tmodification_release(&mod);\n+\treturn err;\n+}\n+\n+static void strbuf_trim_component(struct strbuf *sl)\n+{\n+\twhile (sl->len > 0) {\n+\t\tint is_slash = (sl->buf[sl->len - 1] == '/');\n+\t\tstrbuf_setlen(sl, sl->len - 1);\n+\t\tif (is_slash)\n+\t\t\tbreak;\n+\t}\n+}\n+\n+int modification_validate(struct modification *mod)\n+{\n+\tstruct strbuf slashed = STRBUF_INIT;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (; i < mod->add_len; i++) {\n+\t\terr = validate_refname(mod->add[i]);\n+\t\tif (err)\n+\t\t\tgoto done;\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\tstrbuf_addstr(&slashed, \"/\");\n+\n+\t\terr = modification_has_ref_with_prefix(mod, slashed.buf);\n+\t\tif (err == 0) {\n+\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\tgoto done;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tstrbuf_reset(&slashed);\n+\t\tstrbuf_addstr(&slashed, mod->add[i]);\n+\t\twhile (slashed.len) {\n+\t\t\tstrbuf_trim_component(&slashed);\n+\t\t\terr = modification_has_ref(mod, slashed.buf);\n+\t\t\tif (err == 0) {\n+\t\t\t\terr = REFTABLE_NAME_CONFLICT;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\terr = 0;\n+done:\n+\tstrbuf_release(&slashed);\n+\treturn err;\n+}\ndiff --git a/reftable/refname.h b/reftable/refname.h\nnew file mode 100644\nindex 000000000000..a24b40fcb428\n--- /dev/null\n+++ b/reftable/refname.h\n@@ -0,0 +1,29 @@\n+/*\n+ Copyright 2020 Google LLC\n+\n+ Use of this source code is governed by a BSD-style\n+ license that can be found in the LICENSE file or at\n+ https://developers.google.com/open-source/licenses/bsd\n+*/\n+#ifndef REFNAME_H\n+#define REFNAME_H\n+\n+#include \"reftable-record.h\"\n+#include \"reftable-generic.h\"\n+\n+struct modification {\n+\tstruct reftable_table tab;\n+\n+\tchar **add;\n+\tsize_t add_len;\n+\n+\tchar **del;\n+\tsize_t del_len;\n+};\n+\n+int validate_ref_record_addition(struct reftable_table tab,\n+\t\t\t\t struct reftable_ref_record *recs, size_t sz);\n+\n+int modification_validate(struct modification *mod);\n+\n+#endif\ndiff --git a/reftable/refname_test.c b/reftable/refname_test.c\nnew file mode 100644\nindex 000000000000..8645cd93bbd8\n--- /dev/null\n+++ b/reftable/refname_test.c\n@@ -0,0 +1,102 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"basics.h\"\n+#include \"block.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"refname.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-writer.h\"\n+#include \"system.h\"\n+\n+#include \"test_framework.h\"\n+#include \"reftable-tests.h\"\n+\n+struct testcase {\n+\tchar *add;\n+\tchar *del;\n+\tint error_code;\n+};\n+\n+static void test_conflict(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"a/b\",\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = \"destination\", /* make sure it's not a symref.\n+\t\t\t\t\t\t*/\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tint i;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_table tab = { NULL };\n+\tstruct testcase cases[] = {\n+\t\t{ \"a/b/c\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"b\", NULL, 0 },\n+\t\t{ \"a\", NULL, REFTABLE_NAME_CONFLICT },\n+\t\t{ \"a\", \"a/b\", 0 },\n+\n+\t\t{ \"p/\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p//q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/./q\", NULL, REFTABLE_REFNAME_ERROR },\n+\t\t{ \"p/../q\", NULL, REFTABLE_REFNAME_ERROR },\n+\n+\t\t{ \"a/b/c\", \"a/b\", 0 },\n+\t\t{ NULL, \"a//b\", 0 },\n+\t};\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\treftable_table_from_reader(&tab, rd);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(cases); i++) {\n+\t\tstruct modification mod = {\n+\t\t\t.tab = tab,\n+\t\t};\n+\n+\t\tif (cases[i].add) {\n+\t\t\tmod.add = &cases[i].add;\n+\t\t\tmod.add_len = 1;\n+\t\t}\n+\t\tif (cases[i].del) {\n+\t\t\tmod.del = &cases[i].del;\n+\t\t\tmod.del_len = 1;\n+\t\t}\n+\n+\t\terr = modification_validate(&mod);\n+\t\tEXPECT(err == cases[i].error_code);\n+\t}\n+\n+\treftable_reader_free(rd);\n+\tstrbuf_release(&buf);\n+}\n+\n+int refname_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_conflict);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 8087f2da4e6a..c8db6852c353 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -8,6 +8,7 @@ int cmd__reftable(int argc, const char **argv)\n \tmerged_test_main(argc, argv);\n \tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n+\trefname_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n \treturn 0;\n-- \ngitgitgadget\n\n"},{"id":"422287","messageId":"e3a776f2076cad992eecc22678a01a26e51c59b7.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 19/28] reftable: add merged table view","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:47Z","receivedAt":"2021-04-19T11:40:53Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis adds an abstract, read-only interface to the ref database.\n\nThis primitive is used to construct the read view of the ref database\n(the read view is constructed by merging several *.ref files). It also\nprovides the mechanism to provide a unified view of the refs in the main\nrepository and the per-worktree refs.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/merged.c | 362 +++++++++++++++++++++++++++++++++++++\n reftable/merged.h | 35 ++++\n reftable/merged_test.c | 292 ++++++++++++++++++++++++++++++\n reftable/reftable-merged.h | 72 ++++++++\n t/helper/test-reftable.c | 1 +\n 6 files changed, 764 insertions(+)\n create mode 100644 reftable/merged.c\n create mode 100644 reftable/merged.h\n create mode 100644 reftable/merged_test.c\n create mode 100644 reftable/reftable-merged.h\n\ndiff --git a/Makefile b/Makefile\nindex 07814f92fb36..6139f02faabe 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2414,6 +2414,7 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/merged.o\n REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n@@ -2425,6 +2426,7 @@ REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\ndiff --git a/reftable/merged.c b/reftable/merged.c\nnew file mode 100644\nindex 000000000000..e5b53da6db3f\n--- /dev/null\n+++ b/reftable/merged.c\n@@ -0,0 +1,362 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"constants.h\"\n+#include \"iter.h\"\n+#include \"pq.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"generic.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-error.h\"\n+#include \"system.h\"\n+\n+static int merged_iter_init(struct merged_iter *mi)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\t\tint err = iterator_next(&mi->stack[i], &rec);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\n+\t\tif (err > 0) {\n+\t\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t\t\treftable_record_destroy(&rec);\n+\t\t} else {\n+\t\t\tstruct pq_entry e = {\n+\t\t\t\t.rec = rec,\n+\t\t\t\t.index = i,\n+\t\t\t};\n+\t\t\tmerged_iter_pqueue_add(&mi->pq, e);\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void merged_iter_close(void *p)\n+{\n+\tstruct merged_iter *mi = p;\n+\tint i = 0;\n+\tmerged_iter_pqueue_release(&mi->pq);\n+\tfor (i = 0; i < mi->stack_len; i++) {\n+\t\treftable_iterator_destroy(&mi->stack[i]);\n+\t}\n+\treftable_free(mi->stack);\n+}\n+\n+static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,\n+\t\t\t\t\t size_t idx)\n+{\n+\tstruct reftable_record rec = reftable_new_record(mi->typ);\n+\tstruct pq_entry e = {\n+\t\t.rec = rec,\n+\t\t.index = idx,\n+\t};\n+\tint err = iterator_next(&mi->stack[idx], &rec);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\tif (err > 0) {\n+\t\treftable_iterator_destroy(&mi->stack[idx]);\n+\t\treftable_record_destroy(&rec);\n+\t\treturn 0;\n+\t}\n+\n+\tmerged_iter_pqueue_add(&mi->pq, e);\n+\treturn 0;\n+}\n+\n+static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)\n+{\n+\tif (iterator_is_null(&mi->stack[idx]))\n+\t\treturn 0;\n+\treturn merged_iter_advance_nonnull_subiter(mi, idx);\n+}\n+\n+static int merged_iter_next_entry(struct merged_iter *mi,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct strbuf entry_key = STRBUF_INIT;\n+\tstruct pq_entry entry = { 0 };\n+\tint err = 0;\n+\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\tentry = merged_iter_pqueue_remove(&mi->pq);\n+\terr = merged_iter_advance_subiter(mi, entry.index);\n+\tif (err < 0)\n+\t\treturn err;\n+\n+\t/*\n+\t One can also use reftable as datacenter-local storage, where the ref\n+\t database is maintained in globally consistent database (eg.\n+\t CockroachDB or Spanner). In this scenario, replication delays together\n+\t with compaction may cause newer tables to contain older entries. In\n+\t such a deployment, the loop below must be changed to collect all\n+\t entries for the same key, and return new the newest one.\n+\t*/\n+\treftable_record_key(&entry.rec, &entry_key);\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq)) {\n+\t\tstruct pq_entry top = merged_iter_pqueue_top(mi->pq);\n+\t\tstruct strbuf k = STRBUF_INIT;\n+\t\tint err = 0, cmp = 0;\n+\n+\t\treftable_record_key(&top.rec, &k);\n+\n+\t\tcmp = strbuf_cmp(&k, &entry_key);\n+\t\tstrbuf_release(&k);\n+\n+\t\tif (cmp > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\t\terr = merged_iter_advance_subiter(mi, top.index);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t\treftable_record_destroy(&top.rec);\n+\t}\n+\n+\treftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));\n+\treftable_record_destroy(&entry.rec);\n+\tstrbuf_release(&entry_key);\n+\treturn 0;\n+}\n+\n+static int merged_iter_next(struct merged_iter *mi, struct reftable_record *rec)\n+{\n+\twhile (1) {\n+\t\tint err = merged_iter_next_entry(mi, rec);\n+\t\tif (err == 0 && mi->suppress_deletions &&\n+\t\t reftable_record_is_deletion(rec)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\treturn err;\n+\t}\n+}\n+\n+static int merged_iter_next_void(void *p, struct reftable_record *rec)\n+{\n+\tstruct merged_iter *mi = p;\n+\tif (merged_iter_pqueue_is_empty(mi->pq))\n+\t\treturn 1;\n+\n+\treturn merged_iter_next(mi, rec);\n+}\n+\n+static struct reftable_iterator_vtable merged_iter_vtable = {\n+\t.next = &merged_iter_next_void,\n+\t.close = &merged_iter_close,\n+};\n+\n+static void iterator_from_merged_iter(struct reftable_iterator *it,\n+\t\t\t\t struct merged_iter *mi)\n+{\n+\tassert(!it->ops);\n+\tit->iter_arg = mi;\n+\tit->ops = &merged_iter_vtable;\n+}\n+\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id)\n+{\n+\tstruct reftable_merged_table *m = NULL;\n+\tuint64_t last_max = 0;\n+\tuint64_t first_min = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t min = reftable_table_min_update_index(&stack[i]);\n+\t\tuint64_t max = reftable_table_max_update_index(&stack[i]);\n+\n+\t\tif (reftable_table_hash_id(&stack[i]) != hash_id) {\n+\t\t\treturn REFTABLE_FORMAT_ERROR;\n+\t\t}\n+\t\tif (i == 0 || min < first_min) {\n+\t\t\tfirst_min = min;\n+\t\t}\n+\t\tif (i == 0 || max > last_max) {\n+\t\t\tlast_max = max;\n+\t\t}\n+\t}\n+\n+\tm = reftable_calloc(sizeof(struct reftable_merged_table));\n+\tm->stack = stack;\n+\tm->stack_len = n;\n+\tm->min = first_min;\n+\tm->max = last_max;\n+\tm->hash_id = hash_id;\n+\t*dest = m;\n+\treturn 0;\n+}\n+\n+/* clears the list of subtable, without affecting the readers themselves. */\n+void merged_table_release(struct reftable_merged_table *mt)\n+{\n+\tFREE_AND_NULL(mt->stack);\n+\tmt->stack_len = 0;\n+}\n+\n+void reftable_merged_table_free(struct reftable_merged_table *mt)\n+{\n+\tif (!mt) {\n+\t\treturn;\n+\t}\n+\tmerged_table_release(mt);\n+\treftable_free(mt);\n+}\n+\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->max;\n+}\n+\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt)\n+{\n+\treturn mt->min;\n+}\n+\n+static int reftable_table_seek_record(struct reftable_table *tab,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn tab->ops->seek_record(tab->table_arg, it, rec);\n+}\n+\n+static int merged_table_seek_record(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t struct reftable_record *rec)\n+{\n+\tstruct reftable_iterator *iters = reftable_calloc(\n+\t\tsizeof(struct reftable_iterator) * mt->stack_len);\n+\tstruct merged_iter merged = {\n+\t\t.stack = iters,\n+\t\t.typ = reftable_record_type(rec),\n+\t\t.hash_id = mt->hash_id,\n+\t\t.suppress_deletions = mt->suppress_deletions,\n+\t};\n+\tint n = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\tfor (i = 0; i < mt->stack_len && err == 0; i++) {\n+\t\tint e = reftable_table_seek_record(&mt->stack[i], &iters[n],\n+\t\t\t\t\t\t rec);\n+\t\tif (e < 0) {\n+\t\t\terr = e;\n+\t\t}\n+\t\tif (e == 0) {\n+\t\t\tn++;\n+\t\t}\n+\t}\n+\tif (err < 0) {\n+\t\tint i = 0;\n+\t\tfor (i = 0; i < n; i++) {\n+\t\t\treftable_iterator_destroy(&iters[i]);\n+\t\t}\n+\t\treftable_free(iters);\n+\t\treturn err;\n+\t}\n+\n+\tmerged.stack_len = n;\n+\terr = merged_iter_init(&merged);\n+\tif (err < 0) {\n+\t\tmerged_iter_close(&merged);\n+\t\treturn err;\n+\t} else {\n+\t\tstruct merged_iter *p =\n+\t\t\treftable_malloc(sizeof(struct merged_iter));\n+\t\t*p = merged;\n+\t\titerator_from_merged_iter(it, p);\n+\t}\n+\treturn 0;\n+}\n+\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)name,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_ref(&rec, &ref);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index)\n+{\n+\tstruct reftable_log_record log = {\n+\t\t.refname = (char *)name,\n+\t\t.update_index = update_index,\n+\t};\n+\tstruct reftable_record rec = { NULL };\n+\treftable_record_from_log(&rec, &log);\n+\treturn merged_table_seek_record(mt, it, &rec);\n+}\n+\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name)\n+{\n+\tuint64_t max = ~((uint64_t)0);\n+\treturn reftable_merged_table_seek_log_at(mt, it, name, max);\n+}\n+\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\n+{\n+\treturn mt->hash_id;\n+}\n+\n+static int reftable_merged_table_seek_void(void *tab,\n+\t\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t\t struct reftable_record *rec)\n+{\n+\treturn merged_table_seek_record(tab, it, rec);\n+}\n+\n+static uint32_t reftable_merged_table_hash_id_void(void *tab)\n+{\n+\treturn reftable_merged_table_hash_id(tab);\n+}\n+\n+static uint64_t reftable_merged_table_min_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_min_update_index(tab);\n+}\n+\n+static uint64_t reftable_merged_table_max_update_index_void(void *tab)\n+{\n+\treturn reftable_merged_table_max_update_index(tab);\n+}\n+\n+static struct reftable_table_vtable merged_table_vtable = {\n+\t.seek_record = reftable_merged_table_seek_void,\n+\t.hash_id = reftable_merged_table_hash_id_void,\n+\t.min_update_index = reftable_merged_table_min_update_index_void,\n+\t.max_update_index = reftable_merged_table_max_update_index_void,\n+};\n+\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *merged)\n+{\n+\tassert(!tab->ops);\n+\ttab->ops = &merged_table_vtable;\n+\ttab->table_arg = merged;\n+}\ndiff --git a/reftable/merged.h b/reftable/merged.h\nnew file mode 100644\nindex 000000000000..8c4d4d58d77a\n--- /dev/null\n+++ b/reftable/merged.h\n@@ -0,0 +1,35 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef MERGED_H\n+#define MERGED_H\n+\n+#include \"pq.h\"\n+\n+struct reftable_merged_table {\n+\tstruct reftable_table *stack;\n+\tsize_t stack_len;\n+\tuint32_t hash_id;\n+\tint suppress_deletions;\n+\n+\tuint64_t min;\n+\tuint64_t max;\n+};\n+\n+struct merged_iter {\n+\tstruct reftable_iterator *stack;\n+\tuint32_t hash_id;\n+\tsize_t stack_len;\n+\tuint8_t typ;\n+\tint suppress_deletions;\n+\tstruct merged_iter_pqueue pq;\n+};\n+\n+void merged_table_release(struct reftable_merged_table *mt);\n+\n+#endif\ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nnew file mode 100644\nindex 000000000000..a53251c24f8d\n--- /dev/null\n+++ b/reftable/merged_test.c\n@@ -0,0 +1,292 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"merged.h\"\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"blocksource.h\"\n+#include \"constants.h\"\n+#include \"reader.h\"\n+#include \"record.h\"\n+#include \"test_framework.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-generic.h\"\n+#include \"reftable-writer.h\"\n+\n+static void write_test_table(struct strbuf *buf,\n+\t\t\t struct reftable_ref_record refs[], int n)\n+{\n+\tint min = 0xffffffff;\n+\tint max = 0;\n+\tint i = 0;\n+\tint err;\n+\n+\tstruct reftable_write_options opts = {\n+\t\t.block_size = 256,\n+\t};\n+\tstruct reftable_writer *w = NULL;\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t ui = refs[i].update_index;\n+\t\tif (ui > max) {\n+\t\t\tmax = ui;\n+\t\t}\n+\t\tif (ui < min) {\n+\t\t\tmin = ui;\n+\t\t}\n+\t}\n+\n+\tw = reftable_new_writer(&strbuf_add_void, buf, &opts);\n+\treftable_writer_set_limits(w, min, max);\n+\n+\tfor (i = 0; i < n; i++) {\n+\t\tuint64_t before = refs[i].update_index;\n+\t\tint n = reftable_writer_add_ref(w, &refs[i]);\n+\t\tassert(n == 0);\n+\t\tassert(before == refs[i].update_index);\n+\t}\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\n+\treftable_writer_free(w);\n+}\n+\n+static struct reftable_merged_table *\n+merged_table_from_records(struct reftable_ref_record **refs,\n+\t\t\t struct reftable_block_source **source,\n+\t\t\t struct reftable_reader ***readers, int *sizes,\n+\t\t\t struct strbuf *buf, int n)\n+{\n+\tint i = 0;\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err;\n+\tstruct reftable_table *tabs =\n+\t\treftable_calloc(n * sizeof(struct reftable_table));\n+\t*readers = reftable_calloc(n * sizeof(struct reftable_reader *));\n+\t*source = reftable_calloc(n * sizeof(**source));\n+\tfor (i = 0; i < n; i++) {\n+\t\twrite_test_table(&buf[i], refs[i], sizes[i]);\n+\t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n+\n+\t\terr = reftable_new_reader(&(*readers)[i], &(*source)[i],\n+\t\t\t\t\t \"name\");\n+\t\tEXPECT_ERR(err);\n+\t\treftable_table_from_reader(&tabs[i], (*readers)[i]);\n+\t}\n+\n+\terr = reftable_new_merged_table(&mt, tabs, n, GIT_SHA1_HASH_ID);\n+\tEXPECT_ERR(err);\n+\treturn mt;\n+}\n+\n+static void readers_destroy(struct reftable_reader **readers, size_t n)\n+{\n+\tint i = 0;\n+\tfor (; i < n; i++)\n+\t\treftable_reader_free(readers[i]);\n+\treftable_free(readers);\n+}\n+\n+static void test_merged_between(void)\n+{\n+\tuint8_t hash1[GIT_SHA1_RAWSZ] = { 1, 2, 3, 0 };\n+\n+\tstruct reftable_ref_record r1[] = { {\n+\t\t.refname = \"b\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t.value.val1 = hash1,\n+\t} };\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_DELETION,\n+\t} };\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2 };\n+\tint sizes[] = { 1, 1 };\n+\tstruct strbuf bufs[2] = { STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 2);\n+\tint i;\n+\tstruct reftable_ref_record ref = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_iterator_next_ref(&it, &ref);\n+\tEXPECT_ERR(err);\n+\tEXPECT(ref.update_index == 2);\n+\treftable_ref_record_release(&ref);\n+\treftable_iterator_destroy(&it);\n+\treaders_destroy(readers, 2);\n+\treftable_merged_table_free(mt);\n+\tfor (i = 0; i < ARRAY_SIZE(bufs); i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treftable_free(bs);\n+}\n+\n+static void test_merged(void)\n+{\n+\tuint8_t hash1[GIT_SHA1_RAWSZ] = { 1 };\n+\tuint8_t hash2[GIT_SHA1_RAWSZ] = { 2 };\n+\tstruct reftable_ref_record r1[] = {\n+\t\t{\n+\t\t\t.refname = \"a\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"b\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t}\n+\t};\n+\tstruct reftable_ref_record r2[] = { {\n+\t\t.refname = \"a\",\n+\t\t.update_index = 2,\n+\t\t.value_type = REFTABLE_REF_DELETION,\n+\t} };\n+\tstruct reftable_ref_record r3[] = {\n+\t\t{\n+\t\t\t.refname = \"c\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash2,\n+\t\t},\n+\t\t{\n+\t\t\t.refname = \"d\",\n+\t\t\t.update_index = 3,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = hash1,\n+\t\t},\n+\t};\n+\n+\tstruct reftable_ref_record want[] = {\n+\t\tr2[0],\n+\t\tr1[1],\n+\t\tr3[0],\n+\t\tr3[1],\n+\t};\n+\n+\tstruct reftable_ref_record *refs[] = { r1, r2, r3 };\n+\tint sizes[3] = { 3, 1, 2 };\n+\tstruct strbuf bufs[3] = { STRBUF_INIT, STRBUF_INIT, STRBUF_INIT };\n+\tstruct reftable_block_source *bs = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_merged_table *mt =\n+\t\tmerged_table_from_records(refs, &bs, &readers, sizes, bufs, 3);\n+\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_merged_table_seek_ref(mt, &it, \"a\");\n+\tstruct reftable_ref_record *out = NULL;\n+\tsize_t len = 0;\n+\tsize_t cap = 0;\n+\tint i = 0;\n+\n+\tEXPECT_ERR(err);\n+\twhile (len < 100) { /* cap loops/recursion. */\n+\t\tstruct reftable_ref_record ref = { NULL };\n+\t\tint err = reftable_iterator_next_ref(&it, &ref);\n+\t\tif (err > 0) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tout = reftable_realloc(\n+\t\t\t\tout, sizeof(struct reftable_ref_record) * cap);\n+\t\t}\n+\t\tout[len++] = ref;\n+\t}\n+\treftable_iterator_destroy(&it);\n+\n+\tassert(ARRAY_SIZE(want) == len);\n+\tfor (i = 0; i < len; i++) {\n+\t\tassert(reftable_ref_record_equal(&want[i], &out[i],\n+\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t}\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_ref_record_release(&out[i]);\n+\t}\n+\treftable_free(out);\n+\n+\tfor (i = 0; i < 3; i++) {\n+\t\tstrbuf_release(&bufs[i]);\n+\t}\n+\treaders_destroy(readers, 3);\n+\treftable_merged_table_free(mt);\n+\treftable_free(bs);\n+}\n+\n+static void test_default_write_opts(void)\n+{\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct reftable_writer *w =\n+\t\treftable_new_writer(&strbuf_add_void, &buf, &opts);\n+\n+\tstruct reftable_ref_record rec = {\n+\t\t.refname = \"master\",\n+\t\t.update_index = 1,\n+\t};\n+\tint err;\n+\tstruct reftable_block_source source = { NULL };\n+\tstruct reftable_table *tab = reftable_calloc(sizeof(*tab) * 1);\n+\tuint32_t hash_id;\n+\tstruct reftable_reader *rd = NULL;\n+\tstruct reftable_merged_table *merged = NULL;\n+\n+\treftable_writer_set_limits(w, 1, 1);\n+\n+\terr = reftable_writer_add_ref(w, &rec);\n+\tEXPECT_ERR(err);\n+\n+\terr = reftable_writer_close(w);\n+\tEXPECT_ERR(err);\n+\treftable_writer_free(w);\n+\n+\tblock_source_from_strbuf(&source, &buf);\n+\n+\terr = reftable_new_reader(&rd, &source, \"filename\");\n+\tEXPECT_ERR(err);\n+\n+\thash_id = reftable_reader_hash_id(rd);\n+\tassert(hash_id == GIT_SHA1_HASH_ID);\n+\n+\treftable_table_from_reader(&tab[0], rd);\n+\terr = reftable_new_merged_table(&merged, tab, 1, GIT_SHA1_HASH_ID);\n+\tEXPECT_ERR(err);\n+\n+\treftable_reader_free(rd);\n+\treftable_merged_table_free(merged);\n+\tstrbuf_release(&buf);\n+}\n+\n+/* XXX test refs_for(oid) */\n+\n+int merged_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_merged_between);\n+\tRUN_TEST(test_merged);\n+\tRUN_TEST(test_default_write_opts);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nnew file mode 100644\nindex 000000000000..1a6d16915ab4\n--- /dev/null\n+++ b/reftable/reftable-merged.h\n@@ -0,0 +1,72 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_MERGED_H\n+#define REFTABLE_MERGED_H\n+\n+#include \"reftable-iterator.h\"\n+\n+/*\n+ * Merged tables\n+ *\n+ * A ref database kept in a sequence of table files. The merged_table presents a\n+ * unified view to reading (seeking, iterating) a sequence of immutable tables.\n+ *\n+ * The merged tables are on purpose kept disconnected from their actual storage\n+ * (eg. files on disk), because it is useful to merge tables aren't files. For\n+ * example, the per-workspace and global ref namespace can be implemented as a\n+ * merged table of two stacks of file-backed reftables.\n+ */\n+\n+/* A merged table is implements seeking/iterating over a stack of tables. */\n+struct reftable_merged_table;\n+\n+/* A generic reftable; see below. */\n+struct reftable_table;\n+\n+/* reftable_new_merged_table creates a new merged table. It takes ownership of\n+ the stack array.\n+*/\n+int reftable_new_merged_table(struct reftable_merged_table **dest,\n+\t\t\t struct reftable_table *stack, int n,\n+\t\t\t uint32_t hash_id);\n+\n+/* returns an iterator positioned just before 'name' */\n+int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns an iterator for log entry, at given update_index */\n+int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name, uint64_t update_index);\n+\n+/* like reftable_merged_table_seek_log_at but look for the newest entry. */\n+int reftable_merged_table_seek_log(struct reftable_merged_table *mt,\n+\t\t\t\t struct reftable_iterator *it,\n+\t\t\t\t const char *name);\n+\n+/* returns the max update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_max_update_index(struct reftable_merged_table *mt);\n+\n+/* returns the min update_index covered by this merged table. */\n+uint64_t\n+reftable_merged_table_min_update_index(struct reftable_merged_table *mt);\n+\n+/* releases memory for the merged_table */\n+void reftable_merged_table_free(struct reftable_merged_table *m);\n+\n+/* return the hash ID of the merged table. */\n+uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m);\n+\n+/* create a generic table from reftable_merged_table */\n+void reftable_table_from_merged_table(struct reftable_table *tab,\n+\t\t\t\t struct reftable_merged_table *table);\n+\n+#endif\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 0b5a1701df15..8087f2da4e6a 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -5,6 +5,7 @@ int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n+\tmerged_test_main(argc, argv);\n \tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n-- \ngitgitgadget\n\n"},{"id":"422288","messageId":"84578b666a40068c739d8131ddcd53136d83719b.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 18/28] reftable: add a heap-based priority queue for reftable records","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:46Z","receivedAt":"2021-04-19T11:40:58Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis is needed to create a merged view multiple reftables\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 2 +\n reftable/pq.c | 115 ++++++++++++++++++++++++++++++++++++++\n reftable/pq.h | 32 +++++++++++\n reftable/pq_test.c | 72 ++++++++++++++++++++++++\n reftable/reftable-tests.h | 1 +\n t/helper/test-reftable.c | 1 +\n 6 files changed, 223 insertions(+)\n create mode 100644 reftable/pq.c\n create mode 100644 reftable/pq.h\n create mode 100644 reftable/pq_test.c\n\ndiff --git a/Makefile b/Makefile\nindex 26a805afb5ad..07814f92fb36 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2414,6 +2414,7 @@ REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n REFTABLE_OBJS += reftable/publicbasics.o\n+REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/refname.o\n@@ -2424,6 +2425,7 @@ REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\n REFTABLE_TEST_OBJS += reftable/readwrite_test.o\n REFTABLE_TEST_OBJS += reftable/test_framework.o\ndiff --git a/reftable/pq.c b/reftable/pq.c\nnew file mode 100644\nindex 000000000000..8918d158e2d4\n--- /dev/null\n+++ b/reftable/pq.c\n@@ -0,0 +1,115 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"pq.h\"\n+\n+#include \"reftable-record.h\"\n+#include \"system.h\"\n+#include \"basics.h\"\n+\n+static int pq_less(struct pq_entry a, struct pq_entry b)\n+{\n+\tstruct strbuf ak = STRBUF_INIT;\n+\tstruct strbuf bk = STRBUF_INIT;\n+\tint cmp = 0;\n+\treftable_record_key(&a.rec, &ak);\n+\treftable_record_key(&b.rec, &bk);\n+\n+\tcmp = strbuf_cmp(&ak, &bk);\n+\n+\tstrbuf_release(&ak);\n+\tstrbuf_release(&bk);\n+\n+\tif (cmp == 0)\n+\t\treturn a.index > b.index;\n+\n+\treturn cmp < 0;\n+}\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.heap[0];\n+}\n+\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq)\n+{\n+\treturn pq.len == 0;\n+}\n+\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq)\n+{\n+\tint i = 0;\n+\tfor (i = 1; i < pq.len; i++) {\n+\t\tint parent = (i - 1) / 2;\n+\n+\t\tassert(pq_less(pq.heap[parent], pq.heap[i]));\n+\t}\n+}\n+\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tstruct pq_entry e = pq->heap[0];\n+\tpq->heap[0] = pq->heap[pq->len - 1];\n+\tpq->len--;\n+\n+\ti = 0;\n+\twhile (i < pq->len) {\n+\t\tint min = i;\n+\t\tint j = 2 * i + 1;\n+\t\tint k = 2 * i + 2;\n+\t\tif (j < pq->len && pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tmin = j;\n+\t\t}\n+\t\tif (k < pq->len && pq_less(pq->heap[k], pq->heap[min])) {\n+\t\t\tmin = k;\n+\t\t}\n+\n+\t\tif (min == i) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[i], pq->heap[min]);\n+\t\ti = min;\n+\t}\n+\n+\treturn e;\n+}\n+\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)\n+{\n+\tint i = 0;\n+\tif (pq->len == pq->cap) {\n+\t\tpq->cap = 2 * pq->cap + 1;\n+\t\tpq->heap = reftable_realloc(pq->heap,\n+\t\t\t\t\t pq->cap * sizeof(struct pq_entry));\n+\t}\n+\n+\tpq->heap[pq->len++] = e;\n+\ti = pq->len - 1;\n+\twhile (i > 0) {\n+\t\tint j = (i - 1) / 2;\n+\t\tif (pq_less(pq->heap[j], pq->heap[i])) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tSWAP(pq->heap[j], pq->heap[i]);\n+\n+\t\ti = j;\n+\t}\n+}\n+\n+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)\n+{\n+\tint i = 0;\n+\tfor (i = 0; i < pq->len; i++) {\n+\t\treftable_record_destroy(&pq->heap[i].rec);\n+\t}\n+\tFREE_AND_NULL(pq->heap);\n+\tpq->len = pq->cap = 0;\n+}\ndiff --git a/reftable/pq.h b/reftable/pq.h\nnew file mode 100644\nindex 000000000000..385d2fb139a6\n--- /dev/null\n+++ b/reftable/pq.h\n@@ -0,0 +1,32 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef PQ_H\n+#define PQ_H\n+\n+#include \"record.h\"\n+\n+struct pq_entry {\n+\tint index;\n+\tstruct reftable_record rec;\n+};\n+\n+struct merged_iter_pqueue {\n+\tstruct pq_entry *heap;\n+\tsize_t len;\n+\tsize_t cap;\n+};\n+\n+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq);\n+int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq);\n+void merged_iter_pqueue_check(struct merged_iter_pqueue pq);\n+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e);\n+void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n+\n+#endif\ndiff --git a/reftable/pq_test.c b/reftable/pq_test.c\nnew file mode 100644\nindex 000000000000..ad21673e8546\n--- /dev/null\n+++ b/reftable/pq_test.c\n@@ -0,0 +1,72 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \"system.h\"\n+\n+#include \"basics.h\"\n+#include \"constants.h\"\n+#include \"pq.h\"\n+#include \"record.h\"\n+#include \"reftable-tests.h\"\n+#include \"test_framework.h\"\n+\n+static void test_pq(void)\n+{\n+\tchar *names[54] = { NULL };\n+\tint N = ARRAY_SIZE(names) - 1;\n+\n+\tstruct merged_iter_pqueue pq = { NULL };\n+\tconst char *last = NULL;\n+\n+\tint i = 0;\n+\tfor (i = 0; i < N; i++) {\n+\t\tchar name[100];\n+\t\tsnprintf(name, sizeof(name), \"%02d\", i);\n+\t\tnames[i] = xstrdup(name);\n+\t}\n+\n+\ti = 1;\n+\tdo {\n+\t\tstruct reftable_record rec =\n+\t\t\treftable_new_record(BLOCK_TYPE_REF);\n+\t\tstruct pq_entry e = { 0 };\n+\n+\t\treftable_record_as_ref(&rec)->refname = names[i];\n+\t\te.rec = rec;\n+\t\tmerged_iter_pqueue_add(&pq, e);\n+\t\tmerged_iter_pqueue_check(pq);\n+\t\ti = (i * 7) % N;\n+\t} while (i != 1);\n+\n+\twhile (!merged_iter_pqueue_is_empty(pq)) {\n+\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct reftable_ref_record *ref =\n+\t\t\treftable_record_as_ref(&e.rec);\n+\n+\t\tmerged_iter_pqueue_check(pq);\n+\n+\t\tif (last) {\n+\t\t\tassert(strcmp(last, ref->refname) < 0);\n+\t\t}\n+\t\tlast = ref->refname;\n+\t\tref->refname = NULL;\n+\t\treftable_free(ref);\n+\t}\n+\n+\tfor (i = 0; i < N; i++) {\n+\t\treftable_free(names[i]);\n+\t}\n+\n+\tmerged_iter_pqueue_release(&pq);\n+}\n+\n+int pq_test_main(int argc, const char *argv[])\n+{\n+\tRUN_TEST(test_pq);\n+\treturn 0;\n+}\ndiff --git a/reftable/reftable-tests.h b/reftable/reftable-tests.h\nindex 3d541fa5c0ce..0019cbcfa498 100644\n--- a/reftable/reftable-tests.h\n+++ b/reftable/reftable-tests.h\n@@ -12,6 +12,7 @@ license that can be found in the LICENSE file or at\n int basics_test_main(int argc, const char **argv);\n int block_test_main(int argc, const char **argv);\n int merged_test_main(int argc, const char **argv);\n+int pq_test_main(int argc, const char **argv);\n int record_test_main(int argc, const char **argv);\n int refname_test_main(int argc, const char **argv);\n int readwrite_test_main(int argc, const char **argv);\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 898aba836fd1..0b5a1701df15 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -5,6 +5,7 @@ int cmd__reftable(int argc, const char **argv)\n {\n \tbasics_test_main(argc, argv);\n \tblock_test_main(argc, argv);\n+\tpq_test_main(argc, argv);\n \trecord_test_main(argc, argv);\n \treadwrite_test_main(argc, argv);\n \ttree_test_main(argc, argv);\n-- \ngitgitgadget\n\n"},{"id":"422290","messageId":"2fd7cb8c0983501e2af2f195aec81d7c17fb80e1.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 23/28] Reftable support for git-core","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:51Z","receivedAt":"2021-04-19T11:41:00Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nFor background, see the previous commit introducing the library.\n\nThis introduces the file refs/reftable-backend.c containing a reftable-powered\nref storage backend.\n\nIt can be activated by passing --ref-storage=reftable to \"git init\", or setting\nGIT_TEST_REFTABLE in the environment.\n\nExample use: see t/t0031-reftable.sh\n\nSigned-off-by: Han-Wen Nienhuys \nSigned-off-by: Johannes Schindelin \nHelped-by: Johannes Schindelin \nHelped-by: Junio Hamano \nHelped-by: Patrick Steinhardt \nCo-authored-by: Jeff King \n---\n Documentation/config/extensions.txt | 9 +\n .../technical/repository-version.txt | 7 +\n Makefile | 1 +\n builtin/clone.c | 5 +-\n builtin/init-db.c | 55 +-\n builtin/stash.c | 8 +-\n builtin/worktree.c | 27 +-\n cache.h | 8 +-\n config.mak.uname | 2 +-\n contrib/buildsystems/Generators/Vcxproj.pm | 11 +-\n refs.c | 26 +-\n refs.h | 3 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 1616 +++++++++++++++++\n repository.c | 2 +\n repository.h | 3 +\n setup.c | 9 +-\n t/t0031-reftable.sh | 271 +++\n t/t1409-avoid-packing-refs.sh | 6 +\n t/t1450-fsck.sh | 6 +\n t/t3210-pack-refs.sh | 6 +\n t/test-lib.sh | 5 +\n 22 files changed, 2053 insertions(+), 34 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100755 t/t0031-reftable.sh\n\ndiff --git a/Documentation/config/extensions.txt b/Documentation/config/extensions.txt\nindex 4e23d73cdcad..82c5940f1434 100644\n--- a/Documentation/config/extensions.txt\n+++ b/Documentation/config/extensions.txt\n@@ -6,3 +6,12 @@ extensions.objectFormat::\n Note that this setting should only be set by linkgit:git-init[1] or\n linkgit:git-clone[1]. Trying to change it after initialization will not\n work and will produce hard-to-diagnose issues.\n++\n+extensions.refStorage::\n+\tSpecify the ref storage mechanism to use. The acceptable values are `files` and\n+\t`reftable`. If not specified, `files` is assumed. It is an error to specify\n+\tthis key unless `core.repositoryFormatVersion` is 1.\n++\n+Note that this setting should only be set by linkgit:git-init[1] or\n+linkgit:git-clone[1]. Trying to change it after initialization will not\n+work and will produce hard-to-diagnose issues.\ndiff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\nindex 7844ef30ffde..725762358333 100644\n--- a/Documentation/technical/repository-version.txt\n+++ b/Documentation/technical/repository-version.txt\n@@ -100,3 +100,10 @@ If set, by default \"git config\" reads from both \"config\" and\n multiple working directory mode, \"config\" file is shared while\n \"config.worktree\" is per-working directory (i.e., it's in\n GIT_COMMON_DIR/worktrees//config.worktree)\n+\n+==== `refStorage`\n+\n+Specifies the file format for the ref database. Values are `files`\n+(for the traditional packed + loose ref format) and `reftable` for the\n+binary reftable format. See https://github.com/google/reftable for\n+more information.\ndiff --git a/Makefile b/Makefile\nindex f78d1cbc2d3c..cffcf3496978 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -978,6 +978,7 @@ LIB_OBJS += reflog-walk.o\n LIB_OBJS += refs.o\n LIB_OBJS += refs/debug.o\n LIB_OBJS += refs/files-backend.o\n+LIB_OBJS += refs/reftable-backend.o\n LIB_OBJS += refs/iterator.o\n LIB_OBJS += refs/packed-backend.o\n LIB_OBJS += refs/ref-cache.o\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex f6b0c48beda5..bb2509fc1268 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1148,7 +1148,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t}\n \n \tinit_db(git_dir, real_git_dir, option_template, GIT_HASH_UNKNOWN, NULL,\n-\t\tINIT_DB_QUIET);\n+\t\tdefault_ref_storage(), INIT_DB_QUIET);\n \n \tif (real_git_dir)\n \t\tgit_dir = real_git_dir;\n@@ -1299,7 +1299,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t * Now that we know what algorithm the remote side is using,\n \t\t * let's set ours to the same thing.\n \t\t */\n-\t\tinitialize_repository_version(hash_algo, 1);\n+\t\tinitialize_repository_version(hash_algo, 1,\n+\t\t\t\t\t default_ref_storage());\n \t\trepo_set_hash_algo(the_repository, hash_algo);\n \n \t\tmapped_refs = wanted_peer_refs(refs, &remote->fetch);\ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex ff06fe4b9a49..31b718259a64 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -167,12 +167,14 @@ static int needs_work_tree_config(const char *git_dir, const char *work_tree)\n \treturn 1;\n }\n \n-void initialize_repository_version(int hash_algo, int reinit)\n+void initialize_repository_version(int hash_algo, int reinit,\n+\t\t\t\t const char *ref_storage_format)\n {\n \tchar repo_version_string[10];\n \tint repo_version = GIT_REPO_VERSION;\n \n-\tif (hash_algo != GIT_HASH_SHA1)\n+\tif (hash_algo != GIT_HASH_SHA1 ||\n+\t !strcmp(ref_storage_format, \"reftable\"))\n \t\trepo_version = GIT_REPO_VERSION_READ;\n \n \t/* This forces creation of new config file */\n@@ -225,6 +227,7 @@ static int create_default_files(const char *template_path,\n \tis_bare_repository_cfg = init_is_bare_repository || !work_tree;\n \tif (init_shared_repository != -1)\n \t\tset_shared_repository(init_shared_repository);\n+\tthe_repository->ref_storage_format = xstrdup(fmt->ref_storage);\n \n \t/*\n \t * We would have created the above under user's umask -- under\n@@ -234,6 +237,24 @@ static int create_default_files(const char *template_path,\n \t\tadjust_shared_perm(get_git_dir());\n \t}\n \n+\t/*\n+\t * Check to see if .git/HEAD exists; this must happen before\n+\t * initializing the ref db, because we want to see if there is an\n+\t * existing HEAD.\n+\t */\n+\tpath = git_path_buf(&buf, \"HEAD\");\n+\treinit = (!access(path, R_OK) ||\n+\t\t readlink(path, junk, sizeof(junk) - 1) != -1);\n+\n+\t/*\n+\t * refs/heads is a file when using reftable. We can't reinitialize with\n+\t * a reftable because it will overwrite HEAD\n+\t */\n+\tif (reinit && (!strcmp(fmt->ref_storage, \"reftable\")) ==\n+\t\t\t is_directory(git_path_buf(&buf, \"refs/heads\"))) {\n+\t\tdie(\"cannot switch ref storage format.\");\n+\t}\n+\n \t/*\n \t * We need to create a \"refs\" dir in any case so that older\n \t * versions of git can tell that this is a repository.\n@@ -248,9 +269,6 @@ static int create_default_files(const char *template_path,\n \t * Point the HEAD symref to the initial branch with if HEAD does\n \t * not yet exist.\n \t */\n-\tpath = git_path_buf(&buf, \"HEAD\");\n-\treinit = (!access(path, R_OK)\n-\t\t || readlink(path, junk, sizeof(junk)-1) != -1);\n \tif (!reinit) {\n \t\tchar *ref;\n \n@@ -267,7 +285,7 @@ static int create_default_files(const char *template_path,\n \t\tfree(ref);\n \t}\n \n-\tinitialize_repository_version(fmt->hash_algo, 0);\n+\tinitialize_repository_version(fmt->hash_algo, 0, fmt->ref_storage);\n \n \t/* Check filemode trustability */\n \tpath = git_path_buf(&buf, \"config\");\n@@ -382,7 +400,7 @@ static void validate_hash_algorithm(struct repository_format *repo_fmt, int hash\n \n int init_db(const char *git_dir, const char *real_git_dir,\n \t const char *template_dir, int hash, const char *initial_branch,\n-\t unsigned int flags)\n+\t const char *ref_storage_format, unsigned int flags)\n {\n \tint reinit;\n \tint exist_ok = flags & INIT_DB_EXIST_OK;\n@@ -421,6 +439,7 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \t * is an attempt to reinitialize new repository with an old tool.\n \t */\n \tcheck_repository_format(&repo_fmt);\n+\trepo_fmt.ref_storage = xstrdup(ref_storage_format);\n \n \tvalidate_hash_algorithm(&repo_fmt, hash);\n \n@@ -475,6 +494,9 @@ int init_db(const char *git_dir, const char *real_git_dir,\n \t\tgit_config_set(\"receive.denyNonFastforwards\", \"true\");\n \t}\n \n+\tif (!strcmp(ref_storage_format, \"reftable\"))\n+\t\tgit_config_set(\"extensions.refStorage\", ref_storage_format);\n+\n \tif (!(flags & INIT_DB_QUIET)) {\n \t\tint len = strlen(git_dir);\n \n@@ -548,6 +570,7 @@ static const char *const init_db_usage[] = {\n int cmd_init_db(int argc, const char **argv, const char *prefix)\n {\n \tconst char *git_dir;\n+\tconst char *ref_storage_format = default_ref_storage();\n \tconst char *real_git_dir = NULL;\n \tconst char *work_tree;\n \tconst char *template_dir = NULL;\n@@ -556,15 +579,18 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n \tconst char *initial_branch = NULL;\n \tint hash_algo = GIT_HASH_UNKNOWN;\n \tconst struct option init_db_options[] = {\n-\t\tOPT_STRING(0, \"template\", &template_dir, N_(\"template-directory\"),\n-\t\t\t\tN_(\"directory from which templates will be used\")),\n+\t\tOPT_STRING(0, \"template\", &template_dir,\n+\t\t\t N_(\"template-directory\"),\n+\t\t\t N_(\"directory from which templates will be used\")),\n \t\tOPT_SET_INT(0, \"bare\", &is_bare_repository_cfg,\n-\t\t\t\tN_(\"create a bare repository\"), 1),\n+\t\t\t N_(\"create a bare repository\"), 1),\n \t\t{ OPTION_CALLBACK, 0, \"shared\", &init_shared_repository,\n-\t\t\tN_(\"permissions\"),\n-\t\t\tN_(\"specify that the git repository is to be shared amongst several users\"),\n-\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0},\n+\t\t N_(\"permissions\"),\n+\t\t N_(\"specify that the git repository is to be shared amongst several users\"),\n+\t\t PARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0 },\n \t\tOPT_BIT('q', \"quiet\", &flags, N_(\"be quiet\"), INIT_DB_QUIET),\n+\t\tOPT_STRING(0, \"ref-storage\", &ref_storage_format, N_(\"backend\"),\n+\t\t\t N_(\"the ref storage format to use\")),\n \t\tOPT_STRING(0, \"separate-git-dir\", &real_git_dir, N_(\"gitdir\"),\n \t\t\t N_(\"separate git dir from working tree\")),\n \t\tOPT_STRING('b', \"initial-branch\", &initial_branch, N_(\"name\"),\n@@ -707,10 +733,11 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)\n \t}\n \n \tUNLEAK(real_git_dir);\n+\tUNLEAK(ref_storage_format);\n \tUNLEAK(git_dir);\n \tUNLEAK(work_tree);\n \n \tflags |= INIT_DB_EXIST_OK;\n \treturn init_db(git_dir, real_git_dir, template_dir, hash_algo,\n-\t\t initial_branch, flags);\n+\t\t initial_branch, ref_storage_format, flags);\n }\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex c56fed33546b..2066b85a3903 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -207,10 +207,16 @@ static int get_stash_info(struct stash_info *info, int argc, const char **argv)\n static int do_clear_stash(void)\n {\n \tstruct object_id obj;\n+\tint result;\n \tif (get_oid(ref_stash, &obj))\n \t\treturn 0;\n \n-\treturn delete_ref(NULL, ref_stash, &obj, 0);\n+\tresult = delete_ref(NULL, ref_stash, &obj, 0);\n+\n+\t/* Ignore error; this is necessary for reftable, which keeps reflogs\n+\t * even when refs are deleted. */\n+\tdelete_reflog(ref_stash);\n+\treturn result;\n }\n \n static int clear_stash(int argc, const char **argv, const char *prefix)\ndiff --git a/builtin/worktree.c b/builtin/worktree.c\nindex 877145349381..fd389c86f6fa 100644\n--- a/builtin/worktree.c\n+++ b/builtin/worktree.c\n@@ -13,6 +13,7 @@\n #include \"utf8.h\"\n #include \"worktree.h\"\n #include \"quote.h\"\n+#include \"../refs/refs-internal.h\"\n \n static const char * const worktree_usage[] = {\n \tN_(\"git worktree add [] []\"),\n@@ -330,9 +331,29 @@ static int add_worktree(const char *path, const char *refname,\n \t * worktree.\n \t */\n \tstrbuf_reset(&sb);\n-\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n-\twrite_file(sb.buf, \"%s\", oid_to_hex(&null_oid));\n-\tstrbuf_reset(&sb);\n+\tif (get_main_ref_store(the_repository)->be == &refs_be_reftable) {\n+\t\t/* XXX this is cut & paste from reftable_init_db. */\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"%s\", \"ref: refs/heads/.invalid\\n\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/refs/heads\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\t\tstrbuf_reset(&sb);\n+\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", sb_repo.buf);\n+\t\tsafe_create_dir(sb.buf, 1);\n+\t\tstrbuf_reset(&sb);\n+\t} else {\n+\t\tstrbuf_addf(&sb, \"%s/HEAD\", sb_repo.buf);\n+\t\twrite_file(sb.buf, \"%s\", oid_to_hex(&null_oid));\n+\t\tstrbuf_reset(&sb);\n+\t}\n+\n \tstrbuf_addf(&sb, \"%s/commondir\", sb_repo.buf);\n \twrite_file(sb.buf, \"../..\");\n \ndiff --git a/cache.h b/cache.h\nindex 148d9ab5f185..f299535b0b03 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -629,9 +629,10 @@ int path_inside_repo(const char *prefix, const char *path);\n #define INIT_DB_EXIST_OK 0x0002\n \n int init_db(const char *git_dir, const char *real_git_dir,\n-\t const char *template_dir, int hash_algo,\n-\t const char *initial_branch, unsigned int flags);\n-void initialize_repository_version(int hash_algo, int reinit);\n+\t const char *template_dir, int hash_algo, const char *initial_branch,\n+\t const char *ref_storage_format, unsigned int flags);\n+void initialize_repository_version(int hash_algo, int reinit,\n+\t\t\t\t const char *ref_storage_format);\n \n void sanitize_stdfds(void);\n int daemonize(void);\n@@ -1045,6 +1046,7 @@ struct repository_format {\n \tint is_bare;\n \tint hash_algo;\n \tchar *work_tree;\n+\tchar *ref_storage;\n \tstruct string_list unknown_extensions;\n \tstruct string_list v1_only_extensions;\n };\ndiff --git a/config.mak.uname b/config.mak.uname\nindex cb443b4e023a..96c1268693f1 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -701,7 +701,7 @@ vcxproj:\n \t# Make .vcxproj files and add them\n \tunset QUIET_GEN QUIET_BUILT_IN; \\\n \tperl contrib/buildsystems/generate -g Vcxproj\n-\tgit add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj\n+\tgit add -f git.sln {*,*/lib,*/libreftable,t/helper/*}/*.vcxproj\n \n \t# Generate the LinkOrCopyBuiltins.targets and LinkOrCopyRemoteHttp.targets file\n \t(echo '' && \\\ndiff --git a/contrib/buildsystems/Generators/Vcxproj.pm b/contrib/buildsystems/Generators/Vcxproj.pm\nindex d2584450ba17..1a25789d2851 100644\n--- a/contrib/buildsystems/Generators/Vcxproj.pm\n+++ b/contrib/buildsystems/Generators/Vcxproj.pm\n@@ -77,7 +77,7 @@ sub createProject {\n my $libs_release = \"\\n \";\n my $libs_debug = \"\\n \";\n if (!$static_library) {\n- $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n+ $libs_release = join(\";\", sort(grep /^(?!libgit\\.lib|xdiff\\/lib\\.lib|vcs-svn\\/lib\\.lib|reftable\\/libreftable\\.lib)/, @{$$build_structure{\"$prefix${name}_LIBS\"}}));\n $libs_debug = $libs_release;\n $libs_debug =~ s/zlib\\.lib/zlibd\\.lib/g;\n $libs_debug =~ s/libexpat\\.lib/libexpatd\\.lib/g;\n@@ -232,6 +232,7 @@ sub createProject {\n EOM\n if (!$static_library || $target =~ 'vcs-svn' || $target =~ 'xdiff') {\n my $uuid_libgit = $$build_structure{\"LIBS_libgit_GUID\"};\n+ my $uuid_libreftable = $$build_structure{\"LIBS_reftable/libreftable_GUID\"};\n my $uuid_xdiff_lib = $$build_structure{\"LIBS_xdiff/lib_GUID\"};\n \n print F << \"EOM\";\n@@ -241,6 +242,14 @@ sub createProject {\n false\n \n EOM\n+ if (!($name =~ /xdiff|libreftable/)) {\n+ print F << \"EOM\";\n+ \n+ $uuid_libreftable\n+ false\n+ \n+EOM\n+ }\n if (!($name =~ 'xdiff')) {\n print F << \"EOM\";\n \ndiff --git a/refs.c b/refs.c\nindex 8873854a44fb..7292c80a26ab 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -19,10 +19,15 @@\n #include \"repository.h\"\n #include \"sigchain.h\"\n \n+const char *default_ref_storage(void)\n+{\n+\treturn git_env_bool(\"GIT_TEST_REFTABLE\", 0) ? \"reftable\" : \"files\";\n+}\n+\n /*\n * List of all available backends\n */\n-static struct ref_storage_be *refs_backends = &refs_be_files;\n+static struct ref_storage_be *refs_backends = &refs_be_reftable;\n \n static struct ref_storage_be *find_ref_storage_backend(const char *name)\n {\n@@ -1875,13 +1880,13 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,\n * Create, record, and return a ref_store instance for the specified\n * gitdir.\n */\n-static struct ref_store *ref_store_init(const char *gitdir,\n+static struct ref_store *ref_store_init(const char *gitdir, const char *be_name,\n \t\t\t\t\tunsigned int flags)\n {\n-\tconst char *be_name = \"files\";\n-\tstruct ref_storage_be *be = find_ref_storage_backend(be_name);\n+\tstruct ref_storage_be *be;\n \tstruct ref_store *refs;\n \n+\tbe = find_ref_storage_backend(be_name);\n \tif (!be)\n \t\tBUG(\"reference backend %s is unknown\", be_name);\n \n@@ -1897,7 +1902,11 @@ struct ref_store *get_main_ref_store(struct repository *r)\n \tif (!r->gitdir)\n \t\tBUG(\"attempting to get main_ref_store outside of repository\");\n \n-\tr->refs_private = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);\n+\tr->refs_private = ref_store_init(r->gitdir,\n+\t\t\t\t\t r->ref_storage_format ?\n+\t\t\t\t\t\t r->ref_storage_format :\n+\t\t\t\t\t\t default_ref_storage(),\n+\t\t\t\t\t REF_STORE_ALL_CAPS);\n \tr->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private);\n \treturn r->refs_private;\n }\n@@ -1953,7 +1962,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \t\tgoto done;\n \n \t/* assume that add_submodule_odb() has been called */\n-\trefs = ref_store_init(submodule_sb.buf,\n+\trefs = ref_store_init(submodule_sb.buf, default_ref_storage(),\n \t\t\t REF_STORE_READ | REF_STORE_ODB);\n \tregister_ref_store_map(&submodule_ref_stores, \"submodule\",\n \t\t\t refs, submodule);\n@@ -1967,6 +1976,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \n struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n {\n+\tconst char *format = default_ref_storage();\n \tstruct ref_store *refs;\n \tconst char *id;\n \n@@ -1980,9 +1990,9 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n \n \tif (wt->id)\n \t\trefs = ref_store_init(git_common_path(\"worktrees/%s\", wt->id),\n-\t\t\t\t REF_STORE_ALL_CAPS);\n+\t\t\t\t format, REF_STORE_ALL_CAPS);\n \telse\n-\t\trefs = ref_store_init(get_git_common_dir(),\n+\t\trefs = ref_store_init(get_git_common_dir(), format,\n \t\t\t\t REF_STORE_ALL_CAPS);\n \n \tif (refs)\ndiff --git a/refs.h b/refs.h\nindex 48970dfc7e0f..5a6d4ca9fa8d 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -11,6 +11,9 @@ struct string_list;\n struct string_list_item;\n struct worktree;\n \n+/* Returns the ref storage backend to use by default. */\n+const char *default_ref_storage(void);\n+\n /*\n * Resolve a reference, recursively following symbolic refererences.\n *\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex a31c1f465beb..7b427b58f61f 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -676,6 +676,7 @@ struct ref_storage_be {\n };\n \n extern struct ref_storage_be refs_be_files;\n+extern struct ref_storage_be refs_be_reftable;\n extern struct ref_storage_be refs_be_packed;\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nnew file mode 100644\nindex 000000000000..55d053e5ca65\n--- /dev/null\n+++ b/refs/reftable-backend.c\n@@ -0,0 +1,1616 @@\n+#include \"../cache.h\"\n+#include \"../chdir-notify.h\"\n+#include \"../config.h\"\n+#include \"../iterator.h\"\n+#include \"../lockfile.h\"\n+#include \"../refs.h\"\n+#include \"../reftable/reftable-stack.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-blocksource.h\"\n+#include \"../reftable/reftable-reader.h\"\n+#include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-merged.h\"\n+#include \"../reftable/reftable-generic.h\"\n+#include \"../worktree.h\"\n+#include \"refs-internal.h\"\n+\n+extern struct ref_storage_be refs_be_reftable;\n+\n+struct git_reftable_ref_store {\n+\tstruct ref_store base;\n+\tunsigned int store_flags;\n+\n+\tint err;\n+\tchar *repo_dir;\n+\n+\tchar *reftable_dir;\n+\tchar *worktree_reftable_dir;\n+\n+\tstruct reftable_stack *main_stack;\n+\tstruct reftable_stack *worktree_stack;\n+};\n+\n+static struct reftable_stack *stack_for(struct git_reftable_ref_store *store,\n+\t\t\t\t\tconst char *refname)\n+{\n+\tif (store->worktree_stack == NULL || refname == NULL)\n+\t\treturn store->main_stack;\n+\n+\tswitch (ref_type(refname)) {\n+\tcase REF_TYPE_PER_WORKTREE:\n+\tcase REF_TYPE_PSEUDOREF:\n+\tcase REF_TYPE_OTHER_PSEUDOREF:\n+\t\treturn store->worktree_stack;\n+\tdefault:\n+\tcase REF_TYPE_MAIN_PSEUDOREF:\n+\tcase REF_TYPE_NORMAL:\n+\t\treturn store->main_stack;\n+\t}\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type);\n+\n+static void clear_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tlog->refname = NULL;\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\tlog->update.old_hash = NULL;\n+\t\tlog->update.new_hash = NULL;\n+\t\tlog->update.message = NULL;\n+\t\tbreak;\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\t}\n+\treftable_log_record_release(log);\n+}\n+\n+static void fill_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tconst char *info = git_committer_info(0);\n+\tstruct ident_split split = { NULL };\n+\tint result = split_ident_line(&split, info, strlen(info));\n+\tint sign = 1;\n+\tassert(0 == result);\n+\n+\treftable_log_record_release(log);\n+\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\tlog->update.name =\n+\t\txstrndup(split.name_begin, split.name_end - split.name_begin);\n+\tlog->update.email =\n+\t\txstrndup(split.mail_begin, split.mail_end - split.mail_begin);\n+\tlog->update.time = atol(split.date_begin);\n+\tif (*split.tz_begin == '-') {\n+\t\tsign = -1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\tif (*split.tz_begin == '+') {\n+\t\tsign = 1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\n+\tlog->update.tz_offset = sign * atoi(split.tz_begin);\n+}\n+\n+static int has_suffix(struct strbuf *b, const char *suffix)\n+{\n+\tsize_t len = strlen(suffix);\n+\n+\tif (len > b->len) {\n+\t\treturn 0;\n+\t}\n+\n+\treturn 0 == strncmp(b->buf + b->len - len, suffix, len);\n+}\n+\n+/* trims the last path component of b. Returns -1 if it is not\n+ * present, or 0 on success\n+ */\n+static int trim_component(struct strbuf *b)\n+{\n+\tchar *last;\n+\tlast = strrchr(b->buf, '/');\n+\tif (!last)\n+\t\treturn -1;\n+\tstrbuf_setlen(b, last - b->buf);\n+\treturn 0;\n+}\n+\n+/* Returns whether `b` is a worktree path, trimming it to the gitdir\n+ */\n+static int is_worktree(struct strbuf *b)\n+{\n+\tif (trim_component(b) < 0) {\n+\t\treturn 0;\n+\t}\n+\tif (!has_suffix(b, \"/worktrees\")) {\n+\t\treturn 0;\n+\t}\n+\ttrim_component(b);\n+\treturn 1;\n+}\n+\n+static struct ref_store *git_reftable_ref_store_create(const char *path,\n+\t\t\t\t\t\t unsigned int store_flags)\n+{\n+\tstruct git_reftable_ref_store *refs = xcalloc(1, sizeof(*refs));\n+\tstruct ref_store *ref_store = (struct ref_store *)refs;\n+\tstruct reftable_write_options cfg = {\n+\t\t.block_size = 4096,\n+\t\t.hash_id = the_hash_algo->format_id,\n+\t};\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tconst char *gitdir = path;\n+\tstruct strbuf wt_buf = STRBUF_INIT;\n+\tint wt = 0;\n+\n+\tstrbuf_addstr(&wt_buf, path);\n+\n+\t/* this is clumsy, but the official worktree functions (eg.\n+\t * get_worktrees()) function will try to initialize a ref storage\n+\t * backend, leading to infinite recursion. */\n+\twt = is_worktree(&wt_buf);\n+\tif (wt) {\n+\t\tgitdir = wt_buf.buf;\n+\t}\n+\n+\tbase_ref_store_init(ref_store, &refs_be_reftable);\n+\tref_store->gitdir = xstrdup(gitdir);\n+\trefs->store_flags = store_flags;\n+\tstrbuf_addf(&sb, \"%s/reftable\", gitdir);\n+\trefs->reftable_dir = xstrdup(sb.buf);\n+\tstrbuf_reset(&sb);\n+\n+\trefs->err =\n+\t\treftable_new_stack(&refs->main_stack, refs->reftable_dir, cfg);\n+\tassert(refs->err != REFTABLE_API_ERROR);\n+\n+\tif (refs->err == 0 && wt) {\n+\t\tstrbuf_addf(&sb, \"%s/reftable\", path);\n+\t\trefs->worktree_reftable_dir = xstrdup(sb.buf);\n+\n+\t\trefs->err = reftable_new_stack(&refs->worktree_stack,\n+\t\t\t\t\t refs->worktree_reftable_dir,\n+\t\t\t\t\t cfg);\n+\t\tassert(refs->err != REFTABLE_API_ERROR);\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\tstrbuf_release(&wt_buf);\n+\treturn ref_store;\n+}\n+\n+static int git_reftable_init_db(struct ref_store *ref_store, struct strbuf *err)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\tsafe_create_dir(refs->reftable_dir, 1);\n+\tassert(refs->worktree_reftable_dir == NULL);\n+\n+\tstrbuf_addf(&sb, \"%s/HEAD\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"ref: refs/heads/.invalid\");\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs\", refs->base.gitdir);\n+\tsafe_create_dir(sb.buf, 1);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs/heads\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\n+\treturn 0;\n+}\n+\n+struct git_reftable_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_ref_record ref;\n+\tstruct object_id oid;\n+\tstruct ref_store *ref_store;\n+\n+\t/* In case we must iterate over 2 stacks, this is non-null. */\n+\tstruct reftable_merged_table *merged;\n+\tunsigned int flags;\n+\tint err;\n+\tconst char *prefix;\n+};\n+\n+static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\twhile (ri->err == 0) {\n+\t\tri->err = reftable_iterator_next_ref(&ri->iter, &ri->ref);\n+\t\tif (ri->err) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (ref_type(ri->ref.refname) == REF_TYPE_PSEUDOREF) {\n+\t\t\t/*\n+\t\t\t pseudorefs, eg. HEAD, FETCH_HEAD should not be\n+\t\t\t produced, by default.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\t\tri->base.refname = ri->ref.refname;\n+\t\tif (ri->prefix != NULL &&\n+\t\t strncmp(ri->prefix, ri->ref.refname, strlen(ri->prefix))) {\n+\t\t\tri->err = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (ri->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n+\t\t ref_type(ri->base.refname) != REF_TYPE_PER_WORKTREE)\n+\t\t\tcontinue;\n+\n+\t\tri->base.flags = 0;\n+\t\tswitch (ri->ref.value_type) {\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\toidread(&ri->oid, ri->ref.value.val1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\toidread(&ri->oid, ri->ref.value.val2.value);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF: {\n+\t\t\tint out_flags = 0;\n+\t\t\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\t\t\tri->ref_store, ri->ref.refname,\n+\t\t\t\tRESOLVE_REF_READING, &ri->oid, &out_flags);\n+\t\t\tri->base.flags = out_flags;\n+\t\t\tif (resolved == NULL &&\n+\t\t\t !(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t\t (ri->base.flags & REF_ISBROKEN)) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\t}\n+\t\tdefault:\n+\t\t\tabort();\n+\t\t}\n+\n+\t\tri->base.oid = &ri->oid;\n+\t\tif (!(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t !ref_resolves_to_object(ri->base.refname, ri->base.oid,\n+\t\t\t\t\t ri->base.flags)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tbreak;\n+\t}\n+\n+\tif (ri->err > 0) {\n+\t\treturn ITER_DONE;\n+\t}\n+\tif (ri->err < 0) {\n+\t\treturn ITER_ERROR;\n+\t}\n+\n+\treturn ITER_OK;\n+}\n+\n+static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\tif (ri->ref.value_type == REFTABLE_REF_VAL2) {\n+\t\toidread(peeled, ri->ref.value.val2.target_value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n+}\n+\n+static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_iterator *ri =\n+\t\t(struct git_reftable_iterator *)ref_iterator;\n+\treftable_ref_record_release(&ri->ref);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged) {\n+\t\treftable_merged_table_free(ri->merged);\n+\t}\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n+\treftable_ref_iterator_advance, reftable_ref_iterator_peel,\n+\treftable_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,\n+\t\t\t\tunsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct git_reftable_iterator *ri = xcalloc(1, sizeof(*ri));\n+\n+\tif (refs->err < 0) {\n+\t\tri->err = refs->err;\n+\t} else if (refs->worktree_stack == NULL) {\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tri->err = reftable_merged_table_seek_ref(mt, &ri->iter, prefix);\n+\t} else {\n+\t\tstruct reftable_merged_table *mt1 =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tstruct reftable_merged_table *mt2 =\n+\t\t\treftable_stack_merged_table(refs->worktree_stack);\n+\t\tstruct reftable_table *tabs =\n+\t\t\txcalloc(2, sizeof(struct reftable_table));\n+\t\treftable_table_from_merged_table(&tabs[0], mt1);\n+\t\treftable_table_from_merged_table(&tabs[1], mt2);\n+\t\tri->err = reftable_new_merged_table(&ri->merged, tabs, 2,\n+\t\t\t\t\t\t the_hash_algo->format_id);\n+\t\tif (ri->err == 0)\n+\t\t\tri->err = reftable_merged_table_seek_ref(\n+\t\t\t\tri->merged, &ri->iter, prefix);\n+\t}\n+\n+\tbase_ref_iterator_init(&ri->base, &reftable_ref_iterator_vtable, 1);\n+\tri->prefix = prefix;\n+\tri->base.oid = &ri->oid;\n+\tri->flags = flags;\n+\tri->ref_store = ref_store;\n+\treturn &ri->base;\n+}\n+\n+static int fixup_symrefs(struct ref_store *ref_store,\n+\t\t\t struct ref_transaction *transaction)\n+{\n+\tstruct strbuf referent = STRBUF_INIT;\n+\tint i = 0;\n+\tint err = 0;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *update = transaction->updates[i];\n+\t\tstruct object_id old_oid;\n+\n+\t\terr = git_reftable_read_raw_ref(ref_store, update->refname,\n+\t\t\t\t\t\t&old_oid, &referent,\n+\t\t\t\t\t\t/* mutate input, like\n+\t\t\t\t\t\t files-backend.c */\n+\t\t\t\t\t\t&update->type);\n+\t\tif (err < 0 && errno == ENOENT &&\n+\t\t is_null_oid(&update->old_oid)) {\n+\t\t\terr = 0;\n+\t\t}\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\n+\t\tif (!(update->type & REF_ISSYMREF))\n+\t\t\tcontinue;\n+\n+\t\tif (update->flags & REF_NO_DEREF) {\n+\t\t\t/* what should happen here? See files-backend.c\n+\t\t\t * lock_ref_for_update. */\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t If we are updating a symref (eg. HEAD), we should also\n+\t\t\t update the branch that the symref points to.\n+\n+\t\t\t This is generic functionality, and would be better\n+\t\t\t done in refs.c, but the current implementation is\n+\t\t\t intertwined with the locking in files-backend.c.\n+\t\t\t*/\n+\t\t\tint new_flags = update->flags;\n+\t\t\tstruct ref_update *new_update = NULL;\n+\n+\t\t\t/* if this is an update for HEAD, should also record a\n+\t\t\t log entry for HEAD? See files-backend.c,\n+\t\t\t split_head_update()\n+\t\t\t*/\n+\t\t\tnew_update = ref_transaction_add_update(\n+\t\t\t\ttransaction, referent.buf, new_flags,\n+\t\t\t\t&update->new_oid, &update->old_oid,\n+\t\t\t\tupdate->msg);\n+\t\t\tnew_update->parent_update = update;\n+\n+\t\t\t/* files-backend sets REF_LOG_ONLY here. */\n+\t\t\tupdate->flags |= REF_NO_DEREF | REF_LOG_ONLY;\n+\t\t\tupdate->flags &= ~REF_HAVE_OLD;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tstrbuf_release(&referent);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_prepare(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errbuf)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_addition *add = NULL;\n+\tstruct reftable_stack *stack =\n+\t\tstack_for(refs,\n+\t\t\t transaction->nr ? transaction->updates[0]->refname : NULL);\n+\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_new_addition(&add, stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = fixup_symrefs(ref_store, transaction);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\ttransaction->backend_data = add;\n+\ttransaction->state = REF_TRANSACTION_PREPARED;\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\tif (err < 0) {\n+\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\t\tstrbuf_addf(errbuf, \"reftable: transaction prepare: %s\",\n+\t\t\t reftable_error_str(err));\n+\t}\n+\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_abort(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\treftable_addition_destroy(add);\n+\ttransaction->backend_data = NULL;\n+\treturn 0;\n+}\n+\n+static int reftable_check_old_oid(struct ref_store *refs, const char *refname,\n+\t\t\t\t struct object_id *want_oid)\n+{\n+\tstruct object_id out_oid;\n+\tint out_flags = 0;\n+\tconst char *resolved = refs_resolve_ref_unsafe(\n+\t\trefs, refname, RESOLVE_REF_READING, &out_oid, &out_flags);\n+\tif (is_null_oid(want_oid) != (resolved == NULL)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\tif (resolved != NULL && !oideq(&out_oid, want_oid)) {\n+\t\treturn REFTABLE_LOCK_ERROR;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int ref_update_cmp(const void *a, const void *b)\n+{\n+\treturn strcmp((*(struct ref_update **)a)->refname,\n+\t\t (*(struct ref_update **)b)->refname);\n+}\n+\n+static int write_transaction_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct ref_transaction *transaction = (struct ref_transaction *)arg;\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)transaction->ref_store;\n+\tstruct reftable_stack *stack =\n+\t\tstack_for(refs, transaction->updates[0]->refname);\n+\tuint64_t ts = reftable_stack_next_update_index(stack);\n+\tint err = 0;\n+\tint i = 0;\n+\tstruct reftable_log_record *logs =\n+\t\tcalloc(transaction->nr, sizeof(*logs));\n+\tstruct ref_update **sorted =\n+\t\tmalloc(transaction->nr * sizeof(struct ref_update *));\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_table tab = {NULL};\n+\tstruct reftable_ref_record ref = {NULL};\n+\treftable_table_from_merged_table(&tab, mt);\n+\tCOPY_ARRAY(sorted, transaction->updates, transaction->nr);\n+\tQSORT(sorted, transaction->nr, ref_update_cmp);\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = sorted[i];\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tstruct object_id old_id;\n+\t\tfill_reftable_log_record(log);\n+\t\tlog->update_index = ts;\n+\t\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\t\tlog->refname = (char *)u->refname;\n+\t\tlog->update.new_hash = u->new_oid.hash;\n+\t\tlog->update.message = u->msg;\n+\n+\t\terr = reftable_table_read_ref(&tab, u->refname, &ref);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t\telse if (err > 0) {\n+\t\t\told_id = null_oid;\n+\t\t} else {\n+\t\t\toidread(&old_id, ref.value.val1);\n+\t\t}\n+\n+\t\t/* XXX fold together with the old_id check below? */\n+\n+\t\tlog->update.old_hash = old_id.hash;\n+\t\tif (u->flags & REF_LOG_ONLY) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (u->flags & REF_HAVE_NEW) {\n+\t\t\tstruct reftable_ref_record ref = { NULL };\n+\t\t\tstruct object_id peeled;\n+\n+\t\t\tint peel_error = peel_object(&u->new_oid, &peeled);\n+\t\t\tref.refname = (char *)u->refname;\n+\t\t\tref.update_index = ts;\n+\n+\t\t\tif (!peel_error) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\t\t\tref.value.val2.target_value = peeled.hash;\n+\t\t\t\tref.value.val2.value = u->new_oid.hash;\n+\t\t\t} else if (!is_null_oid(&u->new_oid)) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\t\t\tref.value.val1 = u->new_oid.hash;\n+\t\t\t}\n+\n+\t\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\terr = reftable_writer_add_log(writer, &logs[i]);\n+\t\tclear_reftable_log_record(&logs[i]);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\tfree(logs);\n+\tfree(sorted);\n+\treturn err;\n+}\n+\n+static int git_reftable_transaction_finish(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *errmsg)\n+{\n+\tstruct reftable_addition *add =\n+\t\t(struct reftable_addition *)transaction->backend_data;\n+\tint err = 0;\n+\tint i;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = transaction->updates[i];\n+\t\tif (u->flags & REF_HAVE_OLD) {\n+\t\t\terr = reftable_check_old_oid(transaction->ref_store,\n+\t\t\t\t\t\t u->refname, &u->old_oid);\n+\t\t\tif (err < 0) {\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (transaction->nr) {\n+\t\terr = reftable_addition_add(add, &write_transaction_table,\n+\t\t\t\t\t transaction);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\terr = reftable_addition_commit(add);\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_addition_destroy(add);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\ttransaction->backend_data = NULL;\n+\tif (err) {\n+\t\tstrbuf_addf(errmsg, \"reftable: transaction failure: %s\",\n+\t\t\t reftable_error_str(err));\n+\t\treturn -1;\n+\t}\n+\treturn err;\n+}\n+\n+static int\n+git_reftable_transaction_initial_commit(struct ref_store *ref_store,\n+\t\t\t\t\tstruct ref_transaction *transaction,\n+\t\t\t\t\tstruct strbuf *errmsg)\n+{\n+\tint err = git_reftable_transaction_prepare(ref_store, transaction,\n+\t\t\t\t\t\t errmsg);\n+\tif (err)\n+\t\treturn err;\n+\n+\treturn git_reftable_transaction_finish(ref_store, transaction, errmsg);\n+}\n+\n+struct write_delete_refs_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct string_list *refnames;\n+\tconst char *logmsg;\n+\tunsigned int flags;\n+};\n+\n+static int write_delete_refs_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_delete_refs_arg *arg =\n+\t\t(struct write_delete_refs_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint err = 0;\n+\tint i = 0;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_ref_record ref = {\n+\t\t\t.refname = (char *)arg->refnames->items[i].string,\n+\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\terr = reftable_writer_add_ref(writer, &ref);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < arg->refnames->nr; i++) {\n+\t\tstruct reftable_log_record log = {\n+\t\t\t.update_index = ts,\n+\t\t};\n+\t\tstruct reftable_ref_record current = { NULL };\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.update_index = ts;\n+\t\tlog.refname = (char *)arg->refnames->items[i].string;\n+\n+\t\tlog.update.message = xstrdup(arg->logmsg);\n+\t\tlog.update.new_hash = NULL;\n+\t\tlog.update.old_hash = NULL;\n+\t\tif (reftable_stack_read_ref(arg->stack, log.refname,\n+\t\t\t\t\t ¤t) == 0) {\n+\t\t\tlog.update.old_hash =\n+\t\t\t\treftable_ref_record_val1(¤t);\n+\t\t}\n+\t\terr = reftable_writer_add_log(writer, &log);\n+\t\tlog.update.old_hash = NULL;\n+\t\treftable_ref_record_release(¤t);\n+\n+\t\tclear_reftable_log_record(&log);\n+\t\tif (err < 0) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int git_reftable_delete_refs(struct ref_store *ref_store,\n+\t\t\t\t const char *msg,\n+\t\t\t\t struct string_list *refnames,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack =\n+\t\tstack_for(refs, refnames->nr ? refnames->items[0].string : NULL);\n+\tstruct write_delete_refs_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refnames = refnames,\n+\t\t.logmsg = msg,\n+\t\t.flags = flags,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tstring_list_sort(refnames);\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_delete_refs_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_pack_refs(struct ref_store *ref_store,\n+\t\t\t\t unsigned int flags)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\treturn err;\n+\t}\n+\terr = reftable_stack_compact_all(refs->main_stack, NULL);\n+\tif (err == 0 && refs->worktree_stack != NULL)\n+\t\terr = reftable_stack_compact_all(refs->worktree_stack, NULL);\n+\tif (err == 0)\n+\t\terr = reftable_stack_clean(refs->main_stack);\n+\tif (err == 0 && refs->worktree_stack != NULL)\n+\t\terr = reftable_stack_clean(refs->worktree_stack);\n+\n+\treturn err;\n+}\n+\n+struct write_create_symref_arg {\n+\tstruct git_reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+\tconst char *target;\n+\tconst char *logmsg;\n+};\n+\n+static int write_create_symref_table(struct reftable_writer *writer, void *arg)\n+{\n+\tstruct write_create_symref_arg *create =\n+\t\t(struct write_create_symref_arg *)arg;\n+\tuint64_t ts = reftable_stack_next_update_index(create->stack);\n+\tint err = 0;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)create->refname,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = (char *)create->target,\n+\t\t.update_index = ts,\n+\t};\n+\treftable_writer_set_limits(writer, ts, ts);\n+\terr = reftable_writer_add_ref(writer, &ref);\n+\tif (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\tstruct object_id new_oid;\n+\t\tstruct object_id old_oid;\n+\n+\t\tfill_reftable_log_record(&log);\n+\t\tlog.refname = (char *)create->refname;\n+\t\tlog.update_index = ts;\n+\t\tlog.update.message = (char *)create->logmsg;\n+\t\tif (refs_resolve_ref_unsafe(\n+\t\t\t (struct ref_store *)create->refs, create->refname,\n+\t\t\t RESOLVE_REF_READING, &old_oid, NULL) != NULL) {\n+\t\t\tlog.update.old_hash = old_oid.hash;\n+\t\t}\n+\n+\t\tif (refs_resolve_ref_unsafe((struct ref_store *)create->refs,\n+\t\t\t\t\t create->target, RESOLVE_REF_READING,\n+\t\t\t\t\t &new_oid, NULL) != NULL) {\n+\t\t\tlog.update.new_hash = new_oid.hash;\n+\t\t}\n+\n+\t\tif (log.update.old_hash != NULL ||\n+\t\t log.update.new_hash != NULL) {\n+\t\t\terr = reftable_writer_add_log(writer, &log);\n+\t\t}\n+\t\tlog.refname = NULL;\n+\t\tlog.update.message = NULL;\n+\t\tlog.update.old_hash = NULL;\n+\t\tlog.update.new_hash = NULL;\n+\t\tclear_reftable_log_record(&log);\n+\t}\n+\treturn err;\n+}\n+\n+static int git_reftable_create_symref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, const char *target,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_create_symref_arg arg = { .refs = refs,\n+\t\t\t\t\t .stack = stack,\n+\t\t\t\t\t .refname = refname,\n+\t\t\t\t\t .target = target,\n+\t\t\t\t\t .logmsg = logmsg };\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_add(stack, &write_create_symref_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct write_rename_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *oldname;\n+\tconst char *newname;\n+\tconst char *logmsg;\n+};\n+\n+static int write_rename_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_ref_record old_ref = { NULL };\n+\tstruct reftable_ref_record new_ref = { NULL };\n+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref);\n+\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* git-branch supports a --force, but the check is not atomic. */\n+\tif (reftable_stack_read_ref(arg->stack, arg->newname, &new_ref) == 0) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\t{\n+\t\tstruct reftable_ref_record todo[2] = {\n+\t\t\t{\n+\t\t\t\t.refname = (char *)arg->oldname,\n+\t\t\t\t.update_index = ts,\n+\t\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t},\n+\t\t\told_ref,\n+\t\t};\n+\t\ttodo[1].update_index = ts;\n+\t\tfree(todo[1].refname);\n+\t\ttodo[1].refname = strdup(arg->newname);\n+\n+\t\terr = reftable_writer_add_refs(writer, todo, 2);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\tif (reftable_ref_record_val1(&old_ref)) {\n+\t\tuint8_t *val1 = reftable_ref_record_val1(&old_ref);\n+\t\tstruct reftable_log_record todo[2] = { { NULL } };\n+\t\tfill_reftable_log_record(&todo[0]);\n+\t\tfill_reftable_log_record(&todo[1]);\n+\n+\t\ttodo[0].refname = (char *)arg->oldname;\n+\t\ttodo[0].update_index = ts;\n+\t\ttodo[0].update.message = (char *)arg->logmsg;\n+\t\ttodo[0].update.old_hash = val1;\n+\t\ttodo[0].update.new_hash = NULL;\n+\n+\t\ttodo[1].refname = (char *)arg->newname;\n+\t\ttodo[1].update_index = ts;\n+\t\ttodo[1].update.old_hash = NULL;\n+\t\ttodo[1].update.new_hash = val1;\n+\t\ttodo[1].update.message = (char *)arg->logmsg;\n+\n+\t\terr = reftable_writer_add_logs(writer, todo, 2);\n+\n+\t\tclear_reftable_log_record(&todo[0]);\n+\t\tclear_reftable_log_record(&todo[1]);\n+\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t} else {\n+\t\t/* XXX symrefs? */\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&new_ref);\n+\treftable_ref_record_release(&old_ref);\n+\treturn err;\n+}\n+\n+static int write_copy_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_rename_arg *arg = (struct write_rename_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_ref_record old_ref = { NULL };\n+\tstruct reftable_ref_record new_ref = { NULL };\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tint err = reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* git-branch supports a --force, but the check is not atomic. */\n+\tif (reftable_stack_read_ref(arg->stack, arg->newname, &new_ref) == 0) {\n+\t\tgoto done;\n+\t}\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tFREE_AND_NULL(old_ref.refname);\n+\told_ref.refname = xstrdup(arg->newname);\n+\told_ref.update_index = ts;\n+\terr = reftable_writer_add_ref(writer, &old_ref);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\t/* this copies the entire reflog history. Is this the right semantics? */\n+\t/* XXX should clear out existing reflog entries for oldname? */\n+\terr = reftable_merged_table_seek_log(reftable_stack_merged_table(arg->stack), &it, arg->oldname);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (err > 0 || strcmp(log.refname, arg->oldname)) {\n+\t\t\tbreak;\n+\t\t}\n+\t\tFREE_AND_NULL(log.refname);\n+\t\tlog.refname = xstrdup(arg->newname);\n+\t\treftable_writer_add_log(writer, &log);\n+\t\treftable_log_record_release(&log);\n+\t}\n+\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&new_ref);\n+\treftable_ref_record_release(&old_ref);\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_rename_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname,\n+\t\t\t\t const char *newrefname, const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n+\tstruct write_rename_arg arg = {\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_rename_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+static int git_reftable_copy_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname, const char *newrefname,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname);\n+\tstruct write_rename_arg arg = {\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint err = refs->err;\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_add(stack, &write_copy_table, &arg);\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct git_reftable_reflog_ref_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_log_record log;\n+\tstruct object_id oid;\n+\n+\t/* Used when iterating over worktree & main */\n+\tstruct reftable_merged_table *merged;\n+\tchar *last_name;\n+};\n+\n+static int\n+git_reftable_reflog_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\n+\twhile (1) {\n+\t\tint err = reftable_iterator_next_log(&ri->iter, &ri->log);\n+\t\tif (err > 0) {\n+\t\t\treturn ITER_DONE;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\treturn ITER_ERROR;\n+\t\t}\n+\n+\t\tri->base.refname = ri->log.refname;\n+\t\tif (ri->last_name != NULL &&\n+\t\t !strcmp(ri->log.refname, ri->last_name)) {\n+\t\t\t/* we want the refnames that we have reflogs for, so we\n+\t\t\t * skip if we've already produced this name. This could\n+\t\t\t * be faster by seeking directly to\n+\t\t\t * reflog@update_index==0.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfree(ri->last_name);\n+\t\tri->last_name = xstrdup(ri->log.refname);\n+\t\toidread(&ri->oid, ri->log.update.new_hash);\n+\t\treturn ITER_OK;\n+\t}\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tBUG(\"not supported.\");\n+\treturn -1;\n+}\n+\n+static int\n+git_reftable_reflog_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri =\n+\t\t(struct git_reftable_reflog_ref_iterator *)ref_iterator;\n+\treftable_log_record_release(&ri->log);\n+\treftable_iterator_destroy(&ri->iter);\n+\tif (ri->merged)\n+\t\treftable_merged_table_free(ri->merged);\n+\treturn 0;\n+}\n+\n+static struct ref_iterator_vtable git_reftable_reflog_ref_iterator_vtable = {\n+\tgit_reftable_reflog_ref_iterator_advance,\n+\tgit_reftable_reflog_ref_iterator_peel,\n+\tgit_reftable_reflog_ref_iterator_abort\n+};\n+\n+static struct ref_iterator *\n+git_reftable_reflog_iterator_begin(struct ref_store *ref_store)\n+{\n+\tstruct git_reftable_reflog_ref_iterator *ri = xcalloc(1, sizeof(*ri));\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\n+\tif (refs->worktree_stack == NULL) {\n+\t\tstruct reftable_stack *stack = refs->main_stack;\n+\t\tstruct reftable_merged_table *mt =\n+\t\t\treftable_stack_merged_table(stack);\n+\t\tint err = reftable_merged_table_seek_log(mt, &ri->iter, \"\");\n+\t\tif (err < 0) {\n+\t\t\tfree(ri);\n+\t\t\t/* XXX is this allowed? */\n+\t\t\treturn NULL;\n+\t\t}\n+\t} else {\n+\t\tstruct reftable_merged_table *mt1 =\n+\t\t\treftable_stack_merged_table(refs->main_stack);\n+\t\tstruct reftable_merged_table *mt2 =\n+\t\t\treftable_stack_merged_table(refs->worktree_stack);\n+\t\tstruct reftable_table *tabs =\n+\t\t\txcalloc(2, sizeof(struct reftable_table));\n+\t\tint err = 0;\n+\t\treftable_table_from_merged_table(&tabs[0], mt1);\n+\t\treftable_table_from_merged_table(&tabs[1], mt2);\n+\t\terr = reftable_new_merged_table(&ri->merged, tabs, 2,\n+\t\t\t\t\t\tthe_hash_algo->format_id);\n+\t\tif (err < 0) {\n+\t\t\tfree(tabs);\n+\t\t\t/* XXX see above */\n+\t\t\treturn NULL;\n+\t\t}\n+\t\terr = reftable_merged_table_seek_ref(ri->merged, &ri->iter, \"\");\n+\t\tif (err < 0) {\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\tbase_ref_iterator_init(&ri->base,\n+\t\t\t &git_reftable_reflog_ref_iterator_vtable, 1);\n+\tri->base.oid = &ri->oid;\n+\n+\treturn (struct ref_iterator *)ri;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_newest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tint err = 0;\n+\tstruct reftable_log_record log = { NULL };\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (err == 0) {\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\toidread(&old_oid, log.update.old_hash);\n+\t\toidread(&new_oid, log.update.new_hash);\n+\n+\t\tfull_committer = fmt_ident(log.update.name, log.update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT,\n+\t\t\t\t\t /*date*/ NULL, IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer, log.update.time,\n+\t\t\t log.update.tz_offset, log.update.message, cb_data);\n+\t\tif (err)\n+\t\t\tbreak;\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_for_each_reflog_ent_oldest_first(\n+\tstruct ref_store *ref_store, const char *refname, each_reflog_ent_fn fn,\n+\tvoid *cb_data)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record *logs = NULL;\n+\tint cap = 0;\n+\tint len = 0;\n+\tint err = 0;\n+\tint i = 0;\n+\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\n+\twhile (err == 0) {\n+\t\tstruct reftable_log_record log = { NULL };\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (err < 0) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (len == cap) {\n+\t\t\tcap = 2 * cap + 1;\n+\t\t\tlogs = realloc(logs, cap * sizeof(*logs));\n+\t\t}\n+\n+\t\tlogs[len++] = log;\n+\t}\n+\n+\tfor (i = len; i--;) {\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tstruct object_id old_oid;\n+\t\tstruct object_id new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\toidread(&old_oid, log->update.old_hash);\n+\t\toidread(&new_oid, log->update.new_hash);\n+\n+\t\tfull_committer = fmt_ident(log->update.name, log->update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT, NULL,\n+\t\t\t\t\t IDENT_NO_DATE);\n+\t\terr = fn(&old_oid, &new_oid, full_committer, log->update.time,\n+\t\t\t log->update.tz_offset, log->update.message, cb_data);\n+\t\tif (err) {\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < len; i++) {\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tfree(logs);\n+\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int git_reftable_reflog_exists(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record log = { NULL };\n+\tint err = refs->err;\n+\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\terr = reftable_iterator_next_log(&it, &log);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (strcmp(log.refname, refname)) {\n+\t\terr = 1;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\treturn !err;\n+}\n+\n+static int git_reftable_create_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, int force_create,\n+\t\t\t\t struct strbuf *err)\n+{\n+\treturn 0;\n+}\n+\n+struct write_reflog_delete_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+};\n+\n+static int write_reflog_delete_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct write_reflog_delete_arg *arg = argv;\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(arg->stack);\n+\tstruct reftable_log_record log = { NULL };\n+\tstruct reftable_iterator it = { NULL };\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint err = reftable_merged_table_seek_log(mt, &it, arg->refname);\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\twhile (err == 0) {\n+\t\tstruct reftable_log_record tombstone = {\n+\t\t\t.refname = (char*)arg->refname,\n+\t\t\t.update_index = REFTABLE_LOG_DELETION,\n+\t\t};\n+\t\terr = reftable_iterator_next_log(&it, &log);\n+\t\tif (err > 0) {\n+\t\t\terr = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (err < 0 || strcmp(log.refname, arg->refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\t\ttombstone.update_index = log.update_index;\n+\t\terr = reftable_writer_add_log(writer, &tombstone);\n+\t}\n+\n+\treturn err;\n+}\n+\n+\n+static int git_reftable_delete_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct write_reflog_delete_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tint err = reftable_stack_add(stack, &write_reflog_delete_table, &arg);\n+\tassert(err != REFTABLE_API_ERROR);\n+\treturn err;\n+}\n+\n+struct reflog_expiry_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_log_record *records;\n+\tint len;\n+};\n+\n+static int write_reflog_expiry_table(struct reftable_writer *writer, void *argv)\n+{\n+\tstruct reflog_expiry_arg *arg = (struct reflog_expiry_arg *)argv;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint i = 0;\n+\treftable_writer_set_limits(writer, ts, ts);\n+\tfor (i = 0; i < arg->len; i++) {\n+\t\tint err = reftable_writer_add_log(writer, &arg->records[i]);\n+\t\tif (err) {\n+\t\t\treturn err;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int\n+git_reftable_reflog_expire(struct ref_store *ref_store, const char *refname,\n+\t\t\t const struct object_id *oid, unsigned int flags,\n+\t\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t void *policy_cb_data)\n+{\n+\t/*\n+\t For log expiry, we write tombstones in place of the expired entries,\n+\t This means that the entries are still retrievable by delving into the\n+\t stack, and expiring entries paradoxically takes extra memory.\n+\n+\t This memory is only reclaimed when some operation issues a\n+\t git_reftable_pack_refs(), which will compact the entire stack and get\n+\t rid of deletion entries.\n+\n+\t It would be better if the refs backend supported an API that sets a\n+\t criterion for all refs, passing the criterion to pack_refs().\n+\t*/\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reflog_expiry_arg arg = {\n+\t\t.stack = stack,\n+\t};\n+\tstruct reftable_log_record *logs = NULL;\n+\tstruct reftable_log_record *rewritten = NULL;\n+\tint logs_len = 0;\n+\tint logs_cap = 0;\n+\tint i = 0;\n+\tuint8_t *last_hash = NULL;\n+\tstruct reftable_iterator it = { NULL };\n+\tstruct reftable_addition *add = NULL;\n+\tint err = 0;\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\terr = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_new_addition(&add, stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\tprepare_fn(refname, oid, policy_cb_data);\n+\twhile (1) {\n+\t\tstruct reftable_log_record log = {NULL};\n+\t\tint err = reftable_iterator_next_log(&it, &log);\n+\t\tif (err < 0) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (err > 0 || strcmp(log.refname, refname)) {\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (logs_len >= logs_cap) {\n+\t\t\tint new_cap = logs_cap * 2 + 1;\n+\t\t\tlogs = realloc(logs, new_cap * sizeof(*logs));\n+\t\t\tlogs_cap = new_cap;\n+\t\t}\n+\t\tlogs[logs_len++] = log;\n+\t}\n+\n+\trewritten = calloc(logs_len, sizeof(*rewritten));\n+\tfor (i = logs_len-1; i >= 0; i--) {\n+\t\tstruct object_id ooid;\n+\t\tstruct object_id noid;\n+\t\tstruct reftable_log_record *dest = &rewritten[i];\n+\n+\t\t*dest = logs[i];\n+\t\toidread(&ooid, logs[i].update.old_hash);\n+\t\toidread(&noid, logs[i].update.new_hash);\n+\n+\t\tif (should_prune_fn(&ooid, &noid, logs[i].update.email,\n+\t\t\t\t (timestamp_t)logs[i].update.time,\n+\t\t\t\t logs[i].update.tz_offset, logs[i].update.message,\n+\t\t\t\t policy_cb_data)) {\n+\t\t\tdest->value_type = REFTABLE_LOG_DELETION;\n+\t\t} else {\n+\t\t\tif ((flags & EXPIRE_REFLOGS_REWRITE) && last_hash != NULL) {\n+\t\t\t\tdest->update.old_hash = last_hash;\n+\t\t\t}\n+\t\t\tlast_hash = logs[i].update.new_hash;\n+\t\t}\n+\t}\n+\n+\targ.records = rewritten;\n+\targ.len = logs_len;\n+\terr = reftable_addition_add(add, &write_reflog_expiry_table, &arg);\n+\tif (err < 0) {\n+\t\tgoto done;\n+\t}\n+\n+\tif (!(flags & EXPIRE_REFLOGS_DRY_RUN)) {\n+\t\t/* XXX - skip writing records that were not changed. */\n+\t\terr = reftable_addition_commit(add);\n+\t} else {\n+\t\t/* XXX - print something */\n+\t}\n+\n+done:\n+\tif (add) {\n+\t\tcleanup_fn(policy_cb_data);\n+\t}\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_addition_destroy(add);\n+\tfor (i = 0; i < logs_len; i++)\n+\t\treftable_log_record_release(&logs[i]);\n+\tfree(logs);\n+\tfree(rewritten);\n+\treftable_iterator_destroy(&it);\n+\treturn err;\n+}\n+\n+static int reftable_error_to_errno(int err)\n+{\n+\tswitch (err) {\n+\tcase REFTABLE_IO_ERROR:\n+\t\treturn EIO;\n+\tcase REFTABLE_FORMAT_ERROR:\n+\t\treturn EFAULT;\n+\tcase REFTABLE_NOT_EXIST_ERROR:\n+\t\treturn ENOENT;\n+\tcase REFTABLE_LOCK_ERROR:\n+\t\treturn EBUSY;\n+\tcase REFTABLE_API_ERROR:\n+\t\treturn EINVAL;\n+\tcase REFTABLE_ZLIB_ERROR:\n+\t\treturn EDOM;\n+\tdefault:\n+\t\treturn ERANGE;\n+\t}\n+}\n+\n+static int git_reftable_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname, struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type)\n+{\n+\tstruct git_reftable_ref_store *refs =\n+\t\t(struct git_reftable_ref_store *)ref_store;\n+\tstruct reftable_stack *stack = stack_for(refs, refname);\n+\n+\tstruct reftable_ref_record ref = { NULL };\n+\tint err = 0;\n+\tif (refs->err < 0) {\n+\t\treturn refs->err;\n+\t}\n+\n+\t/* This is usually not needed, but Git doesn't signal to ref backend if\n+\t a subprocess updated the ref DB. So we always check.\n+\t*/\n+\terr = reftable_stack_reload(stack);\n+\tif (err) {\n+\t\tgoto done;\n+\t}\n+\n+\terr = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (err > 0) {\n+\t\terrno = ENOENT;\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\tif (err < 0) {\n+\t\terrno = reftable_error_to_errno(err);\n+\t\terr = -1;\n+\t\tgoto done;\n+\t}\n+\n+\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (reftable_ref_record_val1(&ref) != NULL) {\n+\t\toidread(oid, reftable_ref_record_val1(&ref));\n+\t} else {\n+\t\t*type |= REF_ISBROKEN;\n+\t\terrno = EINVAL;\n+\t\terr = -1;\n+\t}\n+done:\n+\tassert(err != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn err;\n+}\n+\n+struct ref_storage_be refs_be_reftable = {\n+\t&refs_be_files,\n+\t\"reftable\",\n+\tgit_reftable_ref_store_create,\n+\tgit_reftable_init_db,\n+\tgit_reftable_transaction_prepare,\n+\tgit_reftable_transaction_finish,\n+\tgit_reftable_transaction_abort,\n+\tgit_reftable_transaction_initial_commit,\n+\n+\tgit_reftable_pack_refs,\n+\tgit_reftable_create_symref,\n+\tgit_reftable_delete_refs,\n+\tgit_reftable_rename_ref,\n+\tgit_reftable_copy_ref,\n+\n+\tgit_reftable_ref_iterator_begin,\n+\tgit_reftable_read_raw_ref,\n+\n+\tgit_reftable_reflog_iterator_begin,\n+\tgit_reftable_for_each_reflog_ent_oldest_first,\n+\tgit_reftable_for_each_reflog_ent_newest_first,\n+\tgit_reftable_reflog_exists,\n+\tgit_reftable_create_reflog,\n+\tgit_reftable_delete_reflog,\n+\tgit_reftable_reflog_expire,\n+};\ndiff --git a/repository.c b/repository.c\nindex 87b355e7a659..e9f78496d338 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -174,6 +174,8 @@ int repo_init(struct repository *repo,\n \tif (worktree)\n \t\trepo_set_worktree(repo, worktree);\n \n+\trepo->ref_storage_format = xstrdup_or_null(format.ref_storage);\n+\n \tclear_repository_format(&format);\n \treturn 0;\n \ndiff --git a/repository.h b/repository.h\nindex b385ca3c94b6..8019a7d0a1fc 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -78,6 +78,9 @@ struct repository {\n \t */\n \tstruct ref_store *refs_private;\n \n+\t/* The format to use for the ref database. */\n+\tchar *ref_storage_format;\n+\n \t/*\n \t * Contains path to often used file names.\n \t */\ndiff --git a/setup.c b/setup.c\nindex 59e2facd9db6..3a4fe1ccebab 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -500,6 +500,9 @@ static enum extension_result handle_extension(const char *var,\n \t\t\treturn error(\"invalid value for 'extensions.objectformat'\");\n \t\tdata->hash_algo = format;\n \t\treturn EXTENSION_OK;\n+\t} else if (!strcmp(ext, \"refstorage\")) {\n+\t\tdata->ref_storage = xstrdup(value);\n+\t\treturn EXTENSION_OK;\n \t}\n \treturn EXTENSION_UNKNOWN;\n }\n@@ -651,6 +654,7 @@ void clear_repository_format(struct repository_format *format)\n \tstring_list_clear(&format->v1_only_extensions, 0);\n \tfree(format->work_tree);\n \tfree(format->partial_clone);\n+\tfree(format->ref_storage);\n \tinit_repository_format(format);\n }\n \n@@ -1300,8 +1304,11 @@ const char *setup_git_directory_gently(int *nongit_ok)\n \t\t\t\tgitdir = DEFAULT_GIT_DIR_ENVIRONMENT;\n \t\t\tsetup_git_env(gitdir);\n \t\t}\n-\t\tif (startup_info->have_repository)\n+\t\tif (startup_info->have_repository) {\n \t\t\trepo_set_hash_algo(the_repository, repo_fmt.hash_algo);\n+\t\t\tthe_repository->ref_storage_format =\n+\t\t\t\txstrdup_or_null(repo_fmt.ref_storage);\n+\t\t}\n \t}\n \t/*\n \t * Since precompose_string_if_needed() needs to look at\ndiff --git a/t/t0031-reftable.sh b/t/t0031-reftable.sh\nnew file mode 100755\nindex 000000000000..4685b9484214\n--- /dev/null\n+++ b/t/t0031-reftable.sh\n@@ -0,0 +1,271 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable basics'\n+\n+. ./test-lib.sh\n+\n+INVALID_SHA1=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n+\n+git_init () {\n+\tgit init -b primary \"$@\"\n+}\n+\n+initialize () {\n+\trm -rf .git &&\n+\tgit_init --ref-storage=reftable &&\n+\tmv .git/hooks .git/hooks-disabled\n+}\n+\n+write_script fake_editor <<\\EOF\n+echo \"$MSG\" >\"$1\"\n+echo \"$MSG\" >&2\n+EOF\n+GIT_EDITOR=./fake_editor\n+export GIT_EDITOR\n+\n+test_expect_success 'read existing old OID if REF_HAVE_OLD is not set' '\n+\tinitialize &&\n+\ttest_commit 1st &&\n+\ttest_commit 2nd &&\n+\tMSG=b4 git notes add &&\n+\tMSG=b3 git notes edit &&\n+\techo b4 >expect &&\n+\tgit notes --ref commits@{1} show >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git reflog delete' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_commit file2 &&\n+\ttest_commit file3 &&\n+\ttest_commit file4 &&\n+\tgit reflog delete HEAD@{1} &&\n+\tgit reflog > output &&\n+\t! grep file3 output\n+'\n+\n+test_expect_success 'branch -D delete nonexistent branch' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_must_fail git branch -D ../../my-private-file\n+'\n+\n+test_expect_success 'branch copy' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\ttest_commit file2 &&\n+\tgit branch src &&\n+\tgit reflog src > expect &&\n+\tgit branch -c src dst &&\n+\tgit reflog dst | sed \"s/dst/src/g\" > actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git stash' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttouch actual expected &&\n+\tgit -c status.showStash=true status >expected &&\n+\techo hoi >> file.t &&\n+\tgit stash push -m stashed &&\n+\tgit stash clear &&\n+\tgit -c status.showStash=true status >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'rename branch' '\n+\tinitialize &&\n+\tgit symbolic-ref HEAD refs/heads/before &&\n+\ttest_commit file &&\n+\tgit show-ref | sed s/before/after/g > expected &&\n+\tgit branch -M after &&\n+\tgit show-ref > actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'SHA256 support, env' '\n+\trm -rf .git &&\n+\tGIT_DEFAULT_HASH=sha256 && export GIT_DEFAULT_HASH &&\n+\tgit_init --ref-storage=reftable &&\n+\tmv .git/hooks .git/hooks-disabled &&\n+\ttest_commit file\n+'\n+\n+test_expect_success 'SHA256 support, option' '\n+\trm -rf .git &&\n+\tgit_init --ref-storage=reftable --object-format=sha256 &&\n+\tmv .git/hooks .git/hooks-disabled &&\n+\ttest_commit file\n+'\n+\n+test_expect_success 'delete ref' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\tSHA=$(git show-ref -s --verify HEAD) &&\n+\ttest_write_lines \"$SHA refs/heads/primary\" \"$SHA refs/tags/file\" >expect &&\n+\tgit show-ref >actual &&\n+\t! git update-ref -d refs/tags/file $INVALID_SHA1 &&\n+\ttest_cmp expect actual &&\n+\tgit update-ref -d refs/tags/file $SHA &&\n+\ttest_write_lines \"$SHA refs/heads/primary\" >expect &&\n+\tgit show-ref >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+\n+test_expect_success 'clone calls transaction_initial_commit' '\n+\ttest_commit message1 file1 &&\n+\tgit clone . cloned &&\n+\t(test -f cloned/file1 || echo \"Fixme.\")\n+'\n+\n+test_expect_success 'basic operation of reftable storage: commit, show-ref' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\ttest_write_lines refs/heads/primary refs/tags/file >expect &&\n+\tgit show-ref &&\n+\tgit show-ref | cut -f2 -d\" \" >actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'reflog, repack' '\n+\tinitialize &&\n+\tfor count in $(test_seq 1 10)\n+\tdo\n+\t\ttest_commit \"number $count\" file.t $count number-$count ||\n+\t\treturn 1\n+\tdone &&\n+\tgit pack-refs &&\n+\tls -1 .git/reftable >table-files &&\n+\ttest_line_count = 2 table-files &&\n+\tgit reflog refs/heads/primary >output &&\n+\ttest_line_count = 10 output &&\n+\tgrep \"commit (initial): number 1\" output &&\n+\tgrep \"commit: number 10\" output &&\n+\tgit gc &&\n+\tgit reflog refs/heads/primary >output &&\n+\ttest_line_count = 0 output\n+'\n+\n+test_expect_success 'branch switch in reflog output' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tgit checkout -b branch1 &&\n+\ttest_commit file2 &&\n+\tgit checkout -b branch2 &&\n+\tgit switch - &&\n+\tgit rev-parse --symbolic-full-name HEAD >actual &&\n+\techo refs/heads/branch1 >expect &&\n+\ttest_cmp actual expect\n+'\n+\n+\n+# This matches show-ref's output\n+print_ref() {\n+\techo \"$(git rev-parse \"$1\") $1\"\n+}\n+\n+test_expect_success 'peeled tags are stored' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\tgit tag -m \"annotated tag\" test_tag HEAD &&\n+\t{\n+\t\tprint_ref \"refs/heads/primary\" &&\n+\t\tprint_ref \"refs/tags/file\" &&\n+\t\tprint_ref \"refs/tags/test_tag\" &&\n+\t\tprint_ref \"refs/tags/test_tag^{}\"\n+\t} >expect &&\n+\tgit show-ref -d >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'show-ref works on fresh repo' '\n+\tinitialize &&\n+\trm -rf .git &&\n+\tgit_init --ref-storage=reftable &&\n+\t>expect &&\n+\t! git show-ref >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'checkout unborn branch' '\n+\tinitialize &&\n+\tgit checkout -b primary\n+'\n+\n+\n+test_expect_success 'dir/file conflict' '\n+\tinitialize &&\n+\ttest_commit file &&\n+\t! git branch primary/forbidden\n+'\n+\n+\n+test_expect_success 'do not clobber existing repo' '\n+\trm -rf .git &&\n+\tgit_init --ref-storage=files &&\n+\tcat .git/HEAD >expect &&\n+\ttest_commit file &&\n+\t(git_init --ref-storage=reftable || true) &&\n+\tcat .git/HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# cherry-pick uses a pseudo ref.\n+test_expect_success 'pseudo refs' '\n+\tinitialize &&\n+\ttest_commit message1 file1 &&\n+\ttest_commit message2 file2 &&\n+\tgit branch source &&\n+\tgit checkout HEAD^ &&\n+\ttest_commit message3 file3 &&\n+\tgit cherry-pick source &&\n+\ttest -f file2\n+'\n+\n+# cherry-pick uses a pseudo ref.\n+test_expect_success 'rebase' '\n+\tinitialize &&\n+\ttest_commit message1 file1 &&\n+\ttest_commit message2 file2 &&\n+\tgit branch source &&\n+\tgit checkout HEAD^ &&\n+\ttest_commit message3 file3 &&\n+\tgit rebase source &&\n+\ttest -f file2\n+'\n+\n+test_expect_success 'worktrees' '\n+\tgit_init --ref-storage=reftable start &&\n+\t(cd start && test_commit file1 && git checkout -b branch1 &&\n+\tgit checkout -b branch2 &&\n+\tgit worktree add ../wt\n+\t) &&\n+\tcd wt &&\n+\tgit checkout branch1 &&\n+\tgit branch\n+'\n+\n+test_expect_success 'worktrees 2' '\n+\tinitialize &&\n+\ttest_commit file1 &&\n+\tmkdir existing_empty &&\n+\tgit worktree add --detach existing_empty primary\n+'\n+\n+test_expect_success 'FETCH_HEAD' '\n+\tinitialize &&\n+\ttest_commit one &&\n+\t(git_init sub && cd sub && test_commit two) &&\n+\tgit --git-dir sub/.git rev-parse HEAD >expect &&\n+\tgit fetch sub &&\n+\tgit checkout FETCH_HEAD &&\n+\tgit rev-parse HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/t1409-avoid-packing-refs.sh b/t/t1409-avoid-packing-refs.sh\nindex be12fb63506e..c6f783255633 100755\n--- a/t/t1409-avoid-packing-refs.sh\n+++ b/t/t1409-avoid-packing-refs.sh\n@@ -4,6 +4,12 @@ test_description='avoid rewriting packed-refs unnecessarily'\n \n . ./test-lib.sh\n \n+if test_have_prereq REFTABLE\n+then\n+ skip_all='skipping pack-refs tests; incompatible with reftable'\n+ test_done\n+fi\n+\n # Add an identifying mark to the packed-refs file header line. This\n # shouldn't upset readers, and it should be omitted if the file is\n # ever rewritten.\ndiff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh\nindex 5071ac63a5b5..16440d58d6bb 100755\n--- a/t/t1450-fsck.sh\n+++ b/t/t1450-fsck.sh\n@@ -8,6 +8,12 @@ test_description='git fsck random collection of tests\n \n . ./test-lib.sh\n \n+if test_have_prereq REFTABLE\n+then\n+ skip_all='skipping tests; incompatible with reftable'\n+ test_done\n+fi\n+\n test_expect_success setup '\n \tgit config gc.auto 0 &&\n \tgit config i18n.commitencoding ISO-8859-1 &&\ndiff --git a/t/t3210-pack-refs.sh b/t/t3210-pack-refs.sh\nindex 3b7cdc56ec84..667a23affe36 100755\n--- a/t/t3210-pack-refs.sh\n+++ b/t/t3210-pack-refs.sh\n@@ -14,6 +14,12 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n . ./test-lib.sh\n \n+if test_have_prereq REFTABLE\n+then\n+ skip_all='skipping pack-refs tests; incompatible with reftable'\n+ test_done\n+fi\n+\n test_expect_success 'enable reflogs' '\n \tgit config core.logallrefupdates true\n '\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex d3f6af6a6545..943aa9713961 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1481,6 +1481,11 @@ parisc* | hppa*)\n \t;;\n esac\n \n+if test -n \"$GIT_TEST_REFTABLE\"\n+then\n+ test_set_prereq REFTABLE\n+fi\n+\n ( COLUMNS=1 && test $COLUMNS = 1 ) && test_set_prereq COLUMNS_CAN_BE_1\n test -z \"$NO_PERL\" && test_set_prereq PERL\n test -z \"$NO_PTHREADS\" && test_set_prereq PTHREADS\n-- \ngitgitgadget\n\n"},{"id":"422289","messageId":"b2ac95a25af804c172d474cd2d33b83cf3bdae4f.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 22/28] reftable: add dump utility","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:50Z","receivedAt":"2021-04-19T11:41:01Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nprovide a command-line utility for inspecting individual tables, and\ninspecting a complete ref database\n\nSigned-off-by: Han-Wen Nienhuys \n---\n reftable/dump.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 100 insertions(+)\n create mode 100644 reftable/dump.c\n\ndiff --git a/reftable/dump.c b/reftable/dump.c\nnew file mode 100644\nindex 000000000000..c4201f826b0c\n--- /dev/null\n+++ b/reftable/dump.c\n@@ -0,0 +1,100 @@\n+/*\n+Copyright 2020 Google LLC\n+\n+Use of this source code is governed by a BSD-style\n+license that can be found in the LICENSE file or at\n+https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+\n+#include \"reftable-blocksource.h\"\n+#include \"reftable-error.h\"\n+#include \"reftable-merged.h\"\n+#include \"reftable-record.h\"\n+#include \"reftable-tests.h\"\n+#include \"reftable-writer.h\"\n+#include \"reftable-iterator.h\"\n+#include \"reftable-reader.h\"\n+#include \"reftable-stack.h\"\n+#include \"reftable-generic.h\"\n+\n+static int compact_stack(const char *stackdir)\n+{\n+\tstruct reftable_stack *stack = NULL;\n+\tstruct reftable_write_options cfg = { 0 };\n+\n+\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_stack_compact_all(stack, NULL);\n+\tif (err < 0)\n+\t\tgoto done;\n+done:\n+\tif (stack) {\n+\t\treftable_stack_destroy(stack);\n+\t}\n+\treturn err;\n+}\n+\n+static void print_help(void)\n+{\n+\tprintf(\"usage: dump [-cst] arg\\n\\n\"\n+\t \"options: \\n\"\n+\t \" -c compact\\n\"\n+\t \" -t dump table\\n\"\n+\t \" -s dump stack\\n\"\n+\t \" -h this help\\n\"\n+\t \"\\n\");\n+}\n+\n+int reftable_dump_main(int argc, char *const *argv)\n+{\n+\tint err = 0;\n+\tint opt_dump_table = 0;\n+\tint opt_dump_stack = 0;\n+\tint opt_compact = 0;\n+\tconst char *arg = NULL, *argv0 = argv[0];\n+\n+\tfor (; argc > 1; argv++, argc--)\n+\t\tif (*argv[1] != '-')\n+\t\t\tbreak;\n+\t\telse if (!strcmp(\"-t\", argv[1]))\n+\t\t\topt_dump_table = 1;\n+\t\telse if (!strcmp(\"-s\", argv[1]))\n+\t\t\topt_dump_stack = 1;\n+\t\telse if (!strcmp(\"-c\", argv[1]))\n+\t\t\topt_compact = 1;\n+\t\telse if (!strcmp(\"-?\", argv[1]) || !strcmp(\"-h\", argv[1])) {\n+\t\t\tprint_help();\n+\t\t\treturn 2;\n+\t\t}\n+\n+\tif (argc != 2) {\n+\t\tfprintf(stderr, \"need argument\\n\");\n+\t\tprint_help();\n+\t\treturn 2;\n+\t}\n+\n+\targ = argv[1];\n+\n+\tif (opt_dump_table) {\n+\t\terr = reftable_reader_print_file(arg);\n+\t} else if (opt_dump_stack) {\n+\t\terr = reftable_stack_print_directory(arg);\n+\t} else if (opt_compact) {\n+\t\terr = compact_stack(arg);\n+\t}\n+\n+\tif (err < 0) {\n+\t\tfprintf(stderr, \"%s: %s: %s\\n\", argv0, arg,\n+\t\t\treftable_error_str(err));\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n-- \ngitgitgadget\n\n"},{"id":"422291","messageId":"ae88f41de9a92fb0335216a708edc0a8e5e77c2c.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 26/28] t1301: document what needs to be done for REFTABLE","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:54Z","receivedAt":"2021-04-19T11:41:04Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n t/t1301-shared-repo.sh | 8 +++++---\n 1 file changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t1301-shared-repo.sh b/t/t1301-shared-repo.sh\nindex ac947bff9fcf..194fa54ea3e7 100755\n--- a/t/t1301-shared-repo.sh\n+++ b/t/t1301-shared-repo.sh\n@@ -22,9 +22,10 @@ test_expect_success 'shared = 0400 (faulty permission u-w)' '\n \t)\n '\n \n+# TODO(hanwen): for REFTABLE should inspect group-readable of .git/reftable/\n for u in 002 022\n do\n-\ttest_expect_success POSIXPERM \"shared=1 does not clear bits preset by umask $u\" '\n+\ttest_expect_success REFFILES,POSIXPERM \"shared=1 does not clear bits preset by umask $u\" '\n \t\tmkdir sub && (\n \t\t\tcd sub &&\n \t\t\tumask $u &&\n@@ -114,7 +115,8 @@ test_expect_success POSIXPERM 'info/refs respects umask in unshared repo' '\n \ttest_cmp expect actual\n '\n \n-test_expect_success POSIXPERM 'git reflog expire honors core.sharedRepository' '\n+# For reftable, the check on .git/reftable/ is sufficient.\n+test_expect_success REFFILES,POSIXPERM 'git reflog expire honors core.sharedRepository' '\n \tumask 077 &&\n \tgit config core.sharedRepository group &&\n \tgit reflog expire --all &&\n@@ -201,7 +203,7 @@ test_expect_success POSIXPERM 're-init respects core.sharedrepository (remote)'\n \ttest_cmp expect actual\n '\n \n-test_expect_success POSIXPERM 'template can set core.sharedrepository' '\n+test_expect_success REFFILES,POSIXPERM 'template can set core.sharedrepository' '\n \trm -rf child.git &&\n \tumask 0022 &&\n \tgit config core.sharedrepository 0666 &&\n-- \ngitgitgadget\n\n"},{"id":"422292","messageId":"aca82763e72caa7b4b5eb93ceb428a6c18ab8b31.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 25/28] Add \"test-tool dump-reftable\" command.","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:53Z","receivedAt":"2021-04-19T11:41:13Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nThis command dumps individual tables or a stack of of tables.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n Makefile | 1 +\n t/helper/test-reftable.c | 5 +++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 8 insertions(+)\n\ndiff --git a/Makefile b/Makefile\nindex cffcf3496978..115ba0ccba0a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2427,6 +2427,7 @@ REFTABLE_OBJS += reftable/writer.o\n \n REFTABLE_TEST_OBJS += reftable/basics_test.o\n REFTABLE_TEST_OBJS += reftable/block_test.o\n+REFTABLE_TEST_OBJS += reftable/dump.o\n REFTABLE_TEST_OBJS += reftable/merged_test.o\n REFTABLE_TEST_OBJS += reftable/pq_test.o\n REFTABLE_TEST_OBJS += reftable/record_test.o\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 996da85f7b5c..26b03d7b7892 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -14,3 +14,8 @@ int cmd__reftable(int argc, const char **argv)\n \ttree_test_main(argc, argv);\n \treturn 0;\n }\n+\n+int cmd__dump_reftable(int argc, const char **argv)\n+{\n+\treturn reftable_dump_main(argc, (char *const *)argv);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex bea3285eed9f..6922f64c1683 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -58,6 +58,7 @@ static struct test_cmd cmds[] = {\n \t{ \"read-midx\", cmd__read_midx },\n \t{ \"ref-store\", cmd__ref_store },\n \t{ \"reftable\", cmd__reftable },\n+\t{ \"dump-reftable\", cmd__dump_reftable },\n \t{ \"regex\", cmd__regex },\n \t{ \"repository\", cmd__repository },\n \t{ \"revision-walking\", cmd__revision_walking },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 90f935310788..3ccc869f4595 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -19,6 +19,7 @@ int cmd__dump_cache_tree(int argc, const char **argv);\n int cmd__dump_fsmonitor(int argc, const char **argv);\n int cmd__dump_split_index(int argc, const char **argv);\n int cmd__dump_untracked_cache(int argc, const char **argv);\n+int cmd__dump_reftable(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__fast_rebase(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n-- \ngitgitgadget\n\n"},{"id":"422293","messageId":"742d716e451061540ec3a1df53e0c9dbb08d9124.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 24/28] git-prompt: prepare for reftable refs backend","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:52Z","receivedAt":"2021-04-19T11:41:14Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nIn our git-prompt script we strive to use Bash builtins wherever\npossible, because fork()-ing subshells for command substitutions and\nfork()+exec()-ing Git commands are expensive on some platforms. We\neven read and parse '.git/HEAD' using Bash builtins to get the name of\nthe current branch [1]. However, the upcoming reftable refs backend\nwon't use '.git/HEAD' at all, but will write an invalid refname as\nplaceholder for backwards compatibility instead, which will break our\ngit-prompt script.\n\nUpdate the git-prompt script to recognize the placeholder '.git/HEAD'\nwritten by the reftable backend (its content is specified in the\nreftable specs), and then fall back to use 'git symbolic-ref' to get\nthe name of the current branch.\n\n[1] 3a43c4b5bd (bash prompt: use bash builtins to find out current\n branch, 2011-03-31)\n\nSigned-off-by: SZEDER Gábor \n---\n contrib/completion/git-prompt.sh | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/completion/git-prompt.sh b/contrib/completion/git-prompt.sh\nindex 4640a1535d15..2e5a5d80271d 100644\n--- a/contrib/completion/git-prompt.sh\n+++ b/contrib/completion/git-prompt.sh\n@@ -478,10 +478,15 @@ __git_ps1 ()\n \t\t\tif ! __git_eread \"$g/HEAD\" head; then\n \t\t\t\treturn $exit\n \t\t\tfi\n-\t\t\t# is it a symbolic ref?\n \t\t\tb=\"${head#ref: }\"\n \t\t\tif [ \"$head\" = \"$b\" ]; then\n \t\t\t\tdetached=yes\n+\t\t\telif [ \"$b\" = \"refs/heads/.invalid\" ]; then\n+\t\t\t\t# Reftable\n+\t\t\t\tb=\"$(git symbolic-ref HEAD 2>/dev/null)\" ||\n+\t\t\t\tdetached=yes\n+\t\t\tfi\n+\t\t\tif [ \"$detached\" = yes ]; then\n \t\t\t\tb=\"$(\n \t\t\t\tcase \"${GIT_PS1_DESCRIBE_STYLE-}\" in\n \t\t\t\t(contains)\n-- \ngitgitgadget\n\n"},{"id":"422295","messageId":"dd8ffbf53157bde0c8dcc7d778b6f8d5849499fd.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 28/28] t1404: annotate test cases with REFFILES","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:56Z","receivedAt":"2021-04-19T11:41:14Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\n* Reftable for now lacks detailed error messages for directory/file conflicts.\n Skip message comparisons.\n\n* Mark tests that muck with .git directly as REFFILES.\n\nSigned-off-by: Han-Wen Nienhuys \n---\n t/t1404-update-ref-errors.sh | 86 ++++++++++++++++++++++++------------\n 1 file changed, 57 insertions(+), 29 deletions(-)\n\ndiff --git a/t/t1404-update-ref-errors.sh b/t/t1404-update-ref-errors.sh\nindex 8b51c4efc135..811d5bb56d41 100755\n--- a/t/t1404-update-ref-errors.sh\n+++ b/t/t1404-update-ref-errors.sh\n@@ -27,7 +27,9 @@ test_update_rejected () {\n \tfi &&\n \tprintf \"create $prefix/%s $C\\n\" $create >input &&\n \ttest_must_fail git update-ref --stdin output.err &&\n-\ttest_i18ngrep -F \"$error\" output.err &&\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_i18ngrep -F \"$error\" output.err\n+\tfi &&\n \tgit for-each-ref $prefix >actual &&\n \ttest_cmp unchanged actual\n }\n@@ -101,7 +103,9 @@ df_test() {\n \t\tprintf \"%s\\n\" \"delete $delname\" \"create $addname $D\"\n \tfi >commands &&\n \ttest_must_fail git update-ref --stdin output.err &&\n-\ttest_cmp expected-err output.err &&\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected-err output.err\n+\tfi &&\n \tprintf \"%s\\n\" \"$C $delref\" >expected-refs &&\n \tgit for-each-ref --format=\"%(objectname) %(refname)\" $prefix/r >actual-refs &&\n \ttest_cmp expected-refs actual-refs\n@@ -189,7 +193,7 @@ test_expect_success 'one new ref is a simple prefix of another' '\n \n '\n \n-test_expect_success 'empty directory should not fool rev-parse' '\n+test_expect_success REFFILES 'empty directory should not fool rev-parse' '\n \tprefix=refs/e-rev-parse &&\n \tgit update-ref $prefix/foo $C &&\n \tgit pack-refs --all &&\n@@ -199,7 +203,7 @@ test_expect_success 'empty directory should not fool rev-parse' '\n \ttest_cmp expected actual\n '\n \n-test_expect_success 'empty directory should not fool for-each-ref' '\n+test_expect_success REFFILES 'empty directory should not fool for-each-ref' '\n \tprefix=refs/e-for-each-ref &&\n \tgit update-ref $prefix/foo $C &&\n \tgit for-each-ref $prefix >expected &&\n@@ -209,14 +213,14 @@ test_expect_success 'empty directory should not fool for-each-ref' '\n \ttest_cmp expected actual\n '\n \n-test_expect_success 'empty directory should not fool create' '\n+test_expect_success REFFILES 'empty directory should not fool create' '\n \tprefix=refs/e-create &&\n \tmkdir -p .git/$prefix/foo/bar/baz &&\n \tprintf \"create %s $C\\n\" $prefix/foo |\n \tgit update-ref --stdin\n '\n \n-test_expect_success 'empty directory should not fool verify' '\n+test_expect_success REFFILES 'empty directory should not fool verify' '\n \tprefix=refs/e-verify &&\n \tgit update-ref $prefix/foo $C &&\n \tgit pack-refs --all &&\n@@ -225,7 +229,7 @@ test_expect_success 'empty directory should not fool verify' '\n \tgit update-ref --stdin\n '\n \n-test_expect_success 'empty directory should not fool 1-arg update' '\n+test_expect_success REFFILES 'empty directory should not fool 1-arg update' '\n \tprefix=refs/e-update-1 &&\n \tgit update-ref $prefix/foo $C &&\n \tgit pack-refs --all &&\n@@ -234,7 +238,7 @@ test_expect_success 'empty directory should not fool 1-arg update' '\n \tgit update-ref --stdin\n '\n \n-test_expect_success 'empty directory should not fool 2-arg update' '\n+test_expect_success REFFILES 'empty directory should not fool 2-arg update' '\n \tprefix=refs/e-update-2 &&\n \tgit update-ref $prefix/foo $C &&\n \tgit pack-refs --all &&\n@@ -243,7 +247,7 @@ test_expect_success 'empty directory should not fool 2-arg update' '\n \tgit update-ref --stdin\n '\n \n-test_expect_success 'empty directory should not fool 0-arg delete' '\n+test_expect_success REFFILES 'empty directory should not fool 0-arg delete' '\n \tprefix=refs/e-delete-0 &&\n \tgit update-ref $prefix/foo $C &&\n \tgit pack-refs --all &&\n@@ -252,7 +256,7 @@ test_expect_success 'empty directory should not fool 0-arg delete' '\n \tgit update-ref --stdin\n '\n \n-test_expect_success 'empty directory should not fool 1-arg delete' '\n+test_expect_success REFFILES 'empty directory should not fool 1-arg delete' '\n \tprefix=refs/e-delete-1 &&\n \tgit update-ref $prefix/foo $C &&\n \tgit pack-refs --all &&\n@@ -336,7 +340,9 @@ test_expect_success 'missing old value blocks update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/foo $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks update' '\n@@ -347,7 +353,9 @@ test_expect_success 'incorrect old value blocks update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/foo $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'existing old value blocks create' '\n@@ -358,7 +366,9 @@ test_expect_success 'existing old value blocks create' '\n \tEOF\n \tprintf \"%s\\n\" \"create $prefix/foo $E\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks delete' '\n@@ -369,7 +379,9 @@ test_expect_success 'incorrect old value blocks delete' '\n \tEOF\n \tprintf \"%s\\n\" \"delete $prefix/foo $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'missing old value blocks indirect update' '\n@@ -380,7 +392,9 @@ test_expect_success 'missing old value blocks indirect update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect update' '\n@@ -392,7 +406,9 @@ test_expect_success 'incorrect old value blocks indirect update' '\n \tEOF\n \tprintf \"%s\\n\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'existing old value blocks indirect create' '\n@@ -404,7 +420,9 @@ test_expect_success 'existing old value blocks indirect create' '\n \tEOF\n \tprintf \"%s\\n\" \"create $prefix/symref $E\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect delete' '\n@@ -416,7 +434,9 @@ test_expect_success 'incorrect old value blocks indirect delete' '\n \tEOF\n \tprintf \"%s\\n\" \"delete $prefix/symref $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'missing old value blocks indirect no-deref update' '\n@@ -427,7 +447,9 @@ test_expect_success 'missing old value blocks indirect no-deref update' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect no-deref update' '\n@@ -439,7 +461,9 @@ test_expect_success 'incorrect old value blocks indirect no-deref update' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"update $prefix/symref $E $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'existing old value blocks indirect no-deref create' '\n@@ -451,7 +475,9 @@ test_expect_success 'existing old value blocks indirect no-deref create' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"create $prefix/symref $E\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n test_expect_success 'incorrect old value blocks indirect no-deref delete' '\n@@ -463,10 +489,12 @@ test_expect_success 'incorrect old value blocks indirect no-deref delete' '\n \tEOF\n \tprintf \"%s\\n\" \"option no-deref\" \"delete $prefix/symref $D\" |\n \ttest_must_fail git update-ref --stdin 2>output.err &&\n-\ttest_cmp expected output.err\n+\tif test_have_prereq REFFILES ; then\n+\t\ttest_cmp expected output.err\n+\tfi\n '\n \n-test_expect_success 'non-empty directory blocks create' '\n+test_expect_success REFFILES 'non-empty directory blocks create' '\n \tprefix=refs/ne-create &&\n \tmkdir -p .git/$prefix/foo/bar &&\n \t: >.git/$prefix/foo/bar/baz.lock &&\n@@ -485,7 +513,7 @@ test_expect_success 'non-empty directory blocks create' '\n \ttest_cmp expected output.err\n '\n \n-test_expect_success 'broken reference blocks create' '\n+test_expect_success REFFILES 'broken reference blocks create' '\n \tprefix=refs/broken-create &&\n \tmkdir -p .git/$prefix &&\n \techo \"gobbledigook\" >.git/$prefix/foo &&\n@@ -504,7 +532,7 @@ test_expect_success 'broken reference blocks create' '\n \ttest_cmp expected output.err\n '\n \n-test_expect_success 'non-empty directory blocks indirect create' '\n+test_expect_success REFFILES 'non-empty directory blocks indirect create' '\n \tprefix=refs/ne-indirect-create &&\n \tgit symbolic-ref $prefix/symref $prefix/foo &&\n \tmkdir -p .git/$prefix/foo/bar &&\n@@ -524,7 +552,7 @@ test_expect_success 'non-empty directory blocks indirect create' '\n \ttest_cmp expected output.err\n '\n \n-test_expect_success 'broken reference blocks indirect create' '\n+test_expect_success REFFILES 'broken reference blocks indirect create' '\n \tprefix=refs/broken-indirect-create &&\n \tgit symbolic-ref $prefix/symref $prefix/foo &&\n \techo \"gobbledigook\" >.git/$prefix/foo &&\n@@ -543,7 +571,7 @@ test_expect_success 'broken reference blocks indirect create' '\n \ttest_cmp expected output.err\n '\n \n-test_expect_success 'no bogus intermediate values during delete' '\n+test_expect_success REFFILES 'no bogus intermediate values during delete' '\n \tprefix=refs/slow-transaction &&\n \t# Set up a reference with differing loose and packed versions:\n \tgit update-ref $prefix/foo $C &&\n@@ -600,7 +628,7 @@ test_expect_success 'no bogus intermediate values during delete' '\n \ttest_must_fail git rev-parse --verify --quiet $prefix/foo\n '\n \n-test_expect_success 'delete fails cleanly if packed-refs file is locked' '\n+test_expect_success REFFILES 'delete fails cleanly if packed-refs file is locked' '\n \tprefix=refs/locked-packed-refs &&\n \t# Set up a reference with differing loose and packed versions:\n \tgit update-ref $prefix/foo $C &&\n@@ -616,7 +644,7 @@ test_expect_success 'delete fails cleanly if packed-refs file is locked' '\n \ttest_cmp unchanged actual\n '\n \n-test_expect_success 'delete fails cleanly if packed-refs.new write fails' '\n+test_expect_success REFFILES 'delete fails cleanly if packed-refs.new write fails' '\n \t# Setup and expectations are similar to the test above.\n \tprefix=refs/failed-packed-refs &&\n \tgit update-ref $prefix/foo $C &&\n-- \ngitgitgadget\n"},{"id":"422294","messageId":"0c1eb69900a7aa9650dcc695ef18793602320cce.1618832277.git.gitgitgadget@gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"[PATCH v7 27/28] t1401,t2011: parameterize HEAD.lock for REFTABLE","fromName":"Han-Wen Nienhuys via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-19T11:37:55Z","receivedAt":"2021-04-19T11:41:15Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"From: Han-Wen Nienhuys \n\nSigned-off-by: Han-Wen Nienhuys \n---\n t/t1401-symbolic-ref.sh | 11 +++++++++--\n t/t2011-checkout-invalid-head.sh | 11 +++++++++--\n 2 files changed, 18 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t1401-symbolic-ref.sh b/t/t1401-symbolic-ref.sh\nindex a4ebb0b65fec..4e130776b40f 100755\n--- a/t/t1401-symbolic-ref.sh\n+++ b/t/t1401-symbolic-ref.sh\n@@ -99,9 +99,16 @@ test_expect_success LONG_REF 'we can parse long symbolic ref' '\n \ttest_cmp expect actual\n '\n \n+if test_have_prereq REFTABLE\n+then\n+\tHEAD_LOCK=reftable/tables.list.lock\n+else\n+\tHEAD_LOCK=HEAD.lock\n+fi\n+\n test_expect_success 'symbolic-ref reports failure in exit code' '\n-\ttest_when_finished \"rm -f .git/HEAD.lock\" &&\n-\t>.git/HEAD.lock &&\n+\ttest_when_finished \"rm -f .git/$HEAD_LOCK\" &&\n+\t>.git/$HEAD_LOCK &&\n \ttest_must_fail git symbolic-ref HEAD refs/heads/whatever\n '\n \ndiff --git a/t/t2011-checkout-invalid-head.sh b/t/t2011-checkout-invalid-head.sh\nindex e52022e15229..ef6471cb0954 100755\n--- a/t/t2011-checkout-invalid-head.sh\n+++ b/t/t2011-checkout-invalid-head.sh\n@@ -22,9 +22,16 @@ test_expect_success 'checkout main from invalid HEAD' '\n \tgit checkout main --\n '\n \n+if test_have_prereq REFTABLE\n+then\n+\tHEAD_LOCK=reftable/tables.list.lock\n+else\n+\tHEAD_LOCK=HEAD.lock\n+fi\n+\n test_expect_success 'checkout notices failure to lock HEAD' '\n-\ttest_when_finished \"rm -f .git/HEAD.lock\" &&\n-\t>.git/HEAD.lock &&\n+\ttest_when_finished \"rm -f .git/$HEAD_LOCK\" &&\n+\t>.git/$HEAD_LOCK &&\n \ttest_must_fail git checkout -b other\n '\n \n-- \ngitgitgadget\n\n"},{"id":"422458","messageId":"xmqq35vk7pzd.fsf@gitster.g","threadId":"54245","inReplyTo":"8103a80394aefdd4e8b5061dfbb6a6199fe5fcae.1618832276.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 01/28] refs: ref_iterator_peel returns boolean, rather than peel_status","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-20T18:47:18Z","receivedAt":"2021-04-20T18:47:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Han-Wen Nienhuys via GitGitGadget\" writes:\n\n> From: Han-Wen Nienhuys \n>\n> Before, the cached ref_iterator would return peel_object() output directly. This\n> led to spurious differences in the GIT_TRACE_REFS output, depending on the ref\n> storage backend active.\n>\n> Signed-off-by: Han-Wen Nienhuys \n> ---\n> refs.c | 2 +-\n> refs/ref-cache.c | 2 +-\n> refs/refs-internal.h | 3 +++\n> 3 files changed, 5 insertions(+), 2 deletions(-)\n\nA few observations.\n\n * The ref_iterator_peel() is defined to \"Return 0 on success.\" in\n refs/refs-internal.h and implication of the missing mention of\n what is returned on failure is that it can give any non-zero\n value back, and in this project a failure usually is signaled by\n returning a negative value.\n\n * If the trace output cares that all non-success trace entries to\n look identical, I wonder if that layer should be the one that\n normalizes \"0 is success, any other value is failure\" into \"0 is\n success, 1 is failure\" (even though I find a positive 1 used as a\n failure a bit odd).\n\n * refs.c::peel_object() is defined to return \"enum peel_status\"\n which is not even a simple yes/no.\n\n PEEL_PEELED = 0\n PEEL_INVALID = -1\n PEEL_NON_TAG = -2\n PEEL_IS_SYMREF = -3\n PEEL_BROKEN = -4\n\n The comment in refs/refs-internal.h about \"Reference interators\"\n suggests me that ref_iterator_peel() is supposed to be a moral\n equivalent of (recently gone) peel_ref() which was a thin wrapper\n to still surviving peel_object(), so I expect ref_iterator_peel()\n to allow the caller to differentiate various failure modes. And\n as a way to debug into a running system, I am not sure if losing\n the distinction in the trace output is even desirable.\n\npacked_ref_iterator_peel() does use !!peel_object(), but I have a\nfeeling that that is what should be fixed and not the other way\naround.\n\nI haven't seen a good justification given to help convince me that\nthis is a good change (and I presume it is, as I trust you or any\nother contributores enough not to knowingly make the system worse),\n\n> +/*\n> + * Peels the current ref, returning 0 for success.\n> + */\n\nAnd if it does make sense to squash the peel status down to \"bool\",\nthen the comment should mention the single acceptable value for\nfailure, not just \"0 for success\" which implies \"different negative\nvalues depending on the nature of failure\".\n\n> typedef int ref_iterator_peel_fn(struct ref_iterator *ref_iterator,\n> \t\t\t\t struct object_id *peeled);\n\nThanks.\n"},{"id":"422464","messageId":"xmqqy2dc698w.fsf@gitster.g","threadId":"54245","inReplyTo":"554bb1ac3aed066eb578a305b7fe76f1660f6153.1618832277.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 02/28] refs: document reflog_expire_fn's flag argument","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-20T19:34:07Z","receivedAt":"2021-04-20T19:34:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Han-Wen Nienhuys via GitGitGadget\" writes:\n\n> From: Han-Wen Nienhuys \n>\n> Signed-off-by: Han-Wen Nienhuys \n> ---\n> refs/refs-internal.h | 4 ++++\n> 1 file changed, 4 insertions(+)\n>\n> diff --git a/refs/refs-internal.h b/refs/refs-internal.h\n> index 546a6b965dcc..a31c1f465beb 100644\n> --- a/refs/refs-internal.h\n> +++ b/refs/refs-internal.h\n> @@ -592,6 +592,10 @@ typedef int reflog_exists_fn(struct ref_store *ref_store, const char *refname);\n> typedef int create_reflog_fn(struct ref_store *ref_store, const char *refname,\n> \t\t\t int force_create, struct strbuf *err);\n> typedef int delete_reflog_fn(struct ref_store *ref_store, const char *refname);\n> +\n> +/*\n> + * `flags` accepts a bitmask of `expire_reflog_flags`.\n> + */\n\nOK. It would have been better to say what expire_reflog_flags is\n(i.e. `enum expire_reflog_flags`), though.\n\n> typedef int reflog_expire_fn(struct ref_store *ref_store,\n> \t\t\t const char *refname, const struct object_id *oid,\n> \t\t\t unsigned int flags,\n"},{"id":"422466","messageId":"xmqqsg3k68vv.fsf@gitster.g","threadId":"54245","inReplyTo":"9ae5ddff6aed48184d2a10c569e41441b9199f10.1618832277.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 03/28] refs/debug: trace into reflog expiry too","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-20T19:41:56Z","receivedAt":"2021-04-20T19:42:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Han-Wen Nienhuys via GitGitGadget\" writes:\n\n> From: Han-Wen Nienhuys \n>\n> Signed-off-by: Han-Wen Nienhuys \n> ---\n> refs/debug.c | 47 ++++++++++++++++++++++++++++++++++++++++++++---\n> 1 file changed, 44 insertions(+), 3 deletions(-)\n\nNicely done. \n\nI as a reader of this patch do have to wonder, with the above very\nlimited log message material, how useful did \"debug_reflog_expire()\"\nmachinery used to be without any tracing.\n\nIt just reported the fact that expire method was called and what the\nbackend did as a whole, instead of reporting what the machinery\ndecided for each reflog entry to be pruned (or not pruned)?\n\nNot a problem with this patch at all, and certainly it does not have\nto be part of this series, but it feels very backwards, at least to\nme, to have the method should_prune in ref backends. As a function\nto make a policy decision (e.g. \"this has a timestamp older than X,\nso needs to be pruned\", \"the author of this change I do not like, so\nlet's prune it ;-)\"), it is more natural to have it as independent\nas possible from the individual backends, no?\n"},{"id":"422467","messageId":"xmqqlf9c68iy.fsf@gitster.g","threadId":"54245","inReplyTo":"f0216ae20b6988514bdb60d8fff96e18c2ce9f1d.1618832277.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 04/28] hash.h: provide constants for the hash IDs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-20T19:49:41Z","receivedAt":"2021-04-20T19:49:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Han-Wen Nienhuys via GitGitGadget\" writes:\n\n> From: Han-Wen Nienhuys \n>\n> This will simplify referencing them from code that is not deeply integrated with\n> Git, in particular, the reftable library.\n>\n> Signed-off-by: Han-Wen Nienhuys \n> ---\n> hash.h | 6 ++++++\n> object-file.c | 6 ++----\n> 2 files changed, 8 insertions(+), 4 deletions(-)\n>\n> diff --git a/hash.h b/hash.h\n> index 3fb0c3d4005a..b17fb2927711 100644\n> --- a/hash.h\n> +++ b/hash.h\n> @@ -161,12 +161,18 @@ static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n> \treturn p - hash_algos;\n> }\n> \n> +/* \"sha1\", big-endian */\n> +#define GIT_SHA1_HASH_ID 0x73686131\n> +\n> /* The length in bytes and in hex digits of an object name (SHA-1 value). */\n> #define GIT_SHA1_RAWSZ 20\n> #define GIT_SHA1_HEXSZ (2 * GIT_SHA1_RAWSZ)\n> /* The block size of SHA-1. */\n> #define GIT_SHA1_BLKSZ 64\n> \n> +/* \"s256\", big-endian */\n> +#define GIT_SHA256_HASH_ID 0x73323536\n\nI agree that it makes sense to have symbolic constants defined in\nthis file. These are values that fit in the \".format_id\" member of\n\"struct git_hash_algo\", and it is preferrable to make sure that the\nnames align (if I were designing in void, I would have called the\nmember \"algo_id\" and made the constants GIT_(SHA1|SHA256)_ALGO_ID).\n\nBrian? What's your preference (\"I am fine to store HASH_ID in the\n'.format_id' member\" is perfectly an acceptable answer).\n\nThanks.\n"},{"id":"422487","messageId":"xmqqk0ow37as.fsf@gitster.g","threadId":"54245","inReplyTo":"2fd7cb8c0983501e2af2f195aec81d7c17fb80e1.1618832277.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 23/28] Reftable support for git-core","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-20T22:44:27Z","receivedAt":"2021-04-20T22:44:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Han-Wen Nienhuys via GitGitGadget\" writes:\n\n> +if test -n \"$GIT_TEST_REFTABLE\"\n> +then\n> + test_set_prereq REFTABLE\n> +fi\n\nHow would this interact with what the test prep series did which was\nto unconditionally add\n\ntest_seq_prereq REFFILES\n\naround the same place? Should $GIT_TEST_REFTABLE disable REFFILES?\nIOW, do you want the above to read\n\n\tif test -n \"$GIT_TEST_REFTABLE\"\n\tthen\n\t\ttest_set_prereq REFTABLE\n\telse\n\t\ttest_set_prereq REFFILES\n\tfi\n\nwhen both series are in effect?\n\nThanks.\n"},{"id":"422516","messageId":"YH96CcJhZt4CTPWD@camp.crustytoothpaste.net","threadId":"54245","inReplyTo":"xmqqlf9c68iy.fsf@gitster.g","subject":"Re: [PATCH v7 04/28] hash.h: provide constants for the hash IDs","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-04-21T01:04:09Z","receivedAt":"2021-04-21T01:04:47Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-04-20 at 19:49:41, Junio C Hamano wrote:\n> \"Han-Wen Nienhuys via GitGitGadget\" writes:\n> \n> > From: Han-Wen Nienhuys \n> >\n> > This will simplify referencing them from code that is not deeply integrated with\n> > Git, in particular, the reftable library.\n> >\n> > Signed-off-by: Han-Wen Nienhuys \n> > ---\n> > hash.h | 6 ++++++\n> > object-file.c | 6 ++----\n> > 2 files changed, 8 insertions(+), 4 deletions(-)\n> >\n> > diff --git a/hash.h b/hash.h\n> > index 3fb0c3d4005a..b17fb2927711 100644\n> > --- a/hash.h\n> > +++ b/hash.h\n> > @@ -161,12 +161,18 @@ static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n> > \treturn p - hash_algos;\n> > }\n> > \n> > +/* \"sha1\", big-endian */\n> > +#define GIT_SHA1_HASH_ID 0x73686131\n> > +\n> > /* The length in bytes and in hex digits of an object name (SHA-1 value). */\n> > #define GIT_SHA1_RAWSZ 20\n> > #define GIT_SHA1_HEXSZ (2 * GIT_SHA1_RAWSZ)\n> > /* The block size of SHA-1. */\n> > #define GIT_SHA1_BLKSZ 64\n> > \n> > +/* \"s256\", big-endian */\n> > +#define GIT_SHA256_HASH_ID 0x73323536\n> \n> I agree that it makes sense to have symbolic constants defined in\n> this file. These are values that fit in the \".format_id\" member of\n> \"struct git_hash_algo\", and it is preferrable to make sure that the\n> names align (if I were designing in void, I would have called the\n> member \"algo_id\" and made the constants GIT_(SHA1|SHA256)_ALGO_ID).\n> \n> Brian? What's your preference (\"I am fine to store HASH_ID in the\n> '.format_id' member\" is perfectly an acceptable answer).\n\nI slightly prefer FORMAT_ID because it's consistent (and for that reason\nonly), but if HASH_ID is more convenient, that's fine; I don't have a\nstrong opinion at all. Definitely don't reroll the series because of my\nslight preference. Either way, I think these are fine things to have as\nconstants, and I appreciate you hoisting the comments here.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"},{"id":"422541","messageId":"878s5c3wti.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"pull.847.v7.git.git.1618832276.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 00/28] reftable library","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-21T07:45:29Z","receivedAt":"2021-04-21T07:45:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Apr 19 2021, Han-Wen Nienhuys via GitGitGadget wrote:\n\n> The commits up to \"hash.h: provide constants for the hash IDs\" should be\n> good to merge to 'next'.\n\nWith how Junio queues things up perhaps submitting this as another\n\"prep\" series would be good, to reduce future reviewer fatigue,\ni.e. anything we can make land on master makes re-rolls that much\nsmaller.\n\n> There are several test fixups, but I've put them in another series because\n> GGG enforces max 30 commits.\n\nI left a bunch of comments on the test prep series now. Probably good to\nhave it split up regardless of GGG limits.\n\nRe the comments I left on the test series. I'm very happy to see the\nstart of addressing the \"it must be tested\" concerns I had in\nhttps://lore.kernel.org/git/87wnt2th1v.fsf@evledraar.gmail.com/\n\nBut running the test suite with GIT_TEST_REFTABLE=true still yields a\nwall of failed tests. I tested with a merger of this and\npr-git-1008/hanwen/reffiles-v1, i.e. what's on \"seen\" currently.\n\nI don't see the point of having re-rolls of this topic while the test\nchanges topic it's based on hasn't finished\nmarking/splitting/refactoring the various tests that do and don't depend\non the file backend.\n\nAt least when I review it I'm just left with going in circles digging\ninto one of those failing test, figuring out if it's really\nrefs/files-backend.c specific or not etc., and as long as we can't turn\non GIT_TEST_REFTABLE=true in CI as part of this series I don't see a\npath to making it advance to next->master.\n\n> Due to using uncompress2, this build fails on the Linux32 builder; what is\n> the magic incantation to run autoconf?\n\nDoesn't this just need a NO_UNCOMPRESS2 for Linux32 in ci/lib.sh? See\nthe lines just below that for e.g. NO_REGEX for another CI target.\n"},{"id":"422542","messageId":"875z0g3wp9.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"8a960fb77fa9117009119ce37aaa9885749bd1a5.1618832277.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 06/28] reftable: add LICENSE","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-21T07:48:02Z","receivedAt":"2021-04-21T07:48:09Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Apr 19 2021, Han-Wen Nienhuys via GitGitGadget wrote:\n\n> From: Han-Wen Nienhuys \n>\n> TODO: relicense?\n\nSo, still no resolution to the questions in v6 in [1]?\n\n1. https://lore.kernel.org/git/87o8ei2tfo.fsf@evledraar.gmail.com/\n"},{"id":"422551","messageId":"CAFQ2z_OwTzb03MPsER1QVxro3D06rdbPABb6s3JnjC5F0b0BnA@mail.gmail.com","threadId":"54245","inReplyTo":"875z0g3wp9.fsf@evledraar.gmail.com","subject":"Re: [PATCH v7 06/28] reftable: add LICENSE","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-04-21T09:15:34Z","receivedAt":"2021-04-21T09:15:51Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Wed, Apr 21, 2021 at 9:48 AM Ævar Arnfjörð Bjarmason\n wrote:\n>\n>\n> On Mon, Apr 19 2021, Han-Wen Nienhuys via GitGitGadget wrote:\n>\n> > From: Han-Wen Nienhuys \n> >\n> > TODO: relicense?\n>\n> So, still no resolution to the questions in v6 in [1]?\n\nI reworded the commit message, but that update was lost in one of the\ncopious rebase operations. I recovered it from reflog, and changed it.\nHopefully it survives this again.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"422552","messageId":"CAFQ2z_NAQ02y1gaTymzi21aHxh+bswvr6ko_Eg=1uuHds4mGyg@mail.gmail.com","threadId":"54245","inReplyTo":"YH96CcJhZt4CTPWD@camp.crustytoothpaste.net","subject":"Re: [PATCH v7 04/28] hash.h: provide constants for the hash IDs","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-04-21T09:43:18Z","receivedAt":"2021-04-21T09:43:32Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Wed, Apr 21, 2021 at 3:04 AM brian m. carlson\n wrote:\n> > I agree that it makes sense to have symbolic constants defined in\n> > this file. These are values that fit in the \".format_id\" member of\n> > \"struct git_hash_algo\", and it is preferrable to make sure that the\n> > names align (if I were designing in void, I would have called the\n> > member \"algo_id\" and made the constants GIT_(SHA1|SHA256)_ALGO_ID).\n> >\n> > Brian? What's your preference (\"I am fine to store HASH_ID in the\n> > '.format_id' member\" is perfectly an acceptable answer).\n>\n> I slightly prefer FORMAT_ID because it's consistent (and for that reason\n> only), but if HASH_ID is more convenient, that's fine; I don't have a\n> strong opinion at all. Definitely don't reroll the series because of my\n> slight preference. Either way, I think these are fine things to have as\n> constants, and I appreciate you hoisting the comments here.\n\nEither way is fine for me. I can paint the bikeshed in your favorite color :)\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"422553","messageId":"CAFQ2z_MF3SbKf8yXBvkOcLGg03dVxJW3F=c6ZDUtstEbZUUMCQ@mail.gmail.com","threadId":"54245","inReplyTo":"878s5c3wti.fsf@evledraar.gmail.com","subject":"Re: [PATCH v7 00/28] reftable library","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-04-21T09:52:52Z","receivedAt":"2021-04-21T09:53:07Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Wed, Apr 21, 2021 at 9:45 AM Ævar Arnfjörð Bjarmason\n wrote:\n> > The commits up to \"hash.h: provide constants for the hash IDs\" should be\n> > good to merge to 'next'.\n>\n> With how Junio queues things up perhaps submitting this as another\n> \"prep\" series would be good, to reduce future reviewer fatigue,\n> i.e. anything we can make land on master makes re-rolls that much\n> smaller.\n\nwill do.\n\n> > There are several test fixups, but I've put them in another series because\n> > GGG enforces max 30 commits.\n>\n> I left a bunch of comments on the test prep series now. Probably good to\n> have it split up regardless of GGG limits.\n>\n> Re the comments I left on the test series. I'm very happy to see the\n> start of addressing the \"it must be tested\" concerns I had in\n> https://lore.kernel.org/git/87wnt2th1v.fsf@evledraar.gmail.com/\n\nIt may look like the start, but I've been improving the number of\ntests that pass continuously since I posted the first version of this\ncode over a year ago.\n\n> I don't see the point of having re-rolls of this topic while the test\n> changes topic it's based on hasn't finished\n> marking/splitting/refactoring the various tests that do and don't depend\n> on the file backend.\n>\n> At least when I review it I'm just left with going in circles digging\n> into one of those failing test, figuring out if it's really\n> refs/files-backend.c specific or not etc., and as long as we can't turn\n> on GIT_TEST_REFTABLE=true in CI as part of this series I don't see a\n> path to making it advance to next->master.\n\nThe point of posting updates is to garner feedback, especially from\npeople familiar with the Git code itself. If you would like this\neffort to move forward faster, I am most in need of review for the\npart that glues the library together with the git code itself (ie. the\ncommit introducing refs/reftable-backend.c).\n\n> > Due to using uncompress2, this build fails on the Linux32 builder; what is\n> > the magic incantation to run autoconf?\n>\n> Doesn't this just need a NO_UNCOMPRESS2 for Linux32 in ci/lib.sh? See\n> the lines just below that for e.g. NO_REGEX for another CI target.\n\nThanks!\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"422568","messageId":"CAFQ2z_MaS_jBof5VaqRg8K+0NN2n1GQ7mwnKnvjy__yzchwv1Q@mail.gmail.com","threadId":"54245","inReplyTo":"xmqq35vk7pzd.fsf@gitster.g","subject":"Re: [PATCH v7 01/28] refs: ref_iterator_peel returns boolean, rather than peel_status","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-04-21T10:15:40Z","receivedAt":"2021-04-21T10:15:57Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Tue, Apr 20, 2021 at 8:47 PM Junio C Hamano wrote:\n>\n> \"Han-Wen Nienhuys via GitGitGadget\" writes:\n>\n> > From: Han-Wen Nienhuys \n> >\n> > Before, the cached ref_iterator would return peel_object() output directly. This\n> > led to spurious differences in the GIT_TRACE_REFS output, depending on the ref\n> > storage backend active.\n> >\n> > Signed-off-by: Han-Wen Nienhuys \n> > ---\n> > refs.c | 2 +-\n> > refs/ref-cache.c | 2 +-\n> > refs/refs-internal.h | 3 +++\n> > 3 files changed, 5 insertions(+), 2 deletions(-)\n>\n> A few observations.\n>\n> * The ref_iterator_peel() is defined to \"Return 0 on success.\" in\n> refs/refs-internal.h and implication of the missing mention of\n> what is returned on failure is that it can give any non-zero\n> value back, and in this project a failure usually is signaled by\n> returning a negative value.\n>\n> * If the trace output cares that all non-success trace entries to\n> look identical, I wonder if that layer should be the one that\n> normalizes \"0 is success, any other value is failure\" into \"0 is\n> success, 1 is failure\" (even though I find a positive 1 used as a\n> failure a bit odd).\n\nNo, it should not. The whole point of the trace output is to see\nwhere and how the files backend and the reftable backend return\ndifferent results. By having the trace layer paper over differences, I\nwill miss regressions of the reftable backend.\n\n> * refs.c::peel_object() is defined to return \"enum peel_status\"\n> which is not even a simple yes/no.\n>\n> PEEL_PEELED = 0\n> PEEL_INVALID = -1\n> PEEL_NON_TAG = -2\n> PEEL_IS_SYMREF = -3\n> PEEL_BROKEN = -4\n>\n> The comment in refs/refs-internal.h about \"Reference interators\"\n> suggests me that ref_iterator_peel() is supposed to be a moral\n> equivalent of (recently gone) peel_ref() which was a thin wrapper\n> to still surviving peel_object(), so I expect ref_iterator_peel()\n> to allow the caller to differentiate various failure modes. And\n> as a way to debug into a running system, I am not sure if losing\n> the distinction in the trace output is even desirable.\n>\n> packed_ref_iterator_peel() does use !!peel_object(), but I have a\n> feeling that that is what should be fixed and not the other way\n> around.\n\nAll these functions go into ref_iterator_peel, which is only called\nfrom peel_iterated_oid().\n\nThat function calls !!peel_object(base, peeled) if there is no\niterator. That leads me to conclude\n\nAll callers of peel_iterated_oid use it as a boolean exclusively, i.e.\n\n if (!peel_iterated_oid( .. )) { .. }\n\nAside from these considerations, it is also odd to return peel_status.\nFor example, PEEL_IS_SYMREF is unnecessary, because the symref status\nis already returned from iterator_next().\n\n> I haven't seen a good justification given to help convince me that\n> this is a good change (and I presume it is, as I trust you or any\n> other contributores enough not to knowingly make the system worse),\n>\n> > +/*\n> > + * Peels the current ref, returning 0 for success.\n> > + */\n>\n> And if it does make sense to squash the peel status down to \"bool\",\n> then the comment should mention the single acceptable value for\n> failure, not just \"0 for success\" which implies \"different negative\n> values depending on the nature of failure\".\n\nI can make it return -1 instead.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\nRegistergericht und -nummer: Hamburg, HRB 86891\nSitz der Gesellschaft: Hamburg\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"422570","messageId":"CAFQ2z_Nm4Mixhi0kXTdN+nVj412_WPe=k9v6FpJFyaOx9qMRUA@mail.gmail.com","threadId":"54245","inReplyTo":"xmqqk0ow37as.fsf@gitster.g","subject":"Re: [PATCH v7 23/28] Reftable support for git-core","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-04-21T10:19:09Z","receivedAt":"2021-04-21T10:19:29Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Wed, Apr 21, 2021 at 12:44 AM Junio C Hamano wrote:\n> > +if test -n \"$GIT_TEST_REFTABLE\"\n> > +then\n> > + test_set_prereq REFTABLE\n> > +fi\n>\n> How would this interact with what the test prep series did which was\n> to unconditionally add\n>\n> test_seq_prereq REFFILES\n>\n> around the same place? Should $GIT_TEST_REFTABLE disable REFFILES?\n> IOW, do you want the above to read\n>\n> if test -n \"$GIT_TEST_REFTABLE\"\n> then\n> test_set_prereq REFTABLE\n> else\n> test_set_prereq REFFILES\n> fi\n>\n> when both series are in effect?\n\nYes, but on 2nd thought it's probably better to stick with just\nREFFILES, and rewrite any reftable specific tests as !REFFILES.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"422574","messageId":"87lf9b3mth.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"CAFQ2z_MF3SbKf8yXBvkOcLGg03dVxJW3F=c6ZDUtstEbZUUMCQ@mail.gmail.com","subject":"Re: [PATCH v7 00/28] reftable library","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-21T11:21:30Z","receivedAt":"2021-04-21T11:21:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Apr 21 2021, Han-Wen Nienhuys wrote:\n\n> On Wed, Apr 21, 2021 at 9:45 AM Ævar Arnfjörð Bjarmason\n> wrote:\n>> > The commits up to \"hash.h: provide constants for the hash IDs\" should be\n>> > good to merge to 'next'.\n>>\n>> With how Junio queues things up perhaps submitting this as another\n>> \"prep\" series would be good, to reduce future reviewer fatigue,\n>> i.e. anything we can make land on master makes re-rolls that much\n>> smaller.\n>\n> will do.\n>\n>> > There are several test fixups, but I've put them in another series because\n>> > GGG enforces max 30 commits.\n>>\n>> I left a bunch of comments on the test prep series now. Probably good to\n>> have it split up regardless of GGG limits.\n>>\n>> Re the comments I left on the test series. I'm very happy to see the\n>> start of addressing the \"it must be tested\" concerns I had in\n>> https://lore.kernel.org/git/87wnt2th1v.fsf@evledraar.gmail.com/\n>\n> It may look like the start, but I've been improving the number of\n> tests that pass continuously since I posted the first version of this\n> code over a year ago.\n\nFWIW I meant or meant to say something closer to \"a start at the\nnumerous failures I noted in the v6 discussion\", not \"no work has been\ndone at all on this front\". Sorry.\n\n>> I don't see the point of having re-rolls of this topic while the test\n>> changes topic it's based on hasn't finished\n>> marking/splitting/refactoring the various tests that do and don't depend\n>> on the file backend.\n>>\n>> At least when I review it I'm just left with going in circles digging\n>> into one of those failing test, figuring out if it's really\n>> refs/files-backend.c specific or not etc., and as long as we can't turn\n>> on GIT_TEST_REFTABLE=true in CI as part of this series I don't see a\n>> path to making it advance to next->master.\n>\n> The point of posting updates is to garner feedback, especially from\n> people familiar with the Git code itself.\n\nSo you agree that we should make the tests pass first, then shouldn't\nthese me marked as RFC/PATCH?\n\nI for one would find that a lot less confusing, \"PATCH\" means \"the\nauthor considers this ready to land on master sans undiscovered bugs\netc.\n\n> If you would like this effort to move forward faster,\n\nYes, I'm keen to help move it forward.\n\nI am saying that I think for me or anyone else to do that in any\nsensible way the path forward is to make the tests pass with\nGIT_TEST_REFTABLE=true.\n\nIOW a large part of the feedback you're looking for is already part of\nthe codebase. Nobody can keep all of it in their head, but we've encoded\nall the tricky edge cases we could think of in the tests.\n\nSo in particular per my feedback on the test series: It's only when we\nstart digging into those tests that we discover the interesting bits,\ni.e. how things like .git/SOME_FILE behaves per[1], and my comments\nabout reflog behavior in [2].\n\nAnyway, I think I'm just repeating myself at this point, but part of the\nreason is that I don't think I've gotten a straight answer about the\npoint-by-point questions I had in [3].\n\nI.e. reaching some consensus on things like whether\nGIT_TEST_REFTABLE=true passing under CI being a hard-prereq for this\nseries or not is surely one of the first things to sketch out before\nfiguring out how to move this forward.\n\nAlso, with how unhappy Junio is with my patch-dumping I'm probably the\nlast person to give advice about managing mailing list attention, but\nstill: here's an attempt:\n\nIn v6 I noted that t5510-fetch.sh had a segfault[4], you said you'd\ncheck it out, and reading your cover letter nothing stood out about\nthat, so I assumed it was sorted out somehow.\n\nBut running it now yields a BUG() instead:\n \n BUG: refs.c:1038: free called on a prepared reference transaction\n Aborted\n [...]\n not ok 18 - fetch --atomic aborts all reference updates if hook aborts\n\nAnd trying the whole test suite with --verbose-log yields:\n \n $ grep -e 'Segmentation fault' -e BUG: test-results/*\n test-results/t0210-trace2-normal.out:BUG: t/helper/test-trace2.c:206: the bug message\n test-results/t1400-update-ref.out:BUG: refs.c:1038: free called on a prepared reference transaction\n test-results/t1400-update-ref.out:BUG: refs.c:1038: free called on a prepared reference transaction\n test-results/t4058-diff-duplicates.out:Segmentation fault\n test-results/t4058-diff-duplicates.out:Segmentation fault\n test-results/t4058-diff-duplicates.out:Segmentation fault\n test-results/t4058-diff-duplicates.out:Segmentation fault\n test-results/t5510-fetch.out:BUG: refs.c:1038: free called on a prepared reference transaction\n test-results/t5510-fetch.out:BUG: refs.c:1038: free called on a prepared reference transaction\n test-results/t5600-clone-fail-cleanup.out:Segmentation fault\n test-results/t5600-clone-fail-cleanup.out:Segmentation fault\n test-results/t5600-clone-fail-cleanup.out:Segmentation fault\n test-results/t5601-clone.out:Segmentation fault\n test-results/t6423-merge-rename-directories.out: test_i18ngrep ! BUG: err &&\n\nNot all of that is yours, FWIW \"seen\" is currently doing, and the \"diff\"\nfailures are ac14de13b2 (t4058: explore duplicate tree entry handling in\na bit more detail, 2020-12-11).\n \n $ grep -e 'Segmentation fault' -e BUG: test-results/*\n test-results/t0210-trace2-normal.out:BUG: t/helper/test-trace2.c:206: the bug message\n test-results/t1406-submodule-ref-store.out:BUG: refs/files-backend.c:139: operation pack_refs requires abilities 0x6, but only have 0x5\n test-results/t1406-submodule-ref-store.out:BUG: refs/files-backend.c:139: operation create_symref requires abilities 0x2, but only have 0x5\n test-results/t1406-submodule-ref-store.out:BUG: refs/files-backend.c:139: operation delete_refs requires abilities 0x2, but only have 0x5\n test-results/t1406-submodule-ref-store.out:BUG: refs/files-backend.c:139: operation rename_ref requires abilities 0x2, but only have 0x5\n test-results/t1406-submodule-ref-store.out:BUG: refs/files-backend.c:139: operation delete_reflog requires abilities 0x2, but only have 0x5\n test-results/t1406-submodule-ref-store.out:BUG: refs/files-backend.c:139: operation create_reflog requires abilities 0x2, but only have 0x5\n test-results/t4058-diff-duplicates.out:Segmentation fault\n test-results/t4058-diff-duplicates.out:Segmentation fault\n test-results/t4058-diff-duplicates.out:Segmentation fault\n test-results/t4058-diff-duplicates.out:Segmentation fault\n test-results/t6423-merge-rename-directories.out: test_i18ngrep ! BUG: err &&\n\nAnyway, the \"clone\"/\"fetch\"/\"update-ref\" failures look new with\nGIT_TEST_REFTABLE=true.\n\nSo to close up the attempt at getting feedback: I think we'd probably be\nbetter off still discussing \"I tried this to fix the segfault, but now I\nget this BUG\" rather than a 30 patch re-roll.\n\n> I am most in need of review for the part that glues the library\n> together with the git code itself (ie. the commit introducing\n> refs/reftable-backend.c).\n\nIn other and briefer words I don't think this series is in need of\nreview at this point, it's in need of *addressing* review, where review\nis both the ghosts from the past of failing tests, and outstanding\nsegfaults/BUG()s being hit.\n\n1. https://lore.kernel.org/git/87k0ow2n29.fsf@evledraar.gmail.com/ \n2. https://lore.kernel.org/git/87pmyo3zvw.fsf@evledraar.gmail.com/\n3. https://lore.kernel.org/git/87wnt2th1v.fsf@evledraar.gmail.com/\n4. https://lore.kernel.org/git/87im4qejpk.fsf@evledraar.gmail.com/\n"},{"id":"422641","messageId":"xmqqeef3xlxg.fsf@gitster.g","threadId":"54245","inReplyTo":"CAFQ2z_Nm4Mixhi0kXTdN+nVj412_WPe=k9v6FpJFyaOx9qMRUA@mail.gmail.com","subject":"Re: [PATCH v7 23/28] Reftable support for git-core","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-21T23:22:35Z","receivedAt":"2021-04-21T23:22:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Han-Wen Nienhuys writes:\n\n>> IOW, do you want the above to read\n>>\n>> if test -n \"$GIT_TEST_REFTABLE\"\n>> then\n>> test_set_prereq REFTABLE\n>> else\n>> test_set_prereq REFFILES\n>> fi\n>>\n>> when both series are in effect?\n>\n> Yes, but on 2nd thought it's probably better to stick with just\n> REFFILES, and rewrite any reftable specific tests as !REFFILES.\n\nI guess that is both good enough and simpler. The only possible\ndownside is that it would be cumbersome if we ever need to support\nthe third ref backend, but I expect that it won't happen before the\nreftable support solidifies, and under that assumption, the simpler\nthe better.\n\nThanks.\n"},{"id":"422643","messageId":"xmqqa6prxln3.fsf@gitster.g","threadId":"54245","inReplyTo":"CAFQ2z_MaS_jBof5VaqRg8K+0NN2n1GQ7mwnKnvjy__yzchwv1Q@mail.gmail.com","subject":"Re: [PATCH v7 01/28] refs: ref_iterator_peel returns boolean, rather than peel_status","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-21T23:28:48Z","receivedAt":"2021-04-21T23:28:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Han-Wen Nienhuys writes:\n\n> All callers of peel_iterated_oid use it as a boolean exclusively, i.e.\n>\n> if (!peel_iterated_oid( .. )) { .. }\n>\n> Aside from these considerations, it is also odd to return peel_status.\n> For example, PEEL_IS_SYMREF is unnecessary, because the symref status\n> is already returned from iterator_next().\n>\n>> I haven't seen a good justification given to help convince me that\n>> this is a good change (and I presume it is, as I trust you or any\n>> other contributores enough not to knowingly make the system worse),\n\nI am not sure I agree 100% with\n\n\tif (!do_this()) {\n\t\t...\n\t}\n\ni.e. \"try doing it and do one thing upon success\" is a sign that all\nerror conditions will ever be treated equally and justifies to squash\nall the error codes the current code tries to return, but whether I\nagree with it or not, I'd want to see it recorded in the proposed\nlog message. It would help future developers and allow them to take\ninto account your motivation behind this change, when they need to\nupdate the implementation.\n\n>> > +/*\n>> > + * Peels the current ref, returning 0 for success.\n>> > + */\n>>\n>> And if it does make sense to squash the peel status down to \"bool\",\n>> then the comment should mention the single acceptable value for\n>> failure, not just \"0 for success\" which implies \"different negative\n>> values depending on the nature of failure\".\n>\n> I can make it return -1 instead.\n\nI do care more about _documenting_ it here, not just ending the\nsentence with \", returning 0 for success\", it should also say\n\"returns X upon failure\", whether the value of X is 1 or -1 (and\nobviously our codebase prefers -1 for an error, but that is\nsecondary).\n\nThanks.\n\n\t\n"},{"id":"422697","messageId":"CAFQ2z_PTF=5wLKXmqzFkPiC3RTy-RtzCAtBHUSWekkgaKJekkw@mail.gmail.com","threadId":"54245","inReplyTo":"xmqqsg3k68vv.fsf@gitster.g","subject":"Re: [PATCH v7 03/28] refs/debug: trace into reflog expiry too","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-04-22T17:27:51Z","receivedAt":"2021-04-22T17:28:05Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Tue, Apr 20, 2021 at 9:42 PM Junio C Hamano wrote:\n> Nicely done.\n>\n> I as a reader of this patch do have to wonder, with the above very\n> limited log message material, how useful did \"debug_reflog_expire()\"\n> machinery used to be without any tracing.\n\nIt wasn't; that's why I'm changing it :-)\n\nI noticed a test failure with reftable, and adding this function\nshowed me I was expiring the entries in the reverse order, so that\n\n git reflog delete branch@{1}\n\nwould remove one but oldest entry in the reflog.\n\n> Not a problem with this patch at all, and certainly it does not have\n> to be part of this series, but it feels very backwards, at least to\n> me, to have the method should_prune in ref backends. As a function\n> to make a policy decision (e.g. \"this has a timestamp older than X,\n> so needs to be pruned\", \"the author of this change I do not like, so\n> let's prune it ;-)\"), it is more natural to have it as independent\n> as possible from the individual backends, no?\n\nthe should_prune function is passed in into the ref backend as an\nargument of type reflog_expiry_should_prune_fn to the\nrefs_reflog_expire() function.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\nRegistergericht und -nummer: Hamburg, HRB 86891\nSitz der Gesellschaft: Hamburg\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"422746","messageId":"CAFQ2z_Pco4KhhE7S_8zBZ7hCwOH=DnxHoKE8zus7REdQaXCZiQ@mail.gmail.com","threadId":"54245","inReplyTo":"871ree8eto.fsf@evledraar.gmail.com","subject":"Re: [PATCH v4 13/15] Reftable support for git-core","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-04-23T10:22:04Z","receivedAt":"2021-04-23T10:22:18Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Thu, Jan 21, 2021 at 4:55 PM Ævar Arnfjörð Bjarmason\n wrote:\n> I've been chasing down some edge cases in refs.c as part of another WIP\n> series I have and found that for this particular \"errno\" stuff we don't\n> have any test coverage. And with master & hanwen/reftable if I apply:\n>\n> - errno = EINVAL;\n..\n> All tests pass on both.\n..\n> That I can remove error checking/handling from this place means existing\n> general logic was faithfully copied without checking that we have a test\n> for it, or adding one.\n\nI think that is an unfair characterization. The API documentation of\nread_raw_ref_fn says that implementations should return EINVAL for\ncertain cases, so that's what I did. The point of having a documented\ninternal API is that one doesn't have to double check what is behind\nthe API.\n\n--\nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"422927","messageId":"87lf953yto.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"CAFQ2z_Pco4KhhE7S_8zBZ7hCwOH=DnxHoKE8zus7REdQaXCZiQ@mail.gmail.com","subject":"Re: [PATCH v4 13/15] Reftable support for git-core","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-26T13:23:11Z","receivedAt":"2021-04-26T14:28:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Apr 23 2021, Han-Wen Nienhuys wrote:\n\n> On Thu, Jan 21, 2021 at 4:55 PM Ævar Arnfjörð Bjarmason\n> wrote:\n>> I've been chasing down some edge cases in refs.c as part of another WIP\n>> series I have and found that for this particular \"errno\" stuff we don't\n>> have any test coverage. And with master & hanwen/reftable if I apply:\n>>\n>> - errno = EINVAL;\n> ..\n>> All tests pass on both.\n> ..\n>> That I can remove error checking/handling from this place means existing\n>> general logic was faithfully copied without checking that we have a test\n>> for it, or adding one.\n>\n> I think that is an unfair characterization. The API documentation of\n> read_raw_ref_fn says that implementations should return EINVAL for\n> certain cases, so that's what I did. The point of having a documented\n> internal API is that one doesn't have to double check what is behind\n> the API.\n\nI'm only meaning to point out that the behavior being relied on here\ndoesn't have any tests.\n\nI agree that whether or not we should have any tests at all is a matter\nof opinion and circumstance. I don't think in general that someone using\nsome random internal API needs to be checking what is and isn't tested\nin that API.\n\nI do think in this case that it's worth digging a bit further. The APIs\nin question are using EINVAL and EISDIR to pass up errors that are had a\n1:1 mapping into the FS historically.\n\nAre we really better off faking those up, or is some of that perhaps not\nused at all? Maybe we'd find that this level of abstraction isn't what\nwe need, and it could actually be much simpler.\n\nMy main goal here is not to keep creating some infinite amount of work\nfor you, but to try to help this along into some shape that's better\nreviewable & land-able.\n\nIt seems to me that a good way to get there is to seek some systematic\nways of focusing review onto various edge cases of this series. I.e. to\nbegin with having GIT_TEST_REFTABLE pass as noted elsewhere, and in this\ncase calling attention to some of the underlying assumptions behind this\nseries.\n\nOne of the hardest things I've found about trying to review this has\nbeen closing the gap between things that exist in your mind and commit\nmessages / code.\n\nE.g. in this case sure, you can be of the opinion that since the point\nof an internal API is to provide an interface we can simply assume it's\nOK and working as intended.\n\nI don't think given the history and use of this code that that's an\nassumption I'd be as comfortable making, but anyway. Getting back on\npoint, I do think whatever opinion anyone is of on that subject having\nsomething like this in the commit message (and other applicable commits\netc.) would be *very* valuable:\n\n In functions such as git_reftable_read_raw_ref() (and ????, are\n there more?) we're diligently emulating the documented behavior and\n return values of the refs file backend. According to \"make gcov\" we\n can see we don't have coverage for this, in particular the behavior\n of EINVAL etc.\n\nI.e. per [1] once if and when we have GIT_TEST_REFTABLE passing surely\none of the best way to garner feedback on the rest is to discover those\nparts (using \"make gcov\", after running with/without\nGIT_TEST_REFTABLE=[true|false]) where we still have outstanding blind\nspots between the tests and code.\n\n1. https://lore.kernel.org/git/87lf9b3mth.fsf@evledraar.gmail.com/\n"},{"id":"422963","messageId":"CAFQ2z_Pb62Oe352UwdsPWLxjXd7P97pyuggTMK2URJd2p=OTnA@mail.gmail.com","threadId":"54245","inReplyTo":"87lf953yto.fsf@evledraar.gmail.com","subject":"Re: [PATCH v4 13/15] Reftable support for git-core","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-04-26T16:17:48Z","receivedAt":"2021-04-26T16:18:07Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Mon, Apr 26, 2021 at 4:28 PM Ævar Arnfjörð Bjarmason\n wrote:\n> I agree that whether or not we should have any tests at all is a matter\n> of opinion and circumstance. I don't think in general that someone using\n> some random internal API needs to be checking what is and isn't tested\n> in that API.\n>\n> I do think in this case that it's worth digging a bit further. The APIs\n> in question are using EINVAL and EISDIR to pass up errors that are had a\n> 1:1 mapping into the FS historically.\n>\n> Are we really better off faking those up, or is some of that perhaps not\n> used at all? Maybe we'd find that this level of abstraction isn't what\n> we need, and it could actually be much simpler.\n\nI think none of these error functions are being used at all, and I've\nmade a start at removing them in https://github.com/git/git/pull/1012\n(see also my discussion with peff.)\n\n> It seems to me that a good way to get there is to seek some systematic\n> ways of focusing review onto various edge cases of this series. I.e. to\n> begin with having GIT_TEST_REFTABLE pass as noted elsewhere, and in this\n> case calling attention to some of the underlying assumptions behind this\n> series.\n\n> One of the hardest things I've found about trying to review this has\n> been closing the gap between things that exist in your mind and commit\n> messages / code.\n\nthanks, that is valuable feedback.\n\n> something like this in the commit message (and other applicable commits\n> etc.) would be *very* valuable:\n>\n> In functions such as git_reftable_read_raw_ref() (and ????, are\n> there more?) we're diligently emulating the documented behavior and\n> return values of the refs file backend. According to \"make gcov\" we\n> can see we don't have coverage for this, in particular the behavior\n> of EINVAL etc.\n\nI haven't done this, because a lot of these considerations are\ntransient, and I'd rather not spend a lot of time documenting what I\ndon't know.\n\n> I.e. per [1] once if and when we have GIT_TEST_REFTABLE passing surely\n> one of the best way to garner feedback on the rest is to discover those\n> parts (using \"make gcov\", after running with/without\n> GIT_TEST_REFTABLE=[true|false]) where we still have outstanding blind\n> spots between the tests and code.\n\nGetting GIT_TEST_REFTABLE=1 passing is the hard part, because it means\nhaving to understand exactly how the current code is supposed to\nwork. Once I get to that point (with knowledge being complete and\ntests passing), it will be easy to document what is happening and why.\n\nI was hoping that by posting these series with known test failures,\nand questions marked \"XXX\" in reftable-backend.c, I would get feedback\nfrom the other people who know exactly how this part of the code\nworks. But from your mail, I get the sense that nobody understands\nhow the whole picture fits together?\n\n--\nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"423023","messageId":"CAFQ2z_Ow38_80kA5W3vPyPnZFTSCAF7nwYavgki=GJVm44P7EQ@mail.gmail.com","threadId":"54245","inReplyTo":"87lf9b3mth.fsf@evledraar.gmail.com","subject":"Re: [PATCH v7 00/28] reftable library","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-04-26T17:59:25Z","receivedAt":"2021-04-26T17:59:43Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Wed, Apr 21, 2021 at 1:21 PM Ævar Arnfjörð Bjarmason\n wrote:\n\n> >> I don't see the point of having re-rolls of this topic while the test\n> >> changes topic it's based on hasn't finished\n> >> marking/splitting/refactoring the various tests that do and don't depend\n> >> on the file backend.\n> >>\n> >> At least when I review it I'm just left with going in circles digging\n> >> into one of those failing test, figuring out if it's really\n> >> refs/files-backend.c specific or not etc., and as long as we can't turn\n> >> on GIT_TEST_REFTABLE=true in CI as part of this series I don't see a\n> >> path to making it advance to next->master.\n> >\n> > The point of posting updates is to garner feedback, especially from\n> > people familiar with the Git code itself.\n>\n> So you agree that we should make the tests pass first, then shouldn't\n> these me marked as RFC/PATCH?\n\nI added RFC to the patch introducing reftable-backend.c\n\nI consider the code under reftable/*.c to be of production quality,\neven if it's stylistically different from the git code.\n\n> IOW a large part of the feedback you're looking for is already part of\n> the codebase. Nobody can keep all of it in their head, but we've encoded\n> all the tricky edge cases we could think of in the tests.\n\nThat's disappointing. I know that I can just fix test failures until\nthere are none left, but it's not a terribly efficient method if there\nis someone who knows how this works. Is there no other way to solicit\nfeedback?\n\n> I.e. reaching some consensus on things like whether\n> GIT_TEST_REFTABLE=true passing under CI being a hard-prereq for this\n> series or not is surely one of the first things to sketch out before\n> figuring out how to move this forward.\n\nI've had conflicting feedback about this, but it's ultimately not my\ncall to make. Jonathan Nieder suggested that this was asking too much,\nand it would be OK to have a credible starting point where the effort\nto address remaining test failures could be shared across multiple\npeople. Jun also expressed he didn't want to put all the work on my\nshoulders, but so far I've not received much help.\n\n> In v6 I noted that t5510-fetch.sh had a segfault[4], you said you'd\n> check it out, and reading your cover letter nothing stood out about\n> that, so I assumed it was sorted out somehow.\n\nI fixed a number of segfaults, so it's quite likely (I did see the\nfreed transaction BUG).\n\nOne of the problems of working with a large pending series like this,\nis that it is a project in itself, and keeping track of which change\nfixes which bug is something I really need version control for, but\nthe frequent rebasing/squashing erases this kind of information.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"423110","messageId":"CAFQ2z_PUwjVOShyDi67bw2X7K1x=s66Lpwn14UcWeUrksJSRbw@mail.gmail.com","threadId":"54245","inReplyTo":"xmqqy2dc698w.fsf@gitster.g","subject":"Re: [PATCH v7 02/28] refs: document reflog_expire_fn's flag argument","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-04-27T15:21:25Z","receivedAt":"2021-04-27T15:21:40Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Tue, Apr 20, 2021 at 9:34 PM Junio C Hamano wrote:> > +/*\n> > + * `flags` accepts a bitmask of `expire_reflog_flags`.\n> > + */\n>\n> OK. It would have been better to say what expire_reflog_flags is\n> (i.e. `enum expire_reflog_flags`), though.\n\nDone, in the last version.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"423203","messageId":"87h7jqz7k5.fsf@evledraar.gmail.com","threadId":"54245","inReplyTo":"CAFQ2z_Pb62Oe352UwdsPWLxjXd7P97pyuggTMK2URJd2p=OTnA@mail.gmail.com","subject":"Re: [PATCH v4 13/15] Reftable support for git-core","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-28T16:32:31Z","receivedAt":"2021-04-28T16:40:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Apr 26 2021, Han-Wen Nienhuys wrote:\n\n> On Mon, Apr 26, 2021 at 4:28 PM Ævar Arnfjörð Bjarmason\n> wrote:\n>> I agree that whether or not we should have any tests at all is a matter\n>> of opinion and circumstance. I don't think in general that someone using\n>> some random internal API needs to be checking what is and isn't tested\n>> in that API.\n>>\n>> I do think in this case that it's worth digging a bit further. The APIs\n>> in question are using EINVAL and EISDIR to pass up errors that are had a\n>> 1:1 mapping into the FS historically.\n>>\n>> Are we really better off faking those up, or is some of that perhaps not\n>> used at all? Maybe we'd find that this level of abstraction isn't what\n>> we need, and it could actually be much simpler.\n>\n> I think none of these error functions are being used at all, and I've\n> made a start at removing them in https://github.com/git/git/pull/1012\n> (see also my discussion with peff.)\n>\n>> It seems to me that a good way to get there is to seek some systematic\n>> ways of focusing review onto various edge cases of this series. I.e. to\n>> begin with having GIT_TEST_REFTABLE pass as noted elsewhere, and in this\n>> case calling attention to some of the underlying assumptions behind this\n>> series.\n>\n>> One of the hardest things I've found about trying to review this has\n>> been closing the gap between things that exist in your mind and commit\n>> messages / code.\n>\n> thanks, that is valuable feedback.\n>\n>> something like this in the commit message (and other applicable commits\n>> etc.) would be *very* valuable:\n>>\n>> In functions such as git_reftable_read_raw_ref() (and ????, are\n>> there more?) we're diligently emulating the documented behavior and\n>> return values of the refs file backend. According to \"make gcov\" we\n>> can see we don't have coverage for this, in particular the behavior\n>> of EINVAL etc.\n>\n> I haven't done this, because a lot of these considerations are\n> transient, and I'd rather not spend a lot of time documenting what I\n> don't know.\n>\n>> I.e. per [1] once if and when we have GIT_TEST_REFTABLE passing surely\n>> one of the best way to garner feedback on the rest is to discover those\n>> parts (using \"make gcov\", after running with/without\n>> GIT_TEST_REFTABLE=[true|false]) where we still have outstanding blind\n>> spots between the tests and code.\n>\n> Getting GIT_TEST_REFTABLE=1 passing is the hard part, because it means\n> having to understand exactly how the current code is supposed to\n> work. Once I get to that point (with knowledge being complete and\n> tests passing), it will be easy to document what is happening and why.\n>\n> I was hoping that by posting these series with known test failures,\n> and questions marked \"XXX\" in reftable-backend.c, I would get feedback\n> from the other people who know exactly how this part of the code\n> works. But from your mail, I get the sense that nobody understands\n> how the whole picture fits together?\n\nAlmost definitely not. I don't know about you but when I'm looking at\ncode I wrote 6 months ago handling some special case it takes me a while\nto get up to speed on just knowing what I knew then, and when we're\ntalking about something like refs.c ...\n\nOn the topic of the way forward: I for one would very much be for a plan\nwhere step 0 is to just a series import the reftable code you have\nas-is. I.e. we'd include it as an imported external library, maybe have\nsome light test-tool integration and compile it / run its own tests, but\nnot have/advertise the \"git init\" etc. integration yet.\n\nI.e. my opinion on GIT_TEST_REFTABLE=1 needing to pass is implicit (but\nI now realize I haven't explicitly said this) on that happening before\nthe tree is in a state where we'd code/doc-wise be in a state of\nshipping this to users.\n\nWhereas just importing the library != that, and I think we'd be in much\nbetter shape if we had it in-tree and would incrementally work on\nintegration from that point, v.s. having more re-rolls of\nmostly-the-same big codebase being re-submitted.\n\nI'm sure there's some things that'll need to change as we start the\ntest/integration work, e.g. the reflog topic that's been discussed. But\nthat's surely better done as some patches on top of the already-landed\nlibrary import at this point v.s. trying to get the library perfect\nbefore getting it in-tree.\n\nMaybe Junio disagrees, just my 0.02....\n"},{"id":"423206","messageId":"CAFQ2z_P8vgY0RRT+XSH9K3VDQt39FLqXx6qfeZqaZPkwhq1w+A@mail.gmail.com","threadId":"54245","inReplyTo":"87h7jqz7k5.fsf@evledraar.gmail.com","subject":"Re: [PATCH v4 13/15] Reftable support for git-core","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-04-28T17:40:29Z","receivedAt":"2021-04-28T17:40:58Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Wed, Apr 28, 2021 at 6:40 PM Ævar Arnfjörð Bjarmason\n wrote:\n> >> I.e. per [1] once if and when we have GIT_TEST_REFTABLE passing surely\n> >> one of the best way to garner feedback on the rest is to discover those\n> >> parts (using \"make gcov\", after running with/without\n> >> GIT_TEST_REFTABLE=[true|false]) where we still have outstanding blind\n> >> spots between the tests and code.\n> >\n> > Getting GIT_TEST_REFTABLE=1 passing is the hard part, because it means\n> > having to understand exactly how the current code is supposed to\n> > work. Once I get to that point (with knowledge being complete and\n> > tests passing), it will be easy to document what is happening and why.\n> >\n> > I was hoping that by posting these series with known test failures,\n> > and questions marked \"XXX\" in reftable-backend.c, I would get feedback\n> > from the other people who know exactly how this part of the code\n> > works. But from your mail, I get the sense that nobody understands\n> > how the whole picture fits together?\n>\n> Almost definitely not. I don't know about you but when I'm looking at\n> code I wrote 6 months ago handling some special case it takes me a while\n> to get up to speed on just knowing what I knew then, and when we're\n> talking about something like refs.c ...\n\n:-(\n\non the bright side, once this is in, the expectations will be much\nmore explicit, because there are two backends that have to work in\nroughly the same way.\n\n> On the topic of the way forward: I for one would very much be for a plan\n> where step 0 is to just a series import the reftable code you have\n> as-is. I.e. we'd include it as an imported external library, maybe have\n> some light test-tool integration and compile it / run its own tests, but\n> not have/advertise the \"git init\" etc. integration yet.\n\nI would really like that too, and I support this way forward. The bulk\nof the work and problems are in the refs/reftable-backend.c and\nassorted incompatibilities across the code base. The library itself\nseems pretty solid at this point.\n\n> I'm sure there's some things that'll need to change as we start the\n> test/integration work, e.g. the reflog topic that's been discussed. But\n> that's surely better done as some patches on top of the already-landed\n> library import at this point v.s. trying to get the library perfect\n> before getting it in-tree.\n>\n> Maybe Junio disagrees, just my 0.02....\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"423597","messageId":"ef664c5c-d959-e265-d678-6fc54d23f031@ahunt.org","threadId":"54245","inReplyTo":"028dd13e701bf82523cc6da1e68a9dab480c1b27.1618832277.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 10/28] reftable: (de)serialization for the polymorphic record type.","fromName":"Andrzej Hunt","fromEmail":"andrzej@ahunt.org","sentAt":"2021-05-04T17:23:22Z","receivedAt":"2021-05-04T17:23:33Z","isPatch":true,"sender":{"key":"andrzej@ahunt.org","avatar":"https://avatars.githubusercontent.com/u/1546915?v=4"},"body":"\n\nOn 19/04/2021 13:37, Han-Wen Nienhuys via GitGitGadget wrote:\n> From: Han-Wen Nienhuys \n[...snip...]\n> diff --git a/reftable/record.c b/reftable/record.c\n> new file mode 100644\n> index 000000000000..b111852b667e\n> --- /dev/null\n> +++ b/reftable/record.c[...snip...> +static void reftable_obj_record_copy_from(void *rec, const \nvoid *src_rec,\n> +\t\t\t\t\t int hash_size)\n> +{\n> +\tstruct reftable_obj_record *obj = rec;\n> +\tconst struct reftable_obj_record *src =\n> +\t\t(const struct reftable_obj_record *)src_rec;\n> +\tint olen;\n> +\n> +\treftable_obj_record_release(obj);\n> +\t*obj = *src;\n> +\tobj->hash_prefix = reftable_malloc(obj->hash_prefix_len);\n> +\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n> +\n> +\tolen = obj->offset_len * sizeof(uint64_t);\n> +\tobj->offsets = reftable_malloc(olen);\n> +\tmemcpy(obj->offsets, src->offsets, olen);\n> +}\n\nIf src->offset_len is 0 then offsets will be NULL - and passing NULL \ninto memcpy() results in undefined behaviour. I think we should either \nadd an \"if (src->offset_len)\" check around the memcpy, or perhaps switch \nto COPY_ARRAY() which performs that check for us. (We can probably also \nskip the malloc and hence also olen calculation in this scenario though, \nbecause obj->offsets should already be NULL if src->offsets was NULL?)\n\nSpecifically, if I run t0032 as such:\n\n make CC=clang-11 UBSAN_OPTIONS=print_stacktrace=1 SANITIZE=undefined \nCOPTS=\"-Og -g\" GIT_TEST_OPTS=-v T=t0032-reftable-unittest.sh test\n\nUBSAN reports the following error:\n\nreftable/record.c:475:23: runtime error: null pointer passed as argument \n2, which is declared to never be null\n/usr/include/string.h:43:28: note: nonnull attribute specified here\n #0 0x698e52 in reftable_obj_record_copy_from record.c:475:2\n #1 0x6ac806 in test_copy record_test.c:21:2\n #2 0x6abb73 in test_reftable_obj_record_roundtrip record_test.c:334:3\n #3 0x6abb73 in record_test_main record_test.c:403:2\n #4 0x437f08 in cmd__reftable test-reftable.c:10:2\n #5 0x427e74 in cmd_main test-tool.c:126:11\n #6 0x428043 in main common-main.c:52:11\n #7 0x7f3cd1044349 in __libc_start_main (/lib64/libc.so.6+0x24349)\n #8 0x407209 in _start start.S:120\n\nSUMMARY: UndefinedBehaviorSanitizer: undefined-behavior \nreftable/record.c:475:23 in\n\n[...snip...]\n> +static int reftable_obj_record_decode(void *rec, struct strbuf key,\n> +\t\t\t\t uint8_t val_type, struct string_view in,\n> +\t\t\t\t int hash_size)\n> +{\n> +\tstruct string_view start = in;\n> +\tstruct reftable_obj_record *r = rec;\n> +\tuint64_t count = val_type;\n> +\tint n = 0;\n> +\tuint64_t last;\n> +\tint j;\n> +\tr->hash_prefix = reftable_malloc(key.len);\n> +\tmemcpy(r->hash_prefix, key.buf, key.len);\n> +\tr->hash_prefix_len = key.len;\n> +\n> +\tif (val_type == 0) {\n> +\t\tn = get_var_int(&count, &in);\n> +\t\tif (n < 0) {\n> +\t\t\treturn n;\n> +\t\t}\n> +\n> +\t\tstring_view_consume(&in, n);\n> +\t}\n> +\n> +\tr->offsets = NULL;\n> +\tr->offset_len = 0;\n> +\tif (count == 0)\n> +\t\treturn start.len - in.len;\n\nAnd it looks like creating a reftable_obj_record with offsets = NULL and \noffset_len = 0 is considered valid based on the code above.\n\n[... snipped the rest ...]\n"},{"id":"423598","messageId":"f5711aa9-e70f-8bb1-09d9-4f283b010d46@ahunt.org","threadId":"54245","inReplyTo":"2fd7cb8c0983501e2af2f195aec81d7c17fb80e1.1618832277.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 23/28] Reftable support for git-core","fromName":"Andrzej Hunt","fromEmail":"andrzej@ahunt.org","sentAt":"2021-05-04T17:24:20Z","receivedAt":"2021-05-04T17:24:30Z","isPatch":true,"sender":{"key":"andrzej@ahunt.org","avatar":"https://avatars.githubusercontent.com/u/1546915?v=4"},"body":"\n\nOn 19/04/2021 13:37, Han-Wen Nienhuys via GitGitGadget wrote:\n> From: Han-Wen Nienhuys \n[...snip...]> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> new file mode 100644\n> index 000000000000..55d053e5ca65\n> --- /dev/null\n> +++ b/refs/reftable-backend.c\n[...snip...]\n> +static int write_transaction_table(struct reftable_writer *writer, void *arg)\n> +{\n> +\tstruct ref_transaction *transaction = (struct ref_transaction *)arg;\n> +\tstruct git_reftable_ref_store *refs =\n> +\t\t(struct git_reftable_ref_store *)transaction->ref_store;\n> +\tstruct reftable_stack *stack =\n> +\t\tstack_for(refs, transaction->updates[0]->refname);\n> +\tuint64_t ts = reftable_stack_next_update_index(stack);\n> +\tint err = 0;\n> +\tint i = 0;\n> +\tstruct reftable_log_record *logs =\n> +\t\tcalloc(transaction->nr, sizeof(*logs));\n> +\tstruct ref_update **sorted =\n> +\t\tmalloc(transaction->nr * sizeof(struct ref_update *));\n> +\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n> +\tstruct reftable_table tab = {NULL};\n> +\tstruct reftable_ref_record ref = {NULL};\n> +\treftable_table_from_merged_table(&tab, mt);\n> +\tCOPY_ARRAY(sorted, transaction->updates, transaction->nr);\n> +\tQSORT(sorted, transaction->nr, ref_update_cmp);\n> +\treftable_writer_set_limits(writer, ts, ts);\n> +\n> +\tfor (i = 0; i < transaction->nr; i++) {\n> +\t\tstruct ref_update *u = sorted[i];\n> +\t\tstruct reftable_log_record *log = &logs[i];\n> +\t\tstruct object_id old_id;\n> +\t\tfill_reftable_log_record(log);\n> +\t\tlog->update_index = ts;\n> +\t\tlog->value_type = REFTABLE_LOG_UPDATE;\n> +\t\tlog->refname = (char *)u->refname;\n> +\t\tlog->update.new_hash = u->new_oid.hash;\n> +\t\tlog->update.message = u->msg;\n> +\n> +\t\terr = reftable_table_read_ref(&tab, u->refname, &ref);\n> +\t\tif (err < 0)\n> +\t\t\tgoto done;\n> +\t\telse if (err > 0) {\n> +\t\t\told_id = null_oid;\n> +\t\t} else {\n> +\t\t\toidread(&old_id, ref.value.val1);\n> +\t\t}\n\nThis seems to assume that 'ref.value_type == REFTABLE_REF_VAL1' - but do \nwe expect to have to handle the other types \n(REFTABLE_REF_VAL2/REFTABLE_REF_SYMREF)? When I run tests in seen \nagainst ASAN I see the following errors in t0031, which suggests we're \nrunning this code against REFTABLE_REF_SYMREF too - but I don't know if \nthat means that this code should be able to handle the other ref types \nor if there's a bug higher up the stack. (AFAIUI REFTABLE_REF_DELETION \nis already handled because reftable_table_read_ref() already returns 1 \nfor deletion, but the other cases seem valid?)\n\nASAN error output:\n\n==25352==ERROR: AddressSanitizer: heap-buffer-overflow on address \n0x603000003353 at pc 0x000000499d17 bp 0x7fff0ea0a210 sp 0x7fff0ea099d8\nREAD of size 32 at 0x603000003353 thread T0\n #0 0x499d16 in __asan_memcpy \n../projects/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:22:3\n #1 0x97ed0a in oidread hash.h:292:2\n #2 0x97ed0a in write_transaction_table refs/reftable-backend.c:548:4\n #3 0xb5537a in reftable_addition_add reftable/stack.c:650:8\n #4 0x97b024 in git_reftable_transaction_finish \nrefs/reftable-backend.c:618:9\n #5 0x95f9bf in ref_transaction_commit refs.c:2218:8\n #6 0x9d3aa6 in update_head_with_reflog sequencer.c:1138:6\n #7 0x52ffbf in cmd_commit builtin/commit.c:1814:6\n #8 0x4ce8fe in run_builtin git.c:461:11\n #9 0x4ccbc8 in handle_builtin git.c:718:3\n #10 0x4cb0cc in run_argv git.c:785:4\n #11 0x4cb0cc in cmd_main git.c:916:19\n #12 0x6beded in main common-main.c:52:11\n #13 0x7f415c762349 in __libc_start_main (/lib64/libc.so.6+0x24349)\n #14 0x420769 in _start ../sysdeps/x86_64/start.S:120\n\n0x603000003353 is located 0 bytes to the right of 19-byte region \n[0x603000003340,0x603000003353)\nallocated by thread T0 here:\n #0 0x4868b4 in strdup \n../projects/compiler-rt/lib/asan/asan_interceptors.cpp:452:3\n #1 0xaa08e8 in xstrdup wrapper.c:29:14\n #2 0xb4b280 in reftable_ref_record_copy_from reftable/record.c:229:23\n #3 0xb46754 in merged_iter_next_entry reftable/merged.c:132:2\n #4 0xb46754 in merged_iter_next reftable/merged.c:141:13\n #5 0xb46754 in merged_iter_next_void reftable/merged.c:157:9\n #6 0xb500ae in iterator_next reftable/generic.c:147:9\n #7 0xb500ae in reftable_iterator_next_ref reftable/generic.c:134:9\n #8 0xb500ae in reftable_table_read_ref reftable/generic.c:46:8\n #9 0x97ec67 in write_transaction_table refs/reftable-backend.c:542:9\n #10 0xb5537a in reftable_addition_add reftable/stack.c:650:8\n #11 0x97b024 in git_reftable_transaction_finish \nrefs/reftable-backend.c:618:9\n #12 0x95f9bf in ref_transaction_commit refs.c:2218:8\n #13 0x9d3aa6 in update_head_with_reflog sequencer.c:1138:6\n #14 0x52ffbf in cmd_commit builtin/commit.c:1814:6\n #15 0x4ce8fe in run_builtin git.c:461:11\n #16 0x4ccbc8 in handle_builtin git.c:718:3\n #17 0x4cb0cc in run_argv git.c:785:4\n #18 0x4cb0cc in cmd_main git.c:916:19\n #19 0x6beded in main common-main.c:52:11\n #20 0x7f415c762349 in __libc_start_main (/lib64/libc.so.6+0x24349)\n\nSUMMARY: AddressSanitizer: heap-buffer-overflow \n../projects/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:22:3 in \n__asan_memcpy\n\n\nProduced using:\n\n make CC=clang-11 SANITIZE=address COPTS=\"-Og -g\" GIT_TEST_OPTS=-v \nT=t0031-reftable.sh test\n\n\n> +\n> +\t\t/* XXX fold together with the old_id check below? */\n> +\n> +\t\tlog->update.old_hash = old_id.hash;\n> +\t\tif (u->flags & REF_LOG_ONLY) {\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tif (u->flags & REF_HAVE_NEW) {\n> +\t\t\tstruct reftable_ref_record ref = { NULL };\n> +\t\t\tstruct object_id peeled;\n> +\n> +\t\t\tint peel_error = peel_object(&u->new_oid, &peeled);\n> +\t\t\tref.refname = (char *)u->refname;\n> +\t\t\tref.update_index = ts;\n> +\n> +\t\t\tif (!peel_error) {\n> +\t\t\t\tref.value_type = REFTABLE_REF_VAL2;\n> +\t\t\t\tref.value.val2.target_value = peeled.hash;\n> +\t\t\t\tref.value.val2.value = u->new_oid.hash;\n> +\t\t\t} else if (!is_null_oid(&u->new_oid)) {\n> +\t\t\t\tref.value_type = REFTABLE_REF_VAL1;\n> +\t\t\t\tref.value.val1 = u->new_oid.hash;\n> +\t\t\t}\n> +\n> +\t\t\terr = reftable_writer_add_ref(writer, &ref);\n> +\t\t\tif (err < 0) {\n> +\t\t\t\tgoto done;\n> +\t\t\t}\n> +\t\t}\n> +\t}\n> +\n> +\tfor (i = 0; i < transaction->nr; i++) {\n> +\t\terr = reftable_writer_add_log(writer, &logs[i]);\n> +\t\tclear_reftable_log_record(&logs[i]);\n> +\t\tif (err < 0) {\n> +\t\t\tgoto done;\n> +\t\t}\n> +\t}\n> +\n> +done:\n> +\tassert(err != REFTABLE_API_ERROR);\n> +\treftable_ref_record_release(&ref);\n> +\tfree(logs);\n> +\tfree(sorted);\n> +\treturn err;\n> +}\n> +\n\n[...snip...]\n"},{"id":"424864","messageId":"CAFQ2z_Ot=tsVi9bruFXJ2KgPVFvHpJEKZTRQQh6oHjF0px93Ow@mail.gmail.com","threadId":"54245","inReplyTo":"ef664c5c-d959-e265-d678-6fc54d23f031@ahunt.org","subject":"Re: [PATCH v7 10/28] reftable: (de)serialization for the polymorphic record type.","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-05-18T13:12:11Z","receivedAt":"2021-05-18T13:12:26Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Tue, May 4, 2021 at 7:23 PM Andrzej Hunt wrote:\n> If src->offset_len is 0 then offsets will be NULL - and passing NULL\n> into memcpy() results in undefined behaviour. I think we should either\n> add an \"if (src->offset_len)\" check around the memcpy, or perhaps switch\n> to COPY_ARRAY() which performs that check for us. (We can probably also\n> skip the malloc and hence also olen calculation in this scenario though,\n> because obj->offsets should already be NULL if src->offsets was NULL?)\n\nThanks for the detailed report. I've fixed this for the next round.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"424865","messageId":"CAFQ2z_Os02GHCcPX0Hcin7+va232Di5wFBbdbOBqEehJvTFbFA@mail.gmail.com","threadId":"54245","inReplyTo":"f5711aa9-e70f-8bb1-09d9-4f283b010d46@ahunt.org","subject":"Re: [PATCH v7 23/28] Reftable support for git-core","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-05-18T13:18:21Z","receivedAt":"2021-05-18T13:18:43Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Tue, May 4, 2021 at 7:24 PM Andrzej Hunt wrote:\n> > + } else {\n> > + oidread(&old_id, ref.value.val1);\n> > + }\n>\n> This seems to assume that 'ref.value_type == REFTABLE_REF_VAL1' - but do\n> we expect to have to handle the other types\n> (REFTABLE_REF_VAL2/REFTABLE_REF_SYMREF)? When I run tests in seen\n> against ASAN I see the following errors in t0031, which suggests we're\n> running this code against REFTABLE_REF_SYMREF too - but I don't know if\n> that means that this code should be able to handle the other ref types\n> or if there's a bug higher up the stack. (AFAIUI REFTABLE_REF_DELETION\n> is already handled because reftable_table_read_ref() already returns 1\n> for deletion, but the other cases seem valid?)\n\nCurious. I didn't know it was supported to create symrefs in a\ntransaction, but I've added an assert to make sure this doesn't\ntrigger any sanitizers.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"424868","messageId":"CAFQ2z_Mz65QsFEsdAEKrRkA8LpOcf+i_Yq-_Rb5sRj4GT=5h5A@mail.gmail.com","threadId":"54245","inReplyTo":"CAFQ2z_Os02GHCcPX0Hcin7+va232Di5wFBbdbOBqEehJvTFbFA@mail.gmail.com","subject":"Re: [PATCH v7 23/28] Reftable support for git-core","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-05-18T13:30:04Z","receivedAt":"2021-05-18T13:30:23Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Tue, May 18, 2021 at 3:18 PM Han-Wen Nienhuys wrote:\n> > This seems to assume that 'ref.value_type == REFTABLE_REF_VAL1' - but do\n> > we expect to have to handle the other types\n> > (REFTABLE_REF_VAL2/REFTABLE_REF_SYMREF)? When I run tests in seen\n> > against ASAN I see the following errors in t0031, which suggests we're\n> > running this code against REFTABLE_REF_SYMREF too - but I don't know if\n> > that means that this code should be able to handle the other ref types\n> > or if there's a bug higher up the stack. (AFAIUI REFTABLE_REF_DELETION\n> > is already handled because reftable_table_read_ref() already returns 1\n> > for deletion, but the other cases seem valid?)\n>\n> Curious. I didn't know it was supported to create symrefs in a\n> transaction, but I've added an assert to make sure this doesn't\n> trigger any sanitizers.\n\nThis happens on repo initialization, where we create HEAD and its\nreferents at the same time.\n\nI've fixed this provisionally for now in my series.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\n\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\n\nRegistergericht und -nummer: Hamburg, HRB 86891\n\nSitz der Gesellschaft: Hamburg\n\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"},{"id":"430895","messageId":"CAFQ2z_NmYdckEQ2EfJyhvAYznhq7PdTaMxu+i-BatSkskvC-uw@mail.gmail.com","threadId":"54245","inReplyTo":"CAFQ2z_NAQ02y1gaTymzi21aHxh+bswvr6ko_Eg=1uuHds4mGyg@mail.gmail.com","subject":"Re: [PATCH v7 04/28] hash.h: provide constants for the hash IDs","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@google.com","sentAt":"2021-07-22T08:31:03Z","receivedAt":"2021-07-22T08:31:17Z","isPatch":true,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"On Wed, Apr 21, 2021 at 11:43 AM Han-Wen Nienhuys wrote:\n>\n> On Wed, Apr 21, 2021 at 3:04 AM brian m. carlson\n> wrote:\n> > > I agree that it makes sense to have symbolic constants defined in\n> > > this file. These are values that fit in the \".format_id\" member of\n> > > \"struct git_hash_algo\", and it is preferrable to make sure that the\n> > > names align (if I were designing in void, I would have called the\n> > > member \"algo_id\" and made the constants GIT_(SHA1|SHA256)_ALGO_ID).\n> > >\n> > > Brian? What's your preference (\"I am fine to store HASH_ID in the\n> > > '.format_id' member\" is perfectly an acceptable answer).\n> >\n> > I slightly prefer FORMAT_ID because it's consistent (and for that reason\n> > only), but if HASH_ID is more convenient, that's fine; I don't have a\n> > strong opinion at all. Definitely don't reroll the series because of my\n> > slight preference. Either way, I think these are fine things to have as\n> > constants, and I appreciate you hoisting the comments here.\n>\n> Either way is fine for me. I can paint the bikeshed in your favorite color :)\n\nThis is now the first commit of the reftable series. I think it could\ngraduate separately.\n\n-- \nHan-Wen Nienhuys - Google Munich\nI work 80%. Don't expect answers from me on Fridays.\n--\nGoogle Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich\nRegistergericht und -nummer: Hamburg, HRB 86891\nSitz der Gesellschaft: Hamburg\nGeschäftsführer: Paul Manicle, Halimah DeLaine Prado\n"}]}