{"thread":{"id":"60815","subject":"[PATCH 0/2] refs: introduce reftable backend","startedAt":"2024-01-30T08:05:33Z","lastAt":"2024-02-15T05:39:03Z","messageCount":23,"participants":["Patrick Steinhardt","Junio C Hamano","Karthik Nayak","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"487592","messageId":"cover.1706601199.git.ps@pks.im","threadId":"60815","inReplyTo":null,"subject":"[PATCH 0/2] refs: introduce reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-01-30T08:05:26Z","receivedAt":"2024-01-30T08:05:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nall prequisites for the reftable backend have now been merged or are\nabout to be merged. This patch series thus introduces the reftable\nbackend itself.\n\nThe patch series is built on top of c5b454771e (The eleventh batch,\n2024-01-29). In addition it depends on ps/tests-with-ref-files-backend\nat bbd6106967 (t: mark tests regarding git-pack-refs(1) to be backend\nspecific, 2024-01-29), which is about to be merged to \"next\".\n\nThe patch series is impacted by three in-flight patch series:\n\n - ps/reftable-compacted-tables-permission-fix, which fixes permissions\n on \"tables.list\" when compacting so that it correctly honors the\n \"core.sharedRepository\" setting.\n\n - jc/reftable-core-fsync, which starts to fsync reftable data to disk.\n\n - A follow-up to that patch series [1], which starts to fsync\n \"tables.list\" to disk during compaction.\n\nThe patch series does _not_ depend on those series. Instead, I have\nadded a couple of tests marked with `test_expect_failure` to t0610.\nThese tests will start to pass once those topics land. As I don't expect\nthis patch series to land on its first iteration I very much assume that\nthe in-flight patch series will land before the reftable backend does.\n\nPatrick\n\n[1]: <7bdafc9bd7f53f38a24d69a563615b6ad484e1ba.1706592127.git.ps@pks.im>\n\nPatrick Steinhardt (2):\n refs: introduce reftable backend\n ci: add jobs to test with the reftable backend\n\n .github/workflows/main.yml | 9 +\n .gitlab-ci.yml | 9 +\n Documentation/ref-storage-format.txt | 2 +\n .../technical/repository-version.txt | 5 +-\n Makefile | 1 +\n ci/lib.sh | 2 +-\n ci/run-build-and-tests.sh | 3 +\n contrib/workdir/git-new-workdir | 2 +-\n path.c | 2 +-\n path.h | 1 +\n refs.c | 1 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 2286 +++++++++++++++++\n repository.h | 5 +-\n t/t0610-reftable-basics.sh | 887 +++++++\n t/t0611-reftable-httpd.sh | 26 +\n t/test-lib.sh | 2 +\n 17 files changed, 3237 insertions(+), 7 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100755 t/t0610-reftable-basics.sh\n create mode 100755 t/t0611-reftable-httpd.sh\n\n-- \n2.43.GIT\n\n"},{"id":"487594","messageId":"5598cd13074beef092a61235a505476d0cbceb90.1706601199.git.ps@pks.im","threadId":"60815","inReplyTo":"cover.1706601199.git.ps@pks.im","subject":"[PATCH 1/2] refs: introduce reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-01-30T08:05:31Z","receivedAt":"2024-01-30T08:05:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Due to scalability issues, Shawn Pearce has originally proposed a new\n\"reftable\" format more than six years ago [1]. Initially, this new\nformat was implemented in JGit with promising results. Around two years\nago, we have then added the \"reftable\" library to the Git codebase via\na4bbd13be3 (Merge branch 'hn/reftable', 2021-12-15). With this we have\nlanded all the low-level code to read and write reftables. Notably\nmissing though was the integration of this low-level code into the Git\ncode base in the form of a new ref backend that ties all of this\ntogether.\n\nThis gap is now finally closed by introducing a new \"reftable\" backend\ninto the Git codebase. This new backend promises to bring some notable\nimprovements to Git repositories:\n\n - It becomes possible to do truly atomic writes where either all refs\n are committed to disk or none are. This was not possible with the\n \"files\" backend because ref updates were split across multiple loose\n files.\n\n - The disk space required to store many refs is reduced, both compared\n to loose refs and packed-refs. This is enabled both by the reftable\n format being a binary format, which is more compact, and by prefix\n compression.\n\n - We can ignore filesystem-specific behaviour as ref names are not\n encoded via paths anymore. This means there is no need to handle\n case sensitivity on Windows systems or Unicode precomposition on\n macOS.\n\n - There is no need to rewrite the complete refdb anymore every time a\n ref is being deleted like it was the case for packed-refs. This\n means that ref deletions are now constant time instead of scaling\n linearly with the number of refs.\n\n - We can ignore file/directory conflicts so that it becomes possible\n to store both \"refs/heads/foo\" and \"refs/heads/foo/bar\".\n\n - Due to this property we can retain reflogs for deleted refs. We have\n previously been deleting reflogs together with their refs to avoid\n file/directory conflicts, which is not necessary anymore.\n\nNot all of these improvements are realized with the current \"reftable\"\nbackend implementation. At this point, the new backend is supposed to be\na drop-in replacement for the \"files\" backend that is used by basically\nall Git repositories nowadays. It strives for 1:1 compatibility, which\nmeans that a user can expect the same behaviour regardless of whether\nthey use the \"reftable\" backend or the \"files\" backend for most of the\npart.\n\nMost notably, this means we artificially limit the capabilities of the\n\"reftable\" backend to match the limits of the \"files\" backend. It is not\npossible to create refs that would end up with file/directory conflicts,\nwe do not retain reflogs, we perform stricter-than-necessary checks.\nThis is done intentionally due to two main reasons:\n\n - It makes it significantly easier to land the \"reftable\" backend as\n tests behave the same. It would be tough to argue for each and every\n single test that doesn't pass with the \"reftable\" backend.\n\n - It ensures compatibility between repositories that use the \"files\"\n backend and repositories that use the \"reftable\" backend. Like this,\n hosters can migrate their repositories to use the \"reftable\" backend\n without causing issues for clients that use the \"files\" backend in\n their clones.\n\nIt is expected that these artificial limitations may eventually go away\nin the long term.\n\nPerformance-wise things very much depend on the actual workload. The\nfollowing benchmarks compare the \"files\" and \"reftable\" backends in the\ncurrent version:\n\n - Creating N refs in separate transactions shows that the \"files\"\n backend is ~50% faster. This is not surprising given that creating a\n ref only requires us to create a single loose ref. The \"reftable\"\n backend will also perform auto compaction on updates. In real-world\n workloads we would likely also want to perform pack loose refs,\n which would likely change the picture.\n\n Benchmark 1: update-ref: create refs sequentially (refformat = files)\n Time (mean ± σ): 2.1 ms ± 0.3 ms [User: 0.6 ms, System: 1.7 ms]\n Range (min … max): 1.8 ms … 4.3 ms 133 runs\n\n Benchmark 2: update-ref: create refs sequentially (refformat = reftable)\n Time (mean ± σ): 2.7 ms ± 0.1 ms [User: 0.6 ms, System: 2.2 ms]\n Range (min … max): 2.4 ms … 2.9 ms 132 runs\n\n Benchmark 3: update-ref: create refs sequentially (refformat = files)\n Time (mean ± σ): 1.975 s ± 0.006 s [User: 0.437 s, System: 1.535 s]\n Range (min … max): 1.969 s … 1.980 s 3 runs\n\n Benchmark 4: update-ref: create refs sequentially (refformat = reftable)\n Time (mean ± σ): 2.611 s ± 0.013 s [User: 0.782 s, System: 1.825 s]\n Range (min … max): 2.597 s … 2.622 s 3 runs\n\n Benchmark 5: update-ref: create refs sequentially (refformat = files)\n Time (mean ± σ): 198.442 s ± 0.241 s [User: 43.051 s, System: 155.250 s]\n Range (min … max): 198.189 s … 198.670 s 3 runs\n\n Benchmark 6: update-ref: create refs sequentially (refformat = reftable)\n Time (mean ± σ): 294.509 s ± 4.269 s [User: 104.046 s, System: 190.326 s]\n Range (min … max): 290.223 s … 298.761 s 3 runs\n\n - Creating N refs in a single transaction shows that the \"files\"\n backend is significantly slower once we start to write many refs.\n The \"reftable\" backend only needs to update two files, whereas the\n \"files\" backend needs to write one file per ref.\n\n Benchmark 1: update-ref: create many refs (refformat = files, refcount = 1)\n Time (mean ± σ): 1.9 ms ± 0.1 ms [User: 0.4 ms, System: 1.4 ms]\n Range (min … max): 1.8 ms … 2.6 ms 151 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, refcount = 1)\n Time (mean ± σ): 2.5 ms ± 0.1 ms [User: 0.7 ms, System: 1.7 ms]\n Range (min … max): 2.4 ms … 3.4 ms 148 runs\n\n Benchmark 3: update-ref: create many refs (refformat = files, refcount = 1000)\n Time (mean ± σ): 152.5 ms ± 5.2 ms [User: 19.1 ms, System: 133.1 ms]\n Range (min … max): 148.5 ms … 167.8 ms 15 runs\n\n Benchmark 4: update-ref: create many refs (refformat = reftable, refcount = 1000)\n Time (mean ± σ): 58.0 ms ± 2.5 ms [User: 28.4 ms, System: 29.4 ms]\n Range (min … max): 56.3 ms … 72.9 ms 40 runs\n\n Benchmark 5: update-ref: create many refs (refformat = files, refcount = 1000000)\n Time (mean ± σ): 152.752 s ± 0.710 s [User: 20.315 s, System: 131.310 s]\n Range (min … max): 152.165 s … 153.542 s 3 runs\n\n Benchmark 6: update-ref: create many refs (refformat = reftable, refcount = 1000000)\n Time (mean ± σ): 51.912 s ± 0.127 s [User: 26.483 s, System: 25.424 s]\n Range (min … max): 51.769 s … 52.012 s 3 runs\n\n - Deleting a ref in a fully-packed repository shows that the \"files\"\n backend scales with the number of refs. The \"reftable\" backend has\n constant-time deletions.\n\n Benchmark 1: update-ref: delete ref (refformat = files, refcount = 1)\n Time (mean ± σ): 1.7 ms ± 0.1 ms [User: 0.4 ms, System: 1.2 ms]\n Range (min … max): 1.6 ms … 2.1 ms 316 runs\n\n Benchmark 2: update-ref: delete ref (refformat = reftable, refcount = 1)\n Time (mean ± σ): 1.8 ms ± 0.1 ms [User: 0.4 ms, System: 1.3 ms]\n Range (min … max): 1.7 ms … 2.1 ms 294 runs\n\n Benchmark 3: update-ref: delete ref (refformat = files, refcount = 1000)\n Time (mean ± σ): 2.0 ms ± 0.1 ms [User: 0.5 ms, System: 1.4 ms]\n Range (min … max): 1.9 ms … 2.5 ms 287 runs\n\n Benchmark 4: update-ref: delete ref (refformat = reftable, refcount = 1000)\n Time (mean ± σ): 1.9 ms ± 0.1 ms [User: 0.5 ms, System: 1.3 ms]\n Range (min … max): 1.8 ms … 2.1 ms 217 runs\n\n Benchmark 5: update-ref: delete ref (refformat = files, refcount = 1000000)\n Time (mean ± σ): 229.8 ms ± 7.9 ms [User: 182.6 ms, System: 46.8 ms]\n Range (min … max): 224.6 ms … 245.2 ms 6 runs\n\n Benchmark 6: update-ref: delete ref (refformat = reftable, refcount = 1000000)\n Time (mean ± σ): 2.0 ms ± 0.0 ms [User: 0.6 ms, System: 1.3 ms]\n Range (min … max): 2.0 ms … 2.1 ms 3 runs\n\n - Listing all refs shows no significant advantage for either of the\n backends. The \"files\" backend is a bit faster, but not by a\n significant margin. When repositories are not packed the \"reftable\"\n backend outperforms the \"files\" backend because the \"reftable\"\n backend performs auto-compaction.\n\n Benchmark 1: show-ref: print all refs (refformat = files, refcount = 1, packed = true)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 2.0 ms 1729 runs\n\n Benchmark 2: show-ref: print all refs (refformat = reftable, refcount = 1, packed = true)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 1.8 ms 1816 runs\n\n Benchmark 3: show-ref: print all refs (refformat = files, refcount = 1000, packed = true)\n Time (mean ± σ): 4.3 ms ± 0.1 ms [User: 0.9 ms, System: 3.3 ms]\n Range (min … max): 4.1 ms … 4.6 ms 645 runs\n\n Benchmark 4: show-ref: print all refs (refformat = reftable, refcount = 1000, packed = true)\n Time (mean ± σ): 4.5 ms ± 0.2 ms [User: 1.0 ms, System: 3.3 ms]\n Range (min … max): 4.2 ms … 5.9 ms 643 runs\n\n Benchmark 5: show-ref: print all refs (refformat = files, refcount = 1000000, packed = true)\n Time (mean ± σ): 2.537 s ± 0.034 s [User: 0.488 s, System: 2.048 s]\n Range (min … max): 2.511 s … 2.627 s 10 runs\n\n Benchmark 6: show-ref: print all refs (refformat = reftable, refcount = 1000000, packed = true)\n Time (mean ± σ): 2.712 s ± 0.017 s [User: 0.653 s, System: 2.059 s]\n Range (min … max): 2.692 s … 2.752 s 10 runs\n\n Benchmark 7: show-ref: print all refs (refformat = files, refcount = 1, packed = false)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 1.9 ms 1834 runs\n\n Benchmark 8: show-ref: print all refs (refformat = reftable, refcount = 1, packed = false)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.4 ms … 2.0 ms 1840 runs\n\n Benchmark 9: show-ref: print all refs (refformat = files, refcount = 1000, packed = false)\n Time (mean ± σ): 13.8 ms ± 0.2 ms [User: 2.8 ms, System: 10.8 ms]\n Range (min … max): 13.3 ms … 14.5 ms 208 runs\n\n Benchmark 10: show-ref: print all refs (refformat = reftable, refcount = 1000, packed = false)\n Time (mean ± σ): 4.5 ms ± 0.2 ms [User: 1.2 ms, System: 3.3 ms]\n Range (min … max): 4.3 ms … 6.2 ms 624 runs\n\n Benchmark 11: show-ref: print all refs (refformat = files, refcount = 1000000, packed = false)\n Time (mean ± σ): 12.127 s ± 0.129 s [User: 2.675 s, System: 9.451 s]\n Range (min … max): 11.965 s … 12.370 s 10 runs\n\n Benchmark 12: show-ref: print all refs (refformat = reftable, refcount = 1000000, packed = false)\n Time (mean ± σ): 2.799 s ± 0.022 s [User: 0.735 s, System: 2.063 s]\n Range (min … max): 2.769 s … 2.836 s 10 runs\n\n - Printing a single ref shows no real difference between the \"files\"\n and \"reftable\" backends.\n\n Benchmark 1: show-ref: print single ref (refformat = files, refcount = 1)\n Time (mean ± σ): 1.5 ms ± 0.1 ms [User: 0.4 ms, System: 1.0 ms]\n Range (min … max): 1.4 ms … 1.8 ms 1779 runs\n\n Benchmark 2: show-ref: print single ref (refformat = reftable, refcount = 1)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.4 ms … 2.5 ms 1753 runs\n\n Benchmark 3: show-ref: print single ref (refformat = files, refcount = 1000)\n Time (mean ± σ): 1.5 ms ± 0.1 ms [User: 0.3 ms, System: 1.1 ms]\n Range (min … max): 1.4 ms … 1.9 ms 1840 runs\n\n Benchmark 4: show-ref: print single ref (refformat = reftable, refcount = 1000)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 2.0 ms 1831 runs\n\n Benchmark 5: show-ref: print single ref (refformat = files, refcount = 1000000)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 2.1 ms 1848 runs\n\n Benchmark 6: show-ref: print single ref (refformat = reftable, refcount = 1000000)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 2.1 ms 1762 runs\n\nSo overall, performance depends on the usecases. Except for many\nsequential writes the \"reftable\" backend is roughly on par or\nsignificantly faster than the \"files\" backend though. Given that the\n\"files\" backend has received 18 years of optimizations by now this can\nbe seen as a win. Furthermore, we can expect that the \"reftable\" backend\nwill grow faster over time when attention turns more towards\noptimizations.\n\nThe complete test suite passes, except for those tests explicitly marked\nto require the REFFILES prerequisite. Some tests in t0610 are marked as\nfailing because they depend on still-in-flight bug fixes. Tests can be\nrun with the new backend by setting the GIT_TEST_DEFAULT_REF_FORMAT\nenvironment variable to \"reftable\".\n\nThere is a single known conceptual incompatibility with the dumb HTTP\ntransport. As \"info/refs\" SHOULD NOT contain the HEAD reference, and\nbecause the \"HEAD\" file is not valid anymore, it is impossible for the\nremote client to figure out the default branch without changing the\nprotocol. This shortcoming needs to be handled in a subsequent patch\nseries.\n\nAs the reftable library has already been introduced a while ago, this\ncommit message will not go into the details of how exactly the on-disk\nformat works. Please refer to our preexisting technical documentation at\nDocumentation/technical/reftable for this.\n\n[1]: https://public-inbox.org/git/CAJo=hJtyof=HRy=2sLP0ng0uZ4=S-DpZ5dR1aF+VHVETKG20OQ@mail.gmail.com/\n\nOriginal-idea-by: Shawn Pearce \nBased-on-patch-by: Han-Wen Nienhuys \nSigned-off-by: Patrick Steinhardt \n---\n Documentation/ref-storage-format.txt | 2 +\n .../technical/repository-version.txt | 5 +-\n Makefile | 1 +\n contrib/workdir/git-new-workdir | 2 +-\n path.c | 2 +-\n path.h | 1 +\n refs.c | 1 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 2286 +++++++++++++++++\n repository.h | 5 +-\n t/t0610-reftable-basics.sh | 887 +++++++\n t/t0611-reftable-httpd.sh | 26 +\n t/test-lib.sh | 2 +\n 13 files changed, 3215 insertions(+), 6 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100755 t/t0610-reftable-basics.sh\n create mode 100755 t/t0611-reftable-httpd.sh\n\ndiff --git a/Documentation/ref-storage-format.txt b/Documentation/ref-storage-format.txt\nindex 1a65cac468..14fff8a9c6 100644\n--- a/Documentation/ref-storage-format.txt\n+++ b/Documentation/ref-storage-format.txt\n@@ -1 +1,3 @@\n * `files` for loose files with packed-refs. This is the default.\n+* `reftable` for the reftable format. This format is experimental and its\n+ internals are subject to change.\ndiff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\nindex 27be3741e6..47281420fc 100644\n--- a/Documentation/technical/repository-version.txt\n+++ b/Documentation/technical/repository-version.txt\n@@ -103,5 +103,6 @@ GIT_COMMON_DIR/worktrees//config.worktree)\n \n ==== `refStorage`\n \n-Specifies the file format for the ref database. The only valid value\n-is `files` (loose references with a packed-refs file).\n+Specifies the file format for the ref database. The valid values are\n+`files` (loose references with a packed-refs file) and `reftable` (see\n+Documentation/technical/reftable.txt).\ndiff --git a/Makefile b/Makefile\nindex 0f748a52e6..b302516173 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1127,6 +1127,7 @@ LIB_OBJS += reflog.o\n LIB_OBJS += refs.o\n LIB_OBJS += refs/debug.o\n LIB_OBJS += refs/files-backend.o\n+LIB_OBJS += refs/reftable-backend.o\n LIB_OBJS += refs/iterator.o\n LIB_OBJS += refs/packed-backend.o\n LIB_OBJS += refs/ref-cache.o\ndiff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir\nindex 888c34a521..989197aace 100755\n--- a/contrib/workdir/git-new-workdir\n+++ b/contrib/workdir/git-new-workdir\n@@ -79,7 +79,7 @@ trap cleanup $siglist\n # create the links to the original repo. explicitly exclude index, HEAD and\n # logs/HEAD from the list since they are purely related to the current working\n # directory, and should not be shared.\n-for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn\n+for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn reftable\n do\n \t# create a containing directory if needed\n \tcase $x in\ndiff --git a/path.c b/path.c\nindex 0fb527918b..8bb223c92c 100644\n--- a/path.c\n+++ b/path.c\n@@ -871,7 +871,7 @@ const char *enter_repo(const char *path, int strict)\n \treturn NULL;\n }\n \n-static int calc_shared_perm(int mode)\n+int calc_shared_perm(int mode)\n {\n \tint tweak;\n \ndiff --git a/path.h b/path.h\nindex b3233c51fa..e053effef2 100644\n--- a/path.h\n+++ b/path.h\n@@ -181,6 +181,7 @@ const char *git_path_shallow(struct repository *r);\n int ends_with_path_components(const char *path, const char *components);\n int validate_headref(const char *ref);\n \n+int calc_shared_perm(int mode);\n int adjust_shared_perm(const char *path);\n \n char *interpolate_path(const char *path, int real_home);\ndiff --git a/refs.c b/refs.c\nindex c633abf284..fff343c256 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -35,6 +35,7 @@\n */\n static const struct ref_storage_be *refs_backends[] = {\n \t[REF_STORAGE_FORMAT_FILES] = &refs_be_files,\n+\t[REF_STORAGE_FORMAT_REFTABLE] = &refs_be_reftable,\n };\n \n static const struct ref_storage_be *find_ref_storage_backend(unsigned int ref_storage_format)\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 82219829b0..83e0f0bba3 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -693,6 +693,7 @@ struct ref_storage_be {\n };\n \n extern struct ref_storage_be refs_be_files;\n+extern struct ref_storage_be refs_be_reftable;\n extern struct ref_storage_be refs_be_packed;\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nnew file mode 100644\nindex 0000000000..895de0b273\n--- /dev/null\n+++ b/refs/reftable-backend.c\n@@ -0,0 +1,2286 @@\n+#include \"../git-compat-util.h\"\n+#include \"../abspath.h\"\n+#include \"../chdir-notify.h\"\n+#include \"../config.h\"\n+#include \"../environment.h\"\n+#include \"../gettext.h\"\n+#include \"../hash.h\"\n+#include \"../hex.h\"\n+#include \"../iterator.h\"\n+#include \"../ident.h\"\n+#include \"../lockfile.h\"\n+#include \"../object.h\"\n+#include \"../path.h\"\n+#include \"../refs.h\"\n+#include \"../reftable/reftable-stack.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-blocksource.h\"\n+#include \"../reftable/reftable-reader.h\"\n+#include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-merged.h\"\n+#include \"../reftable/reftable-generic.h\"\n+#include \"../setup.h\"\n+#include \"../strmap.h\"\n+#include \"../worktree.h\"\n+#include \"refs-internal.h\"\n+\n+/*\n+ * Used as a flag in ref_update::flags when the ref_update was via an\n+ * update to HEAD.\n+ */\n+#define REF_UPDATE_VIA_HEAD (1 << 8)\n+\n+struct reftable_ref_store {\n+\tstruct ref_store base;\n+\n+\tstruct reftable_stack *main_stack;\n+\tstruct reftable_stack *worktree_stack;\n+\tstruct strmap worktree_stacks;\n+\tstruct reftable_write_options write_options;\n+\n+\tunsigned int store_flags;\n+\tint err;\n+};\n+\n+/*\n+ * Downcast ref_store to reftable_ref_store. Die if ref_store is not a\n+ * reftable_ref_store. required_flags is compared with ref_store's store_flags\n+ * to ensure the ref_store has all required capabilities. \"caller\" is used in\n+ * any necessary error messages.\n+ */\n+static struct reftable_ref_store *reftable_be_downcast(struct ref_store *ref_store,\n+\t\t\t\t\t\t unsigned int required_flags,\n+\t\t\t\t\t\t const char *caller)\n+{\n+\tstruct reftable_ref_store *refs;\n+\n+\tif (ref_store->be != &refs_be_reftable)\n+\t\tBUG(\"ref_store is type \\\"%s\\\" not \\\"reftables\\\" in %s\",\n+\t\t ref_store->be->name, caller);\n+\n+\trefs = (struct reftable_ref_store *)ref_store;\n+\n+\tif ((refs->store_flags & required_flags) != required_flags)\n+\t\tBUG(\"operation %s requires abilities 0x%x, but only have 0x%x\",\n+\t\t caller, required_flags, refs->store_flags);\n+\n+\treturn refs;\n+}\n+\n+/*\n+ * Some refs are global to the repository (refs/heads/{*}), while others are\n+ * local to the worktree (eg. HEAD, refs/bisect/{*}). We solve this by having\n+ * multiple separate databases (ie. multiple reftable/ directories), one for\n+ * the shared refs, one for the current worktree refs, and one for each\n+ * additional worktree. For reading, we merge the view of both the shared and\n+ * the current worktree's refs, when necessary.\n+ *\n+ * This function also optionally assigns the rewritten reference name that is\n+ * local to the stack. This translation is required when using worktree refs\n+ * like `worktrees/$worktree/refs/heads/foo` as worktree stacks will store\n+ * those references in their normalized form.\n+ */\n+static struct reftable_stack *stack_for(struct reftable_ref_store *store,\n+\t\t\t\t\tconst char *refname,\n+\t\t\t\t\tconst char **rewritten_ref)\n+{\n+\tconst char *wtname;\n+\tint wtname_len;\n+\n+\tif (!refname)\n+\t\treturn store->main_stack;\n+\n+\tswitch (parse_worktree_ref(refname, &wtname, &wtname_len, rewritten_ref)) {\n+\tcase REF_WORKTREE_OTHER: {\n+\t\tstatic struct strbuf wtname_buf = STRBUF_INIT;\n+\t\tstruct strbuf wt_dir = STRBUF_INIT;\n+\t\tstruct reftable_stack *stack;\n+\n+\t\t/*\n+\t\t * We're using a static buffer here so that we don't need to\n+\t\t * allocate the worktree name whenever we look up a reference.\n+\t\t * This could be avoided if the strmap interface knew how to\n+\t\t * handle keys with a length.\n+\t\t */\n+\t\tstrbuf_reset(&wtname_buf);\n+\t\tstrbuf_add(&wtname_buf, wtname, wtname_len);\n+\n+\t\t/*\n+\t\t * There is an edge case here: when the worktree references the\n+\t\t * current worktree, then we set up the stack once via\n+\t\t * `worktree_stacks` and once via `worktree_stack`. This is\n+\t\t * wasteful, but in the reading case it shouldn't matter. And\n+\t\t * in the writing case we would notice that the stack is locked\n+\t\t * already and error out when trying to write a reference via\n+\t\t * both stacks.\n+\t\t */\n+\t\tstack = strmap_get(&store->worktree_stacks, wtname_buf.buf);\n+\t\tif (!stack) {\n+\t\t\tstrbuf_addf(&wt_dir, \"%s/worktrees/%s/reftable\",\n+\t\t\t\t store->base.repo->commondir, wtname_buf.buf);\n+\n+\t\t\tstore->err = reftable_new_stack(&stack, wt_dir.buf,\n+\t\t\t\t\t\t\tstore->write_options);\n+\t\t\tassert(store->err != REFTABLE_API_ERROR);\n+\t\t\tstrmap_put(&store->worktree_stacks, wtname_buf.buf, stack);\n+\t\t}\n+\n+\t\tstrbuf_release(&wt_dir);\n+\t\treturn stack;\n+\t}\n+\tcase REF_WORKTREE_CURRENT:\n+\t\t/*\n+\t\t * If there is no worktree stack then we're currently in the\n+\t\t * main worktree. We thus return the main stack in that case.\n+\t\t */\n+\t\tif (!store->worktree_stack)\n+\t\t\treturn store->main_stack;\n+\t\treturn store->worktree_stack;\n+\tcase REF_WORKTREE_MAIN:\n+\tcase REF_WORKTREE_SHARED:\n+\t\treturn store->main_stack;\n+\tdefault:\n+\t\tBUG(\"unhandled worktree reference type\");\n+\t}\n+}\n+\n+static int should_write_log(struct ref_store *refs, const char *refname)\n+{\n+\tif (log_all_ref_updates == LOG_REFS_UNSET)\n+\t\tlog_all_ref_updates = is_bare_repository() ? LOG_REFS_NONE : LOG_REFS_NORMAL;\n+\n+\tswitch (log_all_ref_updates) {\n+\tcase LOG_REFS_NONE:\n+\t\treturn refs_reflog_exists(refs, refname);\n+\tcase LOG_REFS_ALWAYS:\n+\t\treturn 1;\n+\tcase LOG_REFS_NORMAL:\n+\t\tif (should_autocreate_reflog(refname))\n+\t\t\treturn 1;\n+\t\treturn refs_reflog_exists(refs, refname);\n+\tdefault:\n+\t\tBUG(\"unhandled core.logAllRefUpdates value %d\", log_all_ref_updates);\n+\t}\n+}\n+\n+static void clear_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\t/* when we write log records, the hashes are owned by a struct\n+\t\t * oid */\n+\t\tlog->value.update.old_hash = NULL;\n+\t\tlog->value.update.new_hash = NULL;\n+\t\tbreak;\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\t}\n+\treftable_log_record_release(log);\n+}\n+\n+static void fill_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tconst char *info = git_committer_info(0);\n+\tstruct ident_split split = {0};\n+\tint sign = 1;\n+\n+\tif (split_ident_line(&split, info, strlen(info)))\n+\t\tBUG(\"failed splitting committer info\");\n+\n+\treftable_log_record_release(log);\n+\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\tlog->value.update.name =\n+\t\txstrndup(split.name_begin, split.name_end - split.name_begin);\n+\tlog->value.update.email =\n+\t\txstrndup(split.mail_begin, split.mail_end - split.mail_begin);\n+\tlog->value.update.time = atol(split.date_begin);\n+\tif (*split.tz_begin == '-') {\n+\t\tsign = -1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\tif (*split.tz_begin == '+') {\n+\t\tsign = 1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\n+\tlog->value.update.tz_offset = sign * atoi(split.tz_begin);\n+}\n+\n+static int read_ref_without_reload(struct reftable_stack *stack,\n+\t\t\t\t const char *refname,\n+\t\t\t\t struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type)\n+{\n+\tstruct reftable_ref_record ref = {0};\n+\tint ret;\n+\n+\tret = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (reftable_ref_record_val1(&ref)) {\n+\t\toidread(oid, reftable_ref_record_val1(&ref));\n+\t} else {\n+\t\t/* We got a tombstone, which should not happen. */\n+\t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n+\t}\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn ret;\n+}\n+\n+static struct ref_store *reftable_be_init(struct repository *repo,\n+\t\t\t\t\t const char *gitdir,\n+\t\t\t\t\t unsigned int store_flags)\n+{\n+\tstruct reftable_ref_store *refs = xcalloc(1, sizeof(*refs));\n+\tstruct strbuf path = STRBUF_INIT;\n+\tint is_worktree;\n+\tmode_t mask;\n+\n+\tmask = umask(0);\n+\tumask(mask);\n+\n+\tbase_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);\n+\tstrmap_init(&refs->worktree_stacks);\n+\trefs->store_flags = store_flags;\n+\trefs->write_options.block_size = 4096;\n+\trefs->write_options.hash_id = repo->hash_algo->format_id;\n+\trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n+\n+\t/*\n+\t * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.\n+\t * This stack contains both the shared and the main worktree refs.\n+\t */\n+\tis_worktree = get_common_dir_noenv(&path, gitdir);\n+\tif (!is_worktree) {\n+\t\tstrbuf_reset(&path);\n+\t\tstrbuf_realpath(&path, gitdir, 0);\n+\t}\n+\tstrbuf_addstr(&path, \"/reftable\");\n+\trefs->err = reftable_new_stack(&refs->main_stack, path.buf,\n+\t\t\t\t refs->write_options);\n+\tif (refs->err)\n+\t\tgoto done;\n+\n+\t/*\n+\t * If we're in a worktree we also need to set up the worktree reftable\n+\t * stack that is contained in the per-worktree GIT_DIR.\n+\t */\n+\tif (is_worktree) {\n+\t\tstrbuf_reset(&path);\n+\t\tstrbuf_addf(&path, \"%s/reftable\", gitdir);\n+\n+\t\trefs->err = reftable_new_stack(&refs->worktree_stack, path.buf,\n+\t\t\t\t\t refs->write_options);\n+\t\tif (refs->err)\n+\t\t\tgoto done;\n+\t}\n+\n+\tchdir_notify_reparent(\"reftables-backend $GIT_DIR\", &refs->base.gitdir);\n+\n+done:\n+\tassert(refs->err != REFTABLE_API_ERROR);\n+\tstrbuf_release(&path);\n+\treturn &refs->base;\n+}\n+\n+static int reftable_be_init_db(struct ref_store *ref_store,\n+\t\t\t int flags UNUSED,\n+\t\t\t struct strbuf *err UNUSED)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"init_db\");\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\tstrbuf_addf(&sb, \"%s/reftable\", refs->base.gitdir);\n+\tsafe_create_dir(sb.buf, 1);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/HEAD\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"ref: refs/heads/.invalid\");\n+\tadjust_shared_perm(sb.buf);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs\", refs->base.gitdir);\n+\tsafe_create_dir(sb.buf, 1);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs/heads\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\tadjust_shared_perm(sb.buf);\n+\n+\tstrbuf_release(&sb);\n+\treturn 0;\n+}\n+\n+struct reftable_ref_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_ref_record ref;\n+\tstruct object_id oid;\n+\n+\tconst char *prefix;\n+\tunsigned int flags;\n+\tint err;\n+};\n+\n+static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct reftable_ref_iterator *iter =\n+\t\t(struct reftable_ref_iterator *)ref_iterator;\n+\tstruct reftable_ref_store *refs = iter->refs;\n+\n+\twhile (!iter->err) {\n+\t\tint flags = 0;\n+\n+\t\titer->err = reftable_iterator_next_ref(&iter->iter, &iter->ref);\n+\t\tif (iter->err)\n+\t\t\tbreak;\n+\n+\t\t/*\n+\t\t * The files backend only lists references contained in\n+\t\t * \"refs/\". We emulate the same behaviour here and thus skip\n+\t\t * all references that don't start with this prefix.\n+\t\t */\n+\t\tif (!starts_with(iter->ref.refname, \"refs/\"))\n+\t\t\tcontinue;\n+\n+\t\tif (iter->prefix &&\n+\t\t strncmp(iter->prefix, iter->ref.refname, strlen(iter->prefix))) {\n+\t\t\titer->err = 1;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (iter->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n+\t\t parse_worktree_ref(iter->ref.refname, NULL, NULL, NULL) !=\n+\t\t\t REF_WORKTREE_CURRENT)\n+\t\t\tcontinue;\n+\n+\t\tswitch (iter->ref.value_type) {\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\toidread(&iter->oid, iter->ref.value.val1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\toidread(&iter->oid, iter->ref.value.val2.value);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF:\n+\t\t\tif (!refs_resolve_ref_unsafe(&iter->refs->base, iter->ref.refname,\n+\t\t\t\t\t\t RESOLVE_REF_READING, &iter->oid, &flags))\n+\t\t\t\toidclr(&iter->oid);\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tBUG(\"unhandled reference value type %d\", iter->ref.value_type);\n+\t\t}\n+\n+\t\tif (is_null_oid(&iter->oid))\n+\t\t\tflags |= REF_ISBROKEN;\n+\n+\t\tif (check_refname_format(iter->ref.refname, REFNAME_ALLOW_ONELEVEL)) {\n+\t\t\tif (!refname_is_safe(iter->ref.refname))\n+\t\t\t\tdie(_(\"refname is dangerous: %s\"), iter->ref.refname);\n+\t\t\toidclr(&iter->oid);\n+\t\t\tflags |= REF_BAD_NAME | REF_ISBROKEN;\n+\t\t}\n+\n+\t\tif (iter->flags & DO_FOR_EACH_OMIT_DANGLING_SYMREFS &&\n+\t\t flags & REF_ISSYMREF &&\n+\t\t flags & REF_ISBROKEN)\n+\t\t\tcontinue;\n+\n+\t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t !ref_resolves_to_object(iter->ref.refname, refs->base.repo,\n+\t\t\t\t\t &iter->oid, flags))\n+\t\t\t\tcontinue;\n+\n+\t\titer->base.refname = iter->ref.refname;\n+\t\titer->base.oid = &iter->oid;\n+\t\titer->base.flags = flags;\n+\n+\t\tbreak;\n+\t}\n+\n+\tif (iter->err > 0) {\n+\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n+\t\t\treturn ITER_ERROR;\n+\t\treturn ITER_DONE;\n+\t}\n+\n+\tif (iter->err < 0) {\n+\t\tref_iterator_abort(ref_iterator);\n+\t\treturn ITER_ERROR;\n+\t}\n+\n+\treturn ITER_OK;\n+}\n+\n+static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tstruct reftable_ref_iterator *iter =\n+\t\t(struct reftable_ref_iterator *)ref_iterator;\n+\n+\tif (iter->ref.value_type == REFTABLE_REF_VAL2) {\n+\t\toidread(peeled, iter->ref.value.val2.target_value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\n+\n+static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct reftable_ref_iterator *iter =\n+\t\t(struct reftable_ref_iterator *)ref_iterator;\n+\treftable_ref_record_release(&iter->ref);\n+\treftable_iterator_destroy(&iter->iter);\n+\tfree(iter);\n+\treturn ITER_DONE;\n+}\n+\n+static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n+\t.advance = reftable_ref_iterator_advance,\n+\t.peel = reftable_ref_iterator_peel,\n+\t.abort = reftable_ref_iterator_abort\n+};\n+\n+static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_store *refs,\n+\t\t\t\t\t\t\t struct reftable_stack *stack,\n+\t\t\t\t\t\t\t const char *prefix,\n+\t\t\t\t\t\t\t int flags)\n+{\n+\tstruct reftable_merged_table *merged_table;\n+\tstruct reftable_ref_iterator *iter;\n+\tint ret;\n+\n+\titer = xcalloc(1, sizeof(*iter));\n+\tbase_ref_iterator_init(&iter->base, &reftable_ref_iterator_vtable, 1);\n+\titer->prefix = prefix;\n+\titer->base.oid = &iter->oid;\n+\titer->flags = flags;\n+\titer->refs = refs;\n+\n+\tret = refs->err;\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tmerged_table = reftable_stack_merged_table(stack);\n+\n+\tret = reftable_merged_table_seek_ref(merged_table, &iter->iter, prefix);\n+\tif (ret)\n+\t\tgoto done;\n+\n+done:\n+\titer->err = ret;\n+\treturn iter;\n+}\n+\n+static enum iterator_selection iterator_select(struct ref_iterator *iter_worktree,\n+\t\t\t\t\t struct ref_iterator *iter_common,\n+\t\t\t\t\t void *cb_data UNUSED)\n+{\n+\tif (iter_worktree && !iter_common) {\n+\t\t/*\n+\t\t * Return the worktree ref if there are no more common refs.\n+\t\t */\n+\t\treturn ITER_SELECT_0;\n+\t} else if (iter_common) {\n+\t\t/*\n+\t\t * In case we have pending worktree and common refs we need to\n+\t\t * yield them based on their lexicographical order. Worktree\n+\t\t * refs that have the same name as common refs shadow the\n+\t\t * latter.\n+\t\t */\n+\t\tif (iter_worktree) {\n+\t\t\tint cmp = strcmp(iter_worktree->refname,\n+\t\t\t\t\t iter_common->refname);\n+\t\t\tif (cmp < 0)\n+\t\t\t\treturn ITER_SELECT_0;\n+\t\t\telse if (!cmp)\n+\t\t\t\treturn ITER_SELECT_0_SKIP_1;\n+\t\t}\n+\n+\t\t /*\n+\t\t * Otherwise, if we either have no worktree refs anymore or if\n+\t\t * the common ref sorts before the next worktree ref, we need\n+\t\t * to figure out whether the next common ref belongs to the\n+\t\t * main worktree. In that case, it should be ignored.\n+\t\t */\n+\t\tif (parse_worktree_ref(iter_common->refname, NULL, NULL,\n+\t\t\t\t NULL) == REF_WORKTREE_SHARED)\n+\t\t\treturn ITER_SELECT_1;\n+\n+\t\treturn ITER_SKIP_1;\n+\t} else {\n+\t\treturn ITER_DONE;\n+\t}\n+}\n+\n+static struct ref_iterator *reftable_be_iterator_begin(struct ref_store *ref_store,\n+\t\t\t\t\t\t const char *prefix,\n+\t\t\t\t\t\t const char **exclude_patterns,\n+\t\t\t\t\t\t unsigned int flags)\n+{\n+\tstruct reftable_ref_iterator *main_iter, *worktree_iter;\n+\tstruct reftable_ref_store *refs;\n+\tunsigned int required_flags = REF_STORE_READ;\n+\n+\tif (!(flags & DO_FOR_EACH_INCLUDE_BROKEN))\n+\t\trequired_flags |= REF_STORE_ODB;\n+\trefs = reftable_be_downcast(ref_store, required_flags, \"ref_iterator_begin\");\n+\n+\tmain_iter = ref_iterator_for_stack(refs, refs->main_stack, prefix, flags);\n+\n+\t/*\n+\t * The worktree stack is only set when we're in an actual worktree\n+\t * right now. If we aren't, then we return the common reftable\n+\t * iterator, only.\n+\t */\n+\t if (!refs->worktree_stack)\n+\t\treturn &main_iter->base;\n+\n+\t/*\n+\t * Otherwise we merge both the common and the per-worktree refs into a\n+\t * single iterator.\n+\t */\n+\tworktree_iter = ref_iterator_for_stack(refs, refs->worktree_stack, prefix, flags);\n+\treturn merge_ref_iterator_begin(1, &worktree_iter->base, &main_iter->base,\n+\t\t\t\t\titerator_select, NULL);\n+}\n+\n+static int reftable_be_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname,\n+\t\t\t\t struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type,\n+\t\t\t\t int *failure_errno)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_raw_ref\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tint ret;\n+\n+\tif (refs->err < 0)\n+\t\treturn refs->err;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tret = read_ref_without_reload(stack, refname, oid, referent, type);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\tif (ret > 0) {\n+\t\t*failure_errno = ENOENT;\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int reftable_be_read_symbolic_ref(struct ref_store *ref_store,\n+\t\t\t\t\t const char *refname,\n+\t\t\t\t\t struct strbuf *referent)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_symbolic_ref\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_ref_record ref = {0};\n+\tint ret;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tret = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (ret == 0 && ref.value_type == REFTABLE_REF_SYMREF)\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\telse\n+\t\tret = -1;\n+\n+\treftable_ref_record_release(&ref);\n+\treturn ret;\n+}\n+\n+/*\n+ * Return the refname under which update was originally requested.\n+ */\n+static const char *original_update_refname(struct ref_update *update)\n+{\n+\twhile (update->parent_update)\n+\t\tupdate = update->parent_update;\n+\treturn update->refname;\n+}\n+\n+struct reftable_transaction_update {\n+\tstruct ref_update *update;\n+\tstruct object_id current_oid;\n+};\n+\n+struct write_transaction_table_arg {\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_addition *addition;\n+\tstruct reftable_transaction_update *updates;\n+\tsize_t updates_nr;\n+\tsize_t updates_alloc;\n+\tsize_t updates_expected;\n+};\n+\n+struct reftable_transaction_data {\n+\tstruct write_transaction_table_arg *args;\n+\tsize_t args_nr, args_alloc;\n+};\n+\n+static void free_transaction_data(struct reftable_transaction_data *tx_data)\n+{\n+\tif (!tx_data)\n+\t\treturn;\n+\tfor (size_t i = 0; i < tx_data->args_nr; i++) {\n+\t\treftable_addition_destroy(tx_data->args[i].addition);\n+\t\tfree(tx_data->args[i].updates);\n+\t}\n+\tfree(tx_data->args);\n+\tfree(tx_data);\n+}\n+\n+/*\n+ * Prepare transaction update for the given reference update. This will cause\n+ * us to lock the corresponding reftable stack for concurrent modification.\n+ */\n+static int prepare_transaction_update(struct write_transaction_table_arg **out,\n+\t\t\t\t struct reftable_ref_store *refs,\n+\t\t\t\t struct reftable_transaction_data *tx_data,\n+\t\t\t\t struct ref_update *update,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_stack *stack = stack_for(refs, update->refname, NULL);\n+\tstruct write_transaction_table_arg *arg = NULL;\n+\tsize_t i;\n+\tint ret;\n+\n+\t/*\n+\t * Search for a preexisting stack update. If there is one then we add\n+\t * the update to it, otherwise we set up a new stack update.\n+\t */\n+\tfor (i = 0; !arg && i < tx_data->args_nr; i++)\n+\t\tif (tx_data->args[i].stack == stack)\n+\t\t\targ = &tx_data->args[i];\n+\n+\tif (!arg) {\n+\t\tstruct reftable_addition *addition;\n+\n+\t\tret = reftable_stack_reload(stack);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\n+\t\tret = reftable_stack_new_addition(&addition, stack);\n+\t\tif (ret) {\n+\t\t\tif (ret == REFTABLE_LOCK_ERROR)\n+\t\t\t\tstrbuf_addstr(err, \"cannot lock references\");\n+\t\t\treturn ret;\n+\t\t}\n+\n+\t\tALLOC_GROW(tx_data->args, tx_data->args_nr + 1,\n+\t\t\t tx_data->args_alloc);\n+\t\targ = &tx_data->args[tx_data->args_nr++];\n+\t\targ->refs = refs;\n+\t\targ->stack = stack;\n+\t\targ->addition = addition;\n+\t\targ->updates = NULL;\n+\t\targ->updates_nr = 0;\n+\t\targ->updates_alloc = 0;\n+\t\targ->updates_expected = 0;\n+\t}\n+\n+\targ->updates_expected++;\n+\n+\tif (out)\n+\t\t*out = arg;\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Queue a reference update for the correct stack. We potentially need to\n+ * handle multiple stack updates in a single transaction when it spans across\n+ * multiple worktrees.\n+ */\n+static int queue_transaction_update(struct reftable_ref_store *refs,\n+\t\t\t\t struct reftable_transaction_data *tx_data,\n+\t\t\t\t struct ref_update *update,\n+\t\t\t\t struct object_id *current_oid,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct write_transaction_table_arg *arg = NULL;\n+\tint ret;\n+\n+\tif (update->backend_data)\n+\t\tBUG(\"reference update queued more than once\");\n+\n+\tret = prepare_transaction_update(&arg, refs, tx_data, update, err);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\tALLOC_GROW(arg->updates, arg->updates_nr + 1,\n+\t\t arg->updates_alloc);\n+\targ->updates[arg->updates_nr].update = update;\n+\toidcpy(&arg->updates[arg->updates_nr].current_oid, current_oid);\n+\tupdate->backend_data = &arg->updates[arg->updates_nr++];\n+\n+\treturn 0;\n+}\n+\n+static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE|REF_STORE_MAIN, \"ref_transaction_prepare\");\n+\tstruct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT;\n+\tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct reftable_transaction_data *tx_data = NULL;\n+\tstruct object_id head_oid;\n+\tunsigned int head_type = 0;\n+\tsize_t i;\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\ttx_data = xcalloc(1, sizeof(*tx_data));\n+\n+\t/*\n+\t * Preprocess all updates. For one we check that there are no duplicate\n+\t * reference updates in this transaction. Second, we lock all stacks\n+\t * that will be modified during the transaction.\n+\t */\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tret = prepare_transaction_update(NULL, refs, tx_data,\n+\t\t\t\t\t\t transaction->updates[i], err);\n+\t\tif (ret)\n+\t\t\tgoto done;\n+\n+\t\tstring_list_append(&affected_refnames,\n+\t\t\t\t transaction->updates[i]->refname);\n+\t}\n+\n+\t/*\n+\t * Now that we have counted updates per stack we can preallocate their\n+\t * arrays. This avoids having to reallocate many times.\n+\t */\n+\tfor (i = 0; i < tx_data->args_nr; i++) {\n+\t\tCALLOC_ARRAY(tx_data->args[i].updates, tx_data->args[i].updates_expected);\n+\t\ttx_data->args[i].updates_alloc = tx_data->args[i].updates_expected;\n+\t}\n+\n+\t/*\n+\t * Fail if a refname appears more than once in the transaction.\n+\t * This code is taken from the files backend and is a good candidate to\n+\t * be moved into the generic layer.\n+\t */\n+\tstring_list_sort(&affected_refnames);\n+\tif (ref_update_reject_duplicates(&affected_refnames, err)) {\n+\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tret = read_ref_without_reload(stack_for(refs, \"HEAD\", NULL), \"HEAD\", &head_oid,\n+\t\t\t\t &head_referent, &head_type);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = transaction->updates[i];\n+\t\tstruct object_id current_oid = {0};\n+\t\tstruct reftable_stack *stack;\n+\t\tconst char *rewritten_ref;\n+\n+\t\tstack = stack_for(refs, u->refname, &rewritten_ref);\n+\n+\t\t/* Verify that the new object ID is valid. */\n+\t\tif ((u->flags & REF_HAVE_NEW) && !is_null_oid(&u->new_oid) &&\n+\t\t !(u->flags & REF_SKIP_OID_VERIFICATION) &&\n+\t\t !(u->flags & REF_LOG_ONLY)) {\n+\t\t\tstruct object *o = parse_object(refs->base.repo, &u->new_oid);\n+\t\t\tif (!o) {\n+\t\t\t\tstrbuf_addf(err,\n+\t\t\t\t\t _(\"trying to write ref '%s' with nonexistent object %s\"),\n+\t\t\t\t\t u->refname, oid_to_hex(&u->new_oid));\n+\t\t\t\tret = -1;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\n+\t\t\tif (o->type != OBJ_COMMIT && is_branch(u->refname)) {\n+\t\t\t\tstrbuf_addf(err, _(\"trying to write non-commit object %s to branch '%s'\"),\n+\t\t\t\t\t oid_to_hex(&u->new_oid), u->refname);\n+\t\t\t\tret = -1;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\n+\t\t/*\n+\t\t * When we update the reference that HEAD points to we enqueue\n+\t\t * a second log-only update for HEAD so that its reflog is\n+\t\t * updated accordingly.\n+\t\t */\n+\t\tif (head_type == REF_ISSYMREF &&\n+\t\t !(u->flags & REF_LOG_ONLY) &&\n+\t\t !(u->flags & REF_UPDATE_VIA_HEAD) &&\n+\t\t !strcmp(rewritten_ref, head_referent.buf)) {\n+\t\t\tstruct ref_update *new_update;\n+\n+\t\t\t/*\n+\t\t\t * First make sure that HEAD is not already in the\n+\t\t\t * transaction. This check is O(lg N) in the transaction\n+\t\t\t * size, but it happens at most once per transaction.\n+\t\t\t */\n+\t\t\tif (string_list_has_string(&affected_refnames, \"HEAD\")) {\n+\t\t\t\t/* An entry already existed */\n+\t\t\t\tstrbuf_addf(err,\n+\t\t\t\t\t _(\"multiple updates for 'HEAD' (including one \"\n+\t\t\t\t\t \"via its referent '%s') are not allowed\"),\n+\t\t\t\t\t u->refname);\n+\t\t\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\n+\t\t\tnew_update = ref_transaction_add_update(\n+\t\t\t\t\ttransaction, \"HEAD\",\n+\t\t\t\t\tu->flags | REF_LOG_ONLY | REF_NO_DEREF,\n+\t\t\t\t\t&u->new_oid, &u->old_oid, u->msg);\n+\t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n+\t\t}\n+\n+\t\tret = read_ref_without_reload(stack, rewritten_ref,\n+\t\t\t\t\t ¤t_oid, &referent, &u->type);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tif (ret > 0 && (!(u->flags & REF_HAVE_OLD) || is_null_oid(&u->old_oid))) {\n+\t\t\t/*\n+\t\t\t * The reference does not exist, and we either have no\n+\t\t\t * old object ID or expect the reference to not exist.\n+\t\t\t * We can thus skip below safety checks as well as the\n+\t\t\t * symref splitting. But we do want to verify that\n+\t\t\t * there is no conflicting reference here so that we\n+\t\t\t * can output a proper error message instead of failing\n+\t\t\t * at a later point.\n+\t\t\t */\n+\t\t\tret = refs_verify_refname_available(ref_store, u->refname,\n+\t\t\t\t\t\t\t &affected_refnames, NULL, err);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto done;\n+\n+\t\t\t/*\n+\t\t\t * There is no need to write the reference deletion\n+\t\t\t * when the reference in question doesn't exist.\n+\t\t\t */\n+\t\t\t if (u->flags & REF_HAVE_NEW && !is_null_oid(&u->new_oid)) {\n+\t\t\t\t ret = queue_transaction_update(refs, tx_data, u,\n+\t\t\t\t\t\t\t\t¤t_oid, err);\n+\t\t\t\t if (ret)\n+\t\t\t\t\t goto done;\n+\t\t\t }\n+\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (ret > 0) {\n+\t\t\t/* The reference does not exist, but we expected it to. */\n+\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t \"unable to resolve reference '%s'\"),\n+\t\t\t\t original_update_refname(u), u->refname);\n+\t\t\tret = -1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (u->type & REF_ISSYMREF) {\n+\t\t\tconst char *resolved = refs_resolve_ref_unsafe(&refs->base, u->refname, 0,\n+\t\t\t\t\t\t\t\t ¤t_oid, NULL);\n+\n+\t\t\tif (u->flags & REF_NO_DEREF) {\n+\t\t\t\t/*\n+\t\t\t\t * The reftable stack is locked at this point\n+\t\t\t\t * already, so it should be safe to call\n+\t\t\t\t * `refs_resolve_ref_unsafe()` here.\n+\t\t\t\t */\n+\t\t\t\tif (u->flags & REF_HAVE_OLD && !resolved) {\n+\t\t\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t\t\t \"error reading reference\"), u->refname);\n+\t\t\t\t\tret = -1;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n+\t\t\t} else {\n+\t\t\t\tstruct ref_update *new_update;\n+\t\t\t\tint new_flags;\n+\n+\t\t\t\tnew_flags = u->flags;\n+\t\t\t\tif (!strcmp(rewritten_ref, \"HEAD\"))\n+\t\t\t\t\tnew_flags |= REF_UPDATE_VIA_HEAD;\n+\n+\t\t\t\t/*\n+\t\t\t\t * If we are updating a symref (eg. HEAD), we should also\n+\t\t\t\t * update the branch that the symref points to.\n+\t\t\t\t *\n+\t\t\t\t * This is generic functionality, and would be better\n+\t\t\t\t * done in refs.c, but the current implementation is\n+\t\t\t\t * intertwined with the locking in files-backend.c.\n+\t\t\t\t */\n+\t\t\t\tnew_update = ref_transaction_add_update(\n+\t\t\t\t\t\ttransaction, referent.buf, new_flags,\n+\t\t\t\t\t\t&u->new_oid, &u->old_oid, u->msg);\n+\t\t\t\tnew_update->parent_update = u;\n+\n+\t\t\t\t/*\n+\t\t\t\t * Change the symbolic ref update to log only. Also, it\n+\t\t\t\t * doesn't need to check its old OID value, as that will be\n+\t\t\t\t * done when new_update is processed.\n+\t\t\t\t */\n+\t\t\t\tu->flags |= REF_LOG_ONLY | REF_NO_DEREF;\n+\t\t\t\tu->flags &= ~REF_HAVE_OLD;\n+\n+\t\t\t\tif (string_list_has_string(&affected_refnames, new_update->refname)) {\n+\t\t\t\t\tstrbuf_addf(err,\n+\t\t\t\t\t\t _(\"multiple updates for '%s' (including one \"\n+\t\t\t\t\t\t \"via symref '%s') are not allowed\"),\n+\t\t\t\t\t\t referent.buf, u->refname);\n+\t\t\t\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n+\t\t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n+\t\t\t}\n+\t\t}\n+\n+\t\t/*\n+\t\t * Verify that the old object matches our expectations. Note\n+\t\t * that the error messages here do not make a lot of sense in\n+\t\t * the context of the reftable backend as we never lock\n+\t\t * individual refs. But the error messages match what the files\n+\t\t * backend returns, which keeps our tests happy.\n+\t\t */\n+\t\tif (u->flags & REF_HAVE_OLD && !oideq(¤t_oid, &u->old_oid)) {\n+\t\t\tif (is_null_oid(&u->old_oid))\n+\t\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t\t \"reference already exists\"),\n+\t\t\t\t\t original_update_refname(u));\n+\t\t\telse if (is_null_oid(¤t_oid))\n+\t\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t\t \"reference is missing but expected %s\"),\n+\t\t\t\t\t original_update_refname(u),\n+\t\t\t\t\t oid_to_hex(&u->old_oid));\n+\t\t\telse\n+\t\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t\t \"is at %s but expected %s\"),\n+\t\t\t\t\t original_update_refname(u),\n+\t\t\t\t\t oid_to_hex(¤t_oid),\n+\t\t\t\t\t oid_to_hex(&u->old_oid));\n+\t\t\tret = -1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\t/*\n+\t\t * If all of the following conditions are true:\n+\t\t *\n+\t\t * - We're not about to write a symref.\n+\t\t * - We're not about to write a log-only entry.\n+\t\t * - Old and new object ID are different.\n+\t\t *\n+\t\t * Then we're essentially doing a no-op update that can be\n+\t\t * skipped. This is not only for the sake of efficiency, but\n+\t\t * also skips writing unneeded reflog entries.\n+\t\t */\n+\t\tif ((u->type & REF_ISSYMREF) ||\n+\t\t (u->flags & REF_LOG_ONLY) ||\n+\t\t (u->flags & REF_HAVE_NEW && !oideq(¤t_oid, &u->new_oid))) {\n+\t\t\tret = queue_transaction_update(refs, tx_data, u,\n+\t\t\t\t\t\t ¤t_oid, err);\n+\t\t\tif (ret)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\ttransaction->backend_data = tx_data;\n+\ttransaction->state = REF_TRANSACTION_PREPARED;\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\tif (ret < 0) {\n+\t\tfree_transaction_data(tx_data);\n+\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\t\tif (!err->len)\n+\t\t\tstrbuf_addf(err, _(\"reftable: transaction prepare: %s\"),\n+\t\t\t\t reftable_error_str(ret));\n+\t}\n+\tstring_list_clear(&affected_refnames, 0);\n+\tstrbuf_release(&referent);\n+\tstrbuf_release(&head_referent);\n+\n+\treturn ret;\n+}\n+\n+static int reftable_be_transaction_abort(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_transaction_data *tx_data = transaction->backend_data;\n+\tfree_transaction_data(tx_data);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\treturn 0;\n+}\n+\n+static int transaction_update_cmp(const void *a, const void *b)\n+{\n+\treturn strcmp(((struct reftable_transaction_update *)a)->update->refname,\n+\t\t ((struct reftable_transaction_update *)b)->update->refname);\n+}\n+\n+static int write_transaction_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct write_transaction_table_arg *arg = cb_data;\n+\tstruct reftable_merged_table *mt =\n+\t\treftable_stack_merged_table(arg->stack);\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_log_record *logs = NULL;\n+\tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tint ret = 0;\n+\n+\tQSORT(arg->updates, arg->updates_nr, transaction_update_cmp);\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tfor (i = 0; i < arg->updates_nr; i++) {\n+\t\tstruct reftable_transaction_update *tx_update = &arg->updates[i];\n+\t\tstruct ref_update *u = tx_update->update;\n+\n+\t\t/*\n+\t\t * Write a reflog entry when updating a ref to point to\n+\t\t * something new in either of the following cases:\n+\t\t *\n+\t\t * - The reference is about to be deleted. We always want to\n+\t\t * delete the reflog in that case.\n+\t\t * - REF_FORCE_CREATE_REFLOG is set, asking us to always create\n+\t\t * the reflog entry.\n+\t\t * - `core.logAllRefUpdates` tells us to create the reflog for\n+\t\t * the given ref.\n+\t\t */\n+\t\tif (u->flags & REF_HAVE_NEW && !(u->type & REF_ISSYMREF) && is_null_oid(&u->new_oid)) {\n+\t\t\tstruct reftable_log_record log = {0};\n+\t\t\tstruct reftable_iterator it = {0};\n+\n+\t\t\t/*\n+\t\t\t * When deleting refs we also delete all reflog entries\n+\t\t\t * with them. While it is not strictly required to\n+\t\t\t * delete reflogs together with their refs, this\n+\t\t\t * matches the behaviour of the files backend.\n+\t\t\t *\n+\t\t\t * Unfortunately, we have no better way than to delete\n+\t\t\t * all reflog entries one by one.\n+\t\t\t */\n+\t\t\tret = reftable_merged_table_seek_log(mt, &it, u->refname);\n+\t\t\twhile (ret == 0) {\n+\t\t\t\tstruct reftable_log_record *tombstone;\n+\n+\t\t\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\t\t\tif (ret < 0)\n+\t\t\t\t\tbreak;\n+\t\t\t\tif (ret > 0 || strcmp(log.refname, u->refname)) {\n+\t\t\t\t\tret = 0;\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\n+\t\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\t\t\ttombstone = &logs[logs_nr++];\n+\t\t\t\ttombstone->refname = xstrdup(u->refname);\n+\t\t\t\ttombstone->value_type = REFTABLE_LOG_DELETION,\n+\t\t\t\ttombstone->update_index = log.update_index;\n+\t\t\t}\n+\n+\t\t\treftable_log_record_release(&log);\n+\t\t\treftable_iterator_destroy(&it);\n+\n+\t\t\tif (ret)\n+\t\t\t\tgoto done;\n+\t\t} else if (u->flags & REF_HAVE_NEW &&\n+\t\t\t (u->flags & REF_FORCE_CREATE_REFLOG ||\n+\t\t\t should_write_log(&arg->refs->base, u->refname))) {\n+\t\t\tstruct reftable_log_record *log;\n+\n+\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\t\tlog = &logs[logs_nr++];\n+\t\t\tmemset(log, 0, sizeof(*log));\n+\n+\t\t\tfill_reftable_log_record(log);\n+\t\t\tlog->update_index = ts;\n+\t\t\tlog->refname = xstrdup(u->refname);\n+\t\t\tlog->value.update.new_hash = u->new_oid.hash;\n+\t\t\tlog->value.update.old_hash = tx_update->current_oid.hash;\n+\t\t\tlog->value.update.message =\n+\t\t\t\txstrndup(u->msg, arg->refs->write_options.block_size / 2);\n+\t\t}\n+\n+\t\tif (u->flags & REF_LOG_ONLY)\n+\t\t\tcontinue;\n+\n+\t\tif (u->flags & REF_HAVE_NEW && is_null_oid(&u->new_oid)) {\n+\t\t\tstruct reftable_ref_record ref = {\n+\t\t\t\t.refname = (char *)u->refname,\n+\t\t\t\t.update_index = ts,\n+\t\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t};\n+\n+\t\t\tret = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto done;\n+\t\t} else if (u->flags & REF_HAVE_NEW) {\n+\t\t\tstruct reftable_ref_record ref = {0};\n+\t\t\tstruct object_id peeled;\n+\t\t\tint peel_error;\n+\n+\t\t\tref.refname = (char *)u->refname;\n+\t\t\tref.update_index = ts;\n+\n+\t\t\tpeel_error = peel_object(&u->new_oid, &peeled);\n+\t\t\tif (!peel_error) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\t\t\tmemcpy(ref.value.val2.target_value, peeled.hash, GIT_MAX_RAWSZ);\n+\t\t\t\tmemcpy(ref.value.val2.value, u->new_oid.hash, GIT_MAX_RAWSZ);\n+\t\t\t} else if (!is_null_oid(&u->new_oid)) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\t\t\tmemcpy(ref.value.val1, u->new_oid.hash, GIT_MAX_RAWSZ);\n+\t\t\t}\n+\n+\t\t\tret = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Logs are written at the end so that we do not have intermixed ref\n+\t * and log blocks.\n+\t */\n+\tif (logs) {\n+\t\tret = reftable_writer_add_logs(writer, logs, logs_nr);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\tfor (i = 0; i < logs_nr; i++)\n+\t\tclear_reftable_log_record(&logs[i]);\n+\tfree(logs);\n+\treturn ret;\n+}\n+\n+static int reftable_be_transaction_finish(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_transaction_data *tx_data = transaction->backend_data;\n+\tint ret = 0;\n+\n+\tfor (size_t i = 0; i < tx_data->args_nr; i++) {\n+\t\tret = reftable_addition_add(tx_data->args[i].addition,\n+\t\t\t\t\t write_transaction_table, &tx_data->args[i]);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\n+\t\tret = reftable_addition_commit(tx_data->args[i].addition);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\tfree_transaction_data(tx_data);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\n+\tif (ret) {\n+\t\tstrbuf_addf(err, _(\"reftable: transaction failure: %s\"),\n+\t\t\t reftable_error_str(ret));\n+\t\treturn -1;\n+\t}\n+\treturn ret;\n+}\n+\n+static int reftable_be_initial_transaction_commit(struct ref_store *ref_store UNUSED,\n+\t\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t\t struct strbuf *err)\n+{\n+\treturn ref_transaction_commit(transaction, err);\n+}\n+\n+static int reftable_be_pack_refs(struct ref_store *ref_store,\n+\t\t\t\t struct pack_refs_opts *opts)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE | REF_STORE_ODB, \"pack_refs\");\n+\tstruct reftable_stack *stack;\n+\tint ret;\n+\n+\tif (refs->err)\n+\t\treturn refs->err;\n+\n+\tstack = refs->worktree_stack;\n+\tif (!stack)\n+\t\tstack = refs->main_stack;\n+\n+\tret = reftable_stack_compact_all(stack, NULL);\n+\tif (ret)\n+\t\tgoto out;\n+\tret = reftable_stack_clean(stack);\n+\tif (ret)\n+\t\tgoto out;\n+\n+out:\n+\treturn ret;\n+}\n+\n+struct write_create_symref_arg {\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+\tconst char *target;\n+\tconst char *logmsg;\n+};\n+\n+static int write_create_symref_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct write_create_symref_arg *create = cb_data;\n+\tuint64_t ts = reftable_stack_next_update_index(create->stack);\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)create->refname,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = (char *)create->target,\n+\t\t.update_index = ts,\n+\t};\n+\tstruct reftable_log_record log = {0};\n+\tstruct object_id new_oid;\n+\tstruct object_id old_oid;\n+\tint ret;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tret = reftable_writer_add_ref(writer, &ref);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\t/*\n+\t * Note that it is important to try and resolve the reference before we\n+\t * write the log entry. This is because `should_write_log()` will munge\n+\t * `core.logAllRefUpdates`, which is undesirable when we create a new\n+\t * repository because it would be written into the config. As HEAD will\n+\t * not resolve for new repositories this ordering will ensure that this\n+\t * never happens.\n+\t */\n+\tif (!create->logmsg ||\n+\t !refs_resolve_ref_unsafe(&create->refs->base, create->target,\n+\t\t\t\t RESOLVE_REF_READING, &new_oid, NULL) ||\n+\t !should_write_log(&create->refs->base, create->refname))\n+\t\treturn 0;\n+\n+\tfill_reftable_log_record(&log);\n+\tlog.refname = xstrdup(create->refname);\n+\tlog.update_index = ts;\n+\tlog.value.update.message = xstrndup(create->logmsg,\n+\t\t\t\t\t create->refs->write_options.block_size / 2);\n+\tlog.value.update.new_hash = new_oid.hash;\n+\tif (refs_resolve_ref_unsafe(&create->refs->base, create->refname,\n+\t\t\t\t RESOLVE_REF_READING, &old_oid, NULL))\n+\t\tlog.value.update.old_hash = old_oid.hash;\n+\n+\tret = reftable_writer_add_log(writer, &log);\n+\tclear_reftable_log_record(&log);\n+\treturn ret;\n+}\n+\n+static int reftable_be_create_symref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const char *target,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"create_symref\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct write_create_symref_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t\t.target = target,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_add(stack, &write_create_symref_table, &arg);\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\tif (ret)\n+\t\terror(\"unable to write symref for %s: %s\", refname,\n+\t\t reftable_error_str(ret));\n+\treturn ret;\n+}\n+\n+struct write_copy_arg {\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *oldname;\n+\tconst char *newname;\n+\tconst char *logmsg;\n+\tint delete_old;\n+};\n+\n+static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct write_copy_arg *arg = cb_data;\n+\tuint64_t deletion_ts, creation_ts;\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(arg->stack);\n+\tstruct reftable_ref_record old_ref = {0}, refs[2] = {0};\n+\tstruct reftable_log_record old_log = {0}, *logs = NULL;\n+\tstruct reftable_iterator it = {0};\n+\tstruct string_list skip = STRING_LIST_INIT_NODUP;\n+\tstruct strbuf errbuf = STRBUF_INIT;\n+\tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tint ret;\n+\n+\tif (reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref)) {\n+\t\tret = error(_(\"refname %s not found\"), arg->oldname);\n+\t\tgoto done;\n+\t}\n+\tif (old_ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tret = error(_(\"refname %s is a symbolic ref, copying it is not supported\"),\n+\t\t\t arg->oldname);\n+\t\tgoto done;\n+\t}\n+\n+\t/*\n+\t * There's nothing to do in case the old and new name are the same, so\n+\t * we exit early in that case.\n+\t */\n+\tif (!strcmp(arg->oldname, arg->newname)) {\n+\t\tret = 0;\n+\t\tgoto done;\n+\t}\n+\n+\t/*\n+\t * Verify that the new refname is available.\n+\t */\n+\tstring_list_insert(&skip, arg->oldname);\n+\tret = refs_verify_refname_available(&arg->refs->base, arg->newname,\n+\t\t\t\t\t NULL, &skip, &errbuf);\n+\tif (ret < 0) {\n+\t\terror(\"%s\", errbuf.buf);\n+\t\tgoto done;\n+\t}\n+\n+\t/*\n+\t * When deleting the old reference we have to use two update indices:\n+\t * one to delete the old ref and its reflog, and once to create the new\n+\t * ref and its reflog. They need to be staged with two separate indices\n+\t * because the new reflog needs to encode both the deletion of the old\n+\t * branch and the creation of the new branch, and we cannot do two\n+\t * changes to a reflog in a single update.\n+\t */\n+\tdeletion_ts = creation_ts = reftable_stack_next_update_index(arg->stack);\n+\tif (arg->delete_old)\n+\t\tcreation_ts++;\n+\treftable_writer_set_limits(writer, deletion_ts, creation_ts);\n+\n+\t/*\n+\t * Add the new reference. If this is a rename then we also delete the\n+\t * old reference.\n+\t */\n+\trefs[0] = old_ref;\n+\trefs[0].refname = (char *)arg->newname;\n+\trefs[0].update_index = creation_ts;\n+\tif (arg->delete_old) {\n+\t\trefs[1].refname = (char *)arg->oldname;\n+\t\trefs[1].value_type = REFTABLE_REF_DELETION;\n+\t\trefs[1].update_index = deletion_ts;\n+\t}\n+\tret = reftable_writer_add_refs(writer, refs, arg->delete_old ? 2 : 1);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\t/*\n+\t * When deleting the old branch we need to create a reflog entry on the\n+\t * new branch name that indicates that the old branch has been deleted\n+\t * and then recreated. This is a tad weird, but matches what the files\n+\t * backend does.\n+\t */\n+\tif (arg->delete_old) {\n+\t\tstruct strbuf head_referent = STRBUF_INIT;\n+\t\tstruct object_id head_oid;\n+\t\tint append_head_reflog;\n+\t\tunsigned head_type = 0;\n+\n+\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\tmemset(&logs[logs_nr], 0, sizeof(logs[logs_nr]));\n+\t\tfill_reftable_log_record(&logs[logs_nr]);\n+\t\tlogs[logs_nr].refname = (char *)arg->newname;\n+\t\tlogs[logs_nr].update_index = deletion_ts;\n+\t\tlogs[logs_nr].value.update.message =\n+\t\t\txstrndup(arg->logmsg, arg->refs->write_options.block_size / 2);\n+\t\tlogs[logs_nr].value.update.old_hash = old_ref.value.val1;\n+\t\tlogs_nr++;\n+\n+\t\tret = read_ref_without_reload(arg->stack, \"HEAD\", &head_oid, &head_referent, &head_type);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tappend_head_reflog = (head_type & REF_ISSYMREF) && !strcmp(head_referent.buf, arg->oldname);\n+\t\tstrbuf_release(&head_referent);\n+\n+\t\t/*\n+\t\t * The files backend uses `refs_delete_ref()` to delete the old\n+\t\t * branch name, which will append a reflog entry for HEAD in\n+\t\t * case it points to the old branch.\n+\t\t */\n+\t\tif (append_head_reflog) {\n+\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\t\tlogs[logs_nr] = logs[logs_nr - 1];\n+\t\t\tlogs[logs_nr].refname = \"HEAD\";\n+\t\t\tlogs_nr++;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Create the reflog entry for the newly created branch.\n+\t */\n+\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\tmemset(&logs[logs_nr], 0, sizeof(logs[logs_nr]));\n+\tfill_reftable_log_record(&logs[logs_nr]);\n+\tlogs[logs_nr].refname = (char *)arg->newname;\n+\tlogs[logs_nr].update_index = creation_ts;\n+\tlogs[logs_nr].value.update.message =\n+\t\txstrndup(arg->logmsg, arg->refs->write_options.block_size / 2);\n+\tlogs[logs_nr].value.update.new_hash = old_ref.value.val1;\n+\tlogs_nr++;\n+\n+\t/*\n+\t * In addition to writing the reflog entry for the new branch, we also\n+\t * copy over all log entries from the old reflog. Last but not least,\n+\t * when renaming we also have to delete all the old reflog entries.\n+\t */\n+\tret = reftable_merged_table_seek_log(mt, &it, arg->oldname);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\twhile (1) {\n+\t\tret = reftable_iterator_next_log(&it, &old_log);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tif (ret > 0 || strcmp(old_log.refname, arg->oldname)) {\n+\t\t\tret = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tfree(old_log.refname);\n+\n+\t\t/*\n+\t\t * Copy over the old reflog entry with the new refname.\n+\t\t */\n+\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\tlogs[logs_nr] = old_log;\n+\t\tlogs[logs_nr].refname = (char *)arg->newname;\n+\t\tlogs_nr++;\n+\n+\t\t/*\n+\t\t * Delete the old reflog entry in case we are renaming.\n+\t\t */\n+\t\tif (arg->delete_old) {\n+\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\t\tmemset(&logs[logs_nr], 0, sizeof(logs[logs_nr]));\n+\t\t\tlogs[logs_nr].refname = (char *)arg->oldname;\n+\t\t\tlogs[logs_nr].value_type = REFTABLE_LOG_DELETION;\n+\t\t\tlogs[logs_nr].update_index = old_log.update_index;\n+\t\t\tlogs_nr++;\n+\t\t}\n+\n+\t\t/*\n+\t\t * Transfer ownership of the log record we're iterating over to\n+\t\t * the array of log records. Otherwise, the pointers would get\n+\t\t * free'd or reallocated by the iterator.\n+\t\t */\n+\t\tmemset(&old_log, 0, sizeof(old_log));\n+\t}\n+\n+\tret = reftable_writer_add_logs(writer, logs, logs_nr);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treftable_iterator_destroy(&it);\n+\tstring_list_clear(&skip, 0);\n+\tstrbuf_release(&errbuf);\n+\tfor (i = 0; i < logs_nr; i++) {\n+\t\tif (!strcmp(logs[i].refname, \"HEAD\"))\n+\t\t\tcontinue;\n+\t\tif (logs[i].value.update.old_hash == old_ref.value.val1)\n+\t\t\tlogs[i].value.update.old_hash = NULL;\n+\t\tif (logs[i].value.update.new_hash == old_ref.value.val1)\n+\t\t\tlogs[i].value.update.new_hash = NULL;\n+\t\tlogs[i].refname = NULL;\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tfree(logs);\n+\treftable_ref_record_release(&old_ref);\n+\treftable_log_record_release(&old_log);\n+\treturn ret;\n+}\n+\n+static int reftable_be_rename_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname,\n+\t\t\t\t const char *newrefname,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"rename_ref\");\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname, &newrefname);\n+\tstruct write_copy_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t\t.delete_old = 1,\n+\t};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\tret = reftable_stack_add(stack, &write_copy_table, &arg);\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treturn ret;\n+}\n+\n+static int reftable_be_copy_ref(struct ref_store *ref_store,\n+\t\t\t\tconst char *oldrefname,\n+\t\t\t\tconst char *newrefname,\n+\t\t\t\tconst char *logmsg)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"copy_ref\");\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname, &newrefname);\n+\tstruct write_copy_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\tret = reftable_stack_add(stack, &write_copy_table, &arg);\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treturn ret;\n+}\n+\n+struct reftable_reflog_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_log_record log;\n+\tstruct object_id oid;\n+\tchar *last_name;\n+\tint err;\n+};\n+\n+static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct reftable_reflog_iterator *iter =\n+\t\t(struct reftable_reflog_iterator *)ref_iterator;\n+\n+\tif (iter->err) {\n+\t\tref_iterator_abort(ref_iterator);\n+\t\treturn iter->err;\n+\t}\n+\n+\twhile (1) {\n+\t\tint flags, ret;\n+\n+\t\tret = reftable_iterator_next_log(&iter->iter, &iter->log);\n+\t\tif (ret < 0) {\n+\t\t\tref_iterator_abort(ref_iterator);\n+\t\t\treturn ITER_ERROR;\n+\t\t}\n+\t\tif (ret > 0) {\n+\t\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n+\t\t\t\treturn ITER_ERROR;\n+\t\t\treturn ITER_DONE;\n+\t\t}\n+\n+\t\t/*\n+\t\t * We want the refnames that we have reflogs for, so we skip if\n+\t\t * we've already produced this name. This could be faster by\n+\t\t * seeking directly to reflog@update_index==0.\n+\t\t */\n+\t\tif (iter->last_name && !strcmp(iter->log.refname, iter->last_name))\n+\t\t\tcontinue;\n+\n+\t\tif (!refs_resolve_ref_unsafe(&iter->refs->base, iter->log.refname,\n+\t\t\t\t\t 0, &iter->oid, &flags)) {\n+\t\t\terror(_(\"bad ref for %s\"), iter->log.refname);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfree(iter->last_name);\n+\t\titer->last_name = xstrdup(iter->log.refname);\n+\t\titer->base.refname = iter->log.refname;\n+\t\titer->base.oid = &iter->oid;\n+\t\titer->base.flags = flags;\n+\t\treturn ITER_OK;\n+\t}\n+}\n+\n+static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t\t\t struct object_id *peeled)\n+{\n+\tBUG(\"reftable reflog iterator cannot be peeled\");\n+\treturn -1;\n+}\n+\n+static int reftable_reflog_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct reftable_reflog_iterator *iter =\n+\t\t(struct reftable_reflog_iterator *)ref_iterator;\n+\treftable_log_record_release(&iter->log);\n+\treftable_iterator_destroy(&iter->iter);\n+\tfree(iter->last_name);\n+\tfree(iter);\n+\treturn ITER_DONE;\n+}\n+\n+static struct ref_iterator_vtable reftable_reflog_iterator_vtable = {\n+\t.advance = reftable_reflog_iterator_advance,\n+\t.peel = reftable_reflog_iterator_peel,\n+\t.abort = reftable_reflog_iterator_abort\n+};\n+\n+static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftable_ref_store *refs,\n+\t\t\t\t\t\t\t\t struct reftable_stack *stack)\n+{\n+\tstruct reftable_reflog_iterator *iter;\n+\tstruct reftable_merged_table *mt;\n+\tint ret;\n+\n+\titer = xcalloc(1, sizeof(*iter));\n+\tbase_ref_iterator_init(&iter->base, &reftable_reflog_iterator_vtable, 1);\n+\titer->refs = refs;\n+\titer->base.oid = &iter->oid;\n+\n+\tret = reftable_stack_reload(refs->main_stack);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\tret = reftable_merged_table_seek_log(mt, &iter->iter, \"\");\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+done:\n+\titer->err = ret;\n+\treturn iter;\n+}\n+\n+static struct ref_iterator *reftable_be_reflog_iterator_begin(struct ref_store *ref_store)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_iterator_begin\");\n+\tstruct reftable_reflog_iterator *main_iter, *worktree_iter;\n+\n+\tmain_iter = reflog_iterator_for_stack(refs, refs->main_stack);\n+\tif (!refs->worktree_stack)\n+\t\treturn &main_iter->base;\n+\n+\tworktree_iter = reflog_iterator_for_stack(refs, refs->worktree_stack);\n+\n+\treturn merge_ref_iterator_begin(1, &worktree_iter->base, &main_iter->base,\n+\t\t\t\t\titerator_select, NULL);\n+}\n+\n+static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n+\t\t\t\t\t\t const char *refname,\n+\t\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t\t void *cb_data)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent_reverse\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record log = {0};\n+\tstruct reftable_iterator it = {0};\n+\tint ret;\n+\n+\tif (refs->err < 0)\n+\t\treturn refs->err;\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\tret = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (!ret) {\n+\t\tstruct object_id old_oid, new_oid;\n+\t\tconst char *full_committer;\n+\n+\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\tif (ret < 0)\n+\t\t\tbreak;\n+\t\tif (ret > 0 || strcmp(log.refname, refname)) {\n+\t\t\tret = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\toidread(&old_oid, log.value.update.old_hash);\n+\t\toidread(&new_oid, log.value.update.new_hash);\n+\n+\t\t/*\n+\t\t * When both the old object ID and the new object ID are null\n+\t\t * then this is the reflog existence marker. The caller must\n+\t\t * not be aware of it.\n+\t\t */\n+\t\tif (is_null_oid(&old_oid) && is_null_oid(&new_oid))\n+\t\t\tcontinue;\n+\n+\t\tfull_committer = fmt_ident(log.value.update.name, log.value.update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT, NULL, IDENT_NO_DATE);\n+\t\tret = fn(&old_oid, &new_oid, full_committer,\n+\t\t\t log.value.update.time, log.value.update.tz_offset,\n+\t\t\t log.value.update.message, cb_data);\n+\t\tif (ret)\n+\t\t\tbreak;\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn ret;\n+}\n+\n+static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store,\n+\t\t\t\t\t const char *refname,\n+\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t void *cb_data)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record *logs = NULL;\n+\tstruct reftable_iterator it = {0};\n+\tsize_t logs_alloc = 0, logs_nr = 0, i;\n+\tint ret;\n+\n+\tif (refs->err < 0)\n+\t\treturn refs->err;\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\tret = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (!ret) {\n+\t\tstruct reftable_log_record log = {0};\n+\n+\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\tif (ret < 0)\n+\t\t\tbreak;\n+\t\tif (ret > 0 || strcmp(log.refname, refname)) {\n+\t\t\treftable_log_record_release(&log);\n+\t\t\tret = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\tlogs[logs_nr++] = log;\n+\t}\n+\n+\tfor (i = logs_nr; i--;) {\n+\t\tstruct reftable_log_record *log = &logs[i];\n+\t\tstruct object_id old_oid, new_oid;\n+\t\tconst char *full_committer = \"\";\n+\n+\t\toidread(&old_oid, log->value.update.old_hash);\n+\t\toidread(&new_oid, log->value.update.new_hash);\n+\n+\t\t/*\n+\t\t * When both the old object ID and the new object ID are null\n+\t\t * then this is the reflog existence marker. The caller must\n+\t\t * not be aware of it.\n+\t\t */\n+\t\tif (is_null_oid(&old_oid) && is_null_oid(&new_oid))\n+\t\t\tcontinue;\n+\n+\t\tfull_committer = fmt_ident(log->value.update.name, log->value.update.email,\n+\t\t\t\t\t WANT_COMMITTER_IDENT, NULL, IDENT_NO_DATE);\n+\t\tret = fn(&old_oid, &new_oid, full_committer,\n+\t\t\t log->value.update.time, log->value.update.tz_offset,\n+\t\t\t log->value.update.message, cb_data);\n+\t\tif (ret)\n+\t\t\tbreak;\n+\t}\n+\n+\treftable_iterator_destroy(&it);\n+\tfor (i = 0; i < logs_nr; i++)\n+\t\treftable_log_record_release(&logs[i]);\n+\tfree(logs);\n+\treturn ret;\n+}\n+\n+static int reftable_be_reflog_exists(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_exists\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record log = {0};\n+\tstruct reftable_iterator it = {0};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\t/*\n+\t * Seek the reflog to see whether it contains any reflog entries which\n+\t * aren't marked for deletion.\n+\t */\n+\twhile (1) {\n+\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tif (ret > 0 || strcmp(log.refname, refname)) {\n+\t\t\tret = 0;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tret = 1;\n+\t\tbreak;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\tif (ret < 0)\n+\t\tret = 0;\n+\treturn ret;\n+}\n+\n+struct write_reflog_existence_arg {\n+\tstruct reftable_ref_store *refs;\n+\tconst char *refname;\n+\tstruct reftable_stack *stack;\n+};\n+\n+static int write_reflog_existence_table(struct reftable_writer *writer,\n+\t\t\t\t\tvoid *cb_data)\n+{\n+\tstruct write_reflog_existence_arg *arg = cb_data;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_log_record log = {0};\n+\tint ret;\n+\n+\tret = reftable_stack_read_log(arg->stack, arg->refname, &log);\n+\tif (ret <= 0)\n+\t\tgoto done;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\t/*\n+\t * The existence entry has both old and new object ID set to the the\n+\t * null object ID. Our iterators are aware of this and will not present\n+\t * them to their callers.\n+\t */\n+\tlog.refname = xstrdup(arg->refname);\n+\tlog.update_index = ts;\n+\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\tret = reftable_writer_add_log(writer, &log);\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treftable_log_record_release(&log);\n+\treturn ret;\n+}\n+\n+static int reftable_be_create_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname,\n+\t\t\t\t struct strbuf *errmsg)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"create_reflog\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct write_reflog_existence_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_add(stack, &write_reflog_existence_table, &arg);\n+\n+done:\n+\treturn ret;\n+}\n+\n+struct write_reflog_delete_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+};\n+\n+static int write_reflog_delete_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct write_reflog_delete_arg *arg = cb_data;\n+\tstruct reftable_merged_table *mt =\n+\t\treftable_stack_merged_table(arg->stack);\n+\tstruct reftable_log_record log = {0}, tombstone = {0};\n+\tstruct reftable_iterator it = {0};\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint ret;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\t/*\n+\t * In order to delete a table we need to delete all reflog entries one\n+\t * by one. This is inefficient, but the reftable format does not have a\n+\t * better marker right now.\n+\t */\n+\tret = reftable_merged_table_seek_log(mt, &it, arg->refname);\n+\twhile (ret == 0) {\n+\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\tif (ret < 0)\n+\t\t\tbreak;\n+\t\tif (ret > 0 || strcmp(log.refname, arg->refname)) {\n+\t\t\tret = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttombstone.refname = (char *)arg->refname,\n+\t\ttombstone.value_type = REFTABLE_LOG_DELETION,\n+\t\ttombstone.update_index = log.update_index;\n+\n+\t\tret = reftable_writer_add_log(writer, &tombstone);\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn ret;\n+}\n+\n+static int reftable_be_delete_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"delete_reflog\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct write_reflog_delete_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tint ret;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\treturn ret;\n+\tret = reftable_stack_add(stack, &write_reflog_delete_table, &arg);\n+\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treturn ret;\n+}\n+\n+struct reflog_expiry_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_log_record *records;\n+\tstruct object_id update_oid;\n+\tconst char *refname;\n+\tsize_t len;\n+};\n+\n+static int write_reflog_expiry_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct reflog_expiry_arg *arg = cb_data;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tuint64_t live_records = 0;\n+\tsize_t i;\n+\tint ret;\n+\n+\tfor (i = 0; i < arg->len; i++)\n+\t\tif (arg->records[i].value_type == REFTABLE_LOG_UPDATE)\n+\t\t\tlive_records++;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tif (!is_null_oid(&arg->update_oid)) {\n+\t\tstruct reftable_ref_record ref = {0};\n+\t\tstruct object_id peeled;\n+\n+\t\tref.refname = (char *)arg->refname;\n+\t\tref.update_index = ts;\n+\n+\t\tif (!peel_object(&arg->update_oid, &peeled)) {\n+\t\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\t\tmemcpy(ref.value.val2.target_value, peeled.hash, GIT_MAX_RAWSZ);\n+\t\t\tmemcpy(ref.value.val2.value, arg->update_oid.hash, GIT_MAX_RAWSZ);\n+\t\t} else {\n+\t\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\t\tmemcpy(ref.value.val1, arg->update_oid.hash, GIT_MAX_RAWSZ);\n+\t\t}\n+\n+\t\tret = reftable_writer_add_ref(writer, &ref);\n+\t\tif (ret < 0)\n+\t\t\treturn ret;\n+\t}\n+\n+\t/*\n+\t * When there are no more entries left in the reflog we empty it\n+\t * completely, but write a placeholder reflog entry that indicates that\n+\t * the reflog still exists.\n+\t */\n+\tif (!live_records) {\n+\t\tstruct reftable_log_record log = {\n+\t\t\t.refname = (char *)arg->refname,\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.update_index = ts,\n+\t\t};\n+\n+\t\tret = reftable_writer_add_log(writer, &log);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\n+\tfor (i = 0; i < arg->len; i++) {\n+\t\tret = reftable_writer_add_log(writer, &arg->records[i]);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int reftable_be_reflog_expire(struct ref_store *ref_store,\n+\t\t\t\t const char *refname,\n+\t\t\t\t unsigned int flags,\n+\t\t\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t\t void *policy_cb_data)\n+{\n+\t/*\n+\t * For log expiry, we write tombstones for every single reflog entry\n+\t * that is to be expired. This means that the entries are still\n+\t * retrievable by delving into the stack, and expiring entries\n+\t * paradoxically takes extra memory. This memory is only reclaimed when\n+\t * compacting the reftable stack.\n+\t *\n+\t * It would be better if the refs backend supported an API that sets a\n+\t * criterion for all refs, passing the criterion to pack_refs().\n+\t *\n+\t * On the plus side, because we do the expiration per ref, we can easily\n+\t * insert the reflog existence dummies.\n+\t */\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"reflog_expire\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record *logs = NULL;\n+\tstruct reftable_log_record *rewritten = NULL;\n+\tstruct reftable_ref_record ref_record = {0};\n+\tstruct reftable_iterator it = {0};\n+\tstruct reftable_addition *add = NULL;\n+\tstruct reflog_expiry_arg arg = {0};\n+\tstruct object_id oid = {0};\n+\tuint8_t *last_hash = NULL;\n+\tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tint ret;\n+\n+\tif (refs->err < 0)\n+\t\treturn refs->err;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_new_addition(&add, stack);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_read_ref(stack, refname, &ref_record);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\tif (reftable_ref_record_val1(&ref_record))\n+\t\toidread(&oid, reftable_ref_record_val1(&ref_record));\n+\tprepare_fn(refname, &oid, policy_cb_data);\n+\n+\twhile (1) {\n+\t\tstruct reftable_log_record log = {0};\n+\t\tstruct object_id old_oid, new_oid;\n+\n+\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tif (ret > 0 || strcmp(log.refname, refname)) {\n+\t\t\treftable_log_record_release(&log);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\toidread(&old_oid, log.value.update.old_hash);\n+\t\toidread(&new_oid, log.value.update.new_hash);\n+\n+\t\t/*\n+\t\t * Skip over the reflog existence marker. We will add it back\n+\t\t * in when there are no live reflog records.\n+\t\t */\n+\t\tif (is_null_oid(&old_oid) && is_null_oid(&new_oid)) {\n+\t\t\treftable_log_record_release(&log);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\tlogs[logs_nr++] = log;\n+\t}\n+\n+\t/*\n+\t * We need to rewrite all reflog entries according to the pruning\n+\t * callback function:\n+\t *\n+\t * - If a reflog entry shall be pruned we mark the record for\n+\t * deletion.\n+\t *\n+\t * - Otherwise we may have to rewrite the chain of reflog entries so\n+\t * that gaps created by just-deleted records get backfilled.\n+\t */\n+\tCALLOC_ARRAY(rewritten, logs_nr);\n+\tfor (i = logs_nr; i--;) {\n+\t\tstruct reftable_log_record *dest = &rewritten[i];\n+\t\tstruct object_id old_oid, new_oid;\n+\n+\t\t*dest = logs[i];\n+\t\toidread(&old_oid, logs[i].value.update.old_hash);\n+\t\toidread(&new_oid, logs[i].value.update.new_hash);\n+\n+\t\tif (should_prune_fn(&old_oid, &new_oid, logs[i].value.update.email,\n+\t\t\t\t (timestamp_t)logs[i].value.update.time,\n+\t\t\t\t logs[i].value.update.tz_offset,\n+\t\t\t\t logs[i].value.update.message,\n+\t\t\t\t policy_cb_data)) {\n+\t\t\tdest->value_type = REFTABLE_LOG_DELETION;\n+\t\t} else {\n+\t\t\tif ((flags & EXPIRE_REFLOGS_REWRITE) && last_hash)\n+\t\t\t\tdest->value.update.old_hash = last_hash;\n+\t\t\tlast_hash = logs[i].value.update.new_hash;\n+\t\t}\n+\t}\n+\n+\tif (flags & EXPIRE_REFLOGS_UPDATE_REF && last_hash &&\n+\t reftable_ref_record_val1(&ref_record))\n+\t\toidread(&arg.update_oid, last_hash);\n+\n+\targ.records = rewritten;\n+\targ.len = logs_nr;\n+\targ.stack = stack,\n+\targ.refname = refname,\n+\n+\tret = reftable_addition_add(add, &write_reflog_expiry_table, &arg);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\t/*\n+\t * Future improvement: we could skip writing records that were\n+\t * not changed.\n+\t */\n+\tif (!(flags & EXPIRE_REFLOGS_DRY_RUN))\n+\t\tret = reftable_addition_commit(add);\n+\n+done:\n+\tif (add)\n+\t\tcleanup_fn(policy_cb_data);\n+\tassert(ret != REFTABLE_API_ERROR);\n+\n+\treftable_ref_record_release(&ref_record);\n+\treftable_iterator_destroy(&it);\n+\treftable_addition_destroy(add);\n+\tfor (i = 0; i < logs_nr; i++)\n+\t\treftable_log_record_release(&logs[i]);\n+\tfree(logs);\n+\tfree(rewritten);\n+\treturn ret;\n+}\n+\n+struct ref_storage_be refs_be_reftable = {\n+\t.name = \"reftable\",\n+\t.init = reftable_be_init,\n+\t.init_db = reftable_be_init_db,\n+\t.transaction_prepare = reftable_be_transaction_prepare,\n+\t.transaction_finish = reftable_be_transaction_finish,\n+\t.transaction_abort = reftable_be_transaction_abort,\n+\t.initial_transaction_commit = reftable_be_initial_transaction_commit,\n+\n+\t.pack_refs = reftable_be_pack_refs,\n+\t.create_symref = reftable_be_create_symref,\n+\t.rename_ref = reftable_be_rename_ref,\n+\t.copy_ref = reftable_be_copy_ref,\n+\n+\t.iterator_begin = reftable_be_iterator_begin,\n+\t.read_raw_ref = reftable_be_read_raw_ref,\n+\t.read_symbolic_ref = reftable_be_read_symbolic_ref,\n+\n+\t.reflog_iterator_begin = reftable_be_reflog_iterator_begin,\n+\t.for_each_reflog_ent = reftable_be_for_each_reflog_ent,\n+\t.for_each_reflog_ent_reverse = reftable_be_for_each_reflog_ent_reverse,\n+\t.reflog_exists = reftable_be_reflog_exists,\n+\t.create_reflog = reftable_be_create_reflog,\n+\t.delete_reflog = reftable_be_delete_reflog,\n+\t.reflog_expire = reftable_be_reflog_expire,\n+};\ndiff --git a/repository.h b/repository.h\nindex 7a250a6605..1645cef518 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -24,8 +24,9 @@ enum fetch_negotiation_setting {\n \tFETCH_NEGOTIATION_NOOP,\n };\n \n-#define REF_STORAGE_FORMAT_UNKNOWN 0\n-#define REF_STORAGE_FORMAT_FILES 1\n+#define REF_STORAGE_FORMAT_UNKNOWN 0\n+#define REF_STORAGE_FORMAT_FILES 1\n+#define REF_STORAGE_FORMAT_REFTABLE 2\n \n struct repo_settings {\n \tint initialized;\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nnew file mode 100755\nindex 0000000000..94ee46984a\n--- /dev/null\n+++ b/t/t0610-reftable-basics.sh\n@@ -0,0 +1,887 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable basics'\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+if ! test_have_prereq REFTABLE\n+then\n+\tskip_all='skipping reftable tests; set GIT_TEST_DEFAULT_REF_FORMAT=reftable'\n+\ttest_done\n+fi\n+\n+INVALID_OID=$(test_oid 001)\n+\n+test_expect_success 'init: creates basic reftable structures' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_path_is_dir repo/.git/reftable &&\n+\ttest_path_is_file repo/.git/reftable/tables.list &&\n+\techo reftable >expect &&\n+\tgit -C repo rev-parse --show-ref-format >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init: sha256 object format via environment variable' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tGIT_DEFAULT_HASH=sha256 git init repo &&\n+\tcat >expect <<-EOF &&\n+\tsha256\n+\treftable\n+\tEOF\n+\tgit -C repo rev-parse --show-object-format --show-ref-format >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init: sha256 object format via option' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init --object-format=sha256 repo &&\n+\tcat >expect <<-EOF &&\n+\tsha256\n+\treftable\n+\tEOF\n+\tgit -C repo rev-parse --show-object-format --show-ref-format >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init: reinitializing reftable backend succeeds' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo for-each-ref >expect &&\n+\tgit init --ref-format=reftable repo &&\n+\tgit -C repo for-each-ref >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init: reinitializing files with reftable backend fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init --ref-format=files repo &&\n+\ttest_commit -C repo file &&\n+\n+\tcp repo/.git/HEAD expect &&\n+\ttest_must_fail git init --ref-format=reftable repo &&\n+\ttest_cmp expect repo/.git/HEAD\n+'\n+\n+test_expect_success 'init: reinitializing reftable with files backend fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init --ref-format=reftable repo &&\n+\ttest_commit -C repo file &&\n+\n+\tcp repo/.git/HEAD expect &&\n+\ttest_must_fail git init --ref-format=files repo &&\n+\ttest_cmp expect repo/.git/HEAD\n+'\n+\n+test_expect_perms () {\n+\tlocal perms=\"$1\"\n+\tlocal file=\"$2\"\n+\tlocal actual=$(ls -l \"$file\") &&\n+\n+\tcase \"$actual\" in\n+\t$perms*)\n+\t\t: happy\n+\t\t;;\n+\t*)\n+\t\techo \"$(basename $2) is not $perms but $actual\"\n+\t\tfalse\n+\t\t;;\n+\tesac\n+}\n+\n+for umask in 002 022\n+do\n+\ttest_expect_success POSIXPERM 'init: honors core.sharedRepository' '\n+\t\ttest_when_finished \"rm -rf repo\" &&\n+\t\t(\n+\t\t\tumask $umask &&\n+\t\t\tgit init --shared=true repo &&\n+\t\t\ttest 1 = \"$(git -C repo config core.sharedrepository)\"\n+\t\t) &&\n+\t\ttest_expect_perms \"-rw-rw-r--\" repo/.git/reftable/tables.list &&\n+\t\tfor table in repo/.git/reftable/*.ref\n+\t\tdo\n+\t\t\ttest_expect_perms \"-rw-rw-r--\" \"$table\" ||\n+\t\t\treturn 1\n+\t\tdone\n+\t'\n+done\n+\n+test_expect_success 'clone: can clone reftable repository' '\n+\ttest_when_finished \"rm -rf repo clone\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo message1 file1 &&\n+\n+\tgit clone repo cloned &&\n+\techo reftable >expect &&\n+\tgit -C cloned rev-parse --show-ref-format >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_path_is_file cloned/file1\n+'\n+\n+test_expect_success 'clone: can clone reffiles into reftable repository' '\n+\ttest_when_finished \"rm -rf reffiles reftable\" &&\n+\tgit init --ref-format=files reffiles &&\n+\ttest_commit -C reffiles A &&\n+\tgit clone --ref-format=reftable ./reffiles reftable &&\n+\n+\tgit -C reffiles rev-parse HEAD >expect &&\n+\tgit -C reftable rev-parse HEAD >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C reftable rev-parse --show-ref-format >actual &&\n+\techo reftable >expect &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C reffiles rev-parse --show-ref-format >actual &&\n+\techo files >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'clone: can clone reftable into reffiles repository' '\n+\ttest_when_finished \"rm -rf reffiles reftable\" &&\n+\tgit init --ref-format=reftable reftable &&\n+\ttest_commit -C reftable A &&\n+\tgit clone --ref-format=files ./reftable reffiles &&\n+\n+\tgit -C reftable rev-parse HEAD >expect &&\n+\tgit -C reffiles rev-parse HEAD >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C reftable rev-parse --show-ref-format >actual &&\n+\techo reftable >expect &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C reffiles rev-parse --show-ref-format >actual &&\n+\techo files >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'ref transaction: corrupted tables cause failure' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file1 &&\n+\t\tfor f in .git/reftable/*.ref\n+\t\tdo\n+\t\t\t: >\"$f\" || return 1\n+\t\tdone &&\n+\t\ttest_must_fail git update-ref refs/heads/main HEAD\n+\t)\n+'\n+\n+test_expect_success 'ref transaction: corrupted tables.list cause failure' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file1 &&\n+\t\techo garbage >.git/reftable/tables.list &&\n+\t\ttest_must_fail git update-ref refs/heads/main HEAD\n+\t)\n+'\n+\n+test_expect_success 'ref transaction: refuses to write ref causing F/D conflict' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo file &&\n+\ttest_must_fail git -C repo update-ref refs/heads/main/forbidden\n+'\n+\n+test_expect_success 'ref transaction: deleting ref with invalid name fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo file &&\n+\ttest_must_fail git -C repo update-ref -d ../../my-private-file\n+'\n+\n+test_expect_success 'ref transaction: can skip object ID verification' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_must_fail test-tool -C repo ref-store main update-ref msg refs/heads/branch $INVALID_OID $ZERO_OID 0 &&\n+\ttest-tool -C repo ref-store main update-ref msg refs/heads/branch $INVALID_OID $ZERO_OID REF_SKIP_OID_VERIFICATION\n+'\n+\n+test_expect_success 'ref transaction: updating same ref multiple times fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tcat >updates <<-EOF &&\n+\tupdate refs/heads/main $A\n+\tupdate refs/heads/main $A\n+\tEOF\n+\tcat >expect <<-EOF &&\n+\tfatal: multiple updates for ref ${SQ}refs/heads/main${SQ} not allowed\n+\tEOF\n+\ttest_must_fail git -C repo update-ref --stdin err &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'ref transaction: can delete symbolic self-reference with git-symbolic-ref(1)' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tgit -C repo symbolic-ref refs/heads/self refs/heads/self &&\n+\tgit -C repo symbolic-ref -d refs/heads/self\n+'\n+\n+test_expect_success 'ref transaction: deleting symbolic self-reference without --no-deref fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tgit -C repo symbolic-ref refs/heads/self refs/heads/self &&\n+\tcat >expect <<-EOF &&\n+\terror: multiple updates for ${SQ}refs/heads/self${SQ} (including one via symref ${SQ}refs/heads/self${SQ}) are not allowed\n+\tEOF\n+\ttest_must_fail git -C repo update-ref -d refs/heads/self 2>err &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'ref transaction: deleting symbolic self-reference with --no-deref succeeds' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tgit -C repo symbolic-ref refs/heads/self refs/heads/self &&\n+\tgit -C repo update-ref -d --no-deref refs/heads/self\n+'\n+\n+test_expect_success 'ref transaction: creating symbolic ref fails with F/D conflict' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tcat >expect <<-EOF &&\n+\terror: unable to write symref for refs/heads: file/directory conflict\n+\tEOF\n+\ttest_must_fail git -C repo symbolic-ref refs/heads refs/heads/foo 2>err &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'ref transaction: ref deletion' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file &&\n+\t\tHEAD_OID=$(git show-ref -s --verify HEAD) &&\n+\t\tcat >expect <<-EOF &&\n+\t\t$HEAD_OID refs/heads/main\n+\t\t$HEAD_OID refs/tags/file\n+\t\tEOF\n+\t\tgit show-ref >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\ttest_must_fail git update-ref -d refs/tags/file $INVALID_OID &&\n+\t\tgit show-ref >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit update-ref -d refs/tags/file $HEAD_OID &&\n+\t\techo \"$HEAD_OID refs/heads/main\" >expect &&\n+\t\tgit show-ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'ref transaction: writes cause auto-compaction' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\n+\tgit init repo &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\ttest_commit -C repo --no-tag A &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n+\n+\ttest_commit -C repo --no-tag B &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+check_fsync_events () {\n+\tlocal trace=\"$1\" &&\n+\tshift &&\n+\n+\tcat >expect &&\n+\tsed -n \\\n+\t\t-e '/^{\"event\":\"counter\",.*\"category\":\"fsync\",/ {\n+\t\t\ts/.*\"category\":\"fsync\",//;\n+\t\t\ts/}$//;\n+\t\t\tp;\n+\t\t}' \\\n+\t\t<\"$trace\" >actual &&\n+\ttest_cmp expect actual\n+}\n+\n+# A fix for this is in flight via jc/reftable-core-fsync.\n+test_expect_failure 'ref transaction: writes are synced' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo initial &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\tGIT_TEST_FSYNC=true \\\n+\t\tgit -C repo -c core.fsync=reference update-ref refs/heads/branch HEAD &&\n+\tcheck_fsync_events trace2.txt <<-EOF\n+\t\"name\":\"hardware-flush\",\"count\":2\n+\tEOF\n+'\n+\n+test_expect_success 'pack-refs: compacts tables' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\n+\ttest_commit -C repo A &&\n+\tls -1 repo/.git/reftable >table-files &&\n+\ttest_line_count = 4 table-files &&\n+\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n+\n+\tgit -C repo pack-refs &&\n+\tls -1 repo/.git/reftable >table-files &&\n+\ttest_line_count = 2 table-files &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'pack-refs: prunes stale tables' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttouch repo/.git/reftable/stale-table.ref &&\n+\tgit -C repo pack-refs &&\n+\ttest_path_is_missing repo/.git/reftable/stable-ref.ref\n+'\n+\n+test_expect_success 'pack-refs: does not prune non-table files' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttouch repo/.git/reftable/garbage &&\n+\tgit -C repo pack-refs &&\n+\ttest_path_is_file repo/.git/reftable/garbage\n+'\n+\n+for umask in 002 022\n+do\n+\t# A fix for this is in flight via ps/reftable-compacted-tables-permission-fix.\n+\ttest_expect_failure POSIXPERM 'pack-refs: honors core.sharedRepository' '\n+\t\ttest_when_finished \"rm -rf repo\" &&\n+\t\t(\n+\t\t\tumask $umask &&\n+\t\t\tgit init --shared=true repo &&\n+\t\t\ttest_commit -C repo A &&\n+\t\t\ttest_line_count = 3 repo/.git/reftable/tables.list\n+\t\t) &&\n+\t\tgit -C repo pack-refs &&\n+\t\ttest_expect_perms \"-rw-rw-r--\" repo/.git/reftable/tables.list &&\n+\t\tfor table in repo/.git/reftable/*.ref\n+\t\tdo\n+\t\t\ttest_expect_perms \"-rw-rw-r--\" \"$table\" ||\n+\t\t\treturn 1\n+\t\tdone\n+\t'\n+done\n+\n+# A fix for this is in flight.\n+test_expect_failure 'packed-refs: writes are synced' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo initial &&\n+\ttest_line_count = 2 table-files &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\tGIT_TEST_FSYNC=true \\\n+\t\tgit -C repo -c core.fsync=reference pack-refs &&\n+\tcheck_fsync_events trace2.txt <<-EOF\n+\t\"name\":\"hardware-flush\",\"count\":2\n+\tEOF\n+'\n+\n+test_expect_success 'ref iterator: bogus names are flagged' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit --no-tag file &&\n+\t\ttest-tool ref-store main update-ref msg \"refs/heads/bogus..name\" $(git rev-parse HEAD) $ZERO_OID REF_SKIP_REFNAME_VERIFICATION &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\t$ZERO_OID refs/heads/bogus..name 0xc\n+\t\t$(git rev-parse HEAD) refs/heads/main 0x0\n+\t\tEOF\n+\t\ttest-tool ref-store main for-each-ref \"\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'ref iterator: missing object IDs are not flagged' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest-tool ref-store main update-ref msg \"refs/heads/broken-hash\" $INVALID_OID $ZERO_OID REF_SKIP_OID_VERIFICATION &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\t$INVALID_OID refs/heads/broken-hash 0x0\n+\t\tEOF\n+\t\ttest-tool ref-store main for-each-ref \"\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'basic: commit and list refs' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo file &&\n+\ttest_write_lines refs/heads/main refs/tags/file >expect &&\n+\tgit -C repo for-each-ref --format=\"%(refname)\" >actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'basic: can write large commit message' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tperl -e \"\n+\t\tprint \\\"this is a long commit message\\\" x 50000\n+\t\" >commit-msg &&\n+\tgit -C repo commit --allow-empty --file=../commit-msg\n+'\n+\n+test_expect_success 'basic: show-ref fails with empty repository' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_must_fail git -C repo show-ref >actual &&\n+\ttest_must_be_empty actual\n+'\n+\n+test_expect_success 'basic: can check out unborn branch' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tgit -C repo checkout -b main\n+'\n+\n+test_expect_success 'basic: peeled tags are stored' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo file &&\n+\tgit -C repo tag -m \"annotated tag\" test_tag HEAD &&\n+\tfor ref in refs/heads/main refs/tags/file refs/tags/test_tag refs/tags/test_tag^{}\n+\tdo\n+\t\techo \"$(git -C repo rev-parse \"$ref\") $ref\" || return 1\n+\tdone >expect &&\n+\tgit -C repo show-ref -d >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'basic: for-each-ref can print symrefs' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file &&\n+\t\tgit branch &&\n+\t\tgit symbolic-ref refs/heads/sym refs/heads/main &&\n+\t\tcat >expected <<-EOF &&\n+\t\trefs/heads/main\n+\t\tEOF\n+\t\tgit for-each-ref --format=\"%(symref)\" refs/heads/sym >actual &&\n+\t\ttest_cmp expected actual\n+\t)\n+'\n+\n+test_expect_success 'basic: notes' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\twrite_script fake_editor <<-\\EOF &&\n+\t\techo \"$MSG\" >\"$1\"\n+\t\techo \"$MSG\" >&2\n+\t\tEOF\n+\n+\t\ttest_commit 1st &&\n+\t\ttest_commit 2nd &&\n+\t\tGIT_EDITOR=./fake_editor MSG=b4 git notes add &&\n+\t\tGIT_EDITOR=./fake_editor MSG=b3 git notes edit &&\n+\t\techo b4 >expect &&\n+\t\tgit notes --ref commits@{1} show >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'basic: stash' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file &&\n+\t\tgit stash list >expect &&\n+\t\ttest_line_count = 0 expect &&\n+\n+\t\techo hoi >>file.t &&\n+\t\tgit stash push -m stashed &&\n+\t\tgit stash list >expect &&\n+\t\ttest_line_count = 1 expect &&\n+\n+\t\tgit stash clear &&\n+\t\tgit stash list >expect &&\n+\t\ttest_line_count = 0 expect\n+\t)\n+'\n+\n+test_expect_success 'basic: cherry-pick' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit message1 file1 &&\n+\t\ttest_commit message2 file2 &&\n+\t\tgit branch source &&\n+\t\tgit checkout HEAD^ &&\n+\t\ttest_commit message3 file3 &&\n+\t\tgit cherry-pick source &&\n+\t\ttest_path_is_file file2\n+\t)\n+'\n+\n+test_expect_success 'basic: rebase' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit message1 file1 &&\n+\t\ttest_commit message2 file2 &&\n+\t\tgit branch source &&\n+\t\tgit checkout HEAD^ &&\n+\t\ttest_commit message3 file3 &&\n+\t\tgit rebase source &&\n+\t\ttest_path_is_file file2\n+\t)\n+'\n+\n+test_expect_success 'reflog: can delete separate reflog entries' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit file &&\n+\t\ttest_commit file2 &&\n+\t\ttest_commit file3 &&\n+\t\ttest_commit file4 &&\n+\t\tgit reflog >actual &&\n+\t\tgrep file3 actual &&\n+\n+\t\tgit reflog delete HEAD@{1} &&\n+\t\tgit reflog >actual &&\n+\t\t! grep file3 actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: can switch to previous branch' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file1 &&\n+\t\tgit checkout -b branch1 &&\n+\t\ttest_commit file2 &&\n+\t\tgit checkout -b branch2 &&\n+\t\tgit switch - &&\n+\t\tgit rev-parse --symbolic-full-name HEAD >actual &&\n+\t\techo refs/heads/branch1 >expect &&\n+\t\ttest_cmp actual expect\n+\t)\n+'\n+\n+test_expect_success 'reflog: copying branch writes reflog entry' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file1 &&\n+\t\ttest_commit file2 &&\n+\t\toid=$(git rev-parse --short HEAD) &&\n+\t\tgit branch src &&\n+\t\tcat >expect <<-EOF &&\n+\t\t${oid} dst@{0}: Branch: copied refs/heads/src to refs/heads/dst\n+\t\t${oid} dst@{1}: branch: Created from main\n+\t\tEOF\n+\t\tgit branch -c src dst &&\n+\t\tgit reflog dst >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: renaming branch writes reflog entry' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit symbolic-ref HEAD refs/heads/before &&\n+\t\ttest_commit file &&\n+\t\tgit show-ref >expected.refs &&\n+\t\tsed s/before/after/g expected &&\n+\t\tgit branch -M after &&\n+\t\tgit show-ref >actual &&\n+\t\ttest_cmp expected actual &&\n+\t\techo refs/heads/after >expected &&\n+\t\tgit symbolic-ref HEAD >actual &&\n+\t\ttest_cmp expected actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: can store empty logs' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_must_fail test-tool ref-store main reflog-exists refs/heads/branch &&\n+\t\ttest-tool ref-store main create-reflog refs/heads/branch &&\n+\t\ttest-tool ref-store main reflog-exists refs/heads/branch &&\n+\t\ttest-tool ref-store main for-each-reflog-ent-reverse refs/heads/branch >actual &&\n+\t\ttest_must_be_empty actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: expiry empties reflog' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit initial &&\n+\t\tgit checkout -b branch &&\n+\t\ttest_commit fileA &&\n+\t\ttest_commit fileB &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\tcommit: fileB\n+\t\tcommit: fileA\n+\t\tbranch: Created from HEAD\n+\t\tEOF\n+\t\tgit reflog show --format=\"%gs\" refs/heads/branch >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit reflog expire branch --expire=all &&\n+\t\tgit reflog show --format=\"%gs\" refs/heads/branch >actual &&\n+\t\ttest_must_be_empty actual &&\n+\t\ttest-tool ref-store main reflog-exists refs/heads/branch\n+\t)\n+'\n+\n+test_expect_success 'reflog: can be deleted' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\ttest-tool ref-store main reflog-exists refs/heads/main &&\n+\t\ttest-tool ref-store main delete-reflog refs/heads/main &&\n+\t\ttest_must_fail test-tool ref-store main reflog-exists refs/heads/main\n+\t)\n+'\n+\n+test_expect_success 'reflog: garbage collection deletes reflog entries' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\tfor count in $(test_seq 1 10)\n+\t\tdo\n+\t\t\ttest_commit \"number $count\" file.t $count number-$count ||\n+\t\t\treturn 1\n+\t\tdone &&\n+\t\tgit reflog refs/heads/main >actual &&\n+\t\ttest_line_count = 10 actual &&\n+\t\tgrep \"commit (initial): number 1\" actual &&\n+\t\tgrep \"commit: number 10\" actual &&\n+\n+\t\tgit gc &&\n+\t\tgit reflog refs/heads/main >actual &&\n+\t\ttest_line_count = 0 actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: updates via HEAD update HEAD reflog' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit main-one &&\n+\t\tgit checkout -b new-branch &&\n+\t\ttest_commit new-one &&\n+\t\ttest_commit new-two &&\n+\n+\t\techo new-one >expect &&\n+\t\tgit log -1 --format=%s HEAD@{1} >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'worktree: adding worktree creates separate stack' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\ttest_path_is_file repo/.git/worktrees/worktree/refs/heads &&\n+\techo \"ref: refs/heads/.invalid\" >expect &&\n+\ttest_cmp expect repo/.git/worktrees/worktree/HEAD &&\n+\ttest_path_is_dir repo/.git/worktrees/worktree/reftable &&\n+\ttest_path_is_file repo/.git/worktrees/worktree/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: pack-refs in main repo packs main refs' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tgit -C repo worktree add ../worktree &&\n+\n+\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\tgit -C repo pack-refs &&\n+\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: pack-refs in worktree packs worktree refs' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tgit -C repo worktree add ../worktree &&\n+\n+\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: creating shared ref updates main stack' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\tgit -C repo pack-refs &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tgit -C worktree update-ref refs/heads/shared HEAD &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: creating per-worktree ref updates worktree stack' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\tgit -C repo pack-refs &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tgit -C worktree update-ref refs/bisect/per-worktree HEAD &&\n+\ttest_line_count = 2 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: creating per-worktree ref from main repo' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\tgit -C repo pack-refs &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tgit -C repo update-ref worktrees/worktree/refs/bisect/per-worktree HEAD &&\n+\ttest_line_count = 2 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: creating per-worktree ref from second worktree' '\n+\ttest_when_finished \"rm -rf repo wt1 wt2\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../wt1 &&\n+\tgit -C repo worktree add ../wt2 &&\n+\tgit -C repo pack-refs &&\n+\tgit -C wt1 pack-refs &&\n+\tgit -C wt2 pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/wt1/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/worktrees/wt2/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tgit -C wt1 update-ref worktrees/wt2/refs/bisect/per-worktree HEAD &&\n+\ttest_line_count = 1 repo/.git/worktrees/wt1/reftable/tables.list &&\n+\ttest_line_count = 2 repo/.git/worktrees/wt2/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: can create shared and per-worktree ref in one transaction' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\tgit -C repo pack-refs &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tcat >stdin <<-EOF &&\n+\tcreate worktrees/worktree/refs/bisect/per-worktree HEAD\n+\tcreate refs/branches/shared HEAD\n+\tEOF\n+\tgit -C repo update-ref --stdin expect &&\n+\tgit -C worktree symbolic-ref HEAD >actual &&\n+\ttest_cmp expect actual &&\n+\tgit -C worktree checkout branch1\n+'\n+\n+test_expect_success 'worktree: adds worktree with detached HEAD' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tgit -C repo rev-parse main >expect &&\n+\n+\tgit -C repo worktree add --detach ../worktree main &&\n+\tgit -C worktree rev-parse HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'fetch: accessing FETCH_HEAD special ref works' '\n+\ttest_when_finished \"rm -rf repo sub\" &&\n+\n+\tgit init sub &&\n+\ttest_commit -C sub two &&\n+\tgit -C sub rev-parse HEAD >expect &&\n+\n+\tgit init repo &&\n+\ttest_commit -C repo one &&\n+\tgit -C repo fetch ../sub &&\n+\tgit -C repo rev-parse FETCH_HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/t0611-reftable-httpd.sh b/t/t0611-reftable-httpd.sh\nnew file mode 100755\nindex 0000000000..5e05b9c1f2\n--- /dev/null\n+++ b/t/t0611-reftable-httpd.sh\n@@ -0,0 +1,26 @@\n+#!/bin/sh\n+\n+test_description='reftable HTTPD tests'\n+\n+. ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd\n+\n+REPO=\"$HTTPD_DOCUMENT_ROOT_PATH/repo\"\n+\n+test_expect_success 'serving ls-remote' '\n+\tgit init --ref-format=reftable -b main \"$REPO\" &&\n+\tcd \"$REPO\" &&\n+\ttest_commit m1 &&\n+\t>.git/git-daemon-export-ok &&\n+\tgit ls-remote \"http://127.0.0.1:$LIB_HTTPD_PORT/smart/repo\" | cut -f 2-2 -d \"\t\" >actual &&\n+\tcat >expect <<-EOF &&\n+\tHEAD\n+\trefs/heads/main\n+\trefs/tags/m1\n+\tEOF\n+\ttest_cmp actual expect\n+'\n+\n+test_done\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex fc93aa57e6..0fb8757b7d 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1750,6 +1750,8 @@ esac\n case \"$GIT_DEFAULT_REF_FORMAT\" in\n files)\n \ttest_set_prereq REFFILES;;\n+reftable)\n+\ttest_set_prereq REFTABLE;;\n *)\n \techo 2>&1 \"error: unknown ref format $GIT_DEFAULT_REF_FORMAT\"\n \texit 1\n-- \n2.43.GIT\n\n"},{"id":"487593","messageId":"bb694fa132fe602c92a7e96d5ccf3fef7986fff8.1706601199.git.ps@pks.im","threadId":"60815","inReplyTo":"cover.1706601199.git.ps@pks.im","subject":"[PATCH 2/2] ci: add jobs to test with the reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-01-30T08:05:35Z","receivedAt":"2024-01-30T08:05:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Add CI jobs for both GitHub Workflows and GitLab CI to run Git with the\nnew reftable backend.\n\nSigned-off-by: Patrick Steinhardt \n---\n .github/workflows/main.yml | 9 +++++++++\n .gitlab-ci.yml | 9 +++++++++\n ci/lib.sh | 2 +-\n ci/run-build-and-tests.sh | 3 +++\n 4 files changed, 22 insertions(+), 1 deletion(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 4d97da57ec..1b43e49dad 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -266,6 +266,9 @@ jobs:\n - jobname: linux-sha256\n cc: clang\n pool: ubuntu-latest\n+ - jobname: linux-reftable\n+ cc: clang\n+ pool: ubuntu-latest\n - jobname: linux-gcc\n cc: gcc\n cc_package: gcc-8\n@@ -277,6 +280,9 @@ jobs:\n - jobname: osx-clang\n cc: clang\n pool: macos-13\n+ - jobname: osx-reftable\n+ cc: clang\n+ pool: macos-13\n - jobname: osx-gcc\n cc: gcc\n cc_package: gcc-13\n@@ -287,6 +293,9 @@ jobs:\n - jobname: linux-leaks\n cc: gcc\n pool: ubuntu-latest\n+ - jobname: linux-reftable-leaks\n+ cc: gcc\n+ pool: ubuntu-latest\n - jobname: linux-asan-ubsan\n cc: clang\n pool: ubuntu-latest\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nindex 43bfbd8834..c0fa2fe90b 100644\n--- a/.gitlab-ci.yml\n+++ b/.gitlab-ci.yml\n@@ -26,6 +26,9 @@ test:linux:\n - jobname: linux-sha256\n image: ubuntu:latest\n CC: clang\n+ - jobname: linux-reftable\n+ image: ubuntu:latest\n+ CC: clang\n - jobname: linux-gcc\n image: ubuntu:20.04\n CC: gcc\n@@ -40,6 +43,9 @@ test:linux:\n - jobname: linux-leaks\n image: ubuntu:latest\n CC: gcc\n+ - jobname: linux-reftable-leaks\n+ image: ubuntu:latest\n+ CC: gcc\n - jobname: linux-asan-ubsan\n image: ubuntu:latest\n CC: clang\n@@ -79,6 +85,9 @@ test:osx:\n - jobname: osx-clang\n image: macos-13-xcode-14\n CC: clang\n+ - jobname: osx-reftable\n+ image: macos-13-xcode-14\n+ CC: clang\n artifacts:\n paths:\n - t/failed-test-artifacts\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex d5dd2f2697..0a73fc7bd1 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -367,7 +367,7 @@ linux-musl)\n \tMAKEFLAGS=\"$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes\"\n \tMAKEFLAGS=\"$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8\"\n \t;;\n-linux-leaks)\n+linux-leaks|linux-reftable-leaks)\n \texport SANITIZE=leak\n \texport GIT_TEST_PASSING_SANITIZE_LEAK=true\n \texport GIT_TEST_SANITIZE_LEAK_LOG=true\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 7a1466b868..c192bd613c 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -37,6 +37,9 @@ linux-clang)\n linux-sha256)\n \texport GIT_TEST_DEFAULT_HASH=sha256\n \t;;\n+linux-reftable|linux-reftable-leaks|osx-reftable)\n+\texport GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+\t;;\n pedantic)\n \t# Don't run the tests; we only care about whether Git can be\n \t# built.\n-- \n2.43.GIT\n\n"},{"id":"487614","messageId":"xmqqil3aa30a.fsf@gitster.g","threadId":"60815","inReplyTo":"cover.1706601199.git.ps@pks.im","subject":"Re: [PATCH 0/2] refs: introduce reftable backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-01-30T22:08:05Z","receivedAt":"2024-01-30T22:08:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> Patrick Steinhardt (2):\n> refs: introduce reftable backend\n> ci: add jobs to test with the reftable backend\n\nI haven't checked the code in refs/reftable-backend.c at all, but\njudging from the fallout to other files, it issurprisingly cleanly\ndone. I somehow expected that we would need more changes like\nexposing the helper function to compute shared permission bits,\nbut I see that it was the only change needed. Everything else done\nto other files are natural requirement to plug the new thing as a\nnew ref backend to the system. Very nicely done.\n\nThanks.\n\n\n> .github/workflows/main.yml | 9 +\n> .gitlab-ci.yml | 9 +\n> Documentation/ref-storage-format.txt | 2 +\n> .../technical/repository-version.txt | 5 +-\n> Makefile | 1 +\n> ci/lib.sh | 2 +-\n> ci/run-build-and-tests.sh | 3 +\n> contrib/workdir/git-new-workdir | 2 +-\n> path.c | 2 +-\n> path.h | 1 +\n> refs.c | 1 +\n> refs/refs-internal.h | 1 +\n> refs/reftable-backend.c | 2286 +++++++++++++++++\n> repository.h | 5 +-\n> t/t0610-reftable-basics.sh | 887 +++++++\n> t/t0611-reftable-httpd.sh | 26 +\n> t/test-lib.sh | 2 +\n> 17 files changed, 3237 insertions(+), 7 deletions(-)\n> create mode 100644 refs/reftable-backend.c\n> create mode 100755 t/t0610-reftable-basics.sh\n> create mode 100755 t/t0611-reftable-httpd.sh\n"},{"id":"487746","messageId":"CAOLa=ZTrPuRAbkDgg1Dh4M-3o58Rw1KruAMcw47o2V_LrK9KPg@mail.gmail.com","threadId":"60815","inReplyTo":"5598cd13074beef092a61235a505476d0cbceb90.1706601199.git.ps@pks.im","subject":"Re: [PATCH 1/2] refs: introduce reftable backend","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-01T15:17:45Z","receivedAt":"2024-02-01T15:17:47Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Hello,\n\nPatrick Steinhardt writes:\n> - It becomes possible to do truly atomic writes where either all refs\n> are committed to disk or none are. This was not possible with the\n> \"files\" backend because ref updates were split across multiple loose\n> files.\n>\n> - The disk space required to store many refs is reduced, both compared\n> to loose refs and packed-refs. This is enabled both by the reftable\n> format being a binary format, which is more compact, and by prefix\n> compression.\n>\n> - We can ignore filesystem-specific behaviour as ref names are not\n> encoded via paths anymore. This means there is no need to handle\n> case sensitivity on Windows systems or Unicode precomposition on\n> macOS.\n>\n> - There is no need to rewrite the complete refdb anymore every time a\n> ref is being deleted like it was the case for packed-refs. This\n> means that ref deletions are now constant time instead of scaling\n> linearly with the number of refs.\n>\n> - We can ignore file/directory conflicts so that it becomes possible\n> to store both \"refs/heads/foo\" and \"refs/heads/foo/bar\".\n>\n> - Due to this property we can retain reflogs for deleted refs. We have\n> previously been deleting reflogs together with their refs to avoid\n> file/directory conflicts, which is not necessary anymore.\n>\n\nNit: Maybe also a point about how with the current files backend doesn't\nhave a good way to differentiate between regular files and refs. While\nregular refs are in the 'refs/' folder, pseudorefs have no namespace.\nWhile with the reftable implementation we know that everything within\nthe reftable is a ref and there are no other refs we need to consider.\n\n\n> Performance-wise things very much depend on the actual workload. The\n> following benchmarks compare the \"files\" and \"reftable\" backends in the\n> current version:\n\nPretty nice to have these numbers here.\n\n>\n> - Creating N refs in separate transactions shows that the \"files\"\n> backend is ~50% faster. This is not surprising given that creating a\n> ref only requires us to create a single loose ref. The \"reftable\"\n> backend will also perform auto compaction on updates. In real-world\n> workloads we would likely also want to perform pack loose refs,\n> which would likely change the picture.\n>\n> Benchmark 1: update-ref: create refs sequentially (refformat = files)\n> Time (mean ± σ): 2.1 ms ± 0.3 ms [User: 0.6 ms, System: 1.7 ms]\n> Range (min … max): 1.8 ms … 4.3 ms 133 runs\n>\n> Benchmark 2: update-ref: create refs sequentially (refformat = reftable)\n> Time (mean ± σ): 2.7 ms ± 0.1 ms [User: 0.6 ms, System: 2.2 ms]\n> Range (min … max): 2.4 ms … 2.9 ms 132 runs\n>\n> Benchmark 3: update-ref: create refs sequentially (refformat = files)\n> Time (mean ± σ): 1.975 s ± 0.006 s [User: 0.437 s, System: 1.535 s]\n> Range (min … max): 1.969 s … 1.980 s 3 runs\n>\n> Benchmark 4: update-ref: create refs sequentially (refformat = reftable)\n> Time (mean ± σ): 2.611 s ± 0.013 s [User: 0.782 s, System: 1.825 s]\n> Range (min … max): 2.597 s … 2.622 s 3 runs\n>\n> Benchmark 5: update-ref: create refs sequentially (refformat = files)\n> Time (mean ± σ): 198.442 s ± 0.241 s [User: 43.051 s, System: 155.250 s]\n> Range (min … max): 198.189 s … 198.670 s 3 runs\n>\n> Benchmark 6: update-ref: create refs sequentially (refformat = reftable)\n> Time (mean ± σ): 294.509 s ± 4.269 s [User: 104.046 s, System: 190.326 s]\n> Range (min … max): 290.223 s … 298.761 s 3 runs\n>\n\nNit: The refcount is missing in these benchmarks.\n\n> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> new file mode 100644\n> index 0000000000..895de0b273\n> --- /dev/null\n> +++ b/refs/reftable-backend.c\n> @@ -0,0 +1,2286 @@\n> +#include \"../git-compat-util.h\"\n> +#include \"../abspath.h\"\n> +#include \"../chdir-notify.h\"\n> +#include \"../config.h\"\n\nI believe this header can be dropped.\n\nNot sure about removing headers which aren't needed. The coding\nguidelines mentions that we should include headers that declare the\nfunctions and the types used. I found some headers here which are not\nrequired considering this rule.\n\n> +#include \"../environment.h\"\n> +#include \"../gettext.h\"\n> +#include \"../hash.h\"\n> +#include \"../hex.h\"\n> +#include \"../iterator.h\"\n> +#include \"../ident.h\"\n> +#include \"../lockfile.h\"\n> +#include \"../object.h\"\n> +#include \"../path.h\"\n> +#include \"../refs.h\"\n> +#include \"../reftable/reftable-stack.h\"\n> +#include \"../reftable/reftable-record.h\"\n> +#include \"../reftable/reftable-error.h\"\n> +#include \"../reftable/reftable-blocksource.h\"\n\nThis header can be dropped too.\n\n> +#include \"../reftable/reftable-reader.h\"\n\nThis also.\n\n> +#include \"../reftable/reftable-iterator.h\"\n> +#include \"../reftable/reftable-merged.h\"\n> +#include \"../reftable/reftable-generic.h\"\n\nThis one as well.\n\n> +#include \"../setup.h\"\n> +#include \"../strmap.h\"\n> +#include \"../worktree.h\"\n\nThis one too.\n\n> +struct reftable_ref_store {\n> +\tstruct ref_store base;\n> +\n> +\tstruct reftable_stack *main_stack;\n> +\tstruct reftable_stack *worktree_stack;\n> +\tstruct strmap worktree_stacks;\n> +\tstruct reftable_write_options write_options;\n> +\n> +\tunsigned int store_flags;\n> +\tint err;\n> +};\n\nSo, I'm assuming that `main_stack` here would be the primary reference\ndb, even when inside the worktree. I'm wondering why we can't just have\n\n struct reftable_stack *current_stack;\n struct strmap worktree_stacks;\n\nReading further on, it becomes necessary to have both, because the user\ncould\n 1. Request main-worktree/{ref}\n 2. Request worktrees/{worktree}/{ref}\nSo it's important that we have access to both the main worktree, the\ncurrent worktree and also any other worktree.\n\nMaybe we could just drop the `struct reftable_stack *worktree_stack` and\nrely on the map entirely, but I guess `worktree_stack` acts as a caching\nlayer and avoids using the map unless necessary.\n\n> +static struct ref_store *reftable_be_init(struct repository *repo,\n> +\t\t\t\t\t const char *gitdir,\n> +\t\t\t\t\t unsigned int store_flags)\n> +{\n> +\tstruct reftable_ref_store *refs = xcalloc(1, sizeof(*refs));\n> +\tstruct strbuf path = STRBUF_INIT;\n> +\tint is_worktree;\n> +\tmode_t mask;\n> +\n> +\tmask = umask(0);\n> +\tumask(mask);\n\nTook me more than a glance to understand this. For other readers who\ndidn't know: there is no single API to get the current umask. But\numask(2) sets the umask and returns the prev umask. So we set a random\nvalue (here 0) to get the prev umask and reset it.\n\n> +\n> +\tbase_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);\n> +\tstrmap_init(&refs->worktree_stacks);\n> +\trefs->store_flags = store_flags;\n> +\trefs->write_options.block_size = 4096;\n>\n\nPerhaps use `DEFAULT_BLOCK_SIZE` from \"reftable/constants.h\" here.\n\n> +\trefs->write_options.hash_id = repo->hash_algo->format_id;\n> +\trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n> +\n> +\t/*\n> +\t * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.\n> +\t * This stack contains both the shared and the main worktree refs.\n> +\t */\n> +\tis_worktree = get_common_dir_noenv(&path, gitdir);\n> +\tif (!is_worktree) {\n> +\t\tstrbuf_reset(&path);\n> +\t\tstrbuf_realpath(&path, gitdir, 0);\n> +\t}\n\nNit: would be nice to have a comment that for wortrees, `gitdir` would\nalready be an absolute path.\n\n> +\tstrbuf_addstr(&path, \"/reftable\");\n> +\trefs->err = reftable_new_stack(&refs->main_stack, path.buf,\n> +\t\t\t\t refs->write_options);\n> +\tif (refs->err)\n> +\t\tgoto done;\n> +\n> +\t/*\n> +\t * If we're in a worktree we also need to set up the worktree reftable\n> +\t * stack that is contained in the per-worktree GIT_DIR.\n> +\t */\n> +\tif (is_worktree) {\n> +\t\tstrbuf_reset(&path);\n> +\t\tstrbuf_addf(&path, \"%s/reftable\", gitdir);\n> +\n> +\t\trefs->err = reftable_new_stack(&refs->worktree_stack, path.buf,\n> +\t\t\t\t\t refs->write_options);\n> +\t\tif (refs->err)\n> +\t\t\tgoto done;\n> +\t}\n\nWondering if we should also add this to the `refs->worktree_stacks`.\n\n> +struct reftable_ref_iterator {\n> +\tstruct ref_iterator base;\n> +\tstruct reftable_ref_store *refs;\n> +\tstruct reftable_iterator iter;\n> +\tstruct reftable_ref_record ref;\n> +\tstruct object_id oid;\n> +\n> +\tconst char *prefix;\n> +\tunsigned int flags;\n> +\tint err;\n> +};\n\nSo the `flags` in this structure are for the iterator itself but the\n`flags` in `base` are flags related to the current ref stored in `base`.\n\n> +static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> +{\n> +\tstruct reftable_ref_iterator *iter =\n> +\t\t(struct reftable_ref_iterator *)ref_iterator;\n> +\tstruct reftable_ref_store *refs = iter->refs;\n> +\n> +\twhile (!iter->err) {\n> +\t\tint flags = 0;\n> +\n\nNit: I think the syntax in `reftable_reflog_iterator_advance` is similar but\nnicer to read, the usage of `while(1)` and returning within the while\nloop is better than breaking here and returning outside, either ways, I\nthink it'd be nicer to make both of them consistent.\n\n> +\t\titer->err = reftable_iterator_next_ref(&iter->iter, &iter->ref);\n> +\t\tif (iter->err)\n> +\t\t\tbreak;\n> +\n> +\t\t/*\n> +\t\t * The files backend only lists references contained in\n> +\t\t * \"refs/\". We emulate the same behaviour here and thus skip\n> +\t\t * all references that don't start with this prefix.\n> +\t\t */\n> +\t\tif (!starts_with(iter->ref.refname, \"refs/\"))\n> +\t\t\tcontinue;\n> +\n\nSince my patch series [1] to print all refs is now merged to next, maybe\nyou could add this in?\n\n diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n index 895de0b273..3f4f905292 100644\n --- a/refs/reftable-backend.c\n +++ b/refs/reftable-backend.c\n @@ -348,11 +348,10 @@ static int\nreftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\tbreak;\n\n \t\t/*\n -\t\t * The files backend only lists references contained in\n -\t\t * \"refs/\". We emulate the same behaviour here and thus skip\n -\t\t * all references that don't start with this prefix.\n +\t\t * Unless the `DO_FOR_EACH_INCLUDE_ALL_REFS` flag is use, we only\n +\t\t * list references contained in \"refs/\" to mimic the file-backend.\n \t\t */\n -\t\tif (!starts_with(iter->ref.refname, \"refs/\"))\n +\t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_ALL_REFS) &&\n!starts_with(iter->ref.refname, \"refs/\"))\n \t\t\tcontinue;\n\n \t\tif (iter->prefix &&\n\n> +static enum iterator_selection iterator_select(struct ref_iterator *iter_worktree,\n> +\t\t\t\t\t struct ref_iterator *iter_common,\n> +\t\t\t\t\t void *cb_data UNUSED)\n> +{\n> +\tif (iter_worktree && !iter_common) {\n> +\t\t/*\n> +\t\t * Return the worktree ref if there are no more common refs.\n> +\t\t */\n> +\t\treturn ITER_SELECT_0;\n> +\t} else if (iter_common) {\n> +\t\t/*\n> +\t\t * In case we have pending worktree and common refs we need to\n> +\t\t * yield them based on their lexicographical order. Worktree\n> +\t\t * refs that have the same name as common refs shadow the\n> +\t\t * latter.\n> +\t\t */\n> +\t\tif (iter_worktree) {\n> +\t\t\tint cmp = strcmp(iter_worktree->refname,\n> +\t\t\t\t\t iter_common->refname);\n> +\t\t\tif (cmp < 0)\n> +\t\t\t\treturn ITER_SELECT_0;\n> +\t\t\telse if (!cmp)\n> +\t\t\t\treturn ITER_SELECT_0_SKIP_1;\n> +\t\t}\n> +\n> +\t\t /*\n> +\t\t * Otherwise, if we either have no worktree refs anymore or if\n> +\t\t * the common ref sorts before the next worktree ref, we need\n> +\t\t * to figure out whether the next common ref belongs to the\n> +\t\t * main worktree. In that case, it should be ignored.\n> +\t\t */\n> +\t\tif (parse_worktree_ref(iter_common->refname, NULL, NULL,\n> +\t\t\t\t NULL) == REF_WORKTREE_SHARED)\n> +\t\t\treturn ITER_SELECT_1;\n> +\n\nI'm not sure I understand this. When would this situation occur?\n\n> +\t\treturn ITER_SKIP_1;\n> +\t} else {\n> +\t\treturn ITER_DONE;\n> +\t}\n> +}\n> +\n> +\n> +static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n> +\t\t\t\t\t struct ref_transaction *transaction,\n> +\t\t\t\t\t struct strbuf *err)\n> +{\n> +\tstruct reftable_ref_store *refs =\n> +\t\treftable_be_downcast(ref_store, REF_STORE_WRITE|REF_STORE_MAIN, \"ref_transaction_prepare\");\n> +\tstruct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT;\n> +\tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n> +\tstruct reftable_transaction_data *tx_data = NULL;\n> +\tstruct object_id head_oid;\n> +\tunsigned int head_type = 0;\n> +\tsize_t i;\n> +\tint ret;\n> +\n> +\tret = refs->err;\n> +\tif (ret < 0)\n> +\t\tgoto done;\n> +\n> +\ttx_data = xcalloc(1, sizeof(*tx_data));\n> +\n> +\t/*\n> +\t * Preprocess all updates. For one we check that there are no duplicate\n> +\t * reference updates in this transaction. Second, we lock all stacks\n> +\t * that will be modified during the transaction.\n> +\t */\n> +\tfor (i = 0; i < transaction->nr; i++) {\n> +\t\tret = prepare_transaction_update(NULL, refs, tx_data,\n> +\t\t\t\t\t\t transaction->updates[i], err);\n> +\t\tif (ret)\n> +\t\t\tgoto done;\n> +\n> +\t\tstring_list_append(&affected_refnames,\n> +\t\t\t\t transaction->updates[i]->refname);\n> +\t}\n> +\n> +\t/*\n> +\t * Now that we have counted updates per stack we can preallocate their\n> +\t * arrays. This avoids having to reallocate many times.\n> +\t */\n> +\tfor (i = 0; i < tx_data->args_nr; i++) {\n> +\t\tCALLOC_ARRAY(tx_data->args[i].updates, tx_data->args[i].updates_expected);\n> +\t\ttx_data->args[i].updates_alloc = tx_data->args[i].updates_expected;\n> +\t}\n> +\n> +\t/*\n> +\t * Fail if a refname appears more than once in the transaction.\n> +\t * This code is taken from the files backend and is a good candidate to\n> +\t * be moved into the generic layer.\n> +\t */\n> +\tstring_list_sort(&affected_refnames);\n> +\tif (ref_update_reject_duplicates(&affected_refnames, err)) {\n> +\t\tret = TRANSACTION_GENERIC_ERROR;\n> +\t\tgoto done;\n> +\t}\n> +\n> +\tret = read_ref_without_reload(stack_for(refs, \"HEAD\", NULL), \"HEAD\", &head_oid,\n> +\t\t\t\t &head_referent, &head_type);\n> +\tif (ret < 0)\n> +\t\tgoto done;\n> +\n> +\tfor (i = 0; i < transaction->nr; i++) {\n> +\t\tstruct ref_update *u = transaction->updates[i];\n> +\t\tstruct object_id current_oid = {0};\n> +\t\tstruct reftable_stack *stack;\n> +\t\tconst char *rewritten_ref;\n> +\n> +\t\tstack = stack_for(refs, u->refname, &rewritten_ref);\n>\n\nWondering why we didn't just iterate over `tx_data.args` and\n`args[i].updates` since there we already have the required stack.\n\nUpdate: we don't save rewritten_ref in tx_data, so that's one blocker\nfor this.\n\n> +\n> +\t\tif (u->type & REF_ISSYMREF) {\n> +\t\t\tconst char *resolved = refs_resolve_ref_unsafe(&refs->base, u->refname, 0,\n> +\t\t\t\t\t\t\t\t ¤t_oid, NULL);\n> +\n> +\t\t\tif (u->flags & REF_NO_DEREF) {\n> +\t\t\t\t/*\n> +\t\t\t\t * The reftable stack is locked at this point\n> +\t\t\t\t * already, so it should be safe to call\n> +\t\t\t\t * `refs_resolve_ref_unsafe()` here.\n> +\t\t\t\t */\n\nShouldn't this comment be a few lines before?\n\n> +\n> +static int write_transaction_table(struct reftable_writer *writer, void *cb_data)\n> +{\n> +\tstruct write_transaction_table_arg *arg = cb_data;\n> +\tstruct reftable_merged_table *mt =\n> +\t\treftable_stack_merged_table(arg->stack);\n> +\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n> +\tstruct reftable_log_record *logs = NULL;\n> +\tsize_t logs_nr = 0, logs_alloc = 0, i;\n> +\tint ret = 0;\n> +\n> +\tQSORT(arg->updates, arg->updates_nr, transaction_update_cmp);\n> +\n> +\treftable_writer_set_limits(writer, ts, ts);\n> +\n> +\tfor (i = 0; i < arg->updates_nr; i++) {\n> +\t\tstruct reftable_transaction_update *tx_update = &arg->updates[i];\n> +\t\tstruct ref_update *u = tx_update->update;\n> +\n> +\t\t/*\n> +\t\t * Write a reflog entry when updating a ref to point to\n> +\t\t * something new in either of the following cases:\n> +\t\t *\n> +\t\t * - The reference is about to be deleted. We always want to\n> +\t\t * delete the reflog in that case.\n> +\t\t * - REF_FORCE_CREATE_REFLOG is set, asking us to always create\n> +\t\t * the reflog entry.\n> +\t\t * - `core.logAllRefUpdates` tells us to create the reflog for\n> +\t\t * the given ref.\n> +\t\t */\n> +\t\tif (u->flags & REF_HAVE_NEW && !(u->type & REF_ISSYMREF) && is_null_oid(&u->new_oid)) {\n> +\t\t\tstruct reftable_log_record log = {0};\n> +\t\t\tstruct reftable_iterator it = {0};\n> +\n> +\t\t\t/*\n> +\t\t\t * When deleting refs we also delete all reflog entries\n> +\t\t\t * with them. While it is not strictly required to\n> +\t\t\t * delete reflogs together with their refs, this\n> +\t\t\t * matches the behaviour of the files backend.\n> +\t\t\t *\n> +\t\t\t * Unfortunately, we have no better way than to delete\n> +\t\t\t * all reflog entries one by one.\n> +\t\t\t */\n> +\t\t\tret = reftable_merged_table_seek_log(mt, &it, u->refname);\n> +\t\t\twhile (ret == 0) {\n> +\t\t\t\tstruct reftable_log_record *tombstone;\n> +\n> +\t\t\t\tret = reftable_iterator_next_log(&it, &log);\n> +\t\t\t\tif (ret < 0)\n> +\t\t\t\t\tbreak;\n> +\t\t\t\tif (ret > 0 || strcmp(log.refname, u->refname)) {\n> +\t\t\t\t\tret = 0;\n> +\t\t\t\t\tbreak;\n> +\t\t\t\t}\n> +\n> +\t\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n> +\t\t\t\ttombstone = &logs[logs_nr++];\n> +\t\t\t\ttombstone->refname = xstrdup(u->refname);\n> +\t\t\t\ttombstone->value_type = REFTABLE_LOG_DELETION,\n\nWhy is this a comma? one other such instance in this file.\n\n> +static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n> +{\n> +\tstruct write_copy_arg *arg = cb_data;\n> +\tuint64_t deletion_ts, creation_ts;\n> +\tstruct reftable_merged_table *mt = reftable_stack_merged_table(arg->stack);\n> +\tstruct reftable_ref_record old_ref = {0}, refs[2] = {0};\n> +\tstruct reftable_log_record old_log = {0}, *logs = NULL;\n> +\tstruct reftable_iterator it = {0};\n> +\tstruct string_list skip = STRING_LIST_INIT_NODUP;\n> +\tstruct strbuf errbuf = STRBUF_INIT;\n> +\tsize_t logs_nr = 0, logs_alloc = 0, i;\n> +\tint ret;\n> +\n> +\tif (reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref)) {\n> +\t\tret = error(_(\"refname %s not found\"), arg->oldname);\n> +\t\tgoto done;\n> +\t}\n> +\tif (old_ref.value_type == REFTABLE_REF_SYMREF) {\n> +\t\tret = error(_(\"refname %s is a symbolic ref, copying it is not supported\"),\n> +\t\t\t arg->oldname);\n> +\t\tgoto done;\n> +\t}\n> +\n> +\t/*\n> +\t * There's nothing to do in case the old and new name are the same, so\n> +\t * we exit early in that case.\n> +\t */\n> +\tif (!strcmp(arg->oldname, arg->newname)) {\n> +\t\tret = 0;\n> +\t\tgoto done;\n> +\t}\n> +\n> +\t/*\n> +\t * Verify that the new refname is available.\n> +\t */\n> +\tstring_list_insert(&skip, arg->oldname);\n> +\tret = refs_verify_refname_available(&arg->refs->base, arg->newname,\n> +\t\t\t\t\t NULL, &skip, &errbuf);\n> +\tif (ret < 0) {\n> +\t\terror(\"%s\", errbuf.buf);\n> +\t\tgoto done;\n> +\t}\n> +\n> +\t/*\n> +\t * When deleting the old reference we have to use two update indices:\n> +\t * one to delete the old ref and its reflog, and once to create the new\n>\n\ns/once/one\n\n> +\t * ref and its reflog. They need to be staged with two separate indices\n> +\t * because the new reflog needs to encode both the deletion of the old\n> +\t * branch and the creation of the new branch, and we cannot do two\n> +\t * changes to a reflog in a single update.\n> +\t */\n> +\tdeletion_ts = creation_ts = reftable_stack_next_update_index(arg->stack);\n> +\tif (arg->delete_old)\n> +\t\tcreation_ts++;\n> +\treftable_writer_set_limits(writer, deletion_ts, creation_ts);\n> +\n> +\t/*\n> +\t * Add the new reference. If this is a rename then we also delete the\n> +\t * old reference.\n> +\t */\n> +\trefs[0] = old_ref;\n> +\trefs[0].refname = (char *)arg->newname;\n> +\trefs[0].update_index = creation_ts;\n> +\tif (arg->delete_old) {\n> +\t\trefs[1].refname = (char *)arg->oldname;\n> +\t\trefs[1].value_type = REFTABLE_REF_DELETION;\n> +\t\trefs[1].update_index = deletion_ts;\n> +\t}\n> +\tret = reftable_writer_add_refs(writer, refs, arg->delete_old ? 2 : 1);\n> +\tif (ret < 0)\n> +\t\tgoto done;\n> +\n> +\t/*\n> +\t * When deleting the old branch we need to create a reflog entry on the\n> +\t * new branch name that indicates that the old branch has been deleted\n> +\t * and then recreated. This is a tad weird, but matches what the files\n> +\t * backend does.\n> +\t */\n> +\tif (arg->delete_old) {\n> +\t\tstruct strbuf head_referent = STRBUF_INIT;\n> +\t\tstruct object_id head_oid;\n> +\t\tint append_head_reflog;\n> +\t\tunsigned head_type = 0;\n> +\n> +\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n> +\t\tmemset(&logs[logs_nr], 0, sizeof(logs[logs_nr]));\n> +\t\tfill_reftable_log_record(&logs[logs_nr]);\n> +\t\tlogs[logs_nr].refname = (char *)arg->newname;\n> +\t\tlogs[logs_nr].update_index = deletion_ts;\n> +\t\tlogs[logs_nr].value.update.message =\n> +\t\t\txstrndup(arg->logmsg, arg->refs->write_options.block_size / 2);\n\nQuestion: here and other places in this file, shouldn't we free memory\nallocated by `xstrndup`?\n\nAlso, why is it `block_size / 2` ?\n\n> +static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftable_ref_store *refs,\n> +\t\t\t\t\t\t\t\t struct reftable_stack *stack)\n> +{\n> +\tstruct reftable_reflog_iterator *iter;\n> +\tstruct reftable_merged_table *mt;\n> +\tint ret;\n> +\n> +\titer = xcalloc(1, sizeof(*iter));\n> +\tbase_ref_iterator_init(&iter->base, &reftable_reflog_iterator_vtable, 1);\n> +\titer->refs = refs;\n> +\titer->base.oid = &iter->oid;\n> +\n> +\tret = reftable_stack_reload(refs->main_stack);\n> +\tif (ret < 0)\n> +\t\tgoto done;\n> +\n> +\tmt = reftable_stack_merged_table(stack);\n> +\tret = reftable_merged_table_seek_log(mt, &iter->iter, \"\");\n> +\tif (ret < 0)\n> +\t\tgoto done;\n> +\n\nKeeping it similar to `ref_iterator_for_stack`, perhaps:\n\n ret = refs->err;\n\tif (ret)\n\t\tgoto done;\n\n\tret = reftable_stack_reload(stack);\n\tif (ret)\n\t\tgoto done;\n\n\tmerged_table = reftable_stack_merged_table(stack);\n\n\tret = reftable_merged_table_seek_ref(merged_table, &iter->iter, prefix);\n\tif (ret)\n\t\tgoto done;\n\n> +static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n> +\t\t\t\t\t\t const char *refname,\n> +\t\t\t\t\t\t each_reflog_ent_fn fn,\n> +\t\t\t\t\t\t void *cb_data)\n> +{\n> +\tstruct reftable_ref_store *refs =\n> +\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent_reverse\");\n> +\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n> +\tstruct reftable_merged_table *mt = NULL;\n> +\tstruct reftable_log_record log = {0};\n> +\tstruct reftable_iterator it = {0};\n> +\tint ret;\n> +\n> +\tif (refs->err < 0)\n> +\t\treturn refs->err;\n\nNit: seems like this function and the one below\n`reftable_be_for_each_reflog_ent`, share the same code apart from\niteration direction. Wonder if it'd be nicer to extract out the common\ncode from them.\n\n> +static int reftable_be_reflog_exists(struct ref_store *ref_store,\n> +\t\t\t\t const char *refname)\n> +{\n> +\tstruct reftable_ref_store *refs =\n> +\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_exists\");\n> +\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n> +\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n> +\tstruct reftable_log_record log = {0};\n> +\tstruct reftable_iterator it = {0};\n> +\tint ret;\n> +\n> +\tret = refs->err;\n> +\tif (ret < 0)\n> +\t\tgoto done;\n> +\n> +\tret = reftable_stack_reload(stack);\n> +\tif (ret)\n> +\t\tgoto done;\n> +\n> +\tret = reftable_merged_table_seek_log(mt, &it, refname);\n> +\tif (ret)\n> +\t\tgoto done;\n> +\n> +\t/*\n> +\t * Seek the reflog to see whether it contains any reflog entries which\n> +\t * aren't marked for deletion.\n> +\t */\n\nShouldn't we be checking `log.value_type` in that case?\n\nThis email only reviews this file, I'll review the others in follow up\nemail[s].\n\n[1]: https://git.kernel.org/pub/scm/git/git.git/commit/?h=next&id=e7a9234a8b39bd69de12b105731413ab8e62942e\n"},{"id":"487792","messageId":"ZbyoL3r7yEVYe35B@tanuki","threadId":"60815","inReplyTo":"CAOLa=ZTrPuRAbkDgg1Dh4M-3o58Rw1KruAMcw47o2V_LrK9KPg@mail.gmail.com","subject":"Re: [PATCH 1/2] refs: introduce reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-02T08:30:39Z","receivedAt":"2024-02-02T08:30:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Feb 01, 2024 at 07:17:45AM -0800, Karthik Nayak wrote:\n> Hello,\n> \n> Patrick Steinhardt writes:\n> > - It becomes possible to do truly atomic writes where either all refs\n> > are committed to disk or none are. This was not possible with the\n> > \"files\" backend because ref updates were split across multiple loose\n> > files.\n> >\n> > - The disk space required to store many refs is reduced, both compared\n> > to loose refs and packed-refs. This is enabled both by the reftable\n> > format being a binary format, which is more compact, and by prefix\n> > compression.\n> >\n> > - We can ignore filesystem-specific behaviour as ref names are not\n> > encoded via paths anymore. This means there is no need to handle\n> > case sensitivity on Windows systems or Unicode precomposition on\n> > macOS.\n> >\n> > - There is no need to rewrite the complete refdb anymore every time a\n> > ref is being deleted like it was the case for packed-refs. This\n> > means that ref deletions are now constant time instead of scaling\n> > linearly with the number of refs.\n> >\n> > - We can ignore file/directory conflicts so that it becomes possible\n> > to store both \"refs/heads/foo\" and \"refs/heads/foo/bar\".\n> >\n> > - Due to this property we can retain reflogs for deleted refs. We have\n> > previously been deleting reflogs together with their refs to avoid\n> > file/directory conflicts, which is not necessary anymore.\n> >\n> \n> Nit: Maybe also a point about how with the current files backend doesn't\n> have a good way to differentiate between regular files and refs. While\n> regular refs are in the 'refs/' folder, pseudorefs have no namespace.\n> While with the reftable implementation we know that everything within\n> the reftable is a ref and there are no other refs we need to consider.\n\nTrue, I'll add that.\n\n> > Performance-wise things very much depend on the actual workload. The\n> > following benchmarks compare the \"files\" and \"reftable\" backends in the\n> > current version:\n> \n> Pretty nice to have these numbers here.\n> \n> >\n> > - Creating N refs in separate transactions shows that the \"files\"\n> > backend is ~50% faster. This is not surprising given that creating a\n> > ref only requires us to create a single loose ref. The \"reftable\"\n> > backend will also perform auto compaction on updates. In real-world\n> > workloads we would likely also want to perform pack loose refs,\n> > which would likely change the picture.\n> >\n> > Benchmark 1: update-ref: create refs sequentially (refformat = files)\n> > Time (mean ± σ): 2.1 ms ± 0.3 ms [User: 0.6 ms, System: 1.7 ms]\n> > Range (min … max): 1.8 ms … 4.3 ms 133 runs\n> >\n> > Benchmark 2: update-ref: create refs sequentially (refformat = reftable)\n> > Time (mean ± σ): 2.7 ms ± 0.1 ms [User: 0.6 ms, System: 2.2 ms]\n> > Range (min … max): 2.4 ms … 2.9 ms 132 runs\n> >\n> > Benchmark 3: update-ref: create refs sequentially (refformat = files)\n> > Time (mean ± σ): 1.975 s ± 0.006 s [User: 0.437 s, System: 1.535 s]\n> > Range (min … max): 1.969 s … 1.980 s 3 runs\n> >\n> > Benchmark 4: update-ref: create refs sequentially (refformat = reftable)\n> > Time (mean ± σ): 2.611 s ± 0.013 s [User: 0.782 s, System: 1.825 s]\n> > Range (min … max): 2.597 s … 2.622 s 3 runs\n> >\n> > Benchmark 5: update-ref: create refs sequentially (refformat = files)\n> > Time (mean ± σ): 198.442 s ± 0.241 s [User: 43.051 s, System: 155.250 s]\n> > Range (min … max): 198.189 s … 198.670 s 3 runs\n> >\n> > Benchmark 6: update-ref: create refs sequentially (refformat = reftable)\n> > Time (mean ± σ): 294.509 s ± 4.269 s [User: 104.046 s, System: 190.326 s]\n> > Range (min … max): 290.223 s … 298.761 s 3 runs\n> >\n> \n> Nit: The refcount is missing in these benchmarks.\n\nGood catch, this is an edge case of hyperfine. Because the refcount is\nused in the command line it's not shown anymore in its output. I'll just\nmassage these manually.\n\n> > diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> > new file mode 100644\n> > index 0000000000..895de0b273\n> > --- /dev/null\n> > +++ b/refs/reftable-backend.c\n> > @@ -0,0 +1,2286 @@\n> > +#include \"../git-compat-util.h\"\n> > +#include \"../abspath.h\"\n> > +#include \"../chdir-notify.h\"\n> > +#include \"../config.h\"\n> \n> I believe this header can be dropped.\n> \n> Not sure about removing headers which aren't needed. The coding\n> guidelines mentions that we should include headers that declare the\n> functions and the types used. I found some headers here which are not\n> required considering this rule.\n\nIt's always good hygiene to not include unneeded headers. Also helps to\nkeep build times lower and reduce unnecessary rebuilds.\n\n> > +#include \"../environment.h\"\n> > +#include \"../gettext.h\"\n> > +#include \"../hash.h\"\n> > +#include \"../hex.h\"\n> > +#include \"../iterator.h\"\n> > +#include \"../ident.h\"\n> > +#include \"../lockfile.h\"\n> > +#include \"../object.h\"\n> > +#include \"../path.h\"\n> > +#include \"../refs.h\"\n> > +#include \"../reftable/reftable-stack.h\"\n> > +#include \"../reftable/reftable-record.h\"\n> > +#include \"../reftable/reftable-error.h\"\n> > +#include \"../reftable/reftable-blocksource.h\"\n> \n> This header can be dropped too.\n> \n> > +#include \"../reftable/reftable-reader.h\"\n> \n> This also.\n> \n> > +#include \"../reftable/reftable-iterator.h\"\n> > +#include \"../reftable/reftable-merged.h\"\n> > +#include \"../reftable/reftable-generic.h\"\n> \n> This one as well.\n> \n> > +#include \"../setup.h\"\n> > +#include \"../strmap.h\"\n> > +#include \"../worktree.h\"\n> \n> This one too.\n> \n> > +struct reftable_ref_store {\n> > +\tstruct ref_store base;\n> > +\n> > +\tstruct reftable_stack *main_stack;\n> > +\tstruct reftable_stack *worktree_stack;\n> > +\tstruct strmap worktree_stacks;\n> > +\tstruct reftable_write_options write_options;\n> > +\n> > +\tunsigned int store_flags;\n> > +\tint err;\n> > +};\n> \n> So, I'm assuming that `main_stack` here would be the primary reference\n> db, even when inside the worktree. I'm wondering why we can't just have\n> \n> struct reftable_stack *current_stack;\n> struct strmap worktree_stacks;\n> \n> Reading further on, it becomes necessary to have both, because the user\n> could\n> 1. Request main-worktree/{ref}\n> 2. Request worktrees/{worktree}/{ref}\n> So it's important that we have access to both the main worktree, the\n> current worktree and also any other worktree.\n\nIndeed. I'll add some comments to point out their respective uses.\n\n> Maybe we could just drop the `struct reftable_stack *worktree_stack` and\n> rely on the map entirely, but I guess `worktree_stack` acts as a caching\n> layer and avoids using the map unless necessary.\n\nI doubt that the map access would be all that expensive in practice. The\nproblem rather is that we don't know about the worktree name when we\ninit the backend, and looking it up feels fragile to me and may involve\nparsing all worktree metadata. So I rather want to avoid that to the\nextent possible.\n\n> > +\tbase_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);\n> > +\tstrmap_init(&refs->worktree_stacks);\n> > +\trefs->store_flags = store_flags;\n> > +\trefs->write_options.block_size = 4096;\n> >\n> \n> Perhaps use `DEFAULT_BLOCK_SIZE` from \"reftable/constants.h\" here.\n\nThis isn't exposed given that only the \"reftable/reftable-*.h\" headers\nare considered to be public. Ideally, we'd just leave it unset and let\nthe default block size kick in. But `reftable_new_stack()` doesn't set\nup the block size at all in that case, and that leads to failures.\n\nI'll leave this as-is for now. We can clean it up in the future.\n\n> > +\tstrbuf_addstr(&path, \"/reftable\");\n> > +\trefs->err = reftable_new_stack(&refs->main_stack, path.buf,\n> > +\t\t\t\t refs->write_options);\n> > +\tif (refs->err)\n> > +\t\tgoto done;\n> > +\n> > +\t/*\n> > +\t * If we're in a worktree we also need to set up the worktree reftable\n> > +\t * stack that is contained in the per-worktree GIT_DIR.\n> > +\t */\n> > +\tif (is_worktree) {\n> > +\t\tstrbuf_reset(&path);\n> > +\t\tstrbuf_addf(&path, \"%s/reftable\", gitdir);\n> > +\n> > +\t\trefs->err = reftable_new_stack(&refs->worktree_stack, path.buf,\n> > +\t\t\t\t\t refs->write_options);\n> > +\t\tif (refs->err)\n> > +\t\t\tgoto done;\n> > +\t}\n> \n> Wondering if we should also add this to the `refs->worktree_stacks`.\n\nIdeally we would, but at this point we have no easy way to figure out\nthe worktree name to the best of my knowledge.\n\n> > +struct reftable_ref_iterator {\n> > +\tstruct ref_iterator base;\n> > +\tstruct reftable_ref_store *refs;\n> > +\tstruct reftable_iterator iter;\n> > +\tstruct reftable_ref_record ref;\n> > +\tstruct object_id oid;\n> > +\n> > +\tconst char *prefix;\n> > +\tunsigned int flags;\n> > +\tint err;\n> > +};\n> \n> So the `flags` in this structure are for the iterator itself but the\n> `flags` in `base` are flags related to the current ref stored in `base`.\n\nYup.\n\n> > +static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> > +{\n> > +\tstruct reftable_ref_iterator *iter =\n> > +\t\t(struct reftable_ref_iterator *)ref_iterator;\n> > +\tstruct reftable_ref_store *refs = iter->refs;\n> > +\n> > +\twhile (!iter->err) {\n> > +\t\tint flags = 0;\n> > +\n> \n> Nit: I think the syntax in `reftable_reflog_iterator_advance` is similar but\n> nicer to read, the usage of `while(1)` and returning within the while\n> loop is better than breaking here and returning outside, either ways, I\n> think it'd be nicer to make both of them consistent.\n\nThe problem here is that the ref iterator has multiple exit paths that\nneed to be treated the same, so it is easier to have a common exit path.\nI'll thus touch up the reflog iterator to look the same. I think that's\nthe better option anyway so that there is a single exit path, only.\n\nAlso shows that we didn't store the error into `iter->err` in the reflog\ncase. Not a huge issue because callers aren't supposed to continue\niterating after they've hit an error anyway. But still preferable to\nmake errors persistent.\n\n> > +\t\titer->err = reftable_iterator_next_ref(&iter->iter, &iter->ref);\n> > +\t\tif (iter->err)\n> > +\t\t\tbreak;\n> > +\n> > +\t\t/*\n> > +\t\t * The files backend only lists references contained in\n> > +\t\t * \"refs/\". We emulate the same behaviour here and thus skip\n> > +\t\t * all references that don't start with this prefix.\n> > +\t\t */\n> > +\t\tif (!starts_with(iter->ref.refname, \"refs/\"))\n> > +\t\t\tcontinue;\n> > +\n> \n> Since my patch series [1] to print all refs is now merged to next, maybe\n> you could add this in?\n> \n> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> index 895de0b273..3f4f905292 100644\n> --- a/refs/reftable-backend.c\n> +++ b/refs/reftable-backend.c\n> @@ -348,11 +348,10 @@ static int\n> reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> \t\t\tbreak;\n> \n> \t\t/*\n> -\t\t * The files backend only lists references contained in\n> -\t\t * \"refs/\". We emulate the same behaviour here and thus skip\n> -\t\t * all references that don't start with this prefix.\n> +\t\t * Unless the `DO_FOR_EACH_INCLUDE_ALL_REFS` flag is use, we only\n> +\t\t * list references contained in \"refs/\" to mimic the file-backend.\n> \t\t */\n> -\t\tif (!starts_with(iter->ref.refname, \"refs/\"))\n> +\t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_ALL_REFS) &&\n> !starts_with(iter->ref.refname, \"refs/\"))\n> \t\t\tcontinue;\n> \n> \t\tif (iter->prefix &&\n\nI'm still waiting for your patch series to hit `next` before making it a\ndependency of this series, but thanks for the patch!\n\n> > +static enum iterator_selection iterator_select(struct ref_iterator *iter_worktree,\n> > +\t\t\t\t\t struct ref_iterator *iter_common,\n> > +\t\t\t\t\t void *cb_data UNUSED)\n> > +{\n> > +\tif (iter_worktree && !iter_common) {\n> > +\t\t/*\n> > +\t\t * Return the worktree ref if there are no more common refs.\n> > +\t\t */\n> > +\t\treturn ITER_SELECT_0;\n> > +\t} else if (iter_common) {\n> > +\t\t/*\n> > +\t\t * In case we have pending worktree and common refs we need to\n> > +\t\t * yield them based on their lexicographical order. Worktree\n> > +\t\t * refs that have the same name as common refs shadow the\n> > +\t\t * latter.\n> > +\t\t */\n> > +\t\tif (iter_worktree) {\n> > +\t\t\tint cmp = strcmp(iter_worktree->refname,\n> > +\t\t\t\t\t iter_common->refname);\n> > +\t\t\tif (cmp < 0)\n> > +\t\t\t\treturn ITER_SELECT_0;\n> > +\t\t\telse if (!cmp)\n> > +\t\t\t\treturn ITER_SELECT_0_SKIP_1;\n> > +\t\t}\n> > +\n> > +\t\t /*\n> > +\t\t * Otherwise, if we either have no worktree refs anymore or if\n> > +\t\t * the common ref sorts before the next worktree ref, we need\n> > +\t\t * to figure out whether the next common ref belongs to the\n> > +\t\t * main worktree. In that case, it should be ignored.\n> > +\t\t */\n> > +\t\tif (parse_worktree_ref(iter_common->refname, NULL, NULL,\n> > +\t\t\t\t NULL) == REF_WORKTREE_SHARED)\n> > +\t\t\treturn ITER_SELECT_1;\n> > +\n> \n> I'm not sure I understand this. When would this situation occur?\n\nThere are two cases here:\n\n - The worktree iterator is exhausted because there are common refs\n which sort lexicographically after all worktree refs.\n\n - Both iterators still have refs, and the next lexicographic ref is\n from the common ref store.\n\nIn both case we want to return only those refs which are shared, but we\nwant to skip over all per-worktree refs from the common ref store. This\nis because the per-worktree refs from the common ref store belong to the\nmain worktree, not to ours.\n\nI'll rework the comments a bit.\n\n> > +\t\treturn ITER_SKIP_1;\n> > +\t} else {\n> > +\t\treturn ITER_DONE;\n> > +\t}\n> > +}\n> > +\n> > +\n> > +static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n> > +\t\t\t\t\t struct ref_transaction *transaction,\n> > +\t\t\t\t\t struct strbuf *err)\n> > +{\n> > +\tstruct reftable_ref_store *refs =\n> > +\t\treftable_be_downcast(ref_store, REF_STORE_WRITE|REF_STORE_MAIN, \"ref_transaction_prepare\");\n> > +\tstruct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT;\n> > +\tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n> > +\tstruct reftable_transaction_data *tx_data = NULL;\n> > +\tstruct object_id head_oid;\n> > +\tunsigned int head_type = 0;\n> > +\tsize_t i;\n> > +\tint ret;\n> > +\n> > +\tret = refs->err;\n> > +\tif (ret < 0)\n> > +\t\tgoto done;\n> > +\n> > +\ttx_data = xcalloc(1, sizeof(*tx_data));\n> > +\n> > +\t/*\n> > +\t * Preprocess all updates. For one we check that there are no duplicate\n> > +\t * reference updates in this transaction. Second, we lock all stacks\n> > +\t * that will be modified during the transaction.\n> > +\t */\n> > +\tfor (i = 0; i < transaction->nr; i++) {\n> > +\t\tret = prepare_transaction_update(NULL, refs, tx_data,\n> > +\t\t\t\t\t\t transaction->updates[i], err);\n> > +\t\tif (ret)\n> > +\t\t\tgoto done;\n> > +\n> > +\t\tstring_list_append(&affected_refnames,\n> > +\t\t\t\t transaction->updates[i]->refname);\n> > +\t}\n> > +\n> > +\t/*\n> > +\t * Now that we have counted updates per stack we can preallocate their\n> > +\t * arrays. This avoids having to reallocate many times.\n> > +\t */\n> > +\tfor (i = 0; i < tx_data->args_nr; i++) {\n> > +\t\tCALLOC_ARRAY(tx_data->args[i].updates, tx_data->args[i].updates_expected);\n> > +\t\ttx_data->args[i].updates_alloc = tx_data->args[i].updates_expected;\n> > +\t}\n> > +\n> > +\t/*\n> > +\t * Fail if a refname appears more than once in the transaction.\n> > +\t * This code is taken from the files backend and is a good candidate to\n> > +\t * be moved into the generic layer.\n> > +\t */\n> > +\tstring_list_sort(&affected_refnames);\n> > +\tif (ref_update_reject_duplicates(&affected_refnames, err)) {\n> > +\t\tret = TRANSACTION_GENERIC_ERROR;\n> > +\t\tgoto done;\n> > +\t}\n> > +\n> > +\tret = read_ref_without_reload(stack_for(refs, \"HEAD\", NULL), \"HEAD\", &head_oid,\n> > +\t\t\t\t &head_referent, &head_type);\n> > +\tif (ret < 0)\n> > +\t\tgoto done;\n> > +\n> > +\tfor (i = 0; i < transaction->nr; i++) {\n> > +\t\tstruct ref_update *u = transaction->updates[i];\n> > +\t\tstruct object_id current_oid = {0};\n> > +\t\tstruct reftable_stack *stack;\n> > +\t\tconst char *rewritten_ref;\n> > +\n> > +\t\tstack = stack_for(refs, u->refname, &rewritten_ref);\n> >\n> \n> Wondering why we didn't just iterate over `tx_data.args` and\n> `args[i].updates` since there we already have the required stack.\n> \n> Update: we don't save rewritten_ref in tx_data, so that's one blocker\n> for this.\n\nRight, that's the reason. I was trying to do that, but also ran into\nsome other problems caused by the ref splitting we do in the loop.\n\n> > +\n> > +\t\tif (u->type & REF_ISSYMREF) {\n> > +\t\t\tconst char *resolved = refs_resolve_ref_unsafe(&refs->base, u->refname, 0,\n> > +\t\t\t\t\t\t\t\t ¤t_oid, NULL);\n> > +\n> > +\t\t\tif (u->flags & REF_NO_DEREF) {\n> > +\t\t\t\t/*\n> > +\t\t\t\t * The reftable stack is locked at this point\n> > +\t\t\t\t * already, so it should be safe to call\n> > +\t\t\t\t * `refs_resolve_ref_unsafe()` here.\n> > +\t\t\t\t */\n> \n> Shouldn't this comment be a few lines before?\n\nGood catch. I moved the `refs_resolve_ref_unsafe()` call at one point\nand forgot to also move the comment.\n\n> > +static int write_transaction_table(struct reftable_writer *writer, void *cb_data)\n> > +{\n> > +\tstruct write_transaction_table_arg *arg = cb_data;\n> > +\tstruct reftable_merged_table *mt =\n> > +\t\treftable_stack_merged_table(arg->stack);\n> > +\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n> > +\tstruct reftable_log_record *logs = NULL;\n> > +\tsize_t logs_nr = 0, logs_alloc = 0, i;\n> > +\tint ret = 0;\n> > +\n> > +\tQSORT(arg->updates, arg->updates_nr, transaction_update_cmp);\n> > +\n> > +\treftable_writer_set_limits(writer, ts, ts);\n> > +\n> > +\tfor (i = 0; i < arg->updates_nr; i++) {\n> > +\t\tstruct reftable_transaction_update *tx_update = &arg->updates[i];\n> > +\t\tstruct ref_update *u = tx_update->update;\n> > +\n> > +\t\t/*\n> > +\t\t * Write a reflog entry when updating a ref to point to\n> > +\t\t * something new in either of the following cases:\n> > +\t\t *\n> > +\t\t * - The reference is about to be deleted. We always want to\n> > +\t\t * delete the reflog in that case.\n> > +\t\t * - REF_FORCE_CREATE_REFLOG is set, asking us to always create\n> > +\t\t * the reflog entry.\n> > +\t\t * - `core.logAllRefUpdates` tells us to create the reflog for\n> > +\t\t * the given ref.\n> > +\t\t */\n> > +\t\tif (u->flags & REF_HAVE_NEW && !(u->type & REF_ISSYMREF) && is_null_oid(&u->new_oid)) {\n> > +\t\t\tstruct reftable_log_record log = {0};\n> > +\t\t\tstruct reftable_iterator it = {0};\n> > +\n> > +\t\t\t/*\n> > +\t\t\t * When deleting refs we also delete all reflog entries\n> > +\t\t\t * with them. While it is not strictly required to\n> > +\t\t\t * delete reflogs together with their refs, this\n> > +\t\t\t * matches the behaviour of the files backend.\n> > +\t\t\t *\n> > +\t\t\t * Unfortunately, we have no better way than to delete\n> > +\t\t\t * all reflog entries one by one.\n> > +\t\t\t */\n> > +\t\t\tret = reftable_merged_table_seek_log(mt, &it, u->refname);\n> > +\t\t\twhile (ret == 0) {\n> > +\t\t\t\tstruct reftable_log_record *tombstone;\n> > +\n> > +\t\t\t\tret = reftable_iterator_next_log(&it, &log);\n> > +\t\t\t\tif (ret < 0)\n> > +\t\t\t\t\tbreak;\n> > +\t\t\t\tif (ret > 0 || strcmp(log.refname, u->refname)) {\n> > +\t\t\t\t\tret = 0;\n> > +\t\t\t\t\tbreak;\n> > +\t\t\t\t}\n> > +\n> > +\t\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n> > +\t\t\t\ttombstone = &logs[logs_nr++];\n> > +\t\t\t\ttombstone->refname = xstrdup(u->refname);\n> > +\t\t\t\ttombstone->value_type = REFTABLE_LOG_DELETION,\n> \n> Why is this a comma? one other such instance in this file.\n\nTypo :) Funny enough this is fine and doesn't make any difference. The\ncomma operator will simply discard the value of the first statement\n(which is the assigned value) and instead return the value of the second\nvalue. We don't use either though, so no problem.\n\nI've also fixed the second instance.\n\n[snip]\n> > +\t/*\n> > +\t * When deleting the old branch we need to create a reflog entry on the\n> > +\t * new branch name that indicates that the old branch has been deleted\n> > +\t * and then recreated. This is a tad weird, but matches what the files\n> > +\t * backend does.\n> > +\t */\n> > +\tif (arg->delete_old) {\n> > +\t\tstruct strbuf head_referent = STRBUF_INIT;\n> > +\t\tstruct object_id head_oid;\n> > +\t\tint append_head_reflog;\n> > +\t\tunsigned head_type = 0;\n> > +\n> > +\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n> > +\t\tmemset(&logs[logs_nr], 0, sizeof(logs[logs_nr]));\n> > +\t\tfill_reftable_log_record(&logs[logs_nr]);\n> > +\t\tlogs[logs_nr].refname = (char *)arg->newname;\n> > +\t\tlogs[logs_nr].update_index = deletion_ts;\n> > +\t\tlogs[logs_nr].value.update.message =\n> > +\t\t\txstrndup(arg->logmsg, arg->refs->write_options.block_size / 2);\n> \n> Question: here and other places in this file, shouldn't we free memory\n> allocated by `xstrndup`?\n\nWe use `reftable_log_record_release()`, which takes care of this.\n\n> Also, why is it `block_size / 2` ?\n\nThat's a very good question indeed. The problem we have here is that the\nmaximum length of a record is limited by the block size. Thus, we cannot\nwrite entries into a block that exceed a certain length and we must\nensure that no record exceeds that size.\n\nUsing `block_size / 2` is thus a mechanism to cull the reflog message so\nthat we can at least write a _partial_ reflog without breaking the\nformat. The division by two is kind of arbitrary here and ensures that\nwe have enough headroom to also write the remaining data of the record.\n\nSo this could be improved to instead trim at the exact block boundary\nminus the overhead of the remaining data. But I dobut that's really\nworth it for now given that nobody will ever write a 2kB reflog entry\n(famous last words).\n\n> > +static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftable_ref_store *refs,\n> > +\t\t\t\t\t\t\t\t struct reftable_stack *stack)\n> > +{\n> > +\tstruct reftable_reflog_iterator *iter;\n> > +\tstruct reftable_merged_table *mt;\n> > +\tint ret;\n> > +\n> > +\titer = xcalloc(1, sizeof(*iter));\n> > +\tbase_ref_iterator_init(&iter->base, &reftable_reflog_iterator_vtable, 1);\n> > +\titer->refs = refs;\n> > +\titer->base.oid = &iter->oid;\n> > +\n> > +\tret = reftable_stack_reload(refs->main_stack);\n> > +\tif (ret < 0)\n> > +\t\tgoto done;\n> > +\n> > +\tmt = reftable_stack_merged_table(stack);\n> > +\tret = reftable_merged_table_seek_log(mt, &iter->iter, \"\");\n> > +\tif (ret < 0)\n> > +\t\tgoto done;\n> > +\n> \n> Keeping it similar to `ref_iterator_for_stack`, perhaps:\n> \n> ret = refs->err;\n> \tif (ret)\n> \t\tgoto done;\n> \n> \tret = reftable_stack_reload(stack);\n> \tif (ret)\n> \t\tgoto done;\n> \n> \tmerged_table = reftable_stack_merged_table(stack);\n> \n> \tret = reftable_merged_table_seek_ref(merged_table, &iter->iter, prefix);\n> \tif (ret)\n> \t\tgoto done;\n\nWill do.\n\n> > +static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n> > +\t\t\t\t\t\t const char *refname,\n> > +\t\t\t\t\t\t each_reflog_ent_fn fn,\n> > +\t\t\t\t\t\t void *cb_data)\n> > +{\n> > +\tstruct reftable_ref_store *refs =\n> > +\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent_reverse\");\n> > +\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n> > +\tstruct reftable_merged_table *mt = NULL;\n> > +\tstruct reftable_log_record log = {0};\n> > +\tstruct reftable_iterator it = {0};\n> > +\tint ret;\n> > +\n> > +\tif (refs->err < 0)\n> > +\t\treturn refs->err;\n> \n> Nit: seems like this function and the one below\n> `reftable_be_for_each_reflog_ent`, share the same code apart from\n> iteration direction. Wonder if it'd be nicer to extract out the common\n> code from them.\n\nHm. I tried to do this once a while ago, but didn't end up with a nice\nsolution because I basically tried to unify the iterator loops, which\ndidn't quite work out in a nice way. But coming back with fresh eyes we\ncan at least easily share the code that yields the log record to the\ncallback function to remove some code duplication.\n\n> > +static int reftable_be_reflog_exists(struct ref_store *ref_store,\n> > +\t\t\t\t const char *refname)\n> > +{\n> > +\tstruct reftable_ref_store *refs =\n> > +\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_exists\");\n> > +\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n> > +\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n> > +\tstruct reftable_log_record log = {0};\n> > +\tstruct reftable_iterator it = {0};\n> > +\tint ret;\n> > +\n> > +\tret = refs->err;\n> > +\tif (ret < 0)\n> > +\t\tgoto done;\n> > +\n> > +\tret = reftable_stack_reload(stack);\n> > +\tif (ret)\n> > +\t\tgoto done;\n> > +\n> > +\tret = reftable_merged_table_seek_log(mt, &it, refname);\n> > +\tif (ret)\n> > +\t\tgoto done;\n> > +\n> > +\t/*\n> > +\t * Seek the reflog to see whether it contains any reflog entries which\n> > +\t * aren't marked for deletion.\n> > +\t */\n> \n> Shouldn't we be checking `log.value_type` in that case?\n\nI think this whole loop doesn't make any sense anymore. I initially\nmisunderstood how reflog deletion works [1]: I assumed that a reflog\nentry with the zero object ID as new value is supposed to mark a reflog\nas deleted, but that's not the case. Instead, deleting a reflog requires\nus to mark each record as deleted individually.\n\nGiven that the reftable stack is marked to suppress deletions (see\n`reftable_stack_reload_once()`, `new_merged->suppress_deletions = 1`),\nwe don't have to check for deletions at all. All we have to do is seek\nthe log once and check whether we get a log record matching the expected\nname.\n\n[1]: https://public-inbox.org/git/ZVy0zKcmc8tjmgzs@tanuki/\n\n> This email only reviews this file, I'll review the others in follow up\n> email[s].\n\nThanks a lot for your review! I know the code here is quite dense, so I\nreally appreciate it.\n\nPatrick\n"},{"id":"487794","messageId":"cover.1706862705.git.ps@pks.im","threadId":"60815","inReplyTo":"cover.1706601199.git.ps@pks.im","subject":"[PATCH v2 0/2] refs: introduce reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-02T08:38:25Z","receivedAt":"2024-02-02T08:38:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the second version of my patch series that introduces the\nreftable backend.\n\nThis version addresses the feedback by Karthik. I don't think it really\nmakes sense to spell out every change here -- the range diff should\nlikely be easier to digest.\n\nThanks!\n\nPatrick\n\nPatrick Steinhardt (2):\n refs: introduce reftable backend\n ci: add jobs to test with the reftable backend\n\n .github/workflows/main.yml | 9 +\n .gitlab-ci.yml | 9 +\n Documentation/ref-storage-format.txt | 2 +\n .../technical/repository-version.txt | 5 +-\n Makefile | 1 +\n ci/lib.sh | 2 +-\n ci/run-build-and-tests.sh | 3 +\n contrib/workdir/git-new-workdir | 2 +-\n path.c | 2 +-\n path.h | 1 +\n refs.c | 1 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 2295 +++++++++++++++++\n repository.h | 5 +-\n t/t0610-reftable-basics.sh | 887 +++++++\n t/t0611-reftable-httpd.sh | 26 +\n t/test-lib.sh | 2 +\n 17 files changed, 3246 insertions(+), 7 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100755 t/t0610-reftable-basics.sh\n create mode 100755 t/t0611-reftable-httpd.sh\n\nRange-diff against v1:\n1: 5598cd1307 ! 1: d6548dcfad refs: introduce reftable backend\n @@ Commit message\n previously been deleting reflogs together with their refs to avoid\n file/directory conflicts, which is not necessary anymore.\n \n + - We can properly enumerate all refs. With the \"files\" backend it is\n + not easily possible to distinguish between refs and non-refs because\n + they may live side by side in the gitdir.\n +\n Not all of these improvements are realized with the current \"reftable\"\n backend implementation. At this point, the new backend is supposed to be\n a drop-in replacement for the \"files\" backend that is used by basically\n @@ Commit message\n workloads we would likely also want to perform pack loose refs,\n which would likely change the picture.\n \n - Benchmark 1: update-ref: create refs sequentially (refformat = files)\n + Benchmark 1: update-ref: create refs sequentially (refformat = files, refcount = 1)\n Time (mean ± σ): 2.1 ms ± 0.3 ms [User: 0.6 ms, System: 1.7 ms]\n Range (min … max): 1.8 ms … 4.3 ms 133 runs\n \n - Benchmark 2: update-ref: create refs sequentially (refformat = reftable)\n + Benchmark 2: update-ref: create refs sequentially (refformat = reftable, refcount = 1)\n Time (mean ± σ): 2.7 ms ± 0.1 ms [User: 0.6 ms, System: 2.2 ms]\n Range (min … max): 2.4 ms … 2.9 ms 132 runs\n \n - Benchmark 3: update-ref: create refs sequentially (refformat = files)\n + Benchmark 3: update-ref: create refs sequentially (refformat = files, refcount = 1000)\n Time (mean ± σ): 1.975 s ± 0.006 s [User: 0.437 s, System: 1.535 s]\n Range (min … max): 1.969 s … 1.980 s 3 runs\n \n - Benchmark 4: update-ref: create refs sequentially (refformat = reftable)\n + Benchmark 4: update-ref: create refs sequentially (refformat = reftable, refcount = 1000)\n Time (mean ± σ): 2.611 s ± 0.013 s [User: 0.782 s, System: 1.825 s]\n Range (min … max): 2.597 s … 2.622 s 3 runs\n \n - Benchmark 5: update-ref: create refs sequentially (refformat = files)\n + Benchmark 5: update-ref: create refs sequentially (refformat = files, refcount = 100000)\n Time (mean ± σ): 198.442 s ± 0.241 s [User: 43.051 s, System: 155.250 s]\n Range (min … max): 198.189 s … 198.670 s 3 runs\n \n - Benchmark 6: update-ref: create refs sequentially (refformat = reftable)\n + Benchmark 6: update-ref: create refs sequentially (refformat = reftable, refcount = 100000)\n Time (mean ± σ): 294.509 s ± 4.269 s [User: 104.046 s, System: 190.326 s]\n Range (min … max): 290.223 s … 298.761 s 3 runs\n \n @@ refs/reftable-backend.c (new)\n +#include \"../git-compat-util.h\"\n +#include \"../abspath.h\"\n +#include \"../chdir-notify.h\"\n -+#include \"../config.h\"\n +#include \"../environment.h\"\n +#include \"../gettext.h\"\n +#include \"../hash.h\"\n @@ refs/reftable-backend.c (new)\n +#include \"../reftable/reftable-stack.h\"\n +#include \"../reftable/reftable-record.h\"\n +#include \"../reftable/reftable-error.h\"\n -+#include \"../reftable/reftable-blocksource.h\"\n -+#include \"../reftable/reftable-reader.h\"\n +#include \"../reftable/reftable-iterator.h\"\n +#include \"../reftable/reftable-merged.h\"\n -+#include \"../reftable/reftable-generic.h\"\n +#include \"../setup.h\"\n +#include \"../strmap.h\"\n -+#include \"../worktree.h\"\n +#include \"refs-internal.h\"\n +\n +/*\n @@ refs/reftable-backend.c (new)\n +struct reftable_ref_store {\n +\tstruct ref_store base;\n +\n ++\t/*\n ++\t * The main stack refers to the common dir and thus contains common\n ++\t * refs as well as refs of the main repository.\n ++\t */\n +\tstruct reftable_stack *main_stack;\n ++\t/*\n ++\t * The worktree stack refers to the gitdir in case the refdb is opened\n ++\t * via a worktree. It thus contains the per-worktree refs.\n ++\t */\n +\tstruct reftable_stack *worktree_stack;\n ++\t/*\n ++\t * Map of worktree stacks by their respective worktree names. The map\n ++\t * is populated lazily when we try to resolve `worktrees/$worktree` refs.\n ++\t */\n +\tstruct strmap worktree_stacks;\n +\tstruct reftable_write_options write_options;\n +\n @@ refs/reftable-backend.c (new)\n +\t/*\n +\t * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.\n +\t * This stack contains both the shared and the main worktree refs.\n ++\t *\n ++\t * Note that we don't try to resolve the path in case we have a\n ++\t * worktree because `get_common_dir_noenv()` already does it for us.\n +\t */\n +\tis_worktree = get_common_dir_noenv(&path, gitdir);\n +\tif (!is_worktree) {\n @@ refs/reftable-backend.c (new)\n +\t/*\n +\t * If we're in a worktree we also need to set up the worktree reftable\n +\t * stack that is contained in the per-worktree GIT_DIR.\n ++\t *\n ++\t * Ideally, we would also add the stack to our worktree stack map. But\n ++\t * we have no way to figure out the worktree name here and thus can't\n ++\t * do it efficiently.\n +\t */\n +\tif (is_worktree) {\n +\t\tstrbuf_reset(&path);\n @@ refs/reftable-backend.c (new)\n +\t\t}\n +\n +\t\t /*\n -+\t\t * Otherwise, if we either have no worktree refs anymore or if\n -+\t\t * the common ref sorts before the next worktree ref, we need\n -+\t\t * to figure out whether the next common ref belongs to the\n -+\t\t * main worktree. In that case, it should be ignored.\n ++\t\t * We now know that the lexicographically-next ref is a common\n ++\t\t * ref. When the common ref is a shared one we return it.\n +\t\t */\n +\t\tif (parse_worktree_ref(iter_common->refname, NULL, NULL,\n +\t\t\t\t NULL) == REF_WORKTREE_SHARED)\n +\t\t\treturn ITER_SELECT_1;\n +\n ++\t\t/*\n ++\t\t * Otherwise, if the common ref is a per-worktree ref we skip\n ++\t\t * it because it would belong to the main worktree, not ours.\n ++\t\t */\n +\t\treturn ITER_SKIP_1;\n +\t} else {\n +\t\treturn ITER_DONE;\n @@ refs/reftable-backend.c (new)\n +\t\t}\n +\n +\t\tif (u->type & REF_ISSYMREF) {\n ++\t\t\t/*\n ++\t\t\t * The reftable stack is locked at this point already,\n ++\t\t\t * so it is safe to call `refs_resolve_ref_unsafe()`\n ++\t\t\t * here without causing races.\n ++\t\t\t */\n +\t\t\tconst char *resolved = refs_resolve_ref_unsafe(&refs->base, u->refname, 0,\n +\t\t\t\t\t\t\t\t ¤t_oid, NULL);\n +\n +\t\t\tif (u->flags & REF_NO_DEREF) {\n -+\t\t\t\t/*\n -+\t\t\t\t * The reftable stack is locked at this point\n -+\t\t\t\t * already, so it should be safe to call\n -+\t\t\t\t * `refs_resolve_ref_unsafe()` here.\n -+\t\t\t\t */\n +\t\t\t\tif (u->flags & REF_HAVE_OLD && !resolved) {\n +\t\t\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n +\t\t\t\t\t\t \"error reading reference\"), u->refname);\n @@ refs/reftable-backend.c (new)\n +\t\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n +\t\t\t\ttombstone = &logs[logs_nr++];\n +\t\t\t\ttombstone->refname = xstrdup(u->refname);\n -+\t\t\t\ttombstone->value_type = REFTABLE_LOG_DELETION,\n ++\t\t\t\ttombstone->value_type = REFTABLE_LOG_DELETION;\n +\t\t\t\ttombstone->update_index = log.update_index;\n +\t\t\t}\n +\n @@ refs/reftable-backend.c (new)\n +\n +\t/*\n +\t * When deleting the old reference we have to use two update indices:\n -+\t * one to delete the old ref and its reflog, and once to create the new\n -+\t * ref and its reflog. They need to be staged with two separate indices\n -+\t * because the new reflog needs to encode both the deletion of the old\n -+\t * branch and the creation of the new branch, and we cannot do two\n -+\t * changes to a reflog in a single update.\n ++\t * once to delete the old ref and its reflog, and once to create the\n ++\t * new ref and its reflog. They need to be staged with two separate\n ++\t * indices because the new reflog needs to encode both the deletion of\n ++\t * the old branch and the creation of the new branch, and we cannot do\n ++\t * two changes to a reflog in a single update.\n +\t */\n +\tdeletion_ts = creation_ts = reftable_stack_next_update_index(arg->stack);\n +\tif (arg->delete_old)\n @@ refs/reftable-backend.c (new)\n +\tstruct reftable_reflog_iterator *iter =\n +\t\t(struct reftable_reflog_iterator *)ref_iterator;\n +\n -+\tif (iter->err) {\n -+\t\tref_iterator_abort(ref_iterator);\n -+\t\treturn iter->err;\n -+\t}\n -+\n -+\twhile (1) {\n -+\t\tint flags, ret;\n ++\twhile (!iter->err) {\n ++\t\tint flags;\n +\n -+\t\tret = reftable_iterator_next_log(&iter->iter, &iter->log);\n -+\t\tif (ret < 0) {\n -+\t\t\tref_iterator_abort(ref_iterator);\n -+\t\t\treturn ITER_ERROR;\n -+\t\t}\n -+\t\tif (ret > 0) {\n -+\t\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n -+\t\t\t\treturn ITER_ERROR;\n -+\t\t\treturn ITER_DONE;\n -+\t\t}\n ++\t\titer->err = reftable_iterator_next_log(&iter->iter, &iter->log);\n ++\t\tif (iter->err)\n ++\t\t\tbreak;\n +\n +\t\t/*\n +\t\t * We want the refnames that we have reflogs for, so we skip if\n @@ refs/reftable-backend.c (new)\n +\t\titer->base.refname = iter->log.refname;\n +\t\titer->base.oid = &iter->oid;\n +\t\titer->base.flags = flags;\n -+\t\treturn ITER_OK;\n ++\n ++\t\tbreak;\n +\t}\n ++\n ++\tif (iter->err > 0) {\n ++\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n ++\t\t\treturn ITER_ERROR;\n ++\t\treturn ITER_DONE;\n ++\t}\n ++\n ++\tif (iter->err < 0) {\n ++\t\tref_iterator_abort(ref_iterator);\n ++\t\treturn ITER_ERROR;\n ++\t}\n ++\n ++\treturn ITER_OK;\n +}\n +\n +static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator,\n @@ refs/reftable-backend.c (new)\n +static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftable_ref_store *refs,\n +\t\t\t\t\t\t\t\t struct reftable_stack *stack)\n +{\n ++\tstruct reftable_merged_table *merged_table;\n +\tstruct reftable_reflog_iterator *iter;\n -+\tstruct reftable_merged_table *mt;\n +\tint ret;\n +\n +\titer = xcalloc(1, sizeof(*iter));\n @@ refs/reftable-backend.c (new)\n +\titer->refs = refs;\n +\titer->base.oid = &iter->oid;\n +\n ++\tret = refs->err;\n ++\tif (ret)\n ++\t\tgoto done;\n ++\n +\tret = reftable_stack_reload(refs->main_stack);\n +\tif (ret < 0)\n +\t\tgoto done;\n +\n -+\tmt = reftable_stack_merged_table(stack);\n -+\tret = reftable_merged_table_seek_log(mt, &iter->iter, \"\");\n ++\tmerged_table = reftable_stack_merged_table(stack);\n ++\n ++\tret = reftable_merged_table_seek_log(merged_table, &iter->iter, \"\");\n +\tif (ret < 0)\n +\t\tgoto done;\n +\n @@ refs/reftable-backend.c (new)\n +\t\t\t\t\titerator_select, NULL);\n +}\n +\n ++static int yield_log_record(struct reftable_log_record *log,\n ++\t\t\t each_reflog_ent_fn fn,\n ++\t\t\t void *cb_data)\n ++{\n ++\tstruct object_id old_oid, new_oid;\n ++\tconst char *full_committer;\n ++\n ++\toidread(&old_oid, log->value.update.old_hash);\n ++\toidread(&new_oid, log->value.update.new_hash);\n ++\n ++\t/*\n ++\t * When both the old object ID and the new object ID are null\n ++\t * then this is the reflog existence marker. The caller must\n ++\t * not be aware of it.\n ++\t */\n ++\tif (is_null_oid(&old_oid) && is_null_oid(&new_oid))\n ++\t\treturn 0;\n ++\n ++\tfull_committer = fmt_ident(log->value.update.name, log->value.update.email,\n ++\t\t\t\t WANT_COMMITTER_IDENT, NULL, IDENT_NO_DATE);\n ++\treturn fn(&old_oid, &new_oid, full_committer,\n ++\t\t log->value.update.time, log->value.update.tz_offset,\n ++\t\t log->value.update.message, cb_data);\n ++}\n ++\n +static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n +\t\t\t\t\t\t const char *refname,\n +\t\t\t\t\t\t each_reflog_ent_fn fn,\n @@ refs/reftable-backend.c (new)\n +\tmt = reftable_stack_merged_table(stack);\n +\tret = reftable_merged_table_seek_log(mt, &it, refname);\n +\twhile (!ret) {\n -+\t\tstruct object_id old_oid, new_oid;\n -+\t\tconst char *full_committer;\n -+\n +\t\tret = reftable_iterator_next_log(&it, &log);\n +\t\tif (ret < 0)\n +\t\t\tbreak;\n @@ refs/reftable-backend.c (new)\n +\t\t\tbreak;\n +\t\t}\n +\n -+\t\toidread(&old_oid, log.value.update.old_hash);\n -+\t\toidread(&new_oid, log.value.update.new_hash);\n -+\n -+\t\t/*\n -+\t\t * When both the old object ID and the new object ID are null\n -+\t\t * then this is the reflog existence marker. The caller must\n -+\t\t * not be aware of it.\n -+\t\t */\n -+\t\tif (is_null_oid(&old_oid) && is_null_oid(&new_oid))\n -+\t\t\tcontinue;\n -+\n -+\t\tfull_committer = fmt_ident(log.value.update.name, log.value.update.email,\n -+\t\t\t\t\t WANT_COMMITTER_IDENT, NULL, IDENT_NO_DATE);\n -+\t\tret = fn(&old_oid, &new_oid, full_committer,\n -+\t\t\t log.value.update.time, log.value.update.tz_offset,\n -+\t\t\t log.value.update.message, cb_data);\n ++\t\tret = yield_log_record(&log, fn, cb_data);\n +\t\tif (ret)\n +\t\t\tbreak;\n +\t}\n @@ refs/reftable-backend.c (new)\n +\n +\t\tret = reftable_iterator_next_log(&it, &log);\n +\t\tif (ret < 0)\n -+\t\t\tbreak;\n ++\t\t\tgoto done;\n +\t\tif (ret > 0 || strcmp(log.refname, refname)) {\n +\t\t\treftable_log_record_release(&log);\n +\t\t\tret = 0;\n @@ refs/reftable-backend.c (new)\n +\t}\n +\n +\tfor (i = logs_nr; i--;) {\n -+\t\tstruct reftable_log_record *log = &logs[i];\n -+\t\tstruct object_id old_oid, new_oid;\n -+\t\tconst char *full_committer = \"\";\n -+\n -+\t\toidread(&old_oid, log->value.update.old_hash);\n -+\t\toidread(&new_oid, log->value.update.new_hash);\n -+\n -+\t\t/*\n -+\t\t * When both the old object ID and the new object ID are null\n -+\t\t * then this is the reflog existence marker. The caller must\n -+\t\t * not be aware of it.\n -+\t\t */\n -+\t\tif (is_null_oid(&old_oid) && is_null_oid(&new_oid))\n -+\t\t\tcontinue;\n -+\n -+\t\tfull_committer = fmt_ident(log->value.update.name, log->value.update.email,\n -+\t\t\t\t\t WANT_COMMITTER_IDENT, NULL, IDENT_NO_DATE);\n -+\t\tret = fn(&old_oid, &new_oid, full_committer,\n -+\t\t\t log->value.update.time, log->value.update.tz_offset,\n -+\t\t\t log->value.update.message, cb_data);\n ++\t\tret = yield_log_record(&logs[i], fn, cb_data);\n +\t\tif (ret)\n -+\t\t\tbreak;\n ++\t\t\tgoto done;\n +\t}\n +\n ++done:\n +\treftable_iterator_destroy(&it);\n +\tfor (i = 0; i < logs_nr; i++)\n +\t\treftable_log_record_release(&logs[i]);\n @@ refs/reftable-backend.c (new)\n +\t\tgoto done;\n +\n +\tret = reftable_stack_reload(stack);\n -+\tif (ret)\n ++\tif (ret < 0)\n +\t\tgoto done;\n +\n +\tret = reftable_merged_table_seek_log(mt, &it, refname);\n -+\tif (ret)\n ++\tif (ret < 0)\n +\t\tgoto done;\n +\n +\t/*\n -+\t * Seek the reflog to see whether it contains any reflog entries which\n -+\t * aren't marked for deletion.\n ++\t * Check whether we get at least one log record for the given ref name.\n ++\t * If so, the reflog exists, otherwise it doesn't.\n +\t */\n -+\twhile (1) {\n -+\t\tret = reftable_iterator_next_log(&it, &log);\n -+\t\tif (ret < 0)\n -+\t\t\tgoto done;\n -+\t\tif (ret > 0 || strcmp(log.refname, refname)) {\n -+\t\t\tret = 0;\n -+\t\t\tgoto done;\n -+\t\t}\n -+\n -+\t\tret = 1;\n -+\t\tbreak;\n ++\tret = reftable_iterator_next_log(&it, &log);\n ++\tif (ret < 0)\n ++\t\tgoto done;\n ++\tif (ret > 0) {\n ++\t\tret = 0;\n ++\t\tgoto done;\n +\t}\n +\n ++\tret = strcmp(log.refname, refname) == 0;\n ++\n +done:\n +\treftable_iterator_destroy(&it);\n +\treftable_log_record_release(&log);\n @@ refs/reftable-backend.c (new)\n +\t\t\tbreak;\n +\t\t}\n +\n -+\t\ttombstone.refname = (char *)arg->refname,\n -+\t\ttombstone.value_type = REFTABLE_LOG_DELETION,\n ++\t\ttombstone.refname = (char *)arg->refname;\n ++\t\ttombstone.value_type = REFTABLE_LOG_DELETION;\n +\t\ttombstone.update_index = log.update_index;\n +\n +\t\tret = reftable_writer_add_log(writer, &tombstone);\n2: bb694fa132 = 2: 63eafc9f5b ci: add jobs to test with the reftable backend\n-- \n2.43.GIT\n\n"},{"id":"487796","messageId":"d6548dcfadbe2a60a30ae9a3c96f966eeb9ad47a.1706862705.git.ps@pks.im","threadId":"60815","inReplyTo":"cover.1706862705.git.ps@pks.im","subject":"[PATCH v2 1/2] refs: introduce reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-02T08:38:29Z","receivedAt":"2024-02-02T08:38:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Due to scalability issues, Shawn Pearce has originally proposed a new\n\"reftable\" format more than six years ago [1]. Initially, this new\nformat was implemented in JGit with promising results. Around two years\nago, we have then added the \"reftable\" library to the Git codebase via\na4bbd13be3 (Merge branch 'hn/reftable', 2021-12-15). With this we have\nlanded all the low-level code to read and write reftables. Notably\nmissing though was the integration of this low-level code into the Git\ncode base in the form of a new ref backend that ties all of this\ntogether.\n\nThis gap is now finally closed by introducing a new \"reftable\" backend\ninto the Git codebase. This new backend promises to bring some notable\nimprovements to Git repositories:\n\n - It becomes possible to do truly atomic writes where either all refs\n are committed to disk or none are. This was not possible with the\n \"files\" backend because ref updates were split across multiple loose\n files.\n\n - The disk space required to store many refs is reduced, both compared\n to loose refs and packed-refs. This is enabled both by the reftable\n format being a binary format, which is more compact, and by prefix\n compression.\n\n - We can ignore filesystem-specific behaviour as ref names are not\n encoded via paths anymore. This means there is no need to handle\n case sensitivity on Windows systems or Unicode precomposition on\n macOS.\n\n - There is no need to rewrite the complete refdb anymore every time a\n ref is being deleted like it was the case for packed-refs. This\n means that ref deletions are now constant time instead of scaling\n linearly with the number of refs.\n\n - We can ignore file/directory conflicts so that it becomes possible\n to store both \"refs/heads/foo\" and \"refs/heads/foo/bar\".\n\n - Due to this property we can retain reflogs for deleted refs. We have\n previously been deleting reflogs together with their refs to avoid\n file/directory conflicts, which is not necessary anymore.\n\n - We can properly enumerate all refs. With the \"files\" backend it is\n not easily possible to distinguish between refs and non-refs because\n they may live side by side in the gitdir.\n\nNot all of these improvements are realized with the current \"reftable\"\nbackend implementation. At this point, the new backend is supposed to be\na drop-in replacement for the \"files\" backend that is used by basically\nall Git repositories nowadays. It strives for 1:1 compatibility, which\nmeans that a user can expect the same behaviour regardless of whether\nthey use the \"reftable\" backend or the \"files\" backend for most of the\npart.\n\nMost notably, this means we artificially limit the capabilities of the\n\"reftable\" backend to match the limits of the \"files\" backend. It is not\npossible to create refs that would end up with file/directory conflicts,\nwe do not retain reflogs, we perform stricter-than-necessary checks.\nThis is done intentionally due to two main reasons:\n\n - It makes it significantly easier to land the \"reftable\" backend as\n tests behave the same. It would be tough to argue for each and every\n single test that doesn't pass with the \"reftable\" backend.\n\n - It ensures compatibility between repositories that use the \"files\"\n backend and repositories that use the \"reftable\" backend. Like this,\n hosters can migrate their repositories to use the \"reftable\" backend\n without causing issues for clients that use the \"files\" backend in\n their clones.\n\nIt is expected that these artificial limitations may eventually go away\nin the long term.\n\nPerformance-wise things very much depend on the actual workload. The\nfollowing benchmarks compare the \"files\" and \"reftable\" backends in the\ncurrent version:\n\n - Creating N refs in separate transactions shows that the \"files\"\n backend is ~50% faster. This is not surprising given that creating a\n ref only requires us to create a single loose ref. The \"reftable\"\n backend will also perform auto compaction on updates. In real-world\n workloads we would likely also want to perform pack loose refs,\n which would likely change the picture.\n\n Benchmark 1: update-ref: create refs sequentially (refformat = files, refcount = 1)\n Time (mean ± σ): 2.1 ms ± 0.3 ms [User: 0.6 ms, System: 1.7 ms]\n Range (min … max): 1.8 ms … 4.3 ms 133 runs\n\n Benchmark 2: update-ref: create refs sequentially (refformat = reftable, refcount = 1)\n Time (mean ± σ): 2.7 ms ± 0.1 ms [User: 0.6 ms, System: 2.2 ms]\n Range (min … max): 2.4 ms … 2.9 ms 132 runs\n\n Benchmark 3: update-ref: create refs sequentially (refformat = files, refcount = 1000)\n Time (mean ± σ): 1.975 s ± 0.006 s [User: 0.437 s, System: 1.535 s]\n Range (min … max): 1.969 s … 1.980 s 3 runs\n\n Benchmark 4: update-ref: create refs sequentially (refformat = reftable, refcount = 1000)\n Time (mean ± σ): 2.611 s ± 0.013 s [User: 0.782 s, System: 1.825 s]\n Range (min … max): 2.597 s … 2.622 s 3 runs\n\n Benchmark 5: update-ref: create refs sequentially (refformat = files, refcount = 100000)\n Time (mean ± σ): 198.442 s ± 0.241 s [User: 43.051 s, System: 155.250 s]\n Range (min … max): 198.189 s … 198.670 s 3 runs\n\n Benchmark 6: update-ref: create refs sequentially (refformat = reftable, refcount = 100000)\n Time (mean ± σ): 294.509 s ± 4.269 s [User: 104.046 s, System: 190.326 s]\n Range (min … max): 290.223 s … 298.761 s 3 runs\n\n - Creating N refs in a single transaction shows that the \"files\"\n backend is significantly slower once we start to write many refs.\n The \"reftable\" backend only needs to update two files, whereas the\n \"files\" backend needs to write one file per ref.\n\n Benchmark 1: update-ref: create many refs (refformat = files, refcount = 1)\n Time (mean ± σ): 1.9 ms ± 0.1 ms [User: 0.4 ms, System: 1.4 ms]\n Range (min … max): 1.8 ms … 2.6 ms 151 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, refcount = 1)\n Time (mean ± σ): 2.5 ms ± 0.1 ms [User: 0.7 ms, System: 1.7 ms]\n Range (min … max): 2.4 ms … 3.4 ms 148 runs\n\n Benchmark 3: update-ref: create many refs (refformat = files, refcount = 1000)\n Time (mean ± σ): 152.5 ms ± 5.2 ms [User: 19.1 ms, System: 133.1 ms]\n Range (min … max): 148.5 ms … 167.8 ms 15 runs\n\n Benchmark 4: update-ref: create many refs (refformat = reftable, refcount = 1000)\n Time (mean ± σ): 58.0 ms ± 2.5 ms [User: 28.4 ms, System: 29.4 ms]\n Range (min … max): 56.3 ms … 72.9 ms 40 runs\n\n Benchmark 5: update-ref: create many refs (refformat = files, refcount = 1000000)\n Time (mean ± σ): 152.752 s ± 0.710 s [User: 20.315 s, System: 131.310 s]\n Range (min … max): 152.165 s … 153.542 s 3 runs\n\n Benchmark 6: update-ref: create many refs (refformat = reftable, refcount = 1000000)\n Time (mean ± σ): 51.912 s ± 0.127 s [User: 26.483 s, System: 25.424 s]\n Range (min … max): 51.769 s … 52.012 s 3 runs\n\n - Deleting a ref in a fully-packed repository shows that the \"files\"\n backend scales with the number of refs. The \"reftable\" backend has\n constant-time deletions.\n\n Benchmark 1: update-ref: delete ref (refformat = files, refcount = 1)\n Time (mean ± σ): 1.7 ms ± 0.1 ms [User: 0.4 ms, System: 1.2 ms]\n Range (min … max): 1.6 ms … 2.1 ms 316 runs\n\n Benchmark 2: update-ref: delete ref (refformat = reftable, refcount = 1)\n Time (mean ± σ): 1.8 ms ± 0.1 ms [User: 0.4 ms, System: 1.3 ms]\n Range (min … max): 1.7 ms … 2.1 ms 294 runs\n\n Benchmark 3: update-ref: delete ref (refformat = files, refcount = 1000)\n Time (mean ± σ): 2.0 ms ± 0.1 ms [User: 0.5 ms, System: 1.4 ms]\n Range (min … max): 1.9 ms … 2.5 ms 287 runs\n\n Benchmark 4: update-ref: delete ref (refformat = reftable, refcount = 1000)\n Time (mean ± σ): 1.9 ms ± 0.1 ms [User: 0.5 ms, System: 1.3 ms]\n Range (min … max): 1.8 ms … 2.1 ms 217 runs\n\n Benchmark 5: update-ref: delete ref (refformat = files, refcount = 1000000)\n Time (mean ± σ): 229.8 ms ± 7.9 ms [User: 182.6 ms, System: 46.8 ms]\n Range (min … max): 224.6 ms … 245.2 ms 6 runs\n\n Benchmark 6: update-ref: delete ref (refformat = reftable, refcount = 1000000)\n Time (mean ± σ): 2.0 ms ± 0.0 ms [User: 0.6 ms, System: 1.3 ms]\n Range (min … max): 2.0 ms … 2.1 ms 3 runs\n\n - Listing all refs shows no significant advantage for either of the\n backends. The \"files\" backend is a bit faster, but not by a\n significant margin. When repositories are not packed the \"reftable\"\n backend outperforms the \"files\" backend because the \"reftable\"\n backend performs auto-compaction.\n\n Benchmark 1: show-ref: print all refs (refformat = files, refcount = 1, packed = true)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 2.0 ms 1729 runs\n\n Benchmark 2: show-ref: print all refs (refformat = reftable, refcount = 1, packed = true)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 1.8 ms 1816 runs\n\n Benchmark 3: show-ref: print all refs (refformat = files, refcount = 1000, packed = true)\n Time (mean ± σ): 4.3 ms ± 0.1 ms [User: 0.9 ms, System: 3.3 ms]\n Range (min … max): 4.1 ms … 4.6 ms 645 runs\n\n Benchmark 4: show-ref: print all refs (refformat = reftable, refcount = 1000, packed = true)\n Time (mean ± σ): 4.5 ms ± 0.2 ms [User: 1.0 ms, System: 3.3 ms]\n Range (min … max): 4.2 ms … 5.9 ms 643 runs\n\n Benchmark 5: show-ref: print all refs (refformat = files, refcount = 1000000, packed = true)\n Time (mean ± σ): 2.537 s ± 0.034 s [User: 0.488 s, System: 2.048 s]\n Range (min … max): 2.511 s … 2.627 s 10 runs\n\n Benchmark 6: show-ref: print all refs (refformat = reftable, refcount = 1000000, packed = true)\n Time (mean ± σ): 2.712 s ± 0.017 s [User: 0.653 s, System: 2.059 s]\n Range (min … max): 2.692 s … 2.752 s 10 runs\n\n Benchmark 7: show-ref: print all refs (refformat = files, refcount = 1, packed = false)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 1.9 ms 1834 runs\n\n Benchmark 8: show-ref: print all refs (refformat = reftable, refcount = 1, packed = false)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.4 ms … 2.0 ms 1840 runs\n\n Benchmark 9: show-ref: print all refs (refformat = files, refcount = 1000, packed = false)\n Time (mean ± σ): 13.8 ms ± 0.2 ms [User: 2.8 ms, System: 10.8 ms]\n Range (min … max): 13.3 ms … 14.5 ms 208 runs\n\n Benchmark 10: show-ref: print all refs (refformat = reftable, refcount = 1000, packed = false)\n Time (mean ± σ): 4.5 ms ± 0.2 ms [User: 1.2 ms, System: 3.3 ms]\n Range (min … max): 4.3 ms … 6.2 ms 624 runs\n\n Benchmark 11: show-ref: print all refs (refformat = files, refcount = 1000000, packed = false)\n Time (mean ± σ): 12.127 s ± 0.129 s [User: 2.675 s, System: 9.451 s]\n Range (min … max): 11.965 s … 12.370 s 10 runs\n\n Benchmark 12: show-ref: print all refs (refformat = reftable, refcount = 1000000, packed = false)\n Time (mean ± σ): 2.799 s ± 0.022 s [User: 0.735 s, System: 2.063 s]\n Range (min … max): 2.769 s … 2.836 s 10 runs\n\n - Printing a single ref shows no real difference between the \"files\"\n and \"reftable\" backends.\n\n Benchmark 1: show-ref: print single ref (refformat = files, refcount = 1)\n Time (mean ± σ): 1.5 ms ± 0.1 ms [User: 0.4 ms, System: 1.0 ms]\n Range (min … max): 1.4 ms … 1.8 ms 1779 runs\n\n Benchmark 2: show-ref: print single ref (refformat = reftable, refcount = 1)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.4 ms … 2.5 ms 1753 runs\n\n Benchmark 3: show-ref: print single ref (refformat = files, refcount = 1000)\n Time (mean ± σ): 1.5 ms ± 0.1 ms [User: 0.3 ms, System: 1.1 ms]\n Range (min … max): 1.4 ms … 1.9 ms 1840 runs\n\n Benchmark 4: show-ref: print single ref (refformat = reftable, refcount = 1000)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 2.0 ms 1831 runs\n\n Benchmark 5: show-ref: print single ref (refformat = files, refcount = 1000000)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 2.1 ms 1848 runs\n\n Benchmark 6: show-ref: print single ref (refformat = reftable, refcount = 1000000)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 2.1 ms 1762 runs\n\nSo overall, performance depends on the usecases. Except for many\nsequential writes the \"reftable\" backend is roughly on par or\nsignificantly faster than the \"files\" backend though. Given that the\n\"files\" backend has received 18 years of optimizations by now this can\nbe seen as a win. Furthermore, we can expect that the \"reftable\" backend\nwill grow faster over time when attention turns more towards\noptimizations.\n\nThe complete test suite passes, except for those tests explicitly marked\nto require the REFFILES prerequisite. Some tests in t0610 are marked as\nfailing because they depend on still-in-flight bug fixes. Tests can be\nrun with the new backend by setting the GIT_TEST_DEFAULT_REF_FORMAT\nenvironment variable to \"reftable\".\n\nThere is a single known conceptual incompatibility with the dumb HTTP\ntransport. As \"info/refs\" SHOULD NOT contain the HEAD reference, and\nbecause the \"HEAD\" file is not valid anymore, it is impossible for the\nremote client to figure out the default branch without changing the\nprotocol. This shortcoming needs to be handled in a subsequent patch\nseries.\n\nAs the reftable library has already been introduced a while ago, this\ncommit message will not go into the details of how exactly the on-disk\nformat works. Please refer to our preexisting technical documentation at\nDocumentation/technical/reftable for this.\n\n[1]: https://public-inbox.org/git/CAJo=hJtyof=HRy=2sLP0ng0uZ4=S-DpZ5dR1aF+VHVETKG20OQ@mail.gmail.com/\n\nOriginal-idea-by: Shawn Pearce \nBased-on-patch-by: Han-Wen Nienhuys \nSigned-off-by: Patrick Steinhardt \n---\n Documentation/ref-storage-format.txt | 2 +\n .../technical/repository-version.txt | 5 +-\n Makefile | 1 +\n contrib/workdir/git-new-workdir | 2 +-\n path.c | 2 +-\n path.h | 1 +\n refs.c | 1 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 2295 +++++++++++++++++\n repository.h | 5 +-\n t/t0610-reftable-basics.sh | 887 +++++++\n t/t0611-reftable-httpd.sh | 26 +\n t/test-lib.sh | 2 +\n 13 files changed, 3224 insertions(+), 6 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100755 t/t0610-reftable-basics.sh\n create mode 100755 t/t0611-reftable-httpd.sh\n\ndiff --git a/Documentation/ref-storage-format.txt b/Documentation/ref-storage-format.txt\nindex 1a65cac468..14fff8a9c6 100644\n--- a/Documentation/ref-storage-format.txt\n+++ b/Documentation/ref-storage-format.txt\n@@ -1 +1,3 @@\n * `files` for loose files with packed-refs. This is the default.\n+* `reftable` for the reftable format. This format is experimental and its\n+ internals are subject to change.\ndiff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\nindex 27be3741e6..47281420fc 100644\n--- a/Documentation/technical/repository-version.txt\n+++ b/Documentation/technical/repository-version.txt\n@@ -103,5 +103,6 @@ GIT_COMMON_DIR/worktrees//config.worktree)\n \n ==== `refStorage`\n \n-Specifies the file format for the ref database. The only valid value\n-is `files` (loose references with a packed-refs file).\n+Specifies the file format for the ref database. The valid values are\n+`files` (loose references with a packed-refs file) and `reftable` (see\n+Documentation/technical/reftable.txt).\ndiff --git a/Makefile b/Makefile\nindex 0f748a52e6..b302516173 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1127,6 +1127,7 @@ LIB_OBJS += reflog.o\n LIB_OBJS += refs.o\n LIB_OBJS += refs/debug.o\n LIB_OBJS += refs/files-backend.o\n+LIB_OBJS += refs/reftable-backend.o\n LIB_OBJS += refs/iterator.o\n LIB_OBJS += refs/packed-backend.o\n LIB_OBJS += refs/ref-cache.o\ndiff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir\nindex 888c34a521..989197aace 100755\n--- a/contrib/workdir/git-new-workdir\n+++ b/contrib/workdir/git-new-workdir\n@@ -79,7 +79,7 @@ trap cleanup $siglist\n # create the links to the original repo. explicitly exclude index, HEAD and\n # logs/HEAD from the list since they are purely related to the current working\n # directory, and should not be shared.\n-for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn\n+for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn reftable\n do\n \t# create a containing directory if needed\n \tcase $x in\ndiff --git a/path.c b/path.c\nindex 0fb527918b..8bb223c92c 100644\n--- a/path.c\n+++ b/path.c\n@@ -871,7 +871,7 @@ const char *enter_repo(const char *path, int strict)\n \treturn NULL;\n }\n \n-static int calc_shared_perm(int mode)\n+int calc_shared_perm(int mode)\n {\n \tint tweak;\n \ndiff --git a/path.h b/path.h\nindex b3233c51fa..e053effef2 100644\n--- a/path.h\n+++ b/path.h\n@@ -181,6 +181,7 @@ const char *git_path_shallow(struct repository *r);\n int ends_with_path_components(const char *path, const char *components);\n int validate_headref(const char *ref);\n \n+int calc_shared_perm(int mode);\n int adjust_shared_perm(const char *path);\n \n char *interpolate_path(const char *path, int real_home);\ndiff --git a/refs.c b/refs.c\nindex c633abf284..fff343c256 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -35,6 +35,7 @@\n */\n static const struct ref_storage_be *refs_backends[] = {\n \t[REF_STORAGE_FORMAT_FILES] = &refs_be_files,\n+\t[REF_STORAGE_FORMAT_REFTABLE] = &refs_be_reftable,\n };\n \n static const struct ref_storage_be *find_ref_storage_backend(unsigned int ref_storage_format)\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 82219829b0..83e0f0bba3 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -693,6 +693,7 @@ struct ref_storage_be {\n };\n \n extern struct ref_storage_be refs_be_files;\n+extern struct ref_storage_be refs_be_reftable;\n extern struct ref_storage_be refs_be_packed;\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nnew file mode 100644\nindex 0000000000..f592afe9ff\n--- /dev/null\n+++ b/refs/reftable-backend.c\n@@ -0,0 +1,2295 @@\n+#include \"../git-compat-util.h\"\n+#include \"../abspath.h\"\n+#include \"../chdir-notify.h\"\n+#include \"../environment.h\"\n+#include \"../gettext.h\"\n+#include \"../hash.h\"\n+#include \"../hex.h\"\n+#include \"../iterator.h\"\n+#include \"../ident.h\"\n+#include \"../lockfile.h\"\n+#include \"../object.h\"\n+#include \"../path.h\"\n+#include \"../refs.h\"\n+#include \"../reftable/reftable-stack.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-merged.h\"\n+#include \"../setup.h\"\n+#include \"../strmap.h\"\n+#include \"refs-internal.h\"\n+\n+/*\n+ * Used as a flag in ref_update::flags when the ref_update was via an\n+ * update to HEAD.\n+ */\n+#define REF_UPDATE_VIA_HEAD (1 << 8)\n+\n+struct reftable_ref_store {\n+\tstruct ref_store base;\n+\n+\t/*\n+\t * The main stack refers to the common dir and thus contains common\n+\t * refs as well as refs of the main repository.\n+\t */\n+\tstruct reftable_stack *main_stack;\n+\t/*\n+\t * The worktree stack refers to the gitdir in case the refdb is opened\n+\t * via a worktree. It thus contains the per-worktree refs.\n+\t */\n+\tstruct reftable_stack *worktree_stack;\n+\t/*\n+\t * Map of worktree stacks by their respective worktree names. The map\n+\t * is populated lazily when we try to resolve `worktrees/$worktree` refs.\n+\t */\n+\tstruct strmap worktree_stacks;\n+\tstruct reftable_write_options write_options;\n+\n+\tunsigned int store_flags;\n+\tint err;\n+};\n+\n+/*\n+ * Downcast ref_store to reftable_ref_store. Die if ref_store is not a\n+ * reftable_ref_store. required_flags is compared with ref_store's store_flags\n+ * to ensure the ref_store has all required capabilities. \"caller\" is used in\n+ * any necessary error messages.\n+ */\n+static struct reftable_ref_store *reftable_be_downcast(struct ref_store *ref_store,\n+\t\t\t\t\t\t unsigned int required_flags,\n+\t\t\t\t\t\t const char *caller)\n+{\n+\tstruct reftable_ref_store *refs;\n+\n+\tif (ref_store->be != &refs_be_reftable)\n+\t\tBUG(\"ref_store is type \\\"%s\\\" not \\\"reftables\\\" in %s\",\n+\t\t ref_store->be->name, caller);\n+\n+\trefs = (struct reftable_ref_store *)ref_store;\n+\n+\tif ((refs->store_flags & required_flags) != required_flags)\n+\t\tBUG(\"operation %s requires abilities 0x%x, but only have 0x%x\",\n+\t\t caller, required_flags, refs->store_flags);\n+\n+\treturn refs;\n+}\n+\n+/*\n+ * Some refs are global to the repository (refs/heads/{*}), while others are\n+ * local to the worktree (eg. HEAD, refs/bisect/{*}). We solve this by having\n+ * multiple separate databases (ie. multiple reftable/ directories), one for\n+ * the shared refs, one for the current worktree refs, and one for each\n+ * additional worktree. For reading, we merge the view of both the shared and\n+ * the current worktree's refs, when necessary.\n+ *\n+ * This function also optionally assigns the rewritten reference name that is\n+ * local to the stack. This translation is required when using worktree refs\n+ * like `worktrees/$worktree/refs/heads/foo` as worktree stacks will store\n+ * those references in their normalized form.\n+ */\n+static struct reftable_stack *stack_for(struct reftable_ref_store *store,\n+\t\t\t\t\tconst char *refname,\n+\t\t\t\t\tconst char **rewritten_ref)\n+{\n+\tconst char *wtname;\n+\tint wtname_len;\n+\n+\tif (!refname)\n+\t\treturn store->main_stack;\n+\n+\tswitch (parse_worktree_ref(refname, &wtname, &wtname_len, rewritten_ref)) {\n+\tcase REF_WORKTREE_OTHER: {\n+\t\tstatic struct strbuf wtname_buf = STRBUF_INIT;\n+\t\tstruct strbuf wt_dir = STRBUF_INIT;\n+\t\tstruct reftable_stack *stack;\n+\n+\t\t/*\n+\t\t * We're using a static buffer here so that we don't need to\n+\t\t * allocate the worktree name whenever we look up a reference.\n+\t\t * This could be avoided if the strmap interface knew how to\n+\t\t * handle keys with a length.\n+\t\t */\n+\t\tstrbuf_reset(&wtname_buf);\n+\t\tstrbuf_add(&wtname_buf, wtname, wtname_len);\n+\n+\t\t/*\n+\t\t * There is an edge case here: when the worktree references the\n+\t\t * current worktree, then we set up the stack once via\n+\t\t * `worktree_stacks` and once via `worktree_stack`. This is\n+\t\t * wasteful, but in the reading case it shouldn't matter. And\n+\t\t * in the writing case we would notice that the stack is locked\n+\t\t * already and error out when trying to write a reference via\n+\t\t * both stacks.\n+\t\t */\n+\t\tstack = strmap_get(&store->worktree_stacks, wtname_buf.buf);\n+\t\tif (!stack) {\n+\t\t\tstrbuf_addf(&wt_dir, \"%s/worktrees/%s/reftable\",\n+\t\t\t\t store->base.repo->commondir, wtname_buf.buf);\n+\n+\t\t\tstore->err = reftable_new_stack(&stack, wt_dir.buf,\n+\t\t\t\t\t\t\tstore->write_options);\n+\t\t\tassert(store->err != REFTABLE_API_ERROR);\n+\t\t\tstrmap_put(&store->worktree_stacks, wtname_buf.buf, stack);\n+\t\t}\n+\n+\t\tstrbuf_release(&wt_dir);\n+\t\treturn stack;\n+\t}\n+\tcase REF_WORKTREE_CURRENT:\n+\t\t/*\n+\t\t * If there is no worktree stack then we're currently in the\n+\t\t * main worktree. We thus return the main stack in that case.\n+\t\t */\n+\t\tif (!store->worktree_stack)\n+\t\t\treturn store->main_stack;\n+\t\treturn store->worktree_stack;\n+\tcase REF_WORKTREE_MAIN:\n+\tcase REF_WORKTREE_SHARED:\n+\t\treturn store->main_stack;\n+\tdefault:\n+\t\tBUG(\"unhandled worktree reference type\");\n+\t}\n+}\n+\n+static int should_write_log(struct ref_store *refs, const char *refname)\n+{\n+\tif (log_all_ref_updates == LOG_REFS_UNSET)\n+\t\tlog_all_ref_updates = is_bare_repository() ? LOG_REFS_NONE : LOG_REFS_NORMAL;\n+\n+\tswitch (log_all_ref_updates) {\n+\tcase LOG_REFS_NONE:\n+\t\treturn refs_reflog_exists(refs, refname);\n+\tcase LOG_REFS_ALWAYS:\n+\t\treturn 1;\n+\tcase LOG_REFS_NORMAL:\n+\t\tif (should_autocreate_reflog(refname))\n+\t\t\treturn 1;\n+\t\treturn refs_reflog_exists(refs, refname);\n+\tdefault:\n+\t\tBUG(\"unhandled core.logAllRefUpdates value %d\", log_all_ref_updates);\n+\t}\n+}\n+\n+static void clear_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\t/* when we write log records, the hashes are owned by a struct\n+\t\t * oid */\n+\t\tlog->value.update.old_hash = NULL;\n+\t\tlog->value.update.new_hash = NULL;\n+\t\tbreak;\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\t}\n+\treftable_log_record_release(log);\n+}\n+\n+static void fill_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tconst char *info = git_committer_info(0);\n+\tstruct ident_split split = {0};\n+\tint sign = 1;\n+\n+\tif (split_ident_line(&split, info, strlen(info)))\n+\t\tBUG(\"failed splitting committer info\");\n+\n+\treftable_log_record_release(log);\n+\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\tlog->value.update.name =\n+\t\txstrndup(split.name_begin, split.name_end - split.name_begin);\n+\tlog->value.update.email =\n+\t\txstrndup(split.mail_begin, split.mail_end - split.mail_begin);\n+\tlog->value.update.time = atol(split.date_begin);\n+\tif (*split.tz_begin == '-') {\n+\t\tsign = -1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\tif (*split.tz_begin == '+') {\n+\t\tsign = 1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\n+\tlog->value.update.tz_offset = sign * atoi(split.tz_begin);\n+}\n+\n+static int read_ref_without_reload(struct reftable_stack *stack,\n+\t\t\t\t const char *refname,\n+\t\t\t\t struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type)\n+{\n+\tstruct reftable_ref_record ref = {0};\n+\tint ret;\n+\n+\tret = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (reftable_ref_record_val1(&ref)) {\n+\t\toidread(oid, reftable_ref_record_val1(&ref));\n+\t} else {\n+\t\t/* We got a tombstone, which should not happen. */\n+\t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n+\t}\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn ret;\n+}\n+\n+static struct ref_store *reftable_be_init(struct repository *repo,\n+\t\t\t\t\t const char *gitdir,\n+\t\t\t\t\t unsigned int store_flags)\n+{\n+\tstruct reftable_ref_store *refs = xcalloc(1, sizeof(*refs));\n+\tstruct strbuf path = STRBUF_INIT;\n+\tint is_worktree;\n+\tmode_t mask;\n+\n+\tmask = umask(0);\n+\tumask(mask);\n+\n+\tbase_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);\n+\tstrmap_init(&refs->worktree_stacks);\n+\trefs->store_flags = store_flags;\n+\trefs->write_options.block_size = 4096;\n+\trefs->write_options.hash_id = repo->hash_algo->format_id;\n+\trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n+\n+\t/*\n+\t * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.\n+\t * This stack contains both the shared and the main worktree refs.\n+\t *\n+\t * Note that we don't try to resolve the path in case we have a\n+\t * worktree because `get_common_dir_noenv()` already does it for us.\n+\t */\n+\tis_worktree = get_common_dir_noenv(&path, gitdir);\n+\tif (!is_worktree) {\n+\t\tstrbuf_reset(&path);\n+\t\tstrbuf_realpath(&path, gitdir, 0);\n+\t}\n+\tstrbuf_addstr(&path, \"/reftable\");\n+\trefs->err = reftable_new_stack(&refs->main_stack, path.buf,\n+\t\t\t\t refs->write_options);\n+\tif (refs->err)\n+\t\tgoto done;\n+\n+\t/*\n+\t * If we're in a worktree we also need to set up the worktree reftable\n+\t * stack that is contained in the per-worktree GIT_DIR.\n+\t *\n+\t * Ideally, we would also add the stack to our worktree stack map. But\n+\t * we have no way to figure out the worktree name here and thus can't\n+\t * do it efficiently.\n+\t */\n+\tif (is_worktree) {\n+\t\tstrbuf_reset(&path);\n+\t\tstrbuf_addf(&path, \"%s/reftable\", gitdir);\n+\n+\t\trefs->err = reftable_new_stack(&refs->worktree_stack, path.buf,\n+\t\t\t\t\t refs->write_options);\n+\t\tif (refs->err)\n+\t\t\tgoto done;\n+\t}\n+\n+\tchdir_notify_reparent(\"reftables-backend $GIT_DIR\", &refs->base.gitdir);\n+\n+done:\n+\tassert(refs->err != REFTABLE_API_ERROR);\n+\tstrbuf_release(&path);\n+\treturn &refs->base;\n+}\n+\n+static int reftable_be_init_db(struct ref_store *ref_store,\n+\t\t\t int flags UNUSED,\n+\t\t\t struct strbuf *err UNUSED)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"init_db\");\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\tstrbuf_addf(&sb, \"%s/reftable\", refs->base.gitdir);\n+\tsafe_create_dir(sb.buf, 1);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/HEAD\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"ref: refs/heads/.invalid\");\n+\tadjust_shared_perm(sb.buf);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs\", refs->base.gitdir);\n+\tsafe_create_dir(sb.buf, 1);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs/heads\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\tadjust_shared_perm(sb.buf);\n+\n+\tstrbuf_release(&sb);\n+\treturn 0;\n+}\n+\n+struct reftable_ref_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_ref_record ref;\n+\tstruct object_id oid;\n+\n+\tconst char *prefix;\n+\tunsigned int flags;\n+\tint err;\n+};\n+\n+static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct reftable_ref_iterator *iter =\n+\t\t(struct reftable_ref_iterator *)ref_iterator;\n+\tstruct reftable_ref_store *refs = iter->refs;\n+\n+\twhile (!iter->err) {\n+\t\tint flags = 0;\n+\n+\t\titer->err = reftable_iterator_next_ref(&iter->iter, &iter->ref);\n+\t\tif (iter->err)\n+\t\t\tbreak;\n+\n+\t\t/*\n+\t\t * The files backend only lists references contained in\n+\t\t * \"refs/\". We emulate the same behaviour here and thus skip\n+\t\t * all references that don't start with this prefix.\n+\t\t */\n+\t\tif (!starts_with(iter->ref.refname, \"refs/\"))\n+\t\t\tcontinue;\n+\n+\t\tif (iter->prefix &&\n+\t\t strncmp(iter->prefix, iter->ref.refname, strlen(iter->prefix))) {\n+\t\t\titer->err = 1;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (iter->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n+\t\t parse_worktree_ref(iter->ref.refname, NULL, NULL, NULL) !=\n+\t\t\t REF_WORKTREE_CURRENT)\n+\t\t\tcontinue;\n+\n+\t\tswitch (iter->ref.value_type) {\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\toidread(&iter->oid, iter->ref.value.val1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\toidread(&iter->oid, iter->ref.value.val2.value);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF:\n+\t\t\tif (!refs_resolve_ref_unsafe(&iter->refs->base, iter->ref.refname,\n+\t\t\t\t\t\t RESOLVE_REF_READING, &iter->oid, &flags))\n+\t\t\t\toidclr(&iter->oid);\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tBUG(\"unhandled reference value type %d\", iter->ref.value_type);\n+\t\t}\n+\n+\t\tif (is_null_oid(&iter->oid))\n+\t\t\tflags |= REF_ISBROKEN;\n+\n+\t\tif (check_refname_format(iter->ref.refname, REFNAME_ALLOW_ONELEVEL)) {\n+\t\t\tif (!refname_is_safe(iter->ref.refname))\n+\t\t\t\tdie(_(\"refname is dangerous: %s\"), iter->ref.refname);\n+\t\t\toidclr(&iter->oid);\n+\t\t\tflags |= REF_BAD_NAME | REF_ISBROKEN;\n+\t\t}\n+\n+\t\tif (iter->flags & DO_FOR_EACH_OMIT_DANGLING_SYMREFS &&\n+\t\t flags & REF_ISSYMREF &&\n+\t\t flags & REF_ISBROKEN)\n+\t\t\tcontinue;\n+\n+\t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t !ref_resolves_to_object(iter->ref.refname, refs->base.repo,\n+\t\t\t\t\t &iter->oid, flags))\n+\t\t\t\tcontinue;\n+\n+\t\titer->base.refname = iter->ref.refname;\n+\t\titer->base.oid = &iter->oid;\n+\t\titer->base.flags = flags;\n+\n+\t\tbreak;\n+\t}\n+\n+\tif (iter->err > 0) {\n+\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n+\t\t\treturn ITER_ERROR;\n+\t\treturn ITER_DONE;\n+\t}\n+\n+\tif (iter->err < 0) {\n+\t\tref_iterator_abort(ref_iterator);\n+\t\treturn ITER_ERROR;\n+\t}\n+\n+\treturn ITER_OK;\n+}\n+\n+static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tstruct reftable_ref_iterator *iter =\n+\t\t(struct reftable_ref_iterator *)ref_iterator;\n+\n+\tif (iter->ref.value_type == REFTABLE_REF_VAL2) {\n+\t\toidread(peeled, iter->ref.value.val2.target_value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\n+\n+static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct reftable_ref_iterator *iter =\n+\t\t(struct reftable_ref_iterator *)ref_iterator;\n+\treftable_ref_record_release(&iter->ref);\n+\treftable_iterator_destroy(&iter->iter);\n+\tfree(iter);\n+\treturn ITER_DONE;\n+}\n+\n+static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n+\t.advance = reftable_ref_iterator_advance,\n+\t.peel = reftable_ref_iterator_peel,\n+\t.abort = reftable_ref_iterator_abort\n+};\n+\n+static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_store *refs,\n+\t\t\t\t\t\t\t struct reftable_stack *stack,\n+\t\t\t\t\t\t\t const char *prefix,\n+\t\t\t\t\t\t\t int flags)\n+{\n+\tstruct reftable_merged_table *merged_table;\n+\tstruct reftable_ref_iterator *iter;\n+\tint ret;\n+\n+\titer = xcalloc(1, sizeof(*iter));\n+\tbase_ref_iterator_init(&iter->base, &reftable_ref_iterator_vtable, 1);\n+\titer->prefix = prefix;\n+\titer->base.oid = &iter->oid;\n+\titer->flags = flags;\n+\titer->refs = refs;\n+\n+\tret = refs->err;\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tmerged_table = reftable_stack_merged_table(stack);\n+\n+\tret = reftable_merged_table_seek_ref(merged_table, &iter->iter, prefix);\n+\tif (ret)\n+\t\tgoto done;\n+\n+done:\n+\titer->err = ret;\n+\treturn iter;\n+}\n+\n+static enum iterator_selection iterator_select(struct ref_iterator *iter_worktree,\n+\t\t\t\t\t struct ref_iterator *iter_common,\n+\t\t\t\t\t void *cb_data UNUSED)\n+{\n+\tif (iter_worktree && !iter_common) {\n+\t\t/*\n+\t\t * Return the worktree ref if there are no more common refs.\n+\t\t */\n+\t\treturn ITER_SELECT_0;\n+\t} else if (iter_common) {\n+\t\t/*\n+\t\t * In case we have pending worktree and common refs we need to\n+\t\t * yield them based on their lexicographical order. Worktree\n+\t\t * refs that have the same name as common refs shadow the\n+\t\t * latter.\n+\t\t */\n+\t\tif (iter_worktree) {\n+\t\t\tint cmp = strcmp(iter_worktree->refname,\n+\t\t\t\t\t iter_common->refname);\n+\t\t\tif (cmp < 0)\n+\t\t\t\treturn ITER_SELECT_0;\n+\t\t\telse if (!cmp)\n+\t\t\t\treturn ITER_SELECT_0_SKIP_1;\n+\t\t}\n+\n+\t\t /*\n+\t\t * We now know that the lexicographically-next ref is a common\n+\t\t * ref. When the common ref is a shared one we return it.\n+\t\t */\n+\t\tif (parse_worktree_ref(iter_common->refname, NULL, NULL,\n+\t\t\t\t NULL) == REF_WORKTREE_SHARED)\n+\t\t\treturn ITER_SELECT_1;\n+\n+\t\t/*\n+\t\t * Otherwise, if the common ref is a per-worktree ref we skip\n+\t\t * it because it would belong to the main worktree, not ours.\n+\t\t */\n+\t\treturn ITER_SKIP_1;\n+\t} else {\n+\t\treturn ITER_DONE;\n+\t}\n+}\n+\n+static struct ref_iterator *reftable_be_iterator_begin(struct ref_store *ref_store,\n+\t\t\t\t\t\t const char *prefix,\n+\t\t\t\t\t\t const char **exclude_patterns,\n+\t\t\t\t\t\t unsigned int flags)\n+{\n+\tstruct reftable_ref_iterator *main_iter, *worktree_iter;\n+\tstruct reftable_ref_store *refs;\n+\tunsigned int required_flags = REF_STORE_READ;\n+\n+\tif (!(flags & DO_FOR_EACH_INCLUDE_BROKEN))\n+\t\trequired_flags |= REF_STORE_ODB;\n+\trefs = reftable_be_downcast(ref_store, required_flags, \"ref_iterator_begin\");\n+\n+\tmain_iter = ref_iterator_for_stack(refs, refs->main_stack, prefix, flags);\n+\n+\t/*\n+\t * The worktree stack is only set when we're in an actual worktree\n+\t * right now. If we aren't, then we return the common reftable\n+\t * iterator, only.\n+\t */\n+\t if (!refs->worktree_stack)\n+\t\treturn &main_iter->base;\n+\n+\t/*\n+\t * Otherwise we merge both the common and the per-worktree refs into a\n+\t * single iterator.\n+\t */\n+\tworktree_iter = ref_iterator_for_stack(refs, refs->worktree_stack, prefix, flags);\n+\treturn merge_ref_iterator_begin(1, &worktree_iter->base, &main_iter->base,\n+\t\t\t\t\titerator_select, NULL);\n+}\n+\n+static int reftable_be_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname,\n+\t\t\t\t struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type,\n+\t\t\t\t int *failure_errno)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_raw_ref\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tint ret;\n+\n+\tif (refs->err < 0)\n+\t\treturn refs->err;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tret = read_ref_without_reload(stack, refname, oid, referent, type);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\tif (ret > 0) {\n+\t\t*failure_errno = ENOENT;\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int reftable_be_read_symbolic_ref(struct ref_store *ref_store,\n+\t\t\t\t\t const char *refname,\n+\t\t\t\t\t struct strbuf *referent)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_symbolic_ref\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_ref_record ref = {0};\n+\tint ret;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tret = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (ret == 0 && ref.value_type == REFTABLE_REF_SYMREF)\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\telse\n+\t\tret = -1;\n+\n+\treftable_ref_record_release(&ref);\n+\treturn ret;\n+}\n+\n+/*\n+ * Return the refname under which update was originally requested.\n+ */\n+static const char *original_update_refname(struct ref_update *update)\n+{\n+\twhile (update->parent_update)\n+\t\tupdate = update->parent_update;\n+\treturn update->refname;\n+}\n+\n+struct reftable_transaction_update {\n+\tstruct ref_update *update;\n+\tstruct object_id current_oid;\n+};\n+\n+struct write_transaction_table_arg {\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_addition *addition;\n+\tstruct reftable_transaction_update *updates;\n+\tsize_t updates_nr;\n+\tsize_t updates_alloc;\n+\tsize_t updates_expected;\n+};\n+\n+struct reftable_transaction_data {\n+\tstruct write_transaction_table_arg *args;\n+\tsize_t args_nr, args_alloc;\n+};\n+\n+static void free_transaction_data(struct reftable_transaction_data *tx_data)\n+{\n+\tif (!tx_data)\n+\t\treturn;\n+\tfor (size_t i = 0; i < tx_data->args_nr; i++) {\n+\t\treftable_addition_destroy(tx_data->args[i].addition);\n+\t\tfree(tx_data->args[i].updates);\n+\t}\n+\tfree(tx_data->args);\n+\tfree(tx_data);\n+}\n+\n+/*\n+ * Prepare transaction update for the given reference update. This will cause\n+ * us to lock the corresponding reftable stack for concurrent modification.\n+ */\n+static int prepare_transaction_update(struct write_transaction_table_arg **out,\n+\t\t\t\t struct reftable_ref_store *refs,\n+\t\t\t\t struct reftable_transaction_data *tx_data,\n+\t\t\t\t struct ref_update *update,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_stack *stack = stack_for(refs, update->refname, NULL);\n+\tstruct write_transaction_table_arg *arg = NULL;\n+\tsize_t i;\n+\tint ret;\n+\n+\t/*\n+\t * Search for a preexisting stack update. If there is one then we add\n+\t * the update to it, otherwise we set up a new stack update.\n+\t */\n+\tfor (i = 0; !arg && i < tx_data->args_nr; i++)\n+\t\tif (tx_data->args[i].stack == stack)\n+\t\t\targ = &tx_data->args[i];\n+\n+\tif (!arg) {\n+\t\tstruct reftable_addition *addition;\n+\n+\t\tret = reftable_stack_reload(stack);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\n+\t\tret = reftable_stack_new_addition(&addition, stack);\n+\t\tif (ret) {\n+\t\t\tif (ret == REFTABLE_LOCK_ERROR)\n+\t\t\t\tstrbuf_addstr(err, \"cannot lock references\");\n+\t\t\treturn ret;\n+\t\t}\n+\n+\t\tALLOC_GROW(tx_data->args, tx_data->args_nr + 1,\n+\t\t\t tx_data->args_alloc);\n+\t\targ = &tx_data->args[tx_data->args_nr++];\n+\t\targ->refs = refs;\n+\t\targ->stack = stack;\n+\t\targ->addition = addition;\n+\t\targ->updates = NULL;\n+\t\targ->updates_nr = 0;\n+\t\targ->updates_alloc = 0;\n+\t\targ->updates_expected = 0;\n+\t}\n+\n+\targ->updates_expected++;\n+\n+\tif (out)\n+\t\t*out = arg;\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Queue a reference update for the correct stack. We potentially need to\n+ * handle multiple stack updates in a single transaction when it spans across\n+ * multiple worktrees.\n+ */\n+static int queue_transaction_update(struct reftable_ref_store *refs,\n+\t\t\t\t struct reftable_transaction_data *tx_data,\n+\t\t\t\t struct ref_update *update,\n+\t\t\t\t struct object_id *current_oid,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct write_transaction_table_arg *arg = NULL;\n+\tint ret;\n+\n+\tif (update->backend_data)\n+\t\tBUG(\"reference update queued more than once\");\n+\n+\tret = prepare_transaction_update(&arg, refs, tx_data, update, err);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\tALLOC_GROW(arg->updates, arg->updates_nr + 1,\n+\t\t arg->updates_alloc);\n+\targ->updates[arg->updates_nr].update = update;\n+\toidcpy(&arg->updates[arg->updates_nr].current_oid, current_oid);\n+\tupdate->backend_data = &arg->updates[arg->updates_nr++];\n+\n+\treturn 0;\n+}\n+\n+static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE|REF_STORE_MAIN, \"ref_transaction_prepare\");\n+\tstruct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT;\n+\tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct reftable_transaction_data *tx_data = NULL;\n+\tstruct object_id head_oid;\n+\tunsigned int head_type = 0;\n+\tsize_t i;\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\ttx_data = xcalloc(1, sizeof(*tx_data));\n+\n+\t/*\n+\t * Preprocess all updates. For one we check that there are no duplicate\n+\t * reference updates in this transaction. Second, we lock all stacks\n+\t * that will be modified during the transaction.\n+\t */\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tret = prepare_transaction_update(NULL, refs, tx_data,\n+\t\t\t\t\t\t transaction->updates[i], err);\n+\t\tif (ret)\n+\t\t\tgoto done;\n+\n+\t\tstring_list_append(&affected_refnames,\n+\t\t\t\t transaction->updates[i]->refname);\n+\t}\n+\n+\t/*\n+\t * Now that we have counted updates per stack we can preallocate their\n+\t * arrays. This avoids having to reallocate many times.\n+\t */\n+\tfor (i = 0; i < tx_data->args_nr; i++) {\n+\t\tCALLOC_ARRAY(tx_data->args[i].updates, tx_data->args[i].updates_expected);\n+\t\ttx_data->args[i].updates_alloc = tx_data->args[i].updates_expected;\n+\t}\n+\n+\t/*\n+\t * Fail if a refname appears more than once in the transaction.\n+\t * This code is taken from the files backend and is a good candidate to\n+\t * be moved into the generic layer.\n+\t */\n+\tstring_list_sort(&affected_refnames);\n+\tif (ref_update_reject_duplicates(&affected_refnames, err)) {\n+\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tret = read_ref_without_reload(stack_for(refs, \"HEAD\", NULL), \"HEAD\", &head_oid,\n+\t\t\t\t &head_referent, &head_type);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = transaction->updates[i];\n+\t\tstruct object_id current_oid = {0};\n+\t\tstruct reftable_stack *stack;\n+\t\tconst char *rewritten_ref;\n+\n+\t\tstack = stack_for(refs, u->refname, &rewritten_ref);\n+\n+\t\t/* Verify that the new object ID is valid. */\n+\t\tif ((u->flags & REF_HAVE_NEW) && !is_null_oid(&u->new_oid) &&\n+\t\t !(u->flags & REF_SKIP_OID_VERIFICATION) &&\n+\t\t !(u->flags & REF_LOG_ONLY)) {\n+\t\t\tstruct object *o = parse_object(refs->base.repo, &u->new_oid);\n+\t\t\tif (!o) {\n+\t\t\t\tstrbuf_addf(err,\n+\t\t\t\t\t _(\"trying to write ref '%s' with nonexistent object %s\"),\n+\t\t\t\t\t u->refname, oid_to_hex(&u->new_oid));\n+\t\t\t\tret = -1;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\n+\t\t\tif (o->type != OBJ_COMMIT && is_branch(u->refname)) {\n+\t\t\t\tstrbuf_addf(err, _(\"trying to write non-commit object %s to branch '%s'\"),\n+\t\t\t\t\t oid_to_hex(&u->new_oid), u->refname);\n+\t\t\t\tret = -1;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\n+\t\t/*\n+\t\t * When we update the reference that HEAD points to we enqueue\n+\t\t * a second log-only update for HEAD so that its reflog is\n+\t\t * updated accordingly.\n+\t\t */\n+\t\tif (head_type == REF_ISSYMREF &&\n+\t\t !(u->flags & REF_LOG_ONLY) &&\n+\t\t !(u->flags & REF_UPDATE_VIA_HEAD) &&\n+\t\t !strcmp(rewritten_ref, head_referent.buf)) {\n+\t\t\tstruct ref_update *new_update;\n+\n+\t\t\t/*\n+\t\t\t * First make sure that HEAD is not already in the\n+\t\t\t * transaction. This check is O(lg N) in the transaction\n+\t\t\t * size, but it happens at most once per transaction.\n+\t\t\t */\n+\t\t\tif (string_list_has_string(&affected_refnames, \"HEAD\")) {\n+\t\t\t\t/* An entry already existed */\n+\t\t\t\tstrbuf_addf(err,\n+\t\t\t\t\t _(\"multiple updates for 'HEAD' (including one \"\n+\t\t\t\t\t \"via its referent '%s') are not allowed\"),\n+\t\t\t\t\t u->refname);\n+\t\t\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\n+\t\t\tnew_update = ref_transaction_add_update(\n+\t\t\t\t\ttransaction, \"HEAD\",\n+\t\t\t\t\tu->flags | REF_LOG_ONLY | REF_NO_DEREF,\n+\t\t\t\t\t&u->new_oid, &u->old_oid, u->msg);\n+\t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n+\t\t}\n+\n+\t\tret = read_ref_without_reload(stack, rewritten_ref,\n+\t\t\t\t\t ¤t_oid, &referent, &u->type);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tif (ret > 0 && (!(u->flags & REF_HAVE_OLD) || is_null_oid(&u->old_oid))) {\n+\t\t\t/*\n+\t\t\t * The reference does not exist, and we either have no\n+\t\t\t * old object ID or expect the reference to not exist.\n+\t\t\t * We can thus skip below safety checks as well as the\n+\t\t\t * symref splitting. But we do want to verify that\n+\t\t\t * there is no conflicting reference here so that we\n+\t\t\t * can output a proper error message instead of failing\n+\t\t\t * at a later point.\n+\t\t\t */\n+\t\t\tret = refs_verify_refname_available(ref_store, u->refname,\n+\t\t\t\t\t\t\t &affected_refnames, NULL, err);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto done;\n+\n+\t\t\t/*\n+\t\t\t * There is no need to write the reference deletion\n+\t\t\t * when the reference in question doesn't exist.\n+\t\t\t */\n+\t\t\t if (u->flags & REF_HAVE_NEW && !is_null_oid(&u->new_oid)) {\n+\t\t\t\t ret = queue_transaction_update(refs, tx_data, u,\n+\t\t\t\t\t\t\t\t¤t_oid, err);\n+\t\t\t\t if (ret)\n+\t\t\t\t\t goto done;\n+\t\t\t }\n+\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (ret > 0) {\n+\t\t\t/* The reference does not exist, but we expected it to. */\n+\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t \"unable to resolve reference '%s'\"),\n+\t\t\t\t original_update_refname(u), u->refname);\n+\t\t\tret = -1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (u->type & REF_ISSYMREF) {\n+\t\t\t/*\n+\t\t\t * The reftable stack is locked at this point already,\n+\t\t\t * so it is safe to call `refs_resolve_ref_unsafe()`\n+\t\t\t * here without causing races.\n+\t\t\t */\n+\t\t\tconst char *resolved = refs_resolve_ref_unsafe(&refs->base, u->refname, 0,\n+\t\t\t\t\t\t\t\t ¤t_oid, NULL);\n+\n+\t\t\tif (u->flags & REF_NO_DEREF) {\n+\t\t\t\tif (u->flags & REF_HAVE_OLD && !resolved) {\n+\t\t\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t\t\t \"error reading reference\"), u->refname);\n+\t\t\t\t\tret = -1;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n+\t\t\t} else {\n+\t\t\t\tstruct ref_update *new_update;\n+\t\t\t\tint new_flags;\n+\n+\t\t\t\tnew_flags = u->flags;\n+\t\t\t\tif (!strcmp(rewritten_ref, \"HEAD\"))\n+\t\t\t\t\tnew_flags |= REF_UPDATE_VIA_HEAD;\n+\n+\t\t\t\t/*\n+\t\t\t\t * If we are updating a symref (eg. HEAD), we should also\n+\t\t\t\t * update the branch that the symref points to.\n+\t\t\t\t *\n+\t\t\t\t * This is generic functionality, and would be better\n+\t\t\t\t * done in refs.c, but the current implementation is\n+\t\t\t\t * intertwined with the locking in files-backend.c.\n+\t\t\t\t */\n+\t\t\t\tnew_update = ref_transaction_add_update(\n+\t\t\t\t\t\ttransaction, referent.buf, new_flags,\n+\t\t\t\t\t\t&u->new_oid, &u->old_oid, u->msg);\n+\t\t\t\tnew_update->parent_update = u;\n+\n+\t\t\t\t/*\n+\t\t\t\t * Change the symbolic ref update to log only. Also, it\n+\t\t\t\t * doesn't need to check its old OID value, as that will be\n+\t\t\t\t * done when new_update is processed.\n+\t\t\t\t */\n+\t\t\t\tu->flags |= REF_LOG_ONLY | REF_NO_DEREF;\n+\t\t\t\tu->flags &= ~REF_HAVE_OLD;\n+\n+\t\t\t\tif (string_list_has_string(&affected_refnames, new_update->refname)) {\n+\t\t\t\t\tstrbuf_addf(err,\n+\t\t\t\t\t\t _(\"multiple updates for '%s' (including one \"\n+\t\t\t\t\t\t \"via symref '%s') are not allowed\"),\n+\t\t\t\t\t\t referent.buf, u->refname);\n+\t\t\t\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n+\t\t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n+\t\t\t}\n+\t\t}\n+\n+\t\t/*\n+\t\t * Verify that the old object matches our expectations. Note\n+\t\t * that the error messages here do not make a lot of sense in\n+\t\t * the context of the reftable backend as we never lock\n+\t\t * individual refs. But the error messages match what the files\n+\t\t * backend returns, which keeps our tests happy.\n+\t\t */\n+\t\tif (u->flags & REF_HAVE_OLD && !oideq(¤t_oid, &u->old_oid)) {\n+\t\t\tif (is_null_oid(&u->old_oid))\n+\t\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t\t \"reference already exists\"),\n+\t\t\t\t\t original_update_refname(u));\n+\t\t\telse if (is_null_oid(¤t_oid))\n+\t\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t\t \"reference is missing but expected %s\"),\n+\t\t\t\t\t original_update_refname(u),\n+\t\t\t\t\t oid_to_hex(&u->old_oid));\n+\t\t\telse\n+\t\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t\t \"is at %s but expected %s\"),\n+\t\t\t\t\t original_update_refname(u),\n+\t\t\t\t\t oid_to_hex(¤t_oid),\n+\t\t\t\t\t oid_to_hex(&u->old_oid));\n+\t\t\tret = -1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\t/*\n+\t\t * If all of the following conditions are true:\n+\t\t *\n+\t\t * - We're not about to write a symref.\n+\t\t * - We're not about to write a log-only entry.\n+\t\t * - Old and new object ID are different.\n+\t\t *\n+\t\t * Then we're essentially doing a no-op update that can be\n+\t\t * skipped. This is not only for the sake of efficiency, but\n+\t\t * also skips writing unneeded reflog entries.\n+\t\t */\n+\t\tif ((u->type & REF_ISSYMREF) ||\n+\t\t (u->flags & REF_LOG_ONLY) ||\n+\t\t (u->flags & REF_HAVE_NEW && !oideq(¤t_oid, &u->new_oid))) {\n+\t\t\tret = queue_transaction_update(refs, tx_data, u,\n+\t\t\t\t\t\t ¤t_oid, err);\n+\t\t\tif (ret)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\ttransaction->backend_data = tx_data;\n+\ttransaction->state = REF_TRANSACTION_PREPARED;\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\tif (ret < 0) {\n+\t\tfree_transaction_data(tx_data);\n+\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\t\tif (!err->len)\n+\t\t\tstrbuf_addf(err, _(\"reftable: transaction prepare: %s\"),\n+\t\t\t\t reftable_error_str(ret));\n+\t}\n+\tstring_list_clear(&affected_refnames, 0);\n+\tstrbuf_release(&referent);\n+\tstrbuf_release(&head_referent);\n+\n+\treturn ret;\n+}\n+\n+static int reftable_be_transaction_abort(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_transaction_data *tx_data = transaction->backend_data;\n+\tfree_transaction_data(tx_data);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\treturn 0;\n+}\n+\n+static int transaction_update_cmp(const void *a, const void *b)\n+{\n+\treturn strcmp(((struct reftable_transaction_update *)a)->update->refname,\n+\t\t ((struct reftable_transaction_update *)b)->update->refname);\n+}\n+\n+static int write_transaction_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct write_transaction_table_arg *arg = cb_data;\n+\tstruct reftable_merged_table *mt =\n+\t\treftable_stack_merged_table(arg->stack);\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_log_record *logs = NULL;\n+\tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tint ret = 0;\n+\n+\tQSORT(arg->updates, arg->updates_nr, transaction_update_cmp);\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tfor (i = 0; i < arg->updates_nr; i++) {\n+\t\tstruct reftable_transaction_update *tx_update = &arg->updates[i];\n+\t\tstruct ref_update *u = tx_update->update;\n+\n+\t\t/*\n+\t\t * Write a reflog entry when updating a ref to point to\n+\t\t * something new in either of the following cases:\n+\t\t *\n+\t\t * - The reference is about to be deleted. We always want to\n+\t\t * delete the reflog in that case.\n+\t\t * - REF_FORCE_CREATE_REFLOG is set, asking us to always create\n+\t\t * the reflog entry.\n+\t\t * - `core.logAllRefUpdates` tells us to create the reflog for\n+\t\t * the given ref.\n+\t\t */\n+\t\tif (u->flags & REF_HAVE_NEW && !(u->type & REF_ISSYMREF) && is_null_oid(&u->new_oid)) {\n+\t\t\tstruct reftable_log_record log = {0};\n+\t\t\tstruct reftable_iterator it = {0};\n+\n+\t\t\t/*\n+\t\t\t * When deleting refs we also delete all reflog entries\n+\t\t\t * with them. While it is not strictly required to\n+\t\t\t * delete reflogs together with their refs, this\n+\t\t\t * matches the behaviour of the files backend.\n+\t\t\t *\n+\t\t\t * Unfortunately, we have no better way than to delete\n+\t\t\t * all reflog entries one by one.\n+\t\t\t */\n+\t\t\tret = reftable_merged_table_seek_log(mt, &it, u->refname);\n+\t\t\twhile (ret == 0) {\n+\t\t\t\tstruct reftable_log_record *tombstone;\n+\n+\t\t\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\t\t\tif (ret < 0)\n+\t\t\t\t\tbreak;\n+\t\t\t\tif (ret > 0 || strcmp(log.refname, u->refname)) {\n+\t\t\t\t\tret = 0;\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\n+\t\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\t\t\ttombstone = &logs[logs_nr++];\n+\t\t\t\ttombstone->refname = xstrdup(u->refname);\n+\t\t\t\ttombstone->value_type = REFTABLE_LOG_DELETION;\n+\t\t\t\ttombstone->update_index = log.update_index;\n+\t\t\t}\n+\n+\t\t\treftable_log_record_release(&log);\n+\t\t\treftable_iterator_destroy(&it);\n+\n+\t\t\tif (ret)\n+\t\t\t\tgoto done;\n+\t\t} else if (u->flags & REF_HAVE_NEW &&\n+\t\t\t (u->flags & REF_FORCE_CREATE_REFLOG ||\n+\t\t\t should_write_log(&arg->refs->base, u->refname))) {\n+\t\t\tstruct reftable_log_record *log;\n+\n+\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\t\tlog = &logs[logs_nr++];\n+\t\t\tmemset(log, 0, sizeof(*log));\n+\n+\t\t\tfill_reftable_log_record(log);\n+\t\t\tlog->update_index = ts;\n+\t\t\tlog->refname = xstrdup(u->refname);\n+\t\t\tlog->value.update.new_hash = u->new_oid.hash;\n+\t\t\tlog->value.update.old_hash = tx_update->current_oid.hash;\n+\t\t\tlog->value.update.message =\n+\t\t\t\txstrndup(u->msg, arg->refs->write_options.block_size / 2);\n+\t\t}\n+\n+\t\tif (u->flags & REF_LOG_ONLY)\n+\t\t\tcontinue;\n+\n+\t\tif (u->flags & REF_HAVE_NEW && is_null_oid(&u->new_oid)) {\n+\t\t\tstruct reftable_ref_record ref = {\n+\t\t\t\t.refname = (char *)u->refname,\n+\t\t\t\t.update_index = ts,\n+\t\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t};\n+\n+\t\t\tret = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto done;\n+\t\t} else if (u->flags & REF_HAVE_NEW) {\n+\t\t\tstruct reftable_ref_record ref = {0};\n+\t\t\tstruct object_id peeled;\n+\t\t\tint peel_error;\n+\n+\t\t\tref.refname = (char *)u->refname;\n+\t\t\tref.update_index = ts;\n+\n+\t\t\tpeel_error = peel_object(&u->new_oid, &peeled);\n+\t\t\tif (!peel_error) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\t\t\tmemcpy(ref.value.val2.target_value, peeled.hash, GIT_MAX_RAWSZ);\n+\t\t\t\tmemcpy(ref.value.val2.value, u->new_oid.hash, GIT_MAX_RAWSZ);\n+\t\t\t} else if (!is_null_oid(&u->new_oid)) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\t\t\tmemcpy(ref.value.val1, u->new_oid.hash, GIT_MAX_RAWSZ);\n+\t\t\t}\n+\n+\t\t\tret = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Logs are written at the end so that we do not have intermixed ref\n+\t * and log blocks.\n+\t */\n+\tif (logs) {\n+\t\tret = reftable_writer_add_logs(writer, logs, logs_nr);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\tfor (i = 0; i < logs_nr; i++)\n+\t\tclear_reftable_log_record(&logs[i]);\n+\tfree(logs);\n+\treturn ret;\n+}\n+\n+static int reftable_be_transaction_finish(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_transaction_data *tx_data = transaction->backend_data;\n+\tint ret = 0;\n+\n+\tfor (size_t i = 0; i < tx_data->args_nr; i++) {\n+\t\tret = reftable_addition_add(tx_data->args[i].addition,\n+\t\t\t\t\t write_transaction_table, &tx_data->args[i]);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\n+\t\tret = reftable_addition_commit(tx_data->args[i].addition);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\tfree_transaction_data(tx_data);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\n+\tif (ret) {\n+\t\tstrbuf_addf(err, _(\"reftable: transaction failure: %s\"),\n+\t\t\t reftable_error_str(ret));\n+\t\treturn -1;\n+\t}\n+\treturn ret;\n+}\n+\n+static int reftable_be_initial_transaction_commit(struct ref_store *ref_store UNUSED,\n+\t\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t\t struct strbuf *err)\n+{\n+\treturn ref_transaction_commit(transaction, err);\n+}\n+\n+static int reftable_be_pack_refs(struct ref_store *ref_store,\n+\t\t\t\t struct pack_refs_opts *opts)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE | REF_STORE_ODB, \"pack_refs\");\n+\tstruct reftable_stack *stack;\n+\tint ret;\n+\n+\tif (refs->err)\n+\t\treturn refs->err;\n+\n+\tstack = refs->worktree_stack;\n+\tif (!stack)\n+\t\tstack = refs->main_stack;\n+\n+\tret = reftable_stack_compact_all(stack, NULL);\n+\tif (ret)\n+\t\tgoto out;\n+\tret = reftable_stack_clean(stack);\n+\tif (ret)\n+\t\tgoto out;\n+\n+out:\n+\treturn ret;\n+}\n+\n+struct write_create_symref_arg {\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+\tconst char *target;\n+\tconst char *logmsg;\n+};\n+\n+static int write_create_symref_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct write_create_symref_arg *create = cb_data;\n+\tuint64_t ts = reftable_stack_next_update_index(create->stack);\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)create->refname,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = (char *)create->target,\n+\t\t.update_index = ts,\n+\t};\n+\tstruct reftable_log_record log = {0};\n+\tstruct object_id new_oid;\n+\tstruct object_id old_oid;\n+\tint ret;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tret = reftable_writer_add_ref(writer, &ref);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\t/*\n+\t * Note that it is important to try and resolve the reference before we\n+\t * write the log entry. This is because `should_write_log()` will munge\n+\t * `core.logAllRefUpdates`, which is undesirable when we create a new\n+\t * repository because it would be written into the config. As HEAD will\n+\t * not resolve for new repositories this ordering will ensure that this\n+\t * never happens.\n+\t */\n+\tif (!create->logmsg ||\n+\t !refs_resolve_ref_unsafe(&create->refs->base, create->target,\n+\t\t\t\t RESOLVE_REF_READING, &new_oid, NULL) ||\n+\t !should_write_log(&create->refs->base, create->refname))\n+\t\treturn 0;\n+\n+\tfill_reftable_log_record(&log);\n+\tlog.refname = xstrdup(create->refname);\n+\tlog.update_index = ts;\n+\tlog.value.update.message = xstrndup(create->logmsg,\n+\t\t\t\t\t create->refs->write_options.block_size / 2);\n+\tlog.value.update.new_hash = new_oid.hash;\n+\tif (refs_resolve_ref_unsafe(&create->refs->base, create->refname,\n+\t\t\t\t RESOLVE_REF_READING, &old_oid, NULL))\n+\t\tlog.value.update.old_hash = old_oid.hash;\n+\n+\tret = reftable_writer_add_log(writer, &log);\n+\tclear_reftable_log_record(&log);\n+\treturn ret;\n+}\n+\n+static int reftable_be_create_symref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const char *target,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"create_symref\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct write_create_symref_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t\t.target = target,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_add(stack, &write_create_symref_table, &arg);\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\tif (ret)\n+\t\terror(\"unable to write symref for %s: %s\", refname,\n+\t\t reftable_error_str(ret));\n+\treturn ret;\n+}\n+\n+struct write_copy_arg {\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *oldname;\n+\tconst char *newname;\n+\tconst char *logmsg;\n+\tint delete_old;\n+};\n+\n+static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct write_copy_arg *arg = cb_data;\n+\tuint64_t deletion_ts, creation_ts;\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(arg->stack);\n+\tstruct reftable_ref_record old_ref = {0}, refs[2] = {0};\n+\tstruct reftable_log_record old_log = {0}, *logs = NULL;\n+\tstruct reftable_iterator it = {0};\n+\tstruct string_list skip = STRING_LIST_INIT_NODUP;\n+\tstruct strbuf errbuf = STRBUF_INIT;\n+\tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tint ret;\n+\n+\tif (reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref)) {\n+\t\tret = error(_(\"refname %s not found\"), arg->oldname);\n+\t\tgoto done;\n+\t}\n+\tif (old_ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tret = error(_(\"refname %s is a symbolic ref, copying it is not supported\"),\n+\t\t\t arg->oldname);\n+\t\tgoto done;\n+\t}\n+\n+\t/*\n+\t * There's nothing to do in case the old and new name are the same, so\n+\t * we exit early in that case.\n+\t */\n+\tif (!strcmp(arg->oldname, arg->newname)) {\n+\t\tret = 0;\n+\t\tgoto done;\n+\t}\n+\n+\t/*\n+\t * Verify that the new refname is available.\n+\t */\n+\tstring_list_insert(&skip, arg->oldname);\n+\tret = refs_verify_refname_available(&arg->refs->base, arg->newname,\n+\t\t\t\t\t NULL, &skip, &errbuf);\n+\tif (ret < 0) {\n+\t\terror(\"%s\", errbuf.buf);\n+\t\tgoto done;\n+\t}\n+\n+\t/*\n+\t * When deleting the old reference we have to use two update indices:\n+\t * once to delete the old ref and its reflog, and once to create the\n+\t * new ref and its reflog. They need to be staged with two separate\n+\t * indices because the new reflog needs to encode both the deletion of\n+\t * the old branch and the creation of the new branch, and we cannot do\n+\t * two changes to a reflog in a single update.\n+\t */\n+\tdeletion_ts = creation_ts = reftable_stack_next_update_index(arg->stack);\n+\tif (arg->delete_old)\n+\t\tcreation_ts++;\n+\treftable_writer_set_limits(writer, deletion_ts, creation_ts);\n+\n+\t/*\n+\t * Add the new reference. If this is a rename then we also delete the\n+\t * old reference.\n+\t */\n+\trefs[0] = old_ref;\n+\trefs[0].refname = (char *)arg->newname;\n+\trefs[0].update_index = creation_ts;\n+\tif (arg->delete_old) {\n+\t\trefs[1].refname = (char *)arg->oldname;\n+\t\trefs[1].value_type = REFTABLE_REF_DELETION;\n+\t\trefs[1].update_index = deletion_ts;\n+\t}\n+\tret = reftable_writer_add_refs(writer, refs, arg->delete_old ? 2 : 1);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\t/*\n+\t * When deleting the old branch we need to create a reflog entry on the\n+\t * new branch name that indicates that the old branch has been deleted\n+\t * and then recreated. This is a tad weird, but matches what the files\n+\t * backend does.\n+\t */\n+\tif (arg->delete_old) {\n+\t\tstruct strbuf head_referent = STRBUF_INIT;\n+\t\tstruct object_id head_oid;\n+\t\tint append_head_reflog;\n+\t\tunsigned head_type = 0;\n+\n+\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\tmemset(&logs[logs_nr], 0, sizeof(logs[logs_nr]));\n+\t\tfill_reftable_log_record(&logs[logs_nr]);\n+\t\tlogs[logs_nr].refname = (char *)arg->newname;\n+\t\tlogs[logs_nr].update_index = deletion_ts;\n+\t\tlogs[logs_nr].value.update.message =\n+\t\t\txstrndup(arg->logmsg, arg->refs->write_options.block_size / 2);\n+\t\tlogs[logs_nr].value.update.old_hash = old_ref.value.val1;\n+\t\tlogs_nr++;\n+\n+\t\tret = read_ref_without_reload(arg->stack, \"HEAD\", &head_oid, &head_referent, &head_type);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tappend_head_reflog = (head_type & REF_ISSYMREF) && !strcmp(head_referent.buf, arg->oldname);\n+\t\tstrbuf_release(&head_referent);\n+\n+\t\t/*\n+\t\t * The files backend uses `refs_delete_ref()` to delete the old\n+\t\t * branch name, which will append a reflog entry for HEAD in\n+\t\t * case it points to the old branch.\n+\t\t */\n+\t\tif (append_head_reflog) {\n+\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\t\tlogs[logs_nr] = logs[logs_nr - 1];\n+\t\t\tlogs[logs_nr].refname = \"HEAD\";\n+\t\t\tlogs_nr++;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Create the reflog entry for the newly created branch.\n+\t */\n+\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\tmemset(&logs[logs_nr], 0, sizeof(logs[logs_nr]));\n+\tfill_reftable_log_record(&logs[logs_nr]);\n+\tlogs[logs_nr].refname = (char *)arg->newname;\n+\tlogs[logs_nr].update_index = creation_ts;\n+\tlogs[logs_nr].value.update.message =\n+\t\txstrndup(arg->logmsg, arg->refs->write_options.block_size / 2);\n+\tlogs[logs_nr].value.update.new_hash = old_ref.value.val1;\n+\tlogs_nr++;\n+\n+\t/*\n+\t * In addition to writing the reflog entry for the new branch, we also\n+\t * copy over all log entries from the old reflog. Last but not least,\n+\t * when renaming we also have to delete all the old reflog entries.\n+\t */\n+\tret = reftable_merged_table_seek_log(mt, &it, arg->oldname);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\twhile (1) {\n+\t\tret = reftable_iterator_next_log(&it, &old_log);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tif (ret > 0 || strcmp(old_log.refname, arg->oldname)) {\n+\t\t\tret = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tfree(old_log.refname);\n+\n+\t\t/*\n+\t\t * Copy over the old reflog entry with the new refname.\n+\t\t */\n+\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\tlogs[logs_nr] = old_log;\n+\t\tlogs[logs_nr].refname = (char *)arg->newname;\n+\t\tlogs_nr++;\n+\n+\t\t/*\n+\t\t * Delete the old reflog entry in case we are renaming.\n+\t\t */\n+\t\tif (arg->delete_old) {\n+\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\t\tmemset(&logs[logs_nr], 0, sizeof(logs[logs_nr]));\n+\t\t\tlogs[logs_nr].refname = (char *)arg->oldname;\n+\t\t\tlogs[logs_nr].value_type = REFTABLE_LOG_DELETION;\n+\t\t\tlogs[logs_nr].update_index = old_log.update_index;\n+\t\t\tlogs_nr++;\n+\t\t}\n+\n+\t\t/*\n+\t\t * Transfer ownership of the log record we're iterating over to\n+\t\t * the array of log records. Otherwise, the pointers would get\n+\t\t * free'd or reallocated by the iterator.\n+\t\t */\n+\t\tmemset(&old_log, 0, sizeof(old_log));\n+\t}\n+\n+\tret = reftable_writer_add_logs(writer, logs, logs_nr);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treftable_iterator_destroy(&it);\n+\tstring_list_clear(&skip, 0);\n+\tstrbuf_release(&errbuf);\n+\tfor (i = 0; i < logs_nr; i++) {\n+\t\tif (!strcmp(logs[i].refname, \"HEAD\"))\n+\t\t\tcontinue;\n+\t\tif (logs[i].value.update.old_hash == old_ref.value.val1)\n+\t\t\tlogs[i].value.update.old_hash = NULL;\n+\t\tif (logs[i].value.update.new_hash == old_ref.value.val1)\n+\t\t\tlogs[i].value.update.new_hash = NULL;\n+\t\tlogs[i].refname = NULL;\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tfree(logs);\n+\treftable_ref_record_release(&old_ref);\n+\treftable_log_record_release(&old_log);\n+\treturn ret;\n+}\n+\n+static int reftable_be_rename_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname,\n+\t\t\t\t const char *newrefname,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"rename_ref\");\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname, &newrefname);\n+\tstruct write_copy_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t\t.delete_old = 1,\n+\t};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\tret = reftable_stack_add(stack, &write_copy_table, &arg);\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treturn ret;\n+}\n+\n+static int reftable_be_copy_ref(struct ref_store *ref_store,\n+\t\t\t\tconst char *oldrefname,\n+\t\t\t\tconst char *newrefname,\n+\t\t\t\tconst char *logmsg)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"copy_ref\");\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname, &newrefname);\n+\tstruct write_copy_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\tret = reftable_stack_add(stack, &write_copy_table, &arg);\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treturn ret;\n+}\n+\n+struct reftable_reflog_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_log_record log;\n+\tstruct object_id oid;\n+\tchar *last_name;\n+\tint err;\n+};\n+\n+static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct reftable_reflog_iterator *iter =\n+\t\t(struct reftable_reflog_iterator *)ref_iterator;\n+\n+\twhile (!iter->err) {\n+\t\tint flags;\n+\n+\t\titer->err = reftable_iterator_next_log(&iter->iter, &iter->log);\n+\t\tif (iter->err)\n+\t\t\tbreak;\n+\n+\t\t/*\n+\t\t * We want the refnames that we have reflogs for, so we skip if\n+\t\t * we've already produced this name. This could be faster by\n+\t\t * seeking directly to reflog@update_index==0.\n+\t\t */\n+\t\tif (iter->last_name && !strcmp(iter->log.refname, iter->last_name))\n+\t\t\tcontinue;\n+\n+\t\tif (!refs_resolve_ref_unsafe(&iter->refs->base, iter->log.refname,\n+\t\t\t\t\t 0, &iter->oid, &flags)) {\n+\t\t\terror(_(\"bad ref for %s\"), iter->log.refname);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfree(iter->last_name);\n+\t\titer->last_name = xstrdup(iter->log.refname);\n+\t\titer->base.refname = iter->log.refname;\n+\t\titer->base.oid = &iter->oid;\n+\t\titer->base.flags = flags;\n+\n+\t\tbreak;\n+\t}\n+\n+\tif (iter->err > 0) {\n+\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n+\t\t\treturn ITER_ERROR;\n+\t\treturn ITER_DONE;\n+\t}\n+\n+\tif (iter->err < 0) {\n+\t\tref_iterator_abort(ref_iterator);\n+\t\treturn ITER_ERROR;\n+\t}\n+\n+\treturn ITER_OK;\n+}\n+\n+static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t\t\t struct object_id *peeled)\n+{\n+\tBUG(\"reftable reflog iterator cannot be peeled\");\n+\treturn -1;\n+}\n+\n+static int reftable_reflog_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct reftable_reflog_iterator *iter =\n+\t\t(struct reftable_reflog_iterator *)ref_iterator;\n+\treftable_log_record_release(&iter->log);\n+\treftable_iterator_destroy(&iter->iter);\n+\tfree(iter->last_name);\n+\tfree(iter);\n+\treturn ITER_DONE;\n+}\n+\n+static struct ref_iterator_vtable reftable_reflog_iterator_vtable = {\n+\t.advance = reftable_reflog_iterator_advance,\n+\t.peel = reftable_reflog_iterator_peel,\n+\t.abort = reftable_reflog_iterator_abort\n+};\n+\n+static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftable_ref_store *refs,\n+\t\t\t\t\t\t\t\t struct reftable_stack *stack)\n+{\n+\tstruct reftable_merged_table *merged_table;\n+\tstruct reftable_reflog_iterator *iter;\n+\tint ret;\n+\n+\titer = xcalloc(1, sizeof(*iter));\n+\tbase_ref_iterator_init(&iter->base, &reftable_reflog_iterator_vtable, 1);\n+\titer->refs = refs;\n+\titer->base.oid = &iter->oid;\n+\n+\tret = refs->err;\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(refs->main_stack);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tmerged_table = reftable_stack_merged_table(stack);\n+\n+\tret = reftable_merged_table_seek_log(merged_table, &iter->iter, \"\");\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+done:\n+\titer->err = ret;\n+\treturn iter;\n+}\n+\n+static struct ref_iterator *reftable_be_reflog_iterator_begin(struct ref_store *ref_store)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_iterator_begin\");\n+\tstruct reftable_reflog_iterator *main_iter, *worktree_iter;\n+\n+\tmain_iter = reflog_iterator_for_stack(refs, refs->main_stack);\n+\tif (!refs->worktree_stack)\n+\t\treturn &main_iter->base;\n+\n+\tworktree_iter = reflog_iterator_for_stack(refs, refs->worktree_stack);\n+\n+\treturn merge_ref_iterator_begin(1, &worktree_iter->base, &main_iter->base,\n+\t\t\t\t\titerator_select, NULL);\n+}\n+\n+static int yield_log_record(struct reftable_log_record *log,\n+\t\t\t each_reflog_ent_fn fn,\n+\t\t\t void *cb_data)\n+{\n+\tstruct object_id old_oid, new_oid;\n+\tconst char *full_committer;\n+\n+\toidread(&old_oid, log->value.update.old_hash);\n+\toidread(&new_oid, log->value.update.new_hash);\n+\n+\t/*\n+\t * When both the old object ID and the new object ID are null\n+\t * then this is the reflog existence marker. The caller must\n+\t * not be aware of it.\n+\t */\n+\tif (is_null_oid(&old_oid) && is_null_oid(&new_oid))\n+\t\treturn 0;\n+\n+\tfull_committer = fmt_ident(log->value.update.name, log->value.update.email,\n+\t\t\t\t WANT_COMMITTER_IDENT, NULL, IDENT_NO_DATE);\n+\treturn fn(&old_oid, &new_oid, full_committer,\n+\t\t log->value.update.time, log->value.update.tz_offset,\n+\t\t log->value.update.message, cb_data);\n+}\n+\n+static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n+\t\t\t\t\t\t const char *refname,\n+\t\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t\t void *cb_data)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent_reverse\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record log = {0};\n+\tstruct reftable_iterator it = {0};\n+\tint ret;\n+\n+\tif (refs->err < 0)\n+\t\treturn refs->err;\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\tret = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (!ret) {\n+\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\tif (ret < 0)\n+\t\t\tbreak;\n+\t\tif (ret > 0 || strcmp(log.refname, refname)) {\n+\t\t\tret = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tret = yield_log_record(&log, fn, cb_data);\n+\t\tif (ret)\n+\t\t\tbreak;\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn ret;\n+}\n+\n+static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store,\n+\t\t\t\t\t const char *refname,\n+\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t void *cb_data)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record *logs = NULL;\n+\tstruct reftable_iterator it = {0};\n+\tsize_t logs_alloc = 0, logs_nr = 0, i;\n+\tint ret;\n+\n+\tif (refs->err < 0)\n+\t\treturn refs->err;\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\tret = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (!ret) {\n+\t\tstruct reftable_log_record log = {0};\n+\n+\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tif (ret > 0 || strcmp(log.refname, refname)) {\n+\t\t\treftable_log_record_release(&log);\n+\t\t\tret = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\tlogs[logs_nr++] = log;\n+\t}\n+\n+\tfor (i = logs_nr; i--;) {\n+\t\tret = yield_log_record(&logs[i], fn, cb_data);\n+\t\tif (ret)\n+\t\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\tfor (i = 0; i < logs_nr; i++)\n+\t\treftable_log_record_release(&logs[i]);\n+\tfree(logs);\n+\treturn ret;\n+}\n+\n+static int reftable_be_reflog_exists(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_exists\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record log = {0};\n+\tstruct reftable_iterator it = {0};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\t/*\n+\t * Check whether we get at least one log record for the given ref name.\n+\t * If so, the reflog exists, otherwise it doesn't.\n+\t */\n+\tret = reftable_iterator_next_log(&it, &log);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\tif (ret > 0) {\n+\t\tret = 0;\n+\t\tgoto done;\n+\t}\n+\n+\tret = strcmp(log.refname, refname) == 0;\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\tif (ret < 0)\n+\t\tret = 0;\n+\treturn ret;\n+}\n+\n+struct write_reflog_existence_arg {\n+\tstruct reftable_ref_store *refs;\n+\tconst char *refname;\n+\tstruct reftable_stack *stack;\n+};\n+\n+static int write_reflog_existence_table(struct reftable_writer *writer,\n+\t\t\t\t\tvoid *cb_data)\n+{\n+\tstruct write_reflog_existence_arg *arg = cb_data;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_log_record log = {0};\n+\tint ret;\n+\n+\tret = reftable_stack_read_log(arg->stack, arg->refname, &log);\n+\tif (ret <= 0)\n+\t\tgoto done;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\t/*\n+\t * The existence entry has both old and new object ID set to the the\n+\t * null object ID. Our iterators are aware of this and will not present\n+\t * them to their callers.\n+\t */\n+\tlog.refname = xstrdup(arg->refname);\n+\tlog.update_index = ts;\n+\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\tret = reftable_writer_add_log(writer, &log);\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treftable_log_record_release(&log);\n+\treturn ret;\n+}\n+\n+static int reftable_be_create_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname,\n+\t\t\t\t struct strbuf *errmsg)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"create_reflog\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct write_reflog_existence_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_add(stack, &write_reflog_existence_table, &arg);\n+\n+done:\n+\treturn ret;\n+}\n+\n+struct write_reflog_delete_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+};\n+\n+static int write_reflog_delete_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct write_reflog_delete_arg *arg = cb_data;\n+\tstruct reftable_merged_table *mt =\n+\t\treftable_stack_merged_table(arg->stack);\n+\tstruct reftable_log_record log = {0}, tombstone = {0};\n+\tstruct reftable_iterator it = {0};\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint ret;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\t/*\n+\t * In order to delete a table we need to delete all reflog entries one\n+\t * by one. This is inefficient, but the reftable format does not have a\n+\t * better marker right now.\n+\t */\n+\tret = reftable_merged_table_seek_log(mt, &it, arg->refname);\n+\twhile (ret == 0) {\n+\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\tif (ret < 0)\n+\t\t\tbreak;\n+\t\tif (ret > 0 || strcmp(log.refname, arg->refname)) {\n+\t\t\tret = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttombstone.refname = (char *)arg->refname;\n+\t\ttombstone.value_type = REFTABLE_LOG_DELETION;\n+\t\ttombstone.update_index = log.update_index;\n+\n+\t\tret = reftable_writer_add_log(writer, &tombstone);\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn ret;\n+}\n+\n+static int reftable_be_delete_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"delete_reflog\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct write_reflog_delete_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tint ret;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\treturn ret;\n+\tret = reftable_stack_add(stack, &write_reflog_delete_table, &arg);\n+\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treturn ret;\n+}\n+\n+struct reflog_expiry_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_log_record *records;\n+\tstruct object_id update_oid;\n+\tconst char *refname;\n+\tsize_t len;\n+};\n+\n+static int write_reflog_expiry_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct reflog_expiry_arg *arg = cb_data;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tuint64_t live_records = 0;\n+\tsize_t i;\n+\tint ret;\n+\n+\tfor (i = 0; i < arg->len; i++)\n+\t\tif (arg->records[i].value_type == REFTABLE_LOG_UPDATE)\n+\t\t\tlive_records++;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tif (!is_null_oid(&arg->update_oid)) {\n+\t\tstruct reftable_ref_record ref = {0};\n+\t\tstruct object_id peeled;\n+\n+\t\tref.refname = (char *)arg->refname;\n+\t\tref.update_index = ts;\n+\n+\t\tif (!peel_object(&arg->update_oid, &peeled)) {\n+\t\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\t\tmemcpy(ref.value.val2.target_value, peeled.hash, GIT_MAX_RAWSZ);\n+\t\t\tmemcpy(ref.value.val2.value, arg->update_oid.hash, GIT_MAX_RAWSZ);\n+\t\t} else {\n+\t\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\t\tmemcpy(ref.value.val1, arg->update_oid.hash, GIT_MAX_RAWSZ);\n+\t\t}\n+\n+\t\tret = reftable_writer_add_ref(writer, &ref);\n+\t\tif (ret < 0)\n+\t\t\treturn ret;\n+\t}\n+\n+\t/*\n+\t * When there are no more entries left in the reflog we empty it\n+\t * completely, but write a placeholder reflog entry that indicates that\n+\t * the reflog still exists.\n+\t */\n+\tif (!live_records) {\n+\t\tstruct reftable_log_record log = {\n+\t\t\t.refname = (char *)arg->refname,\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.update_index = ts,\n+\t\t};\n+\n+\t\tret = reftable_writer_add_log(writer, &log);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\n+\tfor (i = 0; i < arg->len; i++) {\n+\t\tret = reftable_writer_add_log(writer, &arg->records[i]);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int reftable_be_reflog_expire(struct ref_store *ref_store,\n+\t\t\t\t const char *refname,\n+\t\t\t\t unsigned int flags,\n+\t\t\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t\t void *policy_cb_data)\n+{\n+\t/*\n+\t * For log expiry, we write tombstones for every single reflog entry\n+\t * that is to be expired. This means that the entries are still\n+\t * retrievable by delving into the stack, and expiring entries\n+\t * paradoxically takes extra memory. This memory is only reclaimed when\n+\t * compacting the reftable stack.\n+\t *\n+\t * It would be better if the refs backend supported an API that sets a\n+\t * criterion for all refs, passing the criterion to pack_refs().\n+\t *\n+\t * On the plus side, because we do the expiration per ref, we can easily\n+\t * insert the reflog existence dummies.\n+\t */\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"reflog_expire\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record *logs = NULL;\n+\tstruct reftable_log_record *rewritten = NULL;\n+\tstruct reftable_ref_record ref_record = {0};\n+\tstruct reftable_iterator it = {0};\n+\tstruct reftable_addition *add = NULL;\n+\tstruct reflog_expiry_arg arg = {0};\n+\tstruct object_id oid = {0};\n+\tuint8_t *last_hash = NULL;\n+\tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tint ret;\n+\n+\tif (refs->err < 0)\n+\t\treturn refs->err;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_new_addition(&add, stack);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_read_ref(stack, refname, &ref_record);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\tif (reftable_ref_record_val1(&ref_record))\n+\t\toidread(&oid, reftable_ref_record_val1(&ref_record));\n+\tprepare_fn(refname, &oid, policy_cb_data);\n+\n+\twhile (1) {\n+\t\tstruct reftable_log_record log = {0};\n+\t\tstruct object_id old_oid, new_oid;\n+\n+\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tif (ret > 0 || strcmp(log.refname, refname)) {\n+\t\t\treftable_log_record_release(&log);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\toidread(&old_oid, log.value.update.old_hash);\n+\t\toidread(&new_oid, log.value.update.new_hash);\n+\n+\t\t/*\n+\t\t * Skip over the reflog existence marker. We will add it back\n+\t\t * in when there are no live reflog records.\n+\t\t */\n+\t\tif (is_null_oid(&old_oid) && is_null_oid(&new_oid)) {\n+\t\t\treftable_log_record_release(&log);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\tlogs[logs_nr++] = log;\n+\t}\n+\n+\t/*\n+\t * We need to rewrite all reflog entries according to the pruning\n+\t * callback function:\n+\t *\n+\t * - If a reflog entry shall be pruned we mark the record for\n+\t * deletion.\n+\t *\n+\t * - Otherwise we may have to rewrite the chain of reflog entries so\n+\t * that gaps created by just-deleted records get backfilled.\n+\t */\n+\tCALLOC_ARRAY(rewritten, logs_nr);\n+\tfor (i = logs_nr; i--;) {\n+\t\tstruct reftable_log_record *dest = &rewritten[i];\n+\t\tstruct object_id old_oid, new_oid;\n+\n+\t\t*dest = logs[i];\n+\t\toidread(&old_oid, logs[i].value.update.old_hash);\n+\t\toidread(&new_oid, logs[i].value.update.new_hash);\n+\n+\t\tif (should_prune_fn(&old_oid, &new_oid, logs[i].value.update.email,\n+\t\t\t\t (timestamp_t)logs[i].value.update.time,\n+\t\t\t\t logs[i].value.update.tz_offset,\n+\t\t\t\t logs[i].value.update.message,\n+\t\t\t\t policy_cb_data)) {\n+\t\t\tdest->value_type = REFTABLE_LOG_DELETION;\n+\t\t} else {\n+\t\t\tif ((flags & EXPIRE_REFLOGS_REWRITE) && last_hash)\n+\t\t\t\tdest->value.update.old_hash = last_hash;\n+\t\t\tlast_hash = logs[i].value.update.new_hash;\n+\t\t}\n+\t}\n+\n+\tif (flags & EXPIRE_REFLOGS_UPDATE_REF && last_hash &&\n+\t reftable_ref_record_val1(&ref_record))\n+\t\toidread(&arg.update_oid, last_hash);\n+\n+\targ.records = rewritten;\n+\targ.len = logs_nr;\n+\targ.stack = stack,\n+\targ.refname = refname,\n+\n+\tret = reftable_addition_add(add, &write_reflog_expiry_table, &arg);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\t/*\n+\t * Future improvement: we could skip writing records that were\n+\t * not changed.\n+\t */\n+\tif (!(flags & EXPIRE_REFLOGS_DRY_RUN))\n+\t\tret = reftable_addition_commit(add);\n+\n+done:\n+\tif (add)\n+\t\tcleanup_fn(policy_cb_data);\n+\tassert(ret != REFTABLE_API_ERROR);\n+\n+\treftable_ref_record_release(&ref_record);\n+\treftable_iterator_destroy(&it);\n+\treftable_addition_destroy(add);\n+\tfor (i = 0; i < logs_nr; i++)\n+\t\treftable_log_record_release(&logs[i]);\n+\tfree(logs);\n+\tfree(rewritten);\n+\treturn ret;\n+}\n+\n+struct ref_storage_be refs_be_reftable = {\n+\t.name = \"reftable\",\n+\t.init = reftable_be_init,\n+\t.init_db = reftable_be_init_db,\n+\t.transaction_prepare = reftable_be_transaction_prepare,\n+\t.transaction_finish = reftable_be_transaction_finish,\n+\t.transaction_abort = reftable_be_transaction_abort,\n+\t.initial_transaction_commit = reftable_be_initial_transaction_commit,\n+\n+\t.pack_refs = reftable_be_pack_refs,\n+\t.create_symref = reftable_be_create_symref,\n+\t.rename_ref = reftable_be_rename_ref,\n+\t.copy_ref = reftable_be_copy_ref,\n+\n+\t.iterator_begin = reftable_be_iterator_begin,\n+\t.read_raw_ref = reftable_be_read_raw_ref,\n+\t.read_symbolic_ref = reftable_be_read_symbolic_ref,\n+\n+\t.reflog_iterator_begin = reftable_be_reflog_iterator_begin,\n+\t.for_each_reflog_ent = reftable_be_for_each_reflog_ent,\n+\t.for_each_reflog_ent_reverse = reftable_be_for_each_reflog_ent_reverse,\n+\t.reflog_exists = reftable_be_reflog_exists,\n+\t.create_reflog = reftable_be_create_reflog,\n+\t.delete_reflog = reftable_be_delete_reflog,\n+\t.reflog_expire = reftable_be_reflog_expire,\n+};\ndiff --git a/repository.h b/repository.h\nindex 7a250a6605..1645cef518 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -24,8 +24,9 @@ enum fetch_negotiation_setting {\n \tFETCH_NEGOTIATION_NOOP,\n };\n \n-#define REF_STORAGE_FORMAT_UNKNOWN 0\n-#define REF_STORAGE_FORMAT_FILES 1\n+#define REF_STORAGE_FORMAT_UNKNOWN 0\n+#define REF_STORAGE_FORMAT_FILES 1\n+#define REF_STORAGE_FORMAT_REFTABLE 2\n \n struct repo_settings {\n \tint initialized;\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nnew file mode 100755\nindex 0000000000..94ee46984a\n--- /dev/null\n+++ b/t/t0610-reftable-basics.sh\n@@ -0,0 +1,887 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable basics'\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+if ! test_have_prereq REFTABLE\n+then\n+\tskip_all='skipping reftable tests; set GIT_TEST_DEFAULT_REF_FORMAT=reftable'\n+\ttest_done\n+fi\n+\n+INVALID_OID=$(test_oid 001)\n+\n+test_expect_success 'init: creates basic reftable structures' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_path_is_dir repo/.git/reftable &&\n+\ttest_path_is_file repo/.git/reftable/tables.list &&\n+\techo reftable >expect &&\n+\tgit -C repo rev-parse --show-ref-format >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init: sha256 object format via environment variable' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tGIT_DEFAULT_HASH=sha256 git init repo &&\n+\tcat >expect <<-EOF &&\n+\tsha256\n+\treftable\n+\tEOF\n+\tgit -C repo rev-parse --show-object-format --show-ref-format >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init: sha256 object format via option' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init --object-format=sha256 repo &&\n+\tcat >expect <<-EOF &&\n+\tsha256\n+\treftable\n+\tEOF\n+\tgit -C repo rev-parse --show-object-format --show-ref-format >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init: reinitializing reftable backend succeeds' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo for-each-ref >expect &&\n+\tgit init --ref-format=reftable repo &&\n+\tgit -C repo for-each-ref >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init: reinitializing files with reftable backend fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init --ref-format=files repo &&\n+\ttest_commit -C repo file &&\n+\n+\tcp repo/.git/HEAD expect &&\n+\ttest_must_fail git init --ref-format=reftable repo &&\n+\ttest_cmp expect repo/.git/HEAD\n+'\n+\n+test_expect_success 'init: reinitializing reftable with files backend fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init --ref-format=reftable repo &&\n+\ttest_commit -C repo file &&\n+\n+\tcp repo/.git/HEAD expect &&\n+\ttest_must_fail git init --ref-format=files repo &&\n+\ttest_cmp expect repo/.git/HEAD\n+'\n+\n+test_expect_perms () {\n+\tlocal perms=\"$1\"\n+\tlocal file=\"$2\"\n+\tlocal actual=$(ls -l \"$file\") &&\n+\n+\tcase \"$actual\" in\n+\t$perms*)\n+\t\t: happy\n+\t\t;;\n+\t*)\n+\t\techo \"$(basename $2) is not $perms but $actual\"\n+\t\tfalse\n+\t\t;;\n+\tesac\n+}\n+\n+for umask in 002 022\n+do\n+\ttest_expect_success POSIXPERM 'init: honors core.sharedRepository' '\n+\t\ttest_when_finished \"rm -rf repo\" &&\n+\t\t(\n+\t\t\tumask $umask &&\n+\t\t\tgit init --shared=true repo &&\n+\t\t\ttest 1 = \"$(git -C repo config core.sharedrepository)\"\n+\t\t) &&\n+\t\ttest_expect_perms \"-rw-rw-r--\" repo/.git/reftable/tables.list &&\n+\t\tfor table in repo/.git/reftable/*.ref\n+\t\tdo\n+\t\t\ttest_expect_perms \"-rw-rw-r--\" \"$table\" ||\n+\t\t\treturn 1\n+\t\tdone\n+\t'\n+done\n+\n+test_expect_success 'clone: can clone reftable repository' '\n+\ttest_when_finished \"rm -rf repo clone\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo message1 file1 &&\n+\n+\tgit clone repo cloned &&\n+\techo reftable >expect &&\n+\tgit -C cloned rev-parse --show-ref-format >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_path_is_file cloned/file1\n+'\n+\n+test_expect_success 'clone: can clone reffiles into reftable repository' '\n+\ttest_when_finished \"rm -rf reffiles reftable\" &&\n+\tgit init --ref-format=files reffiles &&\n+\ttest_commit -C reffiles A &&\n+\tgit clone --ref-format=reftable ./reffiles reftable &&\n+\n+\tgit -C reffiles rev-parse HEAD >expect &&\n+\tgit -C reftable rev-parse HEAD >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C reftable rev-parse --show-ref-format >actual &&\n+\techo reftable >expect &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C reffiles rev-parse --show-ref-format >actual &&\n+\techo files >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'clone: can clone reftable into reffiles repository' '\n+\ttest_when_finished \"rm -rf reffiles reftable\" &&\n+\tgit init --ref-format=reftable reftable &&\n+\ttest_commit -C reftable A &&\n+\tgit clone --ref-format=files ./reftable reffiles &&\n+\n+\tgit -C reftable rev-parse HEAD >expect &&\n+\tgit -C reffiles rev-parse HEAD >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C reftable rev-parse --show-ref-format >actual &&\n+\techo reftable >expect &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C reffiles rev-parse --show-ref-format >actual &&\n+\techo files >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'ref transaction: corrupted tables cause failure' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file1 &&\n+\t\tfor f in .git/reftable/*.ref\n+\t\tdo\n+\t\t\t: >\"$f\" || return 1\n+\t\tdone &&\n+\t\ttest_must_fail git update-ref refs/heads/main HEAD\n+\t)\n+'\n+\n+test_expect_success 'ref transaction: corrupted tables.list cause failure' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file1 &&\n+\t\techo garbage >.git/reftable/tables.list &&\n+\t\ttest_must_fail git update-ref refs/heads/main HEAD\n+\t)\n+'\n+\n+test_expect_success 'ref transaction: refuses to write ref causing F/D conflict' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo file &&\n+\ttest_must_fail git -C repo update-ref refs/heads/main/forbidden\n+'\n+\n+test_expect_success 'ref transaction: deleting ref with invalid name fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo file &&\n+\ttest_must_fail git -C repo update-ref -d ../../my-private-file\n+'\n+\n+test_expect_success 'ref transaction: can skip object ID verification' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_must_fail test-tool -C repo ref-store main update-ref msg refs/heads/branch $INVALID_OID $ZERO_OID 0 &&\n+\ttest-tool -C repo ref-store main update-ref msg refs/heads/branch $INVALID_OID $ZERO_OID REF_SKIP_OID_VERIFICATION\n+'\n+\n+test_expect_success 'ref transaction: updating same ref multiple times fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tcat >updates <<-EOF &&\n+\tupdate refs/heads/main $A\n+\tupdate refs/heads/main $A\n+\tEOF\n+\tcat >expect <<-EOF &&\n+\tfatal: multiple updates for ref ${SQ}refs/heads/main${SQ} not allowed\n+\tEOF\n+\ttest_must_fail git -C repo update-ref --stdin err &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'ref transaction: can delete symbolic self-reference with git-symbolic-ref(1)' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tgit -C repo symbolic-ref refs/heads/self refs/heads/self &&\n+\tgit -C repo symbolic-ref -d refs/heads/self\n+'\n+\n+test_expect_success 'ref transaction: deleting symbolic self-reference without --no-deref fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tgit -C repo symbolic-ref refs/heads/self refs/heads/self &&\n+\tcat >expect <<-EOF &&\n+\terror: multiple updates for ${SQ}refs/heads/self${SQ} (including one via symref ${SQ}refs/heads/self${SQ}) are not allowed\n+\tEOF\n+\ttest_must_fail git -C repo update-ref -d refs/heads/self 2>err &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'ref transaction: deleting symbolic self-reference with --no-deref succeeds' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tgit -C repo symbolic-ref refs/heads/self refs/heads/self &&\n+\tgit -C repo update-ref -d --no-deref refs/heads/self\n+'\n+\n+test_expect_success 'ref transaction: creating symbolic ref fails with F/D conflict' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tcat >expect <<-EOF &&\n+\terror: unable to write symref for refs/heads: file/directory conflict\n+\tEOF\n+\ttest_must_fail git -C repo symbolic-ref refs/heads refs/heads/foo 2>err &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'ref transaction: ref deletion' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file &&\n+\t\tHEAD_OID=$(git show-ref -s --verify HEAD) &&\n+\t\tcat >expect <<-EOF &&\n+\t\t$HEAD_OID refs/heads/main\n+\t\t$HEAD_OID refs/tags/file\n+\t\tEOF\n+\t\tgit show-ref >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\ttest_must_fail git update-ref -d refs/tags/file $INVALID_OID &&\n+\t\tgit show-ref >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit update-ref -d refs/tags/file $HEAD_OID &&\n+\t\techo \"$HEAD_OID refs/heads/main\" >expect &&\n+\t\tgit show-ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'ref transaction: writes cause auto-compaction' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\n+\tgit init repo &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\ttest_commit -C repo --no-tag A &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n+\n+\ttest_commit -C repo --no-tag B &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+check_fsync_events () {\n+\tlocal trace=\"$1\" &&\n+\tshift &&\n+\n+\tcat >expect &&\n+\tsed -n \\\n+\t\t-e '/^{\"event\":\"counter\",.*\"category\":\"fsync\",/ {\n+\t\t\ts/.*\"category\":\"fsync\",//;\n+\t\t\ts/}$//;\n+\t\t\tp;\n+\t\t}' \\\n+\t\t<\"$trace\" >actual &&\n+\ttest_cmp expect actual\n+}\n+\n+# A fix for this is in flight via jc/reftable-core-fsync.\n+test_expect_failure 'ref transaction: writes are synced' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo initial &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\tGIT_TEST_FSYNC=true \\\n+\t\tgit -C repo -c core.fsync=reference update-ref refs/heads/branch HEAD &&\n+\tcheck_fsync_events trace2.txt <<-EOF\n+\t\"name\":\"hardware-flush\",\"count\":2\n+\tEOF\n+'\n+\n+test_expect_success 'pack-refs: compacts tables' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\n+\ttest_commit -C repo A &&\n+\tls -1 repo/.git/reftable >table-files &&\n+\ttest_line_count = 4 table-files &&\n+\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n+\n+\tgit -C repo pack-refs &&\n+\tls -1 repo/.git/reftable >table-files &&\n+\ttest_line_count = 2 table-files &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'pack-refs: prunes stale tables' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttouch repo/.git/reftable/stale-table.ref &&\n+\tgit -C repo pack-refs &&\n+\ttest_path_is_missing repo/.git/reftable/stable-ref.ref\n+'\n+\n+test_expect_success 'pack-refs: does not prune non-table files' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttouch repo/.git/reftable/garbage &&\n+\tgit -C repo pack-refs &&\n+\ttest_path_is_file repo/.git/reftable/garbage\n+'\n+\n+for umask in 002 022\n+do\n+\t# A fix for this is in flight via ps/reftable-compacted-tables-permission-fix.\n+\ttest_expect_failure POSIXPERM 'pack-refs: honors core.sharedRepository' '\n+\t\ttest_when_finished \"rm -rf repo\" &&\n+\t\t(\n+\t\t\tumask $umask &&\n+\t\t\tgit init --shared=true repo &&\n+\t\t\ttest_commit -C repo A &&\n+\t\t\ttest_line_count = 3 repo/.git/reftable/tables.list\n+\t\t) &&\n+\t\tgit -C repo pack-refs &&\n+\t\ttest_expect_perms \"-rw-rw-r--\" repo/.git/reftable/tables.list &&\n+\t\tfor table in repo/.git/reftable/*.ref\n+\t\tdo\n+\t\t\ttest_expect_perms \"-rw-rw-r--\" \"$table\" ||\n+\t\t\treturn 1\n+\t\tdone\n+\t'\n+done\n+\n+# A fix for this is in flight.\n+test_expect_failure 'packed-refs: writes are synced' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo initial &&\n+\ttest_line_count = 2 table-files &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\tGIT_TEST_FSYNC=true \\\n+\t\tgit -C repo -c core.fsync=reference pack-refs &&\n+\tcheck_fsync_events trace2.txt <<-EOF\n+\t\"name\":\"hardware-flush\",\"count\":2\n+\tEOF\n+'\n+\n+test_expect_success 'ref iterator: bogus names are flagged' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit --no-tag file &&\n+\t\ttest-tool ref-store main update-ref msg \"refs/heads/bogus..name\" $(git rev-parse HEAD) $ZERO_OID REF_SKIP_REFNAME_VERIFICATION &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\t$ZERO_OID refs/heads/bogus..name 0xc\n+\t\t$(git rev-parse HEAD) refs/heads/main 0x0\n+\t\tEOF\n+\t\ttest-tool ref-store main for-each-ref \"\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'ref iterator: missing object IDs are not flagged' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest-tool ref-store main update-ref msg \"refs/heads/broken-hash\" $INVALID_OID $ZERO_OID REF_SKIP_OID_VERIFICATION &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\t$INVALID_OID refs/heads/broken-hash 0x0\n+\t\tEOF\n+\t\ttest-tool ref-store main for-each-ref \"\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'basic: commit and list refs' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo file &&\n+\ttest_write_lines refs/heads/main refs/tags/file >expect &&\n+\tgit -C repo for-each-ref --format=\"%(refname)\" >actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'basic: can write large commit message' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tperl -e \"\n+\t\tprint \\\"this is a long commit message\\\" x 50000\n+\t\" >commit-msg &&\n+\tgit -C repo commit --allow-empty --file=../commit-msg\n+'\n+\n+test_expect_success 'basic: show-ref fails with empty repository' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_must_fail git -C repo show-ref >actual &&\n+\ttest_must_be_empty actual\n+'\n+\n+test_expect_success 'basic: can check out unborn branch' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tgit -C repo checkout -b main\n+'\n+\n+test_expect_success 'basic: peeled tags are stored' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo file &&\n+\tgit -C repo tag -m \"annotated tag\" test_tag HEAD &&\n+\tfor ref in refs/heads/main refs/tags/file refs/tags/test_tag refs/tags/test_tag^{}\n+\tdo\n+\t\techo \"$(git -C repo rev-parse \"$ref\") $ref\" || return 1\n+\tdone >expect &&\n+\tgit -C repo show-ref -d >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'basic: for-each-ref can print symrefs' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file &&\n+\t\tgit branch &&\n+\t\tgit symbolic-ref refs/heads/sym refs/heads/main &&\n+\t\tcat >expected <<-EOF &&\n+\t\trefs/heads/main\n+\t\tEOF\n+\t\tgit for-each-ref --format=\"%(symref)\" refs/heads/sym >actual &&\n+\t\ttest_cmp expected actual\n+\t)\n+'\n+\n+test_expect_success 'basic: notes' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\twrite_script fake_editor <<-\\EOF &&\n+\t\techo \"$MSG\" >\"$1\"\n+\t\techo \"$MSG\" >&2\n+\t\tEOF\n+\n+\t\ttest_commit 1st &&\n+\t\ttest_commit 2nd &&\n+\t\tGIT_EDITOR=./fake_editor MSG=b4 git notes add &&\n+\t\tGIT_EDITOR=./fake_editor MSG=b3 git notes edit &&\n+\t\techo b4 >expect &&\n+\t\tgit notes --ref commits@{1} show >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'basic: stash' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file &&\n+\t\tgit stash list >expect &&\n+\t\ttest_line_count = 0 expect &&\n+\n+\t\techo hoi >>file.t &&\n+\t\tgit stash push -m stashed &&\n+\t\tgit stash list >expect &&\n+\t\ttest_line_count = 1 expect &&\n+\n+\t\tgit stash clear &&\n+\t\tgit stash list >expect &&\n+\t\ttest_line_count = 0 expect\n+\t)\n+'\n+\n+test_expect_success 'basic: cherry-pick' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit message1 file1 &&\n+\t\ttest_commit message2 file2 &&\n+\t\tgit branch source &&\n+\t\tgit checkout HEAD^ &&\n+\t\ttest_commit message3 file3 &&\n+\t\tgit cherry-pick source &&\n+\t\ttest_path_is_file file2\n+\t)\n+'\n+\n+test_expect_success 'basic: rebase' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit message1 file1 &&\n+\t\ttest_commit message2 file2 &&\n+\t\tgit branch source &&\n+\t\tgit checkout HEAD^ &&\n+\t\ttest_commit message3 file3 &&\n+\t\tgit rebase source &&\n+\t\ttest_path_is_file file2\n+\t)\n+'\n+\n+test_expect_success 'reflog: can delete separate reflog entries' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit file &&\n+\t\ttest_commit file2 &&\n+\t\ttest_commit file3 &&\n+\t\ttest_commit file4 &&\n+\t\tgit reflog >actual &&\n+\t\tgrep file3 actual &&\n+\n+\t\tgit reflog delete HEAD@{1} &&\n+\t\tgit reflog >actual &&\n+\t\t! grep file3 actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: can switch to previous branch' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file1 &&\n+\t\tgit checkout -b branch1 &&\n+\t\ttest_commit file2 &&\n+\t\tgit checkout -b branch2 &&\n+\t\tgit switch - &&\n+\t\tgit rev-parse --symbolic-full-name HEAD >actual &&\n+\t\techo refs/heads/branch1 >expect &&\n+\t\ttest_cmp actual expect\n+\t)\n+'\n+\n+test_expect_success 'reflog: copying branch writes reflog entry' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file1 &&\n+\t\ttest_commit file2 &&\n+\t\toid=$(git rev-parse --short HEAD) &&\n+\t\tgit branch src &&\n+\t\tcat >expect <<-EOF &&\n+\t\t${oid} dst@{0}: Branch: copied refs/heads/src to refs/heads/dst\n+\t\t${oid} dst@{1}: branch: Created from main\n+\t\tEOF\n+\t\tgit branch -c src dst &&\n+\t\tgit reflog dst >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: renaming branch writes reflog entry' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit symbolic-ref HEAD refs/heads/before &&\n+\t\ttest_commit file &&\n+\t\tgit show-ref >expected.refs &&\n+\t\tsed s/before/after/g expected &&\n+\t\tgit branch -M after &&\n+\t\tgit show-ref >actual &&\n+\t\ttest_cmp expected actual &&\n+\t\techo refs/heads/after >expected &&\n+\t\tgit symbolic-ref HEAD >actual &&\n+\t\ttest_cmp expected actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: can store empty logs' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_must_fail test-tool ref-store main reflog-exists refs/heads/branch &&\n+\t\ttest-tool ref-store main create-reflog refs/heads/branch &&\n+\t\ttest-tool ref-store main reflog-exists refs/heads/branch &&\n+\t\ttest-tool ref-store main for-each-reflog-ent-reverse refs/heads/branch >actual &&\n+\t\ttest_must_be_empty actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: expiry empties reflog' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit initial &&\n+\t\tgit checkout -b branch &&\n+\t\ttest_commit fileA &&\n+\t\ttest_commit fileB &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\tcommit: fileB\n+\t\tcommit: fileA\n+\t\tbranch: Created from HEAD\n+\t\tEOF\n+\t\tgit reflog show --format=\"%gs\" refs/heads/branch >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit reflog expire branch --expire=all &&\n+\t\tgit reflog show --format=\"%gs\" refs/heads/branch >actual &&\n+\t\ttest_must_be_empty actual &&\n+\t\ttest-tool ref-store main reflog-exists refs/heads/branch\n+\t)\n+'\n+\n+test_expect_success 'reflog: can be deleted' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\ttest-tool ref-store main reflog-exists refs/heads/main &&\n+\t\ttest-tool ref-store main delete-reflog refs/heads/main &&\n+\t\ttest_must_fail test-tool ref-store main reflog-exists refs/heads/main\n+\t)\n+'\n+\n+test_expect_success 'reflog: garbage collection deletes reflog entries' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\tfor count in $(test_seq 1 10)\n+\t\tdo\n+\t\t\ttest_commit \"number $count\" file.t $count number-$count ||\n+\t\t\treturn 1\n+\t\tdone &&\n+\t\tgit reflog refs/heads/main >actual &&\n+\t\ttest_line_count = 10 actual &&\n+\t\tgrep \"commit (initial): number 1\" actual &&\n+\t\tgrep \"commit: number 10\" actual &&\n+\n+\t\tgit gc &&\n+\t\tgit reflog refs/heads/main >actual &&\n+\t\ttest_line_count = 0 actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: updates via HEAD update HEAD reflog' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit main-one &&\n+\t\tgit checkout -b new-branch &&\n+\t\ttest_commit new-one &&\n+\t\ttest_commit new-two &&\n+\n+\t\techo new-one >expect &&\n+\t\tgit log -1 --format=%s HEAD@{1} >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'worktree: adding worktree creates separate stack' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\ttest_path_is_file repo/.git/worktrees/worktree/refs/heads &&\n+\techo \"ref: refs/heads/.invalid\" >expect &&\n+\ttest_cmp expect repo/.git/worktrees/worktree/HEAD &&\n+\ttest_path_is_dir repo/.git/worktrees/worktree/reftable &&\n+\ttest_path_is_file repo/.git/worktrees/worktree/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: pack-refs in main repo packs main refs' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tgit -C repo worktree add ../worktree &&\n+\n+\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\tgit -C repo pack-refs &&\n+\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: pack-refs in worktree packs worktree refs' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tgit -C repo worktree add ../worktree &&\n+\n+\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: creating shared ref updates main stack' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\tgit -C repo pack-refs &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tgit -C worktree update-ref refs/heads/shared HEAD &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: creating per-worktree ref updates worktree stack' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\tgit -C repo pack-refs &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tgit -C worktree update-ref refs/bisect/per-worktree HEAD &&\n+\ttest_line_count = 2 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: creating per-worktree ref from main repo' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\tgit -C repo pack-refs &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tgit -C repo update-ref worktrees/worktree/refs/bisect/per-worktree HEAD &&\n+\ttest_line_count = 2 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: creating per-worktree ref from second worktree' '\n+\ttest_when_finished \"rm -rf repo wt1 wt2\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../wt1 &&\n+\tgit -C repo worktree add ../wt2 &&\n+\tgit -C repo pack-refs &&\n+\tgit -C wt1 pack-refs &&\n+\tgit -C wt2 pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/wt1/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/worktrees/wt2/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tgit -C wt1 update-ref worktrees/wt2/refs/bisect/per-worktree HEAD &&\n+\ttest_line_count = 1 repo/.git/worktrees/wt1/reftable/tables.list &&\n+\ttest_line_count = 2 repo/.git/worktrees/wt2/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: can create shared and per-worktree ref in one transaction' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\tgit -C repo pack-refs &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tcat >stdin <<-EOF &&\n+\tcreate worktrees/worktree/refs/bisect/per-worktree HEAD\n+\tcreate refs/branches/shared HEAD\n+\tEOF\n+\tgit -C repo update-ref --stdin expect &&\n+\tgit -C worktree symbolic-ref HEAD >actual &&\n+\ttest_cmp expect actual &&\n+\tgit -C worktree checkout branch1\n+'\n+\n+test_expect_success 'worktree: adds worktree with detached HEAD' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tgit -C repo rev-parse main >expect &&\n+\n+\tgit -C repo worktree add --detach ../worktree main &&\n+\tgit -C worktree rev-parse HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'fetch: accessing FETCH_HEAD special ref works' '\n+\ttest_when_finished \"rm -rf repo sub\" &&\n+\n+\tgit init sub &&\n+\ttest_commit -C sub two &&\n+\tgit -C sub rev-parse HEAD >expect &&\n+\n+\tgit init repo &&\n+\ttest_commit -C repo one &&\n+\tgit -C repo fetch ../sub &&\n+\tgit -C repo rev-parse FETCH_HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/t0611-reftable-httpd.sh b/t/t0611-reftable-httpd.sh\nnew file mode 100755\nindex 0000000000..5e05b9c1f2\n--- /dev/null\n+++ b/t/t0611-reftable-httpd.sh\n@@ -0,0 +1,26 @@\n+#!/bin/sh\n+\n+test_description='reftable HTTPD tests'\n+\n+. ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd\n+\n+REPO=\"$HTTPD_DOCUMENT_ROOT_PATH/repo\"\n+\n+test_expect_success 'serving ls-remote' '\n+\tgit init --ref-format=reftable -b main \"$REPO\" &&\n+\tcd \"$REPO\" &&\n+\ttest_commit m1 &&\n+\t>.git/git-daemon-export-ok &&\n+\tgit ls-remote \"http://127.0.0.1:$LIB_HTTPD_PORT/smart/repo\" | cut -f 2-2 -d \"\t\" >actual &&\n+\tcat >expect <<-EOF &&\n+\tHEAD\n+\trefs/heads/main\n+\trefs/tags/m1\n+\tEOF\n+\ttest_cmp actual expect\n+'\n+\n+test_done\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex fc93aa57e6..0fb8757b7d 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1750,6 +1750,8 @@ esac\n case \"$GIT_DEFAULT_REF_FORMAT\" in\n files)\n \ttest_set_prereq REFFILES;;\n+reftable)\n+\ttest_set_prereq REFTABLE;;\n *)\n \techo 2>&1 \"error: unknown ref format $GIT_DEFAULT_REF_FORMAT\"\n \texit 1\n-- \n2.43.GIT\n\n"},{"id":"487795","messageId":"63eafc9f5bd062fb74b4bb9d9c0dcd7a11125766.1706862705.git.ps@pks.im","threadId":"60815","inReplyTo":"cover.1706862705.git.ps@pks.im","subject":"[PATCH v2 2/2] ci: add jobs to test with the reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-02T08:38:33Z","receivedAt":"2024-02-02T08:38:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Add CI jobs for both GitHub Workflows and GitLab CI to run Git with the\nnew reftable backend.\n\nSigned-off-by: Patrick Steinhardt \n---\n .github/workflows/main.yml | 9 +++++++++\n .gitlab-ci.yml | 9 +++++++++\n ci/lib.sh | 2 +-\n ci/run-build-and-tests.sh | 3 +++\n 4 files changed, 22 insertions(+), 1 deletion(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 4d97da57ec..1b43e49dad 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -266,6 +266,9 @@ jobs:\n - jobname: linux-sha256\n cc: clang\n pool: ubuntu-latest\n+ - jobname: linux-reftable\n+ cc: clang\n+ pool: ubuntu-latest\n - jobname: linux-gcc\n cc: gcc\n cc_package: gcc-8\n@@ -277,6 +280,9 @@ jobs:\n - jobname: osx-clang\n cc: clang\n pool: macos-13\n+ - jobname: osx-reftable\n+ cc: clang\n+ pool: macos-13\n - jobname: osx-gcc\n cc: gcc\n cc_package: gcc-13\n@@ -287,6 +293,9 @@ jobs:\n - jobname: linux-leaks\n cc: gcc\n pool: ubuntu-latest\n+ - jobname: linux-reftable-leaks\n+ cc: gcc\n+ pool: ubuntu-latest\n - jobname: linux-asan-ubsan\n cc: clang\n pool: ubuntu-latest\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nindex 43bfbd8834..c0fa2fe90b 100644\n--- a/.gitlab-ci.yml\n+++ b/.gitlab-ci.yml\n@@ -26,6 +26,9 @@ test:linux:\n - jobname: linux-sha256\n image: ubuntu:latest\n CC: clang\n+ - jobname: linux-reftable\n+ image: ubuntu:latest\n+ CC: clang\n - jobname: linux-gcc\n image: ubuntu:20.04\n CC: gcc\n@@ -40,6 +43,9 @@ test:linux:\n - jobname: linux-leaks\n image: ubuntu:latest\n CC: gcc\n+ - jobname: linux-reftable-leaks\n+ image: ubuntu:latest\n+ CC: gcc\n - jobname: linux-asan-ubsan\n image: ubuntu:latest\n CC: clang\n@@ -79,6 +85,9 @@ test:osx:\n - jobname: osx-clang\n image: macos-13-xcode-14\n CC: clang\n+ - jobname: osx-reftable\n+ image: macos-13-xcode-14\n+ CC: clang\n artifacts:\n paths:\n - t/failed-test-artifacts\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex d5dd2f2697..0a73fc7bd1 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -367,7 +367,7 @@ linux-musl)\n \tMAKEFLAGS=\"$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes\"\n \tMAKEFLAGS=\"$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8\"\n \t;;\n-linux-leaks)\n+linux-leaks|linux-reftable-leaks)\n \texport SANITIZE=leak\n \texport GIT_TEST_PASSING_SANITIZE_LEAK=true\n \texport GIT_TEST_SANITIZE_LEAK_LOG=true\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 7a1466b868..c192bd613c 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -37,6 +37,9 @@ linux-clang)\n linux-sha256)\n \texport GIT_TEST_DEFAULT_HASH=sha256\n \t;;\n+linux-reftable|linux-reftable-leaks|osx-reftable)\n+\texport GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+\t;;\n pedantic)\n \t# Don't run the tests; we only care about whether Git can be\n \t# built.\n-- \n2.43.GIT\n\n"},{"id":"487803","messageId":"ZbzJWo-DBpGZiMO4@tanuki","threadId":"60815","inReplyTo":"ZbyoL3r7yEVYe35B@tanuki","subject":"Re: [PATCH 1/2] refs: introduce reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-02T10:52:10Z","receivedAt":"2024-02-02T10:52:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Feb 02, 2024 at 09:30:39AM +0100, Patrick Steinhardt wrote:\n> On Thu, Feb 01, 2024 at 07:17:45AM -0800, Karthik Nayak wrote:\n> > > +\t\titer->err = reftable_iterator_next_ref(&iter->iter, &iter->ref);\n> > > +\t\tif (iter->err)\n> > > +\t\t\tbreak;\n> > > +\n> > > +\t\t/*\n> > > +\t\t * The files backend only lists references contained in\n> > > +\t\t * \"refs/\". We emulate the same behaviour here and thus skip\n> > > +\t\t * all references that don't start with this prefix.\n> > > +\t\t */\n> > > +\t\tif (!starts_with(iter->ref.refname, \"refs/\"))\n> > > +\t\t\tcontinue;\n> > > +\n> > \n> > Since my patch series [1] to print all refs is now merged to next, maybe\n> > you could add this in?\n> > \n> > diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> > index 895de0b273..3f4f905292 100644\n> > --- a/refs/reftable-backend.c\n> > +++ b/refs/reftable-backend.c\n> > @@ -348,11 +348,10 @@ static int\n> > reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> > \t\t\tbreak;\n> > \n> > \t\t/*\n> > -\t\t * The files backend only lists references contained in\n> > -\t\t * \"refs/\". We emulate the same behaviour here and thus skip\n> > -\t\t * all references that don't start with this prefix.\n> > +\t\t * Unless the `DO_FOR_EACH_INCLUDE_ALL_REFS` flag is use, we only\n> > +\t\t * list references contained in \"refs/\" to mimic the file-backend.\n> > \t\t */\n> > -\t\tif (!starts_with(iter->ref.refname, \"refs/\"))\n> > +\t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_ALL_REFS) &&\n> > !starts_with(iter->ref.refname, \"refs/\"))\n> > \t\t\tcontinue;\n> > \n> > \t\tif (iter->prefix &&\n> \n> I'm still waiting for your patch series to hit `next` before making it a\n> dependency of this series, but thanks for the patch!\n\nOops, didn't read this carefully enough and missed that it already is in\n`next`. I'll adapt accordingly in the next version, thanks!\n\nPatrick\n"},{"id":"487813","messageId":"CAOLa=ZS=Qo5i0niayUyCpsFwePSZoGg3NCWdnAuhvvTizmYTyQ@mail.gmail.com","threadId":"60815","inReplyTo":"cover.1706862705.git.ps@pks.im","subject":"Re: [PATCH v2 0/2] refs: introduce reftable backend","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-02T13:02:36Z","receivedAt":"2024-02-02T13:02:42Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Hello,\n\nPatrick Steinhardt writes:\n> Hi,\n>\n> this is the second version of my patch series that introduces the\n> reftable backend.\n>\n\nThe diff looks good to, apart from the tests (which I didn't get around\nto reviewing), everything else looks great!\n\nThanks,\nKarthik\n"},{"id":"487880","messageId":"xmqq8r41qo0y.fsf@gitster.g","threadId":"60815","inReplyTo":"cover.1706862705.git.ps@pks.im","subject":"Re: [PATCH v2 0/2] refs: introduce reftable backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-03T20:41:01Z","receivedAt":"2024-02-03T20:41:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> this is the second version of my patch series that introduces the\n> reftable backend.\n>\n> This version addresses the feedback by Karthik. I don't think it really\n> makes sense to spell out every change here -- the range diff should\n> likely be easier to digest.\n\nThis, when merged to 'seen' (which also has \"for-each-ref now thinks\nan empty string is a signal to show ref-like things outside the\nrefs/ hierarchy\" topic, kn/for-all-refs), seems to break *-reftable\nCI tests, cf. https://github.com/git/git/actions/runs/7765401528\n\nI'll tentatively eject the topic from 'seen', even though I have a\nsuspicion that it byitself would pass all the tests.\n\nThanks.\n\n"},{"id":"487886","messageId":"Zb8n5N3-_kxDyPGs@ncase","threadId":"60815","inReplyTo":"xmqq8r41qo0y.fsf@gitster.g","subject":"Re: [PATCH v2 0/2] refs: introduce reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-04T06:00:04Z","receivedAt":"2024-02-04T06:00:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Feb 03, 2024 at 12:41:01PM -0800, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > this is the second version of my patch series that introduces the\n> > reftable backend.\n> >\n> > This version addresses the feedback by Karthik. I don't think it really\n> > makes sense to spell out every change here -- the range diff should\n> > likely be easier to digest.\n> \n> This, when merged to 'seen' (which also has \"for-each-ref now thinks\n> an empty string is a signal to show ref-like things outside the\n> refs/ hierarchy\" topic, kn/for-all-refs), seems to break *-reftable\n> CI tests, cf. https://github.com/git/git/actions/runs/7765401528\n> \n> I'll tentatively eject the topic from 'seen', even though I have a\n> suspicion that it byitself would pass all the tests.\n> \n> Thanks.\n\nYup, this is known due to the semantic conflict with kn/for-all-refs, as\nyou point out. Karthik already sent a single-line change in this thread\nthat fixes the issue. I'll rebase the patch series tomorrow and pull in\nthe topic as a dependency and adapt accordingly.\n\nThanks!\n\nPatrick\n"},{"id":"487916","messageId":"cover.1707109509.git.ps@pks.im","threadId":"60815","inReplyTo":"cover.1706601199.git.ps@pks.im","subject":"[PATCH v3 0/2] refs: introduce reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-05T06:02:46Z","receivedAt":"2024-02-05T06:02:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the third version of my patch series that introduces the\nreftable backend.\n\nThe only real change in this version is that I've pulled in\nkn/for-all-refs at 693e807311 (for-each-ref: avoid filtering on empty\npattern, 2024-01-29) as a dependency. This series introduces a new\nDO_FOR_EACH_INCLUDE_ALL_REFS flag that we need to handle in the reftable\nbackend. With this change all tests should pass again.\n\nPatrick\n\nPatrick Steinhardt (2):\n refs: introduce reftable backend\n ci: add jobs to test with the reftable backend\n\n .github/workflows/main.yml | 9 +\n .gitlab-ci.yml | 9 +\n Documentation/ref-storage-format.txt | 2 +\n .../technical/repository-version.txt | 5 +-\n Makefile | 1 +\n ci/lib.sh | 2 +-\n ci/run-build-and-tests.sh | 3 +\n contrib/workdir/git-new-workdir | 2 +-\n path.c | 2 +-\n path.h | 1 +\n refs.c | 1 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 2297 +++++++++++++++++\n repository.h | 5 +-\n t/t0610-reftable-basics.sh | 887 +++++++\n t/t0611-reftable-httpd.sh | 26 +\n t/test-lib.sh | 2 +\n 17 files changed, 3248 insertions(+), 7 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100755 t/t0610-reftable-basics.sh\n create mode 100755 t/t0611-reftable-httpd.sh\n\nRange-diff against v2:\n1: d6548dcfad ! 1: d83e66e980 refs: introduce reftable backend\n @@ refs/reftable-backend.c (new)\n +{\n +\tswitch (log->value_type) {\n +\tcase REFTABLE_LOG_UPDATE:\n -+\t\t/* when we write log records, the hashes are owned by a struct\n -+\t\t * oid */\n ++\t\t/*\n ++\t\t * When we write log records, the hashes are owned by the\n ++\t\t * caller and thus shouldn't be free'd.\n ++\t\t */\n +\t\tlog->value.update.old_hash = NULL;\n +\t\tlog->value.update.new_hash = NULL;\n +\t\tbreak;\n @@ refs/reftable-backend.c (new)\n +\t\t\tbreak;\n +\n +\t\t/*\n -+\t\t * The files backend only lists references contained in\n -+\t\t * \"refs/\". We emulate the same behaviour here and thus skip\n -+\t\t * all references that don't start with this prefix.\n ++\t\t * Unless DO_FOR_EACH_INCLUDE_ALL_REFS is set, we only list\n ++\t\t * refs starting with \"refs/\" to mimic the \"files\" backend.\n +\t\t */\n -+\t\tif (!starts_with(iter->ref.refname, \"refs/\"))\n ++\t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_ALL_REFS) &&\n ++\t\t !starts_with(iter->ref.refname, \"refs/\"))\n +\t\t\tcontinue;\n +\n +\t\tif (iter->prefix &&\n2: 63eafc9f5b = 2: 146bb95c03 ci: add jobs to test with the reftable backend\n-- \n2.43.GIT\n\n"},{"id":"487918","messageId":"d83e66e980d4d72b638f481886dada9d0018163d.1707109509.git.ps@pks.im","threadId":"60815","inReplyTo":"cover.1707109509.git.ps@pks.im","subject":"[PATCH v3 1/2] refs: introduce reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-05T06:02:50Z","receivedAt":"2024-02-05T06:02:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Due to scalability issues, Shawn Pearce has originally proposed a new\n\"reftable\" format more than six years ago [1]. Initially, this new\nformat was implemented in JGit with promising results. Around two years\nago, we have then added the \"reftable\" library to the Git codebase via\na4bbd13be3 (Merge branch 'hn/reftable', 2021-12-15). With this we have\nlanded all the low-level code to read and write reftables. Notably\nmissing though was the integration of this low-level code into the Git\ncode base in the form of a new ref backend that ties all of this\ntogether.\n\nThis gap is now finally closed by introducing a new \"reftable\" backend\ninto the Git codebase. This new backend promises to bring some notable\nimprovements to Git repositories:\n\n - It becomes possible to do truly atomic writes where either all refs\n are committed to disk or none are. This was not possible with the\n \"files\" backend because ref updates were split across multiple loose\n files.\n\n - The disk space required to store many refs is reduced, both compared\n to loose refs and packed-refs. This is enabled both by the reftable\n format being a binary format, which is more compact, and by prefix\n compression.\n\n - We can ignore filesystem-specific behaviour as ref names are not\n encoded via paths anymore. This means there is no need to handle\n case sensitivity on Windows systems or Unicode precomposition on\n macOS.\n\n - There is no need to rewrite the complete refdb anymore every time a\n ref is being deleted like it was the case for packed-refs. This\n means that ref deletions are now constant time instead of scaling\n linearly with the number of refs.\n\n - We can ignore file/directory conflicts so that it becomes possible\n to store both \"refs/heads/foo\" and \"refs/heads/foo/bar\".\n\n - Due to this property we can retain reflogs for deleted refs. We have\n previously been deleting reflogs together with their refs to avoid\n file/directory conflicts, which is not necessary anymore.\n\n - We can properly enumerate all refs. With the \"files\" backend it is\n not easily possible to distinguish between refs and non-refs because\n they may live side by side in the gitdir.\n\nNot all of these improvements are realized with the current \"reftable\"\nbackend implementation. At this point, the new backend is supposed to be\na drop-in replacement for the \"files\" backend that is used by basically\nall Git repositories nowadays. It strives for 1:1 compatibility, which\nmeans that a user can expect the same behaviour regardless of whether\nthey use the \"reftable\" backend or the \"files\" backend for most of the\npart.\n\nMost notably, this means we artificially limit the capabilities of the\n\"reftable\" backend to match the limits of the \"files\" backend. It is not\npossible to create refs that would end up with file/directory conflicts,\nwe do not retain reflogs, we perform stricter-than-necessary checks.\nThis is done intentionally due to two main reasons:\n\n - It makes it significantly easier to land the \"reftable\" backend as\n tests behave the same. It would be tough to argue for each and every\n single test that doesn't pass with the \"reftable\" backend.\n\n - It ensures compatibility between repositories that use the \"files\"\n backend and repositories that use the \"reftable\" backend. Like this,\n hosters can migrate their repositories to use the \"reftable\" backend\n without causing issues for clients that use the \"files\" backend in\n their clones.\n\nIt is expected that these artificial limitations may eventually go away\nin the long term.\n\nPerformance-wise things very much depend on the actual workload. The\nfollowing benchmarks compare the \"files\" and \"reftable\" backends in the\ncurrent version:\n\n - Creating N refs in separate transactions shows that the \"files\"\n backend is ~50% faster. This is not surprising given that creating a\n ref only requires us to create a single loose ref. The \"reftable\"\n backend will also perform auto compaction on updates. In real-world\n workloads we would likely also want to perform pack loose refs,\n which would likely change the picture.\n\n Benchmark 1: update-ref: create refs sequentially (refformat = files, refcount = 1)\n Time (mean ± σ): 2.1 ms ± 0.3 ms [User: 0.6 ms, System: 1.7 ms]\n Range (min … max): 1.8 ms … 4.3 ms 133 runs\n\n Benchmark 2: update-ref: create refs sequentially (refformat = reftable, refcount = 1)\n Time (mean ± σ): 2.7 ms ± 0.1 ms [User: 0.6 ms, System: 2.2 ms]\n Range (min … max): 2.4 ms … 2.9 ms 132 runs\n\n Benchmark 3: update-ref: create refs sequentially (refformat = files, refcount = 1000)\n Time (mean ± σ): 1.975 s ± 0.006 s [User: 0.437 s, System: 1.535 s]\n Range (min … max): 1.969 s … 1.980 s 3 runs\n\n Benchmark 4: update-ref: create refs sequentially (refformat = reftable, refcount = 1000)\n Time (mean ± σ): 2.611 s ± 0.013 s [User: 0.782 s, System: 1.825 s]\n Range (min … max): 2.597 s … 2.622 s 3 runs\n\n Benchmark 5: update-ref: create refs sequentially (refformat = files, refcount = 100000)\n Time (mean ± σ): 198.442 s ± 0.241 s [User: 43.051 s, System: 155.250 s]\n Range (min … max): 198.189 s … 198.670 s 3 runs\n\n Benchmark 6: update-ref: create refs sequentially (refformat = reftable, refcount = 100000)\n Time (mean ± σ): 294.509 s ± 4.269 s [User: 104.046 s, System: 190.326 s]\n Range (min … max): 290.223 s … 298.761 s 3 runs\n\n - Creating N refs in a single transaction shows that the \"files\"\n backend is significantly slower once we start to write many refs.\n The \"reftable\" backend only needs to update two files, whereas the\n \"files\" backend needs to write one file per ref.\n\n Benchmark 1: update-ref: create many refs (refformat = files, refcount = 1)\n Time (mean ± σ): 1.9 ms ± 0.1 ms [User: 0.4 ms, System: 1.4 ms]\n Range (min … max): 1.8 ms … 2.6 ms 151 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, refcount = 1)\n Time (mean ± σ): 2.5 ms ± 0.1 ms [User: 0.7 ms, System: 1.7 ms]\n Range (min … max): 2.4 ms … 3.4 ms 148 runs\n\n Benchmark 3: update-ref: create many refs (refformat = files, refcount = 1000)\n Time (mean ± σ): 152.5 ms ± 5.2 ms [User: 19.1 ms, System: 133.1 ms]\n Range (min … max): 148.5 ms … 167.8 ms 15 runs\n\n Benchmark 4: update-ref: create many refs (refformat = reftable, refcount = 1000)\n Time (mean ± σ): 58.0 ms ± 2.5 ms [User: 28.4 ms, System: 29.4 ms]\n Range (min … max): 56.3 ms … 72.9 ms 40 runs\n\n Benchmark 5: update-ref: create many refs (refformat = files, refcount = 1000000)\n Time (mean ± σ): 152.752 s ± 0.710 s [User: 20.315 s, System: 131.310 s]\n Range (min … max): 152.165 s … 153.542 s 3 runs\n\n Benchmark 6: update-ref: create many refs (refformat = reftable, refcount = 1000000)\n Time (mean ± σ): 51.912 s ± 0.127 s [User: 26.483 s, System: 25.424 s]\n Range (min … max): 51.769 s … 52.012 s 3 runs\n\n - Deleting a ref in a fully-packed repository shows that the \"files\"\n backend scales with the number of refs. The \"reftable\" backend has\n constant-time deletions.\n\n Benchmark 1: update-ref: delete ref (refformat = files, refcount = 1)\n Time (mean ± σ): 1.7 ms ± 0.1 ms [User: 0.4 ms, System: 1.2 ms]\n Range (min … max): 1.6 ms … 2.1 ms 316 runs\n\n Benchmark 2: update-ref: delete ref (refformat = reftable, refcount = 1)\n Time (mean ± σ): 1.8 ms ± 0.1 ms [User: 0.4 ms, System: 1.3 ms]\n Range (min … max): 1.7 ms … 2.1 ms 294 runs\n\n Benchmark 3: update-ref: delete ref (refformat = files, refcount = 1000)\n Time (mean ± σ): 2.0 ms ± 0.1 ms [User: 0.5 ms, System: 1.4 ms]\n Range (min … max): 1.9 ms … 2.5 ms 287 runs\n\n Benchmark 4: update-ref: delete ref (refformat = reftable, refcount = 1000)\n Time (mean ± σ): 1.9 ms ± 0.1 ms [User: 0.5 ms, System: 1.3 ms]\n Range (min … max): 1.8 ms … 2.1 ms 217 runs\n\n Benchmark 5: update-ref: delete ref (refformat = files, refcount = 1000000)\n Time (mean ± σ): 229.8 ms ± 7.9 ms [User: 182.6 ms, System: 46.8 ms]\n Range (min … max): 224.6 ms … 245.2 ms 6 runs\n\n Benchmark 6: update-ref: delete ref (refformat = reftable, refcount = 1000000)\n Time (mean ± σ): 2.0 ms ± 0.0 ms [User: 0.6 ms, System: 1.3 ms]\n Range (min … max): 2.0 ms … 2.1 ms 3 runs\n\n - Listing all refs shows no significant advantage for either of the\n backends. The \"files\" backend is a bit faster, but not by a\n significant margin. When repositories are not packed the \"reftable\"\n backend outperforms the \"files\" backend because the \"reftable\"\n backend performs auto-compaction.\n\n Benchmark 1: show-ref: print all refs (refformat = files, refcount = 1, packed = true)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 2.0 ms 1729 runs\n\n Benchmark 2: show-ref: print all refs (refformat = reftable, refcount = 1, packed = true)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 1.8 ms 1816 runs\n\n Benchmark 3: show-ref: print all refs (refformat = files, refcount = 1000, packed = true)\n Time (mean ± σ): 4.3 ms ± 0.1 ms [User: 0.9 ms, System: 3.3 ms]\n Range (min … max): 4.1 ms … 4.6 ms 645 runs\n\n Benchmark 4: show-ref: print all refs (refformat = reftable, refcount = 1000, packed = true)\n Time (mean ± σ): 4.5 ms ± 0.2 ms [User: 1.0 ms, System: 3.3 ms]\n Range (min … max): 4.2 ms … 5.9 ms 643 runs\n\n Benchmark 5: show-ref: print all refs (refformat = files, refcount = 1000000, packed = true)\n Time (mean ± σ): 2.537 s ± 0.034 s [User: 0.488 s, System: 2.048 s]\n Range (min … max): 2.511 s … 2.627 s 10 runs\n\n Benchmark 6: show-ref: print all refs (refformat = reftable, refcount = 1000000, packed = true)\n Time (mean ± σ): 2.712 s ± 0.017 s [User: 0.653 s, System: 2.059 s]\n Range (min … max): 2.692 s … 2.752 s 10 runs\n\n Benchmark 7: show-ref: print all refs (refformat = files, refcount = 1, packed = false)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 1.9 ms 1834 runs\n\n Benchmark 8: show-ref: print all refs (refformat = reftable, refcount = 1, packed = false)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.4 ms … 2.0 ms 1840 runs\n\n Benchmark 9: show-ref: print all refs (refformat = files, refcount = 1000, packed = false)\n Time (mean ± σ): 13.8 ms ± 0.2 ms [User: 2.8 ms, System: 10.8 ms]\n Range (min … max): 13.3 ms … 14.5 ms 208 runs\n\n Benchmark 10: show-ref: print all refs (refformat = reftable, refcount = 1000, packed = false)\n Time (mean ± σ): 4.5 ms ± 0.2 ms [User: 1.2 ms, System: 3.3 ms]\n Range (min … max): 4.3 ms … 6.2 ms 624 runs\n\n Benchmark 11: show-ref: print all refs (refformat = files, refcount = 1000000, packed = false)\n Time (mean ± σ): 12.127 s ± 0.129 s [User: 2.675 s, System: 9.451 s]\n Range (min … max): 11.965 s … 12.370 s 10 runs\n\n Benchmark 12: show-ref: print all refs (refformat = reftable, refcount = 1000000, packed = false)\n Time (mean ± σ): 2.799 s ± 0.022 s [User: 0.735 s, System: 2.063 s]\n Range (min … max): 2.769 s … 2.836 s 10 runs\n\n - Printing a single ref shows no real difference between the \"files\"\n and \"reftable\" backends.\n\n Benchmark 1: show-ref: print single ref (refformat = files, refcount = 1)\n Time (mean ± σ): 1.5 ms ± 0.1 ms [User: 0.4 ms, System: 1.0 ms]\n Range (min … max): 1.4 ms … 1.8 ms 1779 runs\n\n Benchmark 2: show-ref: print single ref (refformat = reftable, refcount = 1)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.4 ms … 2.5 ms 1753 runs\n\n Benchmark 3: show-ref: print single ref (refformat = files, refcount = 1000)\n Time (mean ± σ): 1.5 ms ± 0.1 ms [User: 0.3 ms, System: 1.1 ms]\n Range (min … max): 1.4 ms … 1.9 ms 1840 runs\n\n Benchmark 4: show-ref: print single ref (refformat = reftable, refcount = 1000)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 2.0 ms 1831 runs\n\n Benchmark 5: show-ref: print single ref (refformat = files, refcount = 1000000)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 2.1 ms 1848 runs\n\n Benchmark 6: show-ref: print single ref (refformat = reftable, refcount = 1000000)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 2.1 ms 1762 runs\n\nSo overall, performance depends on the usecases. Except for many\nsequential writes the \"reftable\" backend is roughly on par or\nsignificantly faster than the \"files\" backend though. Given that the\n\"files\" backend has received 18 years of optimizations by now this can\nbe seen as a win. Furthermore, we can expect that the \"reftable\" backend\nwill grow faster over time when attention turns more towards\noptimizations.\n\nThe complete test suite passes, except for those tests explicitly marked\nto require the REFFILES prerequisite. Some tests in t0610 are marked as\nfailing because they depend on still-in-flight bug fixes. Tests can be\nrun with the new backend by setting the GIT_TEST_DEFAULT_REF_FORMAT\nenvironment variable to \"reftable\".\n\nThere is a single known conceptual incompatibility with the dumb HTTP\ntransport. As \"info/refs\" SHOULD NOT contain the HEAD reference, and\nbecause the \"HEAD\" file is not valid anymore, it is impossible for the\nremote client to figure out the default branch without changing the\nprotocol. This shortcoming needs to be handled in a subsequent patch\nseries.\n\nAs the reftable library has already been introduced a while ago, this\ncommit message will not go into the details of how exactly the on-disk\nformat works. Please refer to our preexisting technical documentation at\nDocumentation/technical/reftable for this.\n\n[1]: https://public-inbox.org/git/CAJo=hJtyof=HRy=2sLP0ng0uZ4=S-DpZ5dR1aF+VHVETKG20OQ@mail.gmail.com/\n\nOriginal-idea-by: Shawn Pearce \nBased-on-patch-by: Han-Wen Nienhuys \nSigned-off-by: Patrick Steinhardt \n---\n Documentation/ref-storage-format.txt | 2 +\n .../technical/repository-version.txt | 5 +-\n Makefile | 1 +\n contrib/workdir/git-new-workdir | 2 +-\n path.c | 2 +-\n path.h | 1 +\n refs.c | 1 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 2297 +++++++++++++++++\n repository.h | 5 +-\n t/t0610-reftable-basics.sh | 887 +++++++\n t/t0611-reftable-httpd.sh | 26 +\n t/test-lib.sh | 2 +\n 13 files changed, 3226 insertions(+), 6 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100755 t/t0610-reftable-basics.sh\n create mode 100755 t/t0611-reftable-httpd.sh\n\ndiff --git a/Documentation/ref-storage-format.txt b/Documentation/ref-storage-format.txt\nindex 1a65cac468..14fff8a9c6 100644\n--- a/Documentation/ref-storage-format.txt\n+++ b/Documentation/ref-storage-format.txt\n@@ -1 +1,3 @@\n * `files` for loose files with packed-refs. This is the default.\n+* `reftable` for the reftable format. This format is experimental and its\n+ internals are subject to change.\ndiff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\nindex 27be3741e6..47281420fc 100644\n--- a/Documentation/technical/repository-version.txt\n+++ b/Documentation/technical/repository-version.txt\n@@ -103,5 +103,6 @@ GIT_COMMON_DIR/worktrees//config.worktree)\n \n ==== `refStorage`\n \n-Specifies the file format for the ref database. The only valid value\n-is `files` (loose references with a packed-refs file).\n+Specifies the file format for the ref database. The valid values are\n+`files` (loose references with a packed-refs file) and `reftable` (see\n+Documentation/technical/reftable.txt).\ndiff --git a/Makefile b/Makefile\nindex 0f748a52e6..b302516173 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1127,6 +1127,7 @@ LIB_OBJS += reflog.o\n LIB_OBJS += refs.o\n LIB_OBJS += refs/debug.o\n LIB_OBJS += refs/files-backend.o\n+LIB_OBJS += refs/reftable-backend.o\n LIB_OBJS += refs/iterator.o\n LIB_OBJS += refs/packed-backend.o\n LIB_OBJS += refs/ref-cache.o\ndiff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir\nindex 888c34a521..989197aace 100755\n--- a/contrib/workdir/git-new-workdir\n+++ b/contrib/workdir/git-new-workdir\n@@ -79,7 +79,7 @@ trap cleanup $siglist\n # create the links to the original repo. explicitly exclude index, HEAD and\n # logs/HEAD from the list since they are purely related to the current working\n # directory, and should not be shared.\n-for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn\n+for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn reftable\n do\n \t# create a containing directory if needed\n \tcase $x in\ndiff --git a/path.c b/path.c\nindex 0fb527918b..8bb223c92c 100644\n--- a/path.c\n+++ b/path.c\n@@ -871,7 +871,7 @@ const char *enter_repo(const char *path, int strict)\n \treturn NULL;\n }\n \n-static int calc_shared_perm(int mode)\n+int calc_shared_perm(int mode)\n {\n \tint tweak;\n \ndiff --git a/path.h b/path.h\nindex b3233c51fa..e053effef2 100644\n--- a/path.h\n+++ b/path.h\n@@ -181,6 +181,7 @@ const char *git_path_shallow(struct repository *r);\n int ends_with_path_components(const char *path, const char *components);\n int validate_headref(const char *ref);\n \n+int calc_shared_perm(int mode);\n int adjust_shared_perm(const char *path);\n \n char *interpolate_path(const char *path, int real_home);\ndiff --git a/refs.c b/refs.c\nindex 3190df8d81..7e6970ca1d 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -35,6 +35,7 @@\n */\n static const struct ref_storage_be *refs_backends[] = {\n \t[REF_STORAGE_FORMAT_FILES] = &refs_be_files,\n+\t[REF_STORAGE_FORMAT_REFTABLE] = &refs_be_reftable,\n };\n \n static const struct ref_storage_be *find_ref_storage_backend(unsigned int ref_storage_format)\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex be15c19c02..2285abe1ec 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -700,6 +700,7 @@ struct ref_storage_be {\n };\n \n extern struct ref_storage_be refs_be_files;\n+extern struct ref_storage_be refs_be_reftable;\n extern struct ref_storage_be refs_be_packed;\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nnew file mode 100644\nindex 0000000000..207cf2f8e9\n--- /dev/null\n+++ b/refs/reftable-backend.c\n@@ -0,0 +1,2297 @@\n+#include \"../git-compat-util.h\"\n+#include \"../abspath.h\"\n+#include \"../chdir-notify.h\"\n+#include \"../environment.h\"\n+#include \"../gettext.h\"\n+#include \"../hash.h\"\n+#include \"../hex.h\"\n+#include \"../iterator.h\"\n+#include \"../ident.h\"\n+#include \"../lockfile.h\"\n+#include \"../object.h\"\n+#include \"../path.h\"\n+#include \"../refs.h\"\n+#include \"../reftable/reftable-stack.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-merged.h\"\n+#include \"../setup.h\"\n+#include \"../strmap.h\"\n+#include \"refs-internal.h\"\n+\n+/*\n+ * Used as a flag in ref_update::flags when the ref_update was via an\n+ * update to HEAD.\n+ */\n+#define REF_UPDATE_VIA_HEAD (1 << 8)\n+\n+struct reftable_ref_store {\n+\tstruct ref_store base;\n+\n+\t/*\n+\t * The main stack refers to the common dir and thus contains common\n+\t * refs as well as refs of the main repository.\n+\t */\n+\tstruct reftable_stack *main_stack;\n+\t/*\n+\t * The worktree stack refers to the gitdir in case the refdb is opened\n+\t * via a worktree. It thus contains the per-worktree refs.\n+\t */\n+\tstruct reftable_stack *worktree_stack;\n+\t/*\n+\t * Map of worktree stacks by their respective worktree names. The map\n+\t * is populated lazily when we try to resolve `worktrees/$worktree` refs.\n+\t */\n+\tstruct strmap worktree_stacks;\n+\tstruct reftable_write_options write_options;\n+\n+\tunsigned int store_flags;\n+\tint err;\n+};\n+\n+/*\n+ * Downcast ref_store to reftable_ref_store. Die if ref_store is not a\n+ * reftable_ref_store. required_flags is compared with ref_store's store_flags\n+ * to ensure the ref_store has all required capabilities. \"caller\" is used in\n+ * any necessary error messages.\n+ */\n+static struct reftable_ref_store *reftable_be_downcast(struct ref_store *ref_store,\n+\t\t\t\t\t\t unsigned int required_flags,\n+\t\t\t\t\t\t const char *caller)\n+{\n+\tstruct reftable_ref_store *refs;\n+\n+\tif (ref_store->be != &refs_be_reftable)\n+\t\tBUG(\"ref_store is type \\\"%s\\\" not \\\"reftables\\\" in %s\",\n+\t\t ref_store->be->name, caller);\n+\n+\trefs = (struct reftable_ref_store *)ref_store;\n+\n+\tif ((refs->store_flags & required_flags) != required_flags)\n+\t\tBUG(\"operation %s requires abilities 0x%x, but only have 0x%x\",\n+\t\t caller, required_flags, refs->store_flags);\n+\n+\treturn refs;\n+}\n+\n+/*\n+ * Some refs are global to the repository (refs/heads/{*}), while others are\n+ * local to the worktree (eg. HEAD, refs/bisect/{*}). We solve this by having\n+ * multiple separate databases (ie. multiple reftable/ directories), one for\n+ * the shared refs, one for the current worktree refs, and one for each\n+ * additional worktree. For reading, we merge the view of both the shared and\n+ * the current worktree's refs, when necessary.\n+ *\n+ * This function also optionally assigns the rewritten reference name that is\n+ * local to the stack. This translation is required when using worktree refs\n+ * like `worktrees/$worktree/refs/heads/foo` as worktree stacks will store\n+ * those references in their normalized form.\n+ */\n+static struct reftable_stack *stack_for(struct reftable_ref_store *store,\n+\t\t\t\t\tconst char *refname,\n+\t\t\t\t\tconst char **rewritten_ref)\n+{\n+\tconst char *wtname;\n+\tint wtname_len;\n+\n+\tif (!refname)\n+\t\treturn store->main_stack;\n+\n+\tswitch (parse_worktree_ref(refname, &wtname, &wtname_len, rewritten_ref)) {\n+\tcase REF_WORKTREE_OTHER: {\n+\t\tstatic struct strbuf wtname_buf = STRBUF_INIT;\n+\t\tstruct strbuf wt_dir = STRBUF_INIT;\n+\t\tstruct reftable_stack *stack;\n+\n+\t\t/*\n+\t\t * We're using a static buffer here so that we don't need to\n+\t\t * allocate the worktree name whenever we look up a reference.\n+\t\t * This could be avoided if the strmap interface knew how to\n+\t\t * handle keys with a length.\n+\t\t */\n+\t\tstrbuf_reset(&wtname_buf);\n+\t\tstrbuf_add(&wtname_buf, wtname, wtname_len);\n+\n+\t\t/*\n+\t\t * There is an edge case here: when the worktree references the\n+\t\t * current worktree, then we set up the stack once via\n+\t\t * `worktree_stacks` and once via `worktree_stack`. This is\n+\t\t * wasteful, but in the reading case it shouldn't matter. And\n+\t\t * in the writing case we would notice that the stack is locked\n+\t\t * already and error out when trying to write a reference via\n+\t\t * both stacks.\n+\t\t */\n+\t\tstack = strmap_get(&store->worktree_stacks, wtname_buf.buf);\n+\t\tif (!stack) {\n+\t\t\tstrbuf_addf(&wt_dir, \"%s/worktrees/%s/reftable\",\n+\t\t\t\t store->base.repo->commondir, wtname_buf.buf);\n+\n+\t\t\tstore->err = reftable_new_stack(&stack, wt_dir.buf,\n+\t\t\t\t\t\t\tstore->write_options);\n+\t\t\tassert(store->err != REFTABLE_API_ERROR);\n+\t\t\tstrmap_put(&store->worktree_stacks, wtname_buf.buf, stack);\n+\t\t}\n+\n+\t\tstrbuf_release(&wt_dir);\n+\t\treturn stack;\n+\t}\n+\tcase REF_WORKTREE_CURRENT:\n+\t\t/*\n+\t\t * If there is no worktree stack then we're currently in the\n+\t\t * main worktree. We thus return the main stack in that case.\n+\t\t */\n+\t\tif (!store->worktree_stack)\n+\t\t\treturn store->main_stack;\n+\t\treturn store->worktree_stack;\n+\tcase REF_WORKTREE_MAIN:\n+\tcase REF_WORKTREE_SHARED:\n+\t\treturn store->main_stack;\n+\tdefault:\n+\t\tBUG(\"unhandled worktree reference type\");\n+\t}\n+}\n+\n+static int should_write_log(struct ref_store *refs, const char *refname)\n+{\n+\tif (log_all_ref_updates == LOG_REFS_UNSET)\n+\t\tlog_all_ref_updates = is_bare_repository() ? LOG_REFS_NONE : LOG_REFS_NORMAL;\n+\n+\tswitch (log_all_ref_updates) {\n+\tcase LOG_REFS_NONE:\n+\t\treturn refs_reflog_exists(refs, refname);\n+\tcase LOG_REFS_ALWAYS:\n+\t\treturn 1;\n+\tcase LOG_REFS_NORMAL:\n+\t\tif (should_autocreate_reflog(refname))\n+\t\t\treturn 1;\n+\t\treturn refs_reflog_exists(refs, refname);\n+\tdefault:\n+\t\tBUG(\"unhandled core.logAllRefUpdates value %d\", log_all_ref_updates);\n+\t}\n+}\n+\n+static void clear_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\t/*\n+\t\t * When we write log records, the hashes are owned by the\n+\t\t * caller and thus shouldn't be free'd.\n+\t\t */\n+\t\tlog->value.update.old_hash = NULL;\n+\t\tlog->value.update.new_hash = NULL;\n+\t\tbreak;\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\t}\n+\treftable_log_record_release(log);\n+}\n+\n+static void fill_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tconst char *info = git_committer_info(0);\n+\tstruct ident_split split = {0};\n+\tint sign = 1;\n+\n+\tif (split_ident_line(&split, info, strlen(info)))\n+\t\tBUG(\"failed splitting committer info\");\n+\n+\treftable_log_record_release(log);\n+\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\tlog->value.update.name =\n+\t\txstrndup(split.name_begin, split.name_end - split.name_begin);\n+\tlog->value.update.email =\n+\t\txstrndup(split.mail_begin, split.mail_end - split.mail_begin);\n+\tlog->value.update.time = atol(split.date_begin);\n+\tif (*split.tz_begin == '-') {\n+\t\tsign = -1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\tif (*split.tz_begin == '+') {\n+\t\tsign = 1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\n+\tlog->value.update.tz_offset = sign * atoi(split.tz_begin);\n+}\n+\n+static int read_ref_without_reload(struct reftable_stack *stack,\n+\t\t\t\t const char *refname,\n+\t\t\t\t struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type)\n+{\n+\tstruct reftable_ref_record ref = {0};\n+\tint ret;\n+\n+\tret = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (reftable_ref_record_val1(&ref)) {\n+\t\toidread(oid, reftable_ref_record_val1(&ref));\n+\t} else {\n+\t\t/* We got a tombstone, which should not happen. */\n+\t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n+\t}\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn ret;\n+}\n+\n+static struct ref_store *reftable_be_init(struct repository *repo,\n+\t\t\t\t\t const char *gitdir,\n+\t\t\t\t\t unsigned int store_flags)\n+{\n+\tstruct reftable_ref_store *refs = xcalloc(1, sizeof(*refs));\n+\tstruct strbuf path = STRBUF_INIT;\n+\tint is_worktree;\n+\tmode_t mask;\n+\n+\tmask = umask(0);\n+\tumask(mask);\n+\n+\tbase_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);\n+\tstrmap_init(&refs->worktree_stacks);\n+\trefs->store_flags = store_flags;\n+\trefs->write_options.block_size = 4096;\n+\trefs->write_options.hash_id = repo->hash_algo->format_id;\n+\trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n+\n+\t/*\n+\t * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.\n+\t * This stack contains both the shared and the main worktree refs.\n+\t *\n+\t * Note that we don't try to resolve the path in case we have a\n+\t * worktree because `get_common_dir_noenv()` already does it for us.\n+\t */\n+\tis_worktree = get_common_dir_noenv(&path, gitdir);\n+\tif (!is_worktree) {\n+\t\tstrbuf_reset(&path);\n+\t\tstrbuf_realpath(&path, gitdir, 0);\n+\t}\n+\tstrbuf_addstr(&path, \"/reftable\");\n+\trefs->err = reftable_new_stack(&refs->main_stack, path.buf,\n+\t\t\t\t refs->write_options);\n+\tif (refs->err)\n+\t\tgoto done;\n+\n+\t/*\n+\t * If we're in a worktree we also need to set up the worktree reftable\n+\t * stack that is contained in the per-worktree GIT_DIR.\n+\t *\n+\t * Ideally, we would also add the stack to our worktree stack map. But\n+\t * we have no way to figure out the worktree name here and thus can't\n+\t * do it efficiently.\n+\t */\n+\tif (is_worktree) {\n+\t\tstrbuf_reset(&path);\n+\t\tstrbuf_addf(&path, \"%s/reftable\", gitdir);\n+\n+\t\trefs->err = reftable_new_stack(&refs->worktree_stack, path.buf,\n+\t\t\t\t\t refs->write_options);\n+\t\tif (refs->err)\n+\t\t\tgoto done;\n+\t}\n+\n+\tchdir_notify_reparent(\"reftables-backend $GIT_DIR\", &refs->base.gitdir);\n+\n+done:\n+\tassert(refs->err != REFTABLE_API_ERROR);\n+\tstrbuf_release(&path);\n+\treturn &refs->base;\n+}\n+\n+static int reftable_be_init_db(struct ref_store *ref_store,\n+\t\t\t int flags UNUSED,\n+\t\t\t struct strbuf *err UNUSED)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"init_db\");\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\tstrbuf_addf(&sb, \"%s/reftable\", refs->base.gitdir);\n+\tsafe_create_dir(sb.buf, 1);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/HEAD\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"ref: refs/heads/.invalid\");\n+\tadjust_shared_perm(sb.buf);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs\", refs->base.gitdir);\n+\tsafe_create_dir(sb.buf, 1);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs/heads\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\tadjust_shared_perm(sb.buf);\n+\n+\tstrbuf_release(&sb);\n+\treturn 0;\n+}\n+\n+struct reftable_ref_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_ref_record ref;\n+\tstruct object_id oid;\n+\n+\tconst char *prefix;\n+\tunsigned int flags;\n+\tint err;\n+};\n+\n+static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct reftable_ref_iterator *iter =\n+\t\t(struct reftable_ref_iterator *)ref_iterator;\n+\tstruct reftable_ref_store *refs = iter->refs;\n+\n+\twhile (!iter->err) {\n+\t\tint flags = 0;\n+\n+\t\titer->err = reftable_iterator_next_ref(&iter->iter, &iter->ref);\n+\t\tif (iter->err)\n+\t\t\tbreak;\n+\n+\t\t/*\n+\t\t * Unless DO_FOR_EACH_INCLUDE_ALL_REFS is set, we only list\n+\t\t * refs starting with \"refs/\" to mimic the \"files\" backend.\n+\t\t */\n+\t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_ALL_REFS) &&\n+\t\t !starts_with(iter->ref.refname, \"refs/\"))\n+\t\t\tcontinue;\n+\n+\t\tif (iter->prefix &&\n+\t\t strncmp(iter->prefix, iter->ref.refname, strlen(iter->prefix))) {\n+\t\t\titer->err = 1;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (iter->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n+\t\t parse_worktree_ref(iter->ref.refname, NULL, NULL, NULL) !=\n+\t\t\t REF_WORKTREE_CURRENT)\n+\t\t\tcontinue;\n+\n+\t\tswitch (iter->ref.value_type) {\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\toidread(&iter->oid, iter->ref.value.val1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\toidread(&iter->oid, iter->ref.value.val2.value);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF:\n+\t\t\tif (!refs_resolve_ref_unsafe(&iter->refs->base, iter->ref.refname,\n+\t\t\t\t\t\t RESOLVE_REF_READING, &iter->oid, &flags))\n+\t\t\t\toidclr(&iter->oid);\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tBUG(\"unhandled reference value type %d\", iter->ref.value_type);\n+\t\t}\n+\n+\t\tif (is_null_oid(&iter->oid))\n+\t\t\tflags |= REF_ISBROKEN;\n+\n+\t\tif (check_refname_format(iter->ref.refname, REFNAME_ALLOW_ONELEVEL)) {\n+\t\t\tif (!refname_is_safe(iter->ref.refname))\n+\t\t\t\tdie(_(\"refname is dangerous: %s\"), iter->ref.refname);\n+\t\t\toidclr(&iter->oid);\n+\t\t\tflags |= REF_BAD_NAME | REF_ISBROKEN;\n+\t\t}\n+\n+\t\tif (iter->flags & DO_FOR_EACH_OMIT_DANGLING_SYMREFS &&\n+\t\t flags & REF_ISSYMREF &&\n+\t\t flags & REF_ISBROKEN)\n+\t\t\tcontinue;\n+\n+\t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t !ref_resolves_to_object(iter->ref.refname, refs->base.repo,\n+\t\t\t\t\t &iter->oid, flags))\n+\t\t\t\tcontinue;\n+\n+\t\titer->base.refname = iter->ref.refname;\n+\t\titer->base.oid = &iter->oid;\n+\t\titer->base.flags = flags;\n+\n+\t\tbreak;\n+\t}\n+\n+\tif (iter->err > 0) {\n+\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n+\t\t\treturn ITER_ERROR;\n+\t\treturn ITER_DONE;\n+\t}\n+\n+\tif (iter->err < 0) {\n+\t\tref_iterator_abort(ref_iterator);\n+\t\treturn ITER_ERROR;\n+\t}\n+\n+\treturn ITER_OK;\n+}\n+\n+static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tstruct reftable_ref_iterator *iter =\n+\t\t(struct reftable_ref_iterator *)ref_iterator;\n+\n+\tif (iter->ref.value_type == REFTABLE_REF_VAL2) {\n+\t\toidread(peeled, iter->ref.value.val2.target_value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\n+\n+static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct reftable_ref_iterator *iter =\n+\t\t(struct reftable_ref_iterator *)ref_iterator;\n+\treftable_ref_record_release(&iter->ref);\n+\treftable_iterator_destroy(&iter->iter);\n+\tfree(iter);\n+\treturn ITER_DONE;\n+}\n+\n+static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n+\t.advance = reftable_ref_iterator_advance,\n+\t.peel = reftable_ref_iterator_peel,\n+\t.abort = reftable_ref_iterator_abort\n+};\n+\n+static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_store *refs,\n+\t\t\t\t\t\t\t struct reftable_stack *stack,\n+\t\t\t\t\t\t\t const char *prefix,\n+\t\t\t\t\t\t\t int flags)\n+{\n+\tstruct reftable_merged_table *merged_table;\n+\tstruct reftable_ref_iterator *iter;\n+\tint ret;\n+\n+\titer = xcalloc(1, sizeof(*iter));\n+\tbase_ref_iterator_init(&iter->base, &reftable_ref_iterator_vtable, 1);\n+\titer->prefix = prefix;\n+\titer->base.oid = &iter->oid;\n+\titer->flags = flags;\n+\titer->refs = refs;\n+\n+\tret = refs->err;\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tmerged_table = reftable_stack_merged_table(stack);\n+\n+\tret = reftable_merged_table_seek_ref(merged_table, &iter->iter, prefix);\n+\tif (ret)\n+\t\tgoto done;\n+\n+done:\n+\titer->err = ret;\n+\treturn iter;\n+}\n+\n+static enum iterator_selection iterator_select(struct ref_iterator *iter_worktree,\n+\t\t\t\t\t struct ref_iterator *iter_common,\n+\t\t\t\t\t void *cb_data UNUSED)\n+{\n+\tif (iter_worktree && !iter_common) {\n+\t\t/*\n+\t\t * Return the worktree ref if there are no more common refs.\n+\t\t */\n+\t\treturn ITER_SELECT_0;\n+\t} else if (iter_common) {\n+\t\t/*\n+\t\t * In case we have pending worktree and common refs we need to\n+\t\t * yield them based on their lexicographical order. Worktree\n+\t\t * refs that have the same name as common refs shadow the\n+\t\t * latter.\n+\t\t */\n+\t\tif (iter_worktree) {\n+\t\t\tint cmp = strcmp(iter_worktree->refname,\n+\t\t\t\t\t iter_common->refname);\n+\t\t\tif (cmp < 0)\n+\t\t\t\treturn ITER_SELECT_0;\n+\t\t\telse if (!cmp)\n+\t\t\t\treturn ITER_SELECT_0_SKIP_1;\n+\t\t}\n+\n+\t\t /*\n+\t\t * We now know that the lexicographically-next ref is a common\n+\t\t * ref. When the common ref is a shared one we return it.\n+\t\t */\n+\t\tif (parse_worktree_ref(iter_common->refname, NULL, NULL,\n+\t\t\t\t NULL) == REF_WORKTREE_SHARED)\n+\t\t\treturn ITER_SELECT_1;\n+\n+\t\t/*\n+\t\t * Otherwise, if the common ref is a per-worktree ref we skip\n+\t\t * it because it would belong to the main worktree, not ours.\n+\t\t */\n+\t\treturn ITER_SKIP_1;\n+\t} else {\n+\t\treturn ITER_DONE;\n+\t}\n+}\n+\n+static struct ref_iterator *reftable_be_iterator_begin(struct ref_store *ref_store,\n+\t\t\t\t\t\t const char *prefix,\n+\t\t\t\t\t\t const char **exclude_patterns,\n+\t\t\t\t\t\t unsigned int flags)\n+{\n+\tstruct reftable_ref_iterator *main_iter, *worktree_iter;\n+\tstruct reftable_ref_store *refs;\n+\tunsigned int required_flags = REF_STORE_READ;\n+\n+\tif (!(flags & DO_FOR_EACH_INCLUDE_BROKEN))\n+\t\trequired_flags |= REF_STORE_ODB;\n+\trefs = reftable_be_downcast(ref_store, required_flags, \"ref_iterator_begin\");\n+\n+\tmain_iter = ref_iterator_for_stack(refs, refs->main_stack, prefix, flags);\n+\n+\t/*\n+\t * The worktree stack is only set when we're in an actual worktree\n+\t * right now. If we aren't, then we return the common reftable\n+\t * iterator, only.\n+\t */\n+\t if (!refs->worktree_stack)\n+\t\treturn &main_iter->base;\n+\n+\t/*\n+\t * Otherwise we merge both the common and the per-worktree refs into a\n+\t * single iterator.\n+\t */\n+\tworktree_iter = ref_iterator_for_stack(refs, refs->worktree_stack, prefix, flags);\n+\treturn merge_ref_iterator_begin(1, &worktree_iter->base, &main_iter->base,\n+\t\t\t\t\titerator_select, NULL);\n+}\n+\n+static int reftable_be_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname,\n+\t\t\t\t struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type,\n+\t\t\t\t int *failure_errno)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_raw_ref\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tint ret;\n+\n+\tif (refs->err < 0)\n+\t\treturn refs->err;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tret = read_ref_without_reload(stack, refname, oid, referent, type);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\tif (ret > 0) {\n+\t\t*failure_errno = ENOENT;\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int reftable_be_read_symbolic_ref(struct ref_store *ref_store,\n+\t\t\t\t\t const char *refname,\n+\t\t\t\t\t struct strbuf *referent)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_symbolic_ref\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_ref_record ref = {0};\n+\tint ret;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tret = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (ret == 0 && ref.value_type == REFTABLE_REF_SYMREF)\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\telse\n+\t\tret = -1;\n+\n+\treftable_ref_record_release(&ref);\n+\treturn ret;\n+}\n+\n+/*\n+ * Return the refname under which update was originally requested.\n+ */\n+static const char *original_update_refname(struct ref_update *update)\n+{\n+\twhile (update->parent_update)\n+\t\tupdate = update->parent_update;\n+\treturn update->refname;\n+}\n+\n+struct reftable_transaction_update {\n+\tstruct ref_update *update;\n+\tstruct object_id current_oid;\n+};\n+\n+struct write_transaction_table_arg {\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_addition *addition;\n+\tstruct reftable_transaction_update *updates;\n+\tsize_t updates_nr;\n+\tsize_t updates_alloc;\n+\tsize_t updates_expected;\n+};\n+\n+struct reftable_transaction_data {\n+\tstruct write_transaction_table_arg *args;\n+\tsize_t args_nr, args_alloc;\n+};\n+\n+static void free_transaction_data(struct reftable_transaction_data *tx_data)\n+{\n+\tif (!tx_data)\n+\t\treturn;\n+\tfor (size_t i = 0; i < tx_data->args_nr; i++) {\n+\t\treftable_addition_destroy(tx_data->args[i].addition);\n+\t\tfree(tx_data->args[i].updates);\n+\t}\n+\tfree(tx_data->args);\n+\tfree(tx_data);\n+}\n+\n+/*\n+ * Prepare transaction update for the given reference update. This will cause\n+ * us to lock the corresponding reftable stack for concurrent modification.\n+ */\n+static int prepare_transaction_update(struct write_transaction_table_arg **out,\n+\t\t\t\t struct reftable_ref_store *refs,\n+\t\t\t\t struct reftable_transaction_data *tx_data,\n+\t\t\t\t struct ref_update *update,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_stack *stack = stack_for(refs, update->refname, NULL);\n+\tstruct write_transaction_table_arg *arg = NULL;\n+\tsize_t i;\n+\tint ret;\n+\n+\t/*\n+\t * Search for a preexisting stack update. If there is one then we add\n+\t * the update to it, otherwise we set up a new stack update.\n+\t */\n+\tfor (i = 0; !arg && i < tx_data->args_nr; i++)\n+\t\tif (tx_data->args[i].stack == stack)\n+\t\t\targ = &tx_data->args[i];\n+\n+\tif (!arg) {\n+\t\tstruct reftable_addition *addition;\n+\n+\t\tret = reftable_stack_reload(stack);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\n+\t\tret = reftable_stack_new_addition(&addition, stack);\n+\t\tif (ret) {\n+\t\t\tif (ret == REFTABLE_LOCK_ERROR)\n+\t\t\t\tstrbuf_addstr(err, \"cannot lock references\");\n+\t\t\treturn ret;\n+\t\t}\n+\n+\t\tALLOC_GROW(tx_data->args, tx_data->args_nr + 1,\n+\t\t\t tx_data->args_alloc);\n+\t\targ = &tx_data->args[tx_data->args_nr++];\n+\t\targ->refs = refs;\n+\t\targ->stack = stack;\n+\t\targ->addition = addition;\n+\t\targ->updates = NULL;\n+\t\targ->updates_nr = 0;\n+\t\targ->updates_alloc = 0;\n+\t\targ->updates_expected = 0;\n+\t}\n+\n+\targ->updates_expected++;\n+\n+\tif (out)\n+\t\t*out = arg;\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Queue a reference update for the correct stack. We potentially need to\n+ * handle multiple stack updates in a single transaction when it spans across\n+ * multiple worktrees.\n+ */\n+static int queue_transaction_update(struct reftable_ref_store *refs,\n+\t\t\t\t struct reftable_transaction_data *tx_data,\n+\t\t\t\t struct ref_update *update,\n+\t\t\t\t struct object_id *current_oid,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct write_transaction_table_arg *arg = NULL;\n+\tint ret;\n+\n+\tif (update->backend_data)\n+\t\tBUG(\"reference update queued more than once\");\n+\n+\tret = prepare_transaction_update(&arg, refs, tx_data, update, err);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\tALLOC_GROW(arg->updates, arg->updates_nr + 1,\n+\t\t arg->updates_alloc);\n+\targ->updates[arg->updates_nr].update = update;\n+\toidcpy(&arg->updates[arg->updates_nr].current_oid, current_oid);\n+\tupdate->backend_data = &arg->updates[arg->updates_nr++];\n+\n+\treturn 0;\n+}\n+\n+static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE|REF_STORE_MAIN, \"ref_transaction_prepare\");\n+\tstruct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT;\n+\tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct reftable_transaction_data *tx_data = NULL;\n+\tstruct object_id head_oid;\n+\tunsigned int head_type = 0;\n+\tsize_t i;\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\ttx_data = xcalloc(1, sizeof(*tx_data));\n+\n+\t/*\n+\t * Preprocess all updates. For one we check that there are no duplicate\n+\t * reference updates in this transaction. Second, we lock all stacks\n+\t * that will be modified during the transaction.\n+\t */\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tret = prepare_transaction_update(NULL, refs, tx_data,\n+\t\t\t\t\t\t transaction->updates[i], err);\n+\t\tif (ret)\n+\t\t\tgoto done;\n+\n+\t\tstring_list_append(&affected_refnames,\n+\t\t\t\t transaction->updates[i]->refname);\n+\t}\n+\n+\t/*\n+\t * Now that we have counted updates per stack we can preallocate their\n+\t * arrays. This avoids having to reallocate many times.\n+\t */\n+\tfor (i = 0; i < tx_data->args_nr; i++) {\n+\t\tCALLOC_ARRAY(tx_data->args[i].updates, tx_data->args[i].updates_expected);\n+\t\ttx_data->args[i].updates_alloc = tx_data->args[i].updates_expected;\n+\t}\n+\n+\t/*\n+\t * Fail if a refname appears more than once in the transaction.\n+\t * This code is taken from the files backend and is a good candidate to\n+\t * be moved into the generic layer.\n+\t */\n+\tstring_list_sort(&affected_refnames);\n+\tif (ref_update_reject_duplicates(&affected_refnames, err)) {\n+\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tret = read_ref_without_reload(stack_for(refs, \"HEAD\", NULL), \"HEAD\", &head_oid,\n+\t\t\t\t &head_referent, &head_type);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = transaction->updates[i];\n+\t\tstruct object_id current_oid = {0};\n+\t\tstruct reftable_stack *stack;\n+\t\tconst char *rewritten_ref;\n+\n+\t\tstack = stack_for(refs, u->refname, &rewritten_ref);\n+\n+\t\t/* Verify that the new object ID is valid. */\n+\t\tif ((u->flags & REF_HAVE_NEW) && !is_null_oid(&u->new_oid) &&\n+\t\t !(u->flags & REF_SKIP_OID_VERIFICATION) &&\n+\t\t !(u->flags & REF_LOG_ONLY)) {\n+\t\t\tstruct object *o = parse_object(refs->base.repo, &u->new_oid);\n+\t\t\tif (!o) {\n+\t\t\t\tstrbuf_addf(err,\n+\t\t\t\t\t _(\"trying to write ref '%s' with nonexistent object %s\"),\n+\t\t\t\t\t u->refname, oid_to_hex(&u->new_oid));\n+\t\t\t\tret = -1;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\n+\t\t\tif (o->type != OBJ_COMMIT && is_branch(u->refname)) {\n+\t\t\t\tstrbuf_addf(err, _(\"trying to write non-commit object %s to branch '%s'\"),\n+\t\t\t\t\t oid_to_hex(&u->new_oid), u->refname);\n+\t\t\t\tret = -1;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\n+\t\t/*\n+\t\t * When we update the reference that HEAD points to we enqueue\n+\t\t * a second log-only update for HEAD so that its reflog is\n+\t\t * updated accordingly.\n+\t\t */\n+\t\tif (head_type == REF_ISSYMREF &&\n+\t\t !(u->flags & REF_LOG_ONLY) &&\n+\t\t !(u->flags & REF_UPDATE_VIA_HEAD) &&\n+\t\t !strcmp(rewritten_ref, head_referent.buf)) {\n+\t\t\tstruct ref_update *new_update;\n+\n+\t\t\t/*\n+\t\t\t * First make sure that HEAD is not already in the\n+\t\t\t * transaction. This check is O(lg N) in the transaction\n+\t\t\t * size, but it happens at most once per transaction.\n+\t\t\t */\n+\t\t\tif (string_list_has_string(&affected_refnames, \"HEAD\")) {\n+\t\t\t\t/* An entry already existed */\n+\t\t\t\tstrbuf_addf(err,\n+\t\t\t\t\t _(\"multiple updates for 'HEAD' (including one \"\n+\t\t\t\t\t \"via its referent '%s') are not allowed\"),\n+\t\t\t\t\t u->refname);\n+\t\t\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\n+\t\t\tnew_update = ref_transaction_add_update(\n+\t\t\t\t\ttransaction, \"HEAD\",\n+\t\t\t\t\tu->flags | REF_LOG_ONLY | REF_NO_DEREF,\n+\t\t\t\t\t&u->new_oid, &u->old_oid, u->msg);\n+\t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n+\t\t}\n+\n+\t\tret = read_ref_without_reload(stack, rewritten_ref,\n+\t\t\t\t\t ¤t_oid, &referent, &u->type);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tif (ret > 0 && (!(u->flags & REF_HAVE_OLD) || is_null_oid(&u->old_oid))) {\n+\t\t\t/*\n+\t\t\t * The reference does not exist, and we either have no\n+\t\t\t * old object ID or expect the reference to not exist.\n+\t\t\t * We can thus skip below safety checks as well as the\n+\t\t\t * symref splitting. But we do want to verify that\n+\t\t\t * there is no conflicting reference here so that we\n+\t\t\t * can output a proper error message instead of failing\n+\t\t\t * at a later point.\n+\t\t\t */\n+\t\t\tret = refs_verify_refname_available(ref_store, u->refname,\n+\t\t\t\t\t\t\t &affected_refnames, NULL, err);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto done;\n+\n+\t\t\t/*\n+\t\t\t * There is no need to write the reference deletion\n+\t\t\t * when the reference in question doesn't exist.\n+\t\t\t */\n+\t\t\t if (u->flags & REF_HAVE_NEW && !is_null_oid(&u->new_oid)) {\n+\t\t\t\t ret = queue_transaction_update(refs, tx_data, u,\n+\t\t\t\t\t\t\t\t¤t_oid, err);\n+\t\t\t\t if (ret)\n+\t\t\t\t\t goto done;\n+\t\t\t }\n+\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (ret > 0) {\n+\t\t\t/* The reference does not exist, but we expected it to. */\n+\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t \"unable to resolve reference '%s'\"),\n+\t\t\t\t original_update_refname(u), u->refname);\n+\t\t\tret = -1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (u->type & REF_ISSYMREF) {\n+\t\t\t/*\n+\t\t\t * The reftable stack is locked at this point already,\n+\t\t\t * so it is safe to call `refs_resolve_ref_unsafe()`\n+\t\t\t * here without causing races.\n+\t\t\t */\n+\t\t\tconst char *resolved = refs_resolve_ref_unsafe(&refs->base, u->refname, 0,\n+\t\t\t\t\t\t\t\t ¤t_oid, NULL);\n+\n+\t\t\tif (u->flags & REF_NO_DEREF) {\n+\t\t\t\tif (u->flags & REF_HAVE_OLD && !resolved) {\n+\t\t\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t\t\t \"error reading reference\"), u->refname);\n+\t\t\t\t\tret = -1;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n+\t\t\t} else {\n+\t\t\t\tstruct ref_update *new_update;\n+\t\t\t\tint new_flags;\n+\n+\t\t\t\tnew_flags = u->flags;\n+\t\t\t\tif (!strcmp(rewritten_ref, \"HEAD\"))\n+\t\t\t\t\tnew_flags |= REF_UPDATE_VIA_HEAD;\n+\n+\t\t\t\t/*\n+\t\t\t\t * If we are updating a symref (eg. HEAD), we should also\n+\t\t\t\t * update the branch that the symref points to.\n+\t\t\t\t *\n+\t\t\t\t * This is generic functionality, and would be better\n+\t\t\t\t * done in refs.c, but the current implementation is\n+\t\t\t\t * intertwined with the locking in files-backend.c.\n+\t\t\t\t */\n+\t\t\t\tnew_update = ref_transaction_add_update(\n+\t\t\t\t\t\ttransaction, referent.buf, new_flags,\n+\t\t\t\t\t\t&u->new_oid, &u->old_oid, u->msg);\n+\t\t\t\tnew_update->parent_update = u;\n+\n+\t\t\t\t/*\n+\t\t\t\t * Change the symbolic ref update to log only. Also, it\n+\t\t\t\t * doesn't need to check its old OID value, as that will be\n+\t\t\t\t * done when new_update is processed.\n+\t\t\t\t */\n+\t\t\t\tu->flags |= REF_LOG_ONLY | REF_NO_DEREF;\n+\t\t\t\tu->flags &= ~REF_HAVE_OLD;\n+\n+\t\t\t\tif (string_list_has_string(&affected_refnames, new_update->refname)) {\n+\t\t\t\t\tstrbuf_addf(err,\n+\t\t\t\t\t\t _(\"multiple updates for '%s' (including one \"\n+\t\t\t\t\t\t \"via symref '%s') are not allowed\"),\n+\t\t\t\t\t\t referent.buf, u->refname);\n+\t\t\t\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n+\t\t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n+\t\t\t}\n+\t\t}\n+\n+\t\t/*\n+\t\t * Verify that the old object matches our expectations. Note\n+\t\t * that the error messages here do not make a lot of sense in\n+\t\t * the context of the reftable backend as we never lock\n+\t\t * individual refs. But the error messages match what the files\n+\t\t * backend returns, which keeps our tests happy.\n+\t\t */\n+\t\tif (u->flags & REF_HAVE_OLD && !oideq(¤t_oid, &u->old_oid)) {\n+\t\t\tif (is_null_oid(&u->old_oid))\n+\t\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t\t \"reference already exists\"),\n+\t\t\t\t\t original_update_refname(u));\n+\t\t\telse if (is_null_oid(¤t_oid))\n+\t\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t\t \"reference is missing but expected %s\"),\n+\t\t\t\t\t original_update_refname(u),\n+\t\t\t\t\t oid_to_hex(&u->old_oid));\n+\t\t\telse\n+\t\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t\t \"is at %s but expected %s\"),\n+\t\t\t\t\t original_update_refname(u),\n+\t\t\t\t\t oid_to_hex(¤t_oid),\n+\t\t\t\t\t oid_to_hex(&u->old_oid));\n+\t\t\tret = -1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\t/*\n+\t\t * If all of the following conditions are true:\n+\t\t *\n+\t\t * - We're not about to write a symref.\n+\t\t * - We're not about to write a log-only entry.\n+\t\t * - Old and new object ID are different.\n+\t\t *\n+\t\t * Then we're essentially doing a no-op update that can be\n+\t\t * skipped. This is not only for the sake of efficiency, but\n+\t\t * also skips writing unneeded reflog entries.\n+\t\t */\n+\t\tif ((u->type & REF_ISSYMREF) ||\n+\t\t (u->flags & REF_LOG_ONLY) ||\n+\t\t (u->flags & REF_HAVE_NEW && !oideq(¤t_oid, &u->new_oid))) {\n+\t\t\tret = queue_transaction_update(refs, tx_data, u,\n+\t\t\t\t\t\t ¤t_oid, err);\n+\t\t\tif (ret)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\ttransaction->backend_data = tx_data;\n+\ttransaction->state = REF_TRANSACTION_PREPARED;\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\tif (ret < 0) {\n+\t\tfree_transaction_data(tx_data);\n+\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\t\tif (!err->len)\n+\t\t\tstrbuf_addf(err, _(\"reftable: transaction prepare: %s\"),\n+\t\t\t\t reftable_error_str(ret));\n+\t}\n+\tstring_list_clear(&affected_refnames, 0);\n+\tstrbuf_release(&referent);\n+\tstrbuf_release(&head_referent);\n+\n+\treturn ret;\n+}\n+\n+static int reftable_be_transaction_abort(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_transaction_data *tx_data = transaction->backend_data;\n+\tfree_transaction_data(tx_data);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\treturn 0;\n+}\n+\n+static int transaction_update_cmp(const void *a, const void *b)\n+{\n+\treturn strcmp(((struct reftable_transaction_update *)a)->update->refname,\n+\t\t ((struct reftable_transaction_update *)b)->update->refname);\n+}\n+\n+static int write_transaction_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct write_transaction_table_arg *arg = cb_data;\n+\tstruct reftable_merged_table *mt =\n+\t\treftable_stack_merged_table(arg->stack);\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_log_record *logs = NULL;\n+\tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tint ret = 0;\n+\n+\tQSORT(arg->updates, arg->updates_nr, transaction_update_cmp);\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tfor (i = 0; i < arg->updates_nr; i++) {\n+\t\tstruct reftable_transaction_update *tx_update = &arg->updates[i];\n+\t\tstruct ref_update *u = tx_update->update;\n+\n+\t\t/*\n+\t\t * Write a reflog entry when updating a ref to point to\n+\t\t * something new in either of the following cases:\n+\t\t *\n+\t\t * - The reference is about to be deleted. We always want to\n+\t\t * delete the reflog in that case.\n+\t\t * - REF_FORCE_CREATE_REFLOG is set, asking us to always create\n+\t\t * the reflog entry.\n+\t\t * - `core.logAllRefUpdates` tells us to create the reflog for\n+\t\t * the given ref.\n+\t\t */\n+\t\tif (u->flags & REF_HAVE_NEW && !(u->type & REF_ISSYMREF) && is_null_oid(&u->new_oid)) {\n+\t\t\tstruct reftable_log_record log = {0};\n+\t\t\tstruct reftable_iterator it = {0};\n+\n+\t\t\t/*\n+\t\t\t * When deleting refs we also delete all reflog entries\n+\t\t\t * with them. While it is not strictly required to\n+\t\t\t * delete reflogs together with their refs, this\n+\t\t\t * matches the behaviour of the files backend.\n+\t\t\t *\n+\t\t\t * Unfortunately, we have no better way than to delete\n+\t\t\t * all reflog entries one by one.\n+\t\t\t */\n+\t\t\tret = reftable_merged_table_seek_log(mt, &it, u->refname);\n+\t\t\twhile (ret == 0) {\n+\t\t\t\tstruct reftable_log_record *tombstone;\n+\n+\t\t\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\t\t\tif (ret < 0)\n+\t\t\t\t\tbreak;\n+\t\t\t\tif (ret > 0 || strcmp(log.refname, u->refname)) {\n+\t\t\t\t\tret = 0;\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\n+\t\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\t\t\ttombstone = &logs[logs_nr++];\n+\t\t\t\ttombstone->refname = xstrdup(u->refname);\n+\t\t\t\ttombstone->value_type = REFTABLE_LOG_DELETION;\n+\t\t\t\ttombstone->update_index = log.update_index;\n+\t\t\t}\n+\n+\t\t\treftable_log_record_release(&log);\n+\t\t\treftable_iterator_destroy(&it);\n+\n+\t\t\tif (ret)\n+\t\t\t\tgoto done;\n+\t\t} else if (u->flags & REF_HAVE_NEW &&\n+\t\t\t (u->flags & REF_FORCE_CREATE_REFLOG ||\n+\t\t\t should_write_log(&arg->refs->base, u->refname))) {\n+\t\t\tstruct reftable_log_record *log;\n+\n+\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\t\tlog = &logs[logs_nr++];\n+\t\t\tmemset(log, 0, sizeof(*log));\n+\n+\t\t\tfill_reftable_log_record(log);\n+\t\t\tlog->update_index = ts;\n+\t\t\tlog->refname = xstrdup(u->refname);\n+\t\t\tlog->value.update.new_hash = u->new_oid.hash;\n+\t\t\tlog->value.update.old_hash = tx_update->current_oid.hash;\n+\t\t\tlog->value.update.message =\n+\t\t\t\txstrndup(u->msg, arg->refs->write_options.block_size / 2);\n+\t\t}\n+\n+\t\tif (u->flags & REF_LOG_ONLY)\n+\t\t\tcontinue;\n+\n+\t\tif (u->flags & REF_HAVE_NEW && is_null_oid(&u->new_oid)) {\n+\t\t\tstruct reftable_ref_record ref = {\n+\t\t\t\t.refname = (char *)u->refname,\n+\t\t\t\t.update_index = ts,\n+\t\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t};\n+\n+\t\t\tret = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto done;\n+\t\t} else if (u->flags & REF_HAVE_NEW) {\n+\t\t\tstruct reftable_ref_record ref = {0};\n+\t\t\tstruct object_id peeled;\n+\t\t\tint peel_error;\n+\n+\t\t\tref.refname = (char *)u->refname;\n+\t\t\tref.update_index = ts;\n+\n+\t\t\tpeel_error = peel_object(&u->new_oid, &peeled);\n+\t\t\tif (!peel_error) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\t\t\tmemcpy(ref.value.val2.target_value, peeled.hash, GIT_MAX_RAWSZ);\n+\t\t\t\tmemcpy(ref.value.val2.value, u->new_oid.hash, GIT_MAX_RAWSZ);\n+\t\t\t} else if (!is_null_oid(&u->new_oid)) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\t\t\tmemcpy(ref.value.val1, u->new_oid.hash, GIT_MAX_RAWSZ);\n+\t\t\t}\n+\n+\t\t\tret = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Logs are written at the end so that we do not have intermixed ref\n+\t * and log blocks.\n+\t */\n+\tif (logs) {\n+\t\tret = reftable_writer_add_logs(writer, logs, logs_nr);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\tfor (i = 0; i < logs_nr; i++)\n+\t\tclear_reftable_log_record(&logs[i]);\n+\tfree(logs);\n+\treturn ret;\n+}\n+\n+static int reftable_be_transaction_finish(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_transaction_data *tx_data = transaction->backend_data;\n+\tint ret = 0;\n+\n+\tfor (size_t i = 0; i < tx_data->args_nr; i++) {\n+\t\tret = reftable_addition_add(tx_data->args[i].addition,\n+\t\t\t\t\t write_transaction_table, &tx_data->args[i]);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\n+\t\tret = reftable_addition_commit(tx_data->args[i].addition);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\tfree_transaction_data(tx_data);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\n+\tif (ret) {\n+\t\tstrbuf_addf(err, _(\"reftable: transaction failure: %s\"),\n+\t\t\t reftable_error_str(ret));\n+\t\treturn -1;\n+\t}\n+\treturn ret;\n+}\n+\n+static int reftable_be_initial_transaction_commit(struct ref_store *ref_store UNUSED,\n+\t\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t\t struct strbuf *err)\n+{\n+\treturn ref_transaction_commit(transaction, err);\n+}\n+\n+static int reftable_be_pack_refs(struct ref_store *ref_store,\n+\t\t\t\t struct pack_refs_opts *opts)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE | REF_STORE_ODB, \"pack_refs\");\n+\tstruct reftable_stack *stack;\n+\tint ret;\n+\n+\tif (refs->err)\n+\t\treturn refs->err;\n+\n+\tstack = refs->worktree_stack;\n+\tif (!stack)\n+\t\tstack = refs->main_stack;\n+\n+\tret = reftable_stack_compact_all(stack, NULL);\n+\tif (ret)\n+\t\tgoto out;\n+\tret = reftable_stack_clean(stack);\n+\tif (ret)\n+\t\tgoto out;\n+\n+out:\n+\treturn ret;\n+}\n+\n+struct write_create_symref_arg {\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+\tconst char *target;\n+\tconst char *logmsg;\n+};\n+\n+static int write_create_symref_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct write_create_symref_arg *create = cb_data;\n+\tuint64_t ts = reftable_stack_next_update_index(create->stack);\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)create->refname,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = (char *)create->target,\n+\t\t.update_index = ts,\n+\t};\n+\tstruct reftable_log_record log = {0};\n+\tstruct object_id new_oid;\n+\tstruct object_id old_oid;\n+\tint ret;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tret = reftable_writer_add_ref(writer, &ref);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\t/*\n+\t * Note that it is important to try and resolve the reference before we\n+\t * write the log entry. This is because `should_write_log()` will munge\n+\t * `core.logAllRefUpdates`, which is undesirable when we create a new\n+\t * repository because it would be written into the config. As HEAD will\n+\t * not resolve for new repositories this ordering will ensure that this\n+\t * never happens.\n+\t */\n+\tif (!create->logmsg ||\n+\t !refs_resolve_ref_unsafe(&create->refs->base, create->target,\n+\t\t\t\t RESOLVE_REF_READING, &new_oid, NULL) ||\n+\t !should_write_log(&create->refs->base, create->refname))\n+\t\treturn 0;\n+\n+\tfill_reftable_log_record(&log);\n+\tlog.refname = xstrdup(create->refname);\n+\tlog.update_index = ts;\n+\tlog.value.update.message = xstrndup(create->logmsg,\n+\t\t\t\t\t create->refs->write_options.block_size / 2);\n+\tlog.value.update.new_hash = new_oid.hash;\n+\tif (refs_resolve_ref_unsafe(&create->refs->base, create->refname,\n+\t\t\t\t RESOLVE_REF_READING, &old_oid, NULL))\n+\t\tlog.value.update.old_hash = old_oid.hash;\n+\n+\tret = reftable_writer_add_log(writer, &log);\n+\tclear_reftable_log_record(&log);\n+\treturn ret;\n+}\n+\n+static int reftable_be_create_symref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const char *target,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"create_symref\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct write_create_symref_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t\t.target = target,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_add(stack, &write_create_symref_table, &arg);\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\tif (ret)\n+\t\terror(\"unable to write symref for %s: %s\", refname,\n+\t\t reftable_error_str(ret));\n+\treturn ret;\n+}\n+\n+struct write_copy_arg {\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *oldname;\n+\tconst char *newname;\n+\tconst char *logmsg;\n+\tint delete_old;\n+};\n+\n+static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct write_copy_arg *arg = cb_data;\n+\tuint64_t deletion_ts, creation_ts;\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(arg->stack);\n+\tstruct reftable_ref_record old_ref = {0}, refs[2] = {0};\n+\tstruct reftable_log_record old_log = {0}, *logs = NULL;\n+\tstruct reftable_iterator it = {0};\n+\tstruct string_list skip = STRING_LIST_INIT_NODUP;\n+\tstruct strbuf errbuf = STRBUF_INIT;\n+\tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tint ret;\n+\n+\tif (reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref)) {\n+\t\tret = error(_(\"refname %s not found\"), arg->oldname);\n+\t\tgoto done;\n+\t}\n+\tif (old_ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tret = error(_(\"refname %s is a symbolic ref, copying it is not supported\"),\n+\t\t\t arg->oldname);\n+\t\tgoto done;\n+\t}\n+\n+\t/*\n+\t * There's nothing to do in case the old and new name are the same, so\n+\t * we exit early in that case.\n+\t */\n+\tif (!strcmp(arg->oldname, arg->newname)) {\n+\t\tret = 0;\n+\t\tgoto done;\n+\t}\n+\n+\t/*\n+\t * Verify that the new refname is available.\n+\t */\n+\tstring_list_insert(&skip, arg->oldname);\n+\tret = refs_verify_refname_available(&arg->refs->base, arg->newname,\n+\t\t\t\t\t NULL, &skip, &errbuf);\n+\tif (ret < 0) {\n+\t\terror(\"%s\", errbuf.buf);\n+\t\tgoto done;\n+\t}\n+\n+\t/*\n+\t * When deleting the old reference we have to use two update indices:\n+\t * once to delete the old ref and its reflog, and once to create the\n+\t * new ref and its reflog. They need to be staged with two separate\n+\t * indices because the new reflog needs to encode both the deletion of\n+\t * the old branch and the creation of the new branch, and we cannot do\n+\t * two changes to a reflog in a single update.\n+\t */\n+\tdeletion_ts = creation_ts = reftable_stack_next_update_index(arg->stack);\n+\tif (arg->delete_old)\n+\t\tcreation_ts++;\n+\treftable_writer_set_limits(writer, deletion_ts, creation_ts);\n+\n+\t/*\n+\t * Add the new reference. If this is a rename then we also delete the\n+\t * old reference.\n+\t */\n+\trefs[0] = old_ref;\n+\trefs[0].refname = (char *)arg->newname;\n+\trefs[0].update_index = creation_ts;\n+\tif (arg->delete_old) {\n+\t\trefs[1].refname = (char *)arg->oldname;\n+\t\trefs[1].value_type = REFTABLE_REF_DELETION;\n+\t\trefs[1].update_index = deletion_ts;\n+\t}\n+\tret = reftable_writer_add_refs(writer, refs, arg->delete_old ? 2 : 1);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\t/*\n+\t * When deleting the old branch we need to create a reflog entry on the\n+\t * new branch name that indicates that the old branch has been deleted\n+\t * and then recreated. This is a tad weird, but matches what the files\n+\t * backend does.\n+\t */\n+\tif (arg->delete_old) {\n+\t\tstruct strbuf head_referent = STRBUF_INIT;\n+\t\tstruct object_id head_oid;\n+\t\tint append_head_reflog;\n+\t\tunsigned head_type = 0;\n+\n+\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\tmemset(&logs[logs_nr], 0, sizeof(logs[logs_nr]));\n+\t\tfill_reftable_log_record(&logs[logs_nr]);\n+\t\tlogs[logs_nr].refname = (char *)arg->newname;\n+\t\tlogs[logs_nr].update_index = deletion_ts;\n+\t\tlogs[logs_nr].value.update.message =\n+\t\t\txstrndup(arg->logmsg, arg->refs->write_options.block_size / 2);\n+\t\tlogs[logs_nr].value.update.old_hash = old_ref.value.val1;\n+\t\tlogs_nr++;\n+\n+\t\tret = read_ref_without_reload(arg->stack, \"HEAD\", &head_oid, &head_referent, &head_type);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tappend_head_reflog = (head_type & REF_ISSYMREF) && !strcmp(head_referent.buf, arg->oldname);\n+\t\tstrbuf_release(&head_referent);\n+\n+\t\t/*\n+\t\t * The files backend uses `refs_delete_ref()` to delete the old\n+\t\t * branch name, which will append a reflog entry for HEAD in\n+\t\t * case it points to the old branch.\n+\t\t */\n+\t\tif (append_head_reflog) {\n+\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\t\tlogs[logs_nr] = logs[logs_nr - 1];\n+\t\t\tlogs[logs_nr].refname = \"HEAD\";\n+\t\t\tlogs_nr++;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Create the reflog entry for the newly created branch.\n+\t */\n+\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\tmemset(&logs[logs_nr], 0, sizeof(logs[logs_nr]));\n+\tfill_reftable_log_record(&logs[logs_nr]);\n+\tlogs[logs_nr].refname = (char *)arg->newname;\n+\tlogs[logs_nr].update_index = creation_ts;\n+\tlogs[logs_nr].value.update.message =\n+\t\txstrndup(arg->logmsg, arg->refs->write_options.block_size / 2);\n+\tlogs[logs_nr].value.update.new_hash = old_ref.value.val1;\n+\tlogs_nr++;\n+\n+\t/*\n+\t * In addition to writing the reflog entry for the new branch, we also\n+\t * copy over all log entries from the old reflog. Last but not least,\n+\t * when renaming we also have to delete all the old reflog entries.\n+\t */\n+\tret = reftable_merged_table_seek_log(mt, &it, arg->oldname);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\twhile (1) {\n+\t\tret = reftable_iterator_next_log(&it, &old_log);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tif (ret > 0 || strcmp(old_log.refname, arg->oldname)) {\n+\t\t\tret = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tfree(old_log.refname);\n+\n+\t\t/*\n+\t\t * Copy over the old reflog entry with the new refname.\n+\t\t */\n+\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\tlogs[logs_nr] = old_log;\n+\t\tlogs[logs_nr].refname = (char *)arg->newname;\n+\t\tlogs_nr++;\n+\n+\t\t/*\n+\t\t * Delete the old reflog entry in case we are renaming.\n+\t\t */\n+\t\tif (arg->delete_old) {\n+\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\t\tmemset(&logs[logs_nr], 0, sizeof(logs[logs_nr]));\n+\t\t\tlogs[logs_nr].refname = (char *)arg->oldname;\n+\t\t\tlogs[logs_nr].value_type = REFTABLE_LOG_DELETION;\n+\t\t\tlogs[logs_nr].update_index = old_log.update_index;\n+\t\t\tlogs_nr++;\n+\t\t}\n+\n+\t\t/*\n+\t\t * Transfer ownership of the log record we're iterating over to\n+\t\t * the array of log records. Otherwise, the pointers would get\n+\t\t * free'd or reallocated by the iterator.\n+\t\t */\n+\t\tmemset(&old_log, 0, sizeof(old_log));\n+\t}\n+\n+\tret = reftable_writer_add_logs(writer, logs, logs_nr);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treftable_iterator_destroy(&it);\n+\tstring_list_clear(&skip, 0);\n+\tstrbuf_release(&errbuf);\n+\tfor (i = 0; i < logs_nr; i++) {\n+\t\tif (!strcmp(logs[i].refname, \"HEAD\"))\n+\t\t\tcontinue;\n+\t\tif (logs[i].value.update.old_hash == old_ref.value.val1)\n+\t\t\tlogs[i].value.update.old_hash = NULL;\n+\t\tif (logs[i].value.update.new_hash == old_ref.value.val1)\n+\t\t\tlogs[i].value.update.new_hash = NULL;\n+\t\tlogs[i].refname = NULL;\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tfree(logs);\n+\treftable_ref_record_release(&old_ref);\n+\treftable_log_record_release(&old_log);\n+\treturn ret;\n+}\n+\n+static int reftable_be_rename_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname,\n+\t\t\t\t const char *newrefname,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"rename_ref\");\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname, &newrefname);\n+\tstruct write_copy_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t\t.delete_old = 1,\n+\t};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\tret = reftable_stack_add(stack, &write_copy_table, &arg);\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treturn ret;\n+}\n+\n+static int reftable_be_copy_ref(struct ref_store *ref_store,\n+\t\t\t\tconst char *oldrefname,\n+\t\t\t\tconst char *newrefname,\n+\t\t\t\tconst char *logmsg)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"copy_ref\");\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname, &newrefname);\n+\tstruct write_copy_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\tret = reftable_stack_add(stack, &write_copy_table, &arg);\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treturn ret;\n+}\n+\n+struct reftable_reflog_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_log_record log;\n+\tstruct object_id oid;\n+\tchar *last_name;\n+\tint err;\n+};\n+\n+static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct reftable_reflog_iterator *iter =\n+\t\t(struct reftable_reflog_iterator *)ref_iterator;\n+\n+\twhile (!iter->err) {\n+\t\tint flags;\n+\n+\t\titer->err = reftable_iterator_next_log(&iter->iter, &iter->log);\n+\t\tif (iter->err)\n+\t\t\tbreak;\n+\n+\t\t/*\n+\t\t * We want the refnames that we have reflogs for, so we skip if\n+\t\t * we've already produced this name. This could be faster by\n+\t\t * seeking directly to reflog@update_index==0.\n+\t\t */\n+\t\tif (iter->last_name && !strcmp(iter->log.refname, iter->last_name))\n+\t\t\tcontinue;\n+\n+\t\tif (!refs_resolve_ref_unsafe(&iter->refs->base, iter->log.refname,\n+\t\t\t\t\t 0, &iter->oid, &flags)) {\n+\t\t\terror(_(\"bad ref for %s\"), iter->log.refname);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfree(iter->last_name);\n+\t\titer->last_name = xstrdup(iter->log.refname);\n+\t\titer->base.refname = iter->log.refname;\n+\t\titer->base.oid = &iter->oid;\n+\t\titer->base.flags = flags;\n+\n+\t\tbreak;\n+\t}\n+\n+\tif (iter->err > 0) {\n+\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n+\t\t\treturn ITER_ERROR;\n+\t\treturn ITER_DONE;\n+\t}\n+\n+\tif (iter->err < 0) {\n+\t\tref_iterator_abort(ref_iterator);\n+\t\treturn ITER_ERROR;\n+\t}\n+\n+\treturn ITER_OK;\n+}\n+\n+static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t\t\t struct object_id *peeled)\n+{\n+\tBUG(\"reftable reflog iterator cannot be peeled\");\n+\treturn -1;\n+}\n+\n+static int reftable_reflog_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct reftable_reflog_iterator *iter =\n+\t\t(struct reftable_reflog_iterator *)ref_iterator;\n+\treftable_log_record_release(&iter->log);\n+\treftable_iterator_destroy(&iter->iter);\n+\tfree(iter->last_name);\n+\tfree(iter);\n+\treturn ITER_DONE;\n+}\n+\n+static struct ref_iterator_vtable reftable_reflog_iterator_vtable = {\n+\t.advance = reftable_reflog_iterator_advance,\n+\t.peel = reftable_reflog_iterator_peel,\n+\t.abort = reftable_reflog_iterator_abort\n+};\n+\n+static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftable_ref_store *refs,\n+\t\t\t\t\t\t\t\t struct reftable_stack *stack)\n+{\n+\tstruct reftable_merged_table *merged_table;\n+\tstruct reftable_reflog_iterator *iter;\n+\tint ret;\n+\n+\titer = xcalloc(1, sizeof(*iter));\n+\tbase_ref_iterator_init(&iter->base, &reftable_reflog_iterator_vtable, 1);\n+\titer->refs = refs;\n+\titer->base.oid = &iter->oid;\n+\n+\tret = refs->err;\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(refs->main_stack);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tmerged_table = reftable_stack_merged_table(stack);\n+\n+\tret = reftable_merged_table_seek_log(merged_table, &iter->iter, \"\");\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+done:\n+\titer->err = ret;\n+\treturn iter;\n+}\n+\n+static struct ref_iterator *reftable_be_reflog_iterator_begin(struct ref_store *ref_store)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_iterator_begin\");\n+\tstruct reftable_reflog_iterator *main_iter, *worktree_iter;\n+\n+\tmain_iter = reflog_iterator_for_stack(refs, refs->main_stack);\n+\tif (!refs->worktree_stack)\n+\t\treturn &main_iter->base;\n+\n+\tworktree_iter = reflog_iterator_for_stack(refs, refs->worktree_stack);\n+\n+\treturn merge_ref_iterator_begin(1, &worktree_iter->base, &main_iter->base,\n+\t\t\t\t\titerator_select, NULL);\n+}\n+\n+static int yield_log_record(struct reftable_log_record *log,\n+\t\t\t each_reflog_ent_fn fn,\n+\t\t\t void *cb_data)\n+{\n+\tstruct object_id old_oid, new_oid;\n+\tconst char *full_committer;\n+\n+\toidread(&old_oid, log->value.update.old_hash);\n+\toidread(&new_oid, log->value.update.new_hash);\n+\n+\t/*\n+\t * When both the old object ID and the new object ID are null\n+\t * then this is the reflog existence marker. The caller must\n+\t * not be aware of it.\n+\t */\n+\tif (is_null_oid(&old_oid) && is_null_oid(&new_oid))\n+\t\treturn 0;\n+\n+\tfull_committer = fmt_ident(log->value.update.name, log->value.update.email,\n+\t\t\t\t WANT_COMMITTER_IDENT, NULL, IDENT_NO_DATE);\n+\treturn fn(&old_oid, &new_oid, full_committer,\n+\t\t log->value.update.time, log->value.update.tz_offset,\n+\t\t log->value.update.message, cb_data);\n+}\n+\n+static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n+\t\t\t\t\t\t const char *refname,\n+\t\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t\t void *cb_data)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent_reverse\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record log = {0};\n+\tstruct reftable_iterator it = {0};\n+\tint ret;\n+\n+\tif (refs->err < 0)\n+\t\treturn refs->err;\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\tret = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (!ret) {\n+\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\tif (ret < 0)\n+\t\t\tbreak;\n+\t\tif (ret > 0 || strcmp(log.refname, refname)) {\n+\t\t\tret = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tret = yield_log_record(&log, fn, cb_data);\n+\t\tif (ret)\n+\t\t\tbreak;\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn ret;\n+}\n+\n+static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store,\n+\t\t\t\t\t const char *refname,\n+\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t void *cb_data)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record *logs = NULL;\n+\tstruct reftable_iterator it = {0};\n+\tsize_t logs_alloc = 0, logs_nr = 0, i;\n+\tint ret;\n+\n+\tif (refs->err < 0)\n+\t\treturn refs->err;\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\tret = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (!ret) {\n+\t\tstruct reftable_log_record log = {0};\n+\n+\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tif (ret > 0 || strcmp(log.refname, refname)) {\n+\t\t\treftable_log_record_release(&log);\n+\t\t\tret = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\tlogs[logs_nr++] = log;\n+\t}\n+\n+\tfor (i = logs_nr; i--;) {\n+\t\tret = yield_log_record(&logs[i], fn, cb_data);\n+\t\tif (ret)\n+\t\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\tfor (i = 0; i < logs_nr; i++)\n+\t\treftable_log_record_release(&logs[i]);\n+\tfree(logs);\n+\treturn ret;\n+}\n+\n+static int reftable_be_reflog_exists(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_exists\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record log = {0};\n+\tstruct reftable_iterator it = {0};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\t/*\n+\t * Check whether we get at least one log record for the given ref name.\n+\t * If so, the reflog exists, otherwise it doesn't.\n+\t */\n+\tret = reftable_iterator_next_log(&it, &log);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\tif (ret > 0) {\n+\t\tret = 0;\n+\t\tgoto done;\n+\t}\n+\n+\tret = strcmp(log.refname, refname) == 0;\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\tif (ret < 0)\n+\t\tret = 0;\n+\treturn ret;\n+}\n+\n+struct write_reflog_existence_arg {\n+\tstruct reftable_ref_store *refs;\n+\tconst char *refname;\n+\tstruct reftable_stack *stack;\n+};\n+\n+static int write_reflog_existence_table(struct reftable_writer *writer,\n+\t\t\t\t\tvoid *cb_data)\n+{\n+\tstruct write_reflog_existence_arg *arg = cb_data;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_log_record log = {0};\n+\tint ret;\n+\n+\tret = reftable_stack_read_log(arg->stack, arg->refname, &log);\n+\tif (ret <= 0)\n+\t\tgoto done;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\t/*\n+\t * The existence entry has both old and new object ID set to the the\n+\t * null object ID. Our iterators are aware of this and will not present\n+\t * them to their callers.\n+\t */\n+\tlog.refname = xstrdup(arg->refname);\n+\tlog.update_index = ts;\n+\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\tret = reftable_writer_add_log(writer, &log);\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treftable_log_record_release(&log);\n+\treturn ret;\n+}\n+\n+static int reftable_be_create_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname,\n+\t\t\t\t struct strbuf *errmsg)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"create_reflog\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct write_reflog_existence_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_add(stack, &write_reflog_existence_table, &arg);\n+\n+done:\n+\treturn ret;\n+}\n+\n+struct write_reflog_delete_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+};\n+\n+static int write_reflog_delete_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct write_reflog_delete_arg *arg = cb_data;\n+\tstruct reftable_merged_table *mt =\n+\t\treftable_stack_merged_table(arg->stack);\n+\tstruct reftable_log_record log = {0}, tombstone = {0};\n+\tstruct reftable_iterator it = {0};\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint ret;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\t/*\n+\t * In order to delete a table we need to delete all reflog entries one\n+\t * by one. This is inefficient, but the reftable format does not have a\n+\t * better marker right now.\n+\t */\n+\tret = reftable_merged_table_seek_log(mt, &it, arg->refname);\n+\twhile (ret == 0) {\n+\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\tif (ret < 0)\n+\t\t\tbreak;\n+\t\tif (ret > 0 || strcmp(log.refname, arg->refname)) {\n+\t\t\tret = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttombstone.refname = (char *)arg->refname;\n+\t\ttombstone.value_type = REFTABLE_LOG_DELETION;\n+\t\ttombstone.update_index = log.update_index;\n+\n+\t\tret = reftable_writer_add_log(writer, &tombstone);\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn ret;\n+}\n+\n+static int reftable_be_delete_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"delete_reflog\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct write_reflog_delete_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tint ret;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\treturn ret;\n+\tret = reftable_stack_add(stack, &write_reflog_delete_table, &arg);\n+\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treturn ret;\n+}\n+\n+struct reflog_expiry_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_log_record *records;\n+\tstruct object_id update_oid;\n+\tconst char *refname;\n+\tsize_t len;\n+};\n+\n+static int write_reflog_expiry_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct reflog_expiry_arg *arg = cb_data;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tuint64_t live_records = 0;\n+\tsize_t i;\n+\tint ret;\n+\n+\tfor (i = 0; i < arg->len; i++)\n+\t\tif (arg->records[i].value_type == REFTABLE_LOG_UPDATE)\n+\t\t\tlive_records++;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tif (!is_null_oid(&arg->update_oid)) {\n+\t\tstruct reftable_ref_record ref = {0};\n+\t\tstruct object_id peeled;\n+\n+\t\tref.refname = (char *)arg->refname;\n+\t\tref.update_index = ts;\n+\n+\t\tif (!peel_object(&arg->update_oid, &peeled)) {\n+\t\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\t\tmemcpy(ref.value.val2.target_value, peeled.hash, GIT_MAX_RAWSZ);\n+\t\t\tmemcpy(ref.value.val2.value, arg->update_oid.hash, GIT_MAX_RAWSZ);\n+\t\t} else {\n+\t\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\t\tmemcpy(ref.value.val1, arg->update_oid.hash, GIT_MAX_RAWSZ);\n+\t\t}\n+\n+\t\tret = reftable_writer_add_ref(writer, &ref);\n+\t\tif (ret < 0)\n+\t\t\treturn ret;\n+\t}\n+\n+\t/*\n+\t * When there are no more entries left in the reflog we empty it\n+\t * completely, but write a placeholder reflog entry that indicates that\n+\t * the reflog still exists.\n+\t */\n+\tif (!live_records) {\n+\t\tstruct reftable_log_record log = {\n+\t\t\t.refname = (char *)arg->refname,\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.update_index = ts,\n+\t\t};\n+\n+\t\tret = reftable_writer_add_log(writer, &log);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\n+\tfor (i = 0; i < arg->len; i++) {\n+\t\tret = reftable_writer_add_log(writer, &arg->records[i]);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int reftable_be_reflog_expire(struct ref_store *ref_store,\n+\t\t\t\t const char *refname,\n+\t\t\t\t unsigned int flags,\n+\t\t\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t\t void *policy_cb_data)\n+{\n+\t/*\n+\t * For log expiry, we write tombstones for every single reflog entry\n+\t * that is to be expired. This means that the entries are still\n+\t * retrievable by delving into the stack, and expiring entries\n+\t * paradoxically takes extra memory. This memory is only reclaimed when\n+\t * compacting the reftable stack.\n+\t *\n+\t * It would be better if the refs backend supported an API that sets a\n+\t * criterion for all refs, passing the criterion to pack_refs().\n+\t *\n+\t * On the plus side, because we do the expiration per ref, we can easily\n+\t * insert the reflog existence dummies.\n+\t */\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"reflog_expire\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record *logs = NULL;\n+\tstruct reftable_log_record *rewritten = NULL;\n+\tstruct reftable_ref_record ref_record = {0};\n+\tstruct reftable_iterator it = {0};\n+\tstruct reftable_addition *add = NULL;\n+\tstruct reflog_expiry_arg arg = {0};\n+\tstruct object_id oid = {0};\n+\tuint8_t *last_hash = NULL;\n+\tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tint ret;\n+\n+\tif (refs->err < 0)\n+\t\treturn refs->err;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_new_addition(&add, stack);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_read_ref(stack, refname, &ref_record);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\tif (reftable_ref_record_val1(&ref_record))\n+\t\toidread(&oid, reftable_ref_record_val1(&ref_record));\n+\tprepare_fn(refname, &oid, policy_cb_data);\n+\n+\twhile (1) {\n+\t\tstruct reftable_log_record log = {0};\n+\t\tstruct object_id old_oid, new_oid;\n+\n+\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tif (ret > 0 || strcmp(log.refname, refname)) {\n+\t\t\treftable_log_record_release(&log);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\toidread(&old_oid, log.value.update.old_hash);\n+\t\toidread(&new_oid, log.value.update.new_hash);\n+\n+\t\t/*\n+\t\t * Skip over the reflog existence marker. We will add it back\n+\t\t * in when there are no live reflog records.\n+\t\t */\n+\t\tif (is_null_oid(&old_oid) && is_null_oid(&new_oid)) {\n+\t\t\treftable_log_record_release(&log);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\tlogs[logs_nr++] = log;\n+\t}\n+\n+\t/*\n+\t * We need to rewrite all reflog entries according to the pruning\n+\t * callback function:\n+\t *\n+\t * - If a reflog entry shall be pruned we mark the record for\n+\t * deletion.\n+\t *\n+\t * - Otherwise we may have to rewrite the chain of reflog entries so\n+\t * that gaps created by just-deleted records get backfilled.\n+\t */\n+\tCALLOC_ARRAY(rewritten, logs_nr);\n+\tfor (i = logs_nr; i--;) {\n+\t\tstruct reftable_log_record *dest = &rewritten[i];\n+\t\tstruct object_id old_oid, new_oid;\n+\n+\t\t*dest = logs[i];\n+\t\toidread(&old_oid, logs[i].value.update.old_hash);\n+\t\toidread(&new_oid, logs[i].value.update.new_hash);\n+\n+\t\tif (should_prune_fn(&old_oid, &new_oid, logs[i].value.update.email,\n+\t\t\t\t (timestamp_t)logs[i].value.update.time,\n+\t\t\t\t logs[i].value.update.tz_offset,\n+\t\t\t\t logs[i].value.update.message,\n+\t\t\t\t policy_cb_data)) {\n+\t\t\tdest->value_type = REFTABLE_LOG_DELETION;\n+\t\t} else {\n+\t\t\tif ((flags & EXPIRE_REFLOGS_REWRITE) && last_hash)\n+\t\t\t\tdest->value.update.old_hash = last_hash;\n+\t\t\tlast_hash = logs[i].value.update.new_hash;\n+\t\t}\n+\t}\n+\n+\tif (flags & EXPIRE_REFLOGS_UPDATE_REF && last_hash &&\n+\t reftable_ref_record_val1(&ref_record))\n+\t\toidread(&arg.update_oid, last_hash);\n+\n+\targ.records = rewritten;\n+\targ.len = logs_nr;\n+\targ.stack = stack,\n+\targ.refname = refname,\n+\n+\tret = reftable_addition_add(add, &write_reflog_expiry_table, &arg);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\t/*\n+\t * Future improvement: we could skip writing records that were\n+\t * not changed.\n+\t */\n+\tif (!(flags & EXPIRE_REFLOGS_DRY_RUN))\n+\t\tret = reftable_addition_commit(add);\n+\n+done:\n+\tif (add)\n+\t\tcleanup_fn(policy_cb_data);\n+\tassert(ret != REFTABLE_API_ERROR);\n+\n+\treftable_ref_record_release(&ref_record);\n+\treftable_iterator_destroy(&it);\n+\treftable_addition_destroy(add);\n+\tfor (i = 0; i < logs_nr; i++)\n+\t\treftable_log_record_release(&logs[i]);\n+\tfree(logs);\n+\tfree(rewritten);\n+\treturn ret;\n+}\n+\n+struct ref_storage_be refs_be_reftable = {\n+\t.name = \"reftable\",\n+\t.init = reftable_be_init,\n+\t.init_db = reftable_be_init_db,\n+\t.transaction_prepare = reftable_be_transaction_prepare,\n+\t.transaction_finish = reftable_be_transaction_finish,\n+\t.transaction_abort = reftable_be_transaction_abort,\n+\t.initial_transaction_commit = reftable_be_initial_transaction_commit,\n+\n+\t.pack_refs = reftable_be_pack_refs,\n+\t.create_symref = reftable_be_create_symref,\n+\t.rename_ref = reftable_be_rename_ref,\n+\t.copy_ref = reftable_be_copy_ref,\n+\n+\t.iterator_begin = reftable_be_iterator_begin,\n+\t.read_raw_ref = reftable_be_read_raw_ref,\n+\t.read_symbolic_ref = reftable_be_read_symbolic_ref,\n+\n+\t.reflog_iterator_begin = reftable_be_reflog_iterator_begin,\n+\t.for_each_reflog_ent = reftable_be_for_each_reflog_ent,\n+\t.for_each_reflog_ent_reverse = reftable_be_for_each_reflog_ent_reverse,\n+\t.reflog_exists = reftable_be_reflog_exists,\n+\t.create_reflog = reftable_be_create_reflog,\n+\t.delete_reflog = reftable_be_delete_reflog,\n+\t.reflog_expire = reftable_be_reflog_expire,\n+};\ndiff --git a/repository.h b/repository.h\nindex 7a250a6605..1645cef518 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -24,8 +24,9 @@ enum fetch_negotiation_setting {\n \tFETCH_NEGOTIATION_NOOP,\n };\n \n-#define REF_STORAGE_FORMAT_UNKNOWN 0\n-#define REF_STORAGE_FORMAT_FILES 1\n+#define REF_STORAGE_FORMAT_UNKNOWN 0\n+#define REF_STORAGE_FORMAT_FILES 1\n+#define REF_STORAGE_FORMAT_REFTABLE 2\n \n struct repo_settings {\n \tint initialized;\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nnew file mode 100755\nindex 0000000000..94ee46984a\n--- /dev/null\n+++ b/t/t0610-reftable-basics.sh\n@@ -0,0 +1,887 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable basics'\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+if ! test_have_prereq REFTABLE\n+then\n+\tskip_all='skipping reftable tests; set GIT_TEST_DEFAULT_REF_FORMAT=reftable'\n+\ttest_done\n+fi\n+\n+INVALID_OID=$(test_oid 001)\n+\n+test_expect_success 'init: creates basic reftable structures' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_path_is_dir repo/.git/reftable &&\n+\ttest_path_is_file repo/.git/reftable/tables.list &&\n+\techo reftable >expect &&\n+\tgit -C repo rev-parse --show-ref-format >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init: sha256 object format via environment variable' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tGIT_DEFAULT_HASH=sha256 git init repo &&\n+\tcat >expect <<-EOF &&\n+\tsha256\n+\treftable\n+\tEOF\n+\tgit -C repo rev-parse --show-object-format --show-ref-format >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init: sha256 object format via option' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init --object-format=sha256 repo &&\n+\tcat >expect <<-EOF &&\n+\tsha256\n+\treftable\n+\tEOF\n+\tgit -C repo rev-parse --show-object-format --show-ref-format >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init: reinitializing reftable backend succeeds' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo for-each-ref >expect &&\n+\tgit init --ref-format=reftable repo &&\n+\tgit -C repo for-each-ref >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init: reinitializing files with reftable backend fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init --ref-format=files repo &&\n+\ttest_commit -C repo file &&\n+\n+\tcp repo/.git/HEAD expect &&\n+\ttest_must_fail git init --ref-format=reftable repo &&\n+\ttest_cmp expect repo/.git/HEAD\n+'\n+\n+test_expect_success 'init: reinitializing reftable with files backend fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init --ref-format=reftable repo &&\n+\ttest_commit -C repo file &&\n+\n+\tcp repo/.git/HEAD expect &&\n+\ttest_must_fail git init --ref-format=files repo &&\n+\ttest_cmp expect repo/.git/HEAD\n+'\n+\n+test_expect_perms () {\n+\tlocal perms=\"$1\"\n+\tlocal file=\"$2\"\n+\tlocal actual=$(ls -l \"$file\") &&\n+\n+\tcase \"$actual\" in\n+\t$perms*)\n+\t\t: happy\n+\t\t;;\n+\t*)\n+\t\techo \"$(basename $2) is not $perms but $actual\"\n+\t\tfalse\n+\t\t;;\n+\tesac\n+}\n+\n+for umask in 002 022\n+do\n+\ttest_expect_success POSIXPERM 'init: honors core.sharedRepository' '\n+\t\ttest_when_finished \"rm -rf repo\" &&\n+\t\t(\n+\t\t\tumask $umask &&\n+\t\t\tgit init --shared=true repo &&\n+\t\t\ttest 1 = \"$(git -C repo config core.sharedrepository)\"\n+\t\t) &&\n+\t\ttest_expect_perms \"-rw-rw-r--\" repo/.git/reftable/tables.list &&\n+\t\tfor table in repo/.git/reftable/*.ref\n+\t\tdo\n+\t\t\ttest_expect_perms \"-rw-rw-r--\" \"$table\" ||\n+\t\t\treturn 1\n+\t\tdone\n+\t'\n+done\n+\n+test_expect_success 'clone: can clone reftable repository' '\n+\ttest_when_finished \"rm -rf repo clone\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo message1 file1 &&\n+\n+\tgit clone repo cloned &&\n+\techo reftable >expect &&\n+\tgit -C cloned rev-parse --show-ref-format >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_path_is_file cloned/file1\n+'\n+\n+test_expect_success 'clone: can clone reffiles into reftable repository' '\n+\ttest_when_finished \"rm -rf reffiles reftable\" &&\n+\tgit init --ref-format=files reffiles &&\n+\ttest_commit -C reffiles A &&\n+\tgit clone --ref-format=reftable ./reffiles reftable &&\n+\n+\tgit -C reffiles rev-parse HEAD >expect &&\n+\tgit -C reftable rev-parse HEAD >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C reftable rev-parse --show-ref-format >actual &&\n+\techo reftable >expect &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C reffiles rev-parse --show-ref-format >actual &&\n+\techo files >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'clone: can clone reftable into reffiles repository' '\n+\ttest_when_finished \"rm -rf reffiles reftable\" &&\n+\tgit init --ref-format=reftable reftable &&\n+\ttest_commit -C reftable A &&\n+\tgit clone --ref-format=files ./reftable reffiles &&\n+\n+\tgit -C reftable rev-parse HEAD >expect &&\n+\tgit -C reffiles rev-parse HEAD >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C reftable rev-parse --show-ref-format >actual &&\n+\techo reftable >expect &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C reffiles rev-parse --show-ref-format >actual &&\n+\techo files >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'ref transaction: corrupted tables cause failure' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file1 &&\n+\t\tfor f in .git/reftable/*.ref\n+\t\tdo\n+\t\t\t: >\"$f\" || return 1\n+\t\tdone &&\n+\t\ttest_must_fail git update-ref refs/heads/main HEAD\n+\t)\n+'\n+\n+test_expect_success 'ref transaction: corrupted tables.list cause failure' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file1 &&\n+\t\techo garbage >.git/reftable/tables.list &&\n+\t\ttest_must_fail git update-ref refs/heads/main HEAD\n+\t)\n+'\n+\n+test_expect_success 'ref transaction: refuses to write ref causing F/D conflict' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo file &&\n+\ttest_must_fail git -C repo update-ref refs/heads/main/forbidden\n+'\n+\n+test_expect_success 'ref transaction: deleting ref with invalid name fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo file &&\n+\ttest_must_fail git -C repo update-ref -d ../../my-private-file\n+'\n+\n+test_expect_success 'ref transaction: can skip object ID verification' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_must_fail test-tool -C repo ref-store main update-ref msg refs/heads/branch $INVALID_OID $ZERO_OID 0 &&\n+\ttest-tool -C repo ref-store main update-ref msg refs/heads/branch $INVALID_OID $ZERO_OID REF_SKIP_OID_VERIFICATION\n+'\n+\n+test_expect_success 'ref transaction: updating same ref multiple times fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tcat >updates <<-EOF &&\n+\tupdate refs/heads/main $A\n+\tupdate refs/heads/main $A\n+\tEOF\n+\tcat >expect <<-EOF &&\n+\tfatal: multiple updates for ref ${SQ}refs/heads/main${SQ} not allowed\n+\tEOF\n+\ttest_must_fail git -C repo update-ref --stdin err &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'ref transaction: can delete symbolic self-reference with git-symbolic-ref(1)' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tgit -C repo symbolic-ref refs/heads/self refs/heads/self &&\n+\tgit -C repo symbolic-ref -d refs/heads/self\n+'\n+\n+test_expect_success 'ref transaction: deleting symbolic self-reference without --no-deref fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tgit -C repo symbolic-ref refs/heads/self refs/heads/self &&\n+\tcat >expect <<-EOF &&\n+\terror: multiple updates for ${SQ}refs/heads/self${SQ} (including one via symref ${SQ}refs/heads/self${SQ}) are not allowed\n+\tEOF\n+\ttest_must_fail git -C repo update-ref -d refs/heads/self 2>err &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'ref transaction: deleting symbolic self-reference with --no-deref succeeds' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tgit -C repo symbolic-ref refs/heads/self refs/heads/self &&\n+\tgit -C repo update-ref -d --no-deref refs/heads/self\n+'\n+\n+test_expect_success 'ref transaction: creating symbolic ref fails with F/D conflict' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tcat >expect <<-EOF &&\n+\terror: unable to write symref for refs/heads: file/directory conflict\n+\tEOF\n+\ttest_must_fail git -C repo symbolic-ref refs/heads refs/heads/foo 2>err &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'ref transaction: ref deletion' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file &&\n+\t\tHEAD_OID=$(git show-ref -s --verify HEAD) &&\n+\t\tcat >expect <<-EOF &&\n+\t\t$HEAD_OID refs/heads/main\n+\t\t$HEAD_OID refs/tags/file\n+\t\tEOF\n+\t\tgit show-ref >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\ttest_must_fail git update-ref -d refs/tags/file $INVALID_OID &&\n+\t\tgit show-ref >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit update-ref -d refs/tags/file $HEAD_OID &&\n+\t\techo \"$HEAD_OID refs/heads/main\" >expect &&\n+\t\tgit show-ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'ref transaction: writes cause auto-compaction' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\n+\tgit init repo &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\ttest_commit -C repo --no-tag A &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n+\n+\ttest_commit -C repo --no-tag B &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+check_fsync_events () {\n+\tlocal trace=\"$1\" &&\n+\tshift &&\n+\n+\tcat >expect &&\n+\tsed -n \\\n+\t\t-e '/^{\"event\":\"counter\",.*\"category\":\"fsync\",/ {\n+\t\t\ts/.*\"category\":\"fsync\",//;\n+\t\t\ts/}$//;\n+\t\t\tp;\n+\t\t}' \\\n+\t\t<\"$trace\" >actual &&\n+\ttest_cmp expect actual\n+}\n+\n+# A fix for this is in flight via jc/reftable-core-fsync.\n+test_expect_failure 'ref transaction: writes are synced' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo initial &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\tGIT_TEST_FSYNC=true \\\n+\t\tgit -C repo -c core.fsync=reference update-ref refs/heads/branch HEAD &&\n+\tcheck_fsync_events trace2.txt <<-EOF\n+\t\"name\":\"hardware-flush\",\"count\":2\n+\tEOF\n+'\n+\n+test_expect_success 'pack-refs: compacts tables' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\n+\ttest_commit -C repo A &&\n+\tls -1 repo/.git/reftable >table-files &&\n+\ttest_line_count = 4 table-files &&\n+\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n+\n+\tgit -C repo pack-refs &&\n+\tls -1 repo/.git/reftable >table-files &&\n+\ttest_line_count = 2 table-files &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'pack-refs: prunes stale tables' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttouch repo/.git/reftable/stale-table.ref &&\n+\tgit -C repo pack-refs &&\n+\ttest_path_is_missing repo/.git/reftable/stable-ref.ref\n+'\n+\n+test_expect_success 'pack-refs: does not prune non-table files' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttouch repo/.git/reftable/garbage &&\n+\tgit -C repo pack-refs &&\n+\ttest_path_is_file repo/.git/reftable/garbage\n+'\n+\n+for umask in 002 022\n+do\n+\t# A fix for this is in flight via ps/reftable-compacted-tables-permission-fix.\n+\ttest_expect_failure POSIXPERM 'pack-refs: honors core.sharedRepository' '\n+\t\ttest_when_finished \"rm -rf repo\" &&\n+\t\t(\n+\t\t\tumask $umask &&\n+\t\t\tgit init --shared=true repo &&\n+\t\t\ttest_commit -C repo A &&\n+\t\t\ttest_line_count = 3 repo/.git/reftable/tables.list\n+\t\t) &&\n+\t\tgit -C repo pack-refs &&\n+\t\ttest_expect_perms \"-rw-rw-r--\" repo/.git/reftable/tables.list &&\n+\t\tfor table in repo/.git/reftable/*.ref\n+\t\tdo\n+\t\t\ttest_expect_perms \"-rw-rw-r--\" \"$table\" ||\n+\t\t\treturn 1\n+\t\tdone\n+\t'\n+done\n+\n+# A fix for this is in flight.\n+test_expect_failure 'packed-refs: writes are synced' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo initial &&\n+\ttest_line_count = 2 table-files &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\tGIT_TEST_FSYNC=true \\\n+\t\tgit -C repo -c core.fsync=reference pack-refs &&\n+\tcheck_fsync_events trace2.txt <<-EOF\n+\t\"name\":\"hardware-flush\",\"count\":2\n+\tEOF\n+'\n+\n+test_expect_success 'ref iterator: bogus names are flagged' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit --no-tag file &&\n+\t\ttest-tool ref-store main update-ref msg \"refs/heads/bogus..name\" $(git rev-parse HEAD) $ZERO_OID REF_SKIP_REFNAME_VERIFICATION &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\t$ZERO_OID refs/heads/bogus..name 0xc\n+\t\t$(git rev-parse HEAD) refs/heads/main 0x0\n+\t\tEOF\n+\t\ttest-tool ref-store main for-each-ref \"\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'ref iterator: missing object IDs are not flagged' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest-tool ref-store main update-ref msg \"refs/heads/broken-hash\" $INVALID_OID $ZERO_OID REF_SKIP_OID_VERIFICATION &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\t$INVALID_OID refs/heads/broken-hash 0x0\n+\t\tEOF\n+\t\ttest-tool ref-store main for-each-ref \"\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'basic: commit and list refs' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo file &&\n+\ttest_write_lines refs/heads/main refs/tags/file >expect &&\n+\tgit -C repo for-each-ref --format=\"%(refname)\" >actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'basic: can write large commit message' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tperl -e \"\n+\t\tprint \\\"this is a long commit message\\\" x 50000\n+\t\" >commit-msg &&\n+\tgit -C repo commit --allow-empty --file=../commit-msg\n+'\n+\n+test_expect_success 'basic: show-ref fails with empty repository' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_must_fail git -C repo show-ref >actual &&\n+\ttest_must_be_empty actual\n+'\n+\n+test_expect_success 'basic: can check out unborn branch' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tgit -C repo checkout -b main\n+'\n+\n+test_expect_success 'basic: peeled tags are stored' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo file &&\n+\tgit -C repo tag -m \"annotated tag\" test_tag HEAD &&\n+\tfor ref in refs/heads/main refs/tags/file refs/tags/test_tag refs/tags/test_tag^{}\n+\tdo\n+\t\techo \"$(git -C repo rev-parse \"$ref\") $ref\" || return 1\n+\tdone >expect &&\n+\tgit -C repo show-ref -d >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'basic: for-each-ref can print symrefs' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file &&\n+\t\tgit branch &&\n+\t\tgit symbolic-ref refs/heads/sym refs/heads/main &&\n+\t\tcat >expected <<-EOF &&\n+\t\trefs/heads/main\n+\t\tEOF\n+\t\tgit for-each-ref --format=\"%(symref)\" refs/heads/sym >actual &&\n+\t\ttest_cmp expected actual\n+\t)\n+'\n+\n+test_expect_success 'basic: notes' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\twrite_script fake_editor <<-\\EOF &&\n+\t\techo \"$MSG\" >\"$1\"\n+\t\techo \"$MSG\" >&2\n+\t\tEOF\n+\n+\t\ttest_commit 1st &&\n+\t\ttest_commit 2nd &&\n+\t\tGIT_EDITOR=./fake_editor MSG=b4 git notes add &&\n+\t\tGIT_EDITOR=./fake_editor MSG=b3 git notes edit &&\n+\t\techo b4 >expect &&\n+\t\tgit notes --ref commits@{1} show >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'basic: stash' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file &&\n+\t\tgit stash list >expect &&\n+\t\ttest_line_count = 0 expect &&\n+\n+\t\techo hoi >>file.t &&\n+\t\tgit stash push -m stashed &&\n+\t\tgit stash list >expect &&\n+\t\ttest_line_count = 1 expect &&\n+\n+\t\tgit stash clear &&\n+\t\tgit stash list >expect &&\n+\t\ttest_line_count = 0 expect\n+\t)\n+'\n+\n+test_expect_success 'basic: cherry-pick' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit message1 file1 &&\n+\t\ttest_commit message2 file2 &&\n+\t\tgit branch source &&\n+\t\tgit checkout HEAD^ &&\n+\t\ttest_commit message3 file3 &&\n+\t\tgit cherry-pick source &&\n+\t\ttest_path_is_file file2\n+\t)\n+'\n+\n+test_expect_success 'basic: rebase' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit message1 file1 &&\n+\t\ttest_commit message2 file2 &&\n+\t\tgit branch source &&\n+\t\tgit checkout HEAD^ &&\n+\t\ttest_commit message3 file3 &&\n+\t\tgit rebase source &&\n+\t\ttest_path_is_file file2\n+\t)\n+'\n+\n+test_expect_success 'reflog: can delete separate reflog entries' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit file &&\n+\t\ttest_commit file2 &&\n+\t\ttest_commit file3 &&\n+\t\ttest_commit file4 &&\n+\t\tgit reflog >actual &&\n+\t\tgrep file3 actual &&\n+\n+\t\tgit reflog delete HEAD@{1} &&\n+\t\tgit reflog >actual &&\n+\t\t! grep file3 actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: can switch to previous branch' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file1 &&\n+\t\tgit checkout -b branch1 &&\n+\t\ttest_commit file2 &&\n+\t\tgit checkout -b branch2 &&\n+\t\tgit switch - &&\n+\t\tgit rev-parse --symbolic-full-name HEAD >actual &&\n+\t\techo refs/heads/branch1 >expect &&\n+\t\ttest_cmp actual expect\n+\t)\n+'\n+\n+test_expect_success 'reflog: copying branch writes reflog entry' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file1 &&\n+\t\ttest_commit file2 &&\n+\t\toid=$(git rev-parse --short HEAD) &&\n+\t\tgit branch src &&\n+\t\tcat >expect <<-EOF &&\n+\t\t${oid} dst@{0}: Branch: copied refs/heads/src to refs/heads/dst\n+\t\t${oid} dst@{1}: branch: Created from main\n+\t\tEOF\n+\t\tgit branch -c src dst &&\n+\t\tgit reflog dst >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: renaming branch writes reflog entry' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit symbolic-ref HEAD refs/heads/before &&\n+\t\ttest_commit file &&\n+\t\tgit show-ref >expected.refs &&\n+\t\tsed s/before/after/g expected &&\n+\t\tgit branch -M after &&\n+\t\tgit show-ref >actual &&\n+\t\ttest_cmp expected actual &&\n+\t\techo refs/heads/after >expected &&\n+\t\tgit symbolic-ref HEAD >actual &&\n+\t\ttest_cmp expected actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: can store empty logs' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_must_fail test-tool ref-store main reflog-exists refs/heads/branch &&\n+\t\ttest-tool ref-store main create-reflog refs/heads/branch &&\n+\t\ttest-tool ref-store main reflog-exists refs/heads/branch &&\n+\t\ttest-tool ref-store main for-each-reflog-ent-reverse refs/heads/branch >actual &&\n+\t\ttest_must_be_empty actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: expiry empties reflog' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit initial &&\n+\t\tgit checkout -b branch &&\n+\t\ttest_commit fileA &&\n+\t\ttest_commit fileB &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\tcommit: fileB\n+\t\tcommit: fileA\n+\t\tbranch: Created from HEAD\n+\t\tEOF\n+\t\tgit reflog show --format=\"%gs\" refs/heads/branch >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit reflog expire branch --expire=all &&\n+\t\tgit reflog show --format=\"%gs\" refs/heads/branch >actual &&\n+\t\ttest_must_be_empty actual &&\n+\t\ttest-tool ref-store main reflog-exists refs/heads/branch\n+\t)\n+'\n+\n+test_expect_success 'reflog: can be deleted' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\ttest-tool ref-store main reflog-exists refs/heads/main &&\n+\t\ttest-tool ref-store main delete-reflog refs/heads/main &&\n+\t\ttest_must_fail test-tool ref-store main reflog-exists refs/heads/main\n+\t)\n+'\n+\n+test_expect_success 'reflog: garbage collection deletes reflog entries' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\tfor count in $(test_seq 1 10)\n+\t\tdo\n+\t\t\ttest_commit \"number $count\" file.t $count number-$count ||\n+\t\t\treturn 1\n+\t\tdone &&\n+\t\tgit reflog refs/heads/main >actual &&\n+\t\ttest_line_count = 10 actual &&\n+\t\tgrep \"commit (initial): number 1\" actual &&\n+\t\tgrep \"commit: number 10\" actual &&\n+\n+\t\tgit gc &&\n+\t\tgit reflog refs/heads/main >actual &&\n+\t\ttest_line_count = 0 actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: updates via HEAD update HEAD reflog' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit main-one &&\n+\t\tgit checkout -b new-branch &&\n+\t\ttest_commit new-one &&\n+\t\ttest_commit new-two &&\n+\n+\t\techo new-one >expect &&\n+\t\tgit log -1 --format=%s HEAD@{1} >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'worktree: adding worktree creates separate stack' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\ttest_path_is_file repo/.git/worktrees/worktree/refs/heads &&\n+\techo \"ref: refs/heads/.invalid\" >expect &&\n+\ttest_cmp expect repo/.git/worktrees/worktree/HEAD &&\n+\ttest_path_is_dir repo/.git/worktrees/worktree/reftable &&\n+\ttest_path_is_file repo/.git/worktrees/worktree/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: pack-refs in main repo packs main refs' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tgit -C repo worktree add ../worktree &&\n+\n+\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\tgit -C repo pack-refs &&\n+\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: pack-refs in worktree packs worktree refs' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tgit -C repo worktree add ../worktree &&\n+\n+\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: creating shared ref updates main stack' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\tgit -C repo pack-refs &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tgit -C worktree update-ref refs/heads/shared HEAD &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: creating per-worktree ref updates worktree stack' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\tgit -C repo pack-refs &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tgit -C worktree update-ref refs/bisect/per-worktree HEAD &&\n+\ttest_line_count = 2 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: creating per-worktree ref from main repo' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\tgit -C repo pack-refs &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tgit -C repo update-ref worktrees/worktree/refs/bisect/per-worktree HEAD &&\n+\ttest_line_count = 2 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: creating per-worktree ref from second worktree' '\n+\ttest_when_finished \"rm -rf repo wt1 wt2\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../wt1 &&\n+\tgit -C repo worktree add ../wt2 &&\n+\tgit -C repo pack-refs &&\n+\tgit -C wt1 pack-refs &&\n+\tgit -C wt2 pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/wt1/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/worktrees/wt2/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tgit -C wt1 update-ref worktrees/wt2/refs/bisect/per-worktree HEAD &&\n+\ttest_line_count = 1 repo/.git/worktrees/wt1/reftable/tables.list &&\n+\ttest_line_count = 2 repo/.git/worktrees/wt2/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: can create shared and per-worktree ref in one transaction' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\tgit -C repo pack-refs &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tcat >stdin <<-EOF &&\n+\tcreate worktrees/worktree/refs/bisect/per-worktree HEAD\n+\tcreate refs/branches/shared HEAD\n+\tEOF\n+\tgit -C repo update-ref --stdin expect &&\n+\tgit -C worktree symbolic-ref HEAD >actual &&\n+\ttest_cmp expect actual &&\n+\tgit -C worktree checkout branch1\n+'\n+\n+test_expect_success 'worktree: adds worktree with detached HEAD' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tgit -C repo rev-parse main >expect &&\n+\n+\tgit -C repo worktree add --detach ../worktree main &&\n+\tgit -C worktree rev-parse HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'fetch: accessing FETCH_HEAD special ref works' '\n+\ttest_when_finished \"rm -rf repo sub\" &&\n+\n+\tgit init sub &&\n+\ttest_commit -C sub two &&\n+\tgit -C sub rev-parse HEAD >expect &&\n+\n+\tgit init repo &&\n+\ttest_commit -C repo one &&\n+\tgit -C repo fetch ../sub &&\n+\tgit -C repo rev-parse FETCH_HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/t0611-reftable-httpd.sh b/t/t0611-reftable-httpd.sh\nnew file mode 100755\nindex 0000000000..5e05b9c1f2\n--- /dev/null\n+++ b/t/t0611-reftable-httpd.sh\n@@ -0,0 +1,26 @@\n+#!/bin/sh\n+\n+test_description='reftable HTTPD tests'\n+\n+. ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd\n+\n+REPO=\"$HTTPD_DOCUMENT_ROOT_PATH/repo\"\n+\n+test_expect_success 'serving ls-remote' '\n+\tgit init --ref-format=reftable -b main \"$REPO\" &&\n+\tcd \"$REPO\" &&\n+\ttest_commit m1 &&\n+\t>.git/git-daemon-export-ok &&\n+\tgit ls-remote \"http://127.0.0.1:$LIB_HTTPD_PORT/smart/repo\" | cut -f 2-2 -d \"\t\" >actual &&\n+\tcat >expect <<-EOF &&\n+\tHEAD\n+\trefs/heads/main\n+\trefs/tags/m1\n+\tEOF\n+\ttest_cmp actual expect\n+'\n+\n+test_done\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex fc93aa57e6..0fb8757b7d 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1750,6 +1750,8 @@ esac\n case \"$GIT_DEFAULT_REF_FORMAT\" in\n files)\n \ttest_set_prereq REFFILES;;\n+reftable)\n+\ttest_set_prereq REFTABLE;;\n *)\n \techo 2>&1 \"error: unknown ref format $GIT_DEFAULT_REF_FORMAT\"\n \texit 1\n-- \n2.43.GIT\n\n"},{"id":"487917","messageId":"146bb95c03dd1ac43931f3d43b0a5779493fffc8.1707109509.git.ps@pks.im","threadId":"60815","inReplyTo":"cover.1707109509.git.ps@pks.im","subject":"[PATCH v3 2/2] ci: add jobs to test with the reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-05T06:02:54Z","receivedAt":"2024-02-05T06:02:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Add CI jobs for both GitHub Workflows and GitLab CI to run Git with the\nnew reftable backend.\n\nSigned-off-by: Patrick Steinhardt \n---\n .github/workflows/main.yml | 9 +++++++++\n .gitlab-ci.yml | 9 +++++++++\n ci/lib.sh | 2 +-\n ci/run-build-and-tests.sh | 3 +++\n 4 files changed, 22 insertions(+), 1 deletion(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 4d97da57ec..1b43e49dad 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -266,6 +266,9 @@ jobs:\n - jobname: linux-sha256\n cc: clang\n pool: ubuntu-latest\n+ - jobname: linux-reftable\n+ cc: clang\n+ pool: ubuntu-latest\n - jobname: linux-gcc\n cc: gcc\n cc_package: gcc-8\n@@ -277,6 +280,9 @@ jobs:\n - jobname: osx-clang\n cc: clang\n pool: macos-13\n+ - jobname: osx-reftable\n+ cc: clang\n+ pool: macos-13\n - jobname: osx-gcc\n cc: gcc\n cc_package: gcc-13\n@@ -287,6 +293,9 @@ jobs:\n - jobname: linux-leaks\n cc: gcc\n pool: ubuntu-latest\n+ - jobname: linux-reftable-leaks\n+ cc: gcc\n+ pool: ubuntu-latest\n - jobname: linux-asan-ubsan\n cc: clang\n pool: ubuntu-latest\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nindex 43bfbd8834..c0fa2fe90b 100644\n--- a/.gitlab-ci.yml\n+++ b/.gitlab-ci.yml\n@@ -26,6 +26,9 @@ test:linux:\n - jobname: linux-sha256\n image: ubuntu:latest\n CC: clang\n+ - jobname: linux-reftable\n+ image: ubuntu:latest\n+ CC: clang\n - jobname: linux-gcc\n image: ubuntu:20.04\n CC: gcc\n@@ -40,6 +43,9 @@ test:linux:\n - jobname: linux-leaks\n image: ubuntu:latest\n CC: gcc\n+ - jobname: linux-reftable-leaks\n+ image: ubuntu:latest\n+ CC: gcc\n - jobname: linux-asan-ubsan\n image: ubuntu:latest\n CC: clang\n@@ -79,6 +85,9 @@ test:osx:\n - jobname: osx-clang\n image: macos-13-xcode-14\n CC: clang\n+ - jobname: osx-reftable\n+ image: macos-13-xcode-14\n+ CC: clang\n artifacts:\n paths:\n - t/failed-test-artifacts\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex d5dd2f2697..0a73fc7bd1 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -367,7 +367,7 @@ linux-musl)\n \tMAKEFLAGS=\"$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes\"\n \tMAKEFLAGS=\"$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8\"\n \t;;\n-linux-leaks)\n+linux-leaks|linux-reftable-leaks)\n \texport SANITIZE=leak\n \texport GIT_TEST_PASSING_SANITIZE_LEAK=true\n \texport GIT_TEST_SANITIZE_LEAK_LOG=true\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 7a1466b868..c192bd613c 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -37,6 +37,9 @@ linux-clang)\n linux-sha256)\n \texport GIT_TEST_DEFAULT_HASH=sha256\n \t;;\n+linux-reftable|linux-reftable-leaks|osx-reftable)\n+\texport GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+\t;;\n pedantic)\n \t# Don't run the tests; we only care about whether Git can be\n \t# built.\n-- \n2.43.GIT\n\n"},{"id":"487927","messageId":"CAOLa=ZTtGJyzQqTTYfO2aypDU2-jDn2cehGGR3B4Vg-tFT0duA@mail.gmail.com","threadId":"60815","inReplyTo":"cover.1707109509.git.ps@pks.im","subject":"Re: [PATCH v3 0/2] refs: introduce reftable backend","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-05T13:10:57Z","receivedAt":"2024-02-05T13:10:58Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Hello,\n\nPatrick Steinhardt writes:\n> Hi,\n>\n> this is the third version of my patch series that introduces the\n> reftable backend.\n>\n> The only real change in this version is that I've pulled in\n> kn/for-all-refs at 693e807311 (for-each-ref: avoid filtering on empty\n> pattern, 2024-01-29) as a dependency. This series introduces a new\n> DO_FOR_EACH_INCLUDE_ALL_REFS flag that we need to handle in the reftable\n> backend. With this change all tests should pass again.\n>\n> Patrick\n>\n> Patrick Steinhardt (2):\n> refs: introduce reftable backend\n> ci: add jobs to test with the reftable backend\n>\n> .github/workflows/main.yml | 9 +\n> .gitlab-ci.yml | 9 +\n> Documentation/ref-storage-format.txt | 2 +\n> .../technical/repository-version.txt | 5 +-\n> Makefile | 1 +\n> ci/lib.sh | 2 +-\n> ci/run-build-and-tests.sh | 3 +\n> contrib/workdir/git-new-workdir | 2 +-\n> path.c | 2 +-\n> path.h | 1 +\n> refs.c | 1 +\n> refs/refs-internal.h | 1 +\n> refs/reftable-backend.c | 2297 +++++++++++++++++\n> repository.h | 5 +-\n> t/t0610-reftable-basics.sh | 887 +++++++\n> t/t0611-reftable-httpd.sh | 26 +\n> t/test-lib.sh | 2 +\n> 17 files changed, 3248 insertions(+), 7 deletions(-)\n> create mode 100644 refs/reftable-backend.c\n> create mode 100755 t/t0610-reftable-basics.sh\n> create mode 100755 t/t0611-reftable-httpd.sh\n>\n> Range-diff against v2:\n> 1: d6548dcfad ! 1: d83e66e980 refs: introduce reftable backend\n> @@ refs/reftable-backend.c (new)\n> +{\n> +\tswitch (log->value_type) {\n> +\tcase REFTABLE_LOG_UPDATE:\n> -+\t\t/* when we write log records, the hashes are owned by a struct\n> -+\t\t * oid */\n> ++\t\t/*\n> ++\t\t * When we write log records, the hashes are owned by the\n> ++\t\t * caller and thus shouldn't be free'd.\n> ++\t\t */\n> +\t\tlog->value.update.old_hash = NULL;\n> +\t\tlog->value.update.new_hash = NULL;\n> +\t\tbreak;\n> @@ refs/reftable-backend.c (new)\n> +\t\t\tbreak;\n> +\n> +\t\t/*\n> -+\t\t * The files backend only lists references contained in\n> -+\t\t * \"refs/\". We emulate the same behaviour here and thus skip\n> -+\t\t * all references that don't start with this prefix.\n> ++\t\t * Unless DO_FOR_EACH_INCLUDE_ALL_REFS is set, we only list\n> ++\t\t * refs starting with \"refs/\" to mimic the \"files\" backend.\n> +\t\t */\n> -+\t\tif (!starts_with(iter->ref.refname, \"refs/\"))\n> ++\t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_ALL_REFS) &&\n> ++\t\t !starts_with(iter->ref.refname, \"refs/\"))\n> +\t\t\tcontinue;\n> +\n> +\t\tif (iter->prefix &&\n> 2: 63eafc9f5b = 2: 146bb95c03 ci: add jobs to test with the reftable backend\n\nThis range-diff looks good to me. I'd already reviewed the previous\nversion, so overall looks good now.\n"},{"id":"488120","messageId":"cover.1707288261.git.ps@pks.im","threadId":"60815","inReplyTo":"cover.1706601199.git.ps@pks.im","subject":"[PATCH v4 0/2] refs: introduce reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-07T07:20:27Z","receivedAt":"2024-02-07T07:20:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the fourth version of my patch series that introduces the\nreftable backend.\n\nChanges compared to v3:\n\n - kn/for-all-refs has been reverted due to discussions around its user\n interface. I have thus evicted the patch series as a dependency and\n dropped handling of DO_FOR_EACH_INCLUDE_ALL_REFS.\n\n - ps/reftable-compacted-tables-permission-fix has been merged to\n \"master\" now, fixing permissions on \"tables.list\" when compacting\n reftables. I've marked the corresponding test as succeeding.\n\n - jc/reftable-core-fsync has been merged to \"master\" now, which adds\n fsyncing logic to reftable. I've marked the corresponding test as\n succeeding.\n\n - I noticed that the fsync tests fail on macOS because there we use a\n different fsync method by default. I fixed that by explicitly saying\n which fsync method should be used in the corresponding tests.\n\n - I also noticed that the second fsync test reused \"trace2.txt\" from\n the first fsync test because it only appends to the file. Thus, we\n saw two fsync events instead of one. I fixed that by truncating the\n file.\n\nThe patch series is based on the current \"master\" branch at 235986be82\n(The fourteenth batch, 2024-02-06).\n\nThanks!\n\nPatrick\n\nPatrick Steinhardt (2):\n refs: introduce reftable backend\n ci: add jobs to test with the reftable backend\n\n .github/workflows/main.yml | 9 +\n .gitlab-ci.yml | 9 +\n Documentation/ref-storage-format.txt | 2 +\n .../technical/repository-version.txt | 5 +-\n Makefile | 1 +\n ci/lib.sh | 2 +-\n ci/run-build-and-tests.sh | 3 +\n contrib/workdir/git-new-workdir | 2 +-\n path.c | 2 +-\n path.h | 1 +\n refs.c | 1 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 2297 +++++++++++++++++\n repository.h | 5 +-\n t/t0610-reftable-basics.sh | 887 +++++++\n t/t0611-reftable-httpd.sh | 26 +\n t/test-lib.sh | 2 +\n 17 files changed, 3248 insertions(+), 7 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100755 t/t0610-reftable-basics.sh\n create mode 100755 t/t0611-reftable-httpd.sh\n\nRange-diff against v3:\n1: d83e66e980 ! 1: 5de60d46bd refs: introduce reftable backend\n @@ refs/reftable-backend.c (new)\n +\t\t\tbreak;\n +\n +\t\t/*\n -+\t\t * Unless DO_FOR_EACH_INCLUDE_ALL_REFS is set, we only list\n -+\t\t * refs starting with \"refs/\" to mimic the \"files\" backend.\n ++\t\t * The files backend only lists references contained in\n ++\t\t * \"refs/\". We emulate the same behaviour here and thus skip\n ++\t\t * all references that don't start with this prefix.\n +\t\t */\n -+\t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_ALL_REFS) &&\n -+\t\t !starts_with(iter->ref.refname, \"refs/\"))\n ++\t\tif (!starts_with(iter->ref.refname, \"refs/\"))\n +\t\t\tcontinue;\n +\n +\t\tif (iter->prefix &&\n @@ t/t0610-reftable-basics.sh (new)\n +\ttest_cmp expect actual\n +}\n +\n -+# A fix for this is in flight via jc/reftable-core-fsync.\n -+test_expect_failure 'ref transaction: writes are synced' '\n ++test_expect_success 'ref transaction: writes are synced' '\n +\ttest_when_finished \"rm -rf repo\" &&\n +\tgit init repo &&\n +\ttest_commit -C repo initial &&\n +\n +\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n +\tGIT_TEST_FSYNC=true \\\n -+\t\tgit -C repo -c core.fsync=reference update-ref refs/heads/branch HEAD &&\n ++\t\tgit -C repo -c core.fsync=reference \\\n ++\t\t-c core.fsyncMethod=fsync update-ref refs/heads/branch HEAD &&\n +\tcheck_fsync_events trace2.txt <<-EOF\n +\t\"name\":\"hardware-flush\",\"count\":2\n +\tEOF\n @@ t/t0610-reftable-basics.sh (new)\n +\n +for umask in 002 022\n +do\n -+\t# A fix for this is in flight via ps/reftable-compacted-tables-permission-fix.\n -+\ttest_expect_failure POSIXPERM 'pack-refs: honors core.sharedRepository' '\n ++\ttest_expect_success POSIXPERM 'pack-refs: honors core.sharedRepository' '\n +\t\ttest_when_finished \"rm -rf repo\" &&\n +\t\t(\n +\t\t\tumask $umask &&\n @@ t/t0610-reftable-basics.sh (new)\n +\t'\n +done\n +\n -+# A fix for this is in flight.\n -+test_expect_failure 'packed-refs: writes are synced' '\n ++test_expect_success 'packed-refs: writes are synced' '\n +\ttest_when_finished \"rm -rf repo\" &&\n +\tgit init repo &&\n +\ttest_commit -C repo initial &&\n +\ttest_line_count = 2 table-files &&\n +\n ++\t: >trace2.txt &&\n +\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n +\tGIT_TEST_FSYNC=true \\\n -+\t\tgit -C repo -c core.fsync=reference pack-refs &&\n ++\t\tgit -C repo -c core.fsync=reference \\\n ++\t\t-c core.fsyncMethod=fsync pack-refs &&\n +\tcheck_fsync_events trace2.txt <<-EOF\n +\t\"name\":\"hardware-flush\",\"count\":2\n +\tEOF\n2: 146bb95c03 = 2: 30e5feb28c ci: add jobs to test with the reftable backend\n\nbase-commit: 235986be822c9f8689be2e9a0b7804d0b1b6d821\n-- \n2.43.GIT\n\n"},{"id":"488122","messageId":"5de60d46bdccbfbf0a923abc2f45eda07f30c110.1707288261.git.ps@pks.im","threadId":"60815","inReplyTo":"cover.1707288261.git.ps@pks.im","subject":"[PATCH v4 1/2] refs: introduce reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-07T07:20:31Z","receivedAt":"2024-02-07T07:20:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Due to scalability issues, Shawn Pearce has originally proposed a new\n\"reftable\" format more than six years ago [1]. Initially, this new\nformat was implemented in JGit with promising results. Around two years\nago, we have then added the \"reftable\" library to the Git codebase via\na4bbd13be3 (Merge branch 'hn/reftable', 2021-12-15). With this we have\nlanded all the low-level code to read and write reftables. Notably\nmissing though was the integration of this low-level code into the Git\ncode base in the form of a new ref backend that ties all of this\ntogether.\n\nThis gap is now finally closed by introducing a new \"reftable\" backend\ninto the Git codebase. This new backend promises to bring some notable\nimprovements to Git repositories:\n\n - It becomes possible to do truly atomic writes where either all refs\n are committed to disk or none are. This was not possible with the\n \"files\" backend because ref updates were split across multiple loose\n files.\n\n - The disk space required to store many refs is reduced, both compared\n to loose refs and packed-refs. This is enabled both by the reftable\n format being a binary format, which is more compact, and by prefix\n compression.\n\n - We can ignore filesystem-specific behaviour as ref names are not\n encoded via paths anymore. This means there is no need to handle\n case sensitivity on Windows systems or Unicode precomposition on\n macOS.\n\n - There is no need to rewrite the complete refdb anymore every time a\n ref is being deleted like it was the case for packed-refs. This\n means that ref deletions are now constant time instead of scaling\n linearly with the number of refs.\n\n - We can ignore file/directory conflicts so that it becomes possible\n to store both \"refs/heads/foo\" and \"refs/heads/foo/bar\".\n\n - Due to this property we can retain reflogs for deleted refs. We have\n previously been deleting reflogs together with their refs to avoid\n file/directory conflicts, which is not necessary anymore.\n\n - We can properly enumerate all refs. With the \"files\" backend it is\n not easily possible to distinguish between refs and non-refs because\n they may live side by side in the gitdir.\n\nNot all of these improvements are realized with the current \"reftable\"\nbackend implementation. At this point, the new backend is supposed to be\na drop-in replacement for the \"files\" backend that is used by basically\nall Git repositories nowadays. It strives for 1:1 compatibility, which\nmeans that a user can expect the same behaviour regardless of whether\nthey use the \"reftable\" backend or the \"files\" backend for most of the\npart.\n\nMost notably, this means we artificially limit the capabilities of the\n\"reftable\" backend to match the limits of the \"files\" backend. It is not\npossible to create refs that would end up with file/directory conflicts,\nwe do not retain reflogs, we perform stricter-than-necessary checks.\nThis is done intentionally due to two main reasons:\n\n - It makes it significantly easier to land the \"reftable\" backend as\n tests behave the same. It would be tough to argue for each and every\n single test that doesn't pass with the \"reftable\" backend.\n\n - It ensures compatibility between repositories that use the \"files\"\n backend and repositories that use the \"reftable\" backend. Like this,\n hosters can migrate their repositories to use the \"reftable\" backend\n without causing issues for clients that use the \"files\" backend in\n their clones.\n\nIt is expected that these artificial limitations may eventually go away\nin the long term.\n\nPerformance-wise things very much depend on the actual workload. The\nfollowing benchmarks compare the \"files\" and \"reftable\" backends in the\ncurrent version:\n\n - Creating N refs in separate transactions shows that the \"files\"\n backend is ~50% faster. This is not surprising given that creating a\n ref only requires us to create a single loose ref. The \"reftable\"\n backend will also perform auto compaction on updates. In real-world\n workloads we would likely also want to perform pack loose refs,\n which would likely change the picture.\n\n Benchmark 1: update-ref: create refs sequentially (refformat = files, refcount = 1)\n Time (mean ± σ): 2.1 ms ± 0.3 ms [User: 0.6 ms, System: 1.7 ms]\n Range (min … max): 1.8 ms … 4.3 ms 133 runs\n\n Benchmark 2: update-ref: create refs sequentially (refformat = reftable, refcount = 1)\n Time (mean ± σ): 2.7 ms ± 0.1 ms [User: 0.6 ms, System: 2.2 ms]\n Range (min … max): 2.4 ms … 2.9 ms 132 runs\n\n Benchmark 3: update-ref: create refs sequentially (refformat = files, refcount = 1000)\n Time (mean ± σ): 1.975 s ± 0.006 s [User: 0.437 s, System: 1.535 s]\n Range (min … max): 1.969 s … 1.980 s 3 runs\n\n Benchmark 4: update-ref: create refs sequentially (refformat = reftable, refcount = 1000)\n Time (mean ± σ): 2.611 s ± 0.013 s [User: 0.782 s, System: 1.825 s]\n Range (min … max): 2.597 s … 2.622 s 3 runs\n\n Benchmark 5: update-ref: create refs sequentially (refformat = files, refcount = 100000)\n Time (mean ± σ): 198.442 s ± 0.241 s [User: 43.051 s, System: 155.250 s]\n Range (min … max): 198.189 s … 198.670 s 3 runs\n\n Benchmark 6: update-ref: create refs sequentially (refformat = reftable, refcount = 100000)\n Time (mean ± σ): 294.509 s ± 4.269 s [User: 104.046 s, System: 190.326 s]\n Range (min … max): 290.223 s … 298.761 s 3 runs\n\n - Creating N refs in a single transaction shows that the \"files\"\n backend is significantly slower once we start to write many refs.\n The \"reftable\" backend only needs to update two files, whereas the\n \"files\" backend needs to write one file per ref.\n\n Benchmark 1: update-ref: create many refs (refformat = files, refcount = 1)\n Time (mean ± σ): 1.9 ms ± 0.1 ms [User: 0.4 ms, System: 1.4 ms]\n Range (min … max): 1.8 ms … 2.6 ms 151 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, refcount = 1)\n Time (mean ± σ): 2.5 ms ± 0.1 ms [User: 0.7 ms, System: 1.7 ms]\n Range (min … max): 2.4 ms … 3.4 ms 148 runs\n\n Benchmark 3: update-ref: create many refs (refformat = files, refcount = 1000)\n Time (mean ± σ): 152.5 ms ± 5.2 ms [User: 19.1 ms, System: 133.1 ms]\n Range (min … max): 148.5 ms … 167.8 ms 15 runs\n\n Benchmark 4: update-ref: create many refs (refformat = reftable, refcount = 1000)\n Time (mean ± σ): 58.0 ms ± 2.5 ms [User: 28.4 ms, System: 29.4 ms]\n Range (min … max): 56.3 ms … 72.9 ms 40 runs\n\n Benchmark 5: update-ref: create many refs (refformat = files, refcount = 1000000)\n Time (mean ± σ): 152.752 s ± 0.710 s [User: 20.315 s, System: 131.310 s]\n Range (min … max): 152.165 s … 153.542 s 3 runs\n\n Benchmark 6: update-ref: create many refs (refformat = reftable, refcount = 1000000)\n Time (mean ± σ): 51.912 s ± 0.127 s [User: 26.483 s, System: 25.424 s]\n Range (min … max): 51.769 s … 52.012 s 3 runs\n\n - Deleting a ref in a fully-packed repository shows that the \"files\"\n backend scales with the number of refs. The \"reftable\" backend has\n constant-time deletions.\n\n Benchmark 1: update-ref: delete ref (refformat = files, refcount = 1)\n Time (mean ± σ): 1.7 ms ± 0.1 ms [User: 0.4 ms, System: 1.2 ms]\n Range (min … max): 1.6 ms … 2.1 ms 316 runs\n\n Benchmark 2: update-ref: delete ref (refformat = reftable, refcount = 1)\n Time (mean ± σ): 1.8 ms ± 0.1 ms [User: 0.4 ms, System: 1.3 ms]\n Range (min … max): 1.7 ms … 2.1 ms 294 runs\n\n Benchmark 3: update-ref: delete ref (refformat = files, refcount = 1000)\n Time (mean ± σ): 2.0 ms ± 0.1 ms [User: 0.5 ms, System: 1.4 ms]\n Range (min … max): 1.9 ms … 2.5 ms 287 runs\n\n Benchmark 4: update-ref: delete ref (refformat = reftable, refcount = 1000)\n Time (mean ± σ): 1.9 ms ± 0.1 ms [User: 0.5 ms, System: 1.3 ms]\n Range (min … max): 1.8 ms … 2.1 ms 217 runs\n\n Benchmark 5: update-ref: delete ref (refformat = files, refcount = 1000000)\n Time (mean ± σ): 229.8 ms ± 7.9 ms [User: 182.6 ms, System: 46.8 ms]\n Range (min … max): 224.6 ms … 245.2 ms 6 runs\n\n Benchmark 6: update-ref: delete ref (refformat = reftable, refcount = 1000000)\n Time (mean ± σ): 2.0 ms ± 0.0 ms [User: 0.6 ms, System: 1.3 ms]\n Range (min … max): 2.0 ms … 2.1 ms 3 runs\n\n - Listing all refs shows no significant advantage for either of the\n backends. The \"files\" backend is a bit faster, but not by a\n significant margin. When repositories are not packed the \"reftable\"\n backend outperforms the \"files\" backend because the \"reftable\"\n backend performs auto-compaction.\n\n Benchmark 1: show-ref: print all refs (refformat = files, refcount = 1, packed = true)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 2.0 ms 1729 runs\n\n Benchmark 2: show-ref: print all refs (refformat = reftable, refcount = 1, packed = true)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 1.8 ms 1816 runs\n\n Benchmark 3: show-ref: print all refs (refformat = files, refcount = 1000, packed = true)\n Time (mean ± σ): 4.3 ms ± 0.1 ms [User: 0.9 ms, System: 3.3 ms]\n Range (min … max): 4.1 ms … 4.6 ms 645 runs\n\n Benchmark 4: show-ref: print all refs (refformat = reftable, refcount = 1000, packed = true)\n Time (mean ± σ): 4.5 ms ± 0.2 ms [User: 1.0 ms, System: 3.3 ms]\n Range (min … max): 4.2 ms … 5.9 ms 643 runs\n\n Benchmark 5: show-ref: print all refs (refformat = files, refcount = 1000000, packed = true)\n Time (mean ± σ): 2.537 s ± 0.034 s [User: 0.488 s, System: 2.048 s]\n Range (min … max): 2.511 s … 2.627 s 10 runs\n\n Benchmark 6: show-ref: print all refs (refformat = reftable, refcount = 1000000, packed = true)\n Time (mean ± σ): 2.712 s ± 0.017 s [User: 0.653 s, System: 2.059 s]\n Range (min … max): 2.692 s … 2.752 s 10 runs\n\n Benchmark 7: show-ref: print all refs (refformat = files, refcount = 1, packed = false)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 1.9 ms 1834 runs\n\n Benchmark 8: show-ref: print all refs (refformat = reftable, refcount = 1, packed = false)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.4 ms … 2.0 ms 1840 runs\n\n Benchmark 9: show-ref: print all refs (refformat = files, refcount = 1000, packed = false)\n Time (mean ± σ): 13.8 ms ± 0.2 ms [User: 2.8 ms, System: 10.8 ms]\n Range (min … max): 13.3 ms … 14.5 ms 208 runs\n\n Benchmark 10: show-ref: print all refs (refformat = reftable, refcount = 1000, packed = false)\n Time (mean ± σ): 4.5 ms ± 0.2 ms [User: 1.2 ms, System: 3.3 ms]\n Range (min … max): 4.3 ms … 6.2 ms 624 runs\n\n Benchmark 11: show-ref: print all refs (refformat = files, refcount = 1000000, packed = false)\n Time (mean ± σ): 12.127 s ± 0.129 s [User: 2.675 s, System: 9.451 s]\n Range (min … max): 11.965 s … 12.370 s 10 runs\n\n Benchmark 12: show-ref: print all refs (refformat = reftable, refcount = 1000000, packed = false)\n Time (mean ± σ): 2.799 s ± 0.022 s [User: 0.735 s, System: 2.063 s]\n Range (min … max): 2.769 s … 2.836 s 10 runs\n\n - Printing a single ref shows no real difference between the \"files\"\n and \"reftable\" backends.\n\n Benchmark 1: show-ref: print single ref (refformat = files, refcount = 1)\n Time (mean ± σ): 1.5 ms ± 0.1 ms [User: 0.4 ms, System: 1.0 ms]\n Range (min … max): 1.4 ms … 1.8 ms 1779 runs\n\n Benchmark 2: show-ref: print single ref (refformat = reftable, refcount = 1)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.4 ms … 2.5 ms 1753 runs\n\n Benchmark 3: show-ref: print single ref (refformat = files, refcount = 1000)\n Time (mean ± σ): 1.5 ms ± 0.1 ms [User: 0.3 ms, System: 1.1 ms]\n Range (min … max): 1.4 ms … 1.9 ms 1840 runs\n\n Benchmark 4: show-ref: print single ref (refformat = reftable, refcount = 1000)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 2.0 ms 1831 runs\n\n Benchmark 5: show-ref: print single ref (refformat = files, refcount = 1000000)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 2.1 ms 1848 runs\n\n Benchmark 6: show-ref: print single ref (refformat = reftable, refcount = 1000000)\n Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms]\n Range (min … max): 1.5 ms … 2.1 ms 1762 runs\n\nSo overall, performance depends on the usecases. Except for many\nsequential writes the \"reftable\" backend is roughly on par or\nsignificantly faster than the \"files\" backend though. Given that the\n\"files\" backend has received 18 years of optimizations by now this can\nbe seen as a win. Furthermore, we can expect that the \"reftable\" backend\nwill grow faster over time when attention turns more towards\noptimizations.\n\nThe complete test suite passes, except for those tests explicitly marked\nto require the REFFILES prerequisite. Some tests in t0610 are marked as\nfailing because they depend on still-in-flight bug fixes. Tests can be\nrun with the new backend by setting the GIT_TEST_DEFAULT_REF_FORMAT\nenvironment variable to \"reftable\".\n\nThere is a single known conceptual incompatibility with the dumb HTTP\ntransport. As \"info/refs\" SHOULD NOT contain the HEAD reference, and\nbecause the \"HEAD\" file is not valid anymore, it is impossible for the\nremote client to figure out the default branch without changing the\nprotocol. This shortcoming needs to be handled in a subsequent patch\nseries.\n\nAs the reftable library has already been introduced a while ago, this\ncommit message will not go into the details of how exactly the on-disk\nformat works. Please refer to our preexisting technical documentation at\nDocumentation/technical/reftable for this.\n\n[1]: https://public-inbox.org/git/CAJo=hJtyof=HRy=2sLP0ng0uZ4=S-DpZ5dR1aF+VHVETKG20OQ@mail.gmail.com/\n\nOriginal-idea-by: Shawn Pearce \nBased-on-patch-by: Han-Wen Nienhuys \nSigned-off-by: Patrick Steinhardt \n---\n Documentation/ref-storage-format.txt | 2 +\n .../technical/repository-version.txt | 5 +-\n Makefile | 1 +\n contrib/workdir/git-new-workdir | 2 +-\n path.c | 2 +-\n path.h | 1 +\n refs.c | 1 +\n refs/refs-internal.h | 1 +\n refs/reftable-backend.c | 2297 +++++++++++++++++\n repository.h | 5 +-\n t/t0610-reftable-basics.sh | 887 +++++++\n t/t0611-reftable-httpd.sh | 26 +\n t/test-lib.sh | 2 +\n 13 files changed, 3226 insertions(+), 6 deletions(-)\n create mode 100644 refs/reftable-backend.c\n create mode 100755 t/t0610-reftable-basics.sh\n create mode 100755 t/t0611-reftable-httpd.sh\n\ndiff --git a/Documentation/ref-storage-format.txt b/Documentation/ref-storage-format.txt\nindex 1a65cac468..14fff8a9c6 100644\n--- a/Documentation/ref-storage-format.txt\n+++ b/Documentation/ref-storage-format.txt\n@@ -1 +1,3 @@\n * `files` for loose files with packed-refs. This is the default.\n+* `reftable` for the reftable format. This format is experimental and its\n+ internals are subject to change.\ndiff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\nindex 27be3741e6..47281420fc 100644\n--- a/Documentation/technical/repository-version.txt\n+++ b/Documentation/technical/repository-version.txt\n@@ -103,5 +103,6 @@ GIT_COMMON_DIR/worktrees//config.worktree)\n \n ==== `refStorage`\n \n-Specifies the file format for the ref database. The only valid value\n-is `files` (loose references with a packed-refs file).\n+Specifies the file format for the ref database. The valid values are\n+`files` (loose references with a packed-refs file) and `reftable` (see\n+Documentation/technical/reftable.txt).\ndiff --git a/Makefile b/Makefile\nindex 302b863548..dd7b1cf9dd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1127,6 +1127,7 @@ LIB_OBJS += reflog.o\n LIB_OBJS += refs.o\n LIB_OBJS += refs/debug.o\n LIB_OBJS += refs/files-backend.o\n+LIB_OBJS += refs/reftable-backend.o\n LIB_OBJS += refs/iterator.o\n LIB_OBJS += refs/packed-backend.o\n LIB_OBJS += refs/ref-cache.o\ndiff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir\nindex 888c34a521..989197aace 100755\n--- a/contrib/workdir/git-new-workdir\n+++ b/contrib/workdir/git-new-workdir\n@@ -79,7 +79,7 @@ trap cleanup $siglist\n # create the links to the original repo. explicitly exclude index, HEAD and\n # logs/HEAD from the list since they are purely related to the current working\n # directory, and should not be shared.\n-for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn\n+for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn reftable\n do\n \t# create a containing directory if needed\n \tcase $x in\ndiff --git a/path.c b/path.c\nindex 0fb527918b..8bb223c92c 100644\n--- a/path.c\n+++ b/path.c\n@@ -871,7 +871,7 @@ const char *enter_repo(const char *path, int strict)\n \treturn NULL;\n }\n \n-static int calc_shared_perm(int mode)\n+int calc_shared_perm(int mode)\n {\n \tint tweak;\n \ndiff --git a/path.h b/path.h\nindex b3233c51fa..e053effef2 100644\n--- a/path.h\n+++ b/path.h\n@@ -181,6 +181,7 @@ const char *git_path_shallow(struct repository *r);\n int ends_with_path_components(const char *path, const char *components);\n int validate_headref(const char *ref);\n \n+int calc_shared_perm(int mode);\n int adjust_shared_perm(const char *path);\n \n char *interpolate_path(const char *path, int real_home);\ndiff --git a/refs.c b/refs.c\nindex c633abf284..fff343c256 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -35,6 +35,7 @@\n */\n static const struct ref_storage_be *refs_backends[] = {\n \t[REF_STORAGE_FORMAT_FILES] = &refs_be_files,\n+\t[REF_STORAGE_FORMAT_REFTABLE] = &refs_be_reftable,\n };\n \n static const struct ref_storage_be *find_ref_storage_backend(unsigned int ref_storage_format)\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 82219829b0..83e0f0bba3 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -693,6 +693,7 @@ struct ref_storage_be {\n };\n \n extern struct ref_storage_be refs_be_files;\n+extern struct ref_storage_be refs_be_reftable;\n extern struct ref_storage_be refs_be_packed;\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nnew file mode 100644\nindex 0000000000..85214baa60\n--- /dev/null\n+++ b/refs/reftable-backend.c\n@@ -0,0 +1,2297 @@\n+#include \"../git-compat-util.h\"\n+#include \"../abspath.h\"\n+#include \"../chdir-notify.h\"\n+#include \"../environment.h\"\n+#include \"../gettext.h\"\n+#include \"../hash.h\"\n+#include \"../hex.h\"\n+#include \"../iterator.h\"\n+#include \"../ident.h\"\n+#include \"../lockfile.h\"\n+#include \"../object.h\"\n+#include \"../path.h\"\n+#include \"../refs.h\"\n+#include \"../reftable/reftable-stack.h\"\n+#include \"../reftable/reftable-record.h\"\n+#include \"../reftable/reftable-error.h\"\n+#include \"../reftable/reftable-iterator.h\"\n+#include \"../reftable/reftable-merged.h\"\n+#include \"../setup.h\"\n+#include \"../strmap.h\"\n+#include \"refs-internal.h\"\n+\n+/*\n+ * Used as a flag in ref_update::flags when the ref_update was via an\n+ * update to HEAD.\n+ */\n+#define REF_UPDATE_VIA_HEAD (1 << 8)\n+\n+struct reftable_ref_store {\n+\tstruct ref_store base;\n+\n+\t/*\n+\t * The main stack refers to the common dir and thus contains common\n+\t * refs as well as refs of the main repository.\n+\t */\n+\tstruct reftable_stack *main_stack;\n+\t/*\n+\t * The worktree stack refers to the gitdir in case the refdb is opened\n+\t * via a worktree. It thus contains the per-worktree refs.\n+\t */\n+\tstruct reftable_stack *worktree_stack;\n+\t/*\n+\t * Map of worktree stacks by their respective worktree names. The map\n+\t * is populated lazily when we try to resolve `worktrees/$worktree` refs.\n+\t */\n+\tstruct strmap worktree_stacks;\n+\tstruct reftable_write_options write_options;\n+\n+\tunsigned int store_flags;\n+\tint err;\n+};\n+\n+/*\n+ * Downcast ref_store to reftable_ref_store. Die if ref_store is not a\n+ * reftable_ref_store. required_flags is compared with ref_store's store_flags\n+ * to ensure the ref_store has all required capabilities. \"caller\" is used in\n+ * any necessary error messages.\n+ */\n+static struct reftable_ref_store *reftable_be_downcast(struct ref_store *ref_store,\n+\t\t\t\t\t\t unsigned int required_flags,\n+\t\t\t\t\t\t const char *caller)\n+{\n+\tstruct reftable_ref_store *refs;\n+\n+\tif (ref_store->be != &refs_be_reftable)\n+\t\tBUG(\"ref_store is type \\\"%s\\\" not \\\"reftables\\\" in %s\",\n+\t\t ref_store->be->name, caller);\n+\n+\trefs = (struct reftable_ref_store *)ref_store;\n+\n+\tif ((refs->store_flags & required_flags) != required_flags)\n+\t\tBUG(\"operation %s requires abilities 0x%x, but only have 0x%x\",\n+\t\t caller, required_flags, refs->store_flags);\n+\n+\treturn refs;\n+}\n+\n+/*\n+ * Some refs are global to the repository (refs/heads/{*}), while others are\n+ * local to the worktree (eg. HEAD, refs/bisect/{*}). We solve this by having\n+ * multiple separate databases (ie. multiple reftable/ directories), one for\n+ * the shared refs, one for the current worktree refs, and one for each\n+ * additional worktree. For reading, we merge the view of both the shared and\n+ * the current worktree's refs, when necessary.\n+ *\n+ * This function also optionally assigns the rewritten reference name that is\n+ * local to the stack. This translation is required when using worktree refs\n+ * like `worktrees/$worktree/refs/heads/foo` as worktree stacks will store\n+ * those references in their normalized form.\n+ */\n+static struct reftable_stack *stack_for(struct reftable_ref_store *store,\n+\t\t\t\t\tconst char *refname,\n+\t\t\t\t\tconst char **rewritten_ref)\n+{\n+\tconst char *wtname;\n+\tint wtname_len;\n+\n+\tif (!refname)\n+\t\treturn store->main_stack;\n+\n+\tswitch (parse_worktree_ref(refname, &wtname, &wtname_len, rewritten_ref)) {\n+\tcase REF_WORKTREE_OTHER: {\n+\t\tstatic struct strbuf wtname_buf = STRBUF_INIT;\n+\t\tstruct strbuf wt_dir = STRBUF_INIT;\n+\t\tstruct reftable_stack *stack;\n+\n+\t\t/*\n+\t\t * We're using a static buffer here so that we don't need to\n+\t\t * allocate the worktree name whenever we look up a reference.\n+\t\t * This could be avoided if the strmap interface knew how to\n+\t\t * handle keys with a length.\n+\t\t */\n+\t\tstrbuf_reset(&wtname_buf);\n+\t\tstrbuf_add(&wtname_buf, wtname, wtname_len);\n+\n+\t\t/*\n+\t\t * There is an edge case here: when the worktree references the\n+\t\t * current worktree, then we set up the stack once via\n+\t\t * `worktree_stacks` and once via `worktree_stack`. This is\n+\t\t * wasteful, but in the reading case it shouldn't matter. And\n+\t\t * in the writing case we would notice that the stack is locked\n+\t\t * already and error out when trying to write a reference via\n+\t\t * both stacks.\n+\t\t */\n+\t\tstack = strmap_get(&store->worktree_stacks, wtname_buf.buf);\n+\t\tif (!stack) {\n+\t\t\tstrbuf_addf(&wt_dir, \"%s/worktrees/%s/reftable\",\n+\t\t\t\t store->base.repo->commondir, wtname_buf.buf);\n+\n+\t\t\tstore->err = reftable_new_stack(&stack, wt_dir.buf,\n+\t\t\t\t\t\t\tstore->write_options);\n+\t\t\tassert(store->err != REFTABLE_API_ERROR);\n+\t\t\tstrmap_put(&store->worktree_stacks, wtname_buf.buf, stack);\n+\t\t}\n+\n+\t\tstrbuf_release(&wt_dir);\n+\t\treturn stack;\n+\t}\n+\tcase REF_WORKTREE_CURRENT:\n+\t\t/*\n+\t\t * If there is no worktree stack then we're currently in the\n+\t\t * main worktree. We thus return the main stack in that case.\n+\t\t */\n+\t\tif (!store->worktree_stack)\n+\t\t\treturn store->main_stack;\n+\t\treturn store->worktree_stack;\n+\tcase REF_WORKTREE_MAIN:\n+\tcase REF_WORKTREE_SHARED:\n+\t\treturn store->main_stack;\n+\tdefault:\n+\t\tBUG(\"unhandled worktree reference type\");\n+\t}\n+}\n+\n+static int should_write_log(struct ref_store *refs, const char *refname)\n+{\n+\tif (log_all_ref_updates == LOG_REFS_UNSET)\n+\t\tlog_all_ref_updates = is_bare_repository() ? LOG_REFS_NONE : LOG_REFS_NORMAL;\n+\n+\tswitch (log_all_ref_updates) {\n+\tcase LOG_REFS_NONE:\n+\t\treturn refs_reflog_exists(refs, refname);\n+\tcase LOG_REFS_ALWAYS:\n+\t\treturn 1;\n+\tcase LOG_REFS_NORMAL:\n+\t\tif (should_autocreate_reflog(refname))\n+\t\t\treturn 1;\n+\t\treturn refs_reflog_exists(refs, refname);\n+\tdefault:\n+\t\tBUG(\"unhandled core.logAllRefUpdates value %d\", log_all_ref_updates);\n+\t}\n+}\n+\n+static void clear_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tswitch (log->value_type) {\n+\tcase REFTABLE_LOG_UPDATE:\n+\t\t/*\n+\t\t * When we write log records, the hashes are owned by the\n+\t\t * caller and thus shouldn't be free'd.\n+\t\t */\n+\t\tlog->value.update.old_hash = NULL;\n+\t\tlog->value.update.new_hash = NULL;\n+\t\tbreak;\n+\tcase REFTABLE_LOG_DELETION:\n+\t\tbreak;\n+\t}\n+\treftable_log_record_release(log);\n+}\n+\n+static void fill_reftable_log_record(struct reftable_log_record *log)\n+{\n+\tconst char *info = git_committer_info(0);\n+\tstruct ident_split split = {0};\n+\tint sign = 1;\n+\n+\tif (split_ident_line(&split, info, strlen(info)))\n+\t\tBUG(\"failed splitting committer info\");\n+\n+\treftable_log_record_release(log);\n+\tlog->value_type = REFTABLE_LOG_UPDATE;\n+\tlog->value.update.name =\n+\t\txstrndup(split.name_begin, split.name_end - split.name_begin);\n+\tlog->value.update.email =\n+\t\txstrndup(split.mail_begin, split.mail_end - split.mail_begin);\n+\tlog->value.update.time = atol(split.date_begin);\n+\tif (*split.tz_begin == '-') {\n+\t\tsign = -1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\tif (*split.tz_begin == '+') {\n+\t\tsign = 1;\n+\t\tsplit.tz_begin++;\n+\t}\n+\n+\tlog->value.update.tz_offset = sign * atoi(split.tz_begin);\n+}\n+\n+static int read_ref_without_reload(struct reftable_stack *stack,\n+\t\t\t\t const char *refname,\n+\t\t\t\t struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type)\n+{\n+\tstruct reftable_ref_record ref = {0};\n+\tint ret;\n+\n+\tret = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (reftable_ref_record_val1(&ref)) {\n+\t\toidread(oid, reftable_ref_record_val1(&ref));\n+\t} else {\n+\t\t/* We got a tombstone, which should not happen. */\n+\t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n+\t}\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn ret;\n+}\n+\n+static struct ref_store *reftable_be_init(struct repository *repo,\n+\t\t\t\t\t const char *gitdir,\n+\t\t\t\t\t unsigned int store_flags)\n+{\n+\tstruct reftable_ref_store *refs = xcalloc(1, sizeof(*refs));\n+\tstruct strbuf path = STRBUF_INIT;\n+\tint is_worktree;\n+\tmode_t mask;\n+\n+\tmask = umask(0);\n+\tumask(mask);\n+\n+\tbase_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);\n+\tstrmap_init(&refs->worktree_stacks);\n+\trefs->store_flags = store_flags;\n+\trefs->write_options.block_size = 4096;\n+\trefs->write_options.hash_id = repo->hash_algo->format_id;\n+\trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n+\n+\t/*\n+\t * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.\n+\t * This stack contains both the shared and the main worktree refs.\n+\t *\n+\t * Note that we don't try to resolve the path in case we have a\n+\t * worktree because `get_common_dir_noenv()` already does it for us.\n+\t */\n+\tis_worktree = get_common_dir_noenv(&path, gitdir);\n+\tif (!is_worktree) {\n+\t\tstrbuf_reset(&path);\n+\t\tstrbuf_realpath(&path, gitdir, 0);\n+\t}\n+\tstrbuf_addstr(&path, \"/reftable\");\n+\trefs->err = reftable_new_stack(&refs->main_stack, path.buf,\n+\t\t\t\t refs->write_options);\n+\tif (refs->err)\n+\t\tgoto done;\n+\n+\t/*\n+\t * If we're in a worktree we also need to set up the worktree reftable\n+\t * stack that is contained in the per-worktree GIT_DIR.\n+\t *\n+\t * Ideally, we would also add the stack to our worktree stack map. But\n+\t * we have no way to figure out the worktree name here and thus can't\n+\t * do it efficiently.\n+\t */\n+\tif (is_worktree) {\n+\t\tstrbuf_reset(&path);\n+\t\tstrbuf_addf(&path, \"%s/reftable\", gitdir);\n+\n+\t\trefs->err = reftable_new_stack(&refs->worktree_stack, path.buf,\n+\t\t\t\t\t refs->write_options);\n+\t\tif (refs->err)\n+\t\t\tgoto done;\n+\t}\n+\n+\tchdir_notify_reparent(\"reftables-backend $GIT_DIR\", &refs->base.gitdir);\n+\n+done:\n+\tassert(refs->err != REFTABLE_API_ERROR);\n+\tstrbuf_release(&path);\n+\treturn &refs->base;\n+}\n+\n+static int reftable_be_init_db(struct ref_store *ref_store,\n+\t\t\t int flags UNUSED,\n+\t\t\t struct strbuf *err UNUSED)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"init_db\");\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\tstrbuf_addf(&sb, \"%s/reftable\", refs->base.gitdir);\n+\tsafe_create_dir(sb.buf, 1);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/HEAD\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"ref: refs/heads/.invalid\");\n+\tadjust_shared_perm(sb.buf);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs\", refs->base.gitdir);\n+\tsafe_create_dir(sb.buf, 1);\n+\tstrbuf_reset(&sb);\n+\n+\tstrbuf_addf(&sb, \"%s/refs/heads\", refs->base.gitdir);\n+\twrite_file(sb.buf, \"this repository uses the reftable format\");\n+\tadjust_shared_perm(sb.buf);\n+\n+\tstrbuf_release(&sb);\n+\treturn 0;\n+}\n+\n+struct reftable_ref_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_ref_record ref;\n+\tstruct object_id oid;\n+\n+\tconst char *prefix;\n+\tunsigned int flags;\n+\tint err;\n+};\n+\n+static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct reftable_ref_iterator *iter =\n+\t\t(struct reftable_ref_iterator *)ref_iterator;\n+\tstruct reftable_ref_store *refs = iter->refs;\n+\n+\twhile (!iter->err) {\n+\t\tint flags = 0;\n+\n+\t\titer->err = reftable_iterator_next_ref(&iter->iter, &iter->ref);\n+\t\tif (iter->err)\n+\t\t\tbreak;\n+\n+\t\t/*\n+\t\t * The files backend only lists references contained in\n+\t\t * \"refs/\". We emulate the same behaviour here and thus skip\n+\t\t * all references that don't start with this prefix.\n+\t\t */\n+\t\tif (!starts_with(iter->ref.refname, \"refs/\"))\n+\t\t\tcontinue;\n+\n+\t\tif (iter->prefix &&\n+\t\t strncmp(iter->prefix, iter->ref.refname, strlen(iter->prefix))) {\n+\t\t\titer->err = 1;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (iter->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n+\t\t parse_worktree_ref(iter->ref.refname, NULL, NULL, NULL) !=\n+\t\t\t REF_WORKTREE_CURRENT)\n+\t\t\tcontinue;\n+\n+\t\tswitch (iter->ref.value_type) {\n+\t\tcase REFTABLE_REF_VAL1:\n+\t\t\toidread(&iter->oid, iter->ref.value.val1);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_VAL2:\n+\t\t\toidread(&iter->oid, iter->ref.value.val2.value);\n+\t\t\tbreak;\n+\t\tcase REFTABLE_REF_SYMREF:\n+\t\t\tif (!refs_resolve_ref_unsafe(&iter->refs->base, iter->ref.refname,\n+\t\t\t\t\t\t RESOLVE_REF_READING, &iter->oid, &flags))\n+\t\t\t\toidclr(&iter->oid);\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tBUG(\"unhandled reference value type %d\", iter->ref.value_type);\n+\t\t}\n+\n+\t\tif (is_null_oid(&iter->oid))\n+\t\t\tflags |= REF_ISBROKEN;\n+\n+\t\tif (check_refname_format(iter->ref.refname, REFNAME_ALLOW_ONELEVEL)) {\n+\t\t\tif (!refname_is_safe(iter->ref.refname))\n+\t\t\t\tdie(_(\"refname is dangerous: %s\"), iter->ref.refname);\n+\t\t\toidclr(&iter->oid);\n+\t\t\tflags |= REF_BAD_NAME | REF_ISBROKEN;\n+\t\t}\n+\n+\t\tif (iter->flags & DO_FOR_EACH_OMIT_DANGLING_SYMREFS &&\n+\t\t flags & REF_ISSYMREF &&\n+\t\t flags & REF_ISBROKEN)\n+\t\t\tcontinue;\n+\n+\t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n+\t\t !ref_resolves_to_object(iter->ref.refname, refs->base.repo,\n+\t\t\t\t\t &iter->oid, flags))\n+\t\t\t\tcontinue;\n+\n+\t\titer->base.refname = iter->ref.refname;\n+\t\titer->base.oid = &iter->oid;\n+\t\titer->base.flags = flags;\n+\n+\t\tbreak;\n+\t}\n+\n+\tif (iter->err > 0) {\n+\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n+\t\t\treturn ITER_ERROR;\n+\t\treturn ITER_DONE;\n+\t}\n+\n+\tif (iter->err < 0) {\n+\t\tref_iterator_abort(ref_iterator);\n+\t\treturn ITER_ERROR;\n+\t}\n+\n+\treturn ITER_OK;\n+}\n+\n+static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t struct object_id *peeled)\n+{\n+\tstruct reftable_ref_iterator *iter =\n+\t\t(struct reftable_ref_iterator *)ref_iterator;\n+\n+\tif (iter->ref.value_type == REFTABLE_REF_VAL2) {\n+\t\toidread(peeled, iter->ref.value.val2.target_value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\n+\n+static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct reftable_ref_iterator *iter =\n+\t\t(struct reftable_ref_iterator *)ref_iterator;\n+\treftable_ref_record_release(&iter->ref);\n+\treftable_iterator_destroy(&iter->iter);\n+\tfree(iter);\n+\treturn ITER_DONE;\n+}\n+\n+static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n+\t.advance = reftable_ref_iterator_advance,\n+\t.peel = reftable_ref_iterator_peel,\n+\t.abort = reftable_ref_iterator_abort\n+};\n+\n+static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_store *refs,\n+\t\t\t\t\t\t\t struct reftable_stack *stack,\n+\t\t\t\t\t\t\t const char *prefix,\n+\t\t\t\t\t\t\t int flags)\n+{\n+\tstruct reftable_merged_table *merged_table;\n+\tstruct reftable_ref_iterator *iter;\n+\tint ret;\n+\n+\titer = xcalloc(1, sizeof(*iter));\n+\tbase_ref_iterator_init(&iter->base, &reftable_ref_iterator_vtable, 1);\n+\titer->prefix = prefix;\n+\titer->base.oid = &iter->oid;\n+\titer->flags = flags;\n+\titer->refs = refs;\n+\n+\tret = refs->err;\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tmerged_table = reftable_stack_merged_table(stack);\n+\n+\tret = reftable_merged_table_seek_ref(merged_table, &iter->iter, prefix);\n+\tif (ret)\n+\t\tgoto done;\n+\n+done:\n+\titer->err = ret;\n+\treturn iter;\n+}\n+\n+static enum iterator_selection iterator_select(struct ref_iterator *iter_worktree,\n+\t\t\t\t\t struct ref_iterator *iter_common,\n+\t\t\t\t\t void *cb_data UNUSED)\n+{\n+\tif (iter_worktree && !iter_common) {\n+\t\t/*\n+\t\t * Return the worktree ref if there are no more common refs.\n+\t\t */\n+\t\treturn ITER_SELECT_0;\n+\t} else if (iter_common) {\n+\t\t/*\n+\t\t * In case we have pending worktree and common refs we need to\n+\t\t * yield them based on their lexicographical order. Worktree\n+\t\t * refs that have the same name as common refs shadow the\n+\t\t * latter.\n+\t\t */\n+\t\tif (iter_worktree) {\n+\t\t\tint cmp = strcmp(iter_worktree->refname,\n+\t\t\t\t\t iter_common->refname);\n+\t\t\tif (cmp < 0)\n+\t\t\t\treturn ITER_SELECT_0;\n+\t\t\telse if (!cmp)\n+\t\t\t\treturn ITER_SELECT_0_SKIP_1;\n+\t\t}\n+\n+\t\t /*\n+\t\t * We now know that the lexicographically-next ref is a common\n+\t\t * ref. When the common ref is a shared one we return it.\n+\t\t */\n+\t\tif (parse_worktree_ref(iter_common->refname, NULL, NULL,\n+\t\t\t\t NULL) == REF_WORKTREE_SHARED)\n+\t\t\treturn ITER_SELECT_1;\n+\n+\t\t/*\n+\t\t * Otherwise, if the common ref is a per-worktree ref we skip\n+\t\t * it because it would belong to the main worktree, not ours.\n+\t\t */\n+\t\treturn ITER_SKIP_1;\n+\t} else {\n+\t\treturn ITER_DONE;\n+\t}\n+}\n+\n+static struct ref_iterator *reftable_be_iterator_begin(struct ref_store *ref_store,\n+\t\t\t\t\t\t const char *prefix,\n+\t\t\t\t\t\t const char **exclude_patterns,\n+\t\t\t\t\t\t unsigned int flags)\n+{\n+\tstruct reftable_ref_iterator *main_iter, *worktree_iter;\n+\tstruct reftable_ref_store *refs;\n+\tunsigned int required_flags = REF_STORE_READ;\n+\n+\tif (!(flags & DO_FOR_EACH_INCLUDE_BROKEN))\n+\t\trequired_flags |= REF_STORE_ODB;\n+\trefs = reftable_be_downcast(ref_store, required_flags, \"ref_iterator_begin\");\n+\n+\tmain_iter = ref_iterator_for_stack(refs, refs->main_stack, prefix, flags);\n+\n+\t/*\n+\t * The worktree stack is only set when we're in an actual worktree\n+\t * right now. If we aren't, then we return the common reftable\n+\t * iterator, only.\n+\t */\n+\t if (!refs->worktree_stack)\n+\t\treturn &main_iter->base;\n+\n+\t/*\n+\t * Otherwise we merge both the common and the per-worktree refs into a\n+\t * single iterator.\n+\t */\n+\tworktree_iter = ref_iterator_for_stack(refs, refs->worktree_stack, prefix, flags);\n+\treturn merge_ref_iterator_begin(1, &worktree_iter->base, &main_iter->base,\n+\t\t\t\t\titerator_select, NULL);\n+}\n+\n+static int reftable_be_read_raw_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname,\n+\t\t\t\t struct object_id *oid,\n+\t\t\t\t struct strbuf *referent,\n+\t\t\t\t unsigned int *type,\n+\t\t\t\t int *failure_errno)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_raw_ref\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tint ret;\n+\n+\tif (refs->err < 0)\n+\t\treturn refs->err;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tret = read_ref_without_reload(stack, refname, oid, referent, type);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\tif (ret > 0) {\n+\t\t*failure_errno = ENOENT;\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int reftable_be_read_symbolic_ref(struct ref_store *ref_store,\n+\t\t\t\t\t const char *refname,\n+\t\t\t\t\t struct strbuf *referent)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_symbolic_ref\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_ref_record ref = {0};\n+\tint ret;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tret = reftable_stack_read_ref(stack, refname, &ref);\n+\tif (ret == 0 && ref.value_type == REFTABLE_REF_SYMREF)\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\telse\n+\t\tret = -1;\n+\n+\treftable_ref_record_release(&ref);\n+\treturn ret;\n+}\n+\n+/*\n+ * Return the refname under which update was originally requested.\n+ */\n+static const char *original_update_refname(struct ref_update *update)\n+{\n+\twhile (update->parent_update)\n+\t\tupdate = update->parent_update;\n+\treturn update->refname;\n+}\n+\n+struct reftable_transaction_update {\n+\tstruct ref_update *update;\n+\tstruct object_id current_oid;\n+};\n+\n+struct write_transaction_table_arg {\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_addition *addition;\n+\tstruct reftable_transaction_update *updates;\n+\tsize_t updates_nr;\n+\tsize_t updates_alloc;\n+\tsize_t updates_expected;\n+};\n+\n+struct reftable_transaction_data {\n+\tstruct write_transaction_table_arg *args;\n+\tsize_t args_nr, args_alloc;\n+};\n+\n+static void free_transaction_data(struct reftable_transaction_data *tx_data)\n+{\n+\tif (!tx_data)\n+\t\treturn;\n+\tfor (size_t i = 0; i < tx_data->args_nr; i++) {\n+\t\treftable_addition_destroy(tx_data->args[i].addition);\n+\t\tfree(tx_data->args[i].updates);\n+\t}\n+\tfree(tx_data->args);\n+\tfree(tx_data);\n+}\n+\n+/*\n+ * Prepare transaction update for the given reference update. This will cause\n+ * us to lock the corresponding reftable stack for concurrent modification.\n+ */\n+static int prepare_transaction_update(struct write_transaction_table_arg **out,\n+\t\t\t\t struct reftable_ref_store *refs,\n+\t\t\t\t struct reftable_transaction_data *tx_data,\n+\t\t\t\t struct ref_update *update,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_stack *stack = stack_for(refs, update->refname, NULL);\n+\tstruct write_transaction_table_arg *arg = NULL;\n+\tsize_t i;\n+\tint ret;\n+\n+\t/*\n+\t * Search for a preexisting stack update. If there is one then we add\n+\t * the update to it, otherwise we set up a new stack update.\n+\t */\n+\tfor (i = 0; !arg && i < tx_data->args_nr; i++)\n+\t\tif (tx_data->args[i].stack == stack)\n+\t\t\targ = &tx_data->args[i];\n+\n+\tif (!arg) {\n+\t\tstruct reftable_addition *addition;\n+\n+\t\tret = reftable_stack_reload(stack);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\n+\t\tret = reftable_stack_new_addition(&addition, stack);\n+\t\tif (ret) {\n+\t\t\tif (ret == REFTABLE_LOCK_ERROR)\n+\t\t\t\tstrbuf_addstr(err, \"cannot lock references\");\n+\t\t\treturn ret;\n+\t\t}\n+\n+\t\tALLOC_GROW(tx_data->args, tx_data->args_nr + 1,\n+\t\t\t tx_data->args_alloc);\n+\t\targ = &tx_data->args[tx_data->args_nr++];\n+\t\targ->refs = refs;\n+\t\targ->stack = stack;\n+\t\targ->addition = addition;\n+\t\targ->updates = NULL;\n+\t\targ->updates_nr = 0;\n+\t\targ->updates_alloc = 0;\n+\t\targ->updates_expected = 0;\n+\t}\n+\n+\targ->updates_expected++;\n+\n+\tif (out)\n+\t\t*out = arg;\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Queue a reference update for the correct stack. We potentially need to\n+ * handle multiple stack updates in a single transaction when it spans across\n+ * multiple worktrees.\n+ */\n+static int queue_transaction_update(struct reftable_ref_store *refs,\n+\t\t\t\t struct reftable_transaction_data *tx_data,\n+\t\t\t\t struct ref_update *update,\n+\t\t\t\t struct object_id *current_oid,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct write_transaction_table_arg *arg = NULL;\n+\tint ret;\n+\n+\tif (update->backend_data)\n+\t\tBUG(\"reference update queued more than once\");\n+\n+\tret = prepare_transaction_update(&arg, refs, tx_data, update, err);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\tALLOC_GROW(arg->updates, arg->updates_nr + 1,\n+\t\t arg->updates_alloc);\n+\targ->updates[arg->updates_nr].update = update;\n+\toidcpy(&arg->updates[arg->updates_nr].current_oid, current_oid);\n+\tupdate->backend_data = &arg->updates[arg->updates_nr++];\n+\n+\treturn 0;\n+}\n+\n+static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE|REF_STORE_MAIN, \"ref_transaction_prepare\");\n+\tstruct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT;\n+\tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct reftable_transaction_data *tx_data = NULL;\n+\tstruct object_id head_oid;\n+\tunsigned int head_type = 0;\n+\tsize_t i;\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\ttx_data = xcalloc(1, sizeof(*tx_data));\n+\n+\t/*\n+\t * Preprocess all updates. For one we check that there are no duplicate\n+\t * reference updates in this transaction. Second, we lock all stacks\n+\t * that will be modified during the transaction.\n+\t */\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tret = prepare_transaction_update(NULL, refs, tx_data,\n+\t\t\t\t\t\t transaction->updates[i], err);\n+\t\tif (ret)\n+\t\t\tgoto done;\n+\n+\t\tstring_list_append(&affected_refnames,\n+\t\t\t\t transaction->updates[i]->refname);\n+\t}\n+\n+\t/*\n+\t * Now that we have counted updates per stack we can preallocate their\n+\t * arrays. This avoids having to reallocate many times.\n+\t */\n+\tfor (i = 0; i < tx_data->args_nr; i++) {\n+\t\tCALLOC_ARRAY(tx_data->args[i].updates, tx_data->args[i].updates_expected);\n+\t\ttx_data->args[i].updates_alloc = tx_data->args[i].updates_expected;\n+\t}\n+\n+\t/*\n+\t * Fail if a refname appears more than once in the transaction.\n+\t * This code is taken from the files backend and is a good candidate to\n+\t * be moved into the generic layer.\n+\t */\n+\tstring_list_sort(&affected_refnames);\n+\tif (ref_update_reject_duplicates(&affected_refnames, err)) {\n+\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tret = read_ref_without_reload(stack_for(refs, \"HEAD\", NULL), \"HEAD\", &head_oid,\n+\t\t\t\t &head_referent, &head_type);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tfor (i = 0; i < transaction->nr; i++) {\n+\t\tstruct ref_update *u = transaction->updates[i];\n+\t\tstruct object_id current_oid = {0};\n+\t\tstruct reftable_stack *stack;\n+\t\tconst char *rewritten_ref;\n+\n+\t\tstack = stack_for(refs, u->refname, &rewritten_ref);\n+\n+\t\t/* Verify that the new object ID is valid. */\n+\t\tif ((u->flags & REF_HAVE_NEW) && !is_null_oid(&u->new_oid) &&\n+\t\t !(u->flags & REF_SKIP_OID_VERIFICATION) &&\n+\t\t !(u->flags & REF_LOG_ONLY)) {\n+\t\t\tstruct object *o = parse_object(refs->base.repo, &u->new_oid);\n+\t\t\tif (!o) {\n+\t\t\t\tstrbuf_addf(err,\n+\t\t\t\t\t _(\"trying to write ref '%s' with nonexistent object %s\"),\n+\t\t\t\t\t u->refname, oid_to_hex(&u->new_oid));\n+\t\t\t\tret = -1;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\n+\t\t\tif (o->type != OBJ_COMMIT && is_branch(u->refname)) {\n+\t\t\t\tstrbuf_addf(err, _(\"trying to write non-commit object %s to branch '%s'\"),\n+\t\t\t\t\t oid_to_hex(&u->new_oid), u->refname);\n+\t\t\t\tret = -1;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\n+\t\t/*\n+\t\t * When we update the reference that HEAD points to we enqueue\n+\t\t * a second log-only update for HEAD so that its reflog is\n+\t\t * updated accordingly.\n+\t\t */\n+\t\tif (head_type == REF_ISSYMREF &&\n+\t\t !(u->flags & REF_LOG_ONLY) &&\n+\t\t !(u->flags & REF_UPDATE_VIA_HEAD) &&\n+\t\t !strcmp(rewritten_ref, head_referent.buf)) {\n+\t\t\tstruct ref_update *new_update;\n+\n+\t\t\t/*\n+\t\t\t * First make sure that HEAD is not already in the\n+\t\t\t * transaction. This check is O(lg N) in the transaction\n+\t\t\t * size, but it happens at most once per transaction.\n+\t\t\t */\n+\t\t\tif (string_list_has_string(&affected_refnames, \"HEAD\")) {\n+\t\t\t\t/* An entry already existed */\n+\t\t\t\tstrbuf_addf(err,\n+\t\t\t\t\t _(\"multiple updates for 'HEAD' (including one \"\n+\t\t\t\t\t \"via its referent '%s') are not allowed\"),\n+\t\t\t\t\t u->refname);\n+\t\t\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\n+\t\t\tnew_update = ref_transaction_add_update(\n+\t\t\t\t\ttransaction, \"HEAD\",\n+\t\t\t\t\tu->flags | REF_LOG_ONLY | REF_NO_DEREF,\n+\t\t\t\t\t&u->new_oid, &u->old_oid, u->msg);\n+\t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n+\t\t}\n+\n+\t\tret = read_ref_without_reload(stack, rewritten_ref,\n+\t\t\t\t\t ¤t_oid, &referent, &u->type);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tif (ret > 0 && (!(u->flags & REF_HAVE_OLD) || is_null_oid(&u->old_oid))) {\n+\t\t\t/*\n+\t\t\t * The reference does not exist, and we either have no\n+\t\t\t * old object ID or expect the reference to not exist.\n+\t\t\t * We can thus skip below safety checks as well as the\n+\t\t\t * symref splitting. But we do want to verify that\n+\t\t\t * there is no conflicting reference here so that we\n+\t\t\t * can output a proper error message instead of failing\n+\t\t\t * at a later point.\n+\t\t\t */\n+\t\t\tret = refs_verify_refname_available(ref_store, u->refname,\n+\t\t\t\t\t\t\t &affected_refnames, NULL, err);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto done;\n+\n+\t\t\t/*\n+\t\t\t * There is no need to write the reference deletion\n+\t\t\t * when the reference in question doesn't exist.\n+\t\t\t */\n+\t\t\t if (u->flags & REF_HAVE_NEW && !is_null_oid(&u->new_oid)) {\n+\t\t\t\t ret = queue_transaction_update(refs, tx_data, u,\n+\t\t\t\t\t\t\t\t¤t_oid, err);\n+\t\t\t\t if (ret)\n+\t\t\t\t\t goto done;\n+\t\t\t }\n+\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (ret > 0) {\n+\t\t\t/* The reference does not exist, but we expected it to. */\n+\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t \"unable to resolve reference '%s'\"),\n+\t\t\t\t original_update_refname(u), u->refname);\n+\t\t\tret = -1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (u->type & REF_ISSYMREF) {\n+\t\t\t/*\n+\t\t\t * The reftable stack is locked at this point already,\n+\t\t\t * so it is safe to call `refs_resolve_ref_unsafe()`\n+\t\t\t * here without causing races.\n+\t\t\t */\n+\t\t\tconst char *resolved = refs_resolve_ref_unsafe(&refs->base, u->refname, 0,\n+\t\t\t\t\t\t\t\t ¤t_oid, NULL);\n+\n+\t\t\tif (u->flags & REF_NO_DEREF) {\n+\t\t\t\tif (u->flags & REF_HAVE_OLD && !resolved) {\n+\t\t\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t\t\t \"error reading reference\"), u->refname);\n+\t\t\t\t\tret = -1;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n+\t\t\t} else {\n+\t\t\t\tstruct ref_update *new_update;\n+\t\t\t\tint new_flags;\n+\n+\t\t\t\tnew_flags = u->flags;\n+\t\t\t\tif (!strcmp(rewritten_ref, \"HEAD\"))\n+\t\t\t\t\tnew_flags |= REF_UPDATE_VIA_HEAD;\n+\n+\t\t\t\t/*\n+\t\t\t\t * If we are updating a symref (eg. HEAD), we should also\n+\t\t\t\t * update the branch that the symref points to.\n+\t\t\t\t *\n+\t\t\t\t * This is generic functionality, and would be better\n+\t\t\t\t * done in refs.c, but the current implementation is\n+\t\t\t\t * intertwined with the locking in files-backend.c.\n+\t\t\t\t */\n+\t\t\t\tnew_update = ref_transaction_add_update(\n+\t\t\t\t\t\ttransaction, referent.buf, new_flags,\n+\t\t\t\t\t\t&u->new_oid, &u->old_oid, u->msg);\n+\t\t\t\tnew_update->parent_update = u;\n+\n+\t\t\t\t/*\n+\t\t\t\t * Change the symbolic ref update to log only. Also, it\n+\t\t\t\t * doesn't need to check its old OID value, as that will be\n+\t\t\t\t * done when new_update is processed.\n+\t\t\t\t */\n+\t\t\t\tu->flags |= REF_LOG_ONLY | REF_NO_DEREF;\n+\t\t\t\tu->flags &= ~REF_HAVE_OLD;\n+\n+\t\t\t\tif (string_list_has_string(&affected_refnames, new_update->refname)) {\n+\t\t\t\t\tstrbuf_addf(err,\n+\t\t\t\t\t\t _(\"multiple updates for '%s' (including one \"\n+\t\t\t\t\t\t \"via symref '%s') are not allowed\"),\n+\t\t\t\t\t\t referent.buf, u->refname);\n+\t\t\t\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n+\t\t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n+\t\t\t}\n+\t\t}\n+\n+\t\t/*\n+\t\t * Verify that the old object matches our expectations. Note\n+\t\t * that the error messages here do not make a lot of sense in\n+\t\t * the context of the reftable backend as we never lock\n+\t\t * individual refs. But the error messages match what the files\n+\t\t * backend returns, which keeps our tests happy.\n+\t\t */\n+\t\tif (u->flags & REF_HAVE_OLD && !oideq(¤t_oid, &u->old_oid)) {\n+\t\t\tif (is_null_oid(&u->old_oid))\n+\t\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t\t \"reference already exists\"),\n+\t\t\t\t\t original_update_refname(u));\n+\t\t\telse if (is_null_oid(¤t_oid))\n+\t\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t\t \"reference is missing but expected %s\"),\n+\t\t\t\t\t original_update_refname(u),\n+\t\t\t\t\t oid_to_hex(&u->old_oid));\n+\t\t\telse\n+\t\t\t\tstrbuf_addf(err, _(\"cannot lock ref '%s': \"\n+\t\t\t\t\t \"is at %s but expected %s\"),\n+\t\t\t\t\t original_update_refname(u),\n+\t\t\t\t\t oid_to_hex(¤t_oid),\n+\t\t\t\t\t oid_to_hex(&u->old_oid));\n+\t\t\tret = -1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\t/*\n+\t\t * If all of the following conditions are true:\n+\t\t *\n+\t\t * - We're not about to write a symref.\n+\t\t * - We're not about to write a log-only entry.\n+\t\t * - Old and new object ID are different.\n+\t\t *\n+\t\t * Then we're essentially doing a no-op update that can be\n+\t\t * skipped. This is not only for the sake of efficiency, but\n+\t\t * also skips writing unneeded reflog entries.\n+\t\t */\n+\t\tif ((u->type & REF_ISSYMREF) ||\n+\t\t (u->flags & REF_LOG_ONLY) ||\n+\t\t (u->flags & REF_HAVE_NEW && !oideq(¤t_oid, &u->new_oid))) {\n+\t\t\tret = queue_transaction_update(refs, tx_data, u,\n+\t\t\t\t\t\t ¤t_oid, err);\n+\t\t\tif (ret)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\ttransaction->backend_data = tx_data;\n+\ttransaction->state = REF_TRANSACTION_PREPARED;\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\tif (ret < 0) {\n+\t\tfree_transaction_data(tx_data);\n+\t\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\t\tif (!err->len)\n+\t\t\tstrbuf_addf(err, _(\"reftable: transaction prepare: %s\"),\n+\t\t\t\t reftable_error_str(ret));\n+\t}\n+\tstring_list_clear(&affected_refnames, 0);\n+\tstrbuf_release(&referent);\n+\tstrbuf_release(&head_referent);\n+\n+\treturn ret;\n+}\n+\n+static int reftable_be_transaction_abort(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_transaction_data *tx_data = transaction->backend_data;\n+\tfree_transaction_data(tx_data);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\treturn 0;\n+}\n+\n+static int transaction_update_cmp(const void *a, const void *b)\n+{\n+\treturn strcmp(((struct reftable_transaction_update *)a)->update->refname,\n+\t\t ((struct reftable_transaction_update *)b)->update->refname);\n+}\n+\n+static int write_transaction_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct write_transaction_table_arg *arg = cb_data;\n+\tstruct reftable_merged_table *mt =\n+\t\treftable_stack_merged_table(arg->stack);\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_log_record *logs = NULL;\n+\tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tint ret = 0;\n+\n+\tQSORT(arg->updates, arg->updates_nr, transaction_update_cmp);\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tfor (i = 0; i < arg->updates_nr; i++) {\n+\t\tstruct reftable_transaction_update *tx_update = &arg->updates[i];\n+\t\tstruct ref_update *u = tx_update->update;\n+\n+\t\t/*\n+\t\t * Write a reflog entry when updating a ref to point to\n+\t\t * something new in either of the following cases:\n+\t\t *\n+\t\t * - The reference is about to be deleted. We always want to\n+\t\t * delete the reflog in that case.\n+\t\t * - REF_FORCE_CREATE_REFLOG is set, asking us to always create\n+\t\t * the reflog entry.\n+\t\t * - `core.logAllRefUpdates` tells us to create the reflog for\n+\t\t * the given ref.\n+\t\t */\n+\t\tif (u->flags & REF_HAVE_NEW && !(u->type & REF_ISSYMREF) && is_null_oid(&u->new_oid)) {\n+\t\t\tstruct reftable_log_record log = {0};\n+\t\t\tstruct reftable_iterator it = {0};\n+\n+\t\t\t/*\n+\t\t\t * When deleting refs we also delete all reflog entries\n+\t\t\t * with them. While it is not strictly required to\n+\t\t\t * delete reflogs together with their refs, this\n+\t\t\t * matches the behaviour of the files backend.\n+\t\t\t *\n+\t\t\t * Unfortunately, we have no better way than to delete\n+\t\t\t * all reflog entries one by one.\n+\t\t\t */\n+\t\t\tret = reftable_merged_table_seek_log(mt, &it, u->refname);\n+\t\t\twhile (ret == 0) {\n+\t\t\t\tstruct reftable_log_record *tombstone;\n+\n+\t\t\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\t\t\tif (ret < 0)\n+\t\t\t\t\tbreak;\n+\t\t\t\tif (ret > 0 || strcmp(log.refname, u->refname)) {\n+\t\t\t\t\tret = 0;\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\n+\t\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\t\t\ttombstone = &logs[logs_nr++];\n+\t\t\t\ttombstone->refname = xstrdup(u->refname);\n+\t\t\t\ttombstone->value_type = REFTABLE_LOG_DELETION;\n+\t\t\t\ttombstone->update_index = log.update_index;\n+\t\t\t}\n+\n+\t\t\treftable_log_record_release(&log);\n+\t\t\treftable_iterator_destroy(&it);\n+\n+\t\t\tif (ret)\n+\t\t\t\tgoto done;\n+\t\t} else if (u->flags & REF_HAVE_NEW &&\n+\t\t\t (u->flags & REF_FORCE_CREATE_REFLOG ||\n+\t\t\t should_write_log(&arg->refs->base, u->refname))) {\n+\t\t\tstruct reftable_log_record *log;\n+\n+\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\t\tlog = &logs[logs_nr++];\n+\t\t\tmemset(log, 0, sizeof(*log));\n+\n+\t\t\tfill_reftable_log_record(log);\n+\t\t\tlog->update_index = ts;\n+\t\t\tlog->refname = xstrdup(u->refname);\n+\t\t\tlog->value.update.new_hash = u->new_oid.hash;\n+\t\t\tlog->value.update.old_hash = tx_update->current_oid.hash;\n+\t\t\tlog->value.update.message =\n+\t\t\t\txstrndup(u->msg, arg->refs->write_options.block_size / 2);\n+\t\t}\n+\n+\t\tif (u->flags & REF_LOG_ONLY)\n+\t\t\tcontinue;\n+\n+\t\tif (u->flags & REF_HAVE_NEW && is_null_oid(&u->new_oid)) {\n+\t\t\tstruct reftable_ref_record ref = {\n+\t\t\t\t.refname = (char *)u->refname,\n+\t\t\t\t.update_index = ts,\n+\t\t\t\t.value_type = REFTABLE_REF_DELETION,\n+\t\t\t};\n+\n+\t\t\tret = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto done;\n+\t\t} else if (u->flags & REF_HAVE_NEW) {\n+\t\t\tstruct reftable_ref_record ref = {0};\n+\t\t\tstruct object_id peeled;\n+\t\t\tint peel_error;\n+\n+\t\t\tref.refname = (char *)u->refname;\n+\t\t\tref.update_index = ts;\n+\n+\t\t\tpeel_error = peel_object(&u->new_oid, &peeled);\n+\t\t\tif (!peel_error) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\t\t\tmemcpy(ref.value.val2.target_value, peeled.hash, GIT_MAX_RAWSZ);\n+\t\t\t\tmemcpy(ref.value.val2.value, u->new_oid.hash, GIT_MAX_RAWSZ);\n+\t\t\t} else if (!is_null_oid(&u->new_oid)) {\n+\t\t\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\t\t\tmemcpy(ref.value.val1, u->new_oid.hash, GIT_MAX_RAWSZ);\n+\t\t\t}\n+\n+\t\t\tret = reftable_writer_add_ref(writer, &ref);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Logs are written at the end so that we do not have intermixed ref\n+\t * and log blocks.\n+\t */\n+\tif (logs) {\n+\t\tret = reftable_writer_add_logs(writer, logs, logs_nr);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\tfor (i = 0; i < logs_nr; i++)\n+\t\tclear_reftable_log_record(&logs[i]);\n+\tfree(logs);\n+\treturn ret;\n+}\n+\n+static int reftable_be_transaction_finish(struct ref_store *ref_store,\n+\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t struct strbuf *err)\n+{\n+\tstruct reftable_transaction_data *tx_data = transaction->backend_data;\n+\tint ret = 0;\n+\n+\tfor (size_t i = 0; i < tx_data->args_nr; i++) {\n+\t\tret = reftable_addition_add(tx_data->args[i].addition,\n+\t\t\t\t\t write_transaction_table, &tx_data->args[i]);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\n+\t\tret = reftable_addition_commit(tx_data->args[i].addition);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t}\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\tfree_transaction_data(tx_data);\n+\ttransaction->state = REF_TRANSACTION_CLOSED;\n+\n+\tif (ret) {\n+\t\tstrbuf_addf(err, _(\"reftable: transaction failure: %s\"),\n+\t\t\t reftable_error_str(ret));\n+\t\treturn -1;\n+\t}\n+\treturn ret;\n+}\n+\n+static int reftable_be_initial_transaction_commit(struct ref_store *ref_store UNUSED,\n+\t\t\t\t\t\t struct ref_transaction *transaction,\n+\t\t\t\t\t\t struct strbuf *err)\n+{\n+\treturn ref_transaction_commit(transaction, err);\n+}\n+\n+static int reftable_be_pack_refs(struct ref_store *ref_store,\n+\t\t\t\t struct pack_refs_opts *opts)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE | REF_STORE_ODB, \"pack_refs\");\n+\tstruct reftable_stack *stack;\n+\tint ret;\n+\n+\tif (refs->err)\n+\t\treturn refs->err;\n+\n+\tstack = refs->worktree_stack;\n+\tif (!stack)\n+\t\tstack = refs->main_stack;\n+\n+\tret = reftable_stack_compact_all(stack, NULL);\n+\tif (ret)\n+\t\tgoto out;\n+\tret = reftable_stack_clean(stack);\n+\tif (ret)\n+\t\tgoto out;\n+\n+out:\n+\treturn ret;\n+}\n+\n+struct write_create_symref_arg {\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+\tconst char *target;\n+\tconst char *logmsg;\n+};\n+\n+static int write_create_symref_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct write_create_symref_arg *create = cb_data;\n+\tuint64_t ts = reftable_stack_next_update_index(create->stack);\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *)create->refname,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = (char *)create->target,\n+\t\t.update_index = ts,\n+\t};\n+\tstruct reftable_log_record log = {0};\n+\tstruct object_id new_oid;\n+\tstruct object_id old_oid;\n+\tint ret;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tret = reftable_writer_add_ref(writer, &ref);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\t/*\n+\t * Note that it is important to try and resolve the reference before we\n+\t * write the log entry. This is because `should_write_log()` will munge\n+\t * `core.logAllRefUpdates`, which is undesirable when we create a new\n+\t * repository because it would be written into the config. As HEAD will\n+\t * not resolve for new repositories this ordering will ensure that this\n+\t * never happens.\n+\t */\n+\tif (!create->logmsg ||\n+\t !refs_resolve_ref_unsafe(&create->refs->base, create->target,\n+\t\t\t\t RESOLVE_REF_READING, &new_oid, NULL) ||\n+\t !should_write_log(&create->refs->base, create->refname))\n+\t\treturn 0;\n+\n+\tfill_reftable_log_record(&log);\n+\tlog.refname = xstrdup(create->refname);\n+\tlog.update_index = ts;\n+\tlog.value.update.message = xstrndup(create->logmsg,\n+\t\t\t\t\t create->refs->write_options.block_size / 2);\n+\tlog.value.update.new_hash = new_oid.hash;\n+\tif (refs_resolve_ref_unsafe(&create->refs->base, create->refname,\n+\t\t\t\t RESOLVE_REF_READING, &old_oid, NULL))\n+\t\tlog.value.update.old_hash = old_oid.hash;\n+\n+\tret = reftable_writer_add_log(writer, &log);\n+\tclear_reftable_log_record(&log);\n+\treturn ret;\n+}\n+\n+static int reftable_be_create_symref(struct ref_store *ref_store,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const char *target,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"create_symref\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct write_create_symref_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t\t.target = target,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_add(stack, &write_create_symref_table, &arg);\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\tif (ret)\n+\t\terror(\"unable to write symref for %s: %s\", refname,\n+\t\t reftable_error_str(ret));\n+\treturn ret;\n+}\n+\n+struct write_copy_arg {\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_stack *stack;\n+\tconst char *oldname;\n+\tconst char *newname;\n+\tconst char *logmsg;\n+\tint delete_old;\n+};\n+\n+static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct write_copy_arg *arg = cb_data;\n+\tuint64_t deletion_ts, creation_ts;\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(arg->stack);\n+\tstruct reftable_ref_record old_ref = {0}, refs[2] = {0};\n+\tstruct reftable_log_record old_log = {0}, *logs = NULL;\n+\tstruct reftable_iterator it = {0};\n+\tstruct string_list skip = STRING_LIST_INIT_NODUP;\n+\tstruct strbuf errbuf = STRBUF_INIT;\n+\tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tint ret;\n+\n+\tif (reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref)) {\n+\t\tret = error(_(\"refname %s not found\"), arg->oldname);\n+\t\tgoto done;\n+\t}\n+\tif (old_ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tret = error(_(\"refname %s is a symbolic ref, copying it is not supported\"),\n+\t\t\t arg->oldname);\n+\t\tgoto done;\n+\t}\n+\n+\t/*\n+\t * There's nothing to do in case the old and new name are the same, so\n+\t * we exit early in that case.\n+\t */\n+\tif (!strcmp(arg->oldname, arg->newname)) {\n+\t\tret = 0;\n+\t\tgoto done;\n+\t}\n+\n+\t/*\n+\t * Verify that the new refname is available.\n+\t */\n+\tstring_list_insert(&skip, arg->oldname);\n+\tret = refs_verify_refname_available(&arg->refs->base, arg->newname,\n+\t\t\t\t\t NULL, &skip, &errbuf);\n+\tif (ret < 0) {\n+\t\terror(\"%s\", errbuf.buf);\n+\t\tgoto done;\n+\t}\n+\n+\t/*\n+\t * When deleting the old reference we have to use two update indices:\n+\t * once to delete the old ref and its reflog, and once to create the\n+\t * new ref and its reflog. They need to be staged with two separate\n+\t * indices because the new reflog needs to encode both the deletion of\n+\t * the old branch and the creation of the new branch, and we cannot do\n+\t * two changes to a reflog in a single update.\n+\t */\n+\tdeletion_ts = creation_ts = reftable_stack_next_update_index(arg->stack);\n+\tif (arg->delete_old)\n+\t\tcreation_ts++;\n+\treftable_writer_set_limits(writer, deletion_ts, creation_ts);\n+\n+\t/*\n+\t * Add the new reference. If this is a rename then we also delete the\n+\t * old reference.\n+\t */\n+\trefs[0] = old_ref;\n+\trefs[0].refname = (char *)arg->newname;\n+\trefs[0].update_index = creation_ts;\n+\tif (arg->delete_old) {\n+\t\trefs[1].refname = (char *)arg->oldname;\n+\t\trefs[1].value_type = REFTABLE_REF_DELETION;\n+\t\trefs[1].update_index = deletion_ts;\n+\t}\n+\tret = reftable_writer_add_refs(writer, refs, arg->delete_old ? 2 : 1);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\t/*\n+\t * When deleting the old branch we need to create a reflog entry on the\n+\t * new branch name that indicates that the old branch has been deleted\n+\t * and then recreated. This is a tad weird, but matches what the files\n+\t * backend does.\n+\t */\n+\tif (arg->delete_old) {\n+\t\tstruct strbuf head_referent = STRBUF_INIT;\n+\t\tstruct object_id head_oid;\n+\t\tint append_head_reflog;\n+\t\tunsigned head_type = 0;\n+\n+\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\tmemset(&logs[logs_nr], 0, sizeof(logs[logs_nr]));\n+\t\tfill_reftable_log_record(&logs[logs_nr]);\n+\t\tlogs[logs_nr].refname = (char *)arg->newname;\n+\t\tlogs[logs_nr].update_index = deletion_ts;\n+\t\tlogs[logs_nr].value.update.message =\n+\t\t\txstrndup(arg->logmsg, arg->refs->write_options.block_size / 2);\n+\t\tlogs[logs_nr].value.update.old_hash = old_ref.value.val1;\n+\t\tlogs_nr++;\n+\n+\t\tret = read_ref_without_reload(arg->stack, \"HEAD\", &head_oid, &head_referent, &head_type);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tappend_head_reflog = (head_type & REF_ISSYMREF) && !strcmp(head_referent.buf, arg->oldname);\n+\t\tstrbuf_release(&head_referent);\n+\n+\t\t/*\n+\t\t * The files backend uses `refs_delete_ref()` to delete the old\n+\t\t * branch name, which will append a reflog entry for HEAD in\n+\t\t * case it points to the old branch.\n+\t\t */\n+\t\tif (append_head_reflog) {\n+\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\t\tlogs[logs_nr] = logs[logs_nr - 1];\n+\t\t\tlogs[logs_nr].refname = \"HEAD\";\n+\t\t\tlogs_nr++;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Create the reflog entry for the newly created branch.\n+\t */\n+\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\tmemset(&logs[logs_nr], 0, sizeof(logs[logs_nr]));\n+\tfill_reftable_log_record(&logs[logs_nr]);\n+\tlogs[logs_nr].refname = (char *)arg->newname;\n+\tlogs[logs_nr].update_index = creation_ts;\n+\tlogs[logs_nr].value.update.message =\n+\t\txstrndup(arg->logmsg, arg->refs->write_options.block_size / 2);\n+\tlogs[logs_nr].value.update.new_hash = old_ref.value.val1;\n+\tlogs_nr++;\n+\n+\t/*\n+\t * In addition to writing the reflog entry for the new branch, we also\n+\t * copy over all log entries from the old reflog. Last but not least,\n+\t * when renaming we also have to delete all the old reflog entries.\n+\t */\n+\tret = reftable_merged_table_seek_log(mt, &it, arg->oldname);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\twhile (1) {\n+\t\tret = reftable_iterator_next_log(&it, &old_log);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tif (ret > 0 || strcmp(old_log.refname, arg->oldname)) {\n+\t\t\tret = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tfree(old_log.refname);\n+\n+\t\t/*\n+\t\t * Copy over the old reflog entry with the new refname.\n+\t\t */\n+\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\tlogs[logs_nr] = old_log;\n+\t\tlogs[logs_nr].refname = (char *)arg->newname;\n+\t\tlogs_nr++;\n+\n+\t\t/*\n+\t\t * Delete the old reflog entry in case we are renaming.\n+\t\t */\n+\t\tif (arg->delete_old) {\n+\t\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\t\tmemset(&logs[logs_nr], 0, sizeof(logs[logs_nr]));\n+\t\t\tlogs[logs_nr].refname = (char *)arg->oldname;\n+\t\t\tlogs[logs_nr].value_type = REFTABLE_LOG_DELETION;\n+\t\t\tlogs[logs_nr].update_index = old_log.update_index;\n+\t\t\tlogs_nr++;\n+\t\t}\n+\n+\t\t/*\n+\t\t * Transfer ownership of the log record we're iterating over to\n+\t\t * the array of log records. Otherwise, the pointers would get\n+\t\t * free'd or reallocated by the iterator.\n+\t\t */\n+\t\tmemset(&old_log, 0, sizeof(old_log));\n+\t}\n+\n+\tret = reftable_writer_add_logs(writer, logs, logs_nr);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treftable_iterator_destroy(&it);\n+\tstring_list_clear(&skip, 0);\n+\tstrbuf_release(&errbuf);\n+\tfor (i = 0; i < logs_nr; i++) {\n+\t\tif (!strcmp(logs[i].refname, \"HEAD\"))\n+\t\t\tcontinue;\n+\t\tif (logs[i].value.update.old_hash == old_ref.value.val1)\n+\t\t\tlogs[i].value.update.old_hash = NULL;\n+\t\tif (logs[i].value.update.new_hash == old_ref.value.val1)\n+\t\t\tlogs[i].value.update.new_hash = NULL;\n+\t\tlogs[i].refname = NULL;\n+\t\treftable_log_record_release(&logs[i]);\n+\t}\n+\tfree(logs);\n+\treftable_ref_record_release(&old_ref);\n+\treftable_log_record_release(&old_log);\n+\treturn ret;\n+}\n+\n+static int reftable_be_rename_ref(struct ref_store *ref_store,\n+\t\t\t\t const char *oldrefname,\n+\t\t\t\t const char *newrefname,\n+\t\t\t\t const char *logmsg)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"rename_ref\");\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname, &newrefname);\n+\tstruct write_copy_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t\t.delete_old = 1,\n+\t};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\tret = reftable_stack_add(stack, &write_copy_table, &arg);\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treturn ret;\n+}\n+\n+static int reftable_be_copy_ref(struct ref_store *ref_store,\n+\t\t\t\tconst char *oldrefname,\n+\t\t\t\tconst char *newrefname,\n+\t\t\t\tconst char *logmsg)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"copy_ref\");\n+\tstruct reftable_stack *stack = stack_for(refs, newrefname, &newrefname);\n+\tstruct write_copy_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.oldname = oldrefname,\n+\t\t.newname = newrefname,\n+\t\t.logmsg = logmsg,\n+\t};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\tret = reftable_stack_add(stack, &write_copy_table, &arg);\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treturn ret;\n+}\n+\n+struct reftable_reflog_iterator {\n+\tstruct ref_iterator base;\n+\tstruct reftable_ref_store *refs;\n+\tstruct reftable_iterator iter;\n+\tstruct reftable_log_record log;\n+\tstruct object_id oid;\n+\tchar *last_name;\n+\tint err;\n+};\n+\n+static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n+{\n+\tstruct reftable_reflog_iterator *iter =\n+\t\t(struct reftable_reflog_iterator *)ref_iterator;\n+\n+\twhile (!iter->err) {\n+\t\tint flags;\n+\n+\t\titer->err = reftable_iterator_next_log(&iter->iter, &iter->log);\n+\t\tif (iter->err)\n+\t\t\tbreak;\n+\n+\t\t/*\n+\t\t * We want the refnames that we have reflogs for, so we skip if\n+\t\t * we've already produced this name. This could be faster by\n+\t\t * seeking directly to reflog@update_index==0.\n+\t\t */\n+\t\tif (iter->last_name && !strcmp(iter->log.refname, iter->last_name))\n+\t\t\tcontinue;\n+\n+\t\tif (!refs_resolve_ref_unsafe(&iter->refs->base, iter->log.refname,\n+\t\t\t\t\t 0, &iter->oid, &flags)) {\n+\t\t\terror(_(\"bad ref for %s\"), iter->log.refname);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfree(iter->last_name);\n+\t\titer->last_name = xstrdup(iter->log.refname);\n+\t\titer->base.refname = iter->log.refname;\n+\t\titer->base.oid = &iter->oid;\n+\t\titer->base.flags = flags;\n+\n+\t\tbreak;\n+\t}\n+\n+\tif (iter->err > 0) {\n+\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n+\t\t\treturn ITER_ERROR;\n+\t\treturn ITER_DONE;\n+\t}\n+\n+\tif (iter->err < 0) {\n+\t\tref_iterator_abort(ref_iterator);\n+\t\treturn ITER_ERROR;\n+\t}\n+\n+\treturn ITER_OK;\n+}\n+\n+static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t\t\t\t\t struct object_id *peeled)\n+{\n+\tBUG(\"reftable reflog iterator cannot be peeled\");\n+\treturn -1;\n+}\n+\n+static int reftable_reflog_iterator_abort(struct ref_iterator *ref_iterator)\n+{\n+\tstruct reftable_reflog_iterator *iter =\n+\t\t(struct reftable_reflog_iterator *)ref_iterator;\n+\treftable_log_record_release(&iter->log);\n+\treftable_iterator_destroy(&iter->iter);\n+\tfree(iter->last_name);\n+\tfree(iter);\n+\treturn ITER_DONE;\n+}\n+\n+static struct ref_iterator_vtable reftable_reflog_iterator_vtable = {\n+\t.advance = reftable_reflog_iterator_advance,\n+\t.peel = reftable_reflog_iterator_peel,\n+\t.abort = reftable_reflog_iterator_abort\n+};\n+\n+static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftable_ref_store *refs,\n+\t\t\t\t\t\t\t\t struct reftable_stack *stack)\n+{\n+\tstruct reftable_merged_table *merged_table;\n+\tstruct reftable_reflog_iterator *iter;\n+\tint ret;\n+\n+\titer = xcalloc(1, sizeof(*iter));\n+\tbase_ref_iterator_init(&iter->base, &reftable_reflog_iterator_vtable, 1);\n+\titer->refs = refs;\n+\titer->base.oid = &iter->oid;\n+\n+\tret = refs->err;\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(refs->main_stack);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tmerged_table = reftable_stack_merged_table(stack);\n+\n+\tret = reftable_merged_table_seek_log(merged_table, &iter->iter, \"\");\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+done:\n+\titer->err = ret;\n+\treturn iter;\n+}\n+\n+static struct ref_iterator *reftable_be_reflog_iterator_begin(struct ref_store *ref_store)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_iterator_begin\");\n+\tstruct reftable_reflog_iterator *main_iter, *worktree_iter;\n+\n+\tmain_iter = reflog_iterator_for_stack(refs, refs->main_stack);\n+\tif (!refs->worktree_stack)\n+\t\treturn &main_iter->base;\n+\n+\tworktree_iter = reflog_iterator_for_stack(refs, refs->worktree_stack);\n+\n+\treturn merge_ref_iterator_begin(1, &worktree_iter->base, &main_iter->base,\n+\t\t\t\t\titerator_select, NULL);\n+}\n+\n+static int yield_log_record(struct reftable_log_record *log,\n+\t\t\t each_reflog_ent_fn fn,\n+\t\t\t void *cb_data)\n+{\n+\tstruct object_id old_oid, new_oid;\n+\tconst char *full_committer;\n+\n+\toidread(&old_oid, log->value.update.old_hash);\n+\toidread(&new_oid, log->value.update.new_hash);\n+\n+\t/*\n+\t * When both the old object ID and the new object ID are null\n+\t * then this is the reflog existence marker. The caller must\n+\t * not be aware of it.\n+\t */\n+\tif (is_null_oid(&old_oid) && is_null_oid(&new_oid))\n+\t\treturn 0;\n+\n+\tfull_committer = fmt_ident(log->value.update.name, log->value.update.email,\n+\t\t\t\t WANT_COMMITTER_IDENT, NULL, IDENT_NO_DATE);\n+\treturn fn(&old_oid, &new_oid, full_committer,\n+\t\t log->value.update.time, log->value.update.tz_offset,\n+\t\t log->value.update.message, cb_data);\n+}\n+\n+static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n+\t\t\t\t\t\t const char *refname,\n+\t\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t\t void *cb_data)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent_reverse\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record log = {0};\n+\tstruct reftable_iterator it = {0};\n+\tint ret;\n+\n+\tif (refs->err < 0)\n+\t\treturn refs->err;\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\tret = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (!ret) {\n+\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\tif (ret < 0)\n+\t\t\tbreak;\n+\t\tif (ret > 0 || strcmp(log.refname, refname)) {\n+\t\t\tret = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tret = yield_log_record(&log, fn, cb_data);\n+\t\tif (ret)\n+\t\t\tbreak;\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn ret;\n+}\n+\n+static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store,\n+\t\t\t\t\t const char *refname,\n+\t\t\t\t\t each_reflog_ent_fn fn,\n+\t\t\t\t\t void *cb_data)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_merged_table *mt = NULL;\n+\tstruct reftable_log_record *logs = NULL;\n+\tstruct reftable_iterator it = {0};\n+\tsize_t logs_alloc = 0, logs_nr = 0, i;\n+\tint ret;\n+\n+\tif (refs->err < 0)\n+\t\treturn refs->err;\n+\n+\tmt = reftable_stack_merged_table(stack);\n+\tret = reftable_merged_table_seek_log(mt, &it, refname);\n+\twhile (!ret) {\n+\t\tstruct reftable_log_record log = {0};\n+\n+\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tif (ret > 0 || strcmp(log.refname, refname)) {\n+\t\t\treftable_log_record_release(&log);\n+\t\t\tret = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\tlogs[logs_nr++] = log;\n+\t}\n+\n+\tfor (i = logs_nr; i--;) {\n+\t\tret = yield_log_record(&logs[i], fn, cb_data);\n+\t\tif (ret)\n+\t\t\tgoto done;\n+\t}\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\tfor (i = 0; i < logs_nr; i++)\n+\t\treftable_log_record_release(&logs[i]);\n+\tfree(logs);\n+\treturn ret;\n+}\n+\n+static int reftable_be_reflog_exists(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_exists\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record log = {0};\n+\tstruct reftable_iterator it = {0};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\t/*\n+\t * Check whether we get at least one log record for the given ref name.\n+\t * If so, the reflog exists, otherwise it doesn't.\n+\t */\n+\tret = reftable_iterator_next_log(&it, &log);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\tif (ret > 0) {\n+\t\tret = 0;\n+\t\tgoto done;\n+\t}\n+\n+\tret = strcmp(log.refname, refname) == 0;\n+\n+done:\n+\treftable_iterator_destroy(&it);\n+\treftable_log_record_release(&log);\n+\tif (ret < 0)\n+\t\tret = 0;\n+\treturn ret;\n+}\n+\n+struct write_reflog_existence_arg {\n+\tstruct reftable_ref_store *refs;\n+\tconst char *refname;\n+\tstruct reftable_stack *stack;\n+};\n+\n+static int write_reflog_existence_table(struct reftable_writer *writer,\n+\t\t\t\t\tvoid *cb_data)\n+{\n+\tstruct write_reflog_existence_arg *arg = cb_data;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tstruct reftable_log_record log = {0};\n+\tint ret;\n+\n+\tret = reftable_stack_read_log(arg->stack, arg->refname, &log);\n+\tif (ret <= 0)\n+\t\tgoto done;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\t/*\n+\t * The existence entry has both old and new object ID set to the the\n+\t * null object ID. Our iterators are aware of this and will not present\n+\t * them to their callers.\n+\t */\n+\tlog.refname = xstrdup(arg->refname);\n+\tlog.update_index = ts;\n+\tlog.value_type = REFTABLE_LOG_UPDATE;\n+\tret = reftable_writer_add_log(writer, &log);\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treftable_log_record_release(&log);\n+\treturn ret;\n+}\n+\n+static int reftable_be_create_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname,\n+\t\t\t\t struct strbuf *errmsg)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"create_reflog\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct write_reflog_existence_arg arg = {\n+\t\t.refs = refs,\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tint ret;\n+\n+\tret = refs->err;\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_add(stack, &write_reflog_existence_table, &arg);\n+\n+done:\n+\treturn ret;\n+}\n+\n+struct write_reflog_delete_arg {\n+\tstruct reftable_stack *stack;\n+\tconst char *refname;\n+};\n+\n+static int write_reflog_delete_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct write_reflog_delete_arg *arg = cb_data;\n+\tstruct reftable_merged_table *mt =\n+\t\treftable_stack_merged_table(arg->stack);\n+\tstruct reftable_log_record log = {0}, tombstone = {0};\n+\tstruct reftable_iterator it = {0};\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tint ret;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\t/*\n+\t * In order to delete a table we need to delete all reflog entries one\n+\t * by one. This is inefficient, but the reftable format does not have a\n+\t * better marker right now.\n+\t */\n+\tret = reftable_merged_table_seek_log(mt, &it, arg->refname);\n+\twhile (ret == 0) {\n+\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\tif (ret < 0)\n+\t\t\tbreak;\n+\t\tif (ret > 0 || strcmp(log.refname, arg->refname)) {\n+\t\t\tret = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\ttombstone.refname = (char *)arg->refname;\n+\t\ttombstone.value_type = REFTABLE_LOG_DELETION;\n+\t\ttombstone.update_index = log.update_index;\n+\n+\t\tret = reftable_writer_add_log(writer, &tombstone);\n+\t}\n+\n+\treftable_log_record_release(&log);\n+\treftable_iterator_destroy(&it);\n+\treturn ret;\n+}\n+\n+static int reftable_be_delete_reflog(struct ref_store *ref_store,\n+\t\t\t\t const char *refname)\n+{\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"delete_reflog\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct write_reflog_delete_arg arg = {\n+\t\t.stack = stack,\n+\t\t.refname = refname,\n+\t};\n+\tint ret;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret)\n+\t\treturn ret;\n+\tret = reftable_stack_add(stack, &write_reflog_delete_table, &arg);\n+\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treturn ret;\n+}\n+\n+struct reflog_expiry_arg {\n+\tstruct reftable_stack *stack;\n+\tstruct reftable_log_record *records;\n+\tstruct object_id update_oid;\n+\tconst char *refname;\n+\tsize_t len;\n+};\n+\n+static int write_reflog_expiry_table(struct reftable_writer *writer, void *cb_data)\n+{\n+\tstruct reflog_expiry_arg *arg = cb_data;\n+\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tuint64_t live_records = 0;\n+\tsize_t i;\n+\tint ret;\n+\n+\tfor (i = 0; i < arg->len; i++)\n+\t\tif (arg->records[i].value_type == REFTABLE_LOG_UPDATE)\n+\t\t\tlive_records++;\n+\n+\treftable_writer_set_limits(writer, ts, ts);\n+\n+\tif (!is_null_oid(&arg->update_oid)) {\n+\t\tstruct reftable_ref_record ref = {0};\n+\t\tstruct object_id peeled;\n+\n+\t\tref.refname = (char *)arg->refname;\n+\t\tref.update_index = ts;\n+\n+\t\tif (!peel_object(&arg->update_oid, &peeled)) {\n+\t\t\tref.value_type = REFTABLE_REF_VAL2;\n+\t\t\tmemcpy(ref.value.val2.target_value, peeled.hash, GIT_MAX_RAWSZ);\n+\t\t\tmemcpy(ref.value.val2.value, arg->update_oid.hash, GIT_MAX_RAWSZ);\n+\t\t} else {\n+\t\t\tref.value_type = REFTABLE_REF_VAL1;\n+\t\t\tmemcpy(ref.value.val1, arg->update_oid.hash, GIT_MAX_RAWSZ);\n+\t\t}\n+\n+\t\tret = reftable_writer_add_ref(writer, &ref);\n+\t\tif (ret < 0)\n+\t\t\treturn ret;\n+\t}\n+\n+\t/*\n+\t * When there are no more entries left in the reflog we empty it\n+\t * completely, but write a placeholder reflog entry that indicates that\n+\t * the reflog still exists.\n+\t */\n+\tif (!live_records) {\n+\t\tstruct reftable_log_record log = {\n+\t\t\t.refname = (char *)arg->refname,\n+\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n+\t\t\t.update_index = ts,\n+\t\t};\n+\n+\t\tret = reftable_writer_add_log(writer, &log);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\n+\tfor (i = 0; i < arg->len; i++) {\n+\t\tret = reftable_writer_add_log(writer, &arg->records[i]);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int reftable_be_reflog_expire(struct ref_store *ref_store,\n+\t\t\t\t const char *refname,\n+\t\t\t\t unsigned int flags,\n+\t\t\t\t reflog_expiry_prepare_fn prepare_fn,\n+\t\t\t\t reflog_expiry_should_prune_fn should_prune_fn,\n+\t\t\t\t reflog_expiry_cleanup_fn cleanup_fn,\n+\t\t\t\t void *policy_cb_data)\n+{\n+\t/*\n+\t * For log expiry, we write tombstones for every single reflog entry\n+\t * that is to be expired. This means that the entries are still\n+\t * retrievable by delving into the stack, and expiring entries\n+\t * paradoxically takes extra memory. This memory is only reclaimed when\n+\t * compacting the reftable stack.\n+\t *\n+\t * It would be better if the refs backend supported an API that sets a\n+\t * criterion for all refs, passing the criterion to pack_refs().\n+\t *\n+\t * On the plus side, because we do the expiration per ref, we can easily\n+\t * insert the reflog existence dummies.\n+\t */\n+\tstruct reftable_ref_store *refs =\n+\t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"reflog_expire\");\n+\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_merged_table *mt = reftable_stack_merged_table(stack);\n+\tstruct reftable_log_record *logs = NULL;\n+\tstruct reftable_log_record *rewritten = NULL;\n+\tstruct reftable_ref_record ref_record = {0};\n+\tstruct reftable_iterator it = {0};\n+\tstruct reftable_addition *add = NULL;\n+\tstruct reflog_expiry_arg arg = {0};\n+\tstruct object_id oid = {0};\n+\tuint8_t *last_hash = NULL;\n+\tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tint ret;\n+\n+\tif (refs->err < 0)\n+\t\treturn refs->err;\n+\n+\tret = reftable_stack_reload(stack);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_merged_table_seek_log(mt, &it, refname);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_new_addition(&add, stack);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_read_ref(stack, refname, &ref_record);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\tif (reftable_ref_record_val1(&ref_record))\n+\t\toidread(&oid, reftable_ref_record_val1(&ref_record));\n+\tprepare_fn(refname, &oid, policy_cb_data);\n+\n+\twhile (1) {\n+\t\tstruct reftable_log_record log = {0};\n+\t\tstruct object_id old_oid, new_oid;\n+\n+\t\tret = reftable_iterator_next_log(&it, &log);\n+\t\tif (ret < 0)\n+\t\t\tgoto done;\n+\t\tif (ret > 0 || strcmp(log.refname, refname)) {\n+\t\t\treftable_log_record_release(&log);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\toidread(&old_oid, log.value.update.old_hash);\n+\t\toidread(&new_oid, log.value.update.new_hash);\n+\n+\t\t/*\n+\t\t * Skip over the reflog existence marker. We will add it back\n+\t\t * in when there are no live reflog records.\n+\t\t */\n+\t\tif (is_null_oid(&old_oid) && is_null_oid(&new_oid)) {\n+\t\t\treftable_log_record_release(&log);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n+\t\tlogs[logs_nr++] = log;\n+\t}\n+\n+\t/*\n+\t * We need to rewrite all reflog entries according to the pruning\n+\t * callback function:\n+\t *\n+\t * - If a reflog entry shall be pruned we mark the record for\n+\t * deletion.\n+\t *\n+\t * - Otherwise we may have to rewrite the chain of reflog entries so\n+\t * that gaps created by just-deleted records get backfilled.\n+\t */\n+\tCALLOC_ARRAY(rewritten, logs_nr);\n+\tfor (i = logs_nr; i--;) {\n+\t\tstruct reftable_log_record *dest = &rewritten[i];\n+\t\tstruct object_id old_oid, new_oid;\n+\n+\t\t*dest = logs[i];\n+\t\toidread(&old_oid, logs[i].value.update.old_hash);\n+\t\toidread(&new_oid, logs[i].value.update.new_hash);\n+\n+\t\tif (should_prune_fn(&old_oid, &new_oid, logs[i].value.update.email,\n+\t\t\t\t (timestamp_t)logs[i].value.update.time,\n+\t\t\t\t logs[i].value.update.tz_offset,\n+\t\t\t\t logs[i].value.update.message,\n+\t\t\t\t policy_cb_data)) {\n+\t\t\tdest->value_type = REFTABLE_LOG_DELETION;\n+\t\t} else {\n+\t\t\tif ((flags & EXPIRE_REFLOGS_REWRITE) && last_hash)\n+\t\t\t\tdest->value.update.old_hash = last_hash;\n+\t\t\tlast_hash = logs[i].value.update.new_hash;\n+\t\t}\n+\t}\n+\n+\tif (flags & EXPIRE_REFLOGS_UPDATE_REF && last_hash &&\n+\t reftable_ref_record_val1(&ref_record))\n+\t\toidread(&arg.update_oid, last_hash);\n+\n+\targ.records = rewritten;\n+\targ.len = logs_nr;\n+\targ.stack = stack,\n+\targ.refname = refname,\n+\n+\tret = reftable_addition_add(add, &write_reflog_expiry_table, &arg);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n+\t/*\n+\t * Future improvement: we could skip writing records that were\n+\t * not changed.\n+\t */\n+\tif (!(flags & EXPIRE_REFLOGS_DRY_RUN))\n+\t\tret = reftable_addition_commit(add);\n+\n+done:\n+\tif (add)\n+\t\tcleanup_fn(policy_cb_data);\n+\tassert(ret != REFTABLE_API_ERROR);\n+\n+\treftable_ref_record_release(&ref_record);\n+\treftable_iterator_destroy(&it);\n+\treftable_addition_destroy(add);\n+\tfor (i = 0; i < logs_nr; i++)\n+\t\treftable_log_record_release(&logs[i]);\n+\tfree(logs);\n+\tfree(rewritten);\n+\treturn ret;\n+}\n+\n+struct ref_storage_be refs_be_reftable = {\n+\t.name = \"reftable\",\n+\t.init = reftable_be_init,\n+\t.init_db = reftable_be_init_db,\n+\t.transaction_prepare = reftable_be_transaction_prepare,\n+\t.transaction_finish = reftable_be_transaction_finish,\n+\t.transaction_abort = reftable_be_transaction_abort,\n+\t.initial_transaction_commit = reftable_be_initial_transaction_commit,\n+\n+\t.pack_refs = reftable_be_pack_refs,\n+\t.create_symref = reftable_be_create_symref,\n+\t.rename_ref = reftable_be_rename_ref,\n+\t.copy_ref = reftable_be_copy_ref,\n+\n+\t.iterator_begin = reftable_be_iterator_begin,\n+\t.read_raw_ref = reftable_be_read_raw_ref,\n+\t.read_symbolic_ref = reftable_be_read_symbolic_ref,\n+\n+\t.reflog_iterator_begin = reftable_be_reflog_iterator_begin,\n+\t.for_each_reflog_ent = reftable_be_for_each_reflog_ent,\n+\t.for_each_reflog_ent_reverse = reftable_be_for_each_reflog_ent_reverse,\n+\t.reflog_exists = reftable_be_reflog_exists,\n+\t.create_reflog = reftable_be_create_reflog,\n+\t.delete_reflog = reftable_be_delete_reflog,\n+\t.reflog_expire = reftable_be_reflog_expire,\n+};\ndiff --git a/repository.h b/repository.h\nindex 7a250a6605..1645cef518 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -24,8 +24,9 @@ enum fetch_negotiation_setting {\n \tFETCH_NEGOTIATION_NOOP,\n };\n \n-#define REF_STORAGE_FORMAT_UNKNOWN 0\n-#define REF_STORAGE_FORMAT_FILES 1\n+#define REF_STORAGE_FORMAT_UNKNOWN 0\n+#define REF_STORAGE_FORMAT_FILES 1\n+#define REF_STORAGE_FORMAT_REFTABLE 2\n \n struct repo_settings {\n \tint initialized;\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nnew file mode 100755\nindex 0000000000..6a131e40b8\n--- /dev/null\n+++ b/t/t0610-reftable-basics.sh\n@@ -0,0 +1,887 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2020 Google LLC\n+#\n+\n+test_description='reftable basics'\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+if ! test_have_prereq REFTABLE\n+then\n+\tskip_all='skipping reftable tests; set GIT_TEST_DEFAULT_REF_FORMAT=reftable'\n+\ttest_done\n+fi\n+\n+INVALID_OID=$(test_oid 001)\n+\n+test_expect_success 'init: creates basic reftable structures' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_path_is_dir repo/.git/reftable &&\n+\ttest_path_is_file repo/.git/reftable/tables.list &&\n+\techo reftable >expect &&\n+\tgit -C repo rev-parse --show-ref-format >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init: sha256 object format via environment variable' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tGIT_DEFAULT_HASH=sha256 git init repo &&\n+\tcat >expect <<-EOF &&\n+\tsha256\n+\treftable\n+\tEOF\n+\tgit -C repo rev-parse --show-object-format --show-ref-format >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init: sha256 object format via option' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init --object-format=sha256 repo &&\n+\tcat >expect <<-EOF &&\n+\tsha256\n+\treftable\n+\tEOF\n+\tgit -C repo rev-parse --show-object-format --show-ref-format >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init: reinitializing reftable backend succeeds' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo for-each-ref >expect &&\n+\tgit init --ref-format=reftable repo &&\n+\tgit -C repo for-each-ref >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init: reinitializing files with reftable backend fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init --ref-format=files repo &&\n+\ttest_commit -C repo file &&\n+\n+\tcp repo/.git/HEAD expect &&\n+\ttest_must_fail git init --ref-format=reftable repo &&\n+\ttest_cmp expect repo/.git/HEAD\n+'\n+\n+test_expect_success 'init: reinitializing reftable with files backend fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init --ref-format=reftable repo &&\n+\ttest_commit -C repo file &&\n+\n+\tcp repo/.git/HEAD expect &&\n+\ttest_must_fail git init --ref-format=files repo &&\n+\ttest_cmp expect repo/.git/HEAD\n+'\n+\n+test_expect_perms () {\n+\tlocal perms=\"$1\"\n+\tlocal file=\"$2\"\n+\tlocal actual=$(ls -l \"$file\") &&\n+\n+\tcase \"$actual\" in\n+\t$perms*)\n+\t\t: happy\n+\t\t;;\n+\t*)\n+\t\techo \"$(basename $2) is not $perms but $actual\"\n+\t\tfalse\n+\t\t;;\n+\tesac\n+}\n+\n+for umask in 002 022\n+do\n+\ttest_expect_success POSIXPERM 'init: honors core.sharedRepository' '\n+\t\ttest_when_finished \"rm -rf repo\" &&\n+\t\t(\n+\t\t\tumask $umask &&\n+\t\t\tgit init --shared=true repo &&\n+\t\t\ttest 1 = \"$(git -C repo config core.sharedrepository)\"\n+\t\t) &&\n+\t\ttest_expect_perms \"-rw-rw-r--\" repo/.git/reftable/tables.list &&\n+\t\tfor table in repo/.git/reftable/*.ref\n+\t\tdo\n+\t\t\ttest_expect_perms \"-rw-rw-r--\" \"$table\" ||\n+\t\t\treturn 1\n+\t\tdone\n+\t'\n+done\n+\n+test_expect_success 'clone: can clone reftable repository' '\n+\ttest_when_finished \"rm -rf repo clone\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo message1 file1 &&\n+\n+\tgit clone repo cloned &&\n+\techo reftable >expect &&\n+\tgit -C cloned rev-parse --show-ref-format >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_path_is_file cloned/file1\n+'\n+\n+test_expect_success 'clone: can clone reffiles into reftable repository' '\n+\ttest_when_finished \"rm -rf reffiles reftable\" &&\n+\tgit init --ref-format=files reffiles &&\n+\ttest_commit -C reffiles A &&\n+\tgit clone --ref-format=reftable ./reffiles reftable &&\n+\n+\tgit -C reffiles rev-parse HEAD >expect &&\n+\tgit -C reftable rev-parse HEAD >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C reftable rev-parse --show-ref-format >actual &&\n+\techo reftable >expect &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C reffiles rev-parse --show-ref-format >actual &&\n+\techo files >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'clone: can clone reftable into reffiles repository' '\n+\ttest_when_finished \"rm -rf reffiles reftable\" &&\n+\tgit init --ref-format=reftable reftable &&\n+\ttest_commit -C reftable A &&\n+\tgit clone --ref-format=files ./reftable reffiles &&\n+\n+\tgit -C reftable rev-parse HEAD >expect &&\n+\tgit -C reffiles rev-parse HEAD >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C reftable rev-parse --show-ref-format >actual &&\n+\techo reftable >expect &&\n+\ttest_cmp expect actual &&\n+\n+\tgit -C reffiles rev-parse --show-ref-format >actual &&\n+\techo files >expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'ref transaction: corrupted tables cause failure' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file1 &&\n+\t\tfor f in .git/reftable/*.ref\n+\t\tdo\n+\t\t\t: >\"$f\" || return 1\n+\t\tdone &&\n+\t\ttest_must_fail git update-ref refs/heads/main HEAD\n+\t)\n+'\n+\n+test_expect_success 'ref transaction: corrupted tables.list cause failure' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file1 &&\n+\t\techo garbage >.git/reftable/tables.list &&\n+\t\ttest_must_fail git update-ref refs/heads/main HEAD\n+\t)\n+'\n+\n+test_expect_success 'ref transaction: refuses to write ref causing F/D conflict' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo file &&\n+\ttest_must_fail git -C repo update-ref refs/heads/main/forbidden\n+'\n+\n+test_expect_success 'ref transaction: deleting ref with invalid name fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo file &&\n+\ttest_must_fail git -C repo update-ref -d ../../my-private-file\n+'\n+\n+test_expect_success 'ref transaction: can skip object ID verification' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_must_fail test-tool -C repo ref-store main update-ref msg refs/heads/branch $INVALID_OID $ZERO_OID 0 &&\n+\ttest-tool -C repo ref-store main update-ref msg refs/heads/branch $INVALID_OID $ZERO_OID REF_SKIP_OID_VERIFICATION\n+'\n+\n+test_expect_success 'ref transaction: updating same ref multiple times fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tcat >updates <<-EOF &&\n+\tupdate refs/heads/main $A\n+\tupdate refs/heads/main $A\n+\tEOF\n+\tcat >expect <<-EOF &&\n+\tfatal: multiple updates for ref ${SQ}refs/heads/main${SQ} not allowed\n+\tEOF\n+\ttest_must_fail git -C repo update-ref --stdin err &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'ref transaction: can delete symbolic self-reference with git-symbolic-ref(1)' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tgit -C repo symbolic-ref refs/heads/self refs/heads/self &&\n+\tgit -C repo symbolic-ref -d refs/heads/self\n+'\n+\n+test_expect_success 'ref transaction: deleting symbolic self-reference without --no-deref fails' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tgit -C repo symbolic-ref refs/heads/self refs/heads/self &&\n+\tcat >expect <<-EOF &&\n+\terror: multiple updates for ${SQ}refs/heads/self${SQ} (including one via symref ${SQ}refs/heads/self${SQ}) are not allowed\n+\tEOF\n+\ttest_must_fail git -C repo update-ref -d refs/heads/self 2>err &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'ref transaction: deleting symbolic self-reference with --no-deref succeeds' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tgit -C repo symbolic-ref refs/heads/self refs/heads/self &&\n+\tgit -C repo update-ref -d --no-deref refs/heads/self\n+'\n+\n+test_expect_success 'ref transaction: creating symbolic ref fails with F/D conflict' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tcat >expect <<-EOF &&\n+\terror: unable to write symref for refs/heads: file/directory conflict\n+\tEOF\n+\ttest_must_fail git -C repo symbolic-ref refs/heads refs/heads/foo 2>err &&\n+\ttest_cmp expect err\n+'\n+\n+test_expect_success 'ref transaction: ref deletion' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file &&\n+\t\tHEAD_OID=$(git show-ref -s --verify HEAD) &&\n+\t\tcat >expect <<-EOF &&\n+\t\t$HEAD_OID refs/heads/main\n+\t\t$HEAD_OID refs/tags/file\n+\t\tEOF\n+\t\tgit show-ref >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\ttest_must_fail git update-ref -d refs/tags/file $INVALID_OID &&\n+\t\tgit show-ref >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit update-ref -d refs/tags/file $HEAD_OID &&\n+\t\techo \"$HEAD_OID refs/heads/main\" >expect &&\n+\t\tgit show-ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'ref transaction: writes cause auto-compaction' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\n+\tgit init repo &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\ttest_commit -C repo --no-tag A &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n+\n+\ttest_commit -C repo --no-tag B &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+check_fsync_events () {\n+\tlocal trace=\"$1\" &&\n+\tshift &&\n+\n+\tcat >expect &&\n+\tsed -n \\\n+\t\t-e '/^{\"event\":\"counter\",.*\"category\":\"fsync\",/ {\n+\t\t\ts/.*\"category\":\"fsync\",//;\n+\t\t\ts/}$//;\n+\t\t\tp;\n+\t\t}' \\\n+\t\t<\"$trace\" >actual &&\n+\ttest_cmp expect actual\n+}\n+\n+test_expect_success 'ref transaction: writes are synced' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo initial &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\tGIT_TEST_FSYNC=true \\\n+\t\tgit -C repo -c core.fsync=reference \\\n+\t\t-c core.fsyncMethod=fsync update-ref refs/heads/branch HEAD &&\n+\tcheck_fsync_events trace2.txt <<-EOF\n+\t\"name\":\"hardware-flush\",\"count\":2\n+\tEOF\n+'\n+\n+test_expect_success 'pack-refs: compacts tables' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\n+\ttest_commit -C repo A &&\n+\tls -1 repo/.git/reftable >table-files &&\n+\ttest_line_count = 4 table-files &&\n+\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n+\n+\tgit -C repo pack-refs &&\n+\tls -1 repo/.git/reftable >table-files &&\n+\ttest_line_count = 2 table-files &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'pack-refs: prunes stale tables' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttouch repo/.git/reftable/stale-table.ref &&\n+\tgit -C repo pack-refs &&\n+\ttest_path_is_missing repo/.git/reftable/stable-ref.ref\n+'\n+\n+test_expect_success 'pack-refs: does not prune non-table files' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttouch repo/.git/reftable/garbage &&\n+\tgit -C repo pack-refs &&\n+\ttest_path_is_file repo/.git/reftable/garbage\n+'\n+\n+for umask in 002 022\n+do\n+\ttest_expect_success POSIXPERM 'pack-refs: honors core.sharedRepository' '\n+\t\ttest_when_finished \"rm -rf repo\" &&\n+\t\t(\n+\t\t\tumask $umask &&\n+\t\t\tgit init --shared=true repo &&\n+\t\t\ttest_commit -C repo A &&\n+\t\t\ttest_line_count = 3 repo/.git/reftable/tables.list\n+\t\t) &&\n+\t\tgit -C repo pack-refs &&\n+\t\ttest_expect_perms \"-rw-rw-r--\" repo/.git/reftable/tables.list &&\n+\t\tfor table in repo/.git/reftable/*.ref\n+\t\tdo\n+\t\t\ttest_expect_perms \"-rw-rw-r--\" \"$table\" ||\n+\t\t\treturn 1\n+\t\tdone\n+\t'\n+done\n+\n+test_expect_success 'packed-refs: writes are synced' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo initial &&\n+\ttest_line_count = 2 table-files &&\n+\n+\t: >trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\tGIT_TEST_FSYNC=true \\\n+\t\tgit -C repo -c core.fsync=reference \\\n+\t\t-c core.fsyncMethod=fsync pack-refs &&\n+\tcheck_fsync_events trace2.txt <<-EOF\n+\t\"name\":\"hardware-flush\",\"count\":2\n+\tEOF\n+'\n+\n+test_expect_success 'ref iterator: bogus names are flagged' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit --no-tag file &&\n+\t\ttest-tool ref-store main update-ref msg \"refs/heads/bogus..name\" $(git rev-parse HEAD) $ZERO_OID REF_SKIP_REFNAME_VERIFICATION &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\t$ZERO_OID refs/heads/bogus..name 0xc\n+\t\t$(git rev-parse HEAD) refs/heads/main 0x0\n+\t\tEOF\n+\t\ttest-tool ref-store main for-each-ref \"\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'ref iterator: missing object IDs are not flagged' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest-tool ref-store main update-ref msg \"refs/heads/broken-hash\" $INVALID_OID $ZERO_OID REF_SKIP_OID_VERIFICATION &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\t$INVALID_OID refs/heads/broken-hash 0x0\n+\t\tEOF\n+\t\ttest-tool ref-store main for-each-ref \"\" >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'basic: commit and list refs' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo file &&\n+\ttest_write_lines refs/heads/main refs/tags/file >expect &&\n+\tgit -C repo for-each-ref --format=\"%(refname)\" >actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'basic: can write large commit message' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tperl -e \"\n+\t\tprint \\\"this is a long commit message\\\" x 50000\n+\t\" >commit-msg &&\n+\tgit -C repo commit --allow-empty --file=../commit-msg\n+'\n+\n+test_expect_success 'basic: show-ref fails with empty repository' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_must_fail git -C repo show-ref >actual &&\n+\ttest_must_be_empty actual\n+'\n+\n+test_expect_success 'basic: can check out unborn branch' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\tgit -C repo checkout -b main\n+'\n+\n+test_expect_success 'basic: peeled tags are stored' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo file &&\n+\tgit -C repo tag -m \"annotated tag\" test_tag HEAD &&\n+\tfor ref in refs/heads/main refs/tags/file refs/tags/test_tag refs/tags/test_tag^{}\n+\tdo\n+\t\techo \"$(git -C repo rev-parse \"$ref\") $ref\" || return 1\n+\tdone >expect &&\n+\tgit -C repo show-ref -d >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'basic: for-each-ref can print symrefs' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file &&\n+\t\tgit branch &&\n+\t\tgit symbolic-ref refs/heads/sym refs/heads/main &&\n+\t\tcat >expected <<-EOF &&\n+\t\trefs/heads/main\n+\t\tEOF\n+\t\tgit for-each-ref --format=\"%(symref)\" refs/heads/sym >actual &&\n+\t\ttest_cmp expected actual\n+\t)\n+'\n+\n+test_expect_success 'basic: notes' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\twrite_script fake_editor <<-\\EOF &&\n+\t\techo \"$MSG\" >\"$1\"\n+\t\techo \"$MSG\" >&2\n+\t\tEOF\n+\n+\t\ttest_commit 1st &&\n+\t\ttest_commit 2nd &&\n+\t\tGIT_EDITOR=./fake_editor MSG=b4 git notes add &&\n+\t\tGIT_EDITOR=./fake_editor MSG=b3 git notes edit &&\n+\t\techo b4 >expect &&\n+\t\tgit notes --ref commits@{1} show >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'basic: stash' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file &&\n+\t\tgit stash list >expect &&\n+\t\ttest_line_count = 0 expect &&\n+\n+\t\techo hoi >>file.t &&\n+\t\tgit stash push -m stashed &&\n+\t\tgit stash list >expect &&\n+\t\ttest_line_count = 1 expect &&\n+\n+\t\tgit stash clear &&\n+\t\tgit stash list >expect &&\n+\t\ttest_line_count = 0 expect\n+\t)\n+'\n+\n+test_expect_success 'basic: cherry-pick' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit message1 file1 &&\n+\t\ttest_commit message2 file2 &&\n+\t\tgit branch source &&\n+\t\tgit checkout HEAD^ &&\n+\t\ttest_commit message3 file3 &&\n+\t\tgit cherry-pick source &&\n+\t\ttest_path_is_file file2\n+\t)\n+'\n+\n+test_expect_success 'basic: rebase' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit message1 file1 &&\n+\t\ttest_commit message2 file2 &&\n+\t\tgit branch source &&\n+\t\tgit checkout HEAD^ &&\n+\t\ttest_commit message3 file3 &&\n+\t\tgit rebase source &&\n+\t\ttest_path_is_file file2\n+\t)\n+'\n+\n+test_expect_success 'reflog: can delete separate reflog entries' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit file &&\n+\t\ttest_commit file2 &&\n+\t\ttest_commit file3 &&\n+\t\ttest_commit file4 &&\n+\t\tgit reflog >actual &&\n+\t\tgrep file3 actual &&\n+\n+\t\tgit reflog delete HEAD@{1} &&\n+\t\tgit reflog >actual &&\n+\t\t! grep file3 actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: can switch to previous branch' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file1 &&\n+\t\tgit checkout -b branch1 &&\n+\t\ttest_commit file2 &&\n+\t\tgit checkout -b branch2 &&\n+\t\tgit switch - &&\n+\t\tgit rev-parse --symbolic-full-name HEAD >actual &&\n+\t\techo refs/heads/branch1 >expect &&\n+\t\ttest_cmp actual expect\n+\t)\n+'\n+\n+test_expect_success 'reflog: copying branch writes reflog entry' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit file1 &&\n+\t\ttest_commit file2 &&\n+\t\toid=$(git rev-parse --short HEAD) &&\n+\t\tgit branch src &&\n+\t\tcat >expect <<-EOF &&\n+\t\t${oid} dst@{0}: Branch: copied refs/heads/src to refs/heads/dst\n+\t\t${oid} dst@{1}: branch: Created from main\n+\t\tEOF\n+\t\tgit branch -c src dst &&\n+\t\tgit reflog dst >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: renaming branch writes reflog entry' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\tgit symbolic-ref HEAD refs/heads/before &&\n+\t\ttest_commit file &&\n+\t\tgit show-ref >expected.refs &&\n+\t\tsed s/before/after/g expected &&\n+\t\tgit branch -M after &&\n+\t\tgit show-ref >actual &&\n+\t\ttest_cmp expected actual &&\n+\t\techo refs/heads/after >expected &&\n+\t\tgit symbolic-ref HEAD >actual &&\n+\t\ttest_cmp expected actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: can store empty logs' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_must_fail test-tool ref-store main reflog-exists refs/heads/branch &&\n+\t\ttest-tool ref-store main create-reflog refs/heads/branch &&\n+\t\ttest-tool ref-store main reflog-exists refs/heads/branch &&\n+\t\ttest-tool ref-store main for-each-reflog-ent-reverse refs/heads/branch >actual &&\n+\t\ttest_must_be_empty actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: expiry empties reflog' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit initial &&\n+\t\tgit checkout -b branch &&\n+\t\ttest_commit fileA &&\n+\t\ttest_commit fileB &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\tcommit: fileB\n+\t\tcommit: fileA\n+\t\tbranch: Created from HEAD\n+\t\tEOF\n+\t\tgit reflog show --format=\"%gs\" refs/heads/branch >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit reflog expire branch --expire=all &&\n+\t\tgit reflog show --format=\"%gs\" refs/heads/branch >actual &&\n+\t\ttest_must_be_empty actual &&\n+\t\ttest-tool ref-store main reflog-exists refs/heads/branch\n+\t)\n+'\n+\n+test_expect_success 'reflog: can be deleted' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\ttest-tool ref-store main reflog-exists refs/heads/main &&\n+\t\ttest-tool ref-store main delete-reflog refs/heads/main &&\n+\t\ttest_must_fail test-tool ref-store main reflog-exists refs/heads/main\n+\t)\n+'\n+\n+test_expect_success 'reflog: garbage collection deletes reflog entries' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\tfor count in $(test_seq 1 10)\n+\t\tdo\n+\t\t\ttest_commit \"number $count\" file.t $count number-$count ||\n+\t\t\treturn 1\n+\t\tdone &&\n+\t\tgit reflog refs/heads/main >actual &&\n+\t\ttest_line_count = 10 actual &&\n+\t\tgrep \"commit (initial): number 1\" actual &&\n+\t\tgrep \"commit: number 10\" actual &&\n+\n+\t\tgit gc &&\n+\t\tgit reflog refs/heads/main >actual &&\n+\t\ttest_line_count = 0 actual\n+\t)\n+'\n+\n+test_expect_success 'reflog: updates via HEAD update HEAD reflog' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit main-one &&\n+\t\tgit checkout -b new-branch &&\n+\t\ttest_commit new-one &&\n+\t\ttest_commit new-two &&\n+\n+\t\techo new-one >expect &&\n+\t\tgit log -1 --format=%s HEAD@{1} >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'worktree: adding worktree creates separate stack' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\ttest_path_is_file repo/.git/worktrees/worktree/refs/heads &&\n+\techo \"ref: refs/heads/.invalid\" >expect &&\n+\ttest_cmp expect repo/.git/worktrees/worktree/HEAD &&\n+\ttest_path_is_dir repo/.git/worktrees/worktree/reftable &&\n+\ttest_path_is_file repo/.git/worktrees/worktree/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: pack-refs in main repo packs main refs' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tgit -C repo worktree add ../worktree &&\n+\n+\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\tgit -C repo pack-refs &&\n+\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: pack-refs in worktree packs worktree refs' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tgit -C repo worktree add ../worktree &&\n+\n+\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: creating shared ref updates main stack' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\tgit -C repo pack-refs &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tgit -C worktree update-ref refs/heads/shared HEAD &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: creating per-worktree ref updates worktree stack' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\tgit -C repo pack-refs &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tgit -C worktree update-ref refs/bisect/per-worktree HEAD &&\n+\ttest_line_count = 2 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: creating per-worktree ref from main repo' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\tgit -C repo pack-refs &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tgit -C repo update-ref worktrees/worktree/refs/bisect/per-worktree HEAD &&\n+\ttest_line_count = 2 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: creating per-worktree ref from second worktree' '\n+\ttest_when_finished \"rm -rf repo wt1 wt2\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../wt1 &&\n+\tgit -C repo worktree add ../wt2 &&\n+\tgit -C repo pack-refs &&\n+\tgit -C wt1 pack-refs &&\n+\tgit -C wt2 pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/wt1/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/worktrees/wt2/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tgit -C wt1 update-ref worktrees/wt2/refs/bisect/per-worktree HEAD &&\n+\ttest_line_count = 1 repo/.git/worktrees/wt1/reftable/tables.list &&\n+\ttest_line_count = 2 repo/.git/worktrees/wt2/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n+test_expect_success 'worktree: can create shared and per-worktree ref in one transaction' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tgit -C repo worktree add ../worktree &&\n+\tgit -C repo pack-refs &&\n+\tgit -C worktree pack-refs &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n+\n+\tcat >stdin <<-EOF &&\n+\tcreate worktrees/worktree/refs/bisect/per-worktree HEAD\n+\tcreate refs/branches/shared HEAD\n+\tEOF\n+\tgit -C repo update-ref --stdin expect &&\n+\tgit -C worktree symbolic-ref HEAD >actual &&\n+\ttest_cmp expect actual &&\n+\tgit -C worktree checkout branch1\n+'\n+\n+test_expect_success 'worktree: adds worktree with detached HEAD' '\n+\ttest_when_finished \"rm -rf repo worktree\" &&\n+\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tgit -C repo rev-parse main >expect &&\n+\n+\tgit -C repo worktree add --detach ../worktree main &&\n+\tgit -C worktree rev-parse HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'fetch: accessing FETCH_HEAD special ref works' '\n+\ttest_when_finished \"rm -rf repo sub\" &&\n+\n+\tgit init sub &&\n+\ttest_commit -C sub two &&\n+\tgit -C sub rev-parse HEAD >expect &&\n+\n+\tgit init repo &&\n+\ttest_commit -C repo one &&\n+\tgit -C repo fetch ../sub &&\n+\tgit -C repo rev-parse FETCH_HEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\ndiff --git a/t/t0611-reftable-httpd.sh b/t/t0611-reftable-httpd.sh\nnew file mode 100755\nindex 0000000000..5e05b9c1f2\n--- /dev/null\n+++ b/t/t0611-reftable-httpd.sh\n@@ -0,0 +1,26 @@\n+#!/bin/sh\n+\n+test_description='reftable HTTPD tests'\n+\n+. ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd\n+\n+REPO=\"$HTTPD_DOCUMENT_ROOT_PATH/repo\"\n+\n+test_expect_success 'serving ls-remote' '\n+\tgit init --ref-format=reftable -b main \"$REPO\" &&\n+\tcd \"$REPO\" &&\n+\ttest_commit m1 &&\n+\t>.git/git-daemon-export-ok &&\n+\tgit ls-remote \"http://127.0.0.1:$LIB_HTTPD_PORT/smart/repo\" | cut -f 2-2 -d \"\t\" >actual &&\n+\tcat >expect <<-EOF &&\n+\tHEAD\n+\trefs/heads/main\n+\trefs/tags/m1\n+\tEOF\n+\ttest_cmp actual expect\n+'\n+\n+test_done\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 042f557a6f..c8af8dab79 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1755,6 +1755,8 @@ esac\n case \"$GIT_DEFAULT_REF_FORMAT\" in\n files)\n \ttest_set_prereq REFFILES;;\n+reftable)\n+\ttest_set_prereq REFTABLE;;\n *)\n \techo 2>&1 \"error: unknown ref format $GIT_DEFAULT_REF_FORMAT\"\n \texit 1\n-- \n2.43.GIT\n\n"},{"id":"488121","messageId":"30e5feb28c79154dba4e5f4b399eb929af9ca558.1707288261.git.ps@pks.im","threadId":"60815","inReplyTo":"cover.1707288261.git.ps@pks.im","subject":"[PATCH v4 2/2] ci: add jobs to test with the reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-07T07:20:35Z","receivedAt":"2024-02-07T07:20:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Add CI jobs for both GitHub Workflows and GitLab CI to run Git with the\nnew reftable backend.\n\nSigned-off-by: Patrick Steinhardt \n---\n .github/workflows/main.yml | 9 +++++++++\n .gitlab-ci.yml | 9 +++++++++\n ci/lib.sh | 2 +-\n ci/run-build-and-tests.sh | 3 +++\n 4 files changed, 22 insertions(+), 1 deletion(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 4d97da57ec..1b43e49dad 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -266,6 +266,9 @@ jobs:\n - jobname: linux-sha256\n cc: clang\n pool: ubuntu-latest\n+ - jobname: linux-reftable\n+ cc: clang\n+ pool: ubuntu-latest\n - jobname: linux-gcc\n cc: gcc\n cc_package: gcc-8\n@@ -277,6 +280,9 @@ jobs:\n - jobname: osx-clang\n cc: clang\n pool: macos-13\n+ - jobname: osx-reftable\n+ cc: clang\n+ pool: macos-13\n - jobname: osx-gcc\n cc: gcc\n cc_package: gcc-13\n@@ -287,6 +293,9 @@ jobs:\n - jobname: linux-leaks\n cc: gcc\n pool: ubuntu-latest\n+ - jobname: linux-reftable-leaks\n+ cc: gcc\n+ pool: ubuntu-latest\n - jobname: linux-asan-ubsan\n cc: clang\n pool: ubuntu-latest\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nindex 43bfbd8834..c0fa2fe90b 100644\n--- a/.gitlab-ci.yml\n+++ b/.gitlab-ci.yml\n@@ -26,6 +26,9 @@ test:linux:\n - jobname: linux-sha256\n image: ubuntu:latest\n CC: clang\n+ - jobname: linux-reftable\n+ image: ubuntu:latest\n+ CC: clang\n - jobname: linux-gcc\n image: ubuntu:20.04\n CC: gcc\n@@ -40,6 +43,9 @@ test:linux:\n - jobname: linux-leaks\n image: ubuntu:latest\n CC: gcc\n+ - jobname: linux-reftable-leaks\n+ image: ubuntu:latest\n+ CC: gcc\n - jobname: linux-asan-ubsan\n image: ubuntu:latest\n CC: clang\n@@ -79,6 +85,9 @@ test:osx:\n - jobname: osx-clang\n image: macos-13-xcode-14\n CC: clang\n+ - jobname: osx-reftable\n+ image: macos-13-xcode-14\n+ CC: clang\n artifacts:\n paths:\n - t/failed-test-artifacts\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex d5dd2f2697..0a73fc7bd1 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -367,7 +367,7 @@ linux-musl)\n \tMAKEFLAGS=\"$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes\"\n \tMAKEFLAGS=\"$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8\"\n \t;;\n-linux-leaks)\n+linux-leaks|linux-reftable-leaks)\n \texport SANITIZE=leak\n \texport GIT_TEST_PASSING_SANITIZE_LEAK=true\n \texport GIT_TEST_SANITIZE_LEAK_LOG=true\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 7a1466b868..c192bd613c 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -37,6 +37,9 @@ linux-clang)\n linux-sha256)\n \texport GIT_TEST_DEFAULT_HASH=sha256\n \t;;\n+linux-reftable|linux-reftable-leaks|osx-reftable)\n+\texport GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+\t;;\n pedantic)\n \t# Don't run the tests; we only care about whether Git can be\n \t# built.\n-- \n2.43.GIT\n\n"},{"id":"488193","messageId":"20240207223120.GA537741@coredump.intra.peff.net","threadId":"60815","inReplyTo":"5de60d46bdccbfbf0a923abc2f45eda07f30c110.1707288261.git.ps@pks.im","subject":"Re: [PATCH v4 1/2] refs: introduce reftable backend","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-02-07T22:31:20Z","receivedAt":"2024-02-07T22:31:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 07, 2024 at 08:20:31AM +0100, Patrick Steinhardt wrote:\n\n> +static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n> +{\n> [...]\n> +\t/*\n> +\t * Create the reflog entry for the newly created branch.\n> +\t */\n> +\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n> +\tmemset(&logs[logs_nr], 0, sizeof(logs[logs_nr]));\n> +\tfill_reftable_log_record(&logs[logs_nr]);\n> +\tlogs[logs_nr].refname = (char *)arg->newname;\n> +\tlogs[logs_nr].update_index = creation_ts;\n> +\tlogs[logs_nr].value.update.message =\n> +\t\txstrndup(arg->logmsg, arg->refs->write_options.block_size / 2);\n> +\tlogs[logs_nr].value.update.new_hash = old_ref.value.val1;\n> +\tlogs_nr++;\n> +\n> +\t/*\n> +\t * In addition to writing the reflog entry for the new branch, we also\n> +\t * copy over all log entries from the old reflog. Last but not least,\n> +\t * when renaming we also have to delete all the old reflog entries.\n> +\t */\n> +\tret = reftable_merged_table_seek_log(mt, &it, arg->oldname);\n> +\tif (ret < 0)\n> +\t\treturn ret;\n\nShould this last line be \"goto done\" as is used elsewhere in the\nfunction? Otherwise we are at least leaking the \"logs\" array (and\npossibly some of the other cleanup is important, too).\n\n> +\twhile (1) {\n> +\t\tret = reftable_iterator_next_log(&it, &old_log);\n> +\t\tif (ret < 0)\n> +\t\t\tgoto done;\n> +\t\tif (ret > 0 || strcmp(old_log.refname, arg->oldname)) {\n> +\t\t\tret = 0;\n> +\t\t\tbreak;\n> +\t\t}\n\nThis \"ret = 0\" doesn't have any effect. We break out of the loop, and\nthen...\n\n> +\t}\n> +\n> +\tret = reftable_writer_add_logs(writer, logs, logs_nr);\n> +\tif (ret < 0)\n> +\t\tgoto done;\n\n...the first thing we do is write over it. I dunno if it's worth keeping\nas a maintenance precaution, though (if the code after the loop changed\nto omit that assignment, then setting \"ret\" would become important).\n\nBoth were noticed by Coverity (along with several other false\npositives).\n\n-Peff\n"},{"id":"488206","messageId":"ZcRinffkQJNWyiGZ@tanuki","threadId":"60815","inReplyTo":"20240207223120.GA537741@coredump.intra.peff.net","subject":"Re: [PATCH v4 1/2] refs: introduce reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-08T05:11:57Z","receivedAt":"2024-02-08T05:12:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Feb 07, 2024 at 05:31:20PM -0500, Jeff King wrote:\n> On Wed, Feb 07, 2024 at 08:20:31AM +0100, Patrick Steinhardt wrote:\n> \n> > +static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n> > +{\n> > [...]\n> > +\t/*\n> > +\t * Create the reflog entry for the newly created branch.\n> > +\t */\n> > +\tALLOC_GROW(logs, logs_nr + 1, logs_alloc);\n> > +\tmemset(&logs[logs_nr], 0, sizeof(logs[logs_nr]));\n> > +\tfill_reftable_log_record(&logs[logs_nr]);\n> > +\tlogs[logs_nr].refname = (char *)arg->newname;\n> > +\tlogs[logs_nr].update_index = creation_ts;\n> > +\tlogs[logs_nr].value.update.message =\n> > +\t\txstrndup(arg->logmsg, arg->refs->write_options.block_size / 2);\n> > +\tlogs[logs_nr].value.update.new_hash = old_ref.value.val1;\n> > +\tlogs_nr++;\n> > +\n> > +\t/*\n> > +\t * In addition to writing the reflog entry for the new branch, we also\n> > +\t * copy over all log entries from the old reflog. Last but not least,\n> > +\t * when renaming we also have to delete all the old reflog entries.\n> > +\t */\n> > +\tret = reftable_merged_table_seek_log(mt, &it, arg->oldname);\n> > +\tif (ret < 0)\n> > +\t\treturn ret;\n> \n> Should this last line be \"goto done\" as is used elsewhere in the\n> function? Otherwise we are at least leaking the \"logs\" array (and\n> possibly some of the other cleanup is important, too).\n\nIndeed we should.\n\n> > +\twhile (1) {\n> > +\t\tret = reftable_iterator_next_log(&it, &old_log);\n> > +\t\tif (ret < 0)\n> > +\t\t\tgoto done;\n> > +\t\tif (ret > 0 || strcmp(old_log.refname, arg->oldname)) {\n> > +\t\t\tret = 0;\n> > +\t\t\tbreak;\n> > +\t\t}\n> \n> This \"ret = 0\" doesn't have any effect. We break out of the loop, and\n> then...\n> \n> > +\t}\n> > +\n> > +\tret = reftable_writer_add_logs(writer, logs, logs_nr);\n> > +\tif (ret < 0)\n> > +\t\tgoto done;\n> \n> ...the first thing we do is write over it. I dunno if it's worth keeping\n> as a maintenance precaution, though (if the code after the loop changed\n> to omit that assignment, then setting \"ret\" would become important).\n\nYeah, I think this one we should keep. It's also a repeating pattern\nthat we have in many other places, so it helps lower the mental burden\nwhen it looks similar to all the others.\n\n> Both were noticed by Coverity (along with several other false\n> positives).\n\nIs the Coverity instance publicly accessible? If not, can you maybe\ninvite me so that I get a better feeling for them?\n\nI'll send a patch to fix the missing `goto done` above, thanks!\n\nPatrick\n"},{"id":"488685","messageId":"20240215053902.GF2821179@coredump.intra.peff.net","threadId":"60815","inReplyTo":"ZcRinffkQJNWyiGZ@tanuki","subject":"Re: [PATCH v4 1/2] refs: introduce reftable backend","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-02-15T05:39:02Z","receivedAt":"2024-02-15T05:39:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 08, 2024 at 06:11:57AM +0100, Patrick Steinhardt wrote:\n\n> > > +\tret = reftable_writer_add_logs(writer, logs, logs_nr);\n> > > +\tif (ret < 0)\n> > > +\t\tgoto done;\n> > \n> > ...the first thing we do is write over it. I dunno if it's worth keeping\n> > as a maintenance precaution, though (if the code after the loop changed\n> > to omit that assignment, then setting \"ret\" would become important).\n> \n> Yeah, I think this one we should keep. It's also a repeating pattern\n> that we have in many other places, so it helps lower the mental burden\n> when it looks similar to all the others.\n\nThat sounds quite reasonable to me.\n\n> > Both were noticed by Coverity (along with several other false\n> > positives).\n> \n> Is the Coverity instance publicly accessible? If not, can you maybe\n> invite me so that I get a better feeling for them?\n\nI don't think there's a way to make it publicly accessible. Probably I\ncould invite you to the project if you sign up for a Coverity account.\nBut it's a build of next plus my personal in-progress topics, which is\nprobably not ideal for other people to look at.\n\nThere are instructions for setting up your own scan in a56b6230d0 (ci:\nadd a GitHub workflow to submit Coverity scans, 2023-09-25). The unsaid\npart there is signing up for Coverity and registering a project to get\nthe necessary tokens. Probably start here:\n\n https://scan.coverity.com/users/sign_up\n\nbut I don't remember the details.\n\nThe Actions workflow is in \"master\", so in theory we could register a\nproject for git.git and get automatic builds when Junio pushes to\n'next', etc. In practice I have found that it requires a human looking\nover the results, but if people at least had access they could poke\naround.\n\n(I suspect it would also be easy to port the workflow over to run on\nGitLab CI, as well, if you wanted to).\n\n-Peff\n"}]}