{"thread":{"id":"62395","subject":"[PATCH 0/7] reftable: stop using Git subsystems","startedAt":"2024-10-23T09:55:57Z","lastAt":"2024-11-20T01:09:13Z","messageCount":43,"participants":["Patrick Steinhardt","Eric Sunshine","Justin Tobler","Junio C Hamano","karthik nayak"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"505911","messageId":"cover.1729677003.git.ps@pks.im","threadId":"62395","inReplyTo":null,"subject":"[PATCH 0/7] reftable: stop using Git subsystems","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-23T09:55:52Z","receivedAt":"2024-10-23T09:55:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series is another step towards detangling the reftable\nlibrary from libgit.a. This step removes all uses of Git subsystems like\n\"hash.h\", \"tempfile.h\" and \"lockfile.h\" in favor of either providing our\nown infra for this (\"hash.h\") or by implementing compatibility shims\nthat bridge between Git and the reftable library (\"tempfile.h\",\n\"lockfile.h\").\n\nThis is the second-last step in the overall effort to detangle those two\ncodebases from each other. The last step will be to split out POSIX bits\nfrom \"git-compat-util.h\" into a separate \"compat/posix.h\" header that we\nthen use from the reftable library such that we don't pull in any of the\nGit-specific parts into it.\n\nThis series is built on top of fd3785337b (The third batch, 2024-10-22)\nwith ps/reftable-strbuf at 20590cd287 (reftable: handle trivial\n`reftable_buf` errors, 2024-10-17) merged into it.\n\nThanks!\n\nPatrick\n\nPatrick Steinhardt (7):\n reftable/system: move \"dir.h\" to its only user\n reftable: explicitly handle hash format IDs\n reftable/system: stop depending on \"hash.h\"\n reftable/stack: stop using `fsync_component()` directly\n reftable/system: provide thin wrapper for tempfile subsystem\n reftable/stack: drop only use of `get_locked_file_path()`\n reftable/system: provide thin wrapper for lockfile subsystem\n\n Makefile | 1 +\n refs/reftable-backend.c | 19 +++-\n reftable/basics.c | 13 ++-\n reftable/basics.h | 9 +-\n reftable/merged.c | 4 +-\n reftable/merged.h | 3 +-\n reftable/reader.c | 14 ++-\n reftable/reader.h | 4 +-\n reftable/reftable-basics.h | 13 +++\n reftable/reftable-merged.h | 4 +-\n reftable/reftable-reader.h | 2 +-\n reftable/reftable-record.h | 12 +-\n reftable/reftable-writer.h | 8 +-\n reftable/stack.c | 171 ++++++++++++++--------------\n reftable/system.c | 126 ++++++++++++++++++++\n reftable/system.h | 88 +++++++++++++-\n reftable/writer.c | 20 +++-\n t/helper/test-reftable.c | 4 +-\n t/unit-tests/lib-reftable.c | 5 +-\n t/unit-tests/lib-reftable.h | 2 +-\n t/unit-tests/t-reftable-block.c | 41 +++----\n t/unit-tests/t-reftable-merged.c | 26 ++---\n t/unit-tests/t-reftable-pq.c | 3 +-\n t/unit-tests/t-reftable-reader.c | 4 +-\n t/unit-tests/t-reftable-readwrite.c | 41 +++----\n t/unit-tests/t-reftable-record.c | 59 +++++-----\n t/unit-tests/t-reftable-stack.c | 37 +++---\n 27 files changed, 504 insertions(+), 229 deletions(-)\n create mode 100644 reftable/system.c\n\n-- \n2.47.0.118.gfd3785337b.dirty\n\n"},{"id":"505912","messageId":"036cc8f9d602a817aa579092bdeaef5d1156214e.1729677003.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1729677003.git.ps@pks.im","subject":"[PATCH 1/7] reftable/system: move \"dir.h\" to its only user","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-23T09:55:56Z","receivedAt":"2024-10-23T09:55:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We still include \"dir.h\" in \"reftable/system.h\" evne though it is not\nused by anything but by a single unit test. Move it over into that unit\ntest so that we don't accidentally use any functionality provided by it\nin the reftable codebase.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/system.h | 1 -\n t/unit-tests/t-reftable-stack.c | 1 +\n 2 files changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 5ec85833434..8564213475e 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -15,7 +15,6 @@ license that can be found in the LICENSE file or at\n #include \"lockfile.h\"\n #include \"tempfile.h\"\n #include \"hash.h\" /* hash ID, sizes.*/\n-#include \"dir.h\" /* remove_dir_recursively, for tests.*/\n \n int hash_size(uint32_t id);\n \ndiff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c\nindex 72f6747064f..1b4363a58fc 100644\n--- a/t/unit-tests/t-reftable-stack.c\n+++ b/t/unit-tests/t-reftable-stack.c\n@@ -8,6 +8,7 @@ license that can be found in the LICENSE file or at\n \n #include \"test-lib.h\"\n #include \"lib-reftable.h\"\n+#include \"dir.h\"\n #include \"reftable/merged.h\"\n #include \"reftable/reader.h\"\n #include \"reftable/reftable-error.h\"\n-- \n2.47.0.118.gfd3785337b.dirty\n\n"},{"id":"505913","messageId":"c1bd8e2b3c4c524a21f68e822812e41179827e4d.1729677003.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1729677003.git.ps@pks.im","subject":"[PATCH 2/7] reftable: explicitly handle hash format IDs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-23T09:55:59Z","receivedAt":"2024-10-23T09:56:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The hash format IDs are used for two different things across the\nreftable codebase:\n\n - They are used as a 32 bit unsigned integer when reading and writing\n the header in order to identify the hash function.\n\n - They are used internally to identify which hash function is in use.\n\nWhen one only considers the second usecase one might think that one can\neasily change the representation of those hash IDs. But because those\nIDs end up in the reftable header and footer on disk it is important\nthat those never change.\n\nCreate separate constants `REFTABLE_FORMAT_ID_*` and use them in\ncontexts where we read or write reftable headers. This serves multiple\npurposes:\n\n - It allows us to more easily discern cases where we actually use\n those constants for the on-disk format.\n\n - It detangles us from the same constants that are defined in\n libgit.a, which is another required step to convert the reftable\n library to become standalone.\n\n - It makes the next step easier where we stop using `GIT_*_FORMAT_ID`\n constants in favor of a custom enum.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 7 +++++++\n reftable/reader.c | 10 ++++++----\n reftable/writer.c | 16 +++++++++++++++-\n 3 files changed, 28 insertions(+), 5 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 7aa46d7c30d..86141602e74 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -150,4 +150,11 @@ int common_prefix_size(struct reftable_buf *a, struct reftable_buf *b);\n \n int hash_size(uint32_t id);\n \n+/*\n+ * Format IDs that identify the hash function used by a reftable. Note that\n+ * these constants end up on disk and thus mustn't change.\n+ */\n+#define REFTABLE_FORMAT_ID_SHA1 ((uint32_t) 0x73686131)\n+#define REFTABLE_FORMAT_ID_SHA256 ((uint32_t) 0x73323536)\n+\n #endif\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 90dc950b577..64eb6938efe 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -109,16 +109,18 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n \tif (r->version == 1) {\n \t\tr->hash_id = GIT_SHA1_FORMAT_ID;\n \t} else {\n-\t\tr->hash_id = get_be32(f);\n-\t\tswitch (r->hash_id) {\n-\t\tcase GIT_SHA1_FORMAT_ID:\n+\t\tswitch (get_be32(f)) {\n+\t\tcase REFTABLE_FORMAT_ID_SHA1:\n+\t\t\tr->hash_id = GIT_SHA1_FORMAT_ID;\n \t\t\tbreak;\n-\t\tcase GIT_SHA256_FORMAT_ID:\n+\t\tcase REFTABLE_FORMAT_ID_SHA256:\n+\t\t\tr->hash_id = GIT_SHA256_FORMAT_ID;\n \t\t\tbreak;\n \t\tdefault:\n \t\t\terr = REFTABLE_FORMAT_ERROR;\n \t\t\tgoto done;\n \t\t}\n+\n \t\tf += 4;\n \t}\n \ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex fd136794d5a..9aa45de6340 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -103,8 +103,22 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \tput_be64(dest + 8, w->min_update_index);\n \tput_be64(dest + 16, w->max_update_index);\n \tif (writer_version(w) == 2) {\n-\t\tput_be32(dest + 24, w->opts.hash_id);\n+\t\tuint32_t hash_id;\n+\n+\t\tswitch (w->opts.hash_id) {\n+\t\tcase GIT_SHA1_FORMAT_ID:\n+\t\t\thash_id = REFTABLE_FORMAT_ID_SHA1;\n+\t\t\tbreak;\n+\t\tcase GIT_SHA256_FORMAT_ID:\n+\t\t\thash_id = REFTABLE_FORMAT_ID_SHA256;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tput_be32(dest + 24, hash_id);\n \t}\n+\n \treturn header_size(writer_version(w));\n }\n \n-- \n2.47.0.118.gfd3785337b.dirty\n\n"},{"id":"505914","messageId":"b595668a5cdae0dff4a7271a3547c5821aa6e912.1729677003.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1729677003.git.ps@pks.im","subject":"[PATCH 3/7] reftable/system: stop depending on \"hash.h\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-23T09:56:02Z","receivedAt":"2024-10-23T09:56:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We include \"hash.h\" in \"reftable/system.h\" such that we can use hash\nformat IDs as well as the raw size of SHA1 and SHA256. As we are in the\nprocess of converting the reftable library to become standalone we of\ncourse cannot rely on those constants anymore.\n\nIntroduce a new `enum reftable_hash` to replace internal uses of the\nhash format IDs and new constants that replace internal uses of the hash\nsize. Adapt the reftable backend to set up the correct hash function.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 12 +++++-\n reftable/basics.c | 13 ++++---\n reftable/basics.h | 2 +-\n reftable/merged.c | 4 +-\n reftable/merged.h | 3 +-\n reftable/reader.c | 8 ++--\n reftable/reader.h | 4 +-\n reftable/reftable-basics.h | 13 +++++++\n reftable/reftable-merged.h | 4 +-\n reftable/reftable-reader.h | 2 +-\n reftable/reftable-record.h | 12 +++---\n reftable/reftable-writer.h | 2 +-\n reftable/stack.c | 4 +-\n reftable/system.h | 3 --\n reftable/writer.c | 8 ++--\n t/helper/test-reftable.c | 4 +-\n t/unit-tests/lib-reftable.c | 4 +-\n t/unit-tests/lib-reftable.h | 2 +-\n t/unit-tests/t-reftable-block.c | 40 +++++++++----------\n t/unit-tests/t-reftable-merged.c | 26 ++++++-------\n t/unit-tests/t-reftable-pq.c | 2 +-\n t/unit-tests/t-reftable-reader.c | 4 +-\n t/unit-tests/t-reftable-readwrite.c | 40 +++++++++----------\n t/unit-tests/t-reftable-record.c | 59 +++++++++++++++--------------\n t/unit-tests/t-reftable-stack.c | 34 ++++++++---------\n 25 files changed, 166 insertions(+), 143 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 3c6107c7ce5..7d86d920970 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -15,6 +15,7 @@\n #include \"../object.h\"\n #include \"../path.h\"\n #include \"../refs.h\"\n+#include \"../reftable/reftable-basics.h\"\n #include \"../reftable/reftable-stack.h\"\n #include \"../reftable/reftable-record.h\"\n #include \"../reftable/reftable-error.h\"\n@@ -289,7 +290,16 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \trefs->store_flags = store_flags;\n \trefs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);\n \n-\trefs->write_options.hash_id = repo->hash_algo->format_id;\n+\tswitch (repo->hash_algo->format_id) {\n+\tcase GIT_SHA1_FORMAT_ID:\n+\t\trefs->write_options.hash_id = REFTABLE_HASH_SHA1;\n+\t\tbreak;\n+\tcase GIT_SHA256_FORMAT_ID:\n+\t\trefs->write_options.hash_id = REFTABLE_HASH_SHA256;\n+\t\tbreak;\n+\tdefault:\n+\t\tBUG(\"unknown hash algorithm %d\", repo->hash_algo->format_id);\n+\t}\n \trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n \trefs->write_options.disable_auto_compact =\n \t\t!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1);\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex bc4fcc91446..7d84a5d62de 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -271,14 +271,15 @@ int common_prefix_size(struct reftable_buf *a, struct reftable_buf *b)\n \treturn p;\n }\n \n-int hash_size(uint32_t id)\n+int hash_size(enum reftable_hash id)\n {\n+\tif (!id)\n+\t\treturn REFTABLE_HASH_SIZE_SHA1;\n \tswitch (id) {\n-\tcase 0:\n-\tcase GIT_SHA1_FORMAT_ID:\n-\t\treturn GIT_SHA1_RAWSZ;\n-\tcase GIT_SHA256_FORMAT_ID:\n-\t\treturn GIT_SHA256_RAWSZ;\n+\tcase REFTABLE_HASH_SHA1:\n+\t\treturn REFTABLE_HASH_SIZE_SHA1;\n+\tcase REFTABLE_HASH_SHA256:\n+\t\treturn REFTABLE_HASH_SIZE_SHA256;\n \t}\n \tabort();\n }\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 86141602e74..0b77d047ada 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -148,7 +148,7 @@ char *reftable_strdup(const char *str);\n /* Find the longest shared prefix size of `a` and `b` */\n int common_prefix_size(struct reftable_buf *a, struct reftable_buf *b);\n \n-int hash_size(uint32_t id);\n+int hash_size(enum reftable_hash id);\n \n /*\n * Format IDs that identify the hash function used by a reftable. Note that\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 514d6facf45..5b93e20f429 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -181,7 +181,7 @@ static void iterator_from_merged_iter(struct reftable_iterator *it,\n \n int reftable_merged_table_new(struct reftable_merged_table **dest,\n \t\t\t struct reftable_reader **readers, size_t n,\n-\t\t\t uint32_t hash_id)\n+\t\t\t enum reftable_hash hash_id)\n {\n \tstruct reftable_merged_table *m = NULL;\n \tuint64_t last_max = 0;\n@@ -293,7 +293,7 @@ int reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n \treturn merged_table_init_iter(mt, it, BLOCK_TYPE_LOG);\n }\n \n-uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\n+enum reftable_hash reftable_merged_table_hash_id(struct reftable_merged_table *mt)\n {\n \treturn mt->hash_id;\n }\ndiff --git a/reftable/merged.h b/reftable/merged.h\nindex 89bd0c4b35b..13a5fe4154e 100644\n--- a/reftable/merged.h\n+++ b/reftable/merged.h\n@@ -10,11 +10,12 @@ license that can be found in the LICENSE file or at\n #define MERGED_H\n \n #include \"system.h\"\n+#include \"basics.h\"\n \n struct reftable_merged_table {\n \tstruct reftable_reader **readers;\n \tsize_t readers_len;\n-\tuint32_t hash_id;\n+\tenum reftable_hash hash_id;\n \n \t/* If unset, produce deletions. This is useful for compaction. For the\n \t * full stack, deletions should be produced. */\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 64eb6938efe..ea82955c9bc 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -67,7 +67,7 @@ static int reader_get_block(struct reftable_reader *r,\n \treturn block_source_read_block(&r->source, dest, off, sz);\n }\n \n-uint32_t reftable_reader_hash_id(struct reftable_reader *r)\n+enum reftable_hash reftable_reader_hash_id(struct reftable_reader *r)\n {\n \treturn r->hash_id;\n }\n@@ -107,14 +107,14 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n \tf += 8;\n \n \tif (r->version == 1) {\n-\t\tr->hash_id = GIT_SHA1_FORMAT_ID;\n+\t\tr->hash_id = REFTABLE_HASH_SHA1;\n \t} else {\n \t\tswitch (get_be32(f)) {\n \t\tcase REFTABLE_FORMAT_ID_SHA1:\n-\t\t\tr->hash_id = GIT_SHA1_FORMAT_ID;\n+\t\t\tr->hash_id = REFTABLE_HASH_SHA1;\n \t\t\tbreak;\n \t\tcase REFTABLE_FORMAT_ID_SHA256:\n-\t\t\tr->hash_id = GIT_SHA256_FORMAT_ID;\n+\t\t\tr->hash_id = REFTABLE_HASH_SHA256;\n \t\t\tbreak;\n \t\tdefault:\n \t\t\terr = REFTABLE_FORMAT_ERROR;\ndiff --git a/reftable/reader.h b/reftable/reader.h\nindex 010fbfe8511..d2b48a48499 100644\n--- a/reftable/reader.h\n+++ b/reftable/reader.h\n@@ -37,8 +37,8 @@ struct reftable_reader {\n \t/* Size of the file, excluding the footer. */\n \tuint64_t size;\n \n-\t/* 'sha1' for SHA1, 's256' for SHA-256 */\n-\tuint32_t hash_id;\n+\t/* The hash function used for ref records. */\n+\tenum reftable_hash hash_id;\n \n \tuint32_t block_size;\n \tuint64_t min_update_index;\ndiff --git a/reftable/reftable-basics.h b/reftable/reftable-basics.h\nindex 6e8e636b716..e0397ed5836 100644\n--- a/reftable/reftable-basics.h\n+++ b/reftable/reftable-basics.h\n@@ -11,6 +11,19 @@\n \n #include \n \n+/*\n+ * Hash functions understood by the reftable library. Note that the values are\n+ * arbitrary and somewhat random such that we can easily detect cases where the\n+ * hash hasn't been properly set up.\n+ */\n+enum reftable_hash {\n+\tREFTABLE_HASH_SHA1 = 89,\n+\tREFTABLE_HASH_SHA256 = 247,\n+};\n+#define REFTABLE_HASH_SIZE_SHA1 20\n+#define REFTABLE_HASH_SIZE_SHA256 32\n+#define REFTABLE_HASH_SIZE_MAX REFTABLE_HASH_SIZE_SHA256\n+\n /* Overrides the functions to use for memory management. */\n void reftable_set_alloc(void *(*malloc)(size_t),\n \t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nindex a970d5dd89a..f2d01c3ef82 100644\n--- a/reftable/reftable-merged.h\n+++ b/reftable/reftable-merged.h\n@@ -34,7 +34,7 @@ struct reftable_reader;\n */\n int reftable_merged_table_new(struct reftable_merged_table **dest,\n \t\t\t struct reftable_reader **readers, size_t n,\n-\t\t\t uint32_t hash_id);\n+\t\t\t enum reftable_hash hash_id);\n \n /* Initialize a merged table iterator for reading refs. */\n int reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n@@ -56,6 +56,6 @@ reftable_merged_table_min_update_index(struct reftable_merged_table *mt);\n void reftable_merged_table_free(struct reftable_merged_table *m);\n \n /* return the hash ID of the merged table. */\n-uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m);\n+enum reftable_hash reftable_merged_table_hash_id(struct reftable_merged_table *m);\n \n #endif\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nindex 6a2d0b693f5..0085fbb9032 100644\n--- a/reftable/reftable-reader.h\n+++ b/reftable/reftable-reader.h\n@@ -54,7 +54,7 @@ int reftable_reader_init_log_iterator(struct reftable_reader *r,\n \t\t\t\t struct reftable_iterator *it);\n \n /* returns the hash ID used in this table. */\n-uint32_t reftable_reader_hash_id(struct reftable_reader *r);\n+enum reftable_hash reftable_reader_hash_id(struct reftable_reader *r);\n \n /* return an iterator for the refs pointing to `oid`. */\n int reftable_reader_refs_for(struct reftable_reader *r,\ndiff --git a/reftable/reftable-record.h b/reftable/reftable-record.h\nindex 2d42463c581..ddd48eb5798 100644\n--- a/reftable/reftable-record.h\n+++ b/reftable/reftable-record.h\n@@ -9,7 +9,7 @@ license that can be found in the LICENSE file or at\n #ifndef REFTABLE_RECORD_H\n #define REFTABLE_RECORD_H\n \n-#include \"hash.h\"\n+#include \"reftable-basics.h\"\n #include \n \n /*\n@@ -40,10 +40,10 @@ struct reftable_ref_record {\n #define REFTABLE_NR_REF_VALUETYPES 4\n \t} value_type;\n \tunion {\n-\t\tunsigned char val1[GIT_MAX_RAWSZ];\n+\t\tunsigned char val1[REFTABLE_HASH_SIZE_MAX];\n \t\tstruct {\n-\t\t\tunsigned char value[GIT_MAX_RAWSZ]; /* first hash */\n-\t\t\tunsigned char target_value[GIT_MAX_RAWSZ]; /* second hash */\n+\t\t\tunsigned char value[REFTABLE_HASH_SIZE_MAX]; /* first hash */\n+\t\t\tunsigned char target_value[REFTABLE_HASH_SIZE_MAX]; /* second hash */\n \t\t} val2;\n \t\tchar *symref; /* referent, malloced 0-terminated string */\n \t} value;\n@@ -85,8 +85,8 @@ struct reftable_log_record {\n \n \tunion {\n \t\tstruct {\n-\t\t\tunsigned char new_hash[GIT_MAX_RAWSZ];\n-\t\t\tunsigned char old_hash[GIT_MAX_RAWSZ];\n+\t\t\tunsigned char new_hash[REFTABLE_HASH_SIZE_MAX];\n+\t\t\tunsigned char old_hash[REFTABLE_HASH_SIZE_MAX];\n \t\t\tchar *name;\n \t\t\tchar *email;\n \t\t\tuint64_t time;\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex e4fc9537883..211860d08a4 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -33,7 +33,7 @@ struct reftable_write_options {\n \t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n \t * Defaults to SHA1 if unset\n \t */\n-\tuint32_t hash_id;\n+\tenum reftable_hash hash_id;\n \n \t/* Default mode for creating files. If unset, use 0666 (+umask) */\n \tunsigned int default_permissions;\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex c33979536ef..9ae716ff375 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -73,7 +73,7 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \tif (_opts)\n \t\topts = *_opts;\n \tif (opts.hash_id == 0)\n-\t\topts.hash_id = GIT_SHA1_FORMAT_ID;\n+\t\topts.hash_id = REFTABLE_HASH_SHA1;\n \n \t*dest = NULL;\n \n@@ -1603,7 +1603,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n,\n \n static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n {\n-\tint version = (st->opts.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2;\n+\tint version = (st->opts.hash_id == REFTABLE_HASH_SHA1) ? 1 : 2;\n \tint overhead = header_size(version) - 1;\n \tuint64_t *sizes;\n \ndiff --git a/reftable/system.h b/reftable/system.h\nindex 8564213475e..38d3534620e 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -14,8 +14,5 @@ license that can be found in the LICENSE file or at\n #include \"git-compat-util.h\"\n #include \"lockfile.h\"\n #include \"tempfile.h\"\n-#include \"hash.h\" /* hash ID, sizes.*/\n-\n-int hash_size(uint32_t id);\n \n #endif\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 9aa45de6340..ea2f831fc58 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -79,7 +79,7 @@ static void options_set_defaults(struct reftable_write_options *opts)\n \t}\n \n \tif (opts->hash_id == 0) {\n-\t\topts->hash_id = GIT_SHA1_FORMAT_ID;\n+\t\topts->hash_id = REFTABLE_HASH_SHA1;\n \t}\n \tif (opts->block_size == 0) {\n \t\topts->block_size = DEFAULT_BLOCK_SIZE;\n@@ -88,7 +88,7 @@ static void options_set_defaults(struct reftable_write_options *opts)\n \n static int writer_version(struct reftable_writer *w)\n {\n-\treturn (w->opts.hash_id == 0 || w->opts.hash_id == GIT_SHA1_FORMAT_ID) ?\n+\treturn (w->opts.hash_id == 0 || w->opts.hash_id == REFTABLE_HASH_SHA1) ?\n \t\t\t 1 :\n \t\t\t 2;\n }\n@@ -106,10 +106,10 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \t\tuint32_t hash_id;\n \n \t\tswitch (w->opts.hash_id) {\n-\t\tcase GIT_SHA1_FORMAT_ID:\n+\t\tcase REFTABLE_HASH_SHA1:\n \t\t\thash_id = REFTABLE_FORMAT_ID_SHA1;\n \t\t\tbreak;\n-\t\tcase GIT_SHA256_FORMAT_ID:\n+\t\tcase REFTABLE_HASH_SHA256:\n \t\t\thash_id = REFTABLE_FORMAT_ID_SHA256;\n \t\t\tbreak;\n \t\tdefault:\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 5c8849d115b..3c72ed985b3 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -156,7 +156,7 @@ int cmd__dump_reftable(int argc, const char **argv)\n \tint opt_dump_blocks = 0;\n \tint opt_dump_table = 0;\n \tint opt_dump_stack = 0;\n-\tuint32_t opt_hash_id = GIT_SHA1_FORMAT_ID;\n+\tuint32_t opt_hash_id = REFTABLE_HASH_SHA1;\n \tconst char *arg = NULL, *argv0 = argv[0];\n \n \tfor (; argc > 1; argv++, argc--)\n@@ -167,7 +167,7 @@ int cmd__dump_reftable(int argc, const char **argv)\n \t\telse if (!strcmp(\"-t\", argv[1]))\n \t\t\topt_dump_table = 1;\n \t\telse if (!strcmp(\"-6\", argv[1]))\n-\t\t\topt_hash_id = GIT_SHA256_FORMAT_ID;\n+\t\t\topt_hash_id = REFTABLE_HASH_SHA256;\n \t\telse if (!strcmp(\"-s\", argv[1]))\n \t\t\topt_dump_stack = 1;\n \t\telse if (!strcmp(\"-?\", argv[1]) || !strcmp(\"-h\", argv[1])) {\ndiff --git a/t/unit-tests/lib-reftable.c b/t/unit-tests/lib-reftable.c\nindex 2ddf480588d..c1631f45275 100644\n--- a/t/unit-tests/lib-reftable.c\n+++ b/t/unit-tests/lib-reftable.c\n@@ -3,7 +3,7 @@\n #include \"reftable/constants.h\"\n #include \"reftable/writer.h\"\n \n-void t_reftable_set_hash(uint8_t *p, int i, uint32_t id)\n+void t_reftable_set_hash(uint8_t *p, int i, enum reftable_hash id)\n {\n \tmemset(p, (uint8_t)i, hash_size(id));\n }\n@@ -82,7 +82,7 @@ void t_reftable_write_to_buf(struct reftable_buf *buf,\n \t\tsize_t off = i * (opts.block_size ? opts.block_size\n \t\t\t\t\t\t : DEFAULT_BLOCK_SIZE);\n \t\tif (!off)\n-\t\t\toff = header_size(opts.hash_id == GIT_SHA256_FORMAT_ID ? 2 : 1);\n+\t\t\toff = header_size(opts.hash_id == REFTABLE_HASH_SHA256 ? 2 : 1);\n \t\tcheck_char(buf->buf[off], ==, 'r');\n \t}\n \ndiff --git a/t/unit-tests/lib-reftable.h b/t/unit-tests/lib-reftable.h\nindex d4950fed3da..e4c360fa7ee 100644\n--- a/t/unit-tests/lib-reftable.h\n+++ b/t/unit-tests/lib-reftable.h\n@@ -6,7 +6,7 @@\n \n struct reftable_buf;\n \n-void t_reftable_set_hash(uint8_t *p, int i, uint32_t id);\n+void t_reftable_set_hash(uint8_t *p, int i, enum reftable_hash id);\n \n struct reftable_writer *t_reftable_strbuf_writer(struct reftable_buf *buf,\n \t\t\t\t\t\t struct reftable_write_options *opts);\ndiff --git a/t/unit-tests/t-reftable-block.c b/t/unit-tests/t-reftable-block.c\nindex f9af907117b..13e10807dae 100644\n--- a/t/unit-tests/t-reftable-block.c\n+++ b/t/unit-tests/t-reftable-block.c\n@@ -36,7 +36,7 @@ static void t_ref_block_read_write(void)\n \tblock.len = block_size;\n \tblock_source_from_buf(&block.source ,&buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n-\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\t\t\t\theader_off, hash_size(REFTABLE_HASH_SHA1));\n \tcheck(!ret);\n \n \trec.u.ref.refname = (char *) \"\";\n@@ -47,7 +47,7 @@ static void t_ref_block_read_write(void)\n \tfor (i = 0; i < N; i++) {\n \t\trec.u.ref.refname = xstrfmt(\"branch%02\"PRIuMAX, (uintmax_t)i);\n \t\trec.u.ref.value_type = REFTABLE_REF_VAL1;\n-\t\tmemset(rec.u.ref.value.val1, i, GIT_SHA1_RAWSZ);\n+\t\tmemset(rec.u.ref.value.val1, i, REFTABLE_HASH_SIZE_SHA1);\n \n \t\trecs[i] = rec;\n \t\tret = block_writer_add(&bw, &rec);\n@@ -61,7 +61,7 @@ static void t_ref_block_read_write(void)\n \n \tblock_writer_release(&bw);\n \n-\tblock_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ);\n+\tblock_reader_init(&br, &block, header_off, block_size, REFTABLE_HASH_SIZE_SHA1);\n \n \tblock_iter_seek_start(&it, &br);\n \n@@ -72,7 +72,7 @@ static void t_ref_block_read_write(void)\n \t\t\tcheck_int(i, ==, N);\n \t\t\tbreak;\n \t\t}\n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -85,7 +85,7 @@ static void t_ref_block_read_write(void)\n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n \n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \n \t\twant.len--;\n \t\tret = block_iter_seek_key(&it, &br, &want);\n@@ -93,7 +93,7 @@ static void t_ref_block_read_write(void)\n \n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n-\t\tcheck(reftable_record_equal(&recs[10 * (i / 10)], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[10 * (i / 10)], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tblock_reader_release(&br);\n@@ -130,7 +130,7 @@ static void t_log_block_read_write(void)\n \tblock.len = block_size;\n \tblock_source_from_buf(&block.source ,&buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size,\n-\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\t\t\t\theader_off, hash_size(REFTABLE_HASH_SHA1));\n \tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n@@ -150,7 +150,7 @@ static void t_log_block_read_write(void)\n \n \tblock_writer_release(&bw);\n \n-\tblock_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ);\n+\tblock_reader_init(&br, &block, header_off, block_size, REFTABLE_HASH_SIZE_SHA1);\n \n \tblock_iter_seek_start(&it, &br);\n \n@@ -161,7 +161,7 @@ static void t_log_block_read_write(void)\n \t\t\tcheck_int(i, ==, N);\n \t\t\tbreak;\n \t\t}\n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -175,7 +175,7 @@ static void t_log_block_read_write(void)\n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n \n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \n \t\twant.len--;\n \t\tret = block_iter_seek_key(&it, &br, &want);\n@@ -183,7 +183,7 @@ static void t_log_block_read_write(void)\n \n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n-\t\tcheck(reftable_record_equal(&recs[10 * (i / 10)], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[10 * (i / 10)], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tblock_reader_release(&br);\n@@ -220,7 +220,7 @@ static void t_obj_block_read_write(void)\n \tblock.len = block_size;\n \tblock_source_from_buf(&block.source, &buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size,\n-\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\t\t\t\theader_off, hash_size(REFTABLE_HASH_SHA1));\n \tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n@@ -242,7 +242,7 @@ static void t_obj_block_read_write(void)\n \n \tblock_writer_release(&bw);\n \n-\tblock_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ);\n+\tblock_reader_init(&br, &block, header_off, block_size, REFTABLE_HASH_SIZE_SHA1);\n \n \tblock_iter_seek_start(&it, &br);\n \n@@ -253,7 +253,7 @@ static void t_obj_block_read_write(void)\n \t\t\tcheck_int(i, ==, N);\n \t\t\tbreak;\n \t\t}\n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -266,7 +266,7 @@ static void t_obj_block_read_write(void)\n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n \n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tblock_reader_release(&br);\n@@ -304,7 +304,7 @@ static void t_index_block_read_write(void)\n \tblock.len = block_size;\n \tblock_source_from_buf(&block.source, &buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size,\n-\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\t\t\t\theader_off, hash_size(REFTABLE_HASH_SHA1));\n \tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n@@ -326,7 +326,7 @@ static void t_index_block_read_write(void)\n \n \tblock_writer_release(&bw);\n \n-\tblock_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ);\n+\tblock_reader_init(&br, &block, header_off, block_size, REFTABLE_HASH_SIZE_SHA1);\n \n \tblock_iter_seek_start(&it, &br);\n \n@@ -337,7 +337,7 @@ static void t_index_block_read_write(void)\n \t\t\tcheck_int(i, ==, N);\n \t\t\tbreak;\n \t\t}\n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -350,7 +350,7 @@ static void t_index_block_read_write(void)\n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n \n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \n \t\twant.len--;\n \t\tret = block_iter_seek_key(&it, &br, &want);\n@@ -358,7 +358,7 @@ static void t_index_block_read_write(void)\n \n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n-\t\tcheck(reftable_record_equal(&recs[10 * (i / 10)], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[10 * (i / 10)], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tblock_reader_release(&br);\ndiff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c\nindex 484c18251f3..0573d9470a6 100644\n--- a/t/unit-tests/t-reftable-merged.c\n+++ b/t/unit-tests/t-reftable-merged.c\n@@ -42,7 +42,7 @@ merged_table_from_records(struct reftable_ref_record **refs,\n \t\tcheck(!err);\n \t}\n \n-\terr = reftable_merged_table_new(&mt, *readers, n, GIT_SHA1_FORMAT_ID);\n+\terr = reftable_merged_table_new(&mt, *readers, n, REFTABLE_HASH_SHA1);\n \tcheck(!err);\n \treturn mt;\n }\n@@ -91,7 +91,7 @@ static void t_merged_single_record(void)\n \n \terr = reftable_iterator_next_ref(&it, &ref);\n \tcheck(!err);\n-\tcheck(reftable_ref_record_equal(&r2[0], &ref, GIT_SHA1_RAWSZ));\n+\tcheck(reftable_ref_record_equal(&r2[0], &ref, REFTABLE_HASH_SIZE_SHA1));\n \treftable_ref_record_release(&ref);\n \treftable_iterator_destroy(&it);\n \treaders_destroy(readers, 3);\n@@ -168,7 +168,7 @@ static void t_merged_refs(void)\n \tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"a\");\n \tcheck(!err);\n-\tcheck_int(reftable_merged_table_hash_id(mt), ==, GIT_SHA1_FORMAT_ID);\n+\tcheck_int(reftable_merged_table_hash_id(mt), ==, REFTABLE_HASH_SHA1);\n \tcheck_int(reftable_merged_table_min_update_index(mt), ==, 1);\n \tcheck_int(reftable_merged_table_max_update_index(mt), ==, 3);\n \n@@ -186,7 +186,7 @@ static void t_merged_refs(void)\n \tcheck_int(ARRAY_SIZE(want), ==, len);\n \tfor (i = 0; i < len; i++)\n \t\tcheck(reftable_ref_record_equal(want[i], &out[i],\n-\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1));\n \tfor (i = 0; i < len; i++)\n \t\treftable_ref_record_release(&out[i]);\n \treftable_free(out);\n@@ -252,12 +252,12 @@ static void t_merged_seek_multiple_times(void)\n \n \t\terr = reftable_iterator_next_ref(&it, &rec);\n \t\tcheck(!err);\n-\t\terr = reftable_ref_record_equal(&rec, &r1[1], GIT_SHA1_RAWSZ);\n+\t\terr = reftable_ref_record_equal(&rec, &r1[1], REFTABLE_HASH_SIZE_SHA1);\n \t\tcheck(err == 1);\n \n \t\terr = reftable_iterator_next_ref(&it, &rec);\n \t\tcheck(!err);\n-\t\terr = reftable_ref_record_equal(&rec, &r2[1], GIT_SHA1_RAWSZ);\n+\t\terr = reftable_ref_record_equal(&rec, &r2[1], REFTABLE_HASH_SIZE_SHA1);\n \t\tcheck(err == 1);\n \n \t\terr = reftable_iterator_next_ref(&it, &rec);\n@@ -300,7 +300,7 @@ merged_table_from_log_records(struct reftable_log_record **logs,\n \t\tcheck(!err);\n \t}\n \n-\terr = reftable_merged_table_new(&mt, *readers, n, GIT_SHA1_FORMAT_ID);\n+\terr = reftable_merged_table_new(&mt, *readers, n, REFTABLE_HASH_SHA1);\n \tcheck(!err);\n \treturn mt;\n }\n@@ -377,7 +377,7 @@ static void t_merged_logs(void)\n \tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"a\");\n \tcheck(!err);\n-\tcheck_int(reftable_merged_table_hash_id(mt), ==, GIT_SHA1_FORMAT_ID);\n+\tcheck_int(reftable_merged_table_hash_id(mt), ==, REFTABLE_HASH_SHA1);\n \tcheck_int(reftable_merged_table_min_update_index(mt), ==, 1);\n \tcheck_int(reftable_merged_table_max_update_index(mt), ==, 3);\n \n@@ -395,7 +395,7 @@ static void t_merged_logs(void)\n \tcheck_int(ARRAY_SIZE(want), ==, len);\n \tfor (i = 0; i < len; i++)\n \t\tcheck(reftable_log_record_equal(want[i], &out[i],\n-\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1));\n \n \terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n \tcheck(!err);\n@@ -404,7 +404,7 @@ static void t_merged_logs(void)\n \treftable_log_record_release(&out[0]);\n \terr = reftable_iterator_next_log(&it, &out[0]);\n \tcheck(!err);\n-\tcheck(reftable_log_record_equal(&out[0], &r3[0], GIT_SHA1_RAWSZ));\n+\tcheck(reftable_log_record_equal(&out[0], &r3[0], REFTABLE_HASH_SIZE_SHA1));\n \treftable_iterator_destroy(&it);\n \n \tfor (i = 0; i < len; i++)\n@@ -448,11 +448,11 @@ static void t_default_write_opts(void)\n \tcheck(!err);\n \n \thash_id = reftable_reader_hash_id(rd);\n-\tcheck_int(hash_id, ==, GIT_SHA1_FORMAT_ID);\n+\tcheck_int(hash_id, ==, REFTABLE_HASH_SHA1);\n \n-\terr = reftable_merged_table_new(&merged, &rd, 1, GIT_SHA256_FORMAT_ID);\n+\terr = reftable_merged_table_new(&merged, &rd, 1, REFTABLE_HASH_SHA256);\n \tcheck_int(err, ==, REFTABLE_FORMAT_ERROR);\n-\terr = reftable_merged_table_new(&merged, &rd, 1, GIT_SHA1_FORMAT_ID);\n+\terr = reftable_merged_table_new(&merged, &rd, 1, REFTABLE_HASH_SHA1);\n \tcheck(!err);\n \n \treftable_reader_decref(rd);\ndiff --git a/t/unit-tests/t-reftable-pq.c b/t/unit-tests/t-reftable-pq.c\nindex ada4c19f18a..272da05bea6 100644\n--- a/t/unit-tests/t-reftable-pq.c\n+++ b/t/unit-tests/t-reftable-pq.c\n@@ -132,7 +132,7 @@ static void t_merged_iter_pqueue_top(void)\n \n \t\tmerged_iter_pqueue_check(&pq);\n \t\tcheck(pq_entry_equal(&top, &e));\n-\t\tcheck(reftable_record_equal(top.rec, &recs[i], GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(top.rec, &recs[i], REFTABLE_HASH_SIZE_SHA1));\n \t\tfor (size_t j = 0; i < pq.len; j++) {\n \t\t\tcheck(pq_less(&top, &pq.heap[j]));\n \t\t\tcheck_int(top.index, >, j);\ndiff --git a/t/unit-tests/t-reftable-reader.c b/t/unit-tests/t-reftable-reader.c\nindex 19cb53b6415..546df6005e4 100644\n--- a/t/unit-tests/t-reftable-reader.c\n+++ b/t/unit-tests/t-reftable-reader.c\n@@ -31,7 +31,7 @@ static int t_reader_seek_once(void)\n \tret = reftable_iterator_next_ref(&it, &ref);\n \tcheck(!ret);\n \n-\tret = reftable_ref_record_equal(&ref, &records[0], GIT_SHA1_RAWSZ);\n+\tret = reftable_ref_record_equal(&ref, &records[0], REFTABLE_HASH_SIZE_SHA1);\n \tcheck_int(ret, ==, 1);\n \n \tret = reftable_iterator_next_ref(&it, &ref);\n@@ -74,7 +74,7 @@ static int t_reader_reseek(void)\n \t\tret = reftable_iterator_next_ref(&it, &ref);\n \t\tcheck(!ret);\n \n-\t\tret = reftable_ref_record_equal(&ref, &records[0], GIT_SHA1_RAWSZ);\n+\t\tret = reftable_ref_record_equal(&ref, &records[0], REFTABLE_HASH_SIZE_SHA1);\n \t\tcheck_int(ret, ==, 1);\n \n \t\tret = reftable_iterator_next_ref(&it, &ref);\ndiff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c\nindex d279b86df0a..57896922eb1 100644\n--- a/t/unit-tests/t-reftable-readwrite.c\n+++ b/t/unit-tests/t-reftable-readwrite.c\n@@ -41,7 +41,7 @@ static void t_buffer(void)\n }\n \n static void write_table(char ***names, struct reftable_buf *buf, int N,\n-\t\t\tint block_size, uint32_t hash_id)\n+\t\t\tint block_size, enum reftable_hash hash_id)\n {\n \tstruct reftable_write_options opts = {\n \t\t.block_size = block_size,\n@@ -62,7 +62,7 @@ static void write_table(char ***names, struct reftable_buf *buf, int N,\n \t\trefs[i].refname = (*names)[i] = xstrfmt(\"refs/heads/branch%02d\", i);\n \t\trefs[i].update_index = update_index;\n \t\trefs[i].value_type = REFTABLE_REF_VAL1;\n-\t\tt_reftable_set_hash(refs[i].value.val1, i, GIT_SHA1_FORMAT_ID);\n+\t\tt_reftable_set_hash(refs[i].value.val1, i, REFTABLE_HASH_SHA1);\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -70,7 +70,7 @@ static void write_table(char ***names, struct reftable_buf *buf, int N,\n \t\tlogs[i].update_index = update_index;\n \t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n \t\tt_reftable_set_hash(logs[i].value.update.new_hash, i,\n-\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t REFTABLE_HASH_SHA1);\n \t\tlogs[i].value.update.message = (char *) \"message\";\n \t}\n \n@@ -104,7 +104,7 @@ static void t_log_buffer_size(void)\n \t/* This tests buffer extension for log compression. Must use a random\n \t hash, to ensure that the compressed part is larger than the original.\n \t*/\n-\tfor (i = 0; i < GIT_SHA1_RAWSZ; i++) {\n+\tfor (i = 0; i < REFTABLE_HASH_SIZE_SHA1; i++) {\n \t\tlog.value.update.old_hash[i] = (uint8_t)(git_rand() % 256);\n \t\tlog.value.update.new_hash[i] = (uint8_t)(git_rand() % 256);\n \t}\n@@ -191,9 +191,9 @@ static void t_log_write_read(void)\n \t\tlog.update_index = i;\n \t\tlog.value_type = REFTABLE_LOG_UPDATE;\n \t\tt_reftable_set_hash(log.value.update.old_hash, i,\n-\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t REFTABLE_HASH_SHA1);\n \t\tt_reftable_set_hash(log.value.update.new_hash, i + 1,\n-\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t REFTABLE_HASH_SHA1);\n \n \t\terr = reftable_writer_add_log(w, &log);\n \t\tcheck(!err);\n@@ -326,7 +326,7 @@ static void t_table_read_write_sequential(void)\n \tint err = 0;\n \tint j = 0;\n \n-\twrite_table(&names, &buf, N, 256, GIT_SHA1_FORMAT_ID);\n+\twrite_table(&names, &buf, N, 256, REFTABLE_HASH_SHA1);\n \n \tblock_source_from_buf(&source, &buf);\n \n@@ -361,7 +361,7 @@ static void t_table_write_small_table(void)\n \tchar **names;\n \tstruct reftable_buf buf = REFTABLE_BUF_INIT;\n \tint N = 1;\n-\twrite_table(&names, &buf, N, 4096, GIT_SHA1_FORMAT_ID);\n+\twrite_table(&names, &buf, N, 4096, REFTABLE_HASH_SHA1);\n \tcheck_int(buf.len, <, 200);\n \treftable_buf_release(&buf);\n \tfree_names(names);\n@@ -378,7 +378,7 @@ static void t_table_read_api(void)\n \tstruct reftable_log_record log = { 0 };\n \tstruct reftable_iterator it = { 0 };\n \n-\twrite_table(&names, &buf, N, 256, GIT_SHA1_FORMAT_ID);\n+\twrite_table(&names, &buf, N, 256, REFTABLE_HASH_SHA1);\n \n \tblock_source_from_buf(&source, &buf);\n \n@@ -400,7 +400,7 @@ static void t_table_read_api(void)\n \treftable_buf_release(&buf);\n }\n \n-static void t_table_read_write_seek(int index, int hash_id)\n+static void t_table_read_write_seek(int index, enum reftable_hash hash_id)\n {\n \tchar **names;\n \tstruct reftable_buf buf = REFTABLE_BUF_INIT;\n@@ -467,24 +467,24 @@ static void t_table_read_write_seek(int index, int hash_id)\n \n static void t_table_read_write_seek_linear(void)\n {\n-\tt_table_read_write_seek(0, GIT_SHA1_FORMAT_ID);\n+\tt_table_read_write_seek(0, REFTABLE_HASH_SHA1);\n }\n \n static void t_table_read_write_seek_linear_sha256(void)\n {\n-\tt_table_read_write_seek(0, GIT_SHA256_FORMAT_ID);\n+\tt_table_read_write_seek(0, REFTABLE_HASH_SHA256);\n }\n \n static void t_table_read_write_seek_index(void)\n {\n-\tt_table_read_write_seek(1, GIT_SHA1_FORMAT_ID);\n+\tt_table_read_write_seek(1, REFTABLE_HASH_SHA1);\n }\n \n static void t_table_refs_for(int indexed)\n {\n \tchar **want_names;\n \tint want_names_len = 0;\n-\tuint8_t want_hash[GIT_SHA1_RAWSZ];\n+\tuint8_t want_hash[REFTABLE_HASH_SIZE_SHA1];\n \n \tstruct reftable_write_options opts = {\n \t\t.block_size = 256,\n@@ -500,10 +500,10 @@ static void t_table_refs_for(int indexed)\n \twant_names = reftable_calloc(N + 1, sizeof(*want_names));\n \tcheck(want_names != NULL);\n \n-\tt_reftable_set_hash(want_hash, 4, GIT_SHA1_FORMAT_ID);\n+\tt_reftable_set_hash(want_hash, 4, REFTABLE_HASH_SHA1);\n \n \tfor (i = 0; i < N; i++) {\n-\t\tuint8_t hash[GIT_SHA1_RAWSZ];\n+\t\tuint8_t hash[REFTABLE_HASH_SIZE_SHA1];\n \t\tchar fill[51] = { 0 };\n \t\tchar name[100];\n \t\tstruct reftable_ref_record ref = { 0 };\n@@ -517,9 +517,9 @@ static void t_table_refs_for(int indexed)\n \n \t\tref.value_type = REFTABLE_REF_VAL2;\n \t\tt_reftable_set_hash(ref.value.val2.value, i / 4,\n-\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t REFTABLE_HASH_SHA1);\n \t\tt_reftable_set_hash(ref.value.val2.target_value, 3 + i / 4,\n-\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t REFTABLE_HASH_SHA1);\n \n \t\t/* 80 bytes / entry, so 3 entries per block. Yields 17\n \t\t */\n@@ -527,8 +527,8 @@ static void t_table_refs_for(int indexed)\n \t\tn = reftable_writer_add_ref(w, &ref);\n \t\tcheck_int(n, ==, 0);\n \n-\t\tif (!memcmp(ref.value.val2.value, want_hash, GIT_SHA1_RAWSZ) ||\n-\t\t !memcmp(ref.value.val2.target_value, want_hash, GIT_SHA1_RAWSZ))\n+\t\tif (!memcmp(ref.value.val2.value, want_hash, REFTABLE_HASH_SIZE_SHA1) ||\n+\t\t !memcmp(ref.value.val2.target_value, want_hash, REFTABLE_HASH_SIZE_SHA1))\n \t\t\twant_names[want_names_len++] = xstrdup(name);\n \t}\n \ndiff --git a/t/unit-tests/t-reftable-record.c b/t/unit-tests/t-reftable-record.c\nindex eb98bf2da91..42bc64cec87 100644\n--- a/t/unit-tests/t-reftable-record.c\n+++ b/t/unit-tests/t-reftable-record.c\n@@ -7,6 +7,7 @@\n */\n \n #include \"test-lib.h\"\n+#include \"reftable/basics.h\"\n #include \"reftable/constants.h\"\n #include \"reftable/record.h\"\n \n@@ -17,10 +18,10 @@ static void t_copy(struct reftable_record *rec)\n \n \ttyp = reftable_record_type(rec);\n \treftable_record_init(©, typ);\n-\treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n+\treftable_record_copy_from(©, rec, REFTABLE_HASH_SIZE_SHA1);\n \t/* do it twice to catch memory leaks */\n-\treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n-\tcheck(reftable_record_equal(rec, ©, GIT_SHA1_RAWSZ));\n+\treftable_record_copy_from(©, rec, REFTABLE_HASH_SIZE_SHA1);\n+\tcheck(reftable_record_equal(rec, ©, REFTABLE_HASH_SIZE_SHA1));\n \n \treftable_record_release(©);\n }\n@@ -59,7 +60,7 @@ static void t_varint_roundtrip(void)\n \n static void set_hash(uint8_t *h, int j)\n {\n-\tfor (int i = 0; i < hash_size(GIT_SHA1_FORMAT_ID); i++)\n+\tfor (int i = 0; i < hash_size(REFTABLE_HASH_SHA1); i++)\n \t\th[i] = (j >> i) & 0xff;\n }\n \n@@ -84,14 +85,14 @@ static void t_reftable_ref_record_comparison(void)\n \t\t},\n \t};\n \n-\tcheck(!reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n \n-\tcheck(!reftable_record_equal(&in[1], &in[2], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n \tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n \n \tin[1].u.ref.value_type = in[0].u.ref.value_type;\n-\tcheck(reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n }\n \n@@ -155,15 +156,15 @@ static void t_reftable_ref_record_roundtrip(void)\n \t\tcheck_int(reftable_record_is_deletion(&in), ==, i == REFTABLE_REF_DELETION);\n \n \t\treftable_record_key(&in, &key);\n-\t\tn = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);\n+\t\tn = reftable_record_encode(&in, dest, REFTABLE_HASH_SIZE_SHA1);\n \t\tcheck_int(n, >, 0);\n \n \t\t/* decode into a non-zero reftable_record to test for leaks. */\n-\t\tm = reftable_record_decode(&out, key, i, dest, GIT_SHA1_RAWSZ, &scratch);\n+\t\tm = reftable_record_decode(&out, key, i, dest, REFTABLE_HASH_SIZE_SHA1, &scratch);\n \t\tcheck_int(n, ==, m);\n \n \t\tcheck(reftable_ref_record_equal(&in.u.ref, &out.u.ref,\n-\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_record_release(&in);\n \n \t\treftable_buf_release(&key);\n@@ -193,15 +194,15 @@ static void t_reftable_log_record_comparison(void)\n \t\t},\n \t};\n \n-\tcheck(!reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n-\tcheck(!reftable_record_equal(&in[1], &in[2], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n+\tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n \tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n \t/* comparison should be reversed for equal keys, because\n \t * comparison is now performed on the basis of update indices */\n \tcheck_int(reftable_record_cmp(&in[0], &in[1]), <, 0);\n \n \tin[1].u.log.update_index = in[0].u.log.update_index;\n-\tcheck(reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n }\n \n@@ -303,15 +304,15 @@ static void t_reftable_log_record_roundtrip(void)\n \n \t\treftable_record_key(&rec, &key);\n \n-\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\t\tn = reftable_record_encode(&rec, dest, REFTABLE_HASH_SIZE_SHA1);\n \t\tcheck_int(n, >=, 0);\n \t\tvaltype = reftable_record_val_type(&rec);\n \t\tm = reftable_record_decode(&out, key, valtype, dest,\n-\t\t\t\t\t GIT_SHA1_RAWSZ, &scratch);\n+\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1, &scratch);\n \t\tcheck_int(n, ==, m);\n \n \t\tcheck(reftable_log_record_equal(&in[i], &out.u.log,\n-\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_log_record_release(&in[i]);\n \t\treftable_buf_release(&key);\n \t\treftable_record_release(&out);\n@@ -380,20 +381,20 @@ static void t_reftable_obj_record_comparison(void)\n \t\t},\n \t};\n \n-\tcheck(!reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n \n-\tcheck(!reftable_record_equal(&in[1], &in[2], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n \tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n \n \tin[1].u.obj.offset_len = in[0].u.obj.offset_len;\n-\tcheck(reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n }\n \n static void t_reftable_obj_record_roundtrip(void)\n {\n-\tuint8_t testHash1[GIT_SHA1_RAWSZ] = { 1, 2, 3, 4, 0 };\n+\tuint8_t testHash1[REFTABLE_HASH_SIZE_SHA1] = { 1, 2, 3, 4, 0 };\n \tuint64_t till9[] = { 1, 2, 3, 4, 500, 600, 700, 800, 9000 };\n \tstruct reftable_obj_record recs[3] = {\n \t\t{\n@@ -435,14 +436,14 @@ static void t_reftable_obj_record_roundtrip(void)\n \t\tcheck(!reftable_record_is_deletion(&in));\n \t\tt_copy(&in);\n \t\treftable_record_key(&in, &key);\n-\t\tn = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);\n+\t\tn = reftable_record_encode(&in, dest, REFTABLE_HASH_SIZE_SHA1);\n \t\tcheck_int(n, >, 0);\n \t\textra = reftable_record_val_type(&in);\n \t\tm = reftable_record_decode(&out, key, extra, dest,\n-\t\t\t\t\t GIT_SHA1_RAWSZ, &scratch);\n+\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1, &scratch);\n \t\tcheck_int(n, ==, m);\n \n-\t\tcheck(reftable_record_equal(&in, &out, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&in, &out, REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_buf_release(&key);\n \t\treftable_record_release(&out);\n \t}\n@@ -473,14 +474,14 @@ static void t_reftable_index_record_comparison(void)\n \tcheck(!reftable_buf_addstr(&in[1].u.idx.last_key, \"refs/heads/master\"));\n \tcheck(!reftable_buf_addstr(&in[2].u.idx.last_key, \"refs/heads/branch\"));\n \n-\tcheck(!reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n \n-\tcheck(!reftable_record_equal(&in[1], &in[2], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n \tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n \n \tin[1].u.idx.offset = in[0].u.idx.offset;\n-\tcheck(reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n \n \tfor (size_t i = 0; i < ARRAY_SIZE(in); i++)\n@@ -516,15 +517,15 @@ static void t_reftable_index_record_roundtrip(void)\n \n \tcheck(!reftable_record_is_deletion(&in));\n \tcheck(!reftable_buf_cmp(&key, &in.u.idx.last_key));\n-\tn = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);\n+\tn = reftable_record_encode(&in, dest, REFTABLE_HASH_SIZE_SHA1);\n \tcheck_int(n, >, 0);\n \n \textra = reftable_record_val_type(&in);\n-\tm = reftable_record_decode(&out, key, extra, dest, GIT_SHA1_RAWSZ,\n+\tm = reftable_record_decode(&out, key, extra, dest, REFTABLE_HASH_SIZE_SHA1,\n \t\t\t\t &scratch);\n \tcheck_int(m, ==, n);\n \n-\tcheck(reftable_record_equal(&in, &out, GIT_SHA1_RAWSZ));\n+\tcheck(reftable_record_equal(&in, &out, REFTABLE_HASH_SIZE_SHA1));\n \n \treftable_record_release(&out);\n \treftable_buf_release(&key);\ndiff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c\nindex 1b4363a58fc..13fd8d8f941 100644\n--- a/t/unit-tests/t-reftable-stack.c\n+++ b/t/unit-tests/t-reftable-stack.c\n@@ -121,7 +121,7 @@ static void write_n_ref_tables(struct reftable_stack *st,\n \n \t\tsnprintf(buf, sizeof(buf), \"refs/heads/branch-%04\"PRIuMAX, (uintmax_t)i);\n \t\tref.refname = buf;\n-\t\tt_reftable_set_hash(ref.value.val1, i, GIT_SHA1_FORMAT_ID);\n+\t\tt_reftable_set_hash(ref.value.val1, i, REFTABLE_HASH_SHA1);\n \n \t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n \t\tcheck(!err);\n@@ -169,7 +169,7 @@ static void t_reftable_stack_add_one(void)\n \n \terr = reftable_stack_read_ref(st, ref.refname, &dest);\n \tcheck(!err);\n-\tcheck(reftable_ref_record_equal(&ref, &dest, GIT_SHA1_RAWSZ));\n+\tcheck(reftable_ref_record_equal(&ref, &dest, REFTABLE_HASH_SIZE_SHA1));\n \tcheck_int(st->readers_len, >, 0);\n \n #ifndef GIT_WINDOWS_NATIVE\n@@ -280,7 +280,7 @@ static void t_reftable_stack_transaction_api(void)\n \terr = reftable_stack_read_ref(st, ref.refname, &dest);\n \tcheck(!err);\n \tcheck_int(REFTABLE_REF_SYMREF, ==, dest.value_type);\n-\tcheck(reftable_ref_record_equal(&ref, &dest, GIT_SHA1_RAWSZ));\n+\tcheck(reftable_ref_record_equal(&ref, &dest, REFTABLE_HASH_SIZE_SHA1));\n \n \treftable_ref_record_release(&dest);\n \treftable_stack_destroy(st);\n@@ -340,7 +340,7 @@ static void t_reftable_stack_transaction_with_reload(void)\n \tfor (size_t i = 0; i < ARRAY_SIZE(refs); i++) {\n \t\terr = reftable_stack_read_ref(st2, refs[i].refname, &ref);\n \t\tcheck(!err);\n-\t\tcheck(reftable_ref_record_equal(&refs[i], &ref, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_ref_record_equal(&refs[i], &ref, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \treftable_ref_record_release(&ref);\n@@ -530,13 +530,13 @@ static void t_reftable_stack_add(void)\n \t\trefs[i].refname = xstrdup(buf);\n \t\trefs[i].update_index = i + 1;\n \t\trefs[i].value_type = REFTABLE_REF_VAL1;\n-\t\tt_reftable_set_hash(refs[i].value.val1, i, GIT_SHA1_FORMAT_ID);\n+\t\tt_reftable_set_hash(refs[i].value.val1, i, REFTABLE_HASH_SHA1);\n \n \t\tlogs[i].refname = xstrdup(buf);\n \t\tlogs[i].update_index = N + i + 1;\n \t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n \t\tlogs[i].value.update.email = xstrdup(\"identity@invalid\");\n-\t\tt_reftable_set_hash(logs[i].value.update.new_hash, i, GIT_SHA1_FORMAT_ID);\n+\t\tt_reftable_set_hash(logs[i].value.update.new_hash, i, REFTABLE_HASH_SHA1);\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -562,7 +562,7 @@ static void t_reftable_stack_add(void)\n \t\tint err = reftable_stack_read_ref(st, refs[i].refname, &dest);\n \t\tcheck(!err);\n \t\tcheck(reftable_ref_record_equal(&dest, refs + i,\n-\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_ref_record_release(&dest);\n \t}\n \n@@ -571,7 +571,7 @@ static void t_reftable_stack_add(void)\n \t\tint err = reftable_stack_read_log(st, refs[i].refname, &dest);\n \t\tcheck(!err);\n \t\tcheck(reftable_log_record_equal(&dest, logs + i,\n-\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_log_record_release(&dest);\n \t}\n \n@@ -622,14 +622,14 @@ static void t_reftable_stack_iterator(void)\n \t\trefs[i].refname = xstrfmt(\"branch%02\"PRIuMAX, (uintmax_t)i);\n \t\trefs[i].update_index = i + 1;\n \t\trefs[i].value_type = REFTABLE_REF_VAL1;\n-\t\tt_reftable_set_hash(refs[i].value.val1, i, GIT_SHA1_FORMAT_ID);\n+\t\tt_reftable_set_hash(refs[i].value.val1, i, REFTABLE_HASH_SHA1);\n \n \t\tlogs[i].refname = xstrfmt(\"branch%02\"PRIuMAX, (uintmax_t)i);\n \t\tlogs[i].update_index = i + 1;\n \t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n \t\tlogs[i].value.update.email = xstrdup(\"johndoe@invalid\");\n \t\tlogs[i].value.update.message = xstrdup(\"commit\\n\");\n-\t\tt_reftable_set_hash(logs[i].value.update.new_hash, i, GIT_SHA1_FORMAT_ID);\n+\t\tt_reftable_set_hash(logs[i].value.update.new_hash, i, REFTABLE_HASH_SHA1);\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -656,7 +656,7 @@ static void t_reftable_stack_iterator(void)\n \t\tif (err > 0)\n \t\t\tbreak;\n \t\tcheck(!err);\n-\t\tcheck(reftable_ref_record_equal(&ref, &refs[i], GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_ref_record_equal(&ref, &refs[i], REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_ref_record_release(&ref);\n \t}\n \tcheck_int(i, ==, N);\n@@ -674,7 +674,7 @@ static void t_reftable_stack_iterator(void)\n \t\tif (err > 0)\n \t\t\tbreak;\n \t\tcheck(!err);\n-\t\tcheck(reftable_log_record_equal(&log, &logs[i], GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_log_record_equal(&log, &logs[i], REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_log_record_release(&log);\n \t}\n \tcheck_int(i, ==, N);\n@@ -767,7 +767,7 @@ static void t_reftable_stack_tombstone(void)\n \t\tif (i % 2 == 0) {\n \t\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n \t\t\tt_reftable_set_hash(refs[i].value.val1, i,\n-\t\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t\t REFTABLE_HASH_SHA1);\n \t\t}\n \n \t\tlogs[i].refname = xstrdup(buf);\n@@ -776,7 +776,7 @@ static void t_reftable_stack_tombstone(void)\n \t\tif (i % 2 == 0) {\n \t\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n \t\t\tt_reftable_set_hash(logs[i].value.update.new_hash, i,\n-\t\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t\t REFTABLE_HASH_SHA1);\n \t\t\tlogs[i].value.update.email =\n \t\t\t\txstrdup(\"identity@invalid\");\n \t\t}\n@@ -836,7 +836,7 @@ static void t_reftable_stack_hash_id(void)\n \t\t.value.symref = (char *) \"target\",\n \t\t.update_index = 1,\n \t};\n-\tstruct reftable_write_options opts32 = { .hash_id = GIT_SHA256_FORMAT_ID };\n+\tstruct reftable_write_options opts32 = { .hash_id = REFTABLE_HASH_SHA256 };\n \tstruct reftable_stack *st32 = NULL;\n \tstruct reftable_write_options opts_default = { 0 };\n \tstruct reftable_stack *st_default = NULL;\n@@ -859,7 +859,7 @@ static void t_reftable_stack_hash_id(void)\n \terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n \tcheck(!err);\n \n-\tcheck(reftable_ref_record_equal(&ref, &dest, GIT_SHA1_RAWSZ));\n+\tcheck(reftable_ref_record_equal(&ref, &dest, REFTABLE_HASH_SIZE_SHA1));\n \treftable_ref_record_release(&dest);\n \treftable_stack_destroy(st);\n \treftable_stack_destroy(st_default);\n@@ -909,7 +909,7 @@ static void t_reflog_expire(void)\n \t\tlogs[i].value.update.time = i;\n \t\tlogs[i].value.update.email = xstrdup(\"identity@invalid\");\n \t\tt_reftable_set_hash(logs[i].value.update.new_hash, i,\n-\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t REFTABLE_HASH_SHA1);\n \t}\n \n \tfor (i = 1; i <= N; i++) {\n-- \n2.47.0.118.gfd3785337b.dirty\n\n"},{"id":"505915","messageId":"86269fc4fcad9a97709aa0d080c4c077a85ca667.1729677003.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1729677003.git.ps@pks.im","subject":"[PATCH 4/7] reftable/stack: stop using `fsync_component()` directly","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-23T09:56:04Z","receivedAt":"2024-10-23T09:56:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're executing `fsync_component()` directly in the reftable library so\nthat we can fsync data to disk depending on \"core.fsync\". But as we're\nin the process of converting the reftable library to become standalone\nwe cannot use that function in the library anymore.\n\nRefactor the code such that users of the library can inject a custom\nfsync function via the write options. This allows us to get rid of the\ndependency on \"write-or-die.h\".\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 7 ++++++\n reftable/reftable-writer.h | 6 +++++\n reftable/stack.c | 49 +++++++++++++++++++++++++-------------\n 3 files changed, 45 insertions(+), 17 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 7d86d920970..2e774176eda 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -24,6 +24,7 @@\n #include \"../setup.h\"\n #include \"../strmap.h\"\n #include \"../trace2.h\"\n+#include \"../write-or-die.h\"\n #include \"parse.h\"\n #include \"refs-internal.h\"\n \n@@ -273,6 +274,11 @@ static int reftable_be_config(const char *var, const char *value,\n \treturn 0;\n }\n \n+static int reftable_be_fsync(int fd)\n+{\n+\treturn fsync_component(FSYNC_COMPONENT_REFERENCE, fd);\n+}\n+\n static struct ref_store *reftable_be_init(struct repository *repo,\n \t\t\t\t\t const char *gitdir,\n \t\t\t\t\t unsigned int store_flags)\n@@ -304,6 +310,7 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \trefs->write_options.disable_auto_compact =\n \t\t!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1);\n \trefs->write_options.lock_timeout_ms = 100;\n+\trefs->write_options.fsync = reftable_be_fsync;\n \n \tgit_config(reftable_be_config, &refs->write_options);\n \ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 211860d08a4..c85ef5a5bd1 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -62,6 +62,12 @@ struct reftable_write_options {\n \t * negative value will cause us to block indefinitely.\n \t */\n \tlong lock_timeout_ms;\n+\n+\t/*\n+\t * Optional callback used to fsync files to disk. Falls back to using\n+\t * fsync(3P) when unset.\n+\t */\n+\tint (*fsync)(int fd);\n };\n \n /* reftable_block_stats holds statistics for a single block type */\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 9ae716ff375..df4f3237007 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -8,7 +8,6 @@ license that can be found in the LICENSE file or at\n \n #include \"stack.h\"\n \n-#include \"../write-or-die.h\"\n #include \"system.h\"\n #include \"constants.h\"\n #include \"merged.h\"\n@@ -43,17 +42,28 @@ static int stack_filename(struct reftable_buf *dest, struct reftable_stack *st,\n \treturn 0;\n }\n \n-static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz)\n+static int stack_fsync(struct reftable_stack *st, int fd)\n {\n-\tint *fdp = (int *)arg;\n-\treturn write_in_full(*fdp, data, sz);\n+\tif (st->opts.fsync)\n+\t\treturn st->opts.fsync(fd);\n+\treturn fsync(fd);\n }\n \n-static int reftable_fd_flush(void *arg)\n+struct fd_writer {\n+\tstruct reftable_stack *stack;\n+\tint fd;\n+};\n+\n+static ssize_t fd_writer_write(void *arg, const void *data, size_t sz)\n {\n-\tint *fdp = (int *)arg;\n+\tstruct fd_writer *writer = arg;\n+\treturn write_in_full(writer->fd, data, sz);\n+}\n \n-\treturn fsync_component(FSYNC_COMPONENT_REFERENCE, *fdp);\n+static int fd_writer_flush(void *arg)\n+{\n+\tstruct fd_writer *writer = arg;\n+\treturn stack_fsync(writer->stack, writer->fd);\n }\n \n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n@@ -765,7 +775,7 @@ int reftable_addition_commit(struct reftable_addition *add)\n \t\tgoto done;\n \t}\n \n-\terr = fsync_component(FSYNC_COMPONENT_REFERENCE, lock_file_fd);\n+\terr = stack_fsync(add->stack, lock_file_fd);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n@@ -858,8 +868,10 @@ int reftable_addition_add(struct reftable_addition *add,\n \tstruct reftable_buf next_name = REFTABLE_BUF_INIT;\n \tstruct reftable_writer *wr = NULL;\n \tstruct tempfile *tab_file = NULL;\n+\tstruct fd_writer writer = {\n+\t\t.stack = add->stack,\n+\t};\n \tint err = 0;\n-\tint tab_fd;\n \n \treftable_buf_reset(&next_name);\n \n@@ -887,10 +899,10 @@ int reftable_addition_add(struct reftable_addition *add,\n \t\t\tgoto done;\n \t\t}\n \t}\n-\ttab_fd = get_tempfile_fd(tab_file);\n \n-\terr = reftable_writer_new(&wr, reftable_fd_write, reftable_fd_flush,\n-\t\t\t\t &tab_fd, &add->stack->opts);\n+\twriter.fd = get_tempfile_fd(tab_file);\n+\terr = reftable_writer_new(&wr, fd_writer_write, fd_writer_flush,\n+\t\t\t\t &writer, &add->stack->opts);\n \tif (err < 0)\n \t\tgoto done;\n \n@@ -973,8 +985,11 @@ static int stack_compact_locked(struct reftable_stack *st,\n \tstruct reftable_buf next_name = REFTABLE_BUF_INIT;\n \tstruct reftable_buf tab_file_path = REFTABLE_BUF_INIT;\n \tstruct reftable_writer *wr = NULL;\n+\tstruct fd_writer writer= {\n+\t\t.stack = st,\n+\t};\n \tstruct tempfile *tab_file;\n-\tint tab_fd, err = 0;\n+\tint err = 0;\n \n \terr = format_name(&next_name, reftable_reader_min_update_index(st->readers[first]),\n \t\t\t reftable_reader_max_update_index(st->readers[last]));\n@@ -994,7 +1009,6 @@ static int stack_compact_locked(struct reftable_stack *st,\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n-\ttab_fd = get_tempfile_fd(tab_file);\n \n \tif (st->opts.default_permissions &&\n \t chmod(get_tempfile_path(tab_file), st->opts.default_permissions) < 0) {\n@@ -1002,8 +1016,9 @@ static int stack_compact_locked(struct reftable_stack *st,\n \t\tgoto done;\n \t}\n \n-\terr = reftable_writer_new(&wr, reftable_fd_write, reftable_fd_flush,\n-\t\t\t\t &tab_fd, &st->opts);\n+\twriter.fd = get_tempfile_fd(tab_file);\n+\terr = reftable_writer_new(&wr, fd_writer_write, fd_writer_flush,\n+\t\t\t\t &writer, &st->opts);\n \tif (err < 0)\n \t\tgoto done;\n \n@@ -1460,7 +1475,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\tgoto done;\n \t}\n \n-\terr = fsync_component(FSYNC_COMPONENT_REFERENCE, get_lock_file_fd(&tables_list_lock));\n+\terr = stack_fsync(st, get_lock_file_fd(&tables_list_lock));\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tunlink(new_table_path.buf);\n-- \n2.47.0.118.gfd3785337b.dirty\n\n"},{"id":"505916","messageId":"aca1995556058b840b984dd1c3548214d791e3a6.1729677003.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1729677003.git.ps@pks.im","subject":"[PATCH 5/7] reftable/system: provide thin wrapper for tempfile subsystem","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-23T09:56:07Z","receivedAt":"2024-10-23T09:56:10Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We use the tempfile subsystem to write temporary tables, but given that\nwe're in the process of converting the reftable library to become\nstandalone we cannot use this subsystem directly anymore. While we could\nin theory convert the code to use mkstemp(3p) instead, we'd lose access\nto our infrastructure that automatically prunes tempfiles via atexit(3p)\nor signal handlers.\n\nProvide a thin wrapper for the tempfile subsystem instead. Like this,\nthe compatibility shim is fully self-contained in \"reftable/system.c\".\nDownstream users of the reftable library would have to implement their\nown tempfile shims by replacing \"system.c\" with a custom version.\n\nSigned-off-by: Patrick Steinhardt \n---\n Makefile | 1 +\n reftable/stack.c | 57 +++++++++++++++++++----------------------------\n reftable/system.c | 49 ++++++++++++++++++++++++++++++++++++++++\n reftable/system.h | 41 +++++++++++++++++++++++++++++++++-\n 4 files changed, 113 insertions(+), 35 deletions(-)\n create mode 100644 reftable/system.c\n\ndiff --git a/Makefile b/Makefile\nindex feeed6f9321..50a79ad83fc 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2722,6 +2722,7 @@ REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/stack.o\n+REFTABLE_OBJS += reftable/system.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n \ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex df4f3237007..67b2117a112 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -16,7 +16,6 @@ license that can be found in the LICENSE file or at\n #include \"reftable-record.h\"\n #include \"reftable-merged.h\"\n #include \"writer.h\"\n-#include \"tempfile.h\"\n \n static int stack_try_add(struct reftable_stack *st,\n \t\t\t int (*write_table)(struct reftable_writer *wr,\n@@ -867,7 +866,7 @@ int reftable_addition_add(struct reftable_addition *add,\n \tstruct reftable_buf tab_file_name = REFTABLE_BUF_INIT;\n \tstruct reftable_buf next_name = REFTABLE_BUF_INIT;\n \tstruct reftable_writer *wr = NULL;\n-\tstruct tempfile *tab_file = NULL;\n+\tstruct reftable_tmpfile tab_file = REFTABLE_TMPFILE_INIT;\n \tstruct fd_writer writer = {\n \t\t.stack = add->stack,\n \t};\n@@ -887,20 +886,18 @@ int reftable_addition_add(struct reftable_addition *add,\n \tif (err < 0)\n \t\tgoto done;\n \n-\ttab_file = mks_tempfile(temp_tab_file_name.buf);\n-\tif (!tab_file) {\n-\t\terr = REFTABLE_IO_ERROR;\n+\terr = tmpfile_from_pattern(&tab_file, temp_tab_file_name.buf);\n+\tif (err < 0)\n \t\tgoto done;\n-\t}\n \tif (add->stack->opts.default_permissions) {\n-\t\tif (chmod(get_tempfile_path(tab_file),\n+\t\tif (chmod(tab_file.path,\n \t\t\t add->stack->opts.default_permissions)) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n \t\t}\n \t}\n \n-\twriter.fd = get_tempfile_fd(tab_file);\n+\twriter.fd = tab_file.fd;\n \terr = reftable_writer_new(&wr, fd_writer_write, fd_writer_flush,\n \t\t\t\t &writer, &add->stack->opts);\n \tif (err < 0)\n@@ -918,11 +915,9 @@ int reftable_addition_add(struct reftable_addition *add,\n \tif (err < 0)\n \t\tgoto done;\n \n-\terr = close_tempfile_gently(tab_file);\n-\tif (err < 0) {\n-\t\terr = REFTABLE_IO_ERROR;\n+\terr = tmpfile_close(&tab_file);\n+\tif (err < 0)\n \t\tgoto done;\n-\t}\n \n \tif (wr->min_update_index < add->next_update_index) {\n \t\terr = REFTABLE_API_ERROR;\n@@ -945,11 +940,9 @@ int reftable_addition_add(struct reftable_addition *add,\n \t On windows, this relies on rand() picking a unique destination name.\n \t Maybe we should do retry loop as well?\n \t */\n-\terr = rename_tempfile(&tab_file, tab_file_name.buf);\n-\tif (err < 0) {\n-\t\terr = REFTABLE_IO_ERROR;\n+\terr = tmpfile_rename(&tab_file, tab_file_name.buf);\n+\tif (err < 0)\n \t\tgoto done;\n-\t}\n \n \tREFTABLE_ALLOC_GROW(add->new_tables, add->new_tables_len + 1,\n \t\t\t add->new_tables_cap);\n@@ -960,7 +953,7 @@ int reftable_addition_add(struct reftable_addition *add,\n \tadd->new_tables[add->new_tables_len++] = reftable_buf_detach(&next_name);\n \n done:\n-\tdelete_tempfile(&tab_file);\n+\ttmpfile_delete(&tab_file);\n \treftable_buf_release(&temp_tab_file_name);\n \treftable_buf_release(&tab_file_name);\n \treftable_buf_release(&next_name);\n@@ -980,7 +973,7 @@ uint64_t reftable_stack_next_update_index(struct reftable_stack *st)\n static int stack_compact_locked(struct reftable_stack *st,\n \t\t\t\tsize_t first, size_t last,\n \t\t\t\tstruct reftable_log_expiry_config *config,\n-\t\t\t\tstruct tempfile **tab_file_out)\n+\t\t\t\tstruct reftable_tmpfile *tab_file_out)\n {\n \tstruct reftable_buf next_name = REFTABLE_BUF_INIT;\n \tstruct reftable_buf tab_file_path = REFTABLE_BUF_INIT;\n@@ -988,7 +981,7 @@ static int stack_compact_locked(struct reftable_stack *st,\n \tstruct fd_writer writer= {\n \t\t.stack = st,\n \t};\n-\tstruct tempfile *tab_file;\n+\tstruct reftable_tmpfile tab_file = REFTABLE_TMPFILE_INIT;\n \tint err = 0;\n \n \terr = format_name(&next_name, reftable_reader_min_update_index(st->readers[first]),\n@@ -1004,19 +997,17 @@ static int stack_compact_locked(struct reftable_stack *st,\n \tif (err < 0)\n \t\tgoto done;\n \n-\ttab_file = mks_tempfile(tab_file_path.buf);\n-\tif (!tab_file) {\n-\t\terr = REFTABLE_IO_ERROR;\n+\terr = tmpfile_from_pattern(&tab_file, tab_file_path.buf);\n+\tif (err < 0)\n \t\tgoto done;\n-\t}\n \n \tif (st->opts.default_permissions &&\n-\t chmod(get_tempfile_path(tab_file), st->opts.default_permissions) < 0) {\n+\t chmod(tab_file.path, st->opts.default_permissions) < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n \n-\twriter.fd = get_tempfile_fd(tab_file);\n+\twriter.fd = tab_file.fd;\n \terr = reftable_writer_new(&wr, fd_writer_write, fd_writer_flush,\n \t\t\t\t &writer, &st->opts);\n \tif (err < 0)\n@@ -1030,15 +1021,15 @@ static int stack_compact_locked(struct reftable_stack *st,\n \tif (err < 0)\n \t\tgoto done;\n \n-\terr = close_tempfile_gently(tab_file);\n+\terr = tmpfile_close(&tab_file);\n \tif (err < 0)\n \t\tgoto done;\n \n \t*tab_file_out = tab_file;\n-\ttab_file = NULL;\n+\ttab_file = REFTABLE_TMPFILE_INIT;\n \n done:\n-\tdelete_tempfile(&tab_file);\n+\ttmpfile_delete(&tab_file);\n \treftable_writer_free(wr);\n \treftable_buf_release(&next_name);\n \treftable_buf_release(&tab_file_path);\n@@ -1171,7 +1162,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \tstruct reftable_buf table_name = REFTABLE_BUF_INIT;\n \tstruct lock_file tables_list_lock = LOCK_INIT;\n \tstruct lock_file *table_locks = NULL;\n-\tstruct tempfile *new_table = NULL;\n+\tstruct reftable_tmpfile new_table = REFTABLE_TMPFILE_INIT;\n \tint is_empty_table = 0, err = 0;\n \tsize_t first_to_replace, last_to_replace;\n \tsize_t i, nlocks = 0;\n@@ -1439,11 +1430,9 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\tif (err < 0)\n \t\t\tgoto done;\n \n-\t\terr = rename_tempfile(&new_table, new_table_path.buf);\n-\t\tif (err < 0) {\n-\t\t\terr = REFTABLE_IO_ERROR;\n+\t\terr = tmpfile_rename(&new_table, new_table_path.buf);\n+\t\tif (err < 0)\n \t\t\tgoto done;\n-\t\t}\n \t}\n \n \t/*\n@@ -1515,7 +1504,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\trollback_lock_file(&table_locks[i]);\n \treftable_free(table_locks);\n \n-\tdelete_tempfile(&new_table);\n+\ttmpfile_delete(&new_table);\n \treftable_buf_release(&new_table_name);\n \treftable_buf_release(&new_table_path);\n \treftable_buf_release(&tables_list_buf);\ndiff --git a/reftable/system.c b/reftable/system.c\nnew file mode 100644\nindex 00000000000..01f96f03d84\n--- /dev/null\n+++ b/reftable/system.c\n@@ -0,0 +1,49 @@\n+#include \"system.h\"\n+#include \"basics.h\"\n+#include \"reftable-error.h\"\n+#include \"../tempfile.h\"\n+\n+int tmpfile_from_pattern(struct reftable_tmpfile *out, const char *pattern)\n+{\n+\tstruct tempfile *tempfile;\n+\n+\ttempfile = mks_tempfile(pattern);\n+\tif (!tempfile)\n+\t\treturn REFTABLE_IO_ERROR;\n+\n+\tout->path = tempfile->filename.buf;\n+\tout->fd = tempfile->fd;\n+\tout->priv = tempfile;\n+\n+\treturn 0;\n+}\n+\n+int tmpfile_close(struct reftable_tmpfile *t)\n+{\n+\tstruct tempfile *tempfile = t->priv;\n+\tint ret = close_tempfile_gently(tempfile);\n+\tt->fd = -1;\n+\tif (ret < 0)\n+\t\treturn REFTABLE_IO_ERROR;\n+\treturn 0;\n+}\n+\n+int tmpfile_delete(struct reftable_tmpfile *t)\n+{\n+\tstruct tempfile *tempfile = t->priv;\n+\tint ret = delete_tempfile(&tempfile);\n+\t*t = REFTABLE_TMPFILE_INIT;\n+\tif (ret < 0)\n+\t\treturn REFTABLE_IO_ERROR;\n+\treturn 0;\n+}\n+\n+int tmpfile_rename(struct reftable_tmpfile *t, const char *path)\n+{\n+\tstruct tempfile *tempfile = t->priv;\n+\tint ret = rename_tempfile(&tempfile, path);\n+\t*t = REFTABLE_TMPFILE_INIT;\n+\tif (ret < 0)\n+\t\treturn REFTABLE_IO_ERROR;\n+\treturn 0;\n+}\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 38d3534620e..e7595800907 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -13,6 +13,45 @@ license that can be found in the LICENSE file or at\n \n #include \"git-compat-util.h\"\n #include \"lockfile.h\"\n-#include \"tempfile.h\"\n+\n+/*\n+ * An implementation-specific temporary file. By making this specific to the\n+ * implementation it becomes possible to tie temporary files into any kind of\n+ * signal or atexit handlers for cleanup on abnormal situations.\n+ */\n+struct reftable_tmpfile {\n+\tconst char *path;\n+\tint fd;\n+\tvoid *priv;\n+};\n+#define REFTABLE_TMPFILE_INIT ((struct reftable_tmpfile) { .fd = -1, })\n+\n+/*\n+ * Create a temporary file from a pattern similar to how mkstemp(3p) would.\n+ * The `pattern` shall not be modified. On success, the structure at `out` has\n+ * been initialized such that it is ready for use. Returns 0 on success, a\n+ * reftable error code on error.\n+ */\n+int tmpfile_from_pattern(struct reftable_tmpfile *out, const char *pattern);\n+\n+/*\n+ * Close the temporary file's file descriptor without removing the file itself.\n+ * This is a no-op in case the file has already been closed beforehand. Returns\n+ * 0 on success, a reftable error code on error.\n+ */\n+int tmpfile_close(struct reftable_tmpfile *t);\n+\n+/*\n+ * Close the temporary file and delete it. This is a no-op in case the file has\n+ * already been deleted or renamed beforehand. Returns 0 on success, a reftable\n+ * error code on error.\n+ */\n+int tmpfile_delete(struct reftable_tmpfile *t);\n+\n+/*\n+ * Rename the temporary file to the provided path. The temporary file must be\n+ * active. Return 0 on success, a reftable error code on error.\n+ */\n+int tmpfile_rename(struct reftable_tmpfile *t, const char *path);\n \n #endif\n-- \n2.47.0.118.gfd3785337b.dirty\n\n"},{"id":"505917","messageId":"74afe30974dd800337950c3cb14e97e2a111b635.1729677003.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1729677003.git.ps@pks.im","subject":"[PATCH 6/7] reftable/stack: drop only use of `get_locked_file_path()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-23T09:56:10Z","receivedAt":"2024-10-23T09:56:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We've got a single callsite where we call `get_locked_file_path()`. As\nwe're about to convert our usage of the lockfile subsystem to instead be\nused via a compatibility shim we'd have to implement more logic for this\nsingle callsite. While that would be okay if Git was the only supposed\nuser of the reftable library, it's a bit more awkward when considering\nthat we have to reimplement this functionality for every user of the\nlibrary eventually.\n\nRefactor the code such that we don't call `get_locked_file_path()`\nanymore.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 12 +++++++++---\n 1 file changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 67b2117a112..c1a4e25e3a2 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1493,9 +1493,15 @@ static int stack_compact_range(struct reftable_stack *st,\n \t */\n \tfor (i = 0; i < nlocks; i++) {\n \t\tstruct lock_file *table_lock = &table_locks[i];\n-\t\tchar *table_path = get_locked_file_path(table_lock);\n-\t\tunlink(table_path);\n-\t\treftable_free(table_path);\n+\t\tconst char *lock_path = get_lock_file_path(table_lock);\n+\n+\t\treftable_buf_reset(&table_name);\n+\t\terr = reftable_buf_add(&table_name, lock_path,\n+\t\t\t\t strlen(lock_path) - strlen(\".lock\"));\n+\t\tif (err)\n+\t\t\tcontinue;\n+\n+\t\tunlink(table_name.buf);\n \t}\n \n done:\n-- \n2.47.0.118.gfd3785337b.dirty\n\n"},{"id":"505918","messageId":"71b213d6f8af725b73b5b9950667668f7c699bef.1729677003.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1729677003.git.ps@pks.im","subject":"[PATCH 7/7] reftable/system: provide thin wrapper for lockfile subsystem","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-23T09:56:15Z","receivedAt":"2024-10-23T09:56:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We use the lockfile subsystem to write lockfiles for \"tables.list\". As\nwith the tempfile subsystem, the lockfile subsystem also hooks into our\ninfrastructure to prune stale locks via atexit(3p) or signal handlers.\n\nFurthermore, the lockfile subsystem also handles locking timeouts, which\ndo add quite a bit of logic. Having to reimplement that in the context\nof Git wouldn't make a whole lot of sense, and it is quite likely that\ndownstream users of the reftable library may have a better idea for how\nexactly to implement timeouts.\n\nSo again, provide a thin wrapper for the lockfile subsystem instead such\nthat the compatibility shim is fully self-contained.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 63 ++++++++++-------------\n reftable/system.c | 77 +++++++++++++++++++++++++++++\n reftable/system.h | 45 ++++++++++++++++-\n t/unit-tests/lib-reftable.c | 1 +\n t/unit-tests/t-reftable-block.c | 1 +\n t/unit-tests/t-reftable-pq.c | 1 +\n t/unit-tests/t-reftable-readwrite.c | 1 +\n t/unit-tests/t-reftable-stack.c | 2 +\n 8 files changed, 154 insertions(+), 37 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex c1a4e25e3a2..1fffd756302 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -657,7 +657,7 @@ static int format_name(struct reftable_buf *dest, uint64_t min, uint64_t max)\n }\n \n struct reftable_addition {\n-\tstruct lock_file tables_list_lock;\n+\tstruct reftable_flock tables_list_lock;\n \tstruct reftable_stack *stack;\n \n \tchar **new_tables;\n@@ -676,10 +676,8 @@ static int reftable_stack_init_addition(struct reftable_addition *add,\n \n \tadd->stack = st;\n \n-\terr = hold_lock_file_for_update_timeout(&add->tables_list_lock,\n-\t\t\t\t\t\tst->list_file,\n-\t\t\t\t\t\tLOCK_NO_DEREF,\n-\t\t\t\t\t\tst->opts.lock_timeout_ms);\n+\terr = flock_acquire(&add->tables_list_lock, st->list_file,\n+\t\t\t st->opts.lock_timeout_ms);\n \tif (err < 0) {\n \t\tif (errno == EEXIST) {\n \t\t\terr = REFTABLE_LOCK_ERROR;\n@@ -689,7 +687,7 @@ static int reftable_stack_init_addition(struct reftable_addition *add,\n \t\tgoto done;\n \t}\n \tif (st->opts.default_permissions) {\n-\t\tif (chmod(get_lock_file_path(&add->tables_list_lock),\n+\t\tif (chmod(add->tables_list_lock.path,\n \t\t\t st->opts.default_permissions) < 0) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n@@ -733,7 +731,7 @@ static void reftable_addition_close(struct reftable_addition *add)\n \tadd->new_tables_len = 0;\n \tadd->new_tables_cap = 0;\n \n-\trollback_lock_file(&add->tables_list_lock);\n+\tflock_release(&add->tables_list_lock);\n \treftable_buf_release(&nm);\n }\n \n@@ -749,7 +747,6 @@ void reftable_addition_destroy(struct reftable_addition *add)\n int reftable_addition_commit(struct reftable_addition *add)\n {\n \tstruct reftable_buf table_list = REFTABLE_BUF_INIT;\n-\tint lock_file_fd = get_lock_file_fd(&add->tables_list_lock);\n \tint err = 0;\n \tsize_t i;\n \n@@ -767,20 +764,20 @@ int reftable_addition_commit(struct reftable_addition *add)\n \t\t\tgoto done;\n \t}\n \n-\terr = write_in_full(lock_file_fd, table_list.buf, table_list.len);\n+\terr = write_in_full(add->tables_list_lock.fd, table_list.buf, table_list.len);\n \treftable_buf_release(&table_list);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n \n-\terr = stack_fsync(add->stack, lock_file_fd);\n+\terr = stack_fsync(add->stack, add->tables_list_lock.fd);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n \n-\terr = commit_lock_file(&add->tables_list_lock);\n+\terr = flock_commit(&add->tables_list_lock);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n@@ -1160,8 +1157,8 @@ static int stack_compact_range(struct reftable_stack *st,\n \tstruct reftable_buf new_table_name = REFTABLE_BUF_INIT;\n \tstruct reftable_buf new_table_path = REFTABLE_BUF_INIT;\n \tstruct reftable_buf table_name = REFTABLE_BUF_INIT;\n-\tstruct lock_file tables_list_lock = LOCK_INIT;\n-\tstruct lock_file *table_locks = NULL;\n+\tstruct reftable_flock tables_list_lock = REFTABLE_FLOCK_INIT;\n+\tstruct reftable_flock *table_locks = NULL;\n \tstruct reftable_tmpfile new_table = REFTABLE_TMPFILE_INIT;\n \tint is_empty_table = 0, err = 0;\n \tsize_t first_to_replace, last_to_replace;\n@@ -1179,10 +1176,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * Hold the lock so that we can read \"tables.list\" and lock all tables\n \t * which are part of the user-specified range.\n \t */\n-\terr = hold_lock_file_for_update_timeout(&tables_list_lock,\n-\t\t\t\t\t\tst->list_file,\n-\t\t\t\t\t\tLOCK_NO_DEREF,\n-\t\t\t\t\t\tst->opts.lock_timeout_ms);\n+\terr = flock_acquire(&tables_list_lock, st->list_file, st->opts.lock_timeout_ms);\n \tif (err < 0) {\n \t\tif (errno == EEXIST)\n \t\t\terr = REFTABLE_LOCK_ERROR;\n@@ -1205,19 +1199,20 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * older process is still busy compacting tables which are preexisting\n \t * from the point of view of the newer process.\n \t */\n-\tREFTABLE_CALLOC_ARRAY(table_locks, last - first + 1);\n+\tREFTABLE_ALLOC_ARRAY(table_locks, last - first + 1);\n \tif (!table_locks) {\n \t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n \t\tgoto done;\n \t}\n+\tfor (i = 0; i < last - first + 1; i++)\n+\t\ttable_locks[i] = REFTABLE_FLOCK_INIT;\n \n \tfor (i = last + 1; i > first; i--) {\n \t\terr = stack_filename(&table_name, st, reader_name(st->readers[i - 1]));\n \t\tif (err < 0)\n \t\t\tgoto done;\n \n-\t\terr = hold_lock_file_for_update(&table_locks[nlocks],\n-\t\t\t\t\t\ttable_name.buf, LOCK_NO_DEREF);\n+\t\terr = flock_acquire(&table_locks[nlocks], table_name.buf, 0);\n \t\tif (err < 0) {\n \t\t\t/*\n \t\t\t * When the table is locked already we may do a\n@@ -1253,7 +1248,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\t * run into file descriptor exhaustion when we compress a lot\n \t\t * of tables.\n \t\t */\n-\t\terr = close_lock_file_gently(&table_locks[nlocks++]);\n+\t\terr = flock_close(&table_locks[nlocks++]);\n \t\tif (err < 0) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n@@ -1265,7 +1260,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * \"tables.list\" lock while compacting the locked tables. This allows\n \t * concurrent updates to the stack to proceed.\n \t */\n-\terr = rollback_lock_file(&tables_list_lock);\n+\terr = flock_release(&tables_list_lock);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n@@ -1288,10 +1283,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * \"tables.list\". We'll then replace the compacted range of tables with\n \t * the new table.\n \t */\n-\terr = hold_lock_file_for_update_timeout(&tables_list_lock,\n-\t\t\t\t\t\tst->list_file,\n-\t\t\t\t\t\tLOCK_NO_DEREF,\n-\t\t\t\t\t\tst->opts.lock_timeout_ms);\n+\terr = flock_acquire(&tables_list_lock, st->list_file, st->opts.lock_timeout_ms);\n \tif (err < 0) {\n \t\tif (errno == EEXIST)\n \t\t\terr = REFTABLE_LOCK_ERROR;\n@@ -1301,7 +1293,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t}\n \n \tif (st->opts.default_permissions) {\n-\t\tif (chmod(get_lock_file_path(&tables_list_lock),\n+\t\tif (chmod(tables_list_lock.path,\n \t\t\t st->opts.default_permissions) < 0) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n@@ -1456,7 +1448,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\t\tgoto done;\n \t}\n \n-\terr = write_in_full(get_lock_file_fd(&tables_list_lock),\n+\terr = write_in_full(tables_list_lock.fd,\n \t\t\t tables_list_buf.buf, tables_list_buf.len);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n@@ -1464,14 +1456,14 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\tgoto done;\n \t}\n \n-\terr = stack_fsync(st, get_lock_file_fd(&tables_list_lock));\n+\terr = stack_fsync(st, tables_list_lock.fd);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tunlink(new_table_path.buf);\n \t\tgoto done;\n \t}\n \n-\terr = commit_lock_file(&tables_list_lock);\n+\terr = flock_commit(&tables_list_lock);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tunlink(new_table_path.buf);\n@@ -1492,12 +1484,11 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * readers, so it is expected that unlinking tables may fail.\n \t */\n \tfor (i = 0; i < nlocks; i++) {\n-\t\tstruct lock_file *table_lock = &table_locks[i];\n-\t\tconst char *lock_path = get_lock_file_path(table_lock);\n+\t\tstruct reftable_flock *table_lock = &table_locks[i];\n \n \t\treftable_buf_reset(&table_name);\n-\t\terr = reftable_buf_add(&table_name, lock_path,\n-\t\t\t\t strlen(lock_path) - strlen(\".lock\"));\n+\t\terr = reftable_buf_add(&table_name, table_lock->path,\n+\t\t\t\t strlen(table_lock->path) - strlen(\".lock\"));\n \t\tif (err)\n \t\t\tcontinue;\n \n@@ -1505,9 +1496,9 @@ static int stack_compact_range(struct reftable_stack *st,\n \t}\n \n done:\n-\trollback_lock_file(&tables_list_lock);\n+\tflock_release(&tables_list_lock);\n \tfor (i = 0; table_locks && i < nlocks; i++)\n-\t\trollback_lock_file(&table_locks[i]);\n+\t\tflock_release(&table_locks[i]);\n \treftable_free(table_locks);\n \n \ttmpfile_delete(&new_table);\ndiff --git a/reftable/system.c b/reftable/system.c\nindex 01f96f03d84..adf8e4d30b8 100644\n--- a/reftable/system.c\n+++ b/reftable/system.c\n@@ -1,6 +1,7 @@\n #include \"system.h\"\n #include \"basics.h\"\n #include \"reftable-error.h\"\n+#include \"../lockfile.h\"\n #include \"../tempfile.h\"\n \n int tmpfile_from_pattern(struct reftable_tmpfile *out, const char *pattern)\n@@ -47,3 +48,79 @@ int tmpfile_rename(struct reftable_tmpfile *t, const char *path)\n \t\treturn REFTABLE_IO_ERROR;\n \treturn 0;\n }\n+\n+int flock_acquire(struct reftable_flock *l, const char *target_path,\n+\t\t long timeout_ms)\n+{\n+\tstruct lock_file *lockfile;\n+\tint err;\n+\n+\tlockfile = reftable_malloc(sizeof(*lockfile));\n+\tif (!lockfile)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n+\terr = hold_lock_file_for_update_timeout(lockfile, target_path, LOCK_NO_DEREF,\n+\t\t\t\t\t\ttimeout_ms);\n+\tif (err < 0) {\n+\t\treftable_free(lockfile);\n+\t\tif (errno == EEXIST)\n+\t\t\treturn REFTABLE_LOCK_ERROR;\n+\t\treturn -1;\n+\t}\n+\n+\tl->fd = get_lock_file_fd(lockfile);\n+\tl->path = get_lock_file_path(lockfile);\n+\tl->priv = lockfile;\n+\n+\treturn 0;\n+}\n+\n+int flock_close(struct reftable_flock *l)\n+{\n+\tstruct lock_file *lockfile = l->priv;\n+\tint ret;\n+\n+\tif (!lockfile)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\tret = close_lock_file_gently(lockfile);\n+\tl->fd = -1;\n+\tif (ret < 0)\n+\t\treturn REFTABLE_IO_ERROR;\n+\n+\treturn 0;\n+}\n+\n+int flock_release(struct reftable_flock *l)\n+{\n+\tstruct lock_file *lockfile = l->priv;\n+\tint ret;\n+\n+\tif (!lockfile)\n+\t\treturn 0;\n+\n+\tret = rollback_lock_file(lockfile);\n+\treftable_free(lockfile);\n+\t*l = REFTABLE_FLOCK_INIT;\n+\tif (ret < 0)\n+\t\treturn REFTABLE_IO_ERROR;\n+\n+\treturn 0;\n+}\n+\n+int flock_commit(struct reftable_flock *l)\n+{\n+\tstruct lock_file *lockfile = l->priv;\n+\tint ret;\n+\n+\tif (!lockfile)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\tret = commit_lock_file(lockfile);\n+\treftable_free(lockfile);\n+\t*l = REFTABLE_FLOCK_INIT;\n+\tif (ret < 0)\n+\t\treturn REFTABLE_IO_ERROR;\n+\n+\treturn 0;\n+}\ndiff --git a/reftable/system.h b/reftable/system.h\nindex e7595800907..0859c3539c6 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -12,7 +12,6 @@ license that can be found in the LICENSE file or at\n /* This header glues the reftable library to the rest of Git */\n \n #include \"git-compat-util.h\"\n-#include \"lockfile.h\"\n \n /*\n * An implementation-specific temporary file. By making this specific to the\n@@ -54,4 +53,48 @@ int tmpfile_delete(struct reftable_tmpfile *t);\n */\n int tmpfile_rename(struct reftable_tmpfile *t, const char *path);\n \n+/*\n+ * An implementation-specific file lock. Same as with `reftable_tmpfile`,\n+ * making this specific to the implementation makes it possible to tie this\n+ * into signal or atexit handlers such that we know to clean up stale locks on\n+ * abnormal exits.\n+ */\n+struct reftable_flock {\n+\tconst char *path;\n+\tint fd;\n+\tvoid *priv;\n+};\n+#define REFTABLE_FLOCK_INIT ((struct reftable_flock){ .fd = -1, })\n+\n+/*\n+ * Acquire the lock for the given target path by exclusively creating a file\n+ * with \".lock\" appended to it. If that lock exists, we wait up to `timeout_ms`\n+ * to acquire the lock. If `timeout_ms` is 0 we don't wait, if it is negative\n+ * we block indefinitely.\n+ *\n+ * Retrun 0 on success, a reftable error code on error.\n+ */\n+int flock_acquire(struct reftable_flock *l, const char *target_path,\n+\t\t long timeout_ms);\n+\n+/*\n+ * Close the lockfile's file descriptor without removing the lock itself. This\n+ * is a no-op in case the lockfile has already been closed beforehand. Returns\n+ * 0 on success, a reftable error code on error.\n+ */\n+int flock_close(struct reftable_flock *l);\n+\n+/*\n+ * Release the lock by unlinking the lockfile. This is a no-op in case the\n+ * lockfile has already been released or committed beforehand. Returns 0 on\n+ * success, a reftable error code on error.\n+ */\n+int flock_release(struct reftable_flock *l);\n+\n+/*\n+ * Commit the lock by renaming the lockfile into place. Returns 0 on success, a\n+ * reftable error code on error.\n+ */\n+int flock_commit(struct reftable_flock *l);\n+\n #endif\ndiff --git a/t/unit-tests/lib-reftable.c b/t/unit-tests/lib-reftable.c\nindex c1631f45275..d795dfb7c99 100644\n--- a/t/unit-tests/lib-reftable.c\n+++ b/t/unit-tests/lib-reftable.c\n@@ -2,6 +2,7 @@\n #include \"test-lib.h\"\n #include \"reftable/constants.h\"\n #include \"reftable/writer.h\"\n+#include \"strbuf.h\"\n \n void t_reftable_set_hash(uint8_t *p, int i, enum reftable_hash id)\n {\ndiff --git a/t/unit-tests/t-reftable-block.c b/t/unit-tests/t-reftable-block.c\nindex 13e10807dae..22040aeefa5 100644\n--- a/t/unit-tests/t-reftable-block.c\n+++ b/t/unit-tests/t-reftable-block.c\n@@ -11,6 +11,7 @@ license that can be found in the LICENSE file or at\n #include \"reftable/blocksource.h\"\n #include \"reftable/constants.h\"\n #include \"reftable/reftable-error.h\"\n+#include \"strbuf.h\"\n \n static void t_ref_block_read_write(void)\n {\ndiff --git a/t/unit-tests/t-reftable-pq.c b/t/unit-tests/t-reftable-pq.c\nindex 272da05bea6..f3f8a0cdf38 100644\n--- a/t/unit-tests/t-reftable-pq.c\n+++ b/t/unit-tests/t-reftable-pq.c\n@@ -9,6 +9,7 @@ license that can be found in the LICENSE file or at\n #include \"test-lib.h\"\n #include \"reftable/constants.h\"\n #include \"reftable/pq.h\"\n+#include \"strbuf.h\"\n \n static void merged_iter_pqueue_check(const struct merged_iter_pqueue *pq)\n {\ndiff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c\nindex 57896922eb1..91c881aedfa 100644\n--- a/t/unit-tests/t-reftable-readwrite.c\n+++ b/t/unit-tests/t-reftable-readwrite.c\n@@ -13,6 +13,7 @@ license that can be found in the LICENSE file or at\n #include \"reftable/reader.h\"\n #include \"reftable/reftable-error.h\"\n #include \"reftable/reftable-writer.h\"\n+#include \"strbuf.h\"\n \n static const int update_index = 5;\n \ndiff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c\nindex 13fd8d8f941..b2f6c1c37e9 100644\n--- a/t/unit-tests/t-reftable-stack.c\n+++ b/t/unit-tests/t-reftable-stack.c\n@@ -13,6 +13,8 @@ license that can be found in the LICENSE file or at\n #include \"reftable/reader.h\"\n #include \"reftable/reftable-error.h\"\n #include \"reftable/stack.h\"\n+#include \"strbuf.h\"\n+#include \"tempfile.h\"\n #include \n \n static void clear_dir(const char *dirname)\n-- \n2.47.0.118.gfd3785337b.dirty\n\n"},{"id":"505998","messageId":"CAPig+cQH+Y47KdpjztSMkJOSHxZ3ZNXBWQutEQCfPO348FQr6g@mail.gmail.com","threadId":"62395","inReplyTo":"036cc8f9d602a817aa579092bdeaef5d1156214e.1729677003.git.ps@pks.im","subject":"Re: [PATCH 1/7] reftable/system: move \"dir.h\" to its only user","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2024-10-24T04:18:13Z","receivedAt":"2024-10-24T04:18:25Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Wed, Oct 23, 2024 at 5:57 AM Patrick Steinhardt wrote:\n> We still include \"dir.h\" in \"reftable/system.h\" evne though it is not\n> used by anything but by a single unit test. Move it over into that unit\n> test so that we don't accidentally use any functionality provided by it\n> in the reftable codebase.\n\ns/evne/even/\n\n> Signed-off-by: Patrick Steinhardt \n"},{"id":"506815","messageId":"yvdvlf45g4ahqa3uey4zj4evissv2tfkwiy5o3e3vbqtwmc7o4@utqagwvv6yhe","threadId":"62395","inReplyTo":"c1bd8e2b3c4c524a21f68e822812e41179827e4d.1729677003.git.ps@pks.im","subject":"Re: [PATCH 2/7] reftable: explicitly handle hash format IDs","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-11-07T23:11:28Z","receivedAt":"2024-11-07T23:13:11Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/10/23 11:55AM, Patrick Steinhardt wrote:\n> diff --git a/reftable/basics.h b/reftable/basics.h\n> index 7aa46d7c30d..86141602e74 100644\n> --- a/reftable/basics.h\n> +++ b/reftable/basics.h\n> @@ -150,4 +150,11 @@ int common_prefix_size(struct reftable_buf *a, struct reftable_buf *b);\n> \n> int hash_size(uint32_t id);\n> \n> +/*\n> + * Format IDs that identify the hash function used by a reftable. Note that\n> + * these constants end up on disk and thus mustn't change.\n> + */\n> +#define REFTABLE_FORMAT_ID_SHA1 ((uint32_t) 0x73686131)\n> +#define REFTABLE_FORMAT_ID_SHA256 ((uint32_t) 0x73323536)\n\nThis context is provided in the comments for `GIT_*_FORMAT_ID`, but now\nthat they being decoupled it might be nice to mention here that they\nstand for \"sha1\" and \"s256\" respectively.\n\n> +\n> #endif\n> diff --git a/reftable/reader.c b/reftable/reader.c\n> index 90dc950b577..64eb6938efe 100644\n> --- a/reftable/reader.c\n> +++ b/reftable/reader.c\n> @@ -109,16 +109,18 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n> \tif (r->version == 1) {\n> \t\tr->hash_id = GIT_SHA1_FORMAT_ID;\n> \t} else {\n> -\t\tr->hash_id = get_be32(f);\n> -\t\tswitch (r->hash_id) {\n> -\t\tcase GIT_SHA1_FORMAT_ID:\n> +\t\tswitch (get_be32(f)) {\n> +\t\tcase REFTABLE_FORMAT_ID_SHA1:\n> +\t\t\tr->hash_id = GIT_SHA1_FORMAT_ID;\n> \t\t\tbreak;\n> -\t\tcase GIT_SHA256_FORMAT_ID:\n> +\t\tcase REFTABLE_FORMAT_ID_SHA256:\n> +\t\t\tr->hash_id = GIT_SHA256_FORMAT_ID;\n> \t\t\tbreak;\n> \t\tdefault:\n> \t\t\terr = REFTABLE_FORMAT_ERROR;\n> \t\t\tgoto done;\n> \t\t}\n\nInstead of directly mapping `hash_id` to `GIT_*_FORMAT_ID`, we use the\nnewly defined constants to map to the corresponding format. As an\ninternal identifier, the format ID can now be freely changed without\nrisk of affecting how on-disk reftable headers are parsed. Makes sense\nto me and seems like a good change.\n\n-Justin\n"},{"id":"506817","messageId":"hhlsmyyqt2y6ue3abnlwjmrmsslewwjcoiu44aadp6nqkscict@m55ip4ibqgoq","threadId":"62395","inReplyTo":"b595668a5cdae0dff4a7271a3547c5821aa6e912.1729677003.git.ps@pks.im","subject":"Re: [PATCH 3/7] reftable/system: stop depending on \"hash.h\"","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-11-08T01:10:11Z","receivedAt":"2024-11-08T01:11:56Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/10/23 11:56AM, Patrick Steinhardt wrote:\n> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> index 3c6107c7ce5..7d86d920970 100644\n> --- a/refs/reftable-backend.c\n> +++ b/refs/reftable-backend.c\n> @@ -15,6 +15,7 @@\n> #include \"../object.h\"\n> #include \"../path.h\"\n> #include \"../refs.h\"\n> +#include \"../reftable/reftable-basics.h\"\n> #include \"../reftable/reftable-stack.h\"\n> #include \"../reftable/reftable-record.h\"\n> #include \"../reftable/reftable-error.h\"\n> @@ -289,7 +290,16 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n> \trefs->store_flags = store_flags;\n> \trefs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);\n> \n> -\trefs->write_options.hash_id = repo->hash_algo->format_id;\n> +\tswitch (repo->hash_algo->format_id) {\n> +\tcase GIT_SHA1_FORMAT_ID:\n> +\t\trefs->write_options.hash_id = REFTABLE_HASH_SHA1;\n> +\t\tbreak;\n> +\tcase GIT_SHA256_FORMAT_ID:\n> +\t\trefs->write_options.hash_id = REFTABLE_HASH_SHA256;\n> +\t\tbreak;\n> +\tdefault:\n> +\t\tBUG(\"unknown hash algorithm %d\", repo->hash_algo->format_id);\n> +\t}\n\nHere we define the mapping between the Git's format ID and the reftable\nhash external to the reftable library. This facilitates swapping uses of\n`GIT_*_FORMAT_ID` to `REFTABLE_HASH_*` as done in the rest of the patch\nwhich looks good.\n\n> \trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n> \trefs->write_options.disable_auto_compact =\n> \t\t!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1);\n[snip]\n> diff --git a/reftable/merged.h b/reftable/merged.h\n> index 89bd0c4b35b..13a5fe4154e 100644\n> --- a/reftable/merged.h\n> +++ b/reftable/merged.h\n> @@ -10,11 +10,12 @@ license that can be found in the LICENSE file or at\n> #define MERGED_H\n> \n> #include \"system.h\"\n> +#include \"basics.h\"\n\nNaive question, being that \"merged.h\" only depends on\n`reftable-basics.h:reftable_hash` and not any of the internal reftable\nbasics components, would it be best to instead reference it directly? Or\nbecause \"merged.h\" is also internal do we also prefer to use the\ninternal \"basics.h\"? \n\nProbably doesn't really matter, but I was just curious if there was any\nreasoning. :)\n\n-Justin\n"},{"id":"506820","messageId":"6ghxx5fmuzujegducuva77vpybihz6b5cnk75wdgv4pv2knac5@m3trfmg2dk5s","threadId":"62395","inReplyTo":"86269fc4fcad9a97709aa0d080c4c077a85ca667.1729677003.git.ps@pks.im","subject":"Re: [PATCH 4/7] reftable/stack: stop using `fsync_component()` directly","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-11-08T02:09:21Z","receivedAt":"2024-11-08T02:11:05Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/10/23 11:56AM, Patrick Steinhardt wrote:\n[snip]\n> diff --git a/reftable/stack.c b/reftable/stack.c\n> index 9ae716ff375..df4f3237007 100644\n> --- a/reftable/stack.c\n> +++ b/reftable/stack.c\n> @@ -8,7 +8,6 @@ license that can be found in the LICENSE file or at\n> \n> #include \"stack.h\"\n> \n> -#include \"../write-or-die.h\"\n> #include \"system.h\"\n> #include \"constants.h\"\n> #include \"merged.h\"\n> @@ -43,17 +42,28 @@ static int stack_filename(struct reftable_buf *dest, struct reftable_stack *st,\n> \treturn 0;\n> }\n> \n> -static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz)\n> +static int stack_fsync(struct reftable_stack *st, int fd)\n> {\n> -\tint *fdp = (int *)arg;\n> -\treturn write_in_full(*fdp, data, sz);\n> +\tif (st->opts.fsync)\n> +\t\treturn st->opts.fsync(fd);\n> +\treturn fsync(fd);\n> }\n> \n> -static int reftable_fd_flush(void *arg)\n> +struct fd_writer {\n> +\tstruct reftable_stack *stack;\n\nOut of curiousity, from the stack I think we only need the callback in\nthe options. Any reason we provide the whole stack here?\n\n> +\tint fd;\n> +};\n> +\n> +static ssize_t fd_writer_write(void *arg, const void *data, size_t sz)\n> {\n> -\tint *fdp = (int *)arg;\n> +\tstruct fd_writer *writer = arg;\n> +\treturn write_in_full(writer->fd, data, sz);\n> +}\n> \n> -\treturn fsync_component(FSYNC_COMPONENT_REFERENCE, *fdp);\n\nPreviously when the writer was flushed it would invoke\n`fsync_component()`. Now that a callback can be configured in the stack\noptions, the callback needs to also be propagated to\n`fd_writer_write()` in addition to the file descriptor being synced.\nThis explains why `fd_writer` is now used.\n\nThe rest of the patch updates writer configuration and fsync call sites.\nLooks good.\n\n-Justin\n"},{"id":"506826","messageId":"Zy2swYMe9naG4akA@pks.im","threadId":"62395","inReplyTo":"hhlsmyyqt2y6ue3abnlwjmrmsslewwjcoiu44aadp6nqkscict@m55ip4ibqgoq","subject":"Re: [PATCH 3/7] reftable/system: stop depending on \"hash.h\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-08T06:17:03Z","receivedAt":"2024-11-08T06:17:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Nov 07, 2024 at 07:10:11PM -0600, Justin Tobler wrote:\n> On 24/10/23 11:56AM, Patrick Steinhardt wrote:\n> > diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> > index 3c6107c7ce5..7d86d920970 100644\n> > --- a/refs/reftable-backend.c\n> > +++ b/refs/reftable-backend.c\n> > diff --git a/reftable/merged.h b/reftable/merged.h\n> > index 89bd0c4b35b..13a5fe4154e 100644\n> > --- a/reftable/merged.h\n> > +++ b/reftable/merged.h\n> > @@ -10,11 +10,12 @@ license that can be found in the LICENSE file or at\n> > #define MERGED_H\n> > \n> > #include \"system.h\"\n> > +#include \"basics.h\"\n> \n> Naive question, being that \"merged.h\" only depends on\n> `reftable-basics.h:reftable_hash` and not any of the internal reftable\n> basics components, would it be best to instead reference it directly? Or\n> because \"merged.h\" is also internal do we also prefer to use the\n> internal \"basics.h\"? \n> \n> Probably doesn't really matter, but I was just curious if there was any\n> reasoning. :)\n\nNot really, let's do as you propose.\n\nPatrick\n"},{"id":"506825","messageId":"Zy2s41sv4IlMoISl@pks.im","threadId":"62395","inReplyTo":"6ghxx5fmuzujegducuva77vpybihz6b5cnk75wdgv4pv2knac5@m3trfmg2dk5s","subject":"Re: [PATCH 4/7] reftable/stack: stop using `fsync_component()` directly","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-08T06:17:07Z","receivedAt":"2024-11-08T06:17:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Nov 07, 2024 at 08:09:21PM -0600, Justin Tobler wrote:\n> On 24/10/23 11:56AM, Patrick Steinhardt wrote:\n> [snip]\n> > diff --git a/reftable/stack.c b/reftable/stack.c\n> > index 9ae716ff375..df4f3237007 100644\n> > --- a/reftable/stack.c\n> > +++ b/reftable/stack.c\n> > @@ -43,17 +42,28 @@ static int stack_filename(struct reftable_buf *dest, struct reftable_stack *st,\n> > \treturn 0;\n> > }\n> > \n> > -static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz)\n> > +static int stack_fsync(struct reftable_stack *st, int fd)\n> > {\n> > -\tint *fdp = (int *)arg;\n> > -\treturn write_in_full(*fdp, data, sz);\n> > +\tif (st->opts.fsync)\n> > +\t\treturn st->opts.fsync(fd);\n> > +\treturn fsync(fd);\n> > }\n> > \n> > -static int reftable_fd_flush(void *arg)\n> > +struct fd_writer {\n> > +\tstruct reftable_stack *stack;\n> \n> Out of curiousity, from the stack I think we only need the callback in\n> the options. Any reason we provide the whole stack here?\n\nI just think that passing around function pointers doesn't make for a\ngood calling convention here, as it hides the fact that it is possible\nto call this without a callback. But there isn't a reason to pass in the\nwhole stack, it would also be fine to instead pass in e.g. the write\noptions.\n\nI think I'll do that instead.\n\nPatrick\n"},{"id":"506832","messageId":"cover.1731047193.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1729677003.git.ps@pks.im","subject":"[PATCH v2 0/7] reftable: stop using Git subsystems","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-08T08:17:06Z","receivedAt":"2024-11-08T08:17:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the second version of my patch series that continues to detangle\nthe reftable library from the Git codebase.\n\nChanges compared to v1:\n\n - Fix a commit message typo.\n\n - Document the values of the newly introduced reftable format IDs.\n\n - Include \"reftable-basics.h\" instead of \"basics.h\".\n\n - Adapt `stack_fsync()` to take write options as input instead of the\n whole stack.\n\nThanks!\n\nPatrick\n\nPatrick Steinhardt (7):\n reftable/system: move \"dir.h\" to its only user\n reftable: explicitly handle hash format IDs\n reftable/system: stop depending on \"hash.h\"\n reftable/stack: stop using `fsync_component()` directly\n reftable/system: provide thin wrapper for tempfile subsystem\n reftable/stack: drop only use of `get_locked_file_path()`\n reftable/system: provide thin wrapper for lockfile subsystem\n\n Makefile | 1 +\n refs/reftable-backend.c | 19 +++-\n reftable/basics.c | 13 ++-\n reftable/basics.h | 10 +-\n reftable/merged.c | 4 +-\n reftable/merged.h | 3 +-\n reftable/reader.c | 14 ++-\n reftable/reader.h | 4 +-\n reftable/reftable-basics.h | 13 +++\n reftable/reftable-merged.h | 4 +-\n reftable/reftable-reader.h | 2 +-\n reftable/reftable-record.h | 12 +-\n reftable/reftable-writer.h | 8 +-\n reftable/stack.c | 171 ++++++++++++++--------------\n reftable/system.c | 126 ++++++++++++++++++++\n reftable/system.h | 88 +++++++++++++-\n reftable/writer.c | 20 +++-\n t/helper/test-reftable.c | 4 +-\n t/unit-tests/lib-reftable.c | 5 +-\n t/unit-tests/lib-reftable.h | 2 +-\n t/unit-tests/t-reftable-block.c | 41 +++----\n t/unit-tests/t-reftable-merged.c | 26 ++---\n t/unit-tests/t-reftable-pq.c | 3 +-\n t/unit-tests/t-reftable-reader.c | 4 +-\n t/unit-tests/t-reftable-readwrite.c | 41 +++----\n t/unit-tests/t-reftable-record.c | 59 +++++-----\n t/unit-tests/t-reftable-stack.c | 37 +++---\n 27 files changed, 505 insertions(+), 229 deletions(-)\n create mode 100644 reftable/system.c\n\nRange-diff against v1:\n1: 036cc8f9d60 ! 1: 2b7d4e28529 reftable/system: move \"dir.h\" to its only user\n @@ Metadata\n ## Commit message ##\n reftable/system: move \"dir.h\" to its only user\n \n - We still include \"dir.h\" in \"reftable/system.h\" evne though it is not\n + We still include \"dir.h\" in \"reftable/system.h\" even though it is not\n used by anything but by a single unit test. Move it over into that unit\n test so that we don't accidentally use any functionality provided by it\n in the reftable codebase.\n2: c1bd8e2b3c4 ! 2: 38cfe85bf5b reftable: explicitly handle hash format IDs\n @@ reftable/basics.h: int common_prefix_size(struct reftable_buf *a, struct reftabl\n \n +/*\n + * Format IDs that identify the hash function used by a reftable. Note that\n -+ * these constants end up on disk and thus mustn't change.\n ++ * these constants end up on disk and thus mustn't change. The format IDs are\n ++ * \"sha1\" and \"s256\" in big endian, respectively.\n + */\n +#define REFTABLE_FORMAT_ID_SHA1 ((uint32_t) 0x73686131)\n +#define REFTABLE_FORMAT_ID_SHA256 ((uint32_t) 0x73323536)\n3: b595668a5cd ! 3: 745c1a070dd reftable/system: stop depending on \"hash.h\"\n @@ reftable/merged.h: license that can be found in the LICENSE file or at\n #define MERGED_H\n \n #include \"system.h\"\n -+#include \"basics.h\"\n ++#include \"reftable-basics.h\"\n \n struct reftable_merged_table {\n \tstruct reftable_reader **readers;\n4: 86269fc4fca ! 4: 7782652b975 reftable/stack: stop using `fsync_component()` directly\n @@ reftable/stack.c: static int stack_filename(struct reftable_buf *dest, struct re\n }\n \n -static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz)\n -+static int stack_fsync(struct reftable_stack *st, int fd)\n ++static int stack_fsync(const struct reftable_write_options *opts, int fd)\n {\n -\tint *fdp = (int *)arg;\n -\treturn write_in_full(*fdp, data, sz);\n -+\tif (st->opts.fsync)\n -+\t\treturn st->opts.fsync(fd);\n ++\tif (opts->fsync)\n ++\t\treturn opts->fsync(fd);\n +\treturn fsync(fd);\n }\n \n -static int reftable_fd_flush(void *arg)\n +struct fd_writer {\n -+\tstruct reftable_stack *stack;\n ++\tconst struct reftable_write_options *opts;\n +\tint fd;\n +};\n +\n @@ reftable/stack.c: static int stack_filename(struct reftable_buf *dest, struct re\n +static int fd_writer_flush(void *arg)\n +{\n +\tstruct fd_writer *writer = arg;\n -+\treturn stack_fsync(writer->stack, writer->fd);\n ++\treturn stack_fsync(writer->opts, writer->fd);\n }\n \n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n @@ reftable/stack.c: int reftable_addition_commit(struct reftable_addition *add)\n \t}\n \n -\terr = fsync_component(FSYNC_COMPONENT_REFERENCE, lock_file_fd);\n -+\terr = stack_fsync(add->stack, lock_file_fd);\n ++\terr = stack_fsync(&add->stack->opts, lock_file_fd);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n @@ reftable/stack.c: int reftable_addition_add(struct reftable_addition *add,\n \tstruct reftable_writer *wr = NULL;\n \tstruct tempfile *tab_file = NULL;\n +\tstruct fd_writer writer = {\n -+\t\t.stack = add->stack,\n ++\t\t.opts = &add->stack->opts,\n +\t};\n \tint err = 0;\n -\tint tab_fd;\n @@ reftable/stack.c: static int stack_compact_locked(struct reftable_stack *st,\n \tstruct reftable_buf tab_file_path = REFTABLE_BUF_INIT;\n \tstruct reftable_writer *wr = NULL;\n +\tstruct fd_writer writer= {\n -+\t\t.stack = st,\n ++\t\t.opts = &st->opts,\n +\t};\n \tstruct tempfile *tab_file;\n -\tint tab_fd, err = 0;\n @@ reftable/stack.c: static int stack_compact_range(struct reftable_stack *st,\n \t}\n \n -\terr = fsync_component(FSYNC_COMPONENT_REFERENCE, get_lock_file_fd(&tables_list_lock));\n -+\terr = stack_fsync(st, get_lock_file_fd(&tables_list_lock));\n ++\terr = stack_fsync(&st->opts, get_lock_file_fd(&tables_list_lock));\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tunlink(new_table_path.buf);\n5: aca19955560 ! 5: b15daefbc83 reftable/system: provide thin wrapper for tempfile subsystem\n @@ reftable/stack.c: int reftable_addition_add(struct reftable_addition *add,\n -\tstruct tempfile *tab_file = NULL;\n +\tstruct reftable_tmpfile tab_file = REFTABLE_TMPFILE_INIT;\n \tstruct fd_writer writer = {\n - \t\t.stack = add->stack,\n + \t\t.opts = &add->stack->opts,\n \t};\n @@ reftable/stack.c: int reftable_addition_add(struct reftable_addition *add,\n \tif (err < 0)\n @@ reftable/stack.c: uint64_t reftable_stack_next_update_index(struct reftable_stac\n \tstruct reftable_buf tab_file_path = REFTABLE_BUF_INIT;\n @@ reftable/stack.c: static int stack_compact_locked(struct reftable_stack *st,\n \tstruct fd_writer writer= {\n - \t\t.stack = st,\n + \t\t.opts = &st->opts,\n \t};\n -\tstruct tempfile *tab_file;\n +\tstruct reftable_tmpfile tab_file = REFTABLE_TMPFILE_INIT;\n6: 74afe30974d = 6: 83949837a29 reftable/stack: drop only use of `get_locked_file_path()`\n7: 71b213d6f8a ! 7: 80fe5bc5e10 reftable/system: provide thin wrapper for lockfile subsystem\n @@ reftable/stack.c: int reftable_addition_commit(struct reftable_addition *add)\n \t\tgoto done;\n \t}\n \n --\terr = stack_fsync(add->stack, lock_file_fd);\n -+\terr = stack_fsync(add->stack, add->tables_list_lock.fd);\n +-\terr = stack_fsync(&add->stack->opts, lock_file_fd);\n ++\terr = stack_fsync(&add->stack->opts, add->tables_list_lock.fd);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n @@ reftable/stack.c: static int stack_compact_range(struct reftable_stack *st,\n \t\tgoto done;\n \t}\n \n --\terr = stack_fsync(st, get_lock_file_fd(&tables_list_lock));\n -+\terr = stack_fsync(st, tables_list_lock.fd);\n +-\terr = stack_fsync(&st->opts, get_lock_file_fd(&tables_list_lock));\n ++\terr = stack_fsync(&st->opts, tables_list_lock.fd);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tunlink(new_table_path.buf);\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506830","messageId":"2b7d4e2852977659752f98b782784eaa6780bbdb.1731047193.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1731047193.git.ps@pks.im","subject":"[PATCH v2 1/7] reftable/system: move \"dir.h\" to its only user","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-08T08:17:09Z","receivedAt":"2024-11-08T08:17:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We still include \"dir.h\" in \"reftable/system.h\" even though it is not\nused by anything but by a single unit test. Move it over into that unit\ntest so that we don't accidentally use any functionality provided by it\nin the reftable codebase.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/system.h | 1 -\n t/unit-tests/t-reftable-stack.c | 1 +\n 2 files changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 5ec85833434..8564213475e 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -15,7 +15,6 @@ license that can be found in the LICENSE file or at\n #include \"lockfile.h\"\n #include \"tempfile.h\"\n #include \"hash.h\" /* hash ID, sizes.*/\n-#include \"dir.h\" /* remove_dir_recursively, for tests.*/\n \n int hash_size(uint32_t id);\n \ndiff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c\nindex 72f6747064f..1b4363a58fc 100644\n--- a/t/unit-tests/t-reftable-stack.c\n+++ b/t/unit-tests/t-reftable-stack.c\n@@ -8,6 +8,7 @@ license that can be found in the LICENSE file or at\n \n #include \"test-lib.h\"\n #include \"lib-reftable.h\"\n+#include \"dir.h\"\n #include \"reftable/merged.h\"\n #include \"reftable/reader.h\"\n #include \"reftable/reftable-error.h\"\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506831","messageId":"38cfe85bf5b82c70848e4b295ba6cae33dcfd667.1731047193.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1731047193.git.ps@pks.im","subject":"[PATCH v2 2/7] reftable: explicitly handle hash format IDs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-08T08:17:12Z","receivedAt":"2024-11-08T08:17:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The hash format IDs are used for two different things across the\nreftable codebase:\n\n - They are used as a 32 bit unsigned integer when reading and writing\n the header in order to identify the hash function.\n\n - They are used internally to identify which hash function is in use.\n\nWhen one only considers the second usecase one might think that one can\neasily change the representation of those hash IDs. But because those\nIDs end up in the reftable header and footer on disk it is important\nthat those never change.\n\nCreate separate constants `REFTABLE_FORMAT_ID_*` and use them in\ncontexts where we read or write reftable headers. This serves multiple\npurposes:\n\n - It allows us to more easily discern cases where we actually use\n those constants for the on-disk format.\n\n - It detangles us from the same constants that are defined in\n libgit.a, which is another required step to convert the reftable\n library to become standalone.\n\n - It makes the next step easier where we stop using `GIT_*_FORMAT_ID`\n constants in favor of a custom enum.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 8 ++++++++\n reftable/reader.c | 10 ++++++----\n reftable/writer.c | 16 +++++++++++++++-\n 3 files changed, 29 insertions(+), 5 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 7aa46d7c30d..bcab0b529b0 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -150,4 +150,12 @@ int common_prefix_size(struct reftable_buf *a, struct reftable_buf *b);\n \n int hash_size(uint32_t id);\n \n+/*\n+ * Format IDs that identify the hash function used by a reftable. Note that\n+ * these constants end up on disk and thus mustn't change. The format IDs are\n+ * \"sha1\" and \"s256\" in big endian, respectively.\n+ */\n+#define REFTABLE_FORMAT_ID_SHA1 ((uint32_t) 0x73686131)\n+#define REFTABLE_FORMAT_ID_SHA256 ((uint32_t) 0x73323536)\n+\n #endif\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 90dc950b577..64eb6938efe 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -109,16 +109,18 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n \tif (r->version == 1) {\n \t\tr->hash_id = GIT_SHA1_FORMAT_ID;\n \t} else {\n-\t\tr->hash_id = get_be32(f);\n-\t\tswitch (r->hash_id) {\n-\t\tcase GIT_SHA1_FORMAT_ID:\n+\t\tswitch (get_be32(f)) {\n+\t\tcase REFTABLE_FORMAT_ID_SHA1:\n+\t\t\tr->hash_id = GIT_SHA1_FORMAT_ID;\n \t\t\tbreak;\n-\t\tcase GIT_SHA256_FORMAT_ID:\n+\t\tcase REFTABLE_FORMAT_ID_SHA256:\n+\t\t\tr->hash_id = GIT_SHA256_FORMAT_ID;\n \t\t\tbreak;\n \t\tdefault:\n \t\t\terr = REFTABLE_FORMAT_ERROR;\n \t\t\tgoto done;\n \t\t}\n+\n \t\tf += 4;\n \t}\n \ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex fd136794d5a..9aa45de6340 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -103,8 +103,22 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \tput_be64(dest + 8, w->min_update_index);\n \tput_be64(dest + 16, w->max_update_index);\n \tif (writer_version(w) == 2) {\n-\t\tput_be32(dest + 24, w->opts.hash_id);\n+\t\tuint32_t hash_id;\n+\n+\t\tswitch (w->opts.hash_id) {\n+\t\tcase GIT_SHA1_FORMAT_ID:\n+\t\t\thash_id = REFTABLE_FORMAT_ID_SHA1;\n+\t\t\tbreak;\n+\t\tcase GIT_SHA256_FORMAT_ID:\n+\t\t\thash_id = REFTABLE_FORMAT_ID_SHA256;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tput_be32(dest + 24, hash_id);\n \t}\n+\n \treturn header_size(writer_version(w));\n }\n \n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506833","messageId":"745c1a070ddd7180f0ee89c6a1d057a256894599.1731047193.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1731047193.git.ps@pks.im","subject":"[PATCH v2 3/7] reftable/system: stop depending on \"hash.h\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-08T08:17:14Z","receivedAt":"2024-11-08T08:17:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We include \"hash.h\" in \"reftable/system.h\" such that we can use hash\nformat IDs as well as the raw size of SHA1 and SHA256. As we are in the\nprocess of converting the reftable library to become standalone we of\ncourse cannot rely on those constants anymore.\n\nIntroduce a new `enum reftable_hash` to replace internal uses of the\nhash format IDs and new constants that replace internal uses of the hash\nsize. Adapt the reftable backend to set up the correct hash function.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 12 +++++-\n reftable/basics.c | 13 ++++---\n reftable/basics.h | 2 +-\n reftable/merged.c | 4 +-\n reftable/merged.h | 3 +-\n reftable/reader.c | 8 ++--\n reftable/reader.h | 4 +-\n reftable/reftable-basics.h | 13 +++++++\n reftable/reftable-merged.h | 4 +-\n reftable/reftable-reader.h | 2 +-\n reftable/reftable-record.h | 12 +++---\n reftable/reftable-writer.h | 2 +-\n reftable/stack.c | 4 +-\n reftable/system.h | 3 --\n reftable/writer.c | 8 ++--\n t/helper/test-reftable.c | 4 +-\n t/unit-tests/lib-reftable.c | 4 +-\n t/unit-tests/lib-reftable.h | 2 +-\n t/unit-tests/t-reftable-block.c | 40 +++++++++----------\n t/unit-tests/t-reftable-merged.c | 26 ++++++-------\n t/unit-tests/t-reftable-pq.c | 2 +-\n t/unit-tests/t-reftable-reader.c | 4 +-\n t/unit-tests/t-reftable-readwrite.c | 40 +++++++++----------\n t/unit-tests/t-reftable-record.c | 59 +++++++++++++++--------------\n t/unit-tests/t-reftable-stack.c | 34 ++++++++---------\n 25 files changed, 166 insertions(+), 143 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 3c6107c7ce5..7d86d920970 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -15,6 +15,7 @@\n #include \"../object.h\"\n #include \"../path.h\"\n #include \"../refs.h\"\n+#include \"../reftable/reftable-basics.h\"\n #include \"../reftable/reftable-stack.h\"\n #include \"../reftable/reftable-record.h\"\n #include \"../reftable/reftable-error.h\"\n@@ -289,7 +290,16 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \trefs->store_flags = store_flags;\n \trefs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);\n \n-\trefs->write_options.hash_id = repo->hash_algo->format_id;\n+\tswitch (repo->hash_algo->format_id) {\n+\tcase GIT_SHA1_FORMAT_ID:\n+\t\trefs->write_options.hash_id = REFTABLE_HASH_SHA1;\n+\t\tbreak;\n+\tcase GIT_SHA256_FORMAT_ID:\n+\t\trefs->write_options.hash_id = REFTABLE_HASH_SHA256;\n+\t\tbreak;\n+\tdefault:\n+\t\tBUG(\"unknown hash algorithm %d\", repo->hash_algo->format_id);\n+\t}\n \trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n \trefs->write_options.disable_auto_compact =\n \t\t!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1);\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex bc4fcc91446..7d84a5d62de 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -271,14 +271,15 @@ int common_prefix_size(struct reftable_buf *a, struct reftable_buf *b)\n \treturn p;\n }\n \n-int hash_size(uint32_t id)\n+int hash_size(enum reftable_hash id)\n {\n+\tif (!id)\n+\t\treturn REFTABLE_HASH_SIZE_SHA1;\n \tswitch (id) {\n-\tcase 0:\n-\tcase GIT_SHA1_FORMAT_ID:\n-\t\treturn GIT_SHA1_RAWSZ;\n-\tcase GIT_SHA256_FORMAT_ID:\n-\t\treturn GIT_SHA256_RAWSZ;\n+\tcase REFTABLE_HASH_SHA1:\n+\t\treturn REFTABLE_HASH_SIZE_SHA1;\n+\tcase REFTABLE_HASH_SHA256:\n+\t\treturn REFTABLE_HASH_SIZE_SHA256;\n \t}\n \tabort();\n }\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex bcab0b529b0..36beda2c25a 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -148,7 +148,7 @@ char *reftable_strdup(const char *str);\n /* Find the longest shared prefix size of `a` and `b` */\n int common_prefix_size(struct reftable_buf *a, struct reftable_buf *b);\n \n-int hash_size(uint32_t id);\n+int hash_size(enum reftable_hash id);\n \n /*\n * Format IDs that identify the hash function used by a reftable. Note that\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 514d6facf45..5b93e20f429 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -181,7 +181,7 @@ static void iterator_from_merged_iter(struct reftable_iterator *it,\n \n int reftable_merged_table_new(struct reftable_merged_table **dest,\n \t\t\t struct reftable_reader **readers, size_t n,\n-\t\t\t uint32_t hash_id)\n+\t\t\t enum reftable_hash hash_id)\n {\n \tstruct reftable_merged_table *m = NULL;\n \tuint64_t last_max = 0;\n@@ -293,7 +293,7 @@ int reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n \treturn merged_table_init_iter(mt, it, BLOCK_TYPE_LOG);\n }\n \n-uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\n+enum reftable_hash reftable_merged_table_hash_id(struct reftable_merged_table *mt)\n {\n \treturn mt->hash_id;\n }\ndiff --git a/reftable/merged.h b/reftable/merged.h\nindex 89bd0c4b35b..0b7d939e92b 100644\n--- a/reftable/merged.h\n+++ b/reftable/merged.h\n@@ -10,11 +10,12 @@ license that can be found in the LICENSE file or at\n #define MERGED_H\n \n #include \"system.h\"\n+#include \"reftable-basics.h\"\n \n struct reftable_merged_table {\n \tstruct reftable_reader **readers;\n \tsize_t readers_len;\n-\tuint32_t hash_id;\n+\tenum reftable_hash hash_id;\n \n \t/* If unset, produce deletions. This is useful for compaction. For the\n \t * full stack, deletions should be produced. */\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 64eb6938efe..ea82955c9bc 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -67,7 +67,7 @@ static int reader_get_block(struct reftable_reader *r,\n \treturn block_source_read_block(&r->source, dest, off, sz);\n }\n \n-uint32_t reftable_reader_hash_id(struct reftable_reader *r)\n+enum reftable_hash reftable_reader_hash_id(struct reftable_reader *r)\n {\n \treturn r->hash_id;\n }\n@@ -107,14 +107,14 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n \tf += 8;\n \n \tif (r->version == 1) {\n-\t\tr->hash_id = GIT_SHA1_FORMAT_ID;\n+\t\tr->hash_id = REFTABLE_HASH_SHA1;\n \t} else {\n \t\tswitch (get_be32(f)) {\n \t\tcase REFTABLE_FORMAT_ID_SHA1:\n-\t\t\tr->hash_id = GIT_SHA1_FORMAT_ID;\n+\t\t\tr->hash_id = REFTABLE_HASH_SHA1;\n \t\t\tbreak;\n \t\tcase REFTABLE_FORMAT_ID_SHA256:\n-\t\t\tr->hash_id = GIT_SHA256_FORMAT_ID;\n+\t\t\tr->hash_id = REFTABLE_HASH_SHA256;\n \t\t\tbreak;\n \t\tdefault:\n \t\t\terr = REFTABLE_FORMAT_ERROR;\ndiff --git a/reftable/reader.h b/reftable/reader.h\nindex 010fbfe8511..d2b48a48499 100644\n--- a/reftable/reader.h\n+++ b/reftable/reader.h\n@@ -37,8 +37,8 @@ struct reftable_reader {\n \t/* Size of the file, excluding the footer. */\n \tuint64_t size;\n \n-\t/* 'sha1' for SHA1, 's256' for SHA-256 */\n-\tuint32_t hash_id;\n+\t/* The hash function used for ref records. */\n+\tenum reftable_hash hash_id;\n \n \tuint32_t block_size;\n \tuint64_t min_update_index;\ndiff --git a/reftable/reftable-basics.h b/reftable/reftable-basics.h\nindex 6e8e636b716..e0397ed5836 100644\n--- a/reftable/reftable-basics.h\n+++ b/reftable/reftable-basics.h\n@@ -11,6 +11,19 @@\n \n #include \n \n+/*\n+ * Hash functions understood by the reftable library. Note that the values are\n+ * arbitrary and somewhat random such that we can easily detect cases where the\n+ * hash hasn't been properly set up.\n+ */\n+enum reftable_hash {\n+\tREFTABLE_HASH_SHA1 = 89,\n+\tREFTABLE_HASH_SHA256 = 247,\n+};\n+#define REFTABLE_HASH_SIZE_SHA1 20\n+#define REFTABLE_HASH_SIZE_SHA256 32\n+#define REFTABLE_HASH_SIZE_MAX REFTABLE_HASH_SIZE_SHA256\n+\n /* Overrides the functions to use for memory management. */\n void reftable_set_alloc(void *(*malloc)(size_t),\n \t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nindex a970d5dd89a..f2d01c3ef82 100644\n--- a/reftable/reftable-merged.h\n+++ b/reftable/reftable-merged.h\n@@ -34,7 +34,7 @@ struct reftable_reader;\n */\n int reftable_merged_table_new(struct reftable_merged_table **dest,\n \t\t\t struct reftable_reader **readers, size_t n,\n-\t\t\t uint32_t hash_id);\n+\t\t\t enum reftable_hash hash_id);\n \n /* Initialize a merged table iterator for reading refs. */\n int reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n@@ -56,6 +56,6 @@ reftable_merged_table_min_update_index(struct reftable_merged_table *mt);\n void reftable_merged_table_free(struct reftable_merged_table *m);\n \n /* return the hash ID of the merged table. */\n-uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m);\n+enum reftable_hash reftable_merged_table_hash_id(struct reftable_merged_table *m);\n \n #endif\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nindex 6a2d0b693f5..0085fbb9032 100644\n--- a/reftable/reftable-reader.h\n+++ b/reftable/reftable-reader.h\n@@ -54,7 +54,7 @@ int reftable_reader_init_log_iterator(struct reftable_reader *r,\n \t\t\t\t struct reftable_iterator *it);\n \n /* returns the hash ID used in this table. */\n-uint32_t reftable_reader_hash_id(struct reftable_reader *r);\n+enum reftable_hash reftable_reader_hash_id(struct reftable_reader *r);\n \n /* return an iterator for the refs pointing to `oid`. */\n int reftable_reader_refs_for(struct reftable_reader *r,\ndiff --git a/reftable/reftable-record.h b/reftable/reftable-record.h\nindex 2d42463c581..ddd48eb5798 100644\n--- a/reftable/reftable-record.h\n+++ b/reftable/reftable-record.h\n@@ -9,7 +9,7 @@ license that can be found in the LICENSE file or at\n #ifndef REFTABLE_RECORD_H\n #define REFTABLE_RECORD_H\n \n-#include \"hash.h\"\n+#include \"reftable-basics.h\"\n #include \n \n /*\n@@ -40,10 +40,10 @@ struct reftable_ref_record {\n #define REFTABLE_NR_REF_VALUETYPES 4\n \t} value_type;\n \tunion {\n-\t\tunsigned char val1[GIT_MAX_RAWSZ];\n+\t\tunsigned char val1[REFTABLE_HASH_SIZE_MAX];\n \t\tstruct {\n-\t\t\tunsigned char value[GIT_MAX_RAWSZ]; /* first hash */\n-\t\t\tunsigned char target_value[GIT_MAX_RAWSZ]; /* second hash */\n+\t\t\tunsigned char value[REFTABLE_HASH_SIZE_MAX]; /* first hash */\n+\t\t\tunsigned char target_value[REFTABLE_HASH_SIZE_MAX]; /* second hash */\n \t\t} val2;\n \t\tchar *symref; /* referent, malloced 0-terminated string */\n \t} value;\n@@ -85,8 +85,8 @@ struct reftable_log_record {\n \n \tunion {\n \t\tstruct {\n-\t\t\tunsigned char new_hash[GIT_MAX_RAWSZ];\n-\t\t\tunsigned char old_hash[GIT_MAX_RAWSZ];\n+\t\t\tunsigned char new_hash[REFTABLE_HASH_SIZE_MAX];\n+\t\t\tunsigned char old_hash[REFTABLE_HASH_SIZE_MAX];\n \t\t\tchar *name;\n \t\t\tchar *email;\n \t\t\tuint64_t time;\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex e4fc9537883..211860d08a4 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -33,7 +33,7 @@ struct reftable_write_options {\n \t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n \t * Defaults to SHA1 if unset\n \t */\n-\tuint32_t hash_id;\n+\tenum reftable_hash hash_id;\n \n \t/* Default mode for creating files. If unset, use 0666 (+umask) */\n \tunsigned int default_permissions;\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex c33979536ef..9ae716ff375 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -73,7 +73,7 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \tif (_opts)\n \t\topts = *_opts;\n \tif (opts.hash_id == 0)\n-\t\topts.hash_id = GIT_SHA1_FORMAT_ID;\n+\t\topts.hash_id = REFTABLE_HASH_SHA1;\n \n \t*dest = NULL;\n \n@@ -1603,7 +1603,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n,\n \n static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n {\n-\tint version = (st->opts.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2;\n+\tint version = (st->opts.hash_id == REFTABLE_HASH_SHA1) ? 1 : 2;\n \tint overhead = header_size(version) - 1;\n \tuint64_t *sizes;\n \ndiff --git a/reftable/system.h b/reftable/system.h\nindex 8564213475e..38d3534620e 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -14,8 +14,5 @@ license that can be found in the LICENSE file or at\n #include \"git-compat-util.h\"\n #include \"lockfile.h\"\n #include \"tempfile.h\"\n-#include \"hash.h\" /* hash ID, sizes.*/\n-\n-int hash_size(uint32_t id);\n \n #endif\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 9aa45de6340..ea2f831fc58 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -79,7 +79,7 @@ static void options_set_defaults(struct reftable_write_options *opts)\n \t}\n \n \tif (opts->hash_id == 0) {\n-\t\topts->hash_id = GIT_SHA1_FORMAT_ID;\n+\t\topts->hash_id = REFTABLE_HASH_SHA1;\n \t}\n \tif (opts->block_size == 0) {\n \t\topts->block_size = DEFAULT_BLOCK_SIZE;\n@@ -88,7 +88,7 @@ static void options_set_defaults(struct reftable_write_options *opts)\n \n static int writer_version(struct reftable_writer *w)\n {\n-\treturn (w->opts.hash_id == 0 || w->opts.hash_id == GIT_SHA1_FORMAT_ID) ?\n+\treturn (w->opts.hash_id == 0 || w->opts.hash_id == REFTABLE_HASH_SHA1) ?\n \t\t\t 1 :\n \t\t\t 2;\n }\n@@ -106,10 +106,10 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \t\tuint32_t hash_id;\n \n \t\tswitch (w->opts.hash_id) {\n-\t\tcase GIT_SHA1_FORMAT_ID:\n+\t\tcase REFTABLE_HASH_SHA1:\n \t\t\thash_id = REFTABLE_FORMAT_ID_SHA1;\n \t\t\tbreak;\n-\t\tcase GIT_SHA256_FORMAT_ID:\n+\t\tcase REFTABLE_HASH_SHA256:\n \t\t\thash_id = REFTABLE_FORMAT_ID_SHA256;\n \t\t\tbreak;\n \t\tdefault:\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 5c8849d115b..3c72ed985b3 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -156,7 +156,7 @@ int cmd__dump_reftable(int argc, const char **argv)\n \tint opt_dump_blocks = 0;\n \tint opt_dump_table = 0;\n \tint opt_dump_stack = 0;\n-\tuint32_t opt_hash_id = GIT_SHA1_FORMAT_ID;\n+\tuint32_t opt_hash_id = REFTABLE_HASH_SHA1;\n \tconst char *arg = NULL, *argv0 = argv[0];\n \n \tfor (; argc > 1; argv++, argc--)\n@@ -167,7 +167,7 @@ int cmd__dump_reftable(int argc, const char **argv)\n \t\telse if (!strcmp(\"-t\", argv[1]))\n \t\t\topt_dump_table = 1;\n \t\telse if (!strcmp(\"-6\", argv[1]))\n-\t\t\topt_hash_id = GIT_SHA256_FORMAT_ID;\n+\t\t\topt_hash_id = REFTABLE_HASH_SHA256;\n \t\telse if (!strcmp(\"-s\", argv[1]))\n \t\t\topt_dump_stack = 1;\n \t\telse if (!strcmp(\"-?\", argv[1]) || !strcmp(\"-h\", argv[1])) {\ndiff --git a/t/unit-tests/lib-reftable.c b/t/unit-tests/lib-reftable.c\nindex 2ddf480588d..c1631f45275 100644\n--- a/t/unit-tests/lib-reftable.c\n+++ b/t/unit-tests/lib-reftable.c\n@@ -3,7 +3,7 @@\n #include \"reftable/constants.h\"\n #include \"reftable/writer.h\"\n \n-void t_reftable_set_hash(uint8_t *p, int i, uint32_t id)\n+void t_reftable_set_hash(uint8_t *p, int i, enum reftable_hash id)\n {\n \tmemset(p, (uint8_t)i, hash_size(id));\n }\n@@ -82,7 +82,7 @@ void t_reftable_write_to_buf(struct reftable_buf *buf,\n \t\tsize_t off = i * (opts.block_size ? opts.block_size\n \t\t\t\t\t\t : DEFAULT_BLOCK_SIZE);\n \t\tif (!off)\n-\t\t\toff = header_size(opts.hash_id == GIT_SHA256_FORMAT_ID ? 2 : 1);\n+\t\t\toff = header_size(opts.hash_id == REFTABLE_HASH_SHA256 ? 2 : 1);\n \t\tcheck_char(buf->buf[off], ==, 'r');\n \t}\n \ndiff --git a/t/unit-tests/lib-reftable.h b/t/unit-tests/lib-reftable.h\nindex d4950fed3da..e4c360fa7ee 100644\n--- a/t/unit-tests/lib-reftable.h\n+++ b/t/unit-tests/lib-reftable.h\n@@ -6,7 +6,7 @@\n \n struct reftable_buf;\n \n-void t_reftable_set_hash(uint8_t *p, int i, uint32_t id);\n+void t_reftable_set_hash(uint8_t *p, int i, enum reftable_hash id);\n \n struct reftable_writer *t_reftable_strbuf_writer(struct reftable_buf *buf,\n \t\t\t\t\t\t struct reftable_write_options *opts);\ndiff --git a/t/unit-tests/t-reftable-block.c b/t/unit-tests/t-reftable-block.c\nindex f9af907117b..13e10807dae 100644\n--- a/t/unit-tests/t-reftable-block.c\n+++ b/t/unit-tests/t-reftable-block.c\n@@ -36,7 +36,7 @@ static void t_ref_block_read_write(void)\n \tblock.len = block_size;\n \tblock_source_from_buf(&block.source ,&buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n-\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\t\t\t\theader_off, hash_size(REFTABLE_HASH_SHA1));\n \tcheck(!ret);\n \n \trec.u.ref.refname = (char *) \"\";\n@@ -47,7 +47,7 @@ static void t_ref_block_read_write(void)\n \tfor (i = 0; i < N; i++) {\n \t\trec.u.ref.refname = xstrfmt(\"branch%02\"PRIuMAX, (uintmax_t)i);\n \t\trec.u.ref.value_type = REFTABLE_REF_VAL1;\n-\t\tmemset(rec.u.ref.value.val1, i, GIT_SHA1_RAWSZ);\n+\t\tmemset(rec.u.ref.value.val1, i, REFTABLE_HASH_SIZE_SHA1);\n \n \t\trecs[i] = rec;\n \t\tret = block_writer_add(&bw, &rec);\n@@ -61,7 +61,7 @@ static void t_ref_block_read_write(void)\n \n \tblock_writer_release(&bw);\n \n-\tblock_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ);\n+\tblock_reader_init(&br, &block, header_off, block_size, REFTABLE_HASH_SIZE_SHA1);\n \n \tblock_iter_seek_start(&it, &br);\n \n@@ -72,7 +72,7 @@ static void t_ref_block_read_write(void)\n \t\t\tcheck_int(i, ==, N);\n \t\t\tbreak;\n \t\t}\n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -85,7 +85,7 @@ static void t_ref_block_read_write(void)\n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n \n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \n \t\twant.len--;\n \t\tret = block_iter_seek_key(&it, &br, &want);\n@@ -93,7 +93,7 @@ static void t_ref_block_read_write(void)\n \n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n-\t\tcheck(reftable_record_equal(&recs[10 * (i / 10)], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[10 * (i / 10)], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tblock_reader_release(&br);\n@@ -130,7 +130,7 @@ static void t_log_block_read_write(void)\n \tblock.len = block_size;\n \tblock_source_from_buf(&block.source ,&buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size,\n-\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\t\t\t\theader_off, hash_size(REFTABLE_HASH_SHA1));\n \tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n@@ -150,7 +150,7 @@ static void t_log_block_read_write(void)\n \n \tblock_writer_release(&bw);\n \n-\tblock_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ);\n+\tblock_reader_init(&br, &block, header_off, block_size, REFTABLE_HASH_SIZE_SHA1);\n \n \tblock_iter_seek_start(&it, &br);\n \n@@ -161,7 +161,7 @@ static void t_log_block_read_write(void)\n \t\t\tcheck_int(i, ==, N);\n \t\t\tbreak;\n \t\t}\n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -175,7 +175,7 @@ static void t_log_block_read_write(void)\n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n \n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \n \t\twant.len--;\n \t\tret = block_iter_seek_key(&it, &br, &want);\n@@ -183,7 +183,7 @@ static void t_log_block_read_write(void)\n \n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n-\t\tcheck(reftable_record_equal(&recs[10 * (i / 10)], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[10 * (i / 10)], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tblock_reader_release(&br);\n@@ -220,7 +220,7 @@ static void t_obj_block_read_write(void)\n \tblock.len = block_size;\n \tblock_source_from_buf(&block.source, &buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size,\n-\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\t\t\t\theader_off, hash_size(REFTABLE_HASH_SHA1));\n \tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n@@ -242,7 +242,7 @@ static void t_obj_block_read_write(void)\n \n \tblock_writer_release(&bw);\n \n-\tblock_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ);\n+\tblock_reader_init(&br, &block, header_off, block_size, REFTABLE_HASH_SIZE_SHA1);\n \n \tblock_iter_seek_start(&it, &br);\n \n@@ -253,7 +253,7 @@ static void t_obj_block_read_write(void)\n \t\t\tcheck_int(i, ==, N);\n \t\t\tbreak;\n \t\t}\n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -266,7 +266,7 @@ static void t_obj_block_read_write(void)\n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n \n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tblock_reader_release(&br);\n@@ -304,7 +304,7 @@ static void t_index_block_read_write(void)\n \tblock.len = block_size;\n \tblock_source_from_buf(&block.source, &buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size,\n-\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\t\t\t\theader_off, hash_size(REFTABLE_HASH_SHA1));\n \tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n@@ -326,7 +326,7 @@ static void t_index_block_read_write(void)\n \n \tblock_writer_release(&bw);\n \n-\tblock_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ);\n+\tblock_reader_init(&br, &block, header_off, block_size, REFTABLE_HASH_SIZE_SHA1);\n \n \tblock_iter_seek_start(&it, &br);\n \n@@ -337,7 +337,7 @@ static void t_index_block_read_write(void)\n \t\t\tcheck_int(i, ==, N);\n \t\t\tbreak;\n \t\t}\n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -350,7 +350,7 @@ static void t_index_block_read_write(void)\n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n \n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \n \t\twant.len--;\n \t\tret = block_iter_seek_key(&it, &br, &want);\n@@ -358,7 +358,7 @@ static void t_index_block_read_write(void)\n \n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n-\t\tcheck(reftable_record_equal(&recs[10 * (i / 10)], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[10 * (i / 10)], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tblock_reader_release(&br);\ndiff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c\nindex 484c18251f3..0573d9470a6 100644\n--- a/t/unit-tests/t-reftable-merged.c\n+++ b/t/unit-tests/t-reftable-merged.c\n@@ -42,7 +42,7 @@ merged_table_from_records(struct reftable_ref_record **refs,\n \t\tcheck(!err);\n \t}\n \n-\terr = reftable_merged_table_new(&mt, *readers, n, GIT_SHA1_FORMAT_ID);\n+\terr = reftable_merged_table_new(&mt, *readers, n, REFTABLE_HASH_SHA1);\n \tcheck(!err);\n \treturn mt;\n }\n@@ -91,7 +91,7 @@ static void t_merged_single_record(void)\n \n \terr = reftable_iterator_next_ref(&it, &ref);\n \tcheck(!err);\n-\tcheck(reftable_ref_record_equal(&r2[0], &ref, GIT_SHA1_RAWSZ));\n+\tcheck(reftable_ref_record_equal(&r2[0], &ref, REFTABLE_HASH_SIZE_SHA1));\n \treftable_ref_record_release(&ref);\n \treftable_iterator_destroy(&it);\n \treaders_destroy(readers, 3);\n@@ -168,7 +168,7 @@ static void t_merged_refs(void)\n \tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"a\");\n \tcheck(!err);\n-\tcheck_int(reftable_merged_table_hash_id(mt), ==, GIT_SHA1_FORMAT_ID);\n+\tcheck_int(reftable_merged_table_hash_id(mt), ==, REFTABLE_HASH_SHA1);\n \tcheck_int(reftable_merged_table_min_update_index(mt), ==, 1);\n \tcheck_int(reftable_merged_table_max_update_index(mt), ==, 3);\n \n@@ -186,7 +186,7 @@ static void t_merged_refs(void)\n \tcheck_int(ARRAY_SIZE(want), ==, len);\n \tfor (i = 0; i < len; i++)\n \t\tcheck(reftable_ref_record_equal(want[i], &out[i],\n-\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1));\n \tfor (i = 0; i < len; i++)\n \t\treftable_ref_record_release(&out[i]);\n \treftable_free(out);\n@@ -252,12 +252,12 @@ static void t_merged_seek_multiple_times(void)\n \n \t\terr = reftable_iterator_next_ref(&it, &rec);\n \t\tcheck(!err);\n-\t\terr = reftable_ref_record_equal(&rec, &r1[1], GIT_SHA1_RAWSZ);\n+\t\terr = reftable_ref_record_equal(&rec, &r1[1], REFTABLE_HASH_SIZE_SHA1);\n \t\tcheck(err == 1);\n \n \t\terr = reftable_iterator_next_ref(&it, &rec);\n \t\tcheck(!err);\n-\t\terr = reftable_ref_record_equal(&rec, &r2[1], GIT_SHA1_RAWSZ);\n+\t\terr = reftable_ref_record_equal(&rec, &r2[1], REFTABLE_HASH_SIZE_SHA1);\n \t\tcheck(err == 1);\n \n \t\terr = reftable_iterator_next_ref(&it, &rec);\n@@ -300,7 +300,7 @@ merged_table_from_log_records(struct reftable_log_record **logs,\n \t\tcheck(!err);\n \t}\n \n-\terr = reftable_merged_table_new(&mt, *readers, n, GIT_SHA1_FORMAT_ID);\n+\terr = reftable_merged_table_new(&mt, *readers, n, REFTABLE_HASH_SHA1);\n \tcheck(!err);\n \treturn mt;\n }\n@@ -377,7 +377,7 @@ static void t_merged_logs(void)\n \tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"a\");\n \tcheck(!err);\n-\tcheck_int(reftable_merged_table_hash_id(mt), ==, GIT_SHA1_FORMAT_ID);\n+\tcheck_int(reftable_merged_table_hash_id(mt), ==, REFTABLE_HASH_SHA1);\n \tcheck_int(reftable_merged_table_min_update_index(mt), ==, 1);\n \tcheck_int(reftable_merged_table_max_update_index(mt), ==, 3);\n \n@@ -395,7 +395,7 @@ static void t_merged_logs(void)\n \tcheck_int(ARRAY_SIZE(want), ==, len);\n \tfor (i = 0; i < len; i++)\n \t\tcheck(reftable_log_record_equal(want[i], &out[i],\n-\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1));\n \n \terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n \tcheck(!err);\n@@ -404,7 +404,7 @@ static void t_merged_logs(void)\n \treftable_log_record_release(&out[0]);\n \terr = reftable_iterator_next_log(&it, &out[0]);\n \tcheck(!err);\n-\tcheck(reftable_log_record_equal(&out[0], &r3[0], GIT_SHA1_RAWSZ));\n+\tcheck(reftable_log_record_equal(&out[0], &r3[0], REFTABLE_HASH_SIZE_SHA1));\n \treftable_iterator_destroy(&it);\n \n \tfor (i = 0; i < len; i++)\n@@ -448,11 +448,11 @@ static void t_default_write_opts(void)\n \tcheck(!err);\n \n \thash_id = reftable_reader_hash_id(rd);\n-\tcheck_int(hash_id, ==, GIT_SHA1_FORMAT_ID);\n+\tcheck_int(hash_id, ==, REFTABLE_HASH_SHA1);\n \n-\terr = reftable_merged_table_new(&merged, &rd, 1, GIT_SHA256_FORMAT_ID);\n+\terr = reftable_merged_table_new(&merged, &rd, 1, REFTABLE_HASH_SHA256);\n \tcheck_int(err, ==, REFTABLE_FORMAT_ERROR);\n-\terr = reftable_merged_table_new(&merged, &rd, 1, GIT_SHA1_FORMAT_ID);\n+\terr = reftable_merged_table_new(&merged, &rd, 1, REFTABLE_HASH_SHA1);\n \tcheck(!err);\n \n \treftable_reader_decref(rd);\ndiff --git a/t/unit-tests/t-reftable-pq.c b/t/unit-tests/t-reftable-pq.c\nindex ada4c19f18a..272da05bea6 100644\n--- a/t/unit-tests/t-reftable-pq.c\n+++ b/t/unit-tests/t-reftable-pq.c\n@@ -132,7 +132,7 @@ static void t_merged_iter_pqueue_top(void)\n \n \t\tmerged_iter_pqueue_check(&pq);\n \t\tcheck(pq_entry_equal(&top, &e));\n-\t\tcheck(reftable_record_equal(top.rec, &recs[i], GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(top.rec, &recs[i], REFTABLE_HASH_SIZE_SHA1));\n \t\tfor (size_t j = 0; i < pq.len; j++) {\n \t\t\tcheck(pq_less(&top, &pq.heap[j]));\n \t\t\tcheck_int(top.index, >, j);\ndiff --git a/t/unit-tests/t-reftable-reader.c b/t/unit-tests/t-reftable-reader.c\nindex 19cb53b6415..546df6005e4 100644\n--- a/t/unit-tests/t-reftable-reader.c\n+++ b/t/unit-tests/t-reftable-reader.c\n@@ -31,7 +31,7 @@ static int t_reader_seek_once(void)\n \tret = reftable_iterator_next_ref(&it, &ref);\n \tcheck(!ret);\n \n-\tret = reftable_ref_record_equal(&ref, &records[0], GIT_SHA1_RAWSZ);\n+\tret = reftable_ref_record_equal(&ref, &records[0], REFTABLE_HASH_SIZE_SHA1);\n \tcheck_int(ret, ==, 1);\n \n \tret = reftable_iterator_next_ref(&it, &ref);\n@@ -74,7 +74,7 @@ static int t_reader_reseek(void)\n \t\tret = reftable_iterator_next_ref(&it, &ref);\n \t\tcheck(!ret);\n \n-\t\tret = reftable_ref_record_equal(&ref, &records[0], GIT_SHA1_RAWSZ);\n+\t\tret = reftable_ref_record_equal(&ref, &records[0], REFTABLE_HASH_SIZE_SHA1);\n \t\tcheck_int(ret, ==, 1);\n \n \t\tret = reftable_iterator_next_ref(&it, &ref);\ndiff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c\nindex d279b86df0a..57896922eb1 100644\n--- a/t/unit-tests/t-reftable-readwrite.c\n+++ b/t/unit-tests/t-reftable-readwrite.c\n@@ -41,7 +41,7 @@ static void t_buffer(void)\n }\n \n static void write_table(char ***names, struct reftable_buf *buf, int N,\n-\t\t\tint block_size, uint32_t hash_id)\n+\t\t\tint block_size, enum reftable_hash hash_id)\n {\n \tstruct reftable_write_options opts = {\n \t\t.block_size = block_size,\n@@ -62,7 +62,7 @@ static void write_table(char ***names, struct reftable_buf *buf, int N,\n \t\trefs[i].refname = (*names)[i] = xstrfmt(\"refs/heads/branch%02d\", i);\n \t\trefs[i].update_index = update_index;\n \t\trefs[i].value_type = REFTABLE_REF_VAL1;\n-\t\tt_reftable_set_hash(refs[i].value.val1, i, GIT_SHA1_FORMAT_ID);\n+\t\tt_reftable_set_hash(refs[i].value.val1, i, REFTABLE_HASH_SHA1);\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -70,7 +70,7 @@ static void write_table(char ***names, struct reftable_buf *buf, int N,\n \t\tlogs[i].update_index = update_index;\n \t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n \t\tt_reftable_set_hash(logs[i].value.update.new_hash, i,\n-\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t REFTABLE_HASH_SHA1);\n \t\tlogs[i].value.update.message = (char *) \"message\";\n \t}\n \n@@ -104,7 +104,7 @@ static void t_log_buffer_size(void)\n \t/* This tests buffer extension for log compression. Must use a random\n \t hash, to ensure that the compressed part is larger than the original.\n \t*/\n-\tfor (i = 0; i < GIT_SHA1_RAWSZ; i++) {\n+\tfor (i = 0; i < REFTABLE_HASH_SIZE_SHA1; i++) {\n \t\tlog.value.update.old_hash[i] = (uint8_t)(git_rand() % 256);\n \t\tlog.value.update.new_hash[i] = (uint8_t)(git_rand() % 256);\n \t}\n@@ -191,9 +191,9 @@ static void t_log_write_read(void)\n \t\tlog.update_index = i;\n \t\tlog.value_type = REFTABLE_LOG_UPDATE;\n \t\tt_reftable_set_hash(log.value.update.old_hash, i,\n-\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t REFTABLE_HASH_SHA1);\n \t\tt_reftable_set_hash(log.value.update.new_hash, i + 1,\n-\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t REFTABLE_HASH_SHA1);\n \n \t\terr = reftable_writer_add_log(w, &log);\n \t\tcheck(!err);\n@@ -326,7 +326,7 @@ static void t_table_read_write_sequential(void)\n \tint err = 0;\n \tint j = 0;\n \n-\twrite_table(&names, &buf, N, 256, GIT_SHA1_FORMAT_ID);\n+\twrite_table(&names, &buf, N, 256, REFTABLE_HASH_SHA1);\n \n \tblock_source_from_buf(&source, &buf);\n \n@@ -361,7 +361,7 @@ static void t_table_write_small_table(void)\n \tchar **names;\n \tstruct reftable_buf buf = REFTABLE_BUF_INIT;\n \tint N = 1;\n-\twrite_table(&names, &buf, N, 4096, GIT_SHA1_FORMAT_ID);\n+\twrite_table(&names, &buf, N, 4096, REFTABLE_HASH_SHA1);\n \tcheck_int(buf.len, <, 200);\n \treftable_buf_release(&buf);\n \tfree_names(names);\n@@ -378,7 +378,7 @@ static void t_table_read_api(void)\n \tstruct reftable_log_record log = { 0 };\n \tstruct reftable_iterator it = { 0 };\n \n-\twrite_table(&names, &buf, N, 256, GIT_SHA1_FORMAT_ID);\n+\twrite_table(&names, &buf, N, 256, REFTABLE_HASH_SHA1);\n \n \tblock_source_from_buf(&source, &buf);\n \n@@ -400,7 +400,7 @@ static void t_table_read_api(void)\n \treftable_buf_release(&buf);\n }\n \n-static void t_table_read_write_seek(int index, int hash_id)\n+static void t_table_read_write_seek(int index, enum reftable_hash hash_id)\n {\n \tchar **names;\n \tstruct reftable_buf buf = REFTABLE_BUF_INIT;\n@@ -467,24 +467,24 @@ static void t_table_read_write_seek(int index, int hash_id)\n \n static void t_table_read_write_seek_linear(void)\n {\n-\tt_table_read_write_seek(0, GIT_SHA1_FORMAT_ID);\n+\tt_table_read_write_seek(0, REFTABLE_HASH_SHA1);\n }\n \n static void t_table_read_write_seek_linear_sha256(void)\n {\n-\tt_table_read_write_seek(0, GIT_SHA256_FORMAT_ID);\n+\tt_table_read_write_seek(0, REFTABLE_HASH_SHA256);\n }\n \n static void t_table_read_write_seek_index(void)\n {\n-\tt_table_read_write_seek(1, GIT_SHA1_FORMAT_ID);\n+\tt_table_read_write_seek(1, REFTABLE_HASH_SHA1);\n }\n \n static void t_table_refs_for(int indexed)\n {\n \tchar **want_names;\n \tint want_names_len = 0;\n-\tuint8_t want_hash[GIT_SHA1_RAWSZ];\n+\tuint8_t want_hash[REFTABLE_HASH_SIZE_SHA1];\n \n \tstruct reftable_write_options opts = {\n \t\t.block_size = 256,\n@@ -500,10 +500,10 @@ static void t_table_refs_for(int indexed)\n \twant_names = reftable_calloc(N + 1, sizeof(*want_names));\n \tcheck(want_names != NULL);\n \n-\tt_reftable_set_hash(want_hash, 4, GIT_SHA1_FORMAT_ID);\n+\tt_reftable_set_hash(want_hash, 4, REFTABLE_HASH_SHA1);\n \n \tfor (i = 0; i < N; i++) {\n-\t\tuint8_t hash[GIT_SHA1_RAWSZ];\n+\t\tuint8_t hash[REFTABLE_HASH_SIZE_SHA1];\n \t\tchar fill[51] = { 0 };\n \t\tchar name[100];\n \t\tstruct reftable_ref_record ref = { 0 };\n@@ -517,9 +517,9 @@ static void t_table_refs_for(int indexed)\n \n \t\tref.value_type = REFTABLE_REF_VAL2;\n \t\tt_reftable_set_hash(ref.value.val2.value, i / 4,\n-\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t REFTABLE_HASH_SHA1);\n \t\tt_reftable_set_hash(ref.value.val2.target_value, 3 + i / 4,\n-\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t REFTABLE_HASH_SHA1);\n \n \t\t/* 80 bytes / entry, so 3 entries per block. Yields 17\n \t\t */\n@@ -527,8 +527,8 @@ static void t_table_refs_for(int indexed)\n \t\tn = reftable_writer_add_ref(w, &ref);\n \t\tcheck_int(n, ==, 0);\n \n-\t\tif (!memcmp(ref.value.val2.value, want_hash, GIT_SHA1_RAWSZ) ||\n-\t\t !memcmp(ref.value.val2.target_value, want_hash, GIT_SHA1_RAWSZ))\n+\t\tif (!memcmp(ref.value.val2.value, want_hash, REFTABLE_HASH_SIZE_SHA1) ||\n+\t\t !memcmp(ref.value.val2.target_value, want_hash, REFTABLE_HASH_SIZE_SHA1))\n \t\t\twant_names[want_names_len++] = xstrdup(name);\n \t}\n \ndiff --git a/t/unit-tests/t-reftable-record.c b/t/unit-tests/t-reftable-record.c\nindex eb98bf2da91..42bc64cec87 100644\n--- a/t/unit-tests/t-reftable-record.c\n+++ b/t/unit-tests/t-reftable-record.c\n@@ -7,6 +7,7 @@\n */\n \n #include \"test-lib.h\"\n+#include \"reftable/basics.h\"\n #include \"reftable/constants.h\"\n #include \"reftable/record.h\"\n \n@@ -17,10 +18,10 @@ static void t_copy(struct reftable_record *rec)\n \n \ttyp = reftable_record_type(rec);\n \treftable_record_init(©, typ);\n-\treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n+\treftable_record_copy_from(©, rec, REFTABLE_HASH_SIZE_SHA1);\n \t/* do it twice to catch memory leaks */\n-\treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n-\tcheck(reftable_record_equal(rec, ©, GIT_SHA1_RAWSZ));\n+\treftable_record_copy_from(©, rec, REFTABLE_HASH_SIZE_SHA1);\n+\tcheck(reftable_record_equal(rec, ©, REFTABLE_HASH_SIZE_SHA1));\n \n \treftable_record_release(©);\n }\n@@ -59,7 +60,7 @@ static void t_varint_roundtrip(void)\n \n static void set_hash(uint8_t *h, int j)\n {\n-\tfor (int i = 0; i < hash_size(GIT_SHA1_FORMAT_ID); i++)\n+\tfor (int i = 0; i < hash_size(REFTABLE_HASH_SHA1); i++)\n \t\th[i] = (j >> i) & 0xff;\n }\n \n@@ -84,14 +85,14 @@ static void t_reftable_ref_record_comparison(void)\n \t\t},\n \t};\n \n-\tcheck(!reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n \n-\tcheck(!reftable_record_equal(&in[1], &in[2], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n \tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n \n \tin[1].u.ref.value_type = in[0].u.ref.value_type;\n-\tcheck(reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n }\n \n@@ -155,15 +156,15 @@ static void t_reftable_ref_record_roundtrip(void)\n \t\tcheck_int(reftable_record_is_deletion(&in), ==, i == REFTABLE_REF_DELETION);\n \n \t\treftable_record_key(&in, &key);\n-\t\tn = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);\n+\t\tn = reftable_record_encode(&in, dest, REFTABLE_HASH_SIZE_SHA1);\n \t\tcheck_int(n, >, 0);\n \n \t\t/* decode into a non-zero reftable_record to test for leaks. */\n-\t\tm = reftable_record_decode(&out, key, i, dest, GIT_SHA1_RAWSZ, &scratch);\n+\t\tm = reftable_record_decode(&out, key, i, dest, REFTABLE_HASH_SIZE_SHA1, &scratch);\n \t\tcheck_int(n, ==, m);\n \n \t\tcheck(reftable_ref_record_equal(&in.u.ref, &out.u.ref,\n-\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_record_release(&in);\n \n \t\treftable_buf_release(&key);\n@@ -193,15 +194,15 @@ static void t_reftable_log_record_comparison(void)\n \t\t},\n \t};\n \n-\tcheck(!reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n-\tcheck(!reftable_record_equal(&in[1], &in[2], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n+\tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n \tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n \t/* comparison should be reversed for equal keys, because\n \t * comparison is now performed on the basis of update indices */\n \tcheck_int(reftable_record_cmp(&in[0], &in[1]), <, 0);\n \n \tin[1].u.log.update_index = in[0].u.log.update_index;\n-\tcheck(reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n }\n \n@@ -303,15 +304,15 @@ static void t_reftable_log_record_roundtrip(void)\n \n \t\treftable_record_key(&rec, &key);\n \n-\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\t\tn = reftable_record_encode(&rec, dest, REFTABLE_HASH_SIZE_SHA1);\n \t\tcheck_int(n, >=, 0);\n \t\tvaltype = reftable_record_val_type(&rec);\n \t\tm = reftable_record_decode(&out, key, valtype, dest,\n-\t\t\t\t\t GIT_SHA1_RAWSZ, &scratch);\n+\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1, &scratch);\n \t\tcheck_int(n, ==, m);\n \n \t\tcheck(reftable_log_record_equal(&in[i], &out.u.log,\n-\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_log_record_release(&in[i]);\n \t\treftable_buf_release(&key);\n \t\treftable_record_release(&out);\n@@ -380,20 +381,20 @@ static void t_reftable_obj_record_comparison(void)\n \t\t},\n \t};\n \n-\tcheck(!reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n \n-\tcheck(!reftable_record_equal(&in[1], &in[2], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n \tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n \n \tin[1].u.obj.offset_len = in[0].u.obj.offset_len;\n-\tcheck(reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n }\n \n static void t_reftable_obj_record_roundtrip(void)\n {\n-\tuint8_t testHash1[GIT_SHA1_RAWSZ] = { 1, 2, 3, 4, 0 };\n+\tuint8_t testHash1[REFTABLE_HASH_SIZE_SHA1] = { 1, 2, 3, 4, 0 };\n \tuint64_t till9[] = { 1, 2, 3, 4, 500, 600, 700, 800, 9000 };\n \tstruct reftable_obj_record recs[3] = {\n \t\t{\n@@ -435,14 +436,14 @@ static void t_reftable_obj_record_roundtrip(void)\n \t\tcheck(!reftable_record_is_deletion(&in));\n \t\tt_copy(&in);\n \t\treftable_record_key(&in, &key);\n-\t\tn = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);\n+\t\tn = reftable_record_encode(&in, dest, REFTABLE_HASH_SIZE_SHA1);\n \t\tcheck_int(n, >, 0);\n \t\textra = reftable_record_val_type(&in);\n \t\tm = reftable_record_decode(&out, key, extra, dest,\n-\t\t\t\t\t GIT_SHA1_RAWSZ, &scratch);\n+\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1, &scratch);\n \t\tcheck_int(n, ==, m);\n \n-\t\tcheck(reftable_record_equal(&in, &out, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&in, &out, REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_buf_release(&key);\n \t\treftable_record_release(&out);\n \t}\n@@ -473,14 +474,14 @@ static void t_reftable_index_record_comparison(void)\n \tcheck(!reftable_buf_addstr(&in[1].u.idx.last_key, \"refs/heads/master\"));\n \tcheck(!reftable_buf_addstr(&in[2].u.idx.last_key, \"refs/heads/branch\"));\n \n-\tcheck(!reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n \n-\tcheck(!reftable_record_equal(&in[1], &in[2], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n \tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n \n \tin[1].u.idx.offset = in[0].u.idx.offset;\n-\tcheck(reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n \n \tfor (size_t i = 0; i < ARRAY_SIZE(in); i++)\n@@ -516,15 +517,15 @@ static void t_reftable_index_record_roundtrip(void)\n \n \tcheck(!reftable_record_is_deletion(&in));\n \tcheck(!reftable_buf_cmp(&key, &in.u.idx.last_key));\n-\tn = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);\n+\tn = reftable_record_encode(&in, dest, REFTABLE_HASH_SIZE_SHA1);\n \tcheck_int(n, >, 0);\n \n \textra = reftable_record_val_type(&in);\n-\tm = reftable_record_decode(&out, key, extra, dest, GIT_SHA1_RAWSZ,\n+\tm = reftable_record_decode(&out, key, extra, dest, REFTABLE_HASH_SIZE_SHA1,\n \t\t\t\t &scratch);\n \tcheck_int(m, ==, n);\n \n-\tcheck(reftable_record_equal(&in, &out, GIT_SHA1_RAWSZ));\n+\tcheck(reftable_record_equal(&in, &out, REFTABLE_HASH_SIZE_SHA1));\n \n \treftable_record_release(&out);\n \treftable_buf_release(&key);\ndiff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c\nindex 1b4363a58fc..13fd8d8f941 100644\n--- a/t/unit-tests/t-reftable-stack.c\n+++ b/t/unit-tests/t-reftable-stack.c\n@@ -121,7 +121,7 @@ static void write_n_ref_tables(struct reftable_stack *st,\n \n \t\tsnprintf(buf, sizeof(buf), \"refs/heads/branch-%04\"PRIuMAX, (uintmax_t)i);\n \t\tref.refname = buf;\n-\t\tt_reftable_set_hash(ref.value.val1, i, GIT_SHA1_FORMAT_ID);\n+\t\tt_reftable_set_hash(ref.value.val1, i, REFTABLE_HASH_SHA1);\n \n \t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n \t\tcheck(!err);\n@@ -169,7 +169,7 @@ static void t_reftable_stack_add_one(void)\n \n \terr = reftable_stack_read_ref(st, ref.refname, &dest);\n \tcheck(!err);\n-\tcheck(reftable_ref_record_equal(&ref, &dest, GIT_SHA1_RAWSZ));\n+\tcheck(reftable_ref_record_equal(&ref, &dest, REFTABLE_HASH_SIZE_SHA1));\n \tcheck_int(st->readers_len, >, 0);\n \n #ifndef GIT_WINDOWS_NATIVE\n@@ -280,7 +280,7 @@ static void t_reftable_stack_transaction_api(void)\n \terr = reftable_stack_read_ref(st, ref.refname, &dest);\n \tcheck(!err);\n \tcheck_int(REFTABLE_REF_SYMREF, ==, dest.value_type);\n-\tcheck(reftable_ref_record_equal(&ref, &dest, GIT_SHA1_RAWSZ));\n+\tcheck(reftable_ref_record_equal(&ref, &dest, REFTABLE_HASH_SIZE_SHA1));\n \n \treftable_ref_record_release(&dest);\n \treftable_stack_destroy(st);\n@@ -340,7 +340,7 @@ static void t_reftable_stack_transaction_with_reload(void)\n \tfor (size_t i = 0; i < ARRAY_SIZE(refs); i++) {\n \t\terr = reftable_stack_read_ref(st2, refs[i].refname, &ref);\n \t\tcheck(!err);\n-\t\tcheck(reftable_ref_record_equal(&refs[i], &ref, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_ref_record_equal(&refs[i], &ref, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \treftable_ref_record_release(&ref);\n@@ -530,13 +530,13 @@ static void t_reftable_stack_add(void)\n \t\trefs[i].refname = xstrdup(buf);\n \t\trefs[i].update_index = i + 1;\n \t\trefs[i].value_type = REFTABLE_REF_VAL1;\n-\t\tt_reftable_set_hash(refs[i].value.val1, i, GIT_SHA1_FORMAT_ID);\n+\t\tt_reftable_set_hash(refs[i].value.val1, i, REFTABLE_HASH_SHA1);\n \n \t\tlogs[i].refname = xstrdup(buf);\n \t\tlogs[i].update_index = N + i + 1;\n \t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n \t\tlogs[i].value.update.email = xstrdup(\"identity@invalid\");\n-\t\tt_reftable_set_hash(logs[i].value.update.new_hash, i, GIT_SHA1_FORMAT_ID);\n+\t\tt_reftable_set_hash(logs[i].value.update.new_hash, i, REFTABLE_HASH_SHA1);\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -562,7 +562,7 @@ static void t_reftable_stack_add(void)\n \t\tint err = reftable_stack_read_ref(st, refs[i].refname, &dest);\n \t\tcheck(!err);\n \t\tcheck(reftable_ref_record_equal(&dest, refs + i,\n-\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_ref_record_release(&dest);\n \t}\n \n@@ -571,7 +571,7 @@ static void t_reftable_stack_add(void)\n \t\tint err = reftable_stack_read_log(st, refs[i].refname, &dest);\n \t\tcheck(!err);\n \t\tcheck(reftable_log_record_equal(&dest, logs + i,\n-\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_log_record_release(&dest);\n \t}\n \n@@ -622,14 +622,14 @@ static void t_reftable_stack_iterator(void)\n \t\trefs[i].refname = xstrfmt(\"branch%02\"PRIuMAX, (uintmax_t)i);\n \t\trefs[i].update_index = i + 1;\n \t\trefs[i].value_type = REFTABLE_REF_VAL1;\n-\t\tt_reftable_set_hash(refs[i].value.val1, i, GIT_SHA1_FORMAT_ID);\n+\t\tt_reftable_set_hash(refs[i].value.val1, i, REFTABLE_HASH_SHA1);\n \n \t\tlogs[i].refname = xstrfmt(\"branch%02\"PRIuMAX, (uintmax_t)i);\n \t\tlogs[i].update_index = i + 1;\n \t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n \t\tlogs[i].value.update.email = xstrdup(\"johndoe@invalid\");\n \t\tlogs[i].value.update.message = xstrdup(\"commit\\n\");\n-\t\tt_reftable_set_hash(logs[i].value.update.new_hash, i, GIT_SHA1_FORMAT_ID);\n+\t\tt_reftable_set_hash(logs[i].value.update.new_hash, i, REFTABLE_HASH_SHA1);\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -656,7 +656,7 @@ static void t_reftable_stack_iterator(void)\n \t\tif (err > 0)\n \t\t\tbreak;\n \t\tcheck(!err);\n-\t\tcheck(reftable_ref_record_equal(&ref, &refs[i], GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_ref_record_equal(&ref, &refs[i], REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_ref_record_release(&ref);\n \t}\n \tcheck_int(i, ==, N);\n@@ -674,7 +674,7 @@ static void t_reftable_stack_iterator(void)\n \t\tif (err > 0)\n \t\t\tbreak;\n \t\tcheck(!err);\n-\t\tcheck(reftable_log_record_equal(&log, &logs[i], GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_log_record_equal(&log, &logs[i], REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_log_record_release(&log);\n \t}\n \tcheck_int(i, ==, N);\n@@ -767,7 +767,7 @@ static void t_reftable_stack_tombstone(void)\n \t\tif (i % 2 == 0) {\n \t\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n \t\t\tt_reftable_set_hash(refs[i].value.val1, i,\n-\t\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t\t REFTABLE_HASH_SHA1);\n \t\t}\n \n \t\tlogs[i].refname = xstrdup(buf);\n@@ -776,7 +776,7 @@ static void t_reftable_stack_tombstone(void)\n \t\tif (i % 2 == 0) {\n \t\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n \t\t\tt_reftable_set_hash(logs[i].value.update.new_hash, i,\n-\t\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t\t REFTABLE_HASH_SHA1);\n \t\t\tlogs[i].value.update.email =\n \t\t\t\txstrdup(\"identity@invalid\");\n \t\t}\n@@ -836,7 +836,7 @@ static void t_reftable_stack_hash_id(void)\n \t\t.value.symref = (char *) \"target\",\n \t\t.update_index = 1,\n \t};\n-\tstruct reftable_write_options opts32 = { .hash_id = GIT_SHA256_FORMAT_ID };\n+\tstruct reftable_write_options opts32 = { .hash_id = REFTABLE_HASH_SHA256 };\n \tstruct reftable_stack *st32 = NULL;\n \tstruct reftable_write_options opts_default = { 0 };\n \tstruct reftable_stack *st_default = NULL;\n@@ -859,7 +859,7 @@ static void t_reftable_stack_hash_id(void)\n \terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n \tcheck(!err);\n \n-\tcheck(reftable_ref_record_equal(&ref, &dest, GIT_SHA1_RAWSZ));\n+\tcheck(reftable_ref_record_equal(&ref, &dest, REFTABLE_HASH_SIZE_SHA1));\n \treftable_ref_record_release(&dest);\n \treftable_stack_destroy(st);\n \treftable_stack_destroy(st_default);\n@@ -909,7 +909,7 @@ static void t_reflog_expire(void)\n \t\tlogs[i].value.update.time = i;\n \t\tlogs[i].value.update.email = xstrdup(\"identity@invalid\");\n \t\tt_reftable_set_hash(logs[i].value.update.new_hash, i,\n-\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t REFTABLE_HASH_SHA1);\n \t}\n \n \tfor (i = 1; i <= N; i++) {\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506834","messageId":"7782652b97545f95c83aef08786c70384e5cfa52.1731047193.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1731047193.git.ps@pks.im","subject":"[PATCH v2 4/7] reftable/stack: stop using `fsync_component()` directly","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-08T08:17:17Z","receivedAt":"2024-11-08T08:17:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're executing `fsync_component()` directly in the reftable library so\nthat we can fsync data to disk depending on \"core.fsync\". But as we're\nin the process of converting the reftable library to become standalone\nwe cannot use that function in the library anymore.\n\nRefactor the code such that users of the library can inject a custom\nfsync function via the write options. This allows us to get rid of the\ndependency on \"write-or-die.h\".\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 7 ++++++\n reftable/reftable-writer.h | 6 +++++\n reftable/stack.c | 49 +++++++++++++++++++++++++-------------\n 3 files changed, 45 insertions(+), 17 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 7d86d920970..2e774176eda 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -24,6 +24,7 @@\n #include \"../setup.h\"\n #include \"../strmap.h\"\n #include \"../trace2.h\"\n+#include \"../write-or-die.h\"\n #include \"parse.h\"\n #include \"refs-internal.h\"\n \n@@ -273,6 +274,11 @@ static int reftable_be_config(const char *var, const char *value,\n \treturn 0;\n }\n \n+static int reftable_be_fsync(int fd)\n+{\n+\treturn fsync_component(FSYNC_COMPONENT_REFERENCE, fd);\n+}\n+\n static struct ref_store *reftable_be_init(struct repository *repo,\n \t\t\t\t\t const char *gitdir,\n \t\t\t\t\t unsigned int store_flags)\n@@ -304,6 +310,7 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \trefs->write_options.disable_auto_compact =\n \t\t!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1);\n \trefs->write_options.lock_timeout_ms = 100;\n+\trefs->write_options.fsync = reftable_be_fsync;\n \n \tgit_config(reftable_be_config, &refs->write_options);\n \ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 211860d08a4..c85ef5a5bd1 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -62,6 +62,12 @@ struct reftable_write_options {\n \t * negative value will cause us to block indefinitely.\n \t */\n \tlong lock_timeout_ms;\n+\n+\t/*\n+\t * Optional callback used to fsync files to disk. Falls back to using\n+\t * fsync(3P) when unset.\n+\t */\n+\tint (*fsync)(int fd);\n };\n \n /* reftable_block_stats holds statistics for a single block type */\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 9ae716ff375..c67bdd952ca 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -8,7 +8,6 @@ license that can be found in the LICENSE file or at\n \n #include \"stack.h\"\n \n-#include \"../write-or-die.h\"\n #include \"system.h\"\n #include \"constants.h\"\n #include \"merged.h\"\n@@ -43,17 +42,28 @@ static int stack_filename(struct reftable_buf *dest, struct reftable_stack *st,\n \treturn 0;\n }\n \n-static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz)\n+static int stack_fsync(const struct reftable_write_options *opts, int fd)\n {\n-\tint *fdp = (int *)arg;\n-\treturn write_in_full(*fdp, data, sz);\n+\tif (opts->fsync)\n+\t\treturn opts->fsync(fd);\n+\treturn fsync(fd);\n }\n \n-static int reftable_fd_flush(void *arg)\n+struct fd_writer {\n+\tconst struct reftable_write_options *opts;\n+\tint fd;\n+};\n+\n+static ssize_t fd_writer_write(void *arg, const void *data, size_t sz)\n {\n-\tint *fdp = (int *)arg;\n+\tstruct fd_writer *writer = arg;\n+\treturn write_in_full(writer->fd, data, sz);\n+}\n \n-\treturn fsync_component(FSYNC_COMPONENT_REFERENCE, *fdp);\n+static int fd_writer_flush(void *arg)\n+{\n+\tstruct fd_writer *writer = arg;\n+\treturn stack_fsync(writer->opts, writer->fd);\n }\n \n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n@@ -765,7 +775,7 @@ int reftable_addition_commit(struct reftable_addition *add)\n \t\tgoto done;\n \t}\n \n-\terr = fsync_component(FSYNC_COMPONENT_REFERENCE, lock_file_fd);\n+\terr = stack_fsync(&add->stack->opts, lock_file_fd);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n@@ -858,8 +868,10 @@ int reftable_addition_add(struct reftable_addition *add,\n \tstruct reftable_buf next_name = REFTABLE_BUF_INIT;\n \tstruct reftable_writer *wr = NULL;\n \tstruct tempfile *tab_file = NULL;\n+\tstruct fd_writer writer = {\n+\t\t.opts = &add->stack->opts,\n+\t};\n \tint err = 0;\n-\tint tab_fd;\n \n \treftable_buf_reset(&next_name);\n \n@@ -887,10 +899,10 @@ int reftable_addition_add(struct reftable_addition *add,\n \t\t\tgoto done;\n \t\t}\n \t}\n-\ttab_fd = get_tempfile_fd(tab_file);\n \n-\terr = reftable_writer_new(&wr, reftable_fd_write, reftable_fd_flush,\n-\t\t\t\t &tab_fd, &add->stack->opts);\n+\twriter.fd = get_tempfile_fd(tab_file);\n+\terr = reftable_writer_new(&wr, fd_writer_write, fd_writer_flush,\n+\t\t\t\t &writer, &add->stack->opts);\n \tif (err < 0)\n \t\tgoto done;\n \n@@ -973,8 +985,11 @@ static int stack_compact_locked(struct reftable_stack *st,\n \tstruct reftable_buf next_name = REFTABLE_BUF_INIT;\n \tstruct reftable_buf tab_file_path = REFTABLE_BUF_INIT;\n \tstruct reftable_writer *wr = NULL;\n+\tstruct fd_writer writer= {\n+\t\t.opts = &st->opts,\n+\t};\n \tstruct tempfile *tab_file;\n-\tint tab_fd, err = 0;\n+\tint err = 0;\n \n \terr = format_name(&next_name, reftable_reader_min_update_index(st->readers[first]),\n \t\t\t reftable_reader_max_update_index(st->readers[last]));\n@@ -994,7 +1009,6 @@ static int stack_compact_locked(struct reftable_stack *st,\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n-\ttab_fd = get_tempfile_fd(tab_file);\n \n \tif (st->opts.default_permissions &&\n \t chmod(get_tempfile_path(tab_file), st->opts.default_permissions) < 0) {\n@@ -1002,8 +1016,9 @@ static int stack_compact_locked(struct reftable_stack *st,\n \t\tgoto done;\n \t}\n \n-\terr = reftable_writer_new(&wr, reftable_fd_write, reftable_fd_flush,\n-\t\t\t\t &tab_fd, &st->opts);\n+\twriter.fd = get_tempfile_fd(tab_file);\n+\terr = reftable_writer_new(&wr, fd_writer_write, fd_writer_flush,\n+\t\t\t\t &writer, &st->opts);\n \tif (err < 0)\n \t\tgoto done;\n \n@@ -1460,7 +1475,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\tgoto done;\n \t}\n \n-\terr = fsync_component(FSYNC_COMPONENT_REFERENCE, get_lock_file_fd(&tables_list_lock));\n+\terr = stack_fsync(&st->opts, get_lock_file_fd(&tables_list_lock));\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tunlink(new_table_path.buf);\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506835","messageId":"b15daefbc83383b3ea2b6ab7d2f091d3d756952b.1731047193.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1731047193.git.ps@pks.im","subject":"[PATCH v2 5/7] reftable/system: provide thin wrapper for tempfile subsystem","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-08T08:17:19Z","receivedAt":"2024-11-08T08:17:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We use the tempfile subsystem to write temporary tables, but given that\nwe're in the process of converting the reftable library to become\nstandalone we cannot use this subsystem directly anymore. While we could\nin theory convert the code to use mkstemp(3p) instead, we'd lose access\nto our infrastructure that automatically prunes tempfiles via atexit(3p)\nor signal handlers.\n\nProvide a thin wrapper for the tempfile subsystem instead. Like this,\nthe compatibility shim is fully self-contained in \"reftable/system.c\".\nDownstream users of the reftable library would have to implement their\nown tempfile shims by replacing \"system.c\" with a custom version.\n\nSigned-off-by: Patrick Steinhardt \n---\n Makefile | 1 +\n reftable/stack.c | 57 +++++++++++++++++++----------------------------\n reftable/system.c | 49 ++++++++++++++++++++++++++++++++++++++++\n reftable/system.h | 41 +++++++++++++++++++++++++++++++++-\n 4 files changed, 113 insertions(+), 35 deletions(-)\n create mode 100644 reftable/system.c\n\ndiff --git a/Makefile b/Makefile\nindex feeed6f9321..50a79ad83fc 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2722,6 +2722,7 @@ REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/stack.o\n+REFTABLE_OBJS += reftable/system.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n \ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex c67bdd952ca..2ac6a371516 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -16,7 +16,6 @@ license that can be found in the LICENSE file or at\n #include \"reftable-record.h\"\n #include \"reftable-merged.h\"\n #include \"writer.h\"\n-#include \"tempfile.h\"\n \n static int stack_try_add(struct reftable_stack *st,\n \t\t\t int (*write_table)(struct reftable_writer *wr,\n@@ -867,7 +866,7 @@ int reftable_addition_add(struct reftable_addition *add,\n \tstruct reftable_buf tab_file_name = REFTABLE_BUF_INIT;\n \tstruct reftable_buf next_name = REFTABLE_BUF_INIT;\n \tstruct reftable_writer *wr = NULL;\n-\tstruct tempfile *tab_file = NULL;\n+\tstruct reftable_tmpfile tab_file = REFTABLE_TMPFILE_INIT;\n \tstruct fd_writer writer = {\n \t\t.opts = &add->stack->opts,\n \t};\n@@ -887,20 +886,18 @@ int reftable_addition_add(struct reftable_addition *add,\n \tif (err < 0)\n \t\tgoto done;\n \n-\ttab_file = mks_tempfile(temp_tab_file_name.buf);\n-\tif (!tab_file) {\n-\t\terr = REFTABLE_IO_ERROR;\n+\terr = tmpfile_from_pattern(&tab_file, temp_tab_file_name.buf);\n+\tif (err < 0)\n \t\tgoto done;\n-\t}\n \tif (add->stack->opts.default_permissions) {\n-\t\tif (chmod(get_tempfile_path(tab_file),\n+\t\tif (chmod(tab_file.path,\n \t\t\t add->stack->opts.default_permissions)) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n \t\t}\n \t}\n \n-\twriter.fd = get_tempfile_fd(tab_file);\n+\twriter.fd = tab_file.fd;\n \terr = reftable_writer_new(&wr, fd_writer_write, fd_writer_flush,\n \t\t\t\t &writer, &add->stack->opts);\n \tif (err < 0)\n@@ -918,11 +915,9 @@ int reftable_addition_add(struct reftable_addition *add,\n \tif (err < 0)\n \t\tgoto done;\n \n-\terr = close_tempfile_gently(tab_file);\n-\tif (err < 0) {\n-\t\terr = REFTABLE_IO_ERROR;\n+\terr = tmpfile_close(&tab_file);\n+\tif (err < 0)\n \t\tgoto done;\n-\t}\n \n \tif (wr->min_update_index < add->next_update_index) {\n \t\terr = REFTABLE_API_ERROR;\n@@ -945,11 +940,9 @@ int reftable_addition_add(struct reftable_addition *add,\n \t On windows, this relies on rand() picking a unique destination name.\n \t Maybe we should do retry loop as well?\n \t */\n-\terr = rename_tempfile(&tab_file, tab_file_name.buf);\n-\tif (err < 0) {\n-\t\terr = REFTABLE_IO_ERROR;\n+\terr = tmpfile_rename(&tab_file, tab_file_name.buf);\n+\tif (err < 0)\n \t\tgoto done;\n-\t}\n \n \tREFTABLE_ALLOC_GROW(add->new_tables, add->new_tables_len + 1,\n \t\t\t add->new_tables_cap);\n@@ -960,7 +953,7 @@ int reftable_addition_add(struct reftable_addition *add,\n \tadd->new_tables[add->new_tables_len++] = reftable_buf_detach(&next_name);\n \n done:\n-\tdelete_tempfile(&tab_file);\n+\ttmpfile_delete(&tab_file);\n \treftable_buf_release(&temp_tab_file_name);\n \treftable_buf_release(&tab_file_name);\n \treftable_buf_release(&next_name);\n@@ -980,7 +973,7 @@ uint64_t reftable_stack_next_update_index(struct reftable_stack *st)\n static int stack_compact_locked(struct reftable_stack *st,\n \t\t\t\tsize_t first, size_t last,\n \t\t\t\tstruct reftable_log_expiry_config *config,\n-\t\t\t\tstruct tempfile **tab_file_out)\n+\t\t\t\tstruct reftable_tmpfile *tab_file_out)\n {\n \tstruct reftable_buf next_name = REFTABLE_BUF_INIT;\n \tstruct reftable_buf tab_file_path = REFTABLE_BUF_INIT;\n@@ -988,7 +981,7 @@ static int stack_compact_locked(struct reftable_stack *st,\n \tstruct fd_writer writer= {\n \t\t.opts = &st->opts,\n \t};\n-\tstruct tempfile *tab_file;\n+\tstruct reftable_tmpfile tab_file = REFTABLE_TMPFILE_INIT;\n \tint err = 0;\n \n \terr = format_name(&next_name, reftable_reader_min_update_index(st->readers[first]),\n@@ -1004,19 +997,17 @@ static int stack_compact_locked(struct reftable_stack *st,\n \tif (err < 0)\n \t\tgoto done;\n \n-\ttab_file = mks_tempfile(tab_file_path.buf);\n-\tif (!tab_file) {\n-\t\terr = REFTABLE_IO_ERROR;\n+\terr = tmpfile_from_pattern(&tab_file, tab_file_path.buf);\n+\tif (err < 0)\n \t\tgoto done;\n-\t}\n \n \tif (st->opts.default_permissions &&\n-\t chmod(get_tempfile_path(tab_file), st->opts.default_permissions) < 0) {\n+\t chmod(tab_file.path, st->opts.default_permissions) < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n \n-\twriter.fd = get_tempfile_fd(tab_file);\n+\twriter.fd = tab_file.fd;\n \terr = reftable_writer_new(&wr, fd_writer_write, fd_writer_flush,\n \t\t\t\t &writer, &st->opts);\n \tif (err < 0)\n@@ -1030,15 +1021,15 @@ static int stack_compact_locked(struct reftable_stack *st,\n \tif (err < 0)\n \t\tgoto done;\n \n-\terr = close_tempfile_gently(tab_file);\n+\terr = tmpfile_close(&tab_file);\n \tif (err < 0)\n \t\tgoto done;\n \n \t*tab_file_out = tab_file;\n-\ttab_file = NULL;\n+\ttab_file = REFTABLE_TMPFILE_INIT;\n \n done:\n-\tdelete_tempfile(&tab_file);\n+\ttmpfile_delete(&tab_file);\n \treftable_writer_free(wr);\n \treftable_buf_release(&next_name);\n \treftable_buf_release(&tab_file_path);\n@@ -1171,7 +1162,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \tstruct reftable_buf table_name = REFTABLE_BUF_INIT;\n \tstruct lock_file tables_list_lock = LOCK_INIT;\n \tstruct lock_file *table_locks = NULL;\n-\tstruct tempfile *new_table = NULL;\n+\tstruct reftable_tmpfile new_table = REFTABLE_TMPFILE_INIT;\n \tint is_empty_table = 0, err = 0;\n \tsize_t first_to_replace, last_to_replace;\n \tsize_t i, nlocks = 0;\n@@ -1439,11 +1430,9 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\tif (err < 0)\n \t\t\tgoto done;\n \n-\t\terr = rename_tempfile(&new_table, new_table_path.buf);\n-\t\tif (err < 0) {\n-\t\t\terr = REFTABLE_IO_ERROR;\n+\t\terr = tmpfile_rename(&new_table, new_table_path.buf);\n+\t\tif (err < 0)\n \t\t\tgoto done;\n-\t\t}\n \t}\n \n \t/*\n@@ -1515,7 +1504,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\trollback_lock_file(&table_locks[i]);\n \treftable_free(table_locks);\n \n-\tdelete_tempfile(&new_table);\n+\ttmpfile_delete(&new_table);\n \treftable_buf_release(&new_table_name);\n \treftable_buf_release(&new_table_path);\n \treftable_buf_release(&tables_list_buf);\ndiff --git a/reftable/system.c b/reftable/system.c\nnew file mode 100644\nindex 00000000000..01f96f03d84\n--- /dev/null\n+++ b/reftable/system.c\n@@ -0,0 +1,49 @@\n+#include \"system.h\"\n+#include \"basics.h\"\n+#include \"reftable-error.h\"\n+#include \"../tempfile.h\"\n+\n+int tmpfile_from_pattern(struct reftable_tmpfile *out, const char *pattern)\n+{\n+\tstruct tempfile *tempfile;\n+\n+\ttempfile = mks_tempfile(pattern);\n+\tif (!tempfile)\n+\t\treturn REFTABLE_IO_ERROR;\n+\n+\tout->path = tempfile->filename.buf;\n+\tout->fd = tempfile->fd;\n+\tout->priv = tempfile;\n+\n+\treturn 0;\n+}\n+\n+int tmpfile_close(struct reftable_tmpfile *t)\n+{\n+\tstruct tempfile *tempfile = t->priv;\n+\tint ret = close_tempfile_gently(tempfile);\n+\tt->fd = -1;\n+\tif (ret < 0)\n+\t\treturn REFTABLE_IO_ERROR;\n+\treturn 0;\n+}\n+\n+int tmpfile_delete(struct reftable_tmpfile *t)\n+{\n+\tstruct tempfile *tempfile = t->priv;\n+\tint ret = delete_tempfile(&tempfile);\n+\t*t = REFTABLE_TMPFILE_INIT;\n+\tif (ret < 0)\n+\t\treturn REFTABLE_IO_ERROR;\n+\treturn 0;\n+}\n+\n+int tmpfile_rename(struct reftable_tmpfile *t, const char *path)\n+{\n+\tstruct tempfile *tempfile = t->priv;\n+\tint ret = rename_tempfile(&tempfile, path);\n+\t*t = REFTABLE_TMPFILE_INIT;\n+\tif (ret < 0)\n+\t\treturn REFTABLE_IO_ERROR;\n+\treturn 0;\n+}\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 38d3534620e..e7595800907 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -13,6 +13,45 @@ license that can be found in the LICENSE file or at\n \n #include \"git-compat-util.h\"\n #include \"lockfile.h\"\n-#include \"tempfile.h\"\n+\n+/*\n+ * An implementation-specific temporary file. By making this specific to the\n+ * implementation it becomes possible to tie temporary files into any kind of\n+ * signal or atexit handlers for cleanup on abnormal situations.\n+ */\n+struct reftable_tmpfile {\n+\tconst char *path;\n+\tint fd;\n+\tvoid *priv;\n+};\n+#define REFTABLE_TMPFILE_INIT ((struct reftable_tmpfile) { .fd = -1, })\n+\n+/*\n+ * Create a temporary file from a pattern similar to how mkstemp(3p) would.\n+ * The `pattern` shall not be modified. On success, the structure at `out` has\n+ * been initialized such that it is ready for use. Returns 0 on success, a\n+ * reftable error code on error.\n+ */\n+int tmpfile_from_pattern(struct reftable_tmpfile *out, const char *pattern);\n+\n+/*\n+ * Close the temporary file's file descriptor without removing the file itself.\n+ * This is a no-op in case the file has already been closed beforehand. Returns\n+ * 0 on success, a reftable error code on error.\n+ */\n+int tmpfile_close(struct reftable_tmpfile *t);\n+\n+/*\n+ * Close the temporary file and delete it. This is a no-op in case the file has\n+ * already been deleted or renamed beforehand. Returns 0 on success, a reftable\n+ * error code on error.\n+ */\n+int tmpfile_delete(struct reftable_tmpfile *t);\n+\n+/*\n+ * Rename the temporary file to the provided path. The temporary file must be\n+ * active. Return 0 on success, a reftable error code on error.\n+ */\n+int tmpfile_rename(struct reftable_tmpfile *t, const char *path);\n \n #endif\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506836","messageId":"83949837a2970304c57e252065391f3c4dab956f.1731047193.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1731047193.git.ps@pks.im","subject":"[PATCH v2 6/7] reftable/stack: drop only use of `get_locked_file_path()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-08T08:17:22Z","receivedAt":"2024-11-08T08:17:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We've got a single callsite where we call `get_locked_file_path()`. As\nwe're about to convert our usage of the lockfile subsystem to instead be\nused via a compatibility shim we'd have to implement more logic for this\nsingle callsite. While that would be okay if Git was the only supposed\nuser of the reftable library, it's a bit more awkward when considering\nthat we have to reimplement this functionality for every user of the\nlibrary eventually.\n\nRefactor the code such that we don't call `get_locked_file_path()`\nanymore.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 12 +++++++++---\n 1 file changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 2ac6a371516..223d7c622d9 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1493,9 +1493,15 @@ static int stack_compact_range(struct reftable_stack *st,\n \t */\n \tfor (i = 0; i < nlocks; i++) {\n \t\tstruct lock_file *table_lock = &table_locks[i];\n-\t\tchar *table_path = get_locked_file_path(table_lock);\n-\t\tunlink(table_path);\n-\t\treftable_free(table_path);\n+\t\tconst char *lock_path = get_lock_file_path(table_lock);\n+\n+\t\treftable_buf_reset(&table_name);\n+\t\terr = reftable_buf_add(&table_name, lock_path,\n+\t\t\t\t strlen(lock_path) - strlen(\".lock\"));\n+\t\tif (err)\n+\t\t\tcontinue;\n+\n+\t\tunlink(table_name.buf);\n \t}\n \n done:\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506837","messageId":"80fe5bc5e109ecc23e582f014585c8110bb404da.1731047193.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1731047193.git.ps@pks.im","subject":"[PATCH v2 7/7] reftable/system: provide thin wrapper for lockfile subsystem","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-08T08:17:24Z","receivedAt":"2024-11-08T08:17:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We use the lockfile subsystem to write lockfiles for \"tables.list\". As\nwith the tempfile subsystem, the lockfile subsystem also hooks into our\ninfrastructure to prune stale locks via atexit(3p) or signal handlers.\n\nFurthermore, the lockfile subsystem also handles locking timeouts, which\ndo add quite a bit of logic. Having to reimplement that in the context\nof Git wouldn't make a whole lot of sense, and it is quite likely that\ndownstream users of the reftable library may have a better idea for how\nexactly to implement timeouts.\n\nSo again, provide a thin wrapper for the lockfile subsystem instead such\nthat the compatibility shim is fully self-contained.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 63 ++++++++++-------------\n reftable/system.c | 77 +++++++++++++++++++++++++++++\n reftable/system.h | 45 ++++++++++++++++-\n t/unit-tests/lib-reftable.c | 1 +\n t/unit-tests/t-reftable-block.c | 1 +\n t/unit-tests/t-reftable-pq.c | 1 +\n t/unit-tests/t-reftable-readwrite.c | 1 +\n t/unit-tests/t-reftable-stack.c | 2 +\n 8 files changed, 154 insertions(+), 37 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 223d7c622d9..10d45e89d00 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -657,7 +657,7 @@ static int format_name(struct reftable_buf *dest, uint64_t min, uint64_t max)\n }\n \n struct reftable_addition {\n-\tstruct lock_file tables_list_lock;\n+\tstruct reftable_flock tables_list_lock;\n \tstruct reftable_stack *stack;\n \n \tchar **new_tables;\n@@ -676,10 +676,8 @@ static int reftable_stack_init_addition(struct reftable_addition *add,\n \n \tadd->stack = st;\n \n-\terr = hold_lock_file_for_update_timeout(&add->tables_list_lock,\n-\t\t\t\t\t\tst->list_file,\n-\t\t\t\t\t\tLOCK_NO_DEREF,\n-\t\t\t\t\t\tst->opts.lock_timeout_ms);\n+\terr = flock_acquire(&add->tables_list_lock, st->list_file,\n+\t\t\t st->opts.lock_timeout_ms);\n \tif (err < 0) {\n \t\tif (errno == EEXIST) {\n \t\t\terr = REFTABLE_LOCK_ERROR;\n@@ -689,7 +687,7 @@ static int reftable_stack_init_addition(struct reftable_addition *add,\n \t\tgoto done;\n \t}\n \tif (st->opts.default_permissions) {\n-\t\tif (chmod(get_lock_file_path(&add->tables_list_lock),\n+\t\tif (chmod(add->tables_list_lock.path,\n \t\t\t st->opts.default_permissions) < 0) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n@@ -733,7 +731,7 @@ static void reftable_addition_close(struct reftable_addition *add)\n \tadd->new_tables_len = 0;\n \tadd->new_tables_cap = 0;\n \n-\trollback_lock_file(&add->tables_list_lock);\n+\tflock_release(&add->tables_list_lock);\n \treftable_buf_release(&nm);\n }\n \n@@ -749,7 +747,6 @@ void reftable_addition_destroy(struct reftable_addition *add)\n int reftable_addition_commit(struct reftable_addition *add)\n {\n \tstruct reftable_buf table_list = REFTABLE_BUF_INIT;\n-\tint lock_file_fd = get_lock_file_fd(&add->tables_list_lock);\n \tint err = 0;\n \tsize_t i;\n \n@@ -767,20 +764,20 @@ int reftable_addition_commit(struct reftable_addition *add)\n \t\t\tgoto done;\n \t}\n \n-\terr = write_in_full(lock_file_fd, table_list.buf, table_list.len);\n+\terr = write_in_full(add->tables_list_lock.fd, table_list.buf, table_list.len);\n \treftable_buf_release(&table_list);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n \n-\terr = stack_fsync(&add->stack->opts, lock_file_fd);\n+\terr = stack_fsync(&add->stack->opts, add->tables_list_lock.fd);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n \n-\terr = commit_lock_file(&add->tables_list_lock);\n+\terr = flock_commit(&add->tables_list_lock);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n@@ -1160,8 +1157,8 @@ static int stack_compact_range(struct reftable_stack *st,\n \tstruct reftable_buf new_table_name = REFTABLE_BUF_INIT;\n \tstruct reftable_buf new_table_path = REFTABLE_BUF_INIT;\n \tstruct reftable_buf table_name = REFTABLE_BUF_INIT;\n-\tstruct lock_file tables_list_lock = LOCK_INIT;\n-\tstruct lock_file *table_locks = NULL;\n+\tstruct reftable_flock tables_list_lock = REFTABLE_FLOCK_INIT;\n+\tstruct reftable_flock *table_locks = NULL;\n \tstruct reftable_tmpfile new_table = REFTABLE_TMPFILE_INIT;\n \tint is_empty_table = 0, err = 0;\n \tsize_t first_to_replace, last_to_replace;\n@@ -1179,10 +1176,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * Hold the lock so that we can read \"tables.list\" and lock all tables\n \t * which are part of the user-specified range.\n \t */\n-\terr = hold_lock_file_for_update_timeout(&tables_list_lock,\n-\t\t\t\t\t\tst->list_file,\n-\t\t\t\t\t\tLOCK_NO_DEREF,\n-\t\t\t\t\t\tst->opts.lock_timeout_ms);\n+\terr = flock_acquire(&tables_list_lock, st->list_file, st->opts.lock_timeout_ms);\n \tif (err < 0) {\n \t\tif (errno == EEXIST)\n \t\t\terr = REFTABLE_LOCK_ERROR;\n@@ -1205,19 +1199,20 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * older process is still busy compacting tables which are preexisting\n \t * from the point of view of the newer process.\n \t */\n-\tREFTABLE_CALLOC_ARRAY(table_locks, last - first + 1);\n+\tREFTABLE_ALLOC_ARRAY(table_locks, last - first + 1);\n \tif (!table_locks) {\n \t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n \t\tgoto done;\n \t}\n+\tfor (i = 0; i < last - first + 1; i++)\n+\t\ttable_locks[i] = REFTABLE_FLOCK_INIT;\n \n \tfor (i = last + 1; i > first; i--) {\n \t\terr = stack_filename(&table_name, st, reader_name(st->readers[i - 1]));\n \t\tif (err < 0)\n \t\t\tgoto done;\n \n-\t\terr = hold_lock_file_for_update(&table_locks[nlocks],\n-\t\t\t\t\t\ttable_name.buf, LOCK_NO_DEREF);\n+\t\terr = flock_acquire(&table_locks[nlocks], table_name.buf, 0);\n \t\tif (err < 0) {\n \t\t\t/*\n \t\t\t * When the table is locked already we may do a\n@@ -1253,7 +1248,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\t * run into file descriptor exhaustion when we compress a lot\n \t\t * of tables.\n \t\t */\n-\t\terr = close_lock_file_gently(&table_locks[nlocks++]);\n+\t\terr = flock_close(&table_locks[nlocks++]);\n \t\tif (err < 0) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n@@ -1265,7 +1260,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * \"tables.list\" lock while compacting the locked tables. This allows\n \t * concurrent updates to the stack to proceed.\n \t */\n-\terr = rollback_lock_file(&tables_list_lock);\n+\terr = flock_release(&tables_list_lock);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n@@ -1288,10 +1283,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * \"tables.list\". We'll then replace the compacted range of tables with\n \t * the new table.\n \t */\n-\terr = hold_lock_file_for_update_timeout(&tables_list_lock,\n-\t\t\t\t\t\tst->list_file,\n-\t\t\t\t\t\tLOCK_NO_DEREF,\n-\t\t\t\t\t\tst->opts.lock_timeout_ms);\n+\terr = flock_acquire(&tables_list_lock, st->list_file, st->opts.lock_timeout_ms);\n \tif (err < 0) {\n \t\tif (errno == EEXIST)\n \t\t\terr = REFTABLE_LOCK_ERROR;\n@@ -1301,7 +1293,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t}\n \n \tif (st->opts.default_permissions) {\n-\t\tif (chmod(get_lock_file_path(&tables_list_lock),\n+\t\tif (chmod(tables_list_lock.path,\n \t\t\t st->opts.default_permissions) < 0) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n@@ -1456,7 +1448,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\t\tgoto done;\n \t}\n \n-\terr = write_in_full(get_lock_file_fd(&tables_list_lock),\n+\terr = write_in_full(tables_list_lock.fd,\n \t\t\t tables_list_buf.buf, tables_list_buf.len);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n@@ -1464,14 +1456,14 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\tgoto done;\n \t}\n \n-\terr = stack_fsync(&st->opts, get_lock_file_fd(&tables_list_lock));\n+\terr = stack_fsync(&st->opts, tables_list_lock.fd);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tunlink(new_table_path.buf);\n \t\tgoto done;\n \t}\n \n-\terr = commit_lock_file(&tables_list_lock);\n+\terr = flock_commit(&tables_list_lock);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tunlink(new_table_path.buf);\n@@ -1492,12 +1484,11 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * readers, so it is expected that unlinking tables may fail.\n \t */\n \tfor (i = 0; i < nlocks; i++) {\n-\t\tstruct lock_file *table_lock = &table_locks[i];\n-\t\tconst char *lock_path = get_lock_file_path(table_lock);\n+\t\tstruct reftable_flock *table_lock = &table_locks[i];\n \n \t\treftable_buf_reset(&table_name);\n-\t\terr = reftable_buf_add(&table_name, lock_path,\n-\t\t\t\t strlen(lock_path) - strlen(\".lock\"));\n+\t\terr = reftable_buf_add(&table_name, table_lock->path,\n+\t\t\t\t strlen(table_lock->path) - strlen(\".lock\"));\n \t\tif (err)\n \t\t\tcontinue;\n \n@@ -1505,9 +1496,9 @@ static int stack_compact_range(struct reftable_stack *st,\n \t}\n \n done:\n-\trollback_lock_file(&tables_list_lock);\n+\tflock_release(&tables_list_lock);\n \tfor (i = 0; table_locks && i < nlocks; i++)\n-\t\trollback_lock_file(&table_locks[i]);\n+\t\tflock_release(&table_locks[i]);\n \treftable_free(table_locks);\n \n \ttmpfile_delete(&new_table);\ndiff --git a/reftable/system.c b/reftable/system.c\nindex 01f96f03d84..adf8e4d30b8 100644\n--- a/reftable/system.c\n+++ b/reftable/system.c\n@@ -1,6 +1,7 @@\n #include \"system.h\"\n #include \"basics.h\"\n #include \"reftable-error.h\"\n+#include \"../lockfile.h\"\n #include \"../tempfile.h\"\n \n int tmpfile_from_pattern(struct reftable_tmpfile *out, const char *pattern)\n@@ -47,3 +48,79 @@ int tmpfile_rename(struct reftable_tmpfile *t, const char *path)\n \t\treturn REFTABLE_IO_ERROR;\n \treturn 0;\n }\n+\n+int flock_acquire(struct reftable_flock *l, const char *target_path,\n+\t\t long timeout_ms)\n+{\n+\tstruct lock_file *lockfile;\n+\tint err;\n+\n+\tlockfile = reftable_malloc(sizeof(*lockfile));\n+\tif (!lockfile)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n+\terr = hold_lock_file_for_update_timeout(lockfile, target_path, LOCK_NO_DEREF,\n+\t\t\t\t\t\ttimeout_ms);\n+\tif (err < 0) {\n+\t\treftable_free(lockfile);\n+\t\tif (errno == EEXIST)\n+\t\t\treturn REFTABLE_LOCK_ERROR;\n+\t\treturn -1;\n+\t}\n+\n+\tl->fd = get_lock_file_fd(lockfile);\n+\tl->path = get_lock_file_path(lockfile);\n+\tl->priv = lockfile;\n+\n+\treturn 0;\n+}\n+\n+int flock_close(struct reftable_flock *l)\n+{\n+\tstruct lock_file *lockfile = l->priv;\n+\tint ret;\n+\n+\tif (!lockfile)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\tret = close_lock_file_gently(lockfile);\n+\tl->fd = -1;\n+\tif (ret < 0)\n+\t\treturn REFTABLE_IO_ERROR;\n+\n+\treturn 0;\n+}\n+\n+int flock_release(struct reftable_flock *l)\n+{\n+\tstruct lock_file *lockfile = l->priv;\n+\tint ret;\n+\n+\tif (!lockfile)\n+\t\treturn 0;\n+\n+\tret = rollback_lock_file(lockfile);\n+\treftable_free(lockfile);\n+\t*l = REFTABLE_FLOCK_INIT;\n+\tif (ret < 0)\n+\t\treturn REFTABLE_IO_ERROR;\n+\n+\treturn 0;\n+}\n+\n+int flock_commit(struct reftable_flock *l)\n+{\n+\tstruct lock_file *lockfile = l->priv;\n+\tint ret;\n+\n+\tif (!lockfile)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\tret = commit_lock_file(lockfile);\n+\treftable_free(lockfile);\n+\t*l = REFTABLE_FLOCK_INIT;\n+\tif (ret < 0)\n+\t\treturn REFTABLE_IO_ERROR;\n+\n+\treturn 0;\n+}\ndiff --git a/reftable/system.h b/reftable/system.h\nindex e7595800907..0859c3539c6 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -12,7 +12,6 @@ license that can be found in the LICENSE file or at\n /* This header glues the reftable library to the rest of Git */\n \n #include \"git-compat-util.h\"\n-#include \"lockfile.h\"\n \n /*\n * An implementation-specific temporary file. By making this specific to the\n@@ -54,4 +53,48 @@ int tmpfile_delete(struct reftable_tmpfile *t);\n */\n int tmpfile_rename(struct reftable_tmpfile *t, const char *path);\n \n+/*\n+ * An implementation-specific file lock. Same as with `reftable_tmpfile`,\n+ * making this specific to the implementation makes it possible to tie this\n+ * into signal or atexit handlers such that we know to clean up stale locks on\n+ * abnormal exits.\n+ */\n+struct reftable_flock {\n+\tconst char *path;\n+\tint fd;\n+\tvoid *priv;\n+};\n+#define REFTABLE_FLOCK_INIT ((struct reftable_flock){ .fd = -1, })\n+\n+/*\n+ * Acquire the lock for the given target path by exclusively creating a file\n+ * with \".lock\" appended to it. If that lock exists, we wait up to `timeout_ms`\n+ * to acquire the lock. If `timeout_ms` is 0 we don't wait, if it is negative\n+ * we block indefinitely.\n+ *\n+ * Retrun 0 on success, a reftable error code on error.\n+ */\n+int flock_acquire(struct reftable_flock *l, const char *target_path,\n+\t\t long timeout_ms);\n+\n+/*\n+ * Close the lockfile's file descriptor without removing the lock itself. This\n+ * is a no-op in case the lockfile has already been closed beforehand. Returns\n+ * 0 on success, a reftable error code on error.\n+ */\n+int flock_close(struct reftable_flock *l);\n+\n+/*\n+ * Release the lock by unlinking the lockfile. This is a no-op in case the\n+ * lockfile has already been released or committed beforehand. Returns 0 on\n+ * success, a reftable error code on error.\n+ */\n+int flock_release(struct reftable_flock *l);\n+\n+/*\n+ * Commit the lock by renaming the lockfile into place. Returns 0 on success, a\n+ * reftable error code on error.\n+ */\n+int flock_commit(struct reftable_flock *l);\n+\n #endif\ndiff --git a/t/unit-tests/lib-reftable.c b/t/unit-tests/lib-reftable.c\nindex c1631f45275..d795dfb7c99 100644\n--- a/t/unit-tests/lib-reftable.c\n+++ b/t/unit-tests/lib-reftable.c\n@@ -2,6 +2,7 @@\n #include \"test-lib.h\"\n #include \"reftable/constants.h\"\n #include \"reftable/writer.h\"\n+#include \"strbuf.h\"\n \n void t_reftable_set_hash(uint8_t *p, int i, enum reftable_hash id)\n {\ndiff --git a/t/unit-tests/t-reftable-block.c b/t/unit-tests/t-reftable-block.c\nindex 13e10807dae..22040aeefa5 100644\n--- a/t/unit-tests/t-reftable-block.c\n+++ b/t/unit-tests/t-reftable-block.c\n@@ -11,6 +11,7 @@ license that can be found in the LICENSE file or at\n #include \"reftable/blocksource.h\"\n #include \"reftable/constants.h\"\n #include \"reftable/reftable-error.h\"\n+#include \"strbuf.h\"\n \n static void t_ref_block_read_write(void)\n {\ndiff --git a/t/unit-tests/t-reftable-pq.c b/t/unit-tests/t-reftable-pq.c\nindex 272da05bea6..f3f8a0cdf38 100644\n--- a/t/unit-tests/t-reftable-pq.c\n+++ b/t/unit-tests/t-reftable-pq.c\n@@ -9,6 +9,7 @@ license that can be found in the LICENSE file or at\n #include \"test-lib.h\"\n #include \"reftable/constants.h\"\n #include \"reftable/pq.h\"\n+#include \"strbuf.h\"\n \n static void merged_iter_pqueue_check(const struct merged_iter_pqueue *pq)\n {\ndiff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c\nindex 57896922eb1..91c881aedfa 100644\n--- a/t/unit-tests/t-reftable-readwrite.c\n+++ b/t/unit-tests/t-reftable-readwrite.c\n@@ -13,6 +13,7 @@ license that can be found in the LICENSE file or at\n #include \"reftable/reader.h\"\n #include \"reftable/reftable-error.h\"\n #include \"reftable/reftable-writer.h\"\n+#include \"strbuf.h\"\n \n static const int update_index = 5;\n \ndiff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c\nindex 13fd8d8f941..b2f6c1c37e9 100644\n--- a/t/unit-tests/t-reftable-stack.c\n+++ b/t/unit-tests/t-reftable-stack.c\n@@ -13,6 +13,8 @@ license that can be found in the LICENSE file or at\n #include \"reftable/reader.h\"\n #include \"reftable/reftable-error.h\"\n #include \"reftable/stack.h\"\n+#include \"strbuf.h\"\n+#include \"tempfile.h\"\n #include \n \n static void clear_dir(const char *dirname)\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506884","messageId":"23i62zyglnqknm4qf2lxmdcthrhhg5vsptfqohtmp327mrwsji@h2tutjzrl6jh","threadId":"62395","inReplyTo":"cover.1731047193.git.ps@pks.im","subject":"Re: [PATCH v2 0/7] reftable: stop using Git subsystems","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-11-08T17:39:50Z","receivedAt":"2024-11-08T17:41:33Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/11/08 09:17AM, Patrick Steinhardt wrote:\n> Hi,\n> \n> this is the second version of my patch series that continues to detangle\n> the reftable library from the Git codebase.\n> \n> Changes compared to v1:\n> \n> - Fix a commit message typo.\n> \n> - Document the values of the newly introduced reftable format IDs.\n> \n> - Include \"reftable-basics.h\" instead of \"basics.h\".\n> \n> - Adapt `stack_fsync()` to take write options as input instead of the\n> whole stack.\n> \n> Thanks!\n> \n> Patrick\n\nI've reviewed the range-diff and this version looks good to me. :)\n\n-Justin\n"},{"id":"506954","messageId":"xmqqr07ik3kw.fsf@gitster.g","threadId":"62395","inReplyTo":"23i62zyglnqknm4qf2lxmdcthrhhg5vsptfqohtmp327mrwsji@h2tutjzrl6jh","subject":"Re: [PATCH v2 0/7] reftable: stop using Git subsystems","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-11T00:37:51Z","receivedAt":"2024-11-11T00:37:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Justin Tobler writes:\n\n> On 24/11/08 09:17AM, Patrick Steinhardt wrote:\n>> Hi,\n>> \n>> this is the second version of my patch series that continues to detangle\n>> the reftable library from the Git codebase.\n>> \n>> Changes compared to v1:\n>> \n>> - Fix a commit message typo.\n>> \n>> - Document the values of the newly introduced reftable format IDs.\n>> \n>> - Include \"reftable-basics.h\" instead of \"basics.h\".\n>> \n>> - Adapt `stack_fsync()` to take write options as input instead of the\n>> whole stack.\n>> \n>> Thanks!\n>> \n>> Patrick\n>\n> I've reviewed the range-diff and this version looks good to me. :)\n\nThanks, both of you.\n"},{"id":"507104","messageId":"xmqqh68d6lqt.fsf@gitster.g","threadId":"62395","inReplyTo":"745c1a070ddd7180f0ee89c6a1d057a256894599.1731047193.git.ps@pks.im","subject":"Re: [PATCH v2 3/7] reftable/system: stop depending on \"hash.h\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-12T05:53:46Z","receivedAt":"2024-11-12T05:53:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> index 3c6107c7ce5..7d86d920970 100644\n> --- a/refs/reftable-backend.c\n> +++ b/refs/reftable-backend.c\n> @@ -15,6 +15,7 @@\n> #include \"../object.h\"\n> #include \"../path.h\"\n> #include \"../refs.h\"\n> +#include \"../reftable/reftable-basics.h\"\n> #include \"../reftable/reftable-stack.h\"\n> #include \"../reftable/reftable-record.h\"\n> #include \"../reftable/reftable-error.h\"\n> @@ -289,7 +290,16 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n> \trefs->store_flags = store_flags;\n> \trefs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);\n> \n> -\trefs->write_options.hash_id = repo->hash_algo->format_id;\n> +\tswitch (repo->hash_algo->format_id) {\n> +\tcase GIT_SHA1_FORMAT_ID:\n> +\t\trefs->write_options.hash_id = REFTABLE_HASH_SHA1;\n> +\t\tbreak;\n> +\tcase GIT_SHA256_FORMAT_ID:\n> +\t\trefs->write_options.hash_id = REFTABLE_HASH_SHA256;\n> +\t\tbreak;\n> +\tdefault:\n> +\t\tBUG(\"unknown hash algorithm %d\", repo->hash_algo->format_id);\n> +\t}\n> \trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n> \trefs->write_options.disable_auto_compact =\n> \t\t!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1);\n\nOK. With this step, together with the previous one, we let Git side\nto use GIT_SHA{1,256}_FORMAT_ID to specify what algorithm is in use\nin the repository, and reftable-backend.c layer is responsible for\ntranslating them into REFTABLE_HASH_SHA{1,256}, which is internally\nused in reftable library to convey the same thing.\n\nWhich makes sense.\n\n> diff --git a/reftable/reftable-basics.h b/reftable/reftable-basics.h\n> index 6e8e636b716..e0397ed5836 100644\n> --- a/reftable/reftable-basics.h\n> +++ b/reftable/reftable-basics.h\n> @@ -11,6 +11,19 @@\n> \n> #include \n> \n> +/*\n> + * Hash functions understood by the reftable library. Note that the values are\n> + * arbitrary and somewhat random such that we can easily detect cases where the\n> + * hash hasn't been properly set up.\n> + */\n> +enum reftable_hash {\n> +\tREFTABLE_HASH_SHA1 = 89,\n> +\tREFTABLE_HASH_SHA256 = 247,\n> +};\n\n;-).\n\n> +#define REFTABLE_HASH_SIZE_SHA1 20\n> +#define REFTABLE_HASH_SIZE_SHA256 32\n> +#define REFTABLE_HASH_SIZE_MAX REFTABLE_HASH_SIZE_SHA256\n> +\n> /* Overrides the functions to use for memory management. */\n> void reftable_set_alloc(void *(*malloc)(size_t),\n> \t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n\nOK.\n"},{"id":"507497","messageId":"CAOLa=ZTauvuFOT1cH=N81st19XBNh587Fx84QE9h+MAtK6rDEw@mail.gmail.com","threadId":"62395","inReplyTo":"38cfe85bf5b82c70848e4b295ba6cae33dcfd667.1731047193.git.ps@pks.im","subject":"Re: [PATCH v2 2/7] reftable: explicitly handle hash format IDs","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-11-18T13:47:28Z","receivedAt":"2024-11-18T13:47:29Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> The hash format IDs are used for two different things across the\n> reftable codebase:\n>\n> - They are used as a 32 bit unsigned integer when reading and writing\n> the header in order to identify the hash function.\n>\n> - They are used internally to identify which hash function is in use.\n>\n> When one only considers the second usecase one might think that one can\n> easily change the representation of those hash IDs. But because those\n> IDs end up in the reftable header and footer on disk it is important\n> that those never change.\n>\n> Create separate constants `REFTABLE_FORMAT_ID_*` and use them in\n> contexts where we read or write reftable headers. This serves multiple\n> purposes:\n>\n> - It allows us to more easily discern cases where we actually use\n> those constants for the on-disk format.\n>\n> - It detangles us from the same constants that are defined in\n> libgit.a, which is another required step to convert the reftable\n> library to become standalone.\n>\n> - It makes the next step easier where we stop using `GIT_*_FORMAT_ID`\n> constants in favor of a custom enum.\n>\n\nSo this patch tackles the first scenario mentioned above (for\nreading/writing the header), changing the second scenario\n(identification of which hash function) is handled in the next step by\nintroducing a custom enum. Makes sense.\n\n- Karthik\n"},{"id":"507498","messageId":"CAOLa=ZQTb=RFc_fqUDJZEdqbf80upshMn1+V4zEgsdHw0wWcLA@mail.gmail.com","threadId":"62395","inReplyTo":"745c1a070ddd7180f0ee89c6a1d057a256894599.1731047193.git.ps@pks.im","subject":"Re: [PATCH v2 3/7] reftable/system: stop depending on \"hash.h\"","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-11-18T13:54:18Z","receivedAt":"2024-11-18T13:54:20Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> We include \"hash.h\" in \"reftable/system.h\" such that we can use hash\n> format IDs as well as the raw size of SHA1 and SHA256. As we are in the\n> process of converting the reftable library to become standalone we of\n> course cannot rely on those constants anymore.\n>\n> Introduce a new `enum reftable_hash` to replace internal uses of the\n> hash format IDs and new constants that replace internal uses of the hash\n> size. Adapt the reftable backend to set up the correct hash function.\n>\n\nSo with this patch, we now have:\n - `REFTABLE_FORMAT_ID_SHA1` & `REFTABLE_FORMAT_ID_SHA256` which is\n what is stored on disk and used while reading/writing.\n - `enum reftable_hash` which is used to identify the hash function\n internally in the reftable code.\nBoth of which are separated. Nice!\n\n- Karthik\n"},{"id":"507499","messageId":"CAOLa=ZQEmzZpH2SLXNgcPLD=CThQgwW4THUiDuPMaOkZWfp_LA@mail.gmail.com","threadId":"62395","inReplyTo":"7782652b97545f95c83aef08786c70384e5cfa52.1731047193.git.ps@pks.im","subject":"Re: [PATCH v2 4/7] reftable/stack: stop using `fsync_component()` directly","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-11-18T14:02:54Z","receivedAt":"2024-11-18T14:02:57Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n[snip]\n\n> @@ -43,17 +42,28 @@ static int stack_filename(struct reftable_buf *dest, struct reftable_stack *st,\n> \treturn 0;\n> }\n>\n> -static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz)\n> +static int stack_fsync(const struct reftable_write_options *opts, int fd)\n> {\n> -\tint *fdp = (int *)arg;\n> -\treturn write_in_full(*fdp, data, sz);\n> +\tif (opts->fsync)\n> +\t\treturn opts->fsync(fd);\n> +\treturn fsync(fd);\n> }\n>\n> -static int reftable_fd_flush(void *arg)\n> +struct fd_writer {\n> +\tconst struct reftable_write_options *opts;\n> +\tint fd;\n> +};\n> +\n> +static ssize_t fd_writer_write(void *arg, const void *data, size_t sz)\n> {\n> -\tint *fdp = (int *)arg;\n> +\tstruct fd_writer *writer = arg;\n> +\treturn write_in_full(writer->fd, data, sz);\n\nTangent: eventually we'd also want to get rid of `write_in_full` no?\nSince it is also an internal Git implementation.\n\nThe patch itself looks good to me.\n\n[snip]\n\n- Karthik\n"},{"id":"507500","messageId":"CAOLa=ZQXXR3RR_2zBfCmR06t101W1X5VqN3P_LYYz+AMJKu+=Q@mail.gmail.com","threadId":"62395","inReplyTo":"b15daefbc83383b3ea2b6ab7d2f091d3d756952b.1731047193.git.ps@pks.im","subject":"Re: [PATCH v2 5/7] reftable/system: provide thin wrapper for tempfile subsystem","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-11-18T14:18:37Z","receivedAt":"2024-11-18T14:18:39Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n[snip]\n\n> +\n> +int tmpfile_close(struct reftable_tmpfile *t)\n> +{\n> +\tstruct tempfile *tempfile = t->priv;\n> +\tint ret = close_tempfile_gently(tempfile);\n> +\tt->fd = -1;\n\nNit: below we do `*t = REFTABLE_TMPFILE_INIT` should we keep things\nconsistent and do the same here?\n\n> +\tif (ret < 0)\n> +\t\treturn REFTABLE_IO_ERROR;\n> +\treturn 0;\n> +}\n> +\n\n[snip]\n\n> +/*\n> + * Close the temporary file and delete it. This is a no-op in case the file has\n> + * already been deleted or renamed beforehand. Returns 0 on success, a reftable\n> + * error code on error.\n> + */\n> +int tmpfile_delete(struct reftable_tmpfile *t);\n> +\n> +/*\n> + * Rename the temporary file to the provided path. The temporary file must be\n> + * active. Return 0 on success, a reftable error code on error.\n> + */\n\nNit: would be nice to mention that the `t` is reset here.\n\n> +int tmpfile_rename(struct reftable_tmpfile *t, const char *path);\n>\n> #endif\n> --\n> 2.47.0.229.g8f8d6eee53.dirty\n"},{"id":"507501","messageId":"CAOLa=ZQ9Etqz2t=OnsvVzVSyhSSB1jp6BYQ=FML8-_u8PGGVkw@mail.gmail.com","threadId":"62395","inReplyTo":"CAOLa=ZQXXR3RR_2zBfCmR06t101W1X5VqN3P_LYYz+AMJKu+=Q@mail.gmail.com","subject":"Re: [PATCH v2 5/7] reftable/system: provide thin wrapper for tempfile subsystem","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-11-18T14:29:28Z","receivedAt":"2024-11-18T14:29:30Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"karthik nayak writes:\n\n> Patrick Steinhardt writes:\n>\n> [snip]\n>\n>> +\n>> +int tmpfile_close(struct reftable_tmpfile *t)\n>> +{\n>> +\tstruct tempfile *tempfile = t->priv;\n>> +\tint ret = close_tempfile_gently(tempfile);\n>> +\tt->fd = -1;\n>\n> Nit: below we do `*t = REFTABLE_TMPFILE_INIT` should we keep things\n> consistent and do the same here?\n>\n\nMy bad, we want this to still hold onto `t->priv` which is cleared when\nwe call tmpfile_delete(). So all good here.\n"},{"id":"507502","messageId":"CAOLa=ZQxtQ9zBqN9U8ZnxaQ+uyMqQ1EnfEL9Br918y6CBxtvVQ@mail.gmail.com","threadId":"62395","inReplyTo":"cover.1731047193.git.ps@pks.im","subject":"Re: [PATCH v2 0/7] reftable: stop using Git subsystems","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-11-18T14:30:44Z","receivedAt":"2024-11-18T14:30:45Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Hi,\n>\n> this is the second version of my patch series that continues to detangle\n> the reftable library from the Git codebase.\n>\n> Changes compared to v1:\n>\n> - Fix a commit message typo.\n>\n> - Document the values of the newly introduced reftable format IDs.\n>\n> - Include \"reftable-basics.h\" instead of \"basics.h\".\n>\n> - Adapt `stack_fsync()` to take write options as input instead of the\n> whole stack.\n>\n> Thanks!\n>\n\n[snip]\n\nI went through the patches and I think it looks great!\n\n- Karthik\n"},{"id":"507505","messageId":"Zztcmgta-DnHFpjj@pks.im","threadId":"62395","inReplyTo":"CAOLa=ZQEmzZpH2SLXNgcPLD=CThQgwW4THUiDuPMaOkZWfp_LA@mail.gmail.com","subject":"Re: [PATCH v2 4/7] reftable/stack: stop using `fsync_component()` directly","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-18T15:26:18Z","receivedAt":"2024-11-18T15:26:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Nov 18, 2024 at 02:02:54PM +0000, karthik nayak wrote:\n> Patrick Steinhardt writes:\n> > @@ -43,17 +42,28 @@ static int stack_filename(struct reftable_buf *dest, struct reftable_stack *st,\n> > \treturn 0;\n> > }\n> >\n> > -static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz)\n> > +static int stack_fsync(const struct reftable_write_options *opts, int fd)\n> > {\n> > -\tint *fdp = (int *)arg;\n> > -\treturn write_in_full(*fdp, data, sz);\n> > +\tif (opts->fsync)\n> > +\t\treturn opts->fsync(fd);\n> > +\treturn fsync(fd);\n> > }\n> >\n> > -static int reftable_fd_flush(void *arg)\n> > +struct fd_writer {\n> > +\tconst struct reftable_write_options *opts;\n> > +\tint fd;\n> > +};\n> > +\n> > +static ssize_t fd_writer_write(void *arg, const void *data, size_t sz)\n> > {\n> > -\tint *fdp = (int *)arg;\n> > +\tstruct fd_writer *writer = arg;\n> > +\treturn write_in_full(writer->fd, data, sz);\n> \n> Tangent: eventually we'd also want to get rid of `write_in_full` no?\n> Since it is also an internal Git implementation.\n> \n> The patch itself looks good to me.\n\nYup. This is part of the next and final patch series that I'll send once\nthis series here has landed.\n\nPatrick\n"},{"id":"507506","messageId":"ZztcrOog-UtHx3us@pks.im","threadId":"62395","inReplyTo":"CAOLa=ZQxtQ9zBqN9U8ZnxaQ+uyMqQ1EnfEL9Br918y6CBxtvVQ@mail.gmail.com","subject":"Re: [PATCH v2 0/7] reftable: stop using Git subsystems","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-18T15:26:36Z","receivedAt":"2024-11-18T15:26:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Nov 18, 2024 at 02:30:44PM +0000, karthik nayak wrote:\n> Patrick Steinhardt writes:\n> \n> > Hi,\n> >\n> > this is the second version of my patch series that continues to detangle\n> > the reftable library from the Git codebase.\n> >\n> > Changes compared to v1:\n> >\n> > - Fix a commit message typo.\n> >\n> > - Document the values of the newly introduced reftable format IDs.\n> >\n> > - Include \"reftable-basics.h\" instead of \"basics.h\".\n> >\n> > - Adapt `stack_fsync()` to take write options as input instead of the\n> > whole stack.\n> >\n> > Thanks!\n> >\n> \n> [snip]\n> \n> I went through the patches and I think it looks great!\n\nThanks for your review!\n\nPatrick\n"},{"id":"507514","messageId":"cover.1731943954.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1729677003.git.ps@pks.im","subject":"[PATCH v3 0/7] reftable: stop using Git subsystems","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-18T15:33:49Z","receivedAt":"2024-11-18T15:34:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the third version of my patch series that continues to detangle\nthe reftable library from the Git codebase.\n\nThere is only a single change compared to v2, which is a clarification\nthat `tmpfile_rename()` also deactivates the underlying tmpfile.\n\nThanks!\n\nPatrick\n\nPatrick Steinhardt (7):\n reftable/system: move \"dir.h\" to its only user\n reftable: explicitly handle hash format IDs\n reftable/system: stop depending on \"hash.h\"\n reftable/stack: stop using `fsync_component()` directly\n reftable/system: provide thin wrapper for tempfile subsystem\n reftable/stack: drop only use of `get_locked_file_path()`\n reftable/system: provide thin wrapper for lockfile subsystem\n\n Makefile | 1 +\n refs/reftable-backend.c | 19 +++-\n reftable/basics.c | 13 ++-\n reftable/basics.h | 10 +-\n reftable/merged.c | 4 +-\n reftable/merged.h | 3 +-\n reftable/reader.c | 14 ++-\n reftable/reader.h | 4 +-\n reftable/reftable-basics.h | 13 +++\n reftable/reftable-merged.h | 4 +-\n reftable/reftable-reader.h | 2 +-\n reftable/reftable-record.h | 12 +-\n reftable/reftable-writer.h | 8 +-\n reftable/stack.c | 171 ++++++++++++++--------------\n reftable/system.c | 126 ++++++++++++++++++++\n reftable/system.h | 89 ++++++++++++++-\n reftable/writer.c | 20 +++-\n t/helper/test-reftable.c | 4 +-\n t/unit-tests/lib-reftable.c | 5 +-\n t/unit-tests/lib-reftable.h | 2 +-\n t/unit-tests/t-reftable-block.c | 41 +++----\n t/unit-tests/t-reftable-merged.c | 26 ++---\n t/unit-tests/t-reftable-pq.c | 3 +-\n t/unit-tests/t-reftable-reader.c | 4 +-\n t/unit-tests/t-reftable-readwrite.c | 41 +++----\n t/unit-tests/t-reftable-record.c | 59 +++++-----\n t/unit-tests/t-reftable-stack.c | 37 +++---\n 27 files changed, 506 insertions(+), 229 deletions(-)\n create mode 100644 reftable/system.c\n\nRange-diff against v2:\n1: 2b7d4e28529 = 1: 2b7d4e28529 reftable/system: move \"dir.h\" to its only user\n2: 38cfe85bf5b = 2: 38cfe85bf5b reftable: explicitly handle hash format IDs\n3: 745c1a070dd = 3: 745c1a070dd reftable/system: stop depending on \"hash.h\"\n4: 7782652b975 = 4: 7782652b975 reftable/stack: stop using `fsync_component()` directly\n5: b15daefbc83 ! 5: 430be1045d6 reftable/system: provide thin wrapper for tempfile subsystem\n @@ reftable/system.h: license that can be found in the LICENSE file or at\n +\n +/*\n + * Rename the temporary file to the provided path. The temporary file must be\n -+ * active. Return 0 on success, a reftable error code on error.\n ++ * active. Return 0 on success, a reftable error code on error. Deactivates the\n ++ * temporary file.\n + */\n +int tmpfile_rename(struct reftable_tmpfile *t, const char *path);\n \n6: 83949837a29 = 6: b9ffdc605b9 reftable/stack: drop only use of `get_locked_file_path()`\n7: 80fe5bc5e10 = 7: e1ac1cc2e67 reftable/system: provide thin wrapper for lockfile subsystem\n-- \n2.47.0.274.g962d0b743d.dirty\n\n"},{"id":"507515","messageId":"2b7d4e2852977659752f98b782784eaa6780bbdb.1731943954.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1731943954.git.ps@pks.im","subject":"[PATCH v3 1/7] reftable/system: move \"dir.h\" to its only user","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-18T15:33:52Z","receivedAt":"2024-11-18T15:34:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We still include \"dir.h\" in \"reftable/system.h\" even though it is not\nused by anything but by a single unit test. Move it over into that unit\ntest so that we don't accidentally use any functionality provided by it\nin the reftable codebase.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/system.h | 1 -\n t/unit-tests/t-reftable-stack.c | 1 +\n 2 files changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 5ec85833434..8564213475e 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -15,7 +15,6 @@ license that can be found in the LICENSE file or at\n #include \"lockfile.h\"\n #include \"tempfile.h\"\n #include \"hash.h\" /* hash ID, sizes.*/\n-#include \"dir.h\" /* remove_dir_recursively, for tests.*/\n \n int hash_size(uint32_t id);\n \ndiff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c\nindex 72f6747064f..1b4363a58fc 100644\n--- a/t/unit-tests/t-reftable-stack.c\n+++ b/t/unit-tests/t-reftable-stack.c\n@@ -8,6 +8,7 @@ license that can be found in the LICENSE file or at\n \n #include \"test-lib.h\"\n #include \"lib-reftable.h\"\n+#include \"dir.h\"\n #include \"reftable/merged.h\"\n #include \"reftable/reader.h\"\n #include \"reftable/reftable-error.h\"\n-- \n2.47.0.274.g962d0b743d.dirty\n\n"},{"id":"507516","messageId":"38cfe85bf5b82c70848e4b295ba6cae33dcfd667.1731943954.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1731943954.git.ps@pks.im","subject":"[PATCH v3 2/7] reftable: explicitly handle hash format IDs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-18T15:33:55Z","receivedAt":"2024-11-18T15:34:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The hash format IDs are used for two different things across the\nreftable codebase:\n\n - They are used as a 32 bit unsigned integer when reading and writing\n the header in order to identify the hash function.\n\n - They are used internally to identify which hash function is in use.\n\nWhen one only considers the second usecase one might think that one can\neasily change the representation of those hash IDs. But because those\nIDs end up in the reftable header and footer on disk it is important\nthat those never change.\n\nCreate separate constants `REFTABLE_FORMAT_ID_*` and use them in\ncontexts where we read or write reftable headers. This serves multiple\npurposes:\n\n - It allows us to more easily discern cases where we actually use\n those constants for the on-disk format.\n\n - It detangles us from the same constants that are defined in\n libgit.a, which is another required step to convert the reftable\n library to become standalone.\n\n - It makes the next step easier where we stop using `GIT_*_FORMAT_ID`\n constants in favor of a custom enum.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 8 ++++++++\n reftable/reader.c | 10 ++++++----\n reftable/writer.c | 16 +++++++++++++++-\n 3 files changed, 29 insertions(+), 5 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 7aa46d7c30d..bcab0b529b0 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -150,4 +150,12 @@ int common_prefix_size(struct reftable_buf *a, struct reftable_buf *b);\n \n int hash_size(uint32_t id);\n \n+/*\n+ * Format IDs that identify the hash function used by a reftable. Note that\n+ * these constants end up on disk and thus mustn't change. The format IDs are\n+ * \"sha1\" and \"s256\" in big endian, respectively.\n+ */\n+#define REFTABLE_FORMAT_ID_SHA1 ((uint32_t) 0x73686131)\n+#define REFTABLE_FORMAT_ID_SHA256 ((uint32_t) 0x73323536)\n+\n #endif\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 90dc950b577..64eb6938efe 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -109,16 +109,18 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n \tif (r->version == 1) {\n \t\tr->hash_id = GIT_SHA1_FORMAT_ID;\n \t} else {\n-\t\tr->hash_id = get_be32(f);\n-\t\tswitch (r->hash_id) {\n-\t\tcase GIT_SHA1_FORMAT_ID:\n+\t\tswitch (get_be32(f)) {\n+\t\tcase REFTABLE_FORMAT_ID_SHA1:\n+\t\t\tr->hash_id = GIT_SHA1_FORMAT_ID;\n \t\t\tbreak;\n-\t\tcase GIT_SHA256_FORMAT_ID:\n+\t\tcase REFTABLE_FORMAT_ID_SHA256:\n+\t\t\tr->hash_id = GIT_SHA256_FORMAT_ID;\n \t\t\tbreak;\n \t\tdefault:\n \t\t\terr = REFTABLE_FORMAT_ERROR;\n \t\t\tgoto done;\n \t\t}\n+\n \t\tf += 4;\n \t}\n \ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex fd136794d5a..9aa45de6340 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -103,8 +103,22 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \tput_be64(dest + 8, w->min_update_index);\n \tput_be64(dest + 16, w->max_update_index);\n \tif (writer_version(w) == 2) {\n-\t\tput_be32(dest + 24, w->opts.hash_id);\n+\t\tuint32_t hash_id;\n+\n+\t\tswitch (w->opts.hash_id) {\n+\t\tcase GIT_SHA1_FORMAT_ID:\n+\t\t\thash_id = REFTABLE_FORMAT_ID_SHA1;\n+\t\t\tbreak;\n+\t\tcase GIT_SHA256_FORMAT_ID:\n+\t\t\thash_id = REFTABLE_FORMAT_ID_SHA256;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tput_be32(dest + 24, hash_id);\n \t}\n+\n \treturn header_size(writer_version(w));\n }\n \n-- \n2.47.0.274.g962d0b743d.dirty\n\n"},{"id":"507518","messageId":"745c1a070ddd7180f0ee89c6a1d057a256894599.1731943954.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1731943954.git.ps@pks.im","subject":"[PATCH v3 3/7] reftable/system: stop depending on \"hash.h\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-18T15:33:57Z","receivedAt":"2024-11-18T15:34:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We include \"hash.h\" in \"reftable/system.h\" such that we can use hash\nformat IDs as well as the raw size of SHA1 and SHA256. As we are in the\nprocess of converting the reftable library to become standalone we of\ncourse cannot rely on those constants anymore.\n\nIntroduce a new `enum reftable_hash` to replace internal uses of the\nhash format IDs and new constants that replace internal uses of the hash\nsize. Adapt the reftable backend to set up the correct hash function.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 12 +++++-\n reftable/basics.c | 13 ++++---\n reftable/basics.h | 2 +-\n reftable/merged.c | 4 +-\n reftable/merged.h | 3 +-\n reftable/reader.c | 8 ++--\n reftable/reader.h | 4 +-\n reftable/reftable-basics.h | 13 +++++++\n reftable/reftable-merged.h | 4 +-\n reftable/reftable-reader.h | 2 +-\n reftable/reftable-record.h | 12 +++---\n reftable/reftable-writer.h | 2 +-\n reftable/stack.c | 4 +-\n reftable/system.h | 3 --\n reftable/writer.c | 8 ++--\n t/helper/test-reftable.c | 4 +-\n t/unit-tests/lib-reftable.c | 4 +-\n t/unit-tests/lib-reftable.h | 2 +-\n t/unit-tests/t-reftable-block.c | 40 +++++++++----------\n t/unit-tests/t-reftable-merged.c | 26 ++++++-------\n t/unit-tests/t-reftable-pq.c | 2 +-\n t/unit-tests/t-reftable-reader.c | 4 +-\n t/unit-tests/t-reftable-readwrite.c | 40 +++++++++----------\n t/unit-tests/t-reftable-record.c | 59 +++++++++++++++--------------\n t/unit-tests/t-reftable-stack.c | 34 ++++++++---------\n 25 files changed, 166 insertions(+), 143 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 3c6107c7ce5..7d86d920970 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -15,6 +15,7 @@\n #include \"../object.h\"\n #include \"../path.h\"\n #include \"../refs.h\"\n+#include \"../reftable/reftable-basics.h\"\n #include \"../reftable/reftable-stack.h\"\n #include \"../reftable/reftable-record.h\"\n #include \"../reftable/reftable-error.h\"\n@@ -289,7 +290,16 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \trefs->store_flags = store_flags;\n \trefs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);\n \n-\trefs->write_options.hash_id = repo->hash_algo->format_id;\n+\tswitch (repo->hash_algo->format_id) {\n+\tcase GIT_SHA1_FORMAT_ID:\n+\t\trefs->write_options.hash_id = REFTABLE_HASH_SHA1;\n+\t\tbreak;\n+\tcase GIT_SHA256_FORMAT_ID:\n+\t\trefs->write_options.hash_id = REFTABLE_HASH_SHA256;\n+\t\tbreak;\n+\tdefault:\n+\t\tBUG(\"unknown hash algorithm %d\", repo->hash_algo->format_id);\n+\t}\n \trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n \trefs->write_options.disable_auto_compact =\n \t\t!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1);\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex bc4fcc91446..7d84a5d62de 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -271,14 +271,15 @@ int common_prefix_size(struct reftable_buf *a, struct reftable_buf *b)\n \treturn p;\n }\n \n-int hash_size(uint32_t id)\n+int hash_size(enum reftable_hash id)\n {\n+\tif (!id)\n+\t\treturn REFTABLE_HASH_SIZE_SHA1;\n \tswitch (id) {\n-\tcase 0:\n-\tcase GIT_SHA1_FORMAT_ID:\n-\t\treturn GIT_SHA1_RAWSZ;\n-\tcase GIT_SHA256_FORMAT_ID:\n-\t\treturn GIT_SHA256_RAWSZ;\n+\tcase REFTABLE_HASH_SHA1:\n+\t\treturn REFTABLE_HASH_SIZE_SHA1;\n+\tcase REFTABLE_HASH_SHA256:\n+\t\treturn REFTABLE_HASH_SIZE_SHA256;\n \t}\n \tabort();\n }\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex bcab0b529b0..36beda2c25a 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -148,7 +148,7 @@ char *reftable_strdup(const char *str);\n /* Find the longest shared prefix size of `a` and `b` */\n int common_prefix_size(struct reftable_buf *a, struct reftable_buf *b);\n \n-int hash_size(uint32_t id);\n+int hash_size(enum reftable_hash id);\n \n /*\n * Format IDs that identify the hash function used by a reftable. Note that\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 514d6facf45..5b93e20f429 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -181,7 +181,7 @@ static void iterator_from_merged_iter(struct reftable_iterator *it,\n \n int reftable_merged_table_new(struct reftable_merged_table **dest,\n \t\t\t struct reftable_reader **readers, size_t n,\n-\t\t\t uint32_t hash_id)\n+\t\t\t enum reftable_hash hash_id)\n {\n \tstruct reftable_merged_table *m = NULL;\n \tuint64_t last_max = 0;\n@@ -293,7 +293,7 @@ int reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n \treturn merged_table_init_iter(mt, it, BLOCK_TYPE_LOG);\n }\n \n-uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\n+enum reftable_hash reftable_merged_table_hash_id(struct reftable_merged_table *mt)\n {\n \treturn mt->hash_id;\n }\ndiff --git a/reftable/merged.h b/reftable/merged.h\nindex 89bd0c4b35b..0b7d939e92b 100644\n--- a/reftable/merged.h\n+++ b/reftable/merged.h\n@@ -10,11 +10,12 @@ license that can be found in the LICENSE file or at\n #define MERGED_H\n \n #include \"system.h\"\n+#include \"reftable-basics.h\"\n \n struct reftable_merged_table {\n \tstruct reftable_reader **readers;\n \tsize_t readers_len;\n-\tuint32_t hash_id;\n+\tenum reftable_hash hash_id;\n \n \t/* If unset, produce deletions. This is useful for compaction. For the\n \t * full stack, deletions should be produced. */\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 64eb6938efe..ea82955c9bc 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -67,7 +67,7 @@ static int reader_get_block(struct reftable_reader *r,\n \treturn block_source_read_block(&r->source, dest, off, sz);\n }\n \n-uint32_t reftable_reader_hash_id(struct reftable_reader *r)\n+enum reftable_hash reftable_reader_hash_id(struct reftable_reader *r)\n {\n \treturn r->hash_id;\n }\n@@ -107,14 +107,14 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n \tf += 8;\n \n \tif (r->version == 1) {\n-\t\tr->hash_id = GIT_SHA1_FORMAT_ID;\n+\t\tr->hash_id = REFTABLE_HASH_SHA1;\n \t} else {\n \t\tswitch (get_be32(f)) {\n \t\tcase REFTABLE_FORMAT_ID_SHA1:\n-\t\t\tr->hash_id = GIT_SHA1_FORMAT_ID;\n+\t\t\tr->hash_id = REFTABLE_HASH_SHA1;\n \t\t\tbreak;\n \t\tcase REFTABLE_FORMAT_ID_SHA256:\n-\t\t\tr->hash_id = GIT_SHA256_FORMAT_ID;\n+\t\t\tr->hash_id = REFTABLE_HASH_SHA256;\n \t\t\tbreak;\n \t\tdefault:\n \t\t\terr = REFTABLE_FORMAT_ERROR;\ndiff --git a/reftable/reader.h b/reftable/reader.h\nindex 010fbfe8511..d2b48a48499 100644\n--- a/reftable/reader.h\n+++ b/reftable/reader.h\n@@ -37,8 +37,8 @@ struct reftable_reader {\n \t/* Size of the file, excluding the footer. */\n \tuint64_t size;\n \n-\t/* 'sha1' for SHA1, 's256' for SHA-256 */\n-\tuint32_t hash_id;\n+\t/* The hash function used for ref records. */\n+\tenum reftable_hash hash_id;\n \n \tuint32_t block_size;\n \tuint64_t min_update_index;\ndiff --git a/reftable/reftable-basics.h b/reftable/reftable-basics.h\nindex 6e8e636b716..e0397ed5836 100644\n--- a/reftable/reftable-basics.h\n+++ b/reftable/reftable-basics.h\n@@ -11,6 +11,19 @@\n \n #include \n \n+/*\n+ * Hash functions understood by the reftable library. Note that the values are\n+ * arbitrary and somewhat random such that we can easily detect cases where the\n+ * hash hasn't been properly set up.\n+ */\n+enum reftable_hash {\n+\tREFTABLE_HASH_SHA1 = 89,\n+\tREFTABLE_HASH_SHA256 = 247,\n+};\n+#define REFTABLE_HASH_SIZE_SHA1 20\n+#define REFTABLE_HASH_SIZE_SHA256 32\n+#define REFTABLE_HASH_SIZE_MAX REFTABLE_HASH_SIZE_SHA256\n+\n /* Overrides the functions to use for memory management. */\n void reftable_set_alloc(void *(*malloc)(size_t),\n \t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nindex a970d5dd89a..f2d01c3ef82 100644\n--- a/reftable/reftable-merged.h\n+++ b/reftable/reftable-merged.h\n@@ -34,7 +34,7 @@ struct reftable_reader;\n */\n int reftable_merged_table_new(struct reftable_merged_table **dest,\n \t\t\t struct reftable_reader **readers, size_t n,\n-\t\t\t uint32_t hash_id);\n+\t\t\t enum reftable_hash hash_id);\n \n /* Initialize a merged table iterator for reading refs. */\n int reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n@@ -56,6 +56,6 @@ reftable_merged_table_min_update_index(struct reftable_merged_table *mt);\n void reftable_merged_table_free(struct reftable_merged_table *m);\n \n /* return the hash ID of the merged table. */\n-uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m);\n+enum reftable_hash reftable_merged_table_hash_id(struct reftable_merged_table *m);\n \n #endif\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nindex 6a2d0b693f5..0085fbb9032 100644\n--- a/reftable/reftable-reader.h\n+++ b/reftable/reftable-reader.h\n@@ -54,7 +54,7 @@ int reftable_reader_init_log_iterator(struct reftable_reader *r,\n \t\t\t\t struct reftable_iterator *it);\n \n /* returns the hash ID used in this table. */\n-uint32_t reftable_reader_hash_id(struct reftable_reader *r);\n+enum reftable_hash reftable_reader_hash_id(struct reftable_reader *r);\n \n /* return an iterator for the refs pointing to `oid`. */\n int reftable_reader_refs_for(struct reftable_reader *r,\ndiff --git a/reftable/reftable-record.h b/reftable/reftable-record.h\nindex 2d42463c581..ddd48eb5798 100644\n--- a/reftable/reftable-record.h\n+++ b/reftable/reftable-record.h\n@@ -9,7 +9,7 @@ license that can be found in the LICENSE file or at\n #ifndef REFTABLE_RECORD_H\n #define REFTABLE_RECORD_H\n \n-#include \"hash.h\"\n+#include \"reftable-basics.h\"\n #include \n \n /*\n@@ -40,10 +40,10 @@ struct reftable_ref_record {\n #define REFTABLE_NR_REF_VALUETYPES 4\n \t} value_type;\n \tunion {\n-\t\tunsigned char val1[GIT_MAX_RAWSZ];\n+\t\tunsigned char val1[REFTABLE_HASH_SIZE_MAX];\n \t\tstruct {\n-\t\t\tunsigned char value[GIT_MAX_RAWSZ]; /* first hash */\n-\t\t\tunsigned char target_value[GIT_MAX_RAWSZ]; /* second hash */\n+\t\t\tunsigned char value[REFTABLE_HASH_SIZE_MAX]; /* first hash */\n+\t\t\tunsigned char target_value[REFTABLE_HASH_SIZE_MAX]; /* second hash */\n \t\t} val2;\n \t\tchar *symref; /* referent, malloced 0-terminated string */\n \t} value;\n@@ -85,8 +85,8 @@ struct reftable_log_record {\n \n \tunion {\n \t\tstruct {\n-\t\t\tunsigned char new_hash[GIT_MAX_RAWSZ];\n-\t\t\tunsigned char old_hash[GIT_MAX_RAWSZ];\n+\t\t\tunsigned char new_hash[REFTABLE_HASH_SIZE_MAX];\n+\t\t\tunsigned char old_hash[REFTABLE_HASH_SIZE_MAX];\n \t\t\tchar *name;\n \t\t\tchar *email;\n \t\t\tuint64_t time;\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex e4fc9537883..211860d08a4 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -33,7 +33,7 @@ struct reftable_write_options {\n \t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n \t * Defaults to SHA1 if unset\n \t */\n-\tuint32_t hash_id;\n+\tenum reftable_hash hash_id;\n \n \t/* Default mode for creating files. If unset, use 0666 (+umask) */\n \tunsigned int default_permissions;\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex c33979536ef..9ae716ff375 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -73,7 +73,7 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \tif (_opts)\n \t\topts = *_opts;\n \tif (opts.hash_id == 0)\n-\t\topts.hash_id = GIT_SHA1_FORMAT_ID;\n+\t\topts.hash_id = REFTABLE_HASH_SHA1;\n \n \t*dest = NULL;\n \n@@ -1603,7 +1603,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n,\n \n static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n {\n-\tint version = (st->opts.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2;\n+\tint version = (st->opts.hash_id == REFTABLE_HASH_SHA1) ? 1 : 2;\n \tint overhead = header_size(version) - 1;\n \tuint64_t *sizes;\n \ndiff --git a/reftable/system.h b/reftable/system.h\nindex 8564213475e..38d3534620e 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -14,8 +14,5 @@ license that can be found in the LICENSE file or at\n #include \"git-compat-util.h\"\n #include \"lockfile.h\"\n #include \"tempfile.h\"\n-#include \"hash.h\" /* hash ID, sizes.*/\n-\n-int hash_size(uint32_t id);\n \n #endif\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 9aa45de6340..ea2f831fc58 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -79,7 +79,7 @@ static void options_set_defaults(struct reftable_write_options *opts)\n \t}\n \n \tif (opts->hash_id == 0) {\n-\t\topts->hash_id = GIT_SHA1_FORMAT_ID;\n+\t\topts->hash_id = REFTABLE_HASH_SHA1;\n \t}\n \tif (opts->block_size == 0) {\n \t\topts->block_size = DEFAULT_BLOCK_SIZE;\n@@ -88,7 +88,7 @@ static void options_set_defaults(struct reftable_write_options *opts)\n \n static int writer_version(struct reftable_writer *w)\n {\n-\treturn (w->opts.hash_id == 0 || w->opts.hash_id == GIT_SHA1_FORMAT_ID) ?\n+\treturn (w->opts.hash_id == 0 || w->opts.hash_id == REFTABLE_HASH_SHA1) ?\n \t\t\t 1 :\n \t\t\t 2;\n }\n@@ -106,10 +106,10 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \t\tuint32_t hash_id;\n \n \t\tswitch (w->opts.hash_id) {\n-\t\tcase GIT_SHA1_FORMAT_ID:\n+\t\tcase REFTABLE_HASH_SHA1:\n \t\t\thash_id = REFTABLE_FORMAT_ID_SHA1;\n \t\t\tbreak;\n-\t\tcase GIT_SHA256_FORMAT_ID:\n+\t\tcase REFTABLE_HASH_SHA256:\n \t\t\thash_id = REFTABLE_FORMAT_ID_SHA256;\n \t\t\tbreak;\n \t\tdefault:\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 5c8849d115b..3c72ed985b3 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -156,7 +156,7 @@ int cmd__dump_reftable(int argc, const char **argv)\n \tint opt_dump_blocks = 0;\n \tint opt_dump_table = 0;\n \tint opt_dump_stack = 0;\n-\tuint32_t opt_hash_id = GIT_SHA1_FORMAT_ID;\n+\tuint32_t opt_hash_id = REFTABLE_HASH_SHA1;\n \tconst char *arg = NULL, *argv0 = argv[0];\n \n \tfor (; argc > 1; argv++, argc--)\n@@ -167,7 +167,7 @@ int cmd__dump_reftable(int argc, const char **argv)\n \t\telse if (!strcmp(\"-t\", argv[1]))\n \t\t\topt_dump_table = 1;\n \t\telse if (!strcmp(\"-6\", argv[1]))\n-\t\t\topt_hash_id = GIT_SHA256_FORMAT_ID;\n+\t\t\topt_hash_id = REFTABLE_HASH_SHA256;\n \t\telse if (!strcmp(\"-s\", argv[1]))\n \t\t\topt_dump_stack = 1;\n \t\telse if (!strcmp(\"-?\", argv[1]) || !strcmp(\"-h\", argv[1])) {\ndiff --git a/t/unit-tests/lib-reftable.c b/t/unit-tests/lib-reftable.c\nindex 2ddf480588d..c1631f45275 100644\n--- a/t/unit-tests/lib-reftable.c\n+++ b/t/unit-tests/lib-reftable.c\n@@ -3,7 +3,7 @@\n #include \"reftable/constants.h\"\n #include \"reftable/writer.h\"\n \n-void t_reftable_set_hash(uint8_t *p, int i, uint32_t id)\n+void t_reftable_set_hash(uint8_t *p, int i, enum reftable_hash id)\n {\n \tmemset(p, (uint8_t)i, hash_size(id));\n }\n@@ -82,7 +82,7 @@ void t_reftable_write_to_buf(struct reftable_buf *buf,\n \t\tsize_t off = i * (opts.block_size ? opts.block_size\n \t\t\t\t\t\t : DEFAULT_BLOCK_SIZE);\n \t\tif (!off)\n-\t\t\toff = header_size(opts.hash_id == GIT_SHA256_FORMAT_ID ? 2 : 1);\n+\t\t\toff = header_size(opts.hash_id == REFTABLE_HASH_SHA256 ? 2 : 1);\n \t\tcheck_char(buf->buf[off], ==, 'r');\n \t}\n \ndiff --git a/t/unit-tests/lib-reftable.h b/t/unit-tests/lib-reftable.h\nindex d4950fed3da..e4c360fa7ee 100644\n--- a/t/unit-tests/lib-reftable.h\n+++ b/t/unit-tests/lib-reftable.h\n@@ -6,7 +6,7 @@\n \n struct reftable_buf;\n \n-void t_reftable_set_hash(uint8_t *p, int i, uint32_t id);\n+void t_reftable_set_hash(uint8_t *p, int i, enum reftable_hash id);\n \n struct reftable_writer *t_reftable_strbuf_writer(struct reftable_buf *buf,\n \t\t\t\t\t\t struct reftable_write_options *opts);\ndiff --git a/t/unit-tests/t-reftable-block.c b/t/unit-tests/t-reftable-block.c\nindex f9af907117b..13e10807dae 100644\n--- a/t/unit-tests/t-reftable-block.c\n+++ b/t/unit-tests/t-reftable-block.c\n@@ -36,7 +36,7 @@ static void t_ref_block_read_write(void)\n \tblock.len = block_size;\n \tblock_source_from_buf(&block.source ,&buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n-\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\t\t\t\theader_off, hash_size(REFTABLE_HASH_SHA1));\n \tcheck(!ret);\n \n \trec.u.ref.refname = (char *) \"\";\n@@ -47,7 +47,7 @@ static void t_ref_block_read_write(void)\n \tfor (i = 0; i < N; i++) {\n \t\trec.u.ref.refname = xstrfmt(\"branch%02\"PRIuMAX, (uintmax_t)i);\n \t\trec.u.ref.value_type = REFTABLE_REF_VAL1;\n-\t\tmemset(rec.u.ref.value.val1, i, GIT_SHA1_RAWSZ);\n+\t\tmemset(rec.u.ref.value.val1, i, REFTABLE_HASH_SIZE_SHA1);\n \n \t\trecs[i] = rec;\n \t\tret = block_writer_add(&bw, &rec);\n@@ -61,7 +61,7 @@ static void t_ref_block_read_write(void)\n \n \tblock_writer_release(&bw);\n \n-\tblock_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ);\n+\tblock_reader_init(&br, &block, header_off, block_size, REFTABLE_HASH_SIZE_SHA1);\n \n \tblock_iter_seek_start(&it, &br);\n \n@@ -72,7 +72,7 @@ static void t_ref_block_read_write(void)\n \t\t\tcheck_int(i, ==, N);\n \t\t\tbreak;\n \t\t}\n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -85,7 +85,7 @@ static void t_ref_block_read_write(void)\n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n \n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \n \t\twant.len--;\n \t\tret = block_iter_seek_key(&it, &br, &want);\n@@ -93,7 +93,7 @@ static void t_ref_block_read_write(void)\n \n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n-\t\tcheck(reftable_record_equal(&recs[10 * (i / 10)], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[10 * (i / 10)], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tblock_reader_release(&br);\n@@ -130,7 +130,7 @@ static void t_log_block_read_write(void)\n \tblock.len = block_size;\n \tblock_source_from_buf(&block.source ,&buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size,\n-\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\t\t\t\theader_off, hash_size(REFTABLE_HASH_SHA1));\n \tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n@@ -150,7 +150,7 @@ static void t_log_block_read_write(void)\n \n \tblock_writer_release(&bw);\n \n-\tblock_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ);\n+\tblock_reader_init(&br, &block, header_off, block_size, REFTABLE_HASH_SIZE_SHA1);\n \n \tblock_iter_seek_start(&it, &br);\n \n@@ -161,7 +161,7 @@ static void t_log_block_read_write(void)\n \t\t\tcheck_int(i, ==, N);\n \t\t\tbreak;\n \t\t}\n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -175,7 +175,7 @@ static void t_log_block_read_write(void)\n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n \n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \n \t\twant.len--;\n \t\tret = block_iter_seek_key(&it, &br, &want);\n@@ -183,7 +183,7 @@ static void t_log_block_read_write(void)\n \n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n-\t\tcheck(reftable_record_equal(&recs[10 * (i / 10)], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[10 * (i / 10)], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tblock_reader_release(&br);\n@@ -220,7 +220,7 @@ static void t_obj_block_read_write(void)\n \tblock.len = block_size;\n \tblock_source_from_buf(&block.source, &buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size,\n-\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\t\t\t\theader_off, hash_size(REFTABLE_HASH_SHA1));\n \tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n@@ -242,7 +242,7 @@ static void t_obj_block_read_write(void)\n \n \tblock_writer_release(&bw);\n \n-\tblock_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ);\n+\tblock_reader_init(&br, &block, header_off, block_size, REFTABLE_HASH_SIZE_SHA1);\n \n \tblock_iter_seek_start(&it, &br);\n \n@@ -253,7 +253,7 @@ static void t_obj_block_read_write(void)\n \t\t\tcheck_int(i, ==, N);\n \t\t\tbreak;\n \t\t}\n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -266,7 +266,7 @@ static void t_obj_block_read_write(void)\n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n \n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tblock_reader_release(&br);\n@@ -304,7 +304,7 @@ static void t_index_block_read_write(void)\n \tblock.len = block_size;\n \tblock_source_from_buf(&block.source, &buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size,\n-\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\t\t\t\theader_off, hash_size(REFTABLE_HASH_SHA1));\n \tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n@@ -326,7 +326,7 @@ static void t_index_block_read_write(void)\n \n \tblock_writer_release(&bw);\n \n-\tblock_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ);\n+\tblock_reader_init(&br, &block, header_off, block_size, REFTABLE_HASH_SIZE_SHA1);\n \n \tblock_iter_seek_start(&it, &br);\n \n@@ -337,7 +337,7 @@ static void t_index_block_read_write(void)\n \t\t\tcheck_int(i, ==, N);\n \t\t\tbreak;\n \t\t}\n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -350,7 +350,7 @@ static void t_index_block_read_write(void)\n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n \n-\t\tcheck(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1));\n \n \t\twant.len--;\n \t\tret = block_iter_seek_key(&it, &br, &want);\n@@ -358,7 +358,7 @@ static void t_index_block_read_write(void)\n \n \t\tret = block_iter_next(&it, &rec);\n \t\tcheck_int(ret, ==, 0);\n-\t\tcheck(reftable_record_equal(&recs[10 * (i / 10)], &rec, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&recs[10 * (i / 10)], &rec, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \tblock_reader_release(&br);\ndiff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c\nindex 484c18251f3..0573d9470a6 100644\n--- a/t/unit-tests/t-reftable-merged.c\n+++ b/t/unit-tests/t-reftable-merged.c\n@@ -42,7 +42,7 @@ merged_table_from_records(struct reftable_ref_record **refs,\n \t\tcheck(!err);\n \t}\n \n-\terr = reftable_merged_table_new(&mt, *readers, n, GIT_SHA1_FORMAT_ID);\n+\terr = reftable_merged_table_new(&mt, *readers, n, REFTABLE_HASH_SHA1);\n \tcheck(!err);\n \treturn mt;\n }\n@@ -91,7 +91,7 @@ static void t_merged_single_record(void)\n \n \terr = reftable_iterator_next_ref(&it, &ref);\n \tcheck(!err);\n-\tcheck(reftable_ref_record_equal(&r2[0], &ref, GIT_SHA1_RAWSZ));\n+\tcheck(reftable_ref_record_equal(&r2[0], &ref, REFTABLE_HASH_SIZE_SHA1));\n \treftable_ref_record_release(&ref);\n \treftable_iterator_destroy(&it);\n \treaders_destroy(readers, 3);\n@@ -168,7 +168,7 @@ static void t_merged_refs(void)\n \tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"a\");\n \tcheck(!err);\n-\tcheck_int(reftable_merged_table_hash_id(mt), ==, GIT_SHA1_FORMAT_ID);\n+\tcheck_int(reftable_merged_table_hash_id(mt), ==, REFTABLE_HASH_SHA1);\n \tcheck_int(reftable_merged_table_min_update_index(mt), ==, 1);\n \tcheck_int(reftable_merged_table_max_update_index(mt), ==, 3);\n \n@@ -186,7 +186,7 @@ static void t_merged_refs(void)\n \tcheck_int(ARRAY_SIZE(want), ==, len);\n \tfor (i = 0; i < len; i++)\n \t\tcheck(reftable_ref_record_equal(want[i], &out[i],\n-\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1));\n \tfor (i = 0; i < len; i++)\n \t\treftable_ref_record_release(&out[i]);\n \treftable_free(out);\n@@ -252,12 +252,12 @@ static void t_merged_seek_multiple_times(void)\n \n \t\terr = reftable_iterator_next_ref(&it, &rec);\n \t\tcheck(!err);\n-\t\terr = reftable_ref_record_equal(&rec, &r1[1], GIT_SHA1_RAWSZ);\n+\t\terr = reftable_ref_record_equal(&rec, &r1[1], REFTABLE_HASH_SIZE_SHA1);\n \t\tcheck(err == 1);\n \n \t\terr = reftable_iterator_next_ref(&it, &rec);\n \t\tcheck(!err);\n-\t\terr = reftable_ref_record_equal(&rec, &r2[1], GIT_SHA1_RAWSZ);\n+\t\terr = reftable_ref_record_equal(&rec, &r2[1], REFTABLE_HASH_SIZE_SHA1);\n \t\tcheck(err == 1);\n \n \t\terr = reftable_iterator_next_ref(&it, &rec);\n@@ -300,7 +300,7 @@ merged_table_from_log_records(struct reftable_log_record **logs,\n \t\tcheck(!err);\n \t}\n \n-\terr = reftable_merged_table_new(&mt, *readers, n, GIT_SHA1_FORMAT_ID);\n+\terr = reftable_merged_table_new(&mt, *readers, n, REFTABLE_HASH_SHA1);\n \tcheck(!err);\n \treturn mt;\n }\n@@ -377,7 +377,7 @@ static void t_merged_logs(void)\n \tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"a\");\n \tcheck(!err);\n-\tcheck_int(reftable_merged_table_hash_id(mt), ==, GIT_SHA1_FORMAT_ID);\n+\tcheck_int(reftable_merged_table_hash_id(mt), ==, REFTABLE_HASH_SHA1);\n \tcheck_int(reftable_merged_table_min_update_index(mt), ==, 1);\n \tcheck_int(reftable_merged_table_max_update_index(mt), ==, 3);\n \n@@ -395,7 +395,7 @@ static void t_merged_logs(void)\n \tcheck_int(ARRAY_SIZE(want), ==, len);\n \tfor (i = 0; i < len; i++)\n \t\tcheck(reftable_log_record_equal(want[i], &out[i],\n-\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1));\n \n \terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n \tcheck(!err);\n@@ -404,7 +404,7 @@ static void t_merged_logs(void)\n \treftable_log_record_release(&out[0]);\n \terr = reftable_iterator_next_log(&it, &out[0]);\n \tcheck(!err);\n-\tcheck(reftable_log_record_equal(&out[0], &r3[0], GIT_SHA1_RAWSZ));\n+\tcheck(reftable_log_record_equal(&out[0], &r3[0], REFTABLE_HASH_SIZE_SHA1));\n \treftable_iterator_destroy(&it);\n \n \tfor (i = 0; i < len; i++)\n@@ -448,11 +448,11 @@ static void t_default_write_opts(void)\n \tcheck(!err);\n \n \thash_id = reftable_reader_hash_id(rd);\n-\tcheck_int(hash_id, ==, GIT_SHA1_FORMAT_ID);\n+\tcheck_int(hash_id, ==, REFTABLE_HASH_SHA1);\n \n-\terr = reftable_merged_table_new(&merged, &rd, 1, GIT_SHA256_FORMAT_ID);\n+\terr = reftable_merged_table_new(&merged, &rd, 1, REFTABLE_HASH_SHA256);\n \tcheck_int(err, ==, REFTABLE_FORMAT_ERROR);\n-\terr = reftable_merged_table_new(&merged, &rd, 1, GIT_SHA1_FORMAT_ID);\n+\terr = reftable_merged_table_new(&merged, &rd, 1, REFTABLE_HASH_SHA1);\n \tcheck(!err);\n \n \treftable_reader_decref(rd);\ndiff --git a/t/unit-tests/t-reftable-pq.c b/t/unit-tests/t-reftable-pq.c\nindex ada4c19f18a..272da05bea6 100644\n--- a/t/unit-tests/t-reftable-pq.c\n+++ b/t/unit-tests/t-reftable-pq.c\n@@ -132,7 +132,7 @@ static void t_merged_iter_pqueue_top(void)\n \n \t\tmerged_iter_pqueue_check(&pq);\n \t\tcheck(pq_entry_equal(&top, &e));\n-\t\tcheck(reftable_record_equal(top.rec, &recs[i], GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(top.rec, &recs[i], REFTABLE_HASH_SIZE_SHA1));\n \t\tfor (size_t j = 0; i < pq.len; j++) {\n \t\t\tcheck(pq_less(&top, &pq.heap[j]));\n \t\t\tcheck_int(top.index, >, j);\ndiff --git a/t/unit-tests/t-reftable-reader.c b/t/unit-tests/t-reftable-reader.c\nindex 19cb53b6415..546df6005e4 100644\n--- a/t/unit-tests/t-reftable-reader.c\n+++ b/t/unit-tests/t-reftable-reader.c\n@@ -31,7 +31,7 @@ static int t_reader_seek_once(void)\n \tret = reftable_iterator_next_ref(&it, &ref);\n \tcheck(!ret);\n \n-\tret = reftable_ref_record_equal(&ref, &records[0], GIT_SHA1_RAWSZ);\n+\tret = reftable_ref_record_equal(&ref, &records[0], REFTABLE_HASH_SIZE_SHA1);\n \tcheck_int(ret, ==, 1);\n \n \tret = reftable_iterator_next_ref(&it, &ref);\n@@ -74,7 +74,7 @@ static int t_reader_reseek(void)\n \t\tret = reftable_iterator_next_ref(&it, &ref);\n \t\tcheck(!ret);\n \n-\t\tret = reftable_ref_record_equal(&ref, &records[0], GIT_SHA1_RAWSZ);\n+\t\tret = reftable_ref_record_equal(&ref, &records[0], REFTABLE_HASH_SIZE_SHA1);\n \t\tcheck_int(ret, ==, 1);\n \n \t\tret = reftable_iterator_next_ref(&it, &ref);\ndiff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c\nindex d279b86df0a..57896922eb1 100644\n--- a/t/unit-tests/t-reftable-readwrite.c\n+++ b/t/unit-tests/t-reftable-readwrite.c\n@@ -41,7 +41,7 @@ static void t_buffer(void)\n }\n \n static void write_table(char ***names, struct reftable_buf *buf, int N,\n-\t\t\tint block_size, uint32_t hash_id)\n+\t\t\tint block_size, enum reftable_hash hash_id)\n {\n \tstruct reftable_write_options opts = {\n \t\t.block_size = block_size,\n@@ -62,7 +62,7 @@ static void write_table(char ***names, struct reftable_buf *buf, int N,\n \t\trefs[i].refname = (*names)[i] = xstrfmt(\"refs/heads/branch%02d\", i);\n \t\trefs[i].update_index = update_index;\n \t\trefs[i].value_type = REFTABLE_REF_VAL1;\n-\t\tt_reftable_set_hash(refs[i].value.val1, i, GIT_SHA1_FORMAT_ID);\n+\t\tt_reftable_set_hash(refs[i].value.val1, i, REFTABLE_HASH_SHA1);\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -70,7 +70,7 @@ static void write_table(char ***names, struct reftable_buf *buf, int N,\n \t\tlogs[i].update_index = update_index;\n \t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n \t\tt_reftable_set_hash(logs[i].value.update.new_hash, i,\n-\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t REFTABLE_HASH_SHA1);\n \t\tlogs[i].value.update.message = (char *) \"message\";\n \t}\n \n@@ -104,7 +104,7 @@ static void t_log_buffer_size(void)\n \t/* This tests buffer extension for log compression. Must use a random\n \t hash, to ensure that the compressed part is larger than the original.\n \t*/\n-\tfor (i = 0; i < GIT_SHA1_RAWSZ; i++) {\n+\tfor (i = 0; i < REFTABLE_HASH_SIZE_SHA1; i++) {\n \t\tlog.value.update.old_hash[i] = (uint8_t)(git_rand() % 256);\n \t\tlog.value.update.new_hash[i] = (uint8_t)(git_rand() % 256);\n \t}\n@@ -191,9 +191,9 @@ static void t_log_write_read(void)\n \t\tlog.update_index = i;\n \t\tlog.value_type = REFTABLE_LOG_UPDATE;\n \t\tt_reftable_set_hash(log.value.update.old_hash, i,\n-\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t REFTABLE_HASH_SHA1);\n \t\tt_reftable_set_hash(log.value.update.new_hash, i + 1,\n-\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t REFTABLE_HASH_SHA1);\n \n \t\terr = reftable_writer_add_log(w, &log);\n \t\tcheck(!err);\n@@ -326,7 +326,7 @@ static void t_table_read_write_sequential(void)\n \tint err = 0;\n \tint j = 0;\n \n-\twrite_table(&names, &buf, N, 256, GIT_SHA1_FORMAT_ID);\n+\twrite_table(&names, &buf, N, 256, REFTABLE_HASH_SHA1);\n \n \tblock_source_from_buf(&source, &buf);\n \n@@ -361,7 +361,7 @@ static void t_table_write_small_table(void)\n \tchar **names;\n \tstruct reftable_buf buf = REFTABLE_BUF_INIT;\n \tint N = 1;\n-\twrite_table(&names, &buf, N, 4096, GIT_SHA1_FORMAT_ID);\n+\twrite_table(&names, &buf, N, 4096, REFTABLE_HASH_SHA1);\n \tcheck_int(buf.len, <, 200);\n \treftable_buf_release(&buf);\n \tfree_names(names);\n@@ -378,7 +378,7 @@ static void t_table_read_api(void)\n \tstruct reftable_log_record log = { 0 };\n \tstruct reftable_iterator it = { 0 };\n \n-\twrite_table(&names, &buf, N, 256, GIT_SHA1_FORMAT_ID);\n+\twrite_table(&names, &buf, N, 256, REFTABLE_HASH_SHA1);\n \n \tblock_source_from_buf(&source, &buf);\n \n@@ -400,7 +400,7 @@ static void t_table_read_api(void)\n \treftable_buf_release(&buf);\n }\n \n-static void t_table_read_write_seek(int index, int hash_id)\n+static void t_table_read_write_seek(int index, enum reftable_hash hash_id)\n {\n \tchar **names;\n \tstruct reftable_buf buf = REFTABLE_BUF_INIT;\n@@ -467,24 +467,24 @@ static void t_table_read_write_seek(int index, int hash_id)\n \n static void t_table_read_write_seek_linear(void)\n {\n-\tt_table_read_write_seek(0, GIT_SHA1_FORMAT_ID);\n+\tt_table_read_write_seek(0, REFTABLE_HASH_SHA1);\n }\n \n static void t_table_read_write_seek_linear_sha256(void)\n {\n-\tt_table_read_write_seek(0, GIT_SHA256_FORMAT_ID);\n+\tt_table_read_write_seek(0, REFTABLE_HASH_SHA256);\n }\n \n static void t_table_read_write_seek_index(void)\n {\n-\tt_table_read_write_seek(1, GIT_SHA1_FORMAT_ID);\n+\tt_table_read_write_seek(1, REFTABLE_HASH_SHA1);\n }\n \n static void t_table_refs_for(int indexed)\n {\n \tchar **want_names;\n \tint want_names_len = 0;\n-\tuint8_t want_hash[GIT_SHA1_RAWSZ];\n+\tuint8_t want_hash[REFTABLE_HASH_SIZE_SHA1];\n \n \tstruct reftable_write_options opts = {\n \t\t.block_size = 256,\n@@ -500,10 +500,10 @@ static void t_table_refs_for(int indexed)\n \twant_names = reftable_calloc(N + 1, sizeof(*want_names));\n \tcheck(want_names != NULL);\n \n-\tt_reftable_set_hash(want_hash, 4, GIT_SHA1_FORMAT_ID);\n+\tt_reftable_set_hash(want_hash, 4, REFTABLE_HASH_SHA1);\n \n \tfor (i = 0; i < N; i++) {\n-\t\tuint8_t hash[GIT_SHA1_RAWSZ];\n+\t\tuint8_t hash[REFTABLE_HASH_SIZE_SHA1];\n \t\tchar fill[51] = { 0 };\n \t\tchar name[100];\n \t\tstruct reftable_ref_record ref = { 0 };\n@@ -517,9 +517,9 @@ static void t_table_refs_for(int indexed)\n \n \t\tref.value_type = REFTABLE_REF_VAL2;\n \t\tt_reftable_set_hash(ref.value.val2.value, i / 4,\n-\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t REFTABLE_HASH_SHA1);\n \t\tt_reftable_set_hash(ref.value.val2.target_value, 3 + i / 4,\n-\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t REFTABLE_HASH_SHA1);\n \n \t\t/* 80 bytes / entry, so 3 entries per block. Yields 17\n \t\t */\n@@ -527,8 +527,8 @@ static void t_table_refs_for(int indexed)\n \t\tn = reftable_writer_add_ref(w, &ref);\n \t\tcheck_int(n, ==, 0);\n \n-\t\tif (!memcmp(ref.value.val2.value, want_hash, GIT_SHA1_RAWSZ) ||\n-\t\t !memcmp(ref.value.val2.target_value, want_hash, GIT_SHA1_RAWSZ))\n+\t\tif (!memcmp(ref.value.val2.value, want_hash, REFTABLE_HASH_SIZE_SHA1) ||\n+\t\t !memcmp(ref.value.val2.target_value, want_hash, REFTABLE_HASH_SIZE_SHA1))\n \t\t\twant_names[want_names_len++] = xstrdup(name);\n \t}\n \ndiff --git a/t/unit-tests/t-reftable-record.c b/t/unit-tests/t-reftable-record.c\nindex eb98bf2da91..42bc64cec87 100644\n--- a/t/unit-tests/t-reftable-record.c\n+++ b/t/unit-tests/t-reftable-record.c\n@@ -7,6 +7,7 @@\n */\n \n #include \"test-lib.h\"\n+#include \"reftable/basics.h\"\n #include \"reftable/constants.h\"\n #include \"reftable/record.h\"\n \n@@ -17,10 +18,10 @@ static void t_copy(struct reftable_record *rec)\n \n \ttyp = reftable_record_type(rec);\n \treftable_record_init(©, typ);\n-\treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n+\treftable_record_copy_from(©, rec, REFTABLE_HASH_SIZE_SHA1);\n \t/* do it twice to catch memory leaks */\n-\treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n-\tcheck(reftable_record_equal(rec, ©, GIT_SHA1_RAWSZ));\n+\treftable_record_copy_from(©, rec, REFTABLE_HASH_SIZE_SHA1);\n+\tcheck(reftable_record_equal(rec, ©, REFTABLE_HASH_SIZE_SHA1));\n \n \treftable_record_release(©);\n }\n@@ -59,7 +60,7 @@ static void t_varint_roundtrip(void)\n \n static void set_hash(uint8_t *h, int j)\n {\n-\tfor (int i = 0; i < hash_size(GIT_SHA1_FORMAT_ID); i++)\n+\tfor (int i = 0; i < hash_size(REFTABLE_HASH_SHA1); i++)\n \t\th[i] = (j >> i) & 0xff;\n }\n \n@@ -84,14 +85,14 @@ static void t_reftable_ref_record_comparison(void)\n \t\t},\n \t};\n \n-\tcheck(!reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n \n-\tcheck(!reftable_record_equal(&in[1], &in[2], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n \tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n \n \tin[1].u.ref.value_type = in[0].u.ref.value_type;\n-\tcheck(reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n }\n \n@@ -155,15 +156,15 @@ static void t_reftable_ref_record_roundtrip(void)\n \t\tcheck_int(reftable_record_is_deletion(&in), ==, i == REFTABLE_REF_DELETION);\n \n \t\treftable_record_key(&in, &key);\n-\t\tn = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);\n+\t\tn = reftable_record_encode(&in, dest, REFTABLE_HASH_SIZE_SHA1);\n \t\tcheck_int(n, >, 0);\n \n \t\t/* decode into a non-zero reftable_record to test for leaks. */\n-\t\tm = reftable_record_decode(&out, key, i, dest, GIT_SHA1_RAWSZ, &scratch);\n+\t\tm = reftable_record_decode(&out, key, i, dest, REFTABLE_HASH_SIZE_SHA1, &scratch);\n \t\tcheck_int(n, ==, m);\n \n \t\tcheck(reftable_ref_record_equal(&in.u.ref, &out.u.ref,\n-\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_record_release(&in);\n \n \t\treftable_buf_release(&key);\n@@ -193,15 +194,15 @@ static void t_reftable_log_record_comparison(void)\n \t\t},\n \t};\n \n-\tcheck(!reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n-\tcheck(!reftable_record_equal(&in[1], &in[2], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n+\tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n \tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n \t/* comparison should be reversed for equal keys, because\n \t * comparison is now performed on the basis of update indices */\n \tcheck_int(reftable_record_cmp(&in[0], &in[1]), <, 0);\n \n \tin[1].u.log.update_index = in[0].u.log.update_index;\n-\tcheck(reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n }\n \n@@ -303,15 +304,15 @@ static void t_reftable_log_record_roundtrip(void)\n \n \t\treftable_record_key(&rec, &key);\n \n-\t\tn = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);\n+\t\tn = reftable_record_encode(&rec, dest, REFTABLE_HASH_SIZE_SHA1);\n \t\tcheck_int(n, >=, 0);\n \t\tvaltype = reftable_record_val_type(&rec);\n \t\tm = reftable_record_decode(&out, key, valtype, dest,\n-\t\t\t\t\t GIT_SHA1_RAWSZ, &scratch);\n+\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1, &scratch);\n \t\tcheck_int(n, ==, m);\n \n \t\tcheck(reftable_log_record_equal(&in[i], &out.u.log,\n-\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_log_record_release(&in[i]);\n \t\treftable_buf_release(&key);\n \t\treftable_record_release(&out);\n@@ -380,20 +381,20 @@ static void t_reftable_obj_record_comparison(void)\n \t\t},\n \t};\n \n-\tcheck(!reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n \n-\tcheck(!reftable_record_equal(&in[1], &in[2], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n \tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n \n \tin[1].u.obj.offset_len = in[0].u.obj.offset_len;\n-\tcheck(reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n }\n \n static void t_reftable_obj_record_roundtrip(void)\n {\n-\tuint8_t testHash1[GIT_SHA1_RAWSZ] = { 1, 2, 3, 4, 0 };\n+\tuint8_t testHash1[REFTABLE_HASH_SIZE_SHA1] = { 1, 2, 3, 4, 0 };\n \tuint64_t till9[] = { 1, 2, 3, 4, 500, 600, 700, 800, 9000 };\n \tstruct reftable_obj_record recs[3] = {\n \t\t{\n@@ -435,14 +436,14 @@ static void t_reftable_obj_record_roundtrip(void)\n \t\tcheck(!reftable_record_is_deletion(&in));\n \t\tt_copy(&in);\n \t\treftable_record_key(&in, &key);\n-\t\tn = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);\n+\t\tn = reftable_record_encode(&in, dest, REFTABLE_HASH_SIZE_SHA1);\n \t\tcheck_int(n, >, 0);\n \t\textra = reftable_record_val_type(&in);\n \t\tm = reftable_record_decode(&out, key, extra, dest,\n-\t\t\t\t\t GIT_SHA1_RAWSZ, &scratch);\n+\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1, &scratch);\n \t\tcheck_int(n, ==, m);\n \n-\t\tcheck(reftable_record_equal(&in, &out, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_record_equal(&in, &out, REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_buf_release(&key);\n \t\treftable_record_release(&out);\n \t}\n@@ -473,14 +474,14 @@ static void t_reftable_index_record_comparison(void)\n \tcheck(!reftable_buf_addstr(&in[1].u.idx.last_key, \"refs/heads/master\"));\n \tcheck(!reftable_buf_addstr(&in[2].u.idx.last_key, \"refs/heads/branch\"));\n \n-\tcheck(!reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n \n-\tcheck(!reftable_record_equal(&in[1], &in[2], GIT_SHA1_RAWSZ));\n+\tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n \tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n \n \tin[1].u.idx.offset = in[0].u.idx.offset;\n-\tcheck(reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ));\n+\tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_cmp(&in[0], &in[1]));\n \n \tfor (size_t i = 0; i < ARRAY_SIZE(in); i++)\n@@ -516,15 +517,15 @@ static void t_reftable_index_record_roundtrip(void)\n \n \tcheck(!reftable_record_is_deletion(&in));\n \tcheck(!reftable_buf_cmp(&key, &in.u.idx.last_key));\n-\tn = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);\n+\tn = reftable_record_encode(&in, dest, REFTABLE_HASH_SIZE_SHA1);\n \tcheck_int(n, >, 0);\n \n \textra = reftable_record_val_type(&in);\n-\tm = reftable_record_decode(&out, key, extra, dest, GIT_SHA1_RAWSZ,\n+\tm = reftable_record_decode(&out, key, extra, dest, REFTABLE_HASH_SIZE_SHA1,\n \t\t\t\t &scratch);\n \tcheck_int(m, ==, n);\n \n-\tcheck(reftable_record_equal(&in, &out, GIT_SHA1_RAWSZ));\n+\tcheck(reftable_record_equal(&in, &out, REFTABLE_HASH_SIZE_SHA1));\n \n \treftable_record_release(&out);\n \treftable_buf_release(&key);\ndiff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c\nindex 1b4363a58fc..13fd8d8f941 100644\n--- a/t/unit-tests/t-reftable-stack.c\n+++ b/t/unit-tests/t-reftable-stack.c\n@@ -121,7 +121,7 @@ static void write_n_ref_tables(struct reftable_stack *st,\n \n \t\tsnprintf(buf, sizeof(buf), \"refs/heads/branch-%04\"PRIuMAX, (uintmax_t)i);\n \t\tref.refname = buf;\n-\t\tt_reftable_set_hash(ref.value.val1, i, GIT_SHA1_FORMAT_ID);\n+\t\tt_reftable_set_hash(ref.value.val1, i, REFTABLE_HASH_SHA1);\n \n \t\terr = reftable_stack_add(st, &write_test_ref, &ref);\n \t\tcheck(!err);\n@@ -169,7 +169,7 @@ static void t_reftable_stack_add_one(void)\n \n \terr = reftable_stack_read_ref(st, ref.refname, &dest);\n \tcheck(!err);\n-\tcheck(reftable_ref_record_equal(&ref, &dest, GIT_SHA1_RAWSZ));\n+\tcheck(reftable_ref_record_equal(&ref, &dest, REFTABLE_HASH_SIZE_SHA1));\n \tcheck_int(st->readers_len, >, 0);\n \n #ifndef GIT_WINDOWS_NATIVE\n@@ -280,7 +280,7 @@ static void t_reftable_stack_transaction_api(void)\n \terr = reftable_stack_read_ref(st, ref.refname, &dest);\n \tcheck(!err);\n \tcheck_int(REFTABLE_REF_SYMREF, ==, dest.value_type);\n-\tcheck(reftable_ref_record_equal(&ref, &dest, GIT_SHA1_RAWSZ));\n+\tcheck(reftable_ref_record_equal(&ref, &dest, REFTABLE_HASH_SIZE_SHA1));\n \n \treftable_ref_record_release(&dest);\n \treftable_stack_destroy(st);\n@@ -340,7 +340,7 @@ static void t_reftable_stack_transaction_with_reload(void)\n \tfor (size_t i = 0; i < ARRAY_SIZE(refs); i++) {\n \t\terr = reftable_stack_read_ref(st2, refs[i].refname, &ref);\n \t\tcheck(!err);\n-\t\tcheck(reftable_ref_record_equal(&refs[i], &ref, GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_ref_record_equal(&refs[i], &ref, REFTABLE_HASH_SIZE_SHA1));\n \t}\n \n \treftable_ref_record_release(&ref);\n@@ -530,13 +530,13 @@ static void t_reftable_stack_add(void)\n \t\trefs[i].refname = xstrdup(buf);\n \t\trefs[i].update_index = i + 1;\n \t\trefs[i].value_type = REFTABLE_REF_VAL1;\n-\t\tt_reftable_set_hash(refs[i].value.val1, i, GIT_SHA1_FORMAT_ID);\n+\t\tt_reftable_set_hash(refs[i].value.val1, i, REFTABLE_HASH_SHA1);\n \n \t\tlogs[i].refname = xstrdup(buf);\n \t\tlogs[i].update_index = N + i + 1;\n \t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n \t\tlogs[i].value.update.email = xstrdup(\"identity@invalid\");\n-\t\tt_reftable_set_hash(logs[i].value.update.new_hash, i, GIT_SHA1_FORMAT_ID);\n+\t\tt_reftable_set_hash(logs[i].value.update.new_hash, i, REFTABLE_HASH_SHA1);\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -562,7 +562,7 @@ static void t_reftable_stack_add(void)\n \t\tint err = reftable_stack_read_ref(st, refs[i].refname, &dest);\n \t\tcheck(!err);\n \t\tcheck(reftable_ref_record_equal(&dest, refs + i,\n-\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_ref_record_release(&dest);\n \t}\n \n@@ -571,7 +571,7 @@ static void t_reftable_stack_add(void)\n \t\tint err = reftable_stack_read_log(st, refs[i].refname, &dest);\n \t\tcheck(!err);\n \t\tcheck(reftable_log_record_equal(&dest, logs + i,\n-\t\t\t\t\t\t GIT_SHA1_RAWSZ));\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_log_record_release(&dest);\n \t}\n \n@@ -622,14 +622,14 @@ static void t_reftable_stack_iterator(void)\n \t\trefs[i].refname = xstrfmt(\"branch%02\"PRIuMAX, (uintmax_t)i);\n \t\trefs[i].update_index = i + 1;\n \t\trefs[i].value_type = REFTABLE_REF_VAL1;\n-\t\tt_reftable_set_hash(refs[i].value.val1, i, GIT_SHA1_FORMAT_ID);\n+\t\tt_reftable_set_hash(refs[i].value.val1, i, REFTABLE_HASH_SHA1);\n \n \t\tlogs[i].refname = xstrfmt(\"branch%02\"PRIuMAX, (uintmax_t)i);\n \t\tlogs[i].update_index = i + 1;\n \t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n \t\tlogs[i].value.update.email = xstrdup(\"johndoe@invalid\");\n \t\tlogs[i].value.update.message = xstrdup(\"commit\\n\");\n-\t\tt_reftable_set_hash(logs[i].value.update.new_hash, i, GIT_SHA1_FORMAT_ID);\n+\t\tt_reftable_set_hash(logs[i].value.update.new_hash, i, REFTABLE_HASH_SHA1);\n \t}\n \n \tfor (i = 0; i < N; i++) {\n@@ -656,7 +656,7 @@ static void t_reftable_stack_iterator(void)\n \t\tif (err > 0)\n \t\t\tbreak;\n \t\tcheck(!err);\n-\t\tcheck(reftable_ref_record_equal(&ref, &refs[i], GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_ref_record_equal(&ref, &refs[i], REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_ref_record_release(&ref);\n \t}\n \tcheck_int(i, ==, N);\n@@ -674,7 +674,7 @@ static void t_reftable_stack_iterator(void)\n \t\tif (err > 0)\n \t\t\tbreak;\n \t\tcheck(!err);\n-\t\tcheck(reftable_log_record_equal(&log, &logs[i], GIT_SHA1_RAWSZ));\n+\t\tcheck(reftable_log_record_equal(&log, &logs[i], REFTABLE_HASH_SIZE_SHA1));\n \t\treftable_log_record_release(&log);\n \t}\n \tcheck_int(i, ==, N);\n@@ -767,7 +767,7 @@ static void t_reftable_stack_tombstone(void)\n \t\tif (i % 2 == 0) {\n \t\t\trefs[i].value_type = REFTABLE_REF_VAL1;\n \t\t\tt_reftable_set_hash(refs[i].value.val1, i,\n-\t\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t\t REFTABLE_HASH_SHA1);\n \t\t}\n \n \t\tlogs[i].refname = xstrdup(buf);\n@@ -776,7 +776,7 @@ static void t_reftable_stack_tombstone(void)\n \t\tif (i % 2 == 0) {\n \t\t\tlogs[i].value_type = REFTABLE_LOG_UPDATE;\n \t\t\tt_reftable_set_hash(logs[i].value.update.new_hash, i,\n-\t\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t\t REFTABLE_HASH_SHA1);\n \t\t\tlogs[i].value.update.email =\n \t\t\t\txstrdup(\"identity@invalid\");\n \t\t}\n@@ -836,7 +836,7 @@ static void t_reftable_stack_hash_id(void)\n \t\t.value.symref = (char *) \"target\",\n \t\t.update_index = 1,\n \t};\n-\tstruct reftable_write_options opts32 = { .hash_id = GIT_SHA256_FORMAT_ID };\n+\tstruct reftable_write_options opts32 = { .hash_id = REFTABLE_HASH_SHA256 };\n \tstruct reftable_stack *st32 = NULL;\n \tstruct reftable_write_options opts_default = { 0 };\n \tstruct reftable_stack *st_default = NULL;\n@@ -859,7 +859,7 @@ static void t_reftable_stack_hash_id(void)\n \terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n \tcheck(!err);\n \n-\tcheck(reftable_ref_record_equal(&ref, &dest, GIT_SHA1_RAWSZ));\n+\tcheck(reftable_ref_record_equal(&ref, &dest, REFTABLE_HASH_SIZE_SHA1));\n \treftable_ref_record_release(&dest);\n \treftable_stack_destroy(st);\n \treftable_stack_destroy(st_default);\n@@ -909,7 +909,7 @@ static void t_reflog_expire(void)\n \t\tlogs[i].value.update.time = i;\n \t\tlogs[i].value.update.email = xstrdup(\"identity@invalid\");\n \t\tt_reftable_set_hash(logs[i].value.update.new_hash, i,\n-\t\t\t\t GIT_SHA1_FORMAT_ID);\n+\t\t\t\t REFTABLE_HASH_SHA1);\n \t}\n \n \tfor (i = 1; i <= N; i++) {\n-- \n2.47.0.274.g962d0b743d.dirty\n\n"},{"id":"507519","messageId":"7782652b97545f95c83aef08786c70384e5cfa52.1731943954.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1731943954.git.ps@pks.im","subject":"[PATCH v3 4/7] reftable/stack: stop using `fsync_component()` directly","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-18T15:34:00Z","receivedAt":"2024-11-18T15:34:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're executing `fsync_component()` directly in the reftable library so\nthat we can fsync data to disk depending on \"core.fsync\". But as we're\nin the process of converting the reftable library to become standalone\nwe cannot use that function in the library anymore.\n\nRefactor the code such that users of the library can inject a custom\nfsync function via the write options. This allows us to get rid of the\ndependency on \"write-or-die.h\".\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 7 ++++++\n reftable/reftable-writer.h | 6 +++++\n reftable/stack.c | 49 +++++++++++++++++++++++++-------------\n 3 files changed, 45 insertions(+), 17 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 7d86d920970..2e774176eda 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -24,6 +24,7 @@\n #include \"../setup.h\"\n #include \"../strmap.h\"\n #include \"../trace2.h\"\n+#include \"../write-or-die.h\"\n #include \"parse.h\"\n #include \"refs-internal.h\"\n \n@@ -273,6 +274,11 @@ static int reftable_be_config(const char *var, const char *value,\n \treturn 0;\n }\n \n+static int reftable_be_fsync(int fd)\n+{\n+\treturn fsync_component(FSYNC_COMPONENT_REFERENCE, fd);\n+}\n+\n static struct ref_store *reftable_be_init(struct repository *repo,\n \t\t\t\t\t const char *gitdir,\n \t\t\t\t\t unsigned int store_flags)\n@@ -304,6 +310,7 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \trefs->write_options.disable_auto_compact =\n \t\t!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1);\n \trefs->write_options.lock_timeout_ms = 100;\n+\trefs->write_options.fsync = reftable_be_fsync;\n \n \tgit_config(reftable_be_config, &refs->write_options);\n \ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 211860d08a4..c85ef5a5bd1 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -62,6 +62,12 @@ struct reftable_write_options {\n \t * negative value will cause us to block indefinitely.\n \t */\n \tlong lock_timeout_ms;\n+\n+\t/*\n+\t * Optional callback used to fsync files to disk. Falls back to using\n+\t * fsync(3P) when unset.\n+\t */\n+\tint (*fsync)(int fd);\n };\n \n /* reftable_block_stats holds statistics for a single block type */\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 9ae716ff375..c67bdd952ca 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -8,7 +8,6 @@ license that can be found in the LICENSE file or at\n \n #include \"stack.h\"\n \n-#include \"../write-or-die.h\"\n #include \"system.h\"\n #include \"constants.h\"\n #include \"merged.h\"\n@@ -43,17 +42,28 @@ static int stack_filename(struct reftable_buf *dest, struct reftable_stack *st,\n \treturn 0;\n }\n \n-static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz)\n+static int stack_fsync(const struct reftable_write_options *opts, int fd)\n {\n-\tint *fdp = (int *)arg;\n-\treturn write_in_full(*fdp, data, sz);\n+\tif (opts->fsync)\n+\t\treturn opts->fsync(fd);\n+\treturn fsync(fd);\n }\n \n-static int reftable_fd_flush(void *arg)\n+struct fd_writer {\n+\tconst struct reftable_write_options *opts;\n+\tint fd;\n+};\n+\n+static ssize_t fd_writer_write(void *arg, const void *data, size_t sz)\n {\n-\tint *fdp = (int *)arg;\n+\tstruct fd_writer *writer = arg;\n+\treturn write_in_full(writer->fd, data, sz);\n+}\n \n-\treturn fsync_component(FSYNC_COMPONENT_REFERENCE, *fdp);\n+static int fd_writer_flush(void *arg)\n+{\n+\tstruct fd_writer *writer = arg;\n+\treturn stack_fsync(writer->opts, writer->fd);\n }\n \n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n@@ -765,7 +775,7 @@ int reftable_addition_commit(struct reftable_addition *add)\n \t\tgoto done;\n \t}\n \n-\terr = fsync_component(FSYNC_COMPONENT_REFERENCE, lock_file_fd);\n+\terr = stack_fsync(&add->stack->opts, lock_file_fd);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n@@ -858,8 +868,10 @@ int reftable_addition_add(struct reftable_addition *add,\n \tstruct reftable_buf next_name = REFTABLE_BUF_INIT;\n \tstruct reftable_writer *wr = NULL;\n \tstruct tempfile *tab_file = NULL;\n+\tstruct fd_writer writer = {\n+\t\t.opts = &add->stack->opts,\n+\t};\n \tint err = 0;\n-\tint tab_fd;\n \n \treftable_buf_reset(&next_name);\n \n@@ -887,10 +899,10 @@ int reftable_addition_add(struct reftable_addition *add,\n \t\t\tgoto done;\n \t\t}\n \t}\n-\ttab_fd = get_tempfile_fd(tab_file);\n \n-\terr = reftable_writer_new(&wr, reftable_fd_write, reftable_fd_flush,\n-\t\t\t\t &tab_fd, &add->stack->opts);\n+\twriter.fd = get_tempfile_fd(tab_file);\n+\terr = reftable_writer_new(&wr, fd_writer_write, fd_writer_flush,\n+\t\t\t\t &writer, &add->stack->opts);\n \tif (err < 0)\n \t\tgoto done;\n \n@@ -973,8 +985,11 @@ static int stack_compact_locked(struct reftable_stack *st,\n \tstruct reftable_buf next_name = REFTABLE_BUF_INIT;\n \tstruct reftable_buf tab_file_path = REFTABLE_BUF_INIT;\n \tstruct reftable_writer *wr = NULL;\n+\tstruct fd_writer writer= {\n+\t\t.opts = &st->opts,\n+\t};\n \tstruct tempfile *tab_file;\n-\tint tab_fd, err = 0;\n+\tint err = 0;\n \n \terr = format_name(&next_name, reftable_reader_min_update_index(st->readers[first]),\n \t\t\t reftable_reader_max_update_index(st->readers[last]));\n@@ -994,7 +1009,6 @@ static int stack_compact_locked(struct reftable_stack *st,\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n-\ttab_fd = get_tempfile_fd(tab_file);\n \n \tif (st->opts.default_permissions &&\n \t chmod(get_tempfile_path(tab_file), st->opts.default_permissions) < 0) {\n@@ -1002,8 +1016,9 @@ static int stack_compact_locked(struct reftable_stack *st,\n \t\tgoto done;\n \t}\n \n-\terr = reftable_writer_new(&wr, reftable_fd_write, reftable_fd_flush,\n-\t\t\t\t &tab_fd, &st->opts);\n+\twriter.fd = get_tempfile_fd(tab_file);\n+\terr = reftable_writer_new(&wr, fd_writer_write, fd_writer_flush,\n+\t\t\t\t &writer, &st->opts);\n \tif (err < 0)\n \t\tgoto done;\n \n@@ -1460,7 +1475,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\tgoto done;\n \t}\n \n-\terr = fsync_component(FSYNC_COMPONENT_REFERENCE, get_lock_file_fd(&tables_list_lock));\n+\terr = stack_fsync(&st->opts, get_lock_file_fd(&tables_list_lock));\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tunlink(new_table_path.buf);\n-- \n2.47.0.274.g962d0b743d.dirty\n\n"},{"id":"507520","messageId":"430be1045d6b9728748ed9dcabe0aab3a9b67859.1731943954.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1731943954.git.ps@pks.im","subject":"[PATCH v3 5/7] reftable/system: provide thin wrapper for tempfile subsystem","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-18T15:34:03Z","receivedAt":"2024-11-18T15:34:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We use the tempfile subsystem to write temporary tables, but given that\nwe're in the process of converting the reftable library to become\nstandalone we cannot use this subsystem directly anymore. While we could\nin theory convert the code to use mkstemp(3p) instead, we'd lose access\nto our infrastructure that automatically prunes tempfiles via atexit(3p)\nor signal handlers.\n\nProvide a thin wrapper for the tempfile subsystem instead. Like this,\nthe compatibility shim is fully self-contained in \"reftable/system.c\".\nDownstream users of the reftable library would have to implement their\nown tempfile shims by replacing \"system.c\" with a custom version.\n\nSigned-off-by: Patrick Steinhardt \n---\n Makefile | 1 +\n reftable/stack.c | 57 +++++++++++++++++++----------------------------\n reftable/system.c | 49 ++++++++++++++++++++++++++++++++++++++++\n reftable/system.h | 42 +++++++++++++++++++++++++++++++++-\n 4 files changed, 114 insertions(+), 35 deletions(-)\n create mode 100644 reftable/system.c\n\ndiff --git a/Makefile b/Makefile\nindex feeed6f9321..50a79ad83fc 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2722,6 +2722,7 @@ REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\n REFTABLE_OBJS += reftable/record.o\n REFTABLE_OBJS += reftable/stack.o\n+REFTABLE_OBJS += reftable/system.o\n REFTABLE_OBJS += reftable/tree.o\n REFTABLE_OBJS += reftable/writer.o\n \ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex c67bdd952ca..2ac6a371516 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -16,7 +16,6 @@ license that can be found in the LICENSE file or at\n #include \"reftable-record.h\"\n #include \"reftable-merged.h\"\n #include \"writer.h\"\n-#include \"tempfile.h\"\n \n static int stack_try_add(struct reftable_stack *st,\n \t\t\t int (*write_table)(struct reftable_writer *wr,\n@@ -867,7 +866,7 @@ int reftable_addition_add(struct reftable_addition *add,\n \tstruct reftable_buf tab_file_name = REFTABLE_BUF_INIT;\n \tstruct reftable_buf next_name = REFTABLE_BUF_INIT;\n \tstruct reftable_writer *wr = NULL;\n-\tstruct tempfile *tab_file = NULL;\n+\tstruct reftable_tmpfile tab_file = REFTABLE_TMPFILE_INIT;\n \tstruct fd_writer writer = {\n \t\t.opts = &add->stack->opts,\n \t};\n@@ -887,20 +886,18 @@ int reftable_addition_add(struct reftable_addition *add,\n \tif (err < 0)\n \t\tgoto done;\n \n-\ttab_file = mks_tempfile(temp_tab_file_name.buf);\n-\tif (!tab_file) {\n-\t\terr = REFTABLE_IO_ERROR;\n+\terr = tmpfile_from_pattern(&tab_file, temp_tab_file_name.buf);\n+\tif (err < 0)\n \t\tgoto done;\n-\t}\n \tif (add->stack->opts.default_permissions) {\n-\t\tif (chmod(get_tempfile_path(tab_file),\n+\t\tif (chmod(tab_file.path,\n \t\t\t add->stack->opts.default_permissions)) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n \t\t}\n \t}\n \n-\twriter.fd = get_tempfile_fd(tab_file);\n+\twriter.fd = tab_file.fd;\n \terr = reftable_writer_new(&wr, fd_writer_write, fd_writer_flush,\n \t\t\t\t &writer, &add->stack->opts);\n \tif (err < 0)\n@@ -918,11 +915,9 @@ int reftable_addition_add(struct reftable_addition *add,\n \tif (err < 0)\n \t\tgoto done;\n \n-\terr = close_tempfile_gently(tab_file);\n-\tif (err < 0) {\n-\t\terr = REFTABLE_IO_ERROR;\n+\terr = tmpfile_close(&tab_file);\n+\tif (err < 0)\n \t\tgoto done;\n-\t}\n \n \tif (wr->min_update_index < add->next_update_index) {\n \t\terr = REFTABLE_API_ERROR;\n@@ -945,11 +940,9 @@ int reftable_addition_add(struct reftable_addition *add,\n \t On windows, this relies on rand() picking a unique destination name.\n \t Maybe we should do retry loop as well?\n \t */\n-\terr = rename_tempfile(&tab_file, tab_file_name.buf);\n-\tif (err < 0) {\n-\t\terr = REFTABLE_IO_ERROR;\n+\terr = tmpfile_rename(&tab_file, tab_file_name.buf);\n+\tif (err < 0)\n \t\tgoto done;\n-\t}\n \n \tREFTABLE_ALLOC_GROW(add->new_tables, add->new_tables_len + 1,\n \t\t\t add->new_tables_cap);\n@@ -960,7 +953,7 @@ int reftable_addition_add(struct reftable_addition *add,\n \tadd->new_tables[add->new_tables_len++] = reftable_buf_detach(&next_name);\n \n done:\n-\tdelete_tempfile(&tab_file);\n+\ttmpfile_delete(&tab_file);\n \treftable_buf_release(&temp_tab_file_name);\n \treftable_buf_release(&tab_file_name);\n \treftable_buf_release(&next_name);\n@@ -980,7 +973,7 @@ uint64_t reftable_stack_next_update_index(struct reftable_stack *st)\n static int stack_compact_locked(struct reftable_stack *st,\n \t\t\t\tsize_t first, size_t last,\n \t\t\t\tstruct reftable_log_expiry_config *config,\n-\t\t\t\tstruct tempfile **tab_file_out)\n+\t\t\t\tstruct reftable_tmpfile *tab_file_out)\n {\n \tstruct reftable_buf next_name = REFTABLE_BUF_INIT;\n \tstruct reftable_buf tab_file_path = REFTABLE_BUF_INIT;\n@@ -988,7 +981,7 @@ static int stack_compact_locked(struct reftable_stack *st,\n \tstruct fd_writer writer= {\n \t\t.opts = &st->opts,\n \t};\n-\tstruct tempfile *tab_file;\n+\tstruct reftable_tmpfile tab_file = REFTABLE_TMPFILE_INIT;\n \tint err = 0;\n \n \terr = format_name(&next_name, reftable_reader_min_update_index(st->readers[first]),\n@@ -1004,19 +997,17 @@ static int stack_compact_locked(struct reftable_stack *st,\n \tif (err < 0)\n \t\tgoto done;\n \n-\ttab_file = mks_tempfile(tab_file_path.buf);\n-\tif (!tab_file) {\n-\t\terr = REFTABLE_IO_ERROR;\n+\terr = tmpfile_from_pattern(&tab_file, tab_file_path.buf);\n+\tif (err < 0)\n \t\tgoto done;\n-\t}\n \n \tif (st->opts.default_permissions &&\n-\t chmod(get_tempfile_path(tab_file), st->opts.default_permissions) < 0) {\n+\t chmod(tab_file.path, st->opts.default_permissions) < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n \n-\twriter.fd = get_tempfile_fd(tab_file);\n+\twriter.fd = tab_file.fd;\n \terr = reftable_writer_new(&wr, fd_writer_write, fd_writer_flush,\n \t\t\t\t &writer, &st->opts);\n \tif (err < 0)\n@@ -1030,15 +1021,15 @@ static int stack_compact_locked(struct reftable_stack *st,\n \tif (err < 0)\n \t\tgoto done;\n \n-\terr = close_tempfile_gently(tab_file);\n+\terr = tmpfile_close(&tab_file);\n \tif (err < 0)\n \t\tgoto done;\n \n \t*tab_file_out = tab_file;\n-\ttab_file = NULL;\n+\ttab_file = REFTABLE_TMPFILE_INIT;\n \n done:\n-\tdelete_tempfile(&tab_file);\n+\ttmpfile_delete(&tab_file);\n \treftable_writer_free(wr);\n \treftable_buf_release(&next_name);\n \treftable_buf_release(&tab_file_path);\n@@ -1171,7 +1162,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \tstruct reftable_buf table_name = REFTABLE_BUF_INIT;\n \tstruct lock_file tables_list_lock = LOCK_INIT;\n \tstruct lock_file *table_locks = NULL;\n-\tstruct tempfile *new_table = NULL;\n+\tstruct reftable_tmpfile new_table = REFTABLE_TMPFILE_INIT;\n \tint is_empty_table = 0, err = 0;\n \tsize_t first_to_replace, last_to_replace;\n \tsize_t i, nlocks = 0;\n@@ -1439,11 +1430,9 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\tif (err < 0)\n \t\t\tgoto done;\n \n-\t\terr = rename_tempfile(&new_table, new_table_path.buf);\n-\t\tif (err < 0) {\n-\t\t\terr = REFTABLE_IO_ERROR;\n+\t\terr = tmpfile_rename(&new_table, new_table_path.buf);\n+\t\tif (err < 0)\n \t\t\tgoto done;\n-\t\t}\n \t}\n \n \t/*\n@@ -1515,7 +1504,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\trollback_lock_file(&table_locks[i]);\n \treftable_free(table_locks);\n \n-\tdelete_tempfile(&new_table);\n+\ttmpfile_delete(&new_table);\n \treftable_buf_release(&new_table_name);\n \treftable_buf_release(&new_table_path);\n \treftable_buf_release(&tables_list_buf);\ndiff --git a/reftable/system.c b/reftable/system.c\nnew file mode 100644\nindex 00000000000..01f96f03d84\n--- /dev/null\n+++ b/reftable/system.c\n@@ -0,0 +1,49 @@\n+#include \"system.h\"\n+#include \"basics.h\"\n+#include \"reftable-error.h\"\n+#include \"../tempfile.h\"\n+\n+int tmpfile_from_pattern(struct reftable_tmpfile *out, const char *pattern)\n+{\n+\tstruct tempfile *tempfile;\n+\n+\ttempfile = mks_tempfile(pattern);\n+\tif (!tempfile)\n+\t\treturn REFTABLE_IO_ERROR;\n+\n+\tout->path = tempfile->filename.buf;\n+\tout->fd = tempfile->fd;\n+\tout->priv = tempfile;\n+\n+\treturn 0;\n+}\n+\n+int tmpfile_close(struct reftable_tmpfile *t)\n+{\n+\tstruct tempfile *tempfile = t->priv;\n+\tint ret = close_tempfile_gently(tempfile);\n+\tt->fd = -1;\n+\tif (ret < 0)\n+\t\treturn REFTABLE_IO_ERROR;\n+\treturn 0;\n+}\n+\n+int tmpfile_delete(struct reftable_tmpfile *t)\n+{\n+\tstruct tempfile *tempfile = t->priv;\n+\tint ret = delete_tempfile(&tempfile);\n+\t*t = REFTABLE_TMPFILE_INIT;\n+\tif (ret < 0)\n+\t\treturn REFTABLE_IO_ERROR;\n+\treturn 0;\n+}\n+\n+int tmpfile_rename(struct reftable_tmpfile *t, const char *path)\n+{\n+\tstruct tempfile *tempfile = t->priv;\n+\tint ret = rename_tempfile(&tempfile, path);\n+\t*t = REFTABLE_TMPFILE_INIT;\n+\tif (ret < 0)\n+\t\treturn REFTABLE_IO_ERROR;\n+\treturn 0;\n+}\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 38d3534620e..858189fd55d 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -13,6 +13,46 @@ license that can be found in the LICENSE file or at\n \n #include \"git-compat-util.h\"\n #include \"lockfile.h\"\n-#include \"tempfile.h\"\n+\n+/*\n+ * An implementation-specific temporary file. By making this specific to the\n+ * implementation it becomes possible to tie temporary files into any kind of\n+ * signal or atexit handlers for cleanup on abnormal situations.\n+ */\n+struct reftable_tmpfile {\n+\tconst char *path;\n+\tint fd;\n+\tvoid *priv;\n+};\n+#define REFTABLE_TMPFILE_INIT ((struct reftable_tmpfile) { .fd = -1, })\n+\n+/*\n+ * Create a temporary file from a pattern similar to how mkstemp(3p) would.\n+ * The `pattern` shall not be modified. On success, the structure at `out` has\n+ * been initialized such that it is ready for use. Returns 0 on success, a\n+ * reftable error code on error.\n+ */\n+int tmpfile_from_pattern(struct reftable_tmpfile *out, const char *pattern);\n+\n+/*\n+ * Close the temporary file's file descriptor without removing the file itself.\n+ * This is a no-op in case the file has already been closed beforehand. Returns\n+ * 0 on success, a reftable error code on error.\n+ */\n+int tmpfile_close(struct reftable_tmpfile *t);\n+\n+/*\n+ * Close the temporary file and delete it. This is a no-op in case the file has\n+ * already been deleted or renamed beforehand. Returns 0 on success, a reftable\n+ * error code on error.\n+ */\n+int tmpfile_delete(struct reftable_tmpfile *t);\n+\n+/*\n+ * Rename the temporary file to the provided path. The temporary file must be\n+ * active. Return 0 on success, a reftable error code on error. Deactivates the\n+ * temporary file.\n+ */\n+int tmpfile_rename(struct reftable_tmpfile *t, const char *path);\n \n #endif\n-- \n2.47.0.274.g962d0b743d.dirty\n\n"},{"id":"507521","messageId":"b9ffdc605b9122fe0ba9525db0b8b3aef1a58186.1731943954.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1731943954.git.ps@pks.im","subject":"[PATCH v3 6/7] reftable/stack: drop only use of `get_locked_file_path()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-18T15:34:05Z","receivedAt":"2024-11-18T15:34:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We've got a single callsite where we call `get_locked_file_path()`. As\nwe're about to convert our usage of the lockfile subsystem to instead be\nused via a compatibility shim we'd have to implement more logic for this\nsingle callsite. While that would be okay if Git was the only supposed\nuser of the reftable library, it's a bit more awkward when considering\nthat we have to reimplement this functionality for every user of the\nlibrary eventually.\n\nRefactor the code such that we don't call `get_locked_file_path()`\nanymore.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 12 +++++++++---\n 1 file changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 2ac6a371516..223d7c622d9 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1493,9 +1493,15 @@ static int stack_compact_range(struct reftable_stack *st,\n \t */\n \tfor (i = 0; i < nlocks; i++) {\n \t\tstruct lock_file *table_lock = &table_locks[i];\n-\t\tchar *table_path = get_locked_file_path(table_lock);\n-\t\tunlink(table_path);\n-\t\treftable_free(table_path);\n+\t\tconst char *lock_path = get_lock_file_path(table_lock);\n+\n+\t\treftable_buf_reset(&table_name);\n+\t\terr = reftable_buf_add(&table_name, lock_path,\n+\t\t\t\t strlen(lock_path) - strlen(\".lock\"));\n+\t\tif (err)\n+\t\t\tcontinue;\n+\n+\t\tunlink(table_name.buf);\n \t}\n \n done:\n-- \n2.47.0.274.g962d0b743d.dirty\n\n"},{"id":"507523","messageId":"e1ac1cc2e677fddb03cb3bc37952b7be55652287.1731943954.git.ps@pks.im","threadId":"62395","inReplyTo":"cover.1731943954.git.ps@pks.im","subject":"[PATCH v3 7/7] reftable/system: provide thin wrapper for lockfile subsystem","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-18T15:34:08Z","receivedAt":"2024-11-18T15:34:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We use the lockfile subsystem to write lockfiles for \"tables.list\". As\nwith the tempfile subsystem, the lockfile subsystem also hooks into our\ninfrastructure to prune stale locks via atexit(3p) or signal handlers.\n\nFurthermore, the lockfile subsystem also handles locking timeouts, which\ndo add quite a bit of logic. Having to reimplement that in the context\nof Git wouldn't make a whole lot of sense, and it is quite likely that\ndownstream users of the reftable library may have a better idea for how\nexactly to implement timeouts.\n\nSo again, provide a thin wrapper for the lockfile subsystem instead such\nthat the compatibility shim is fully self-contained.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 63 ++++++++++-------------\n reftable/system.c | 77 +++++++++++++++++++++++++++++\n reftable/system.h | 45 ++++++++++++++++-\n t/unit-tests/lib-reftable.c | 1 +\n t/unit-tests/t-reftable-block.c | 1 +\n t/unit-tests/t-reftable-pq.c | 1 +\n t/unit-tests/t-reftable-readwrite.c | 1 +\n t/unit-tests/t-reftable-stack.c | 2 +\n 8 files changed, 154 insertions(+), 37 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 223d7c622d9..10d45e89d00 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -657,7 +657,7 @@ static int format_name(struct reftable_buf *dest, uint64_t min, uint64_t max)\n }\n \n struct reftable_addition {\n-\tstruct lock_file tables_list_lock;\n+\tstruct reftable_flock tables_list_lock;\n \tstruct reftable_stack *stack;\n \n \tchar **new_tables;\n@@ -676,10 +676,8 @@ static int reftable_stack_init_addition(struct reftable_addition *add,\n \n \tadd->stack = st;\n \n-\terr = hold_lock_file_for_update_timeout(&add->tables_list_lock,\n-\t\t\t\t\t\tst->list_file,\n-\t\t\t\t\t\tLOCK_NO_DEREF,\n-\t\t\t\t\t\tst->opts.lock_timeout_ms);\n+\terr = flock_acquire(&add->tables_list_lock, st->list_file,\n+\t\t\t st->opts.lock_timeout_ms);\n \tif (err < 0) {\n \t\tif (errno == EEXIST) {\n \t\t\terr = REFTABLE_LOCK_ERROR;\n@@ -689,7 +687,7 @@ static int reftable_stack_init_addition(struct reftable_addition *add,\n \t\tgoto done;\n \t}\n \tif (st->opts.default_permissions) {\n-\t\tif (chmod(get_lock_file_path(&add->tables_list_lock),\n+\t\tif (chmod(add->tables_list_lock.path,\n \t\t\t st->opts.default_permissions) < 0) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n@@ -733,7 +731,7 @@ static void reftable_addition_close(struct reftable_addition *add)\n \tadd->new_tables_len = 0;\n \tadd->new_tables_cap = 0;\n \n-\trollback_lock_file(&add->tables_list_lock);\n+\tflock_release(&add->tables_list_lock);\n \treftable_buf_release(&nm);\n }\n \n@@ -749,7 +747,6 @@ void reftable_addition_destroy(struct reftable_addition *add)\n int reftable_addition_commit(struct reftable_addition *add)\n {\n \tstruct reftable_buf table_list = REFTABLE_BUF_INIT;\n-\tint lock_file_fd = get_lock_file_fd(&add->tables_list_lock);\n \tint err = 0;\n \tsize_t i;\n \n@@ -767,20 +764,20 @@ int reftable_addition_commit(struct reftable_addition *add)\n \t\t\tgoto done;\n \t}\n \n-\terr = write_in_full(lock_file_fd, table_list.buf, table_list.len);\n+\terr = write_in_full(add->tables_list_lock.fd, table_list.buf, table_list.len);\n \treftable_buf_release(&table_list);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n \n-\terr = stack_fsync(&add->stack->opts, lock_file_fd);\n+\terr = stack_fsync(&add->stack->opts, add->tables_list_lock.fd);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n \n-\terr = commit_lock_file(&add->tables_list_lock);\n+\terr = flock_commit(&add->tables_list_lock);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n@@ -1160,8 +1157,8 @@ static int stack_compact_range(struct reftable_stack *st,\n \tstruct reftable_buf new_table_name = REFTABLE_BUF_INIT;\n \tstruct reftable_buf new_table_path = REFTABLE_BUF_INIT;\n \tstruct reftable_buf table_name = REFTABLE_BUF_INIT;\n-\tstruct lock_file tables_list_lock = LOCK_INIT;\n-\tstruct lock_file *table_locks = NULL;\n+\tstruct reftable_flock tables_list_lock = REFTABLE_FLOCK_INIT;\n+\tstruct reftable_flock *table_locks = NULL;\n \tstruct reftable_tmpfile new_table = REFTABLE_TMPFILE_INIT;\n \tint is_empty_table = 0, err = 0;\n \tsize_t first_to_replace, last_to_replace;\n@@ -1179,10 +1176,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * Hold the lock so that we can read \"tables.list\" and lock all tables\n \t * which are part of the user-specified range.\n \t */\n-\terr = hold_lock_file_for_update_timeout(&tables_list_lock,\n-\t\t\t\t\t\tst->list_file,\n-\t\t\t\t\t\tLOCK_NO_DEREF,\n-\t\t\t\t\t\tst->opts.lock_timeout_ms);\n+\terr = flock_acquire(&tables_list_lock, st->list_file, st->opts.lock_timeout_ms);\n \tif (err < 0) {\n \t\tif (errno == EEXIST)\n \t\t\terr = REFTABLE_LOCK_ERROR;\n@@ -1205,19 +1199,20 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * older process is still busy compacting tables which are preexisting\n \t * from the point of view of the newer process.\n \t */\n-\tREFTABLE_CALLOC_ARRAY(table_locks, last - first + 1);\n+\tREFTABLE_ALLOC_ARRAY(table_locks, last - first + 1);\n \tif (!table_locks) {\n \t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n \t\tgoto done;\n \t}\n+\tfor (i = 0; i < last - first + 1; i++)\n+\t\ttable_locks[i] = REFTABLE_FLOCK_INIT;\n \n \tfor (i = last + 1; i > first; i--) {\n \t\terr = stack_filename(&table_name, st, reader_name(st->readers[i - 1]));\n \t\tif (err < 0)\n \t\t\tgoto done;\n \n-\t\terr = hold_lock_file_for_update(&table_locks[nlocks],\n-\t\t\t\t\t\ttable_name.buf, LOCK_NO_DEREF);\n+\t\terr = flock_acquire(&table_locks[nlocks], table_name.buf, 0);\n \t\tif (err < 0) {\n \t\t\t/*\n \t\t\t * When the table is locked already we may do a\n@@ -1253,7 +1248,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\t * run into file descriptor exhaustion when we compress a lot\n \t\t * of tables.\n \t\t */\n-\t\terr = close_lock_file_gently(&table_locks[nlocks++]);\n+\t\terr = flock_close(&table_locks[nlocks++]);\n \t\tif (err < 0) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n@@ -1265,7 +1260,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * \"tables.list\" lock while compacting the locked tables. This allows\n \t * concurrent updates to the stack to proceed.\n \t */\n-\terr = rollback_lock_file(&tables_list_lock);\n+\terr = flock_release(&tables_list_lock);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n@@ -1288,10 +1283,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * \"tables.list\". We'll then replace the compacted range of tables with\n \t * the new table.\n \t */\n-\terr = hold_lock_file_for_update_timeout(&tables_list_lock,\n-\t\t\t\t\t\tst->list_file,\n-\t\t\t\t\t\tLOCK_NO_DEREF,\n-\t\t\t\t\t\tst->opts.lock_timeout_ms);\n+\terr = flock_acquire(&tables_list_lock, st->list_file, st->opts.lock_timeout_ms);\n \tif (err < 0) {\n \t\tif (errno == EEXIST)\n \t\t\terr = REFTABLE_LOCK_ERROR;\n@@ -1301,7 +1293,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t}\n \n \tif (st->opts.default_permissions) {\n-\t\tif (chmod(get_lock_file_path(&tables_list_lock),\n+\t\tif (chmod(tables_list_lock.path,\n \t\t\t st->opts.default_permissions) < 0) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n@@ -1456,7 +1448,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\t\tgoto done;\n \t}\n \n-\terr = write_in_full(get_lock_file_fd(&tables_list_lock),\n+\terr = write_in_full(tables_list_lock.fd,\n \t\t\t tables_list_buf.buf, tables_list_buf.len);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n@@ -1464,14 +1456,14 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\tgoto done;\n \t}\n \n-\terr = stack_fsync(&st->opts, get_lock_file_fd(&tables_list_lock));\n+\terr = stack_fsync(&st->opts, tables_list_lock.fd);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tunlink(new_table_path.buf);\n \t\tgoto done;\n \t}\n \n-\terr = commit_lock_file(&tables_list_lock);\n+\terr = flock_commit(&tables_list_lock);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tunlink(new_table_path.buf);\n@@ -1492,12 +1484,11 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * readers, so it is expected that unlinking tables may fail.\n \t */\n \tfor (i = 0; i < nlocks; i++) {\n-\t\tstruct lock_file *table_lock = &table_locks[i];\n-\t\tconst char *lock_path = get_lock_file_path(table_lock);\n+\t\tstruct reftable_flock *table_lock = &table_locks[i];\n \n \t\treftable_buf_reset(&table_name);\n-\t\terr = reftable_buf_add(&table_name, lock_path,\n-\t\t\t\t strlen(lock_path) - strlen(\".lock\"));\n+\t\terr = reftable_buf_add(&table_name, table_lock->path,\n+\t\t\t\t strlen(table_lock->path) - strlen(\".lock\"));\n \t\tif (err)\n \t\t\tcontinue;\n \n@@ -1505,9 +1496,9 @@ static int stack_compact_range(struct reftable_stack *st,\n \t}\n \n done:\n-\trollback_lock_file(&tables_list_lock);\n+\tflock_release(&tables_list_lock);\n \tfor (i = 0; table_locks && i < nlocks; i++)\n-\t\trollback_lock_file(&table_locks[i]);\n+\t\tflock_release(&table_locks[i]);\n \treftable_free(table_locks);\n \n \ttmpfile_delete(&new_table);\ndiff --git a/reftable/system.c b/reftable/system.c\nindex 01f96f03d84..adf8e4d30b8 100644\n--- a/reftable/system.c\n+++ b/reftable/system.c\n@@ -1,6 +1,7 @@\n #include \"system.h\"\n #include \"basics.h\"\n #include \"reftable-error.h\"\n+#include \"../lockfile.h\"\n #include \"../tempfile.h\"\n \n int tmpfile_from_pattern(struct reftable_tmpfile *out, const char *pattern)\n@@ -47,3 +48,79 @@ int tmpfile_rename(struct reftable_tmpfile *t, const char *path)\n \t\treturn REFTABLE_IO_ERROR;\n \treturn 0;\n }\n+\n+int flock_acquire(struct reftable_flock *l, const char *target_path,\n+\t\t long timeout_ms)\n+{\n+\tstruct lock_file *lockfile;\n+\tint err;\n+\n+\tlockfile = reftable_malloc(sizeof(*lockfile));\n+\tif (!lockfile)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n+\terr = hold_lock_file_for_update_timeout(lockfile, target_path, LOCK_NO_DEREF,\n+\t\t\t\t\t\ttimeout_ms);\n+\tif (err < 0) {\n+\t\treftable_free(lockfile);\n+\t\tif (errno == EEXIST)\n+\t\t\treturn REFTABLE_LOCK_ERROR;\n+\t\treturn -1;\n+\t}\n+\n+\tl->fd = get_lock_file_fd(lockfile);\n+\tl->path = get_lock_file_path(lockfile);\n+\tl->priv = lockfile;\n+\n+\treturn 0;\n+}\n+\n+int flock_close(struct reftable_flock *l)\n+{\n+\tstruct lock_file *lockfile = l->priv;\n+\tint ret;\n+\n+\tif (!lockfile)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\tret = close_lock_file_gently(lockfile);\n+\tl->fd = -1;\n+\tif (ret < 0)\n+\t\treturn REFTABLE_IO_ERROR;\n+\n+\treturn 0;\n+}\n+\n+int flock_release(struct reftable_flock *l)\n+{\n+\tstruct lock_file *lockfile = l->priv;\n+\tint ret;\n+\n+\tif (!lockfile)\n+\t\treturn 0;\n+\n+\tret = rollback_lock_file(lockfile);\n+\treftable_free(lockfile);\n+\t*l = REFTABLE_FLOCK_INIT;\n+\tif (ret < 0)\n+\t\treturn REFTABLE_IO_ERROR;\n+\n+\treturn 0;\n+}\n+\n+int flock_commit(struct reftable_flock *l)\n+{\n+\tstruct lock_file *lockfile = l->priv;\n+\tint ret;\n+\n+\tif (!lockfile)\n+\t\treturn REFTABLE_API_ERROR;\n+\n+\tret = commit_lock_file(lockfile);\n+\treftable_free(lockfile);\n+\t*l = REFTABLE_FLOCK_INIT;\n+\tif (ret < 0)\n+\t\treturn REFTABLE_IO_ERROR;\n+\n+\treturn 0;\n+}\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 858189fd55d..7d5f803eeb1 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -12,7 +12,6 @@ license that can be found in the LICENSE file or at\n /* This header glues the reftable library to the rest of Git */\n \n #include \"git-compat-util.h\"\n-#include \"lockfile.h\"\n \n /*\n * An implementation-specific temporary file. By making this specific to the\n@@ -55,4 +54,48 @@ int tmpfile_delete(struct reftable_tmpfile *t);\n */\n int tmpfile_rename(struct reftable_tmpfile *t, const char *path);\n \n+/*\n+ * An implementation-specific file lock. Same as with `reftable_tmpfile`,\n+ * making this specific to the implementation makes it possible to tie this\n+ * into signal or atexit handlers such that we know to clean up stale locks on\n+ * abnormal exits.\n+ */\n+struct reftable_flock {\n+\tconst char *path;\n+\tint fd;\n+\tvoid *priv;\n+};\n+#define REFTABLE_FLOCK_INIT ((struct reftable_flock){ .fd = -1, })\n+\n+/*\n+ * Acquire the lock for the given target path by exclusively creating a file\n+ * with \".lock\" appended to it. If that lock exists, we wait up to `timeout_ms`\n+ * to acquire the lock. If `timeout_ms` is 0 we don't wait, if it is negative\n+ * we block indefinitely.\n+ *\n+ * Retrun 0 on success, a reftable error code on error.\n+ */\n+int flock_acquire(struct reftable_flock *l, const char *target_path,\n+\t\t long timeout_ms);\n+\n+/*\n+ * Close the lockfile's file descriptor without removing the lock itself. This\n+ * is a no-op in case the lockfile has already been closed beforehand. Returns\n+ * 0 on success, a reftable error code on error.\n+ */\n+int flock_close(struct reftable_flock *l);\n+\n+/*\n+ * Release the lock by unlinking the lockfile. This is a no-op in case the\n+ * lockfile has already been released or committed beforehand. Returns 0 on\n+ * success, a reftable error code on error.\n+ */\n+int flock_release(struct reftable_flock *l);\n+\n+/*\n+ * Commit the lock by renaming the lockfile into place. Returns 0 on success, a\n+ * reftable error code on error.\n+ */\n+int flock_commit(struct reftable_flock *l);\n+\n #endif\ndiff --git a/t/unit-tests/lib-reftable.c b/t/unit-tests/lib-reftable.c\nindex c1631f45275..d795dfb7c99 100644\n--- a/t/unit-tests/lib-reftable.c\n+++ b/t/unit-tests/lib-reftable.c\n@@ -2,6 +2,7 @@\n #include \"test-lib.h\"\n #include \"reftable/constants.h\"\n #include \"reftable/writer.h\"\n+#include \"strbuf.h\"\n \n void t_reftable_set_hash(uint8_t *p, int i, enum reftable_hash id)\n {\ndiff --git a/t/unit-tests/t-reftable-block.c b/t/unit-tests/t-reftable-block.c\nindex 13e10807dae..22040aeefa5 100644\n--- a/t/unit-tests/t-reftable-block.c\n+++ b/t/unit-tests/t-reftable-block.c\n@@ -11,6 +11,7 @@ license that can be found in the LICENSE file or at\n #include \"reftable/blocksource.h\"\n #include \"reftable/constants.h\"\n #include \"reftable/reftable-error.h\"\n+#include \"strbuf.h\"\n \n static void t_ref_block_read_write(void)\n {\ndiff --git a/t/unit-tests/t-reftable-pq.c b/t/unit-tests/t-reftable-pq.c\nindex 272da05bea6..f3f8a0cdf38 100644\n--- a/t/unit-tests/t-reftable-pq.c\n+++ b/t/unit-tests/t-reftable-pq.c\n@@ -9,6 +9,7 @@ license that can be found in the LICENSE file or at\n #include \"test-lib.h\"\n #include \"reftable/constants.h\"\n #include \"reftable/pq.h\"\n+#include \"strbuf.h\"\n \n static void merged_iter_pqueue_check(const struct merged_iter_pqueue *pq)\n {\ndiff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c\nindex 57896922eb1..91c881aedfa 100644\n--- a/t/unit-tests/t-reftable-readwrite.c\n+++ b/t/unit-tests/t-reftable-readwrite.c\n@@ -13,6 +13,7 @@ license that can be found in the LICENSE file or at\n #include \"reftable/reader.h\"\n #include \"reftable/reftable-error.h\"\n #include \"reftable/reftable-writer.h\"\n+#include \"strbuf.h\"\n \n static const int update_index = 5;\n \ndiff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c\nindex 13fd8d8f941..b2f6c1c37e9 100644\n--- a/t/unit-tests/t-reftable-stack.c\n+++ b/t/unit-tests/t-reftable-stack.c\n@@ -13,6 +13,8 @@ license that can be found in the LICENSE file or at\n #include \"reftable/reader.h\"\n #include \"reftable/reftable-error.h\"\n #include \"reftable/stack.h\"\n+#include \"strbuf.h\"\n+#include \"tempfile.h\"\n #include \n \n static void clear_dir(const char *dirname)\n-- \n2.47.0.274.g962d0b743d.dirty\n\n"},{"id":"507613","messageId":"CAOLa=ZSbj966S+kXtyr-1iJ3eqnDvDt=vN1kfPsSAFaAENmOJw@mail.gmail.com","threadId":"62395","inReplyTo":"cover.1731943954.git.ps@pks.im","subject":"Re: [PATCH v3 0/7] reftable: stop using Git subsystems","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-11-19T14:23:56Z","receivedAt":"2024-11-19T14:23:58Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n[snip]\n\n> Range-diff against v2:\n> 1: 2b7d4e28529 = 1: 2b7d4e28529 reftable/system: move \"dir.h\" to its only user\n> 2: 38cfe85bf5b = 2: 38cfe85bf5b reftable: explicitly handle hash format IDs\n> 3: 745c1a070dd = 3: 745c1a070dd reftable/system: stop depending on \"hash.h\"\n> 4: 7782652b975 = 4: 7782652b975 reftable/stack: stop using `fsync_component()` directly\n> 5: b15daefbc83 ! 5: 430be1045d6 reftable/system: provide thin wrapper for tempfile subsystem\n> @@ reftable/system.h: license that can be found in the LICENSE file or at\n> +\n> +/*\n> + * Rename the temporary file to the provided path. The temporary file must be\n> -+ * active. Return 0 on success, a reftable error code on error.\n> ++ * active. Return 0 on success, a reftable error code on error. Deactivates the\n> ++ * temporary file.\n> + */\n> +int tmpfile_rename(struct reftable_tmpfile *t, const char *path);\n>\n> 6: 83949837a29 = 6: b9ffdc605b9 reftable/stack: drop only use of `get_locked_file_path()`\n> 7: 80fe5bc5e10 = 7: e1ac1cc2e67 reftable/system: provide thin wrapper for lockfile subsystem\n\nPost my previous review, this range diff looks good. Thanks\n- Karthik\n"},{"id":"507668","messageId":"xmqq5xoibtjd.fsf@gitster.g","threadId":"62395","inReplyTo":"CAOLa=ZSbj966S+kXtyr-1iJ3eqnDvDt=vN1kfPsSAFaAENmOJw@mail.gmail.com","subject":"Re: [PATCH v3 0/7] reftable: stop using Git subsystems","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-20T01:09:10Z","receivedAt":"2024-11-20T01:09:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"karthik nayak writes:\n\n> Patrick Steinhardt writes:\n>\n> [snip]\n>\n>> Range-diff against v2:\n>> 1: 2b7d4e28529 = 1: 2b7d4e28529 reftable/system: move \"dir.h\" to its only user\n>> 2: 38cfe85bf5b = 2: 38cfe85bf5b reftable: explicitly handle hash format IDs\n>> 3: 745c1a070dd = 3: 745c1a070dd reftable/system: stop depending on \"hash.h\"\n>> 4: 7782652b975 = 4: 7782652b975 reftable/stack: stop using `fsync_component()` directly\n>> 5: b15daefbc83 ! 5: 430be1045d6 reftable/system: provide thin wrapper for tempfile subsystem\n>> @@ reftable/system.h: license that can be found in the LICENSE file or at\n>> +\n>> +/*\n>> + * Rename the temporary file to the provided path. The temporary file must be\n>> -+ * active. Return 0 on success, a reftable error code on error.\n>> ++ * active. Return 0 on success, a reftable error code on error. Deactivates the\n>> ++ * temporary file.\n>> + */\n>> +int tmpfile_rename(struct reftable_tmpfile *t, const char *path);\n>>\n>> 6: 83949837a29 = 6: b9ffdc605b9 reftable/stack: drop only use of `get_locked_file_path()`\n>> 7: 80fe5bc5e10 = 7: e1ac1cc2e67 reftable/system: provide thin wrapper for lockfile subsystem\n>\n> Post my previous review, this range diff looks good. Thanks\n\nThanks, both. Queued.\n"}]}