{"thread":{"id":"61406","subject":"[PATCH 00/11] reftable: expose write options as config","startedAt":"2024-05-02T06:51:32Z","lastAt":"2024-05-22T07:19:49Z","messageCount":78,"participants":["Patrick Steinhardt","Junio C Hamano","Justin Tobler","Karthik Nayak"],"isPatch":true,"patchVersion":1,"patchTotal":11},"messages":[{"id":"493877","messageId":"cover.1714630191.git.ps@pks.im","threadId":"61406","inReplyTo":null,"subject":"[PATCH 00/11] reftable: expose write options as config","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-02T06:51:27Z","receivedAt":"2024-05-02T06:51:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthe reftable format has some flexibility with regards to how exactly it\nwrites the respective tables:\n\n - The block size allows you to control how large each block is\n supposed to be. The bigger the block, the more records you can fit\n into it.\n\n - Restart intervals control how often a restart point is written that\n breaks prefix compression. The lower the interval, the less disk\n space savings you get.\n\n - Object indices can be enabled or disabled. These are optional and\n Git doesn't use them right now, so disabling them may be a sensible\n thing to do if you want to save some disk space.\n\n - The geometric factor controls when we compact tables during auto\n compaction.\n\nThis patch series exposes all of these via a new set of configs so that\nthey can be tweaked by the user as-needed. It's not expected that those\nare really of much importance for the \"normal\" user -- the defaults\nshould be good enough. But for edge cases (huge repos with millions of\nrefs) and for hosting providers these knobs can be helpful.\n\nThis patch series applies on top of d4cc1ec35f (Start the 2.46 cycle,\n2024-04-30).\n\nPatrick\n\nPatrick Steinhardt (11):\n reftable: consistently refer to `reftable_write_options` as `opts`\n reftable: consistently pass write opts as value\n reftable/writer: drop static variable used to initialize strbuf\n reftable/writer: improve error when passed an invalid block size\n reftable/dump: support dumping a table's block structure\n refs/reftable: allow configuring block size\n reftable: use `uint16_t` to track restart interval\n refs/reftable: allow configuring restart interval\n refs/reftable: allow disabling writing the object index\n reftable: make the compaction factor configurable\n refs/reftable: allow configuring geometric factor\n\n Documentation/config.txt | 2 +\n Documentation/config/reftable.txt | 49 +++++\n refs/reftable-backend.c | 46 ++++-\n reftable/block.h | 2 +-\n reftable/dump.c | 12 +-\n reftable/merged_test.c | 6 +-\n reftable/reader.c | 63 +++++++\n reftable/readwrite_test.c | 26 +--\n reftable/refname_test.c | 2 +-\n reftable/reftable-reader.h | 2 +\n reftable/reftable-stack.h | 2 +-\n reftable/reftable-writer.h | 10 +-\n reftable/stack.c | 56 +++---\n reftable/stack.h | 5 +-\n reftable/stack_test.c | 118 ++++++------\n reftable/writer.c | 20 +--\n t/t0613-reftable-write-options.sh | 286 ++++++++++++++++++++++++++++++\n 17 files changed, 577 insertions(+), 130 deletions(-)\n create mode 100644 Documentation/config/reftable.txt\n create mode 100755 t/t0613-reftable-write-options.sh\n\n\nbase-commit: d4cc1ec35f3bcce816b69986ca41943f6ce21377\n-- \n2.45.0\n\n"},{"id":"493878","messageId":"47cee6e25ebaac5af9fd0c9eaad7fc50b2547cad.1714630191.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1714630191.git.ps@pks.im","subject":"[PATCH 01/11] reftable: consistently refer to `reftable_write_options` as `opts`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-02T06:51:32Z","receivedAt":"2024-05-02T06:51:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Throughout the reftable library the `reftable_write_options` are\nsometimes referred to as `cfg` and sometimes as `opts`. Unify these to\nconsistently use `opts` to avoid confusion.\n\nWhile at it, touch up the coding style a bit by removing unneeded braces\naround one-line statements and newlines between variable declarations.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/dump.c | 4 +-\n reftable/reftable-stack.h | 2 +-\n reftable/stack.c | 43 +++++++-------\n reftable/stack.h | 2 +-\n reftable/stack_test.c | 114 +++++++++++++++++---------------------\n 5 files changed, 75 insertions(+), 90 deletions(-)\n\ndiff --git a/reftable/dump.c b/reftable/dump.c\nindex 26e0393c7d..9c770a10cc 100644\n--- a/reftable/dump.c\n+++ b/reftable/dump.c\n@@ -27,9 +27,9 @@ license that can be found in the LICENSE file or at\n static int compact_stack(const char *stackdir)\n {\n \tstruct reftable_stack *stack = NULL;\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \n-\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tint err = reftable_new_stack(&stack, stackdir, opts);\n \tif (err < 0)\n \t\tgoto done;\n \ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nindex 1b602dda58..9c8e4eef49 100644\n--- a/reftable/reftable-stack.h\n+++ b/reftable/reftable-stack.h\n@@ -29,7 +29,7 @@ struct reftable_stack;\n * stored in 'dir'. Typically, this should be .git/reftables.\n */\n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n-\t\t struct reftable_write_options config);\n+\t\t struct reftable_write_options opts);\n \n /* returns the update_index at which a next table should be written. */\n uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 80266bcbab..3979657793 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -56,15 +56,14 @@ static int reftable_fd_flush(void *arg)\n }\n \n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n-\t\t struct reftable_write_options config)\n+\t\t struct reftable_write_options opts)\n {\n \tstruct reftable_stack *p = reftable_calloc(1, sizeof(*p));\n \tstruct strbuf list_file_name = STRBUF_INIT;\n \tint err = 0;\n \n-\tif (config.hash_id == 0) {\n-\t\tconfig.hash_id = GIT_SHA1_FORMAT_ID;\n-\t}\n+\tif (opts.hash_id == 0)\n+\t\topts.hash_id = GIT_SHA1_FORMAT_ID;\n \n \t*dest = NULL;\n \n@@ -75,7 +74,7 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \tp->list_file = strbuf_detach(&list_file_name, NULL);\n \tp->list_fd = -1;\n \tp->reftable_dir = xstrdup(dir);\n-\tp->config = config;\n+\tp->opts = opts;\n \n \terr = reftable_stack_reload_maybe_reuse(p, 1);\n \tif (err < 0) {\n@@ -257,7 +256,7 @@ static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n \n \t/* success! */\n \terr = reftable_new_merged_table(&new_merged, new_tables,\n-\t\t\t\t\tnew_readers_len, st->config.hash_id);\n+\t\t\t\t\tnew_readers_len, st->opts.hash_id);\n \tif (err < 0)\n \t\tgoto done;\n \n@@ -580,8 +579,8 @@ static int reftable_stack_init_addition(struct reftable_addition *add,\n \t\t}\n \t\tgoto done;\n \t}\n-\tif (st->config.default_permissions) {\n-\t\tif (chmod(add->lock_file->filename.buf, st->config.default_permissions) < 0) {\n+\tif (st->opts.default_permissions) {\n+\t\tif (chmod(add->lock_file->filename.buf, st->opts.default_permissions) < 0) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n \t\t}\n@@ -680,7 +679,7 @@ int reftable_addition_commit(struct reftable_addition *add)\n \tif (err)\n \t\tgoto done;\n \n-\tif (!add->stack->config.disable_auto_compact) {\n+\tif (!add->stack->opts.disable_auto_compact) {\n \t\t/*\n \t\t * Auto-compact the stack to keep the number of tables in\n \t\t * control. It is possible that a concurrent writer is already\n@@ -758,9 +757,9 @@ int reftable_addition_add(struct reftable_addition *add,\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n-\tif (add->stack->config.default_permissions) {\n+\tif (add->stack->opts.default_permissions) {\n \t\tif (chmod(get_tempfile_path(tab_file),\n-\t\t\t add->stack->config.default_permissions)) {\n+\t\t\t add->stack->opts.default_permissions)) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n \t\t}\n@@ -768,7 +767,7 @@ int reftable_addition_add(struct reftable_addition *add,\n \ttab_fd = get_tempfile_fd(tab_file);\n \n \twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, &tab_fd,\n-\t\t\t\t &add->stack->config);\n+\t\t\t\t &add->stack->opts);\n \terr = write_table(wr, arg);\n \tif (err < 0)\n \t\tgoto done;\n@@ -855,14 +854,14 @@ static int stack_compact_locked(struct reftable_stack *st,\n \t}\n \ttab_fd = get_tempfile_fd(tab_file);\n \n-\tif (st->config.default_permissions &&\n-\t chmod(get_tempfile_path(tab_file), st->config.default_permissions) < 0) {\n+\tif (st->opts.default_permissions &&\n+\t chmod(get_tempfile_path(tab_file), st->opts.default_permissions) < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n \n \twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush,\n-\t\t\t\t &tab_fd, &st->config);\n+\t\t\t\t &tab_fd, &st->opts);\n \terr = stack_write_compact(st, wr, first, last, config);\n \tif (err < 0)\n \t\tgoto done;\n@@ -910,7 +909,7 @@ static int stack_write_compact(struct reftable_stack *st,\n \t\t\t\t st->readers[last]->max_update_index);\n \n \terr = reftable_new_merged_table(&mt, subtabs, subtabs_len,\n-\t\t\t\t\tst->config.hash_id);\n+\t\t\t\t\tst->opts.hash_id);\n \tif (err < 0) {\n \t\treftable_free(subtabs);\n \t\tgoto done;\n@@ -1100,9 +1099,9 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\tgoto done;\n \t}\n \n-\tif (st->config.default_permissions) {\n+\tif (st->opts.default_permissions) {\n \t\tif (chmod(get_lock_file_path(&tables_list_lock),\n-\t\t\t st->config.default_permissions) < 0) {\n+\t\t\t st->opts.default_permissions) < 0) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n \t\t}\n@@ -1292,7 +1291,7 @@ static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n {\n \tuint64_t *sizes =\n \t\treftable_calloc(st->merged->stack_len, sizeof(*sizes));\n-\tint version = (st->config.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2;\n+\tint version = (st->opts.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2;\n \tint overhead = header_size(version) - 1;\n \tint i = 0;\n \tfor (i = 0; i < st->merged->stack_len; i++) {\n@@ -1368,7 +1367,7 @@ static int stack_check_addition(struct reftable_stack *st,\n \tint len = 0;\n \tint i = 0;\n \n-\tif (st->config.skip_name_check)\n+\tif (st->opts.skip_name_check)\n \t\treturn 0;\n \n \terr = reftable_block_source_from_file(&src, new_tab_name);\n@@ -1500,11 +1499,11 @@ int reftable_stack_clean(struct reftable_stack *st)\n int reftable_stack_print_directory(const char *stackdir, uint32_t hash_id)\n {\n \tstruct reftable_stack *stack = NULL;\n-\tstruct reftable_write_options cfg = { .hash_id = hash_id };\n+\tstruct reftable_write_options opts = { .hash_id = hash_id };\n \tstruct reftable_merged_table *merged = NULL;\n \tstruct reftable_table table = { NULL };\n \n-\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tint err = reftable_new_stack(&stack, stackdir, opts);\n \tif (err < 0)\n \t\tgoto done;\n \ndiff --git a/reftable/stack.h b/reftable/stack.h\nindex d43efa4760..97d7ebc043 100644\n--- a/reftable/stack.h\n+++ b/reftable/stack.h\n@@ -20,7 +20,7 @@ struct reftable_stack {\n \n \tchar *reftable_dir;\n \n-\tstruct reftable_write_options config;\n+\tstruct reftable_write_options opts;\n \n \tstruct reftable_reader **readers;\n \tsize_t readers_len;\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 1df3ffce52..3316d55f19 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -150,7 +150,7 @@ static void test_reftable_stack_add_one(void)\n \tchar *dir = get_tmp_dir(__LINE__);\n \tstruct strbuf scratch = STRBUF_INIT;\n \tint mask = umask(002);\n-\tstruct reftable_write_options cfg = {\n+\tstruct reftable_write_options opts = {\n \t\t.default_permissions = 0660,\n \t};\n \tstruct reftable_stack *st = NULL;\n@@ -163,7 +163,7 @@ static void test_reftable_stack_add_one(void)\n \t};\n \tstruct reftable_ref_record dest = { NULL };\n \tstruct stat stat_result = { 0 };\n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, &write_test_ref, &ref);\n@@ -186,7 +186,7 @@ static void test_reftable_stack_add_one(void)\n \tstrbuf_addstr(&scratch, \"/tables.list\");\n \terr = stat(scratch.buf, &stat_result);\n \tEXPECT(!err);\n-\tEXPECT((stat_result.st_mode & 0777) == cfg.default_permissions);\n+\tEXPECT((stat_result.st_mode & 0777) == opts.default_permissions);\n \n \tstrbuf_reset(&scratch);\n \tstrbuf_addstr(&scratch, dir);\n@@ -195,7 +195,7 @@ static void test_reftable_stack_add_one(void)\n \tstrbuf_addstr(&scratch, st->readers[0]->name);\n \terr = stat(scratch.buf, &stat_result);\n \tEXPECT(!err);\n-\tEXPECT((stat_result.st_mode & 0777) == cfg.default_permissions);\n+\tEXPECT((stat_result.st_mode & 0777) == opts.default_permissions);\n #else\n \t(void) stat_result;\n #endif\n@@ -209,7 +209,7 @@ static void test_reftable_stack_add_one(void)\n \n static void test_reftable_stack_uptodate(void)\n {\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st1 = NULL;\n \tstruct reftable_stack *st2 = NULL;\n \tchar *dir = get_tmp_dir(__LINE__);\n@@ -232,10 +232,10 @@ static void test_reftable_stack_uptodate(void)\n \t/* simulate multi-process access to the same stack\n \t by creating two stacks for the same directory.\n \t */\n-\terr = reftable_new_stack(&st1, dir, cfg);\n+\terr = reftable_new_stack(&st1, dir, opts);\n \tEXPECT_ERR(err);\n \n-\terr = reftable_new_stack(&st2, dir, cfg);\n+\terr = reftable_new_stack(&st2, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st1, &write_test_ref, &ref1);\n@@ -257,8 +257,7 @@ static void test_reftable_stack_uptodate(void)\n static void test_reftable_stack_transaction_api(void)\n {\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n \tstruct reftable_addition *add = NULL;\n@@ -271,8 +270,7 @@ static void test_reftable_stack_transaction_api(void)\n \t};\n \tstruct reftable_ref_record dest = { NULL };\n \n-\n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \treftable_addition_destroy(add);\n@@ -301,12 +299,12 @@ static void test_reftable_stack_transaction_api(void)\n static void test_reftable_stack_transaction_api_performs_auto_compaction(void)\n {\n \tchar *dir = get_tmp_dir(__LINE__);\n-\tstruct reftable_write_options cfg = {0};\n+\tstruct reftable_write_options opts = {0};\n \tstruct reftable_addition *add = NULL;\n \tstruct reftable_stack *st = NULL;\n \tint i, n = 20, err;\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i <= n; i++) {\n@@ -325,7 +323,7 @@ static void test_reftable_stack_transaction_api_performs_auto_compaction(void)\n \t\t * we can ensure that we indeed honor this setting and have\n \t\t * better control over when exactly auto compaction runs.\n \t\t */\n-\t\tst->config.disable_auto_compact = i != n;\n+\t\tst->opts.disable_auto_compact = i != n;\n \n \t\terr = reftable_stack_new_addition(&add, st);\n \t\tEXPECT_ERR(err);\n@@ -361,13 +359,13 @@ static void test_reftable_stack_auto_compaction_fails_gracefully(void)\n \t\t.value_type = REFTABLE_REF_VAL1,\n \t\t.value.val1 = {0x01},\n \t};\n-\tstruct reftable_write_options cfg = {0};\n+\tstruct reftable_write_options opts = {0};\n \tstruct reftable_stack *st;\n \tstruct strbuf table_path = STRBUF_INIT;\n \tchar *dir = get_tmp_dir(__LINE__);\n \tint err;\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, write_test_ref, &ref);\n@@ -442,8 +440,7 @@ static int write_error(struct reftable_writer *wr, void *arg)\n static void test_reftable_stack_update_index_check(void)\n {\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n \tstruct reftable_ref_record ref1 = {\n@@ -459,7 +456,7 @@ static void test_reftable_stack_update_index_check(void)\n \t\t.value.symref = \"master\",\n \t};\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, &write_test_ref, &ref1);\n@@ -474,12 +471,11 @@ static void test_reftable_stack_update_index_check(void)\n static void test_reftable_stack_lock_failure(void)\n {\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err, i;\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \tfor (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) {\n \t\terr = reftable_stack_add(st, &write_error, &i);\n@@ -494,7 +490,7 @@ static void test_reftable_stack_add(void)\n {\n \tint i = 0;\n \tint err = 0;\n-\tstruct reftable_write_options cfg = {\n+\tstruct reftable_write_options opts = {\n \t\t.exact_log_message = 1,\n \t\t.default_permissions = 0660,\n \t\t.disable_auto_compact = 1,\n@@ -507,7 +503,7 @@ static void test_reftable_stack_add(void)\n \tstruct stat stat_result;\n \tint N = ARRAY_SIZE(refs);\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i < N; i++) {\n@@ -566,7 +562,7 @@ static void test_reftable_stack_add(void)\n \tstrbuf_addstr(&path, \"/tables.list\");\n \terr = stat(path.buf, &stat_result);\n \tEXPECT(!err);\n-\tEXPECT((stat_result.st_mode & 0777) == cfg.default_permissions);\n+\tEXPECT((stat_result.st_mode & 0777) == opts.default_permissions);\n \n \tstrbuf_reset(&path);\n \tstrbuf_addstr(&path, dir);\n@@ -575,7 +571,7 @@ static void test_reftable_stack_add(void)\n \tstrbuf_addstr(&path, st->readers[0]->name);\n \terr = stat(path.buf, &stat_result);\n \tEXPECT(!err);\n-\tEXPECT((stat_result.st_mode & 0777) == cfg.default_permissions);\n+\tEXPECT((stat_result.st_mode & 0777) == opts.default_permissions);\n #else\n \t(void) stat_result;\n #endif\n@@ -593,7 +589,7 @@ static void test_reftable_stack_add(void)\n static void test_reftable_stack_log_normalize(void)\n {\n \tint err = 0;\n-\tstruct reftable_write_options cfg = {\n+\tstruct reftable_write_options opts = {\n \t\t0,\n \t};\n \tstruct reftable_stack *st = NULL;\n@@ -617,7 +613,7 @@ static void test_reftable_stack_log_normalize(void)\n \t\t.update_index = 1,\n \t};\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \tinput.value.update.message = \"one\\ntwo\";\n@@ -650,8 +646,7 @@ static void test_reftable_stack_tombstone(void)\n {\n \tint i = 0;\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n \tstruct reftable_ref_record refs[2] = { { NULL } };\n@@ -660,8 +655,7 @@ static void test_reftable_stack_tombstone(void)\n \tstruct reftable_ref_record dest = { NULL };\n \tstruct reftable_log_record log_dest = { NULL };\n \n-\n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \t/* even entries add the refs, odd entries delete them. */\n@@ -729,8 +723,7 @@ static void test_reftable_stack_tombstone(void)\n static void test_reftable_stack_hash_id(void)\n {\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n \n@@ -740,24 +733,24 @@ static void test_reftable_stack_hash_id(void)\n \t\t.value.symref = \"target\",\n \t\t.update_index = 1,\n \t};\n-\tstruct reftable_write_options cfg32 = { .hash_id = GIT_SHA256_FORMAT_ID };\n+\tstruct reftable_write_options opts32 = { .hash_id = GIT_SHA256_FORMAT_ID };\n \tstruct reftable_stack *st32 = NULL;\n-\tstruct reftable_write_options cfg_default = { 0 };\n+\tstruct reftable_write_options opts_default = { 0 };\n \tstruct reftable_stack *st_default = NULL;\n \tstruct reftable_ref_record dest = { NULL };\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, &write_test_ref, &ref);\n \tEXPECT_ERR(err);\n \n \t/* can't read it with the wrong hash ID. */\n-\terr = reftable_new_stack(&st32, dir, cfg32);\n+\terr = reftable_new_stack(&st32, dir, opts32);\n \tEXPECT(err == REFTABLE_FORMAT_ERROR);\n \n-\t/* check that we can read it back with default config too. */\n-\terr = reftable_new_stack(&st_default, dir, cfg_default);\n+\t/* check that we can read it back with default opts too. */\n+\terr = reftable_new_stack(&st_default, dir, opts_default);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n@@ -790,8 +783,7 @@ static void test_suggest_compaction_segment_nothing(void)\n static void test_reflog_expire(void)\n {\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tstruct reftable_log_record logs[20] = { { NULL } };\n \tint N = ARRAY_SIZE(logs) - 1;\n@@ -802,8 +794,7 @@ static void test_reflog_expire(void)\n \t};\n \tstruct reftable_log_record log = { NULL };\n \n-\n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 1; i <= N; i++) {\n@@ -866,21 +857,19 @@ static int write_nothing(struct reftable_writer *wr, void *arg)\n \n static void test_empty_add(void)\n {\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n \tstruct reftable_stack *st2 = NULL;\n \n-\n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, &write_nothing, NULL);\n \tEXPECT_ERR(err);\n \n-\terr = reftable_new_stack(&st2, dir, cfg);\n+\terr = reftable_new_stack(&st2, dir, opts);\n \tEXPECT_ERR(err);\n \tclear_dir(dir);\n \treftable_stack_destroy(st);\n@@ -899,16 +888,15 @@ static int fastlog2(uint64_t sz)\n \n static void test_reftable_stack_auto_compaction(void)\n {\n-\tstruct reftable_write_options cfg = {\n+\tstruct reftable_write_options opts = {\n \t\t.disable_auto_compact = 1,\n \t};\n \tstruct reftable_stack *st = NULL;\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n \tint err, i;\n \tint N = 100;\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i < N; i++) {\n@@ -938,13 +926,13 @@ static void test_reftable_stack_auto_compaction(void)\n \n static void test_reftable_stack_add_performs_auto_compaction(void)\n {\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tstruct strbuf refname = STRBUF_INIT;\n \tchar *dir = get_tmp_dir(__LINE__);\n \tint err, i, n = 20;\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i <= n; i++) {\n@@ -959,7 +947,7 @@ static void test_reftable_stack_add_performs_auto_compaction(void)\n \t\t * we can ensure that we indeed honor this setting and have\n \t\t * better control over when exactly auto compaction runs.\n \t\t */\n-\t\tst->config.disable_auto_compact = i != n;\n+\t\tst->opts.disable_auto_compact = i != n;\n \n \t\tstrbuf_reset(&refname);\n \t\tstrbuf_addf(&refname, \"branch-%04d\", i);\n@@ -986,14 +974,13 @@ static void test_reftable_stack_add_performs_auto_compaction(void)\n \n static void test_reftable_stack_compaction_concurrent(void)\n {\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st1 = NULL, *st2 = NULL;\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n \tint err, i;\n \tint N = 3;\n \n-\terr = reftable_new_stack(&st1, dir, cfg);\n+\terr = reftable_new_stack(&st1, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i < N; i++) {\n@@ -1010,7 +997,7 @@ static void test_reftable_stack_compaction_concurrent(void)\n \t\tEXPECT_ERR(err);\n \t}\n \n-\terr = reftable_new_stack(&st2, dir, cfg);\n+\terr = reftable_new_stack(&st2, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_compact_all(st1, NULL);\n@@ -1036,14 +1023,13 @@ static void unclean_stack_close(struct reftable_stack *st)\n \n static void test_reftable_stack_compaction_concurrent_clean(void)\n {\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st1 = NULL, *st2 = NULL, *st3 = NULL;\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n \tint err, i;\n \tint N = 3;\n \n-\terr = reftable_new_stack(&st1, dir, cfg);\n+\terr = reftable_new_stack(&st1, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i < N; i++) {\n@@ -1060,7 +1046,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void)\n \t\tEXPECT_ERR(err);\n \t}\n \n-\terr = reftable_new_stack(&st2, dir, cfg);\n+\terr = reftable_new_stack(&st2, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_compact_all(st1, NULL);\n@@ -1069,7 +1055,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void)\n \tunclean_stack_close(st1);\n \tunclean_stack_close(st2);\n \n-\terr = reftable_new_stack(&st3, dir, cfg);\n+\terr = reftable_new_stack(&st3, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_clean(st3);\n-- \n2.45.0\n\n"},{"id":"493879","messageId":"d8a0764e8783c26cba247cdb8b497636433d199b.1714630191.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1714630191.git.ps@pks.im","subject":"[PATCH 02/11] reftable: consistently pass write opts as value","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-02T06:51:37Z","receivedAt":"2024-05-02T06:51:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We sometimes pass the refatble write options as value and sometimes as a\npointer. This is quite confusing and makes the reader wonder whether the\noptions get modified sometimes.\n\nIn fact, `reftable_new_writer()` does cause the caller-provided options\nto get updated when some values aren't set up. This is quite unexpected,\nbut didn't cause any harm until now.\n\nRefactor the code to consistently pass the options as a value so that\nthey are local to the subsystem they are being passed into so that we\ncan avoid weirdness like this.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged_test.c | 6 +++---\n reftable/readwrite_test.c | 26 +++++++++++++-------------\n reftable/refname_test.c | 2 +-\n reftable/reftable-writer.h | 2 +-\n reftable/stack.c | 4 ++--\n reftable/writer.c | 12 +++++++-----\n 6 files changed, 27 insertions(+), 25 deletions(-)\n\ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nindex 530fc82d1c..4ac81de8d4 100644\n--- a/reftable/merged_test.c\n+++ b/reftable/merged_test.c\n@@ -42,7 +42,7 @@ static void write_test_table(struct strbuf *buf,\n \t\t}\n \t}\n \n-\tw = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, &opts);\n+\tw = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, opts);\n \treftable_writer_set_limits(w, min, max);\n \n \tfor (i = 0; i < n; i++) {\n@@ -70,7 +70,7 @@ static void write_test_log_table(struct strbuf *buf,\n \t\t.exact_log_message = 1,\n \t};\n \tstruct reftable_writer *w = NULL;\n-\tw = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, &opts);\n+\tw = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, opts);\n \treftable_writer_set_limits(w, update_index, update_index);\n \n \tfor (i = 0; i < n; i++) {\n@@ -403,7 +403,7 @@ static void test_default_write_opts(void)\n \tstruct reftable_write_options opts = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \n \tstruct reftable_ref_record rec = {\n \t\t.refname = \"master\",\ndiff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c\nindex a6dbd214c5..27631a041b 100644\n--- a/reftable/readwrite_test.c\n+++ b/reftable/readwrite_test.c\n@@ -51,7 +51,7 @@ static void write_table(char ***names, struct strbuf *buf, int N,\n \t\t.hash_id = hash_id,\n \t};\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, buf, opts);\n \tstruct reftable_ref_record ref = { NULL };\n \tint i = 0, n;\n \tstruct reftable_log_record log = { NULL };\n@@ -129,7 +129,7 @@ static void test_log_buffer_size(void)\n \t\t\t\t\t .message = \"commit: 9\\n\",\n \t\t\t\t } } };\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \n \t/* This tests buffer extension for log compression. Must use a random\n \t hash, to ensure that the compressed part is larger than the original.\n@@ -172,7 +172,7 @@ static void test_log_overflow(void)\n \t\t},\n \t};\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \n \tmemset(msg, 'x', sizeof(msg) - 1);\n \treftable_writer_set_limits(w, update_index, update_index);\n@@ -199,7 +199,7 @@ static void test_log_write_read(void)\n \tstruct reftable_block_source source = { NULL };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \tconst struct reftable_stats *stats = NULL;\n \treftable_writer_set_limits(w, 0, N);\n \tfor (i = 0; i < N; i++) {\n@@ -288,7 +288,7 @@ static void test_log_zlib_corruption(void)\n \tstruct reftable_block_source source = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \tconst struct reftable_stats *stats = NULL;\n \tchar message[100] = { 0 };\n \tint err, i, n;\n@@ -526,7 +526,7 @@ static void test_table_refs_for(int indexed)\n \n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \n \tstruct reftable_iterator it = { NULL };\n \tint j;\n@@ -619,7 +619,7 @@ static void test_write_empty_table(void)\n \tstruct reftable_write_options opts = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \tstruct reftable_block_source source = { NULL };\n \tstruct reftable_reader *rd = NULL;\n \tstruct reftable_ref_record rec = { NULL };\n@@ -657,7 +657,7 @@ static void test_write_object_id_min_length(void)\n \t};\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \tstruct reftable_ref_record ref = {\n \t\t.update_index = 1,\n \t\t.value_type = REFTABLE_REF_VAL1,\n@@ -692,7 +692,7 @@ static void test_write_object_id_length(void)\n \t};\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \tstruct reftable_ref_record ref = {\n \t\t.update_index = 1,\n \t\t.value_type = REFTABLE_REF_VAL1,\n@@ -726,7 +726,7 @@ static void test_write_empty_key(void)\n \tstruct reftable_write_options opts = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \tstruct reftable_ref_record ref = {\n \t\t.refname = \"\",\n \t\t.update_index = 1,\n@@ -749,7 +749,7 @@ static void test_write_key_order(void)\n \tstruct reftable_write_options opts = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \tstruct reftable_ref_record refs[2] = {\n \t\t{\n \t\t\t.refname = \"b\",\n@@ -792,7 +792,7 @@ static void test_write_multiple_indices(void)\n \tstruct reftable_reader *reader;\n \tint err, i;\n \n-\twriter = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, &opts);\n+\twriter = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, opts);\n \treftable_writer_set_limits(writer, 1, 1);\n \tfor (i = 0; i < 100; i++) {\n \t\tstruct reftable_ref_record ref = {\n@@ -869,7 +869,7 @@ static void test_write_multi_level_index(void)\n \tstruct reftable_reader *reader;\n \tint err;\n \n-\twriter = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, &opts);\n+\twriter = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, opts);\n \treftable_writer_set_limits(writer, 1, 1);\n \tfor (size_t i = 0; i < 200; i++) {\n \t\tstruct reftable_ref_record ref = {\ndiff --git a/reftable/refname_test.c b/reftable/refname_test.c\nindex b9cc62554e..3468253be7 100644\n--- a/reftable/refname_test.c\n+++ b/reftable/refname_test.c\n@@ -30,7 +30,7 @@ static void test_conflict(void)\n \tstruct reftable_write_options opts = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \tstruct reftable_ref_record rec = {\n \t\t.refname = \"a/b\",\n \t\t.value_type = REFTABLE_REF_SYMREF,\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 155bf0bbe2..44cb986465 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -92,7 +92,7 @@ struct reftable_stats {\n struct reftable_writer *\n reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\t int (*flush_func)(void *),\n-\t\t void *writer_arg, struct reftable_write_options *opts);\n+\t\t void *writer_arg, struct reftable_write_options opts);\n \n /* Set the range of update indices for the records we will add. When writing a\n table into a stack, the min should be at least\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 3979657793..7b4fff7c9e 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -767,7 +767,7 @@ int reftable_addition_add(struct reftable_addition *add,\n \ttab_fd = get_tempfile_fd(tab_file);\n \n \twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, &tab_fd,\n-\t\t\t\t &add->stack->opts);\n+\t\t\t\t add->stack->opts);\n \terr = write_table(wr, arg);\n \tif (err < 0)\n \t\tgoto done;\n@@ -861,7 +861,7 @@ static int stack_compact_locked(struct reftable_stack *st,\n \t}\n \n \twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush,\n-\t\t\t\t &tab_fd, &st->opts);\n+\t\t\t\t &tab_fd, st->opts);\n \terr = stack_write_compact(st, wr, first, last, config);\n \tif (err < 0)\n \t\tgoto done;\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 1d9ff0fbfa..ad2f2e6c65 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -122,20 +122,22 @@ static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n struct reftable_writer *\n reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\t int (*flush_func)(void *),\n-\t\t void *writer_arg, struct reftable_write_options *opts)\n+\t\t void *writer_arg, struct reftable_write_options opts)\n {\n \tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n \tstrbuf_init(&wp->block_writer_data.last_key, 0);\n-\toptions_set_defaults(opts);\n-\tif (opts->block_size >= (1 << 24)) {\n+\n+\toptions_set_defaults(&opts);\n+\tif (opts.block_size >= (1 << 24)) {\n \t\t/* TODO - error return? */\n \t\tabort();\n \t}\n+\n \twp->last_key = reftable_empty_strbuf;\n-\tREFTABLE_CALLOC_ARRAY(wp->block, opts->block_size);\n+\tREFTABLE_CALLOC_ARRAY(wp->block, opts.block_size);\n \twp->write = writer_func;\n \twp->write_arg = writer_arg;\n-\twp->opts = *opts;\n+\twp->opts = opts;\n \twp->flush = flush_func;\n \twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n \n-- \n2.45.0\n\n"},{"id":"493880","messageId":"c040f81fba66cbbf783e7a9188fde116fe413702.1714630191.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1714630191.git.ps@pks.im","subject":"[PATCH 03/11] reftable/writer: drop static variable used to initialize strbuf","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-02T06:51:42Z","receivedAt":"2024-05-02T06:51:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have a static variable in the reftable writer code that is merely\nused to initialize the `last_key` of the writer. Convert the code to\ninstead use `strbuf_init()` and drop the variable.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/writer.c | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex ad2f2e6c65..7df6e53699 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -117,15 +117,12 @@ static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n \tw->block_writer->restart_interval = w->opts.restart_interval;\n }\n \n-static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n-\n struct reftable_writer *\n reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\t int (*flush_func)(void *),\n \t\t void *writer_arg, struct reftable_write_options opts)\n {\n \tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n-\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n \n \toptions_set_defaults(&opts);\n \tif (opts.block_size >= (1 << 24)) {\n@@ -133,7 +130,8 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\tabort();\n \t}\n \n-\twp->last_key = reftable_empty_strbuf;\n+\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n+\tstrbuf_init(&wp->last_key, 0);\n \tREFTABLE_CALLOC_ARRAY(wp->block, opts.block_size);\n \twp->write = writer_func;\n \twp->write_arg = writer_arg;\n-- \n2.45.0\n\n"},{"id":"493881","messageId":"ef79bb1b7be05303278232a9e4dc1ab100a53fab.1714630191.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1714630191.git.ps@pks.im","subject":"[PATCH 04/11] reftable/writer: improve error when passed an invalid block size","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-02T06:51:47Z","receivedAt":"2024-05-02T06:51:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable format only supports block sizes up to 16MB. When the\nwriter is being passed a value bigger than that it simply calls\nabort(3P), which isn't all that helpful due to the lack of a proper\nerror message.\n\nImprove this by calling `BUG()` instead.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/writer.c | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 7df6e53699..374b7d15ed 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -125,10 +125,8 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n \n \toptions_set_defaults(&opts);\n-\tif (opts.block_size >= (1 << 24)) {\n-\t\t/* TODO - error return? */\n-\t\tabort();\n-\t}\n+\tif (opts.block_size >= (1 << 24))\n+\t\tBUG(\"configured block size exceeds 16MB\");\n \n \tstrbuf_init(&wp->block_writer_data.last_key, 0);\n \tstrbuf_init(&wp->last_key, 0);\n-- \n2.45.0\n\n"},{"id":"493882","messageId":"4d4407d4a468a81f7775595c1ea4c611c1ca2482.1714630191.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1714630191.git.ps@pks.im","subject":"[PATCH 05/11] reftable/dump: support dumping a table's block structure","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-02T06:51:52Z","receivedAt":"2024-05-02T06:51:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're about to introduce new configs that will allow users to have more\ncontrol over how exactly reftables are written. To verify that these\nconfigs are effective we will need to take a peak into the actual blocks\nwritten by the reftable backend.\n\nIntroduce a new mode to the dumping logic that prints out the block\nstructure. This logic can be invoked via `test-tool dump-reftables -b`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/dump.c | 8 ++-\n reftable/reader.c | 63 ++++++++++++++++++\n reftable/reftable-reader.h | 2 +\n t/t0613-reftable-write-options.sh | 102 ++++++++++++++++++++++++++++++\n 4 files changed, 174 insertions(+), 1 deletion(-)\n create mode 100755 t/t0613-reftable-write-options.sh\n\ndiff --git a/reftable/dump.c b/reftable/dump.c\nindex 9c770a10cc..24476cc2a9 100644\n--- a/reftable/dump.c\n+++ b/reftable/dump.c\n@@ -48,6 +48,7 @@ static void print_help(void)\n \tprintf(\"usage: dump [-cst] arg\\n\\n\"\n \t \"options: \\n\"\n \t \" -c compact\\n\"\n+\t \" -b dump blocks\\n\"\n \t \" -t dump table\\n\"\n \t \" -s dump stack\\n\"\n \t \" -6 sha256 hash format\\n\"\n@@ -58,6 +59,7 @@ static void print_help(void)\n int reftable_dump_main(int argc, char *const *argv)\n {\n \tint err = 0;\n+\tint opt_dump_blocks = 0;\n \tint opt_dump_table = 0;\n \tint opt_dump_stack = 0;\n \tint opt_compact = 0;\n@@ -67,6 +69,8 @@ int reftable_dump_main(int argc, char *const *argv)\n \tfor (; argc > 1; argv++, argc--)\n \t\tif (*argv[1] != '-')\n \t\t\tbreak;\n+\t\telse if (!strcmp(\"-b\", argv[1]))\n+\t\t\topt_dump_blocks = 1;\n \t\telse if (!strcmp(\"-t\", argv[1]))\n \t\t\topt_dump_table = 1;\n \t\telse if (!strcmp(\"-6\", argv[1]))\n@@ -88,7 +92,9 @@ int reftable_dump_main(int argc, char *const *argv)\n \n \targ = argv[1];\n \n-\tif (opt_dump_table) {\n+\tif (opt_dump_blocks) {\n+\t\terr = reftable_reader_print_blocks(arg);\n+\t} else if (opt_dump_table) {\n \t\terr = reftable_reader_print_file(arg);\n \t} else if (opt_dump_stack) {\n \t\terr = reftable_stack_print_directory(arg, opt_hash_id);\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 481dff10d4..f23c8523db 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -856,3 +856,66 @@ int reftable_reader_print_file(const char *tablename)\n \treftable_reader_free(r);\n \treturn err;\n }\n+\n+int reftable_reader_print_blocks(const char *tablename)\n+{\n+\tstruct {\n+\t\tconst char *name;\n+\t\tint type;\n+\t} sections[] = {\n+\t\t{\n+\t\t\t.name = \"ref\",\n+\t\t\t.type = BLOCK_TYPE_REF,\n+\t\t},\n+\t\t{\n+\t\t\t.name = \"obj\",\n+\t\t\t.type = BLOCK_TYPE_OBJ,\n+\t\t},\n+\t\t{\n+\t\t\t.name = \"log\",\n+\t\t\t.type = BLOCK_TYPE_LOG,\n+\t\t},\n+\t};\n+\tstruct reftable_block_source src = { 0 };\n+\tstruct table_iter ti = TABLE_ITER_INIT;\n+\tstruct reftable_reader *r = NULL;\n+\tsize_t i;\n+\tint err;\n+\n+\terr = reftable_block_source_from_file(&src, tablename);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&r, &src, tablename);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tprintf(\"header:\\n\");\n+\tprintf(\" block_size: %d\\n\", r->block_size);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(sections); i++) {\n+\t\terr = reader_start(r, &ti, sections[i].type, 0);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t\tif (err > 0)\n+\t\t\tcontinue;\n+\n+\t\tprintf(\"%s:\\n\", sections[i].name);\n+\n+\t\twhile (1) {\n+\t\t\tprintf(\" - length: %u\\n\", ti.br.block_len);\n+\t\t\tprintf(\" restarts: %u\\n\", ti.br.restart_count);\n+\n+\t\t\terr = table_iter_next_block(&ti);\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t\tif (err > 0)\n+\t\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+done:\n+\treftable_reader_free(r);\n+\ttable_iter_close(&ti);\n+\treturn err;\n+}\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nindex 4a4bc2fdf8..4a04857773 100644\n--- a/reftable/reftable-reader.h\n+++ b/reftable/reftable-reader.h\n@@ -97,5 +97,7 @@ void reftable_table_from_reader(struct reftable_table *tab,\n \n /* print table onto stdout for debugging. */\n int reftable_reader_print_file(const char *tablename);\n+/* print blocks onto stdout for debugging. */\n+int reftable_reader_print_blocks(const char *tablename);\n \n #endif\ndiff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh\nnew file mode 100755\nindex 0000000000..462980c37c\n--- /dev/null\n+++ b/t/t0613-reftable-write-options.sh\n@@ -0,0 +1,102 @@\n+#!/bin/sh\n+\n+test_description='reftable write options'\n+\n+GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+# Disable auto-compaction for all tests as we explicitly control repacking of\n+# refs.\n+GIT_TEST_REFTABLE_AUTOCOMPACTION=false\n+export GIT_TEST_REFTABLE_AUTOCOMPACTION\n+# Block sizes depend on the hash function, so we force SHA1 here.\n+GIT_TEST_DEFAULT_HASH=sha1\n+export GIT_TEST_DEFAULT_HASH\n+# Block sizes also depend on the actual refs we write, so we force \"master\" to\n+# be the default initial branch name.\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'default write options' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tgit pack-refs &&\n+\t\tcat >expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 4096\n+\t\tref:\n+\t\t - length: 129\n+\t\t restarts: 2\n+\t\tlog:\n+\t\t - length: 262\n+\t\t restarts: 2\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'disabled reflog writes no log blocks' '\n+\ttest_config_global core.logAllRefUpdates false &&\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tgit pack-refs &&\n+\t\tcat >expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 4096\n+\t\tref:\n+\t\t - length: 129\n+\t\t restarts: 2\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'many refs results in multiple blocks' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tfor i in $(test_seq 200)\n+\t\tdo\n+\t\t\tprintf \"update refs/heads/branch-%d HEAD\\n\" \"$i\" ||\n+\t\t\treturn 1\n+\t\tdone >input &&\n+\t\tgit update-ref --stdin expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 4096\n+\t\tref:\n+\t\t - length: 4049\n+\t\t restarts: 11\n+\t\t - length: 1136\n+\t\t restarts: 3\n+\t\tlog:\n+\t\t - length: 4041\n+\t\t restarts: 4\n+\t\t - length: 4015\n+\t\t restarts: 3\n+\t\t - length: 4014\n+\t\t restarts: 3\n+\t\t - length: 4012\n+\t\t restarts: 3\n+\t\t - length: 3289\n+\t\t restarts: 3\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_done\n-- \n2.45.0\n\n"},{"id":"493883","messageId":"b4e4db5735beb106511980adca48dc416c4b0d95.1714630191.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1714630191.git.ps@pks.im","subject":"[PATCH 06/11] refs/reftable: allow configuring block size","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-02T06:51:56Z","receivedAt":"2024-05-02T06:52:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Add a new option `reftable.blockSize` that allows the user to control\nthe block size used by the reftable library.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config.txt | 2 +\n Documentation/config/reftable.txt | 14 ++++++\n refs/reftable-backend.c | 32 +++++++++++++-\n t/t0613-reftable-write-options.sh | 72 +++++++++++++++++++++++++++++++\n 4 files changed, 119 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/config/reftable.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 70b448b132..fa1469e5e7 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -497,6 +497,8 @@ include::config/rebase.txt[]\n \n include::config/receive.txt[]\n \n+include::config/reftable.txt[]\n+\n include::config/remote.txt[]\n \n include::config/remotes.txt[]\ndiff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt\nnew file mode 100644\nindex 0000000000..fa7c4be014\n--- /dev/null\n+++ b/Documentation/config/reftable.txt\n@@ -0,0 +1,14 @@\n+reftable.blockSize::\n+\tThe size in bytes used by the reftable backend when writing blocks.\n+\tThe block size is determined by the writer, and does not have to be a\n+\tpower of 2. The block size must be larger than the longest reference\n+\tname or log entry used in the repository, as references cannot span\n+\tblocks.\n++\n+Powers of two that are friendly to the virtual memory system or\n+filesystem (such as 4kB or 8kB) are recommended. Larger sizes (64kB) can\n+yield better compression, with a possible increased cost incurred by\n+readers during access.\n++\n+The largest block size is `16777215` bytes (15.99 MiB). The default value is\n+`4096` bytes (4kB). A value of `0` will use the default value.\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 1cda48c504..c2c47a3bc1 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1,6 +1,7 @@\n #include \"../git-compat-util.h\"\n #include \"../abspath.h\"\n #include \"../chdir-notify.h\"\n+#include \"../config.h\"\n #include \"../environment.h\"\n #include \"../gettext.h\"\n #include \"../hash.h\"\n@@ -230,6 +231,23 @@ static int read_ref_without_reload(struct reftable_stack *stack,\n \treturn ret;\n }\n \n+static int reftable_be_config(const char *var, const char *value,\n+\t\t\t const struct config_context *ctx,\n+\t\t\t void *_opts)\n+{\n+\tstruct reftable_write_options *opts = _opts;\n+\n+\tif (!strcmp(var, \"reftable.blocksize\")) {\n+\t\tunsigned long block_size = git_config_ulong(var, value, ctx->kvi);\n+\t\tif (block_size > 16777215)\n+\t\t\tdie(\"reftable block size cannot exceed 16MB\");\n+\t\topts->block_size = block_size;\n+\t\treturn 0;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static struct ref_store *reftable_be_init(struct repository *repo,\n \t\t\t\t\t const char *gitdir,\n \t\t\t\t\t unsigned int store_flags)\n@@ -245,12 +263,24 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \tbase_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);\n \tstrmap_init(&refs->worktree_stacks);\n \trefs->store_flags = store_flags;\n-\trefs->write_options.block_size = 4096;\n+\n \trefs->write_options.hash_id = repo->hash_algo->format_id;\n \trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n \trefs->write_options.disable_auto_compact =\n \t\t!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1);\n \n+\tgit_config(reftable_be_config, &refs->write_options);\n+\n+\t/*\n+\t * It is somewhat unfortunate that we have to mirror the default block\n+\t * size of the reftable library here. But given that the write options\n+\t * wouldn't be updated by the library here, and given that we require\n+\t * the proper block size to trim reflog message so that they fit, we\n+\t * must set up a proper value here.\n+\t */\n+\tif (!refs->write_options.block_size)\n+\t\trefs->write_options.block_size = 4096;\n+\n \t/*\n \t * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.\n \t * This stack contains both the shared and the main worktree refs.\ndiff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh\nindex 462980c37c..8bdbc6ec70 100755\n--- a/t/t0613-reftable-write-options.sh\n+++ b/t/t0613-reftable-write-options.sh\n@@ -99,4 +99,76 @@ test_expect_success 'many refs results in multiple blocks' '\n \t)\n '\n \n+test_expect_success 'tiny block size leads to error' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: unable to compact stack: entry too large\n+\t\tEOF\n+\t\ttest_must_fail git -c reftable.blockSize=50 pack-refs 2>err &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'small block size leads to multiple ref blocks' '\n+\ttest_config_global core.logAllRefUpdates false &&\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\t\tgit -c reftable.blockSize=100 pack-refs &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 100\n+\t\tref:\n+\t\t - length: 53\n+\t\t restarts: 1\n+\t\t - length: 74\n+\t\t restarts: 1\n+\t\t - length: 38\n+\t\t restarts: 1\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'small block size fails with large reflog message' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit A &&\n+\t\tperl -e \"print \\\"a\\\" x 500\" >logmsg &&\n+\t\tcat >expect <<-EOF &&\n+\t\tfatal: update_ref failed for ref ${SQ}refs/heads/logme${SQ}: reftable: transaction failure: entry too large\n+\t\tEOF\n+\t\ttest_must_fail git -c reftable.blockSize=100 \\\n+\t\t\tupdate-ref -m \"$(cat logmsg)\" refs/heads/logme HEAD 2>err &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'block size exceeding maximum supported size' '\n+\ttest_config_global core.logAllRefUpdates false &&\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\t\tcat >expect <<-EOF &&\n+\t\tfatal: reftable block size cannot exceed 16MB\n+\t\tEOF\n+\t\ttest_must_fail git -c reftable.blockSize=16777216 pack-refs 2>err &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n-- \n2.45.0\n\n"},{"id":"493884","messageId":"79d9e07ca92baace4b729b0b9b9227735a84286b.1714630191.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1714630191.git.ps@pks.im","subject":"[PATCH 07/11] reftable: use `uint16_t` to track restart interval","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-02T06:52:01Z","receivedAt":"2024-05-02T06:52:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The restart interval can at most be `UINT16_MAX` as specified in the\ntechnical documentation of the reftable format. Furthermore, it cannot\never be negative. Regardless of that we use an `int` to track the\nrestart interval.\n\nChange the type to use an `uint16_t` instead.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/block.h | 2 +-\n reftable/reftable-writer.h | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/reftable/block.h b/reftable/block.h\nindex ea4384a7e2..cd5577105d 100644\n--- a/reftable/block.h\n+++ b/reftable/block.h\n@@ -25,7 +25,7 @@ struct block_writer {\n \tuint32_t header_off;\n \n \t/* How often to restart keys. */\n-\tint restart_interval;\n+\tuint16_t restart_interval;\n \tint hash_size;\n \n \t/* Offset of next uint8_t to write. */\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 44cb986465..4cd8ebe6c7 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -28,7 +28,7 @@ struct reftable_write_options {\n \tunsigned skip_index_objects : 1;\n \n \t/* how often to write complete keys in each block. */\n-\tint restart_interval;\n+\tuint16_t restart_interval;\n \n \t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n \t * Defaults to SHA1 if unset\n-- \n2.45.0\n\n"},{"id":"493885","messageId":"653ec4dfa59b20e0ca9ef76bb5f2c1725c2bc078.1714630191.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1714630191.git.ps@pks.im","subject":"[PATCH 08/11] refs/reftable: allow configuring restart interval","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-02T06:52:06Z","receivedAt":"2024-05-02T06:52:10Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Add a new option `reftable.restartInterval` that allows the user to\ncontrol the restart interval when writing reftable records used by the\nreftable library.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config/reftable.txt | 19 ++++++++++++++\n refs/reftable-backend.c | 6 +++++\n t/t0613-reftable-write-options.sh | 43 +++++++++++++++++++++++++++++++\n 3 files changed, 68 insertions(+)\n\ndiff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt\nindex fa7c4be014..16b915c75e 100644\n--- a/Documentation/config/reftable.txt\n+++ b/Documentation/config/reftable.txt\n@@ -12,3 +12,22 @@ readers during access.\n +\n The largest block size is `16777215` bytes (15.99 MiB). The default value is\n `4096` bytes (4kB). A value of `0` will use the default value.\n+\n+reftable.restartInterval::\n+\tThe interval at which to create restart points. The reftable backend\n+\tdetermines the restart points at file creation. The process is\n+\tarbitrary, but every 16 or 64 records is recommended. Every 16 may be\n+\tmore suitable for smaller block sizes (4k or 8k), every 64 for larger\n+\tblock sizes (64k).\n++\n+More frequent restart points reduces prefix compression and increases\n+space consumed by the restart table, both of which increase file size.\n++\n+Less frequent restart points makes prefix compression more effective,\n+decreasing overall file size, with increased penalties for readers\n+walking through more records after the binary search step.\n++\n+A maximum of `65535` restart points per block is supported.\n++\n+The default value is to create restart points every 16 records. A value of `0`\n+will use the default value.\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex c2c47a3bc1..a786143de2 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -243,6 +243,12 @@ static int reftable_be_config(const char *var, const char *value,\n \t\t\tdie(\"reftable block size cannot exceed 16MB\");\n \t\topts->block_size = block_size;\n \t\treturn 0;\n+\t} else if (!strcmp(var, \"reftable.restartinterval\")) {\n+\t\tunsigned long restart_interval = git_config_ulong(var, value, ctx->kvi);\n+\t\tif (restart_interval > UINT16_MAX)\n+\t\t\tdie(\"reftable block size cannot exceed %u\", (unsigned)UINT16_MAX);\n+\t\topts->restart_interval = restart_interval;\n+\t\treturn 0;\n \t}\n \n \treturn 0;\ndiff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh\nindex 8bdbc6ec70..e0a5b26f58 100755\n--- a/t/t0613-reftable-write-options.sh\n+++ b/t/t0613-reftable-write-options.sh\n@@ -171,4 +171,47 @@ test_expect_success 'block size exceeding maximum supported size' '\n \t)\n '\n \n+test_expect_success 'restart interval at every single record' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tfor i in $(test_seq 10)\n+\t\tdo\n+\t\t\tprintf \"update refs/heads/branch-%d HEAD\\n\" \"$i\" ||\n+\t\t\treturn 1\n+\t\tdone >input &&\n+\t\tgit update-ref --stdin expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 4096\n+\t\tref:\n+\t\t - length: 566\n+\t\t restarts: 13\n+\t\tlog:\n+\t\t - length: 1393\n+\t\t restarts: 12\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'restart interval exceeding maximum supported interval' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tcat >expect <<-EOF &&\n+\t\tfatal: reftable block size cannot exceed 65535\n+\t\tEOF\n+\t\ttest_must_fail git -c reftable.restartInterval=65536 pack-refs 2>err &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n-- \n2.45.0\n\n"},{"id":"493886","messageId":"6f2c481acce1ba7728a9a559fbf01fff331153a0.1714630191.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1714630191.git.ps@pks.im","subject":"[PATCH 09/11] refs/reftable: allow disabling writing the object index","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-02T06:52:11Z","receivedAt":"2024-05-02T06:52:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Besides the expected \"ref\" and \"log\" records, the reftable library also\nwrites \"obj\" records. These are basically a reverse mapping of object\nIDs to their respective ref records so that it becomes efficient to\nfigure out which references point to a specific object. The motivation\nfor this data structure is the \"uploadpack.allowTipSHA1InWant\" config,\nwhich allows a client to fetch any object by its hash that has a ref\npointing to it.\n\nThis reverse index is not used by Git at all though, and the expectation\nis that most hosters nowadays use \"uploadpack.allowAnySHA1InWant\". It\nmay thus be preferable for many users to disable writing these optional\nobject indices altogether to safe some precious disk space.\n\nAdd a new config \"reftable.indexObjects\" that allows the user to disable\nthe object index altogether.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config/reftable.txt | 6 +++\n refs/reftable-backend.c | 3 ++\n t/t0613-reftable-write-options.sh | 69 +++++++++++++++++++++++++++++++\n 3 files changed, 78 insertions(+)\n\ndiff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt\nindex 16b915c75e..6e4466f3c5 100644\n--- a/Documentation/config/reftable.txt\n+++ b/Documentation/config/reftable.txt\n@@ -31,3 +31,9 @@ A maximum of `65535` restart points per block is supported.\n +\n The default value is to create restart points every 16 records. A value of `0`\n will use the default value.\n+\n+reftable.indexObjects::\n+\tWhether the reftable backend shall write object blocks. Object blocks\n+\tare a reverse mapping of object ID to the references pointing to them.\n++\n+The default value is `true`.\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex a786143de2..5298fcef6e 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -249,6 +249,9 @@ static int reftable_be_config(const char *var, const char *value,\n \t\t\tdie(\"reftable block size cannot exceed %u\", (unsigned)UINT16_MAX);\n \t\topts->restart_interval = restart_interval;\n \t\treturn 0;\n+\t} else if (!strcmp(var, \"reftable.indexobjects\")) {\n+\t\topts->skip_index_objects = !git_config_bool(var, value);\n+\t\treturn 0;\n \t}\n \n \treturn 0;\ndiff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh\nindex e0a5b26f58..e2708e11d5 100755\n--- a/t/t0613-reftable-write-options.sh\n+++ b/t/t0613-reftable-write-options.sh\n@@ -214,4 +214,73 @@ test_expect_success 'restart interval exceeding maximum supported interval' '\n \t)\n '\n \n+test_expect_success 'object index gets written by default with ref index' '\n+\ttest_config_global core.logAllRefUpdates false &&\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tfor i in $(test_seq 5)\n+\t\tdo\n+\t\t\tprintf \"update refs/heads/branch-%d HEAD\\n\" \"$i\" ||\n+\t\t\treturn 1\n+\t\tdone >input &&\n+\t\tgit update-ref --stdin expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 100\n+\t\tref:\n+\t\t - length: 53\n+\t\t restarts: 1\n+\t\t - length: 95\n+\t\t restarts: 1\n+\t\t - length: 71\n+\t\t restarts: 1\n+\t\t - length: 80\n+\t\t restarts: 1\n+\t\tobj:\n+\t\t - length: 11\n+\t\t restarts: 1\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'object index can be disabled' '\n+\ttest_config_global core.logAllRefUpdates false &&\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tfor i in $(test_seq 5)\n+\t\tdo\n+\t\t\tprintf \"update refs/heads/branch-%d HEAD\\n\" \"$i\" ||\n+\t\t\treturn 1\n+\t\tdone >input &&\n+\t\tgit update-ref --stdin expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 100\n+\t\tref:\n+\t\t - length: 53\n+\t\t restarts: 1\n+\t\t - length: 95\n+\t\t restarts: 1\n+\t\t - length: 71\n+\t\t restarts: 1\n+\t\t - length: 80\n+\t\t restarts: 1\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.45.0\n\n"},{"id":"493887","messageId":"30e2e33479a6b6b07a4652e84eebd78f71fb363a.1714630191.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1714630191.git.ps@pks.im","subject":"[PATCH 10/11] reftable: make the compaction factor configurable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-02T06:52:17Z","receivedAt":"2024-05-02T06:52:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When auto-compacting, the reftable library packs references such that\nthe sizes of the tables form a geometric sequence. The factor for this\ngeometric sequence is hardcoded to 2 right now. We're about to expose\nthis as a config option though, so let's expose the factor via write\noptions.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/reftable-writer.h | 6 ++++++\n reftable/stack.c | 13 +++++++++----\n reftable/stack.h | 3 ++-\n reftable/stack_test.c | 4 ++--\n 4 files changed, 19 insertions(+), 7 deletions(-)\n\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 4cd8ebe6c7..155457b042 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -49,6 +49,12 @@ struct reftable_write_options {\n \n \t/* boolean: Prevent auto-compaction of tables. */\n \tunsigned disable_auto_compact : 1;\n+\n+\t/*\n+\t * Geometric sequence factor used by auto-compaction to decide which\n+\t * tables to compact. Defaults to 2 if unset.\n+\t */\n+\tuint8_t auto_compaction_factor;\n };\n \n /* reftable_block_stats holds statistics for a single block type */\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 7b4fff7c9e..6b0f8e13e7 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1215,12 +1215,16 @@ static int segment_size(struct segment *s)\n \treturn s->end - s->start;\n }\n \n-struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n+struct segment suggest_compaction_segment(uint64_t *sizes, size_t n,\n+\t\t\t\t\t uint8_t factor)\n {\n \tstruct segment seg = { 0 };\n \tuint64_t bytes;\n \tsize_t i;\n \n+\tif (!factor)\n+\t\tfactor = 2;\n+\n \t/*\n \t * If there are no tables or only a single one then we don't have to\n \t * compact anything. The sequence is geometric by definition already.\n@@ -1252,7 +1256,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n \t * \t64, 32, 16, 8, 4, 3, 1\n \t */\n \tfor (i = n - 1; i > 0; i--) {\n-\t\tif (sizes[i - 1] < sizes[i] * 2) {\n+\t\tif (sizes[i - 1] < sizes[i] * factor) {\n \t\t\tseg.end = i + 1;\n \t\t\tbytes = sizes[i];\n \t\t\tbreak;\n@@ -1278,7 +1282,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n \t\tuint64_t curr = bytes;\n \t\tbytes += sizes[i - 1];\n \n-\t\tif (sizes[i - 1] < curr * 2) {\n+\t\tif (sizes[i - 1] < curr * factor) {\n \t\t\tseg.start = i - 1;\n \t\t\tseg.bytes = bytes;\n \t\t}\n@@ -1304,7 +1308,8 @@ int reftable_stack_auto_compact(struct reftable_stack *st)\n {\n \tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n \tstruct segment seg =\n-\t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n+\t\tsuggest_compaction_segment(sizes, st->merged->stack_len,\n+\t\t\t\t\t st->opts.auto_compaction_factor);\n \treftable_free(sizes);\n \tif (segment_size(&seg) > 0)\n \t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\ndiff --git a/reftable/stack.h b/reftable/stack.h\nindex 97d7ebc043..5b45cff4f7 100644\n--- a/reftable/stack.h\n+++ b/reftable/stack.h\n@@ -35,6 +35,7 @@ struct segment {\n \tuint64_t bytes;\n };\n \n-struct segment suggest_compaction_segment(uint64_t *sizes, size_t n);\n+struct segment suggest_compaction_segment(uint64_t *sizes, size_t n,\n+\t\t\t\t\t uint8_t factor);\n \n #endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 3316d55f19..f6c11ef18d 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -767,7 +767,7 @@ static void test_suggest_compaction_segment(void)\n {\n \tuint64_t sizes[] = { 512, 64, 17, 16, 9, 9, 9, 16, 2, 16 };\n \tstruct segment min =\n-\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes), 2);\n \tEXPECT(min.start == 1);\n \tEXPECT(min.end == 10);\n }\n@@ -776,7 +776,7 @@ static void test_suggest_compaction_segment_nothing(void)\n {\n \tuint64_t sizes[] = { 64, 32, 16, 8, 4, 2 };\n \tstruct segment result =\n-\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes), 2);\n \tEXPECT(result.start == result.end);\n }\n \n-- \n2.45.0\n\n"},{"id":"493888","messageId":"861f2e72d987d17b00d68cdaf400d743e2f8c118.1714630191.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1714630191.git.ps@pks.im","subject":"[PATCH 11/11] refs/reftable: allow configuring geometric factor","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-02T06:52:23Z","receivedAt":"2024-05-02T06:52:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Allow configuring the geometric factor used by the auto-compaction\nalgorithm whenever a new table is appended to the stack of tables.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config/reftable.txt | 10 ++++++++++\n refs/reftable-backend.c | 5 +++++\n 2 files changed, 15 insertions(+)\n\ndiff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt\nindex 6e4466f3c5..1c381dda04 100644\n--- a/Documentation/config/reftable.txt\n+++ b/Documentation/config/reftable.txt\n@@ -37,3 +37,13 @@ reftable.indexObjects::\n \tare a reverse mapping of object ID to the references pointing to them.\n +\n The default value is `true`.\n+\n+reftable.geometricFactor::\n+\tWhenever the reftable backend appends a new table to the table it\n+\tperforms auto compaction to ensure that there is only a handful of\n+\ttables. The backend does this by ensuring that tables form a geometric\n+\tsequence regarding the respective sizes of each table.\n++\n+By default, the geometric sequence uses a factor of 2, meaning that for any\n+table, the next-biggest table must at least be twice as big. A maximum factor\n+of 256 is supported.\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 5298fcef6e..657d227c12 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -252,6 +252,11 @@ static int reftable_be_config(const char *var, const char *value,\n \t} else if (!strcmp(var, \"reftable.indexobjects\")) {\n \t\topts->skip_index_objects = !git_config_bool(var, value);\n \t\treturn 0;\n+\t} else if (!strcmp(var, \"reftable.geometricfactor\")) {\n+\t\tunsigned long factor = git_config_ulong(var, value, ctx->kvi);\n+\t\tif (factor > UINT8_MAX)\n+\t\t\tdie(\"reftable geometric factor cannot exceed %u\", (unsigned)UINT8_MAX);\n+\t\topts->auto_compaction_factor = factor;\n \t}\n \n \treturn 0;\n-- \n2.45.0\n\n"},{"id":"493889","messageId":"ZjNA2zNH5twH5qj0@tanuki","threadId":"61406","inReplyTo":"cover.1714630191.git.ps@pks.im","subject":"Re: [PATCH 00/11] reftable: expose write options as config","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-02T07:29:31Z","receivedAt":"2024-05-02T07:29:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, May 02, 2024 at 08:51:27AM +0200, Patrick Steinhardt wrote:\n> Hi,\n> \n> the reftable format has some flexibility with regards to how exactly it\n> writes the respective tables:\n> \n> - The block size allows you to control how large each block is\n> supposed to be. The bigger the block, the more records you can fit\n> into it.\n> \n> - Restart intervals control how often a restart point is written that\n> breaks prefix compression. The lower the interval, the less disk\n> space savings you get.\n> \n> - Object indices can be enabled or disabled. These are optional and\n> Git doesn't use them right now, so disabling them may be a sensible\n> thing to do if you want to save some disk space.\n> \n> - The geometric factor controls when we compact tables during auto\n> compaction.\n> \n> This patch series exposes all of these via a new set of configs so that\n> they can be tweaked by the user as-needed. It's not expected that those\n> are really of much importance for the \"normal\" user -- the defaults\n> should be good enough. But for edge cases (huge repos with millions of\n> refs) and for hosting providers these knobs can be helpful.\n> \n> This patch series applies on top of d4cc1ec35f (Start the 2.46 cycle,\n> 2024-04-30).\n\nUgh. I actually intended to pull in ps/reftable-write-optim as a\ndependency because I know it causes conflicts. But I screwed this up as\nI thought that the topic was merged into \"master\" already, even though\nit has only hit \"next\".\n\nI'll refrain from sending a new version immediately though and will wait\nfor reviews first. Once those are in I will pull in the above topic.\n\nPatrick\n"},{"id":"494063","messageId":"xmqqv83u4omc.fsf@gitster.g","threadId":"61406","inReplyTo":"ZjNA2zNH5twH5qj0@tanuki","subject":"Re: [PATCH 00/11] reftable: expose write options as config","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-03T20:38:51Z","receivedAt":"2024-05-03T20:38:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> Ugh. I actually intended to pull in ps/reftable-write-optim as a\n> dependency because I know it causes conflicts. But I screwed this up as\n> I thought that the topic was merged into \"master\" already, even though\n> it has only hit \"next\".\n>\n> I'll refrain from sending a new version immediately though and will wait\n> for reviews first. Once those are in I will pull in the above topic.\n\nI saw [01/11] has changes to stack_check_addition() that disappeared\nby the other topic, and the resolution is to remove the function (as\nnobody calls it anyway). Also [02/11] has changes to refname_test.c\nthat can be resolved by just removing the file.\n\nIf there are other semantic conflict resolution needed, such a\nrebasing is appreciated, but otherwise, there is no strong need to\nrebase.\n\nThe mention of the name of the other topic that has interactions was\nvery much helpful.\n\nThanks.\n"},{"id":"494117","messageId":"Zjh99shqHEI1ehF6@tanuki","threadId":"61406","inReplyTo":"xmqqv83u4omc.fsf@gitster.g","subject":"Re: [PATCH 00/11] reftable: expose write options as config","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-06T06:51:34Z","receivedAt":"2024-05-06T06:51:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, May 03, 2024 at 01:38:51PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > Ugh. I actually intended to pull in ps/reftable-write-optim as a\n> > dependency because I know it causes conflicts. But I screwed this up as\n> > I thought that the topic was merged into \"master\" already, even though\n> > it has only hit \"next\".\n> >\n> > I'll refrain from sending a new version immediately though and will wait\n> > for reviews first. Once those are in I will pull in the above topic.\n> \n> I saw [01/11] has changes to stack_check_addition() that disappeared\n> by the other topic, and the resolution is to remove the function (as\n> nobody calls it anyway). Also [02/11] has changes to refname_test.c\n> that can be resolved by just removing the file.\n\nYup, sounds right to me.\n\n> If there are other semantic conflict resolution needed, such a\n> rebasing is appreciated, but otherwise, there is no strong need to\n> rebase.\n\nNo other semantic conflicts I'm aware of, no. Thanks!\n\nPatrick\n\n> The mention of the name of the other topic that has interactions was\n> very much helpful.\n> \n> Thanks.\n"},{"id":"494205","messageId":"2rozmhwf2jpyqmy5vtf2t664ko7ztinbzdmwirrqxspjwzgj7t@jctxour2o73r","threadId":"61406","inReplyTo":"cover.1714630191.git.ps@pks.im","subject":"Re: [PATCH 00/11] reftable: expose write options as config","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-05-06T21:29:24Z","receivedAt":"2024-05-06T21:30:45Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/05/02 08:51AM, Patrick Steinhardt wrote:\n> Hi,\n> \n> the reftable format has some flexibility with regards to how exactly it\n> writes the respective tables:\n> \n> - The block size allows you to control how large each block is\n> supposed to be. The bigger the block, the more records you can fit\n> into it.\n> \n> - Restart intervals control how often a restart point is written that\n> breaks prefix compression. The lower the interval, the less disk\n> space savings you get.\n> \n> - Object indices can be enabled or disabled. These are optional and\n> Git doesn't use them right now, so disabling them may be a sensible\n> thing to do if you want to save some disk space.\n> \n> - The geometric factor controls when we compact tables during auto\n> compaction.\n> \n> This patch series exposes all of these via a new set of configs so that\n> they can be tweaked by the user as-needed. It's not expected that those\n> are really of much importance for the \"normal\" user -- the defaults\n> should be good enough. But for edge cases (huge repos with millions of\n> refs) and for hosting providers these knobs can be helpful.\n> \n> This patch series applies on top of d4cc1ec35f (Start the 2.46 cycle,\n> 2024-04-30).\n\nI have reviewed these patches and I have nothing to add. Thanks Patrick!\n\n-Justin\n"},{"id":"494412","messageId":"CAOLa=ZQ3m3HOxpAUVK8_KASfQwmhXNraYrm-M-T2BTHUm5c5ZQ@mail.gmail.com","threadId":"61406","inReplyTo":"47cee6e25ebaac5af9fd0c9eaad7fc50b2547cad.1714630191.git.ps@pks.im","subject":"Re: [PATCH 01/11] reftable: consistently refer to `reftable_write_options` as `opts`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-05-10T09:00:31Z","receivedAt":"2024-05-10T09:07:22Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Throughout the reftable library the `reftable_write_options` are\n> sometimes referred to as `cfg` and sometimes as `opts`. Unify these to\n> consistently use `opts` to avoid confusion.\n>\n\nI think one location was missed:\n\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 3316d55f19..40eb793b3c 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -396,7 +396,7 @@ static void\ntest_reftable_stack_auto_compaction_fails_gracefully(void)\n\n static void test_reftable_stack_validate_refname(void)\n {\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n \tchar *dir = get_tmp_dir(__LINE__);\n@@ -410,7 +410,7 @@ static void test_reftable_stack_validate_refname(void)\n \t};\n \tchar *additions[] = { \"a\", \"a/b/c\" };\n\n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n\n \terr = reftable_stack_add(st, &write_test_ref, &ref);\n\nRest of the patch looks good. Thanks\n"},{"id":"494413","messageId":"CAOLa=ZR7=hVXyGr2iSZW2C8mtwR2SHLikuCVKS-qTB7AE+0+hQ@mail.gmail.com","threadId":"61406","inReplyTo":"b4e4db5735beb106511980adca48dc416c4b0d95.1714630191.git.ps@pks.im","subject":"Re: [PATCH 06/11] refs/reftable: allow configuring block size","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-05-10T09:29:19Z","receivedAt":"2024-05-10T09:29:21Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n[snip]\n\n> @@ -230,6 +231,23 @@ static int read_ref_without_reload(struct reftable_stack *stack,\n> \treturn ret;\n> }\n>\n> +static int reftable_be_config(const char *var, const char *value,\n> +\t\t\t const struct config_context *ctx,\n> +\t\t\t void *_opts)\n> +{\n> +\tstruct reftable_write_options *opts = _opts;\n> +\n> +\tif (!strcmp(var, \"reftable.blocksize\")) {\n> +\t\tunsigned long block_size = git_config_ulong(var, value, ctx->kvi);\n> +\t\tif (block_size > 16777215)\n> +\t\t\tdie(\"reftable block size cannot exceed 16MB\");\n> +\t\topts->block_size = block_size;\n> +\t\treturn 0;\n\nnit: unecessary return\n\n> +\t}\n> +\n> +\treturn 0;\n> +}\n> +\n> static struct ref_store *reftable_be_init(struct repository *repo,\n> \t\t\t\t\t const char *gitdir,\n> \t\t\t\t\t unsigned int store_flags)\n> @@ -245,12 +263,24 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n> \tbase_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);\n> \tstrmap_init(&refs->worktree_stacks);\n> \trefs->store_flags = store_flags;\n> -\trefs->write_options.block_size = 4096;\n> +\n\nNit: do we need this newline?\n\n> \trefs->write_options.hash_id = repo->hash_algo->format_id;\n> \trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n> \trefs->write_options.disable_auto_compact =\n> \t\t!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1);\n>\n> +\tgit_config(reftable_be_config, &refs->write_options);\n> +\n> +\t/*\n> +\t * It is somewhat unfortunate that we have to mirror the default block\n> +\t * size of the reftable library here. But given that the write options\n> +\t * wouldn't be updated by the library here, and given that we require\n> +\t * the proper block size to trim reflog message so that they fit, we\n> +\t * must set up a proper value here.\n> +\t */\n> +\tif (!refs->write_options.block_size)\n> +\t\trefs->write_options.block_size = 4096;\n> +\n\nWouldn't it be to import and use `reftable/constants.h` here?\n\n[snip]\n"},{"id":"494415","messageId":"CAOLa=ZQMoYM6L_2jwE9pOVUOV1tLep93m7VGLA00nrpHKdy0Vg@mail.gmail.com","threadId":"61406","inReplyTo":"30e2e33479a6b6b07a4652e84eebd78f71fb363a.1714630191.git.ps@pks.im","subject":"Re: [PATCH 10/11] reftable: make the compaction factor configurable","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-05-10T09:55:10Z","receivedAt":"2024-05-10T09:55:12Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> When auto-compacting, the reftable library packs references such that\n> the sizes of the tables form a geometric sequence. The factor for this\n> geometric sequence is hardcoded to 2 right now. We're about to expose\n> this as a config option though, so let's expose the factor via write\n> options.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> reftable/reftable-writer.h | 6 ++++++\n> reftable/stack.c | 13 +++++++++----\n> reftable/stack.h | 3 ++-\n> reftable/stack_test.c | 4 ++--\n> 4 files changed, 19 insertions(+), 7 deletions(-)\n>\n> diff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\n> index 4cd8ebe6c7..155457b042 100644\n> --- a/reftable/reftable-writer.h\n> +++ b/reftable/reftable-writer.h\n> @@ -49,6 +49,12 @@ struct reftable_write_options {\n>\n> \t/* boolean: Prevent auto-compaction of tables. */\n> \tunsigned disable_auto_compact : 1;\n> +\n> +\t/*\n> +\t * Geometric sequence factor used by auto-compaction to decide which\n> +\t * tables to compact. Defaults to 2 if unset.\n> +\t */\n> +\tuint8_t auto_compaction_factor;\n> };\n>\n> /* reftable_block_stats holds statistics for a single block type */\n> diff --git a/reftable/stack.c b/reftable/stack.c\n> index 7b4fff7c9e..6b0f8e13e7 100644\n> --- a/reftable/stack.c\n> +++ b/reftable/stack.c\n> @@ -1215,12 +1215,16 @@ static int segment_size(struct segment *s)\n> \treturn s->end - s->start;\n> }\n>\n> -struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n> +struct segment suggest_compaction_segment(uint64_t *sizes, size_t n,\n> +\t\t\t\t\t uint8_t factor)\n> {\n> \tstruct segment seg = { 0 };\n> \tuint64_t bytes;\n> \tsize_t i;\n>\n> +\tif (!factor)\n> +\t\tfactor = 2;\n> +\n\nThis should probably go in reftable/constants.h\n\n[snip]\n"},{"id":"494417","messageId":"CAOLa=ZSLx7fAo0kT5OR6wyjyTczDLOpcS82xGd195uWVs7JRFQ@mail.gmail.com","threadId":"61406","inReplyTo":"861f2e72d987d17b00d68cdaf400d743e2f8c118.1714630191.git.ps@pks.im","subject":"Re: [PATCH 11/11] refs/reftable: allow configuring geometric factor","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-05-10T09:58:58Z","receivedAt":"2024-05-10T09:58:59Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Allow configuring the geometric factor used by the auto-compaction\n> algorithm whenever a new table is appended to the stack of tables.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> Documentation/config/reftable.txt | 10 ++++++++++\n> refs/reftable-backend.c | 5 +++++\n> 2 files changed, 15 insertions(+)\n>\n> diff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt\n> index 6e4466f3c5..1c381dda04 100644\n> --- a/Documentation/config/reftable.txt\n> +++ b/Documentation/config/reftable.txt\n> @@ -37,3 +37,13 @@ reftable.indexObjects::\n> \tare a reverse mapping of object ID to the references pointing to them.\n> +\n> The default value is `true`.\n> +\n> +reftable.geometricFactor::\n> +\tWhenever the reftable backend appends a new table to the table it\n\nThis doesn't read right, did you mean 's/to the table/,' perhaps?\n\n[snip]\n"},{"id":"494419","messageId":"CAOLa=ZTz5k8Ge8Ba1f0AUXE1NRE1sO_g5zrtTeYcthKP9VVeCw@mail.gmail.com","threadId":"61406","inReplyTo":"cover.1714630191.git.ps@pks.im","subject":"Re: [PATCH 00/11] reftable: expose write options as config","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-05-10T10:00:41Z","receivedAt":"2024-05-10T10:00:42Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Hello Patrick,\n\nPatrick Steinhardt writes:\n> Hi,\n>\n> the reftable format has some flexibility with regards to how exactly it\n> writes the respective tables:\n>\n> - The block size allows you to control how large each block is\n> supposed to be. The bigger the block, the more records you can fit\n> into it.\n>\n> - Restart intervals control how often a restart point is written that\n> breaks prefix compression. The lower the interval, the less disk\n> space savings you get.\n>\n> - Object indices can be enabled or disabled. These are optional and\n> Git doesn't use them right now, so disabling them may be a sensible\n> thing to do if you want to save some disk space.\n>\n> - The geometric factor controls when we compact tables during auto\n> compaction.\n>\n> This patch series exposes all of these via a new set of configs so that\n> they can be tweaked by the user as-needed. It's not expected that those\n> are really of much importance for the \"normal\" user -- the defaults\n> should be good enough. But for edge cases (huge repos with millions of\n> refs) and for hosting providers these knobs can be helpful.\n>\n> This patch series applies on top of d4cc1ec35f (Start the 2.46 cycle,\n> 2024-04-30).\n>\n> Patrick\n>\n\nI'v gone through the commits and apart from a few small comments, I\nthink it looks great already.\n\nThanks,\nKarthik\n"},{"id":"494421","messageId":"Zj3zT0AQMIVh-Y4P@tanuki","threadId":"61406","inReplyTo":"CAOLa=ZQ3m3HOxpAUVK8_KASfQwmhXNraYrm-M-T2BTHUm5c5ZQ@mail.gmail.com","subject":"Re: [PATCH 01/11] reftable: consistently refer to `reftable_write_options` as `opts`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T10:13:35Z","receivedAt":"2024-05-10T10:13:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, May 10, 2024 at 02:00:31AM -0700, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> \n> > Throughout the reftable library the `reftable_write_options` are\n> > sometimes referred to as `cfg` and sometimes as `opts`. Unify these to\n> > consistently use `opts` to avoid confusion.\n> >\n> \n> I think one location was missed:\n> \n> diff --git a/reftable/stack_test.c b/reftable/stack_test.c\n> index 3316d55f19..40eb793b3c 100644\n> --- a/reftable/stack_test.c\n> +++ b/reftable/stack_test.c\n> @@ -396,7 +396,7 @@ static void\n> test_reftable_stack_auto_compaction_fails_gracefully(void)\n> \n> static void test_reftable_stack_validate_refname(void)\n> {\n> -\tstruct reftable_write_options cfg = { 0 };\n> +\tstruct reftable_write_options opts = { 0 };\n> \tstruct reftable_stack *st = NULL;\n> \tint err;\n> \tchar *dir = get_tmp_dir(__LINE__);\n> @@ -410,7 +410,7 @@ static void test_reftable_stack_validate_refname(void)\n> \t};\n> \tchar *additions[] = { \"a\", \"a/b/c\" };\n> \n> -\terr = reftable_new_stack(&st, dir, cfg);\n> +\terr = reftable_new_stack(&st, dir, opts);\n> \tEXPECT_ERR(err);\n> \n> \terr = reftable_stack_add(st, &write_test_ref, &ref);\n> \n> Rest of the patch looks good. Thanks\n\nThis section has been removed in 485c63cf5c (reftable: remove name\nchecks, 2024-04-08), so it doesn't exist in `master` anymore. Which is\ncheating a bit because the topic does not build on top of anything where\nthat commit would be reachable. But both rebasing the topic and\nwillfully creating a conflict would probably make Junio's life harder\nnow, so I'll just leave it at that and lie by ommission.\n\nBut thanks anyway for reading this carefully and double checking the\nresults!\n\nPatrick\n"},{"id":"494422","messageId":"Zj3zVQbASOx4EPgu@tanuki","threadId":"61406","inReplyTo":"CAOLa=ZR7=hVXyGr2iSZW2C8mtwR2SHLikuCVKS-qTB7AE+0+hQ@mail.gmail.com","subject":"Re: [PATCH 06/11] refs/reftable: allow configuring block size","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T10:13:41Z","receivedAt":"2024-05-10T10:13:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, May 10, 2024 at 02:29:19AM -0700, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> \n> [snip]\n> \n> > @@ -230,6 +231,23 @@ static int read_ref_without_reload(struct reftable_stack *stack,\n> > \treturn ret;\n> > }\n> >\n> > +static int reftable_be_config(const char *var, const char *value,\n> > +\t\t\t const struct config_context *ctx,\n> > +\t\t\t void *_opts)\n> > +{\n> > +\tstruct reftable_write_options *opts = _opts;\n> > +\n> > +\tif (!strcmp(var, \"reftable.blocksize\")) {\n> > +\t\tunsigned long block_size = git_config_ulong(var, value, ctx->kvi);\n> > +\t\tif (block_size > 16777215)\n> > +\t\t\tdie(\"reftable block size cannot exceed 16MB\");\n> > +\t\topts->block_size = block_size;\n> > +\t\treturn 0;\n> \n> nit: unecessary return\n\nIt's unnecessary indeed. I first wanted to defend this, but then I\nnoticed that I'm also being inconsistent here where the last branch\nwon't have `return 0;` at the end of this series.\n\nWill remove.\n\n> > +\t}\n> > +\n> > +\treturn 0;\n> > +}\n> > +\n> > static struct ref_store *reftable_be_init(struct repository *repo,\n> > \t\t\t\t\t const char *gitdir,\n> > \t\t\t\t\t unsigned int store_flags)\n> > @@ -245,12 +263,24 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n> > \tbase_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);\n> > \tstrmap_init(&refs->worktree_stacks);\n> > \trefs->store_flags = store_flags;\n> > -\trefs->write_options.block_size = 4096;\n> > +\n> \n> Nit: do we need this newline?\n\nI think it's easier to read that way.\n\n> > \trefs->write_options.hash_id = repo->hash_algo->format_id;\n> > \trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n> > \trefs->write_options.disable_auto_compact =\n> > \t\t!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1);\n> >\n> > +\tgit_config(reftable_be_config, &refs->write_options);\n> > +\n> > +\t/*\n> > +\t * It is somewhat unfortunate that we have to mirror the default block\n> > +\t * size of the reftable library here. But given that the write options\n> > +\t * wouldn't be updated by the library here, and given that we require\n> > +\t * the proper block size to trim reflog message so that they fit, we\n> > +\t * must set up a proper value here.\n> > +\t */\n> > +\tif (!refs->write_options.block_size)\n> > +\t\trefs->write_options.block_size = 4096;\n> > +\n> \n> Wouldn't it be to import and use `reftable/constants.h` here?\n\nHeaders in the \"reftable/\" directory which do not have a \"reftable-\"\nprefix are considered to be private. So those shouldn't be used.\n\nPatrick\n"},{"id":"494423","messageId":"Zj3zWwkV4CPj78lt@tanuki","threadId":"61406","inReplyTo":"CAOLa=ZQMoYM6L_2jwE9pOVUOV1tLep93m7VGLA00nrpHKdy0Vg@mail.gmail.com","subject":"Re: [PATCH 10/11] reftable: make the compaction factor configurable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T10:13:47Z","receivedAt":"2024-05-10T10:13:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, May 10, 2024 at 04:55:10AM -0500, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n[snip]\n> > diff --git a/reftable/stack.c b/reftable/stack.c\n> > index 7b4fff7c9e..6b0f8e13e7 100644\n> > --- a/reftable/stack.c\n> > +++ b/reftable/stack.c\n> > @@ -1215,12 +1215,16 @@ static int segment_size(struct segment *s)\n> > \treturn s->end - s->start;\n> > }\n> >\n> > -struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n> > +struct segment suggest_compaction_segment(uint64_t *sizes, size_t n,\n> > +\t\t\t\t\t uint8_t factor)\n> > {\n> > \tstruct segment seg = { 0 };\n> > \tuint64_t bytes;\n> > \tsize_t i;\n> >\n> > +\tif (!factor)\n> > +\t\tfactor = 2;\n> > +\n> \n> This should probably go in reftable/constants.h\n\nGood idea, will do.\n\nPatrick\n"},{"id":"494424","messageId":"Zj3zYvr-o3hpnMGU@tanuki","threadId":"61406","inReplyTo":"CAOLa=ZSLx7fAo0kT5OR6wyjyTczDLOpcS82xGd195uWVs7JRFQ@mail.gmail.com","subject":"Re: [PATCH 11/11] refs/reftable: allow configuring geometric factor","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T10:13:54Z","receivedAt":"2024-05-10T10:13:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, May 10, 2024 at 02:58:58AM -0700, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> \n> > Allow configuring the geometric factor used by the auto-compaction\n> > algorithm whenever a new table is appended to the stack of tables.\n> >\n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > Documentation/config/reftable.txt | 10 ++++++++++\n> > refs/reftable-backend.c | 5 +++++\n> > 2 files changed, 15 insertions(+)\n> >\n> > diff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt\n> > index 6e4466f3c5..1c381dda04 100644\n> > --- a/Documentation/config/reftable.txt\n> > +++ b/Documentation/config/reftable.txt\n> > @@ -37,3 +37,13 @@ reftable.indexObjects::\n> > \tare a reverse mapping of object ID to the references pointing to them.\n> > +\n> > The default value is `true`.\n> > +\n> > +reftable.geometricFactor::\n> > +\tWhenever the reftable backend appends a new table to the table it\n> \n> This doesn't read right, did you mean 's/to the table/,' perhaps?\n\nThis should rather be \"to the stack,\". Good catch.\n\nPatrick\n"},{"id":"494425","messageId":"Zj3zapDuams3DlxO@tanuki","threadId":"61406","inReplyTo":"CAOLa=ZTz5k8Ge8Ba1f0AUXE1NRE1sO_g5zrtTeYcthKP9VVeCw@mail.gmail.com","subject":"Re: [PATCH 00/11] reftable: expose write options as config","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T10:14:02Z","receivedAt":"2024-05-10T10:14:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, May 10, 2024 at 03:00:41AM -0700, Karthik Nayak wrote:\n> I'v gone through the commits and apart from a few small comments, I\n> think it looks great already.\n> \n> Thanks,\n> Karthik\n\nThanks for your review!\n\nPatrick\n"},{"id":"494426","messageId":"cover.1715336797.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1714630191.git.ps@pks.im","subject":"[PATCH v2 00/11] reftable: expose write options as config","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T10:29:16Z","receivedAt":"2024-05-10T10:29:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the second version of my patch series that exposes various\noptions of the reftable writer via Git configuration.\n\nChanges compared to v1:\n\n - Drop unneeded return statements.\n\n - Move default geometric factor into \"constants.h\".\n\n - Fix a typo in a commit message.\n\nThanks!\n\nPatrick\n\nPatrick Steinhardt (11):\n reftable: consistently refer to `reftable_write_options` as `opts`\n reftable: consistently pass write opts as value\n reftable/writer: drop static variable used to initialize strbuf\n reftable/writer: improve error when passed an invalid block size\n reftable/dump: support dumping a table's block structure\n refs/reftable: allow configuring block size\n reftable: use `uint16_t` to track restart interval\n refs/reftable: allow configuring restart interval\n refs/reftable: allow disabling writing the object index\n reftable: make the compaction factor configurable\n refs/reftable: allow configuring geometric factor\n\n Documentation/config.txt | 2 +\n Documentation/config/reftable.txt | 49 +++++\n refs/reftable-backend.c | 43 ++++-\n reftable/block.h | 2 +-\n reftable/constants.h | 1 +\n reftable/dump.c | 12 +-\n reftable/merged_test.c | 6 +-\n reftable/reader.c | 63 +++++++\n reftable/readwrite_test.c | 26 +--\n reftable/refname_test.c | 2 +-\n reftable/reftable-reader.h | 2 +\n reftable/reftable-stack.h | 2 +-\n reftable/reftable-writer.h | 10 +-\n reftable/stack.c | 57 +++---\n reftable/stack.h | 5 +-\n reftable/stack_test.c | 118 ++++++------\n reftable/writer.c | 20 +--\n t/t0613-reftable-write-options.sh | 286 ++++++++++++++++++++++++++++++\n 18 files changed, 576 insertions(+), 130 deletions(-)\n create mode 100644 Documentation/config/reftable.txt\n create mode 100755 t/t0613-reftable-write-options.sh\n\nRange-diff against v1:\n 1: 47cee6e25e = 1: 7efa566306 reftable: consistently refer to `reftable_write_options` as `opts`\n 2: d8a0764e87 = 2: e6f8fc09c2 reftable: consistently pass write opts as value\n 3: c040f81fba = 3: aa2903e3e5 reftable/writer: drop static variable used to initialize strbuf\n 4: ef79bb1b7b = 4: 5e7cbb7b19 reftable/writer: improve error when passed an invalid block size\n 5: 4d4407d4a4 = 5: ed1c150d90 reftable/dump: support dumping a table's block structure\n 6: b4e4db5735 ! 6: be5bdc6dc1 refs/reftable: allow configuring block size\n @@ refs/reftable-backend.c: static int read_ref_without_reload(struct reftable_stac\n +\t\tif (block_size > 16777215)\n +\t\t\tdie(\"reftable block size cannot exceed 16MB\");\n +\t\topts->block_size = block_size;\n -+\t\treturn 0;\n +\t}\n +\n +\treturn 0;\n 7: 79d9e07ca9 = 7: 05e8d1df2d reftable: use `uint16_t` to track restart interval\n 8: 653ec4dfa5 ! 8: bc0bf65553 refs/reftable: allow configuring restart interval\n @@ Documentation/config/reftable.txt: readers during access.\n \n ## refs/reftable-backend.c ##\n @@ refs/reftable-backend.c: static int reftable_be_config(const char *var, const char *value,\n + \t\tif (block_size > 16777215)\n \t\t\tdie(\"reftable block size cannot exceed 16MB\");\n \t\topts->block_size = block_size;\n - \t\treturn 0;\n +\t} else if (!strcmp(var, \"reftable.restartinterval\")) {\n +\t\tunsigned long restart_interval = git_config_ulong(var, value, ctx->kvi);\n +\t\tif (restart_interval > UINT16_MAX)\n +\t\t\tdie(\"reftable block size cannot exceed %u\", (unsigned)UINT16_MAX);\n +\t\topts->restart_interval = restart_interval;\n -+\t\treturn 0;\n \t}\n \n \treturn 0;\n 9: 6f2c481acc ! 9: 6bc240fd0c refs/reftable: allow disabling writing the object index\n @@ Documentation/config/reftable.txt: A maximum of `65535` restart points per block\n \n ## refs/reftable-backend.c ##\n @@ refs/reftable-backend.c: static int reftable_be_config(const char *var, const char *value,\n + \t\tif (restart_interval > UINT16_MAX)\n \t\t\tdie(\"reftable block size cannot exceed %u\", (unsigned)UINT16_MAX);\n \t\topts->restart_interval = restart_interval;\n - \t\treturn 0;\n +\t} else if (!strcmp(var, \"reftable.indexobjects\")) {\n +\t\topts->skip_index_objects = !git_config_bool(var, value);\n -+\t\treturn 0;\n \t}\n \n \treturn 0;\n10: 30e2e33479 ! 10: 9d4c1f0340 reftable: make the compaction factor configurable\n @@ Commit message\n \n Signed-off-by: Patrick Steinhardt \n \n + ## reftable/constants.h ##\n +@@ reftable/constants.h: license that can be found in the LICENSE file or at\n + \n + #define MAX_RESTARTS ((1 << 16) - 1)\n + #define DEFAULT_BLOCK_SIZE 4096\n ++#define DEFAULT_GEOMETRIC_FACTOR 2\n + \n + #endif\n +\n ## reftable/reftable-writer.h ##\n @@ reftable/reftable-writer.h: struct reftable_write_options {\n \n @@ reftable/reftable-writer.h: struct reftable_write_options {\n /* reftable_block_stats holds statistics for a single block type */\n \n ## reftable/stack.c ##\n +@@ reftable/stack.c: license that can be found in the LICENSE file or at\n + \n + #include \"../write-or-die.h\"\n + #include \"system.h\"\n ++#include \"constants.h\"\n + #include \"merged.h\"\n + #include \"reader.h\"\n + #include \"refname.h\"\n @@ reftable/stack.c: static int segment_size(struct segment *s)\n \treturn s->end - s->start;\n }\n @@ reftable/stack.c: static int segment_size(struct segment *s)\n \tsize_t i;\n \n +\tif (!factor)\n -+\t\tfactor = 2;\n ++\t\tfactor = DEFAULT_GEOMETRIC_FACTOR;\n +\n \t/*\n \t * If there are no tables or only a single one then we don't have to\n11: 861f2e72d9 ! 11: e1282e53fb refs/reftable: allow configuring geometric factor\n @@ Documentation/config/reftable.txt: reftable.indexObjects::\n The default value is `true`.\n +\n +reftable.geometricFactor::\n -+\tWhenever the reftable backend appends a new table to the table it\n ++\tWhenever the reftable backend appends a new table to the stack, it\n +\tperforms auto compaction to ensure that there is only a handful of\n +\ttables. The backend does this by ensuring that tables form a geometric\n +\tsequence regarding the respective sizes of each table.\n @@ Documentation/config/reftable.txt: reftable.indexObjects::\n \n ## refs/reftable-backend.c ##\n @@ refs/reftable-backend.c: static int reftable_be_config(const char *var, const char *value,\n + \t\topts->restart_interval = restart_interval;\n \t} else if (!strcmp(var, \"reftable.indexobjects\")) {\n \t\topts->skip_index_objects = !git_config_bool(var, value);\n - \t\treturn 0;\n +\t} else if (!strcmp(var, \"reftable.geometricfactor\")) {\n +\t\tunsigned long factor = git_config_ulong(var, value, ctx->kvi);\n +\t\tif (factor > UINT8_MAX)\n\nbase-commit: d4cc1ec35f3bcce816b69986ca41943f6ce21377\n-- \n2.45.0\n\n"},{"id":"494427","messageId":"7efa56630681b6aca64c04fd19fb020ffbb76152.1715336798.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715336797.git.ps@pks.im","subject":"[PATCH v2 01/11] reftable: consistently refer to `reftable_write_options` as `opts`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T10:29:20Z","receivedAt":"2024-05-10T10:29:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Throughout the reftable library the `reftable_write_options` are\nsometimes referred to as `cfg` and sometimes as `opts`. Unify these to\nconsistently use `opts` to avoid confusion.\n\nWhile at it, touch up the coding style a bit by removing unneeded braces\naround one-line statements and newlines between variable declarations.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/dump.c | 4 +-\n reftable/reftable-stack.h | 2 +-\n reftable/stack.c | 43 +++++++-------\n reftable/stack.h | 2 +-\n reftable/stack_test.c | 114 +++++++++++++++++---------------------\n 5 files changed, 75 insertions(+), 90 deletions(-)\n\ndiff --git a/reftable/dump.c b/reftable/dump.c\nindex 26e0393c7d..9c770a10cc 100644\n--- a/reftable/dump.c\n+++ b/reftable/dump.c\n@@ -27,9 +27,9 @@ license that can be found in the LICENSE file or at\n static int compact_stack(const char *stackdir)\n {\n \tstruct reftable_stack *stack = NULL;\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \n-\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tint err = reftable_new_stack(&stack, stackdir, opts);\n \tif (err < 0)\n \t\tgoto done;\n \ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nindex 1b602dda58..9c8e4eef49 100644\n--- a/reftable/reftable-stack.h\n+++ b/reftable/reftable-stack.h\n@@ -29,7 +29,7 @@ struct reftable_stack;\n * stored in 'dir'. Typically, this should be .git/reftables.\n */\n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n-\t\t struct reftable_write_options config);\n+\t\t struct reftable_write_options opts);\n \n /* returns the update_index at which a next table should be written. */\n uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 80266bcbab..3979657793 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -56,15 +56,14 @@ static int reftable_fd_flush(void *arg)\n }\n \n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n-\t\t struct reftable_write_options config)\n+\t\t struct reftable_write_options opts)\n {\n \tstruct reftable_stack *p = reftable_calloc(1, sizeof(*p));\n \tstruct strbuf list_file_name = STRBUF_INIT;\n \tint err = 0;\n \n-\tif (config.hash_id == 0) {\n-\t\tconfig.hash_id = GIT_SHA1_FORMAT_ID;\n-\t}\n+\tif (opts.hash_id == 0)\n+\t\topts.hash_id = GIT_SHA1_FORMAT_ID;\n \n \t*dest = NULL;\n \n@@ -75,7 +74,7 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \tp->list_file = strbuf_detach(&list_file_name, NULL);\n \tp->list_fd = -1;\n \tp->reftable_dir = xstrdup(dir);\n-\tp->config = config;\n+\tp->opts = opts;\n \n \terr = reftable_stack_reload_maybe_reuse(p, 1);\n \tif (err < 0) {\n@@ -257,7 +256,7 @@ static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n \n \t/* success! */\n \terr = reftable_new_merged_table(&new_merged, new_tables,\n-\t\t\t\t\tnew_readers_len, st->config.hash_id);\n+\t\t\t\t\tnew_readers_len, st->opts.hash_id);\n \tif (err < 0)\n \t\tgoto done;\n \n@@ -580,8 +579,8 @@ static int reftable_stack_init_addition(struct reftable_addition *add,\n \t\t}\n \t\tgoto done;\n \t}\n-\tif (st->config.default_permissions) {\n-\t\tif (chmod(add->lock_file->filename.buf, st->config.default_permissions) < 0) {\n+\tif (st->opts.default_permissions) {\n+\t\tif (chmod(add->lock_file->filename.buf, st->opts.default_permissions) < 0) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n \t\t}\n@@ -680,7 +679,7 @@ int reftable_addition_commit(struct reftable_addition *add)\n \tif (err)\n \t\tgoto done;\n \n-\tif (!add->stack->config.disable_auto_compact) {\n+\tif (!add->stack->opts.disable_auto_compact) {\n \t\t/*\n \t\t * Auto-compact the stack to keep the number of tables in\n \t\t * control. It is possible that a concurrent writer is already\n@@ -758,9 +757,9 @@ int reftable_addition_add(struct reftable_addition *add,\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n-\tif (add->stack->config.default_permissions) {\n+\tif (add->stack->opts.default_permissions) {\n \t\tif (chmod(get_tempfile_path(tab_file),\n-\t\t\t add->stack->config.default_permissions)) {\n+\t\t\t add->stack->opts.default_permissions)) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n \t\t}\n@@ -768,7 +767,7 @@ int reftable_addition_add(struct reftable_addition *add,\n \ttab_fd = get_tempfile_fd(tab_file);\n \n \twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, &tab_fd,\n-\t\t\t\t &add->stack->config);\n+\t\t\t\t &add->stack->opts);\n \terr = write_table(wr, arg);\n \tif (err < 0)\n \t\tgoto done;\n@@ -855,14 +854,14 @@ static int stack_compact_locked(struct reftable_stack *st,\n \t}\n \ttab_fd = get_tempfile_fd(tab_file);\n \n-\tif (st->config.default_permissions &&\n-\t chmod(get_tempfile_path(tab_file), st->config.default_permissions) < 0) {\n+\tif (st->opts.default_permissions &&\n+\t chmod(get_tempfile_path(tab_file), st->opts.default_permissions) < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n \n \twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush,\n-\t\t\t\t &tab_fd, &st->config);\n+\t\t\t\t &tab_fd, &st->opts);\n \terr = stack_write_compact(st, wr, first, last, config);\n \tif (err < 0)\n \t\tgoto done;\n@@ -910,7 +909,7 @@ static int stack_write_compact(struct reftable_stack *st,\n \t\t\t\t st->readers[last]->max_update_index);\n \n \terr = reftable_new_merged_table(&mt, subtabs, subtabs_len,\n-\t\t\t\t\tst->config.hash_id);\n+\t\t\t\t\tst->opts.hash_id);\n \tif (err < 0) {\n \t\treftable_free(subtabs);\n \t\tgoto done;\n@@ -1100,9 +1099,9 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\tgoto done;\n \t}\n \n-\tif (st->config.default_permissions) {\n+\tif (st->opts.default_permissions) {\n \t\tif (chmod(get_lock_file_path(&tables_list_lock),\n-\t\t\t st->config.default_permissions) < 0) {\n+\t\t\t st->opts.default_permissions) < 0) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n \t\t}\n@@ -1292,7 +1291,7 @@ static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n {\n \tuint64_t *sizes =\n \t\treftable_calloc(st->merged->stack_len, sizeof(*sizes));\n-\tint version = (st->config.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2;\n+\tint version = (st->opts.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2;\n \tint overhead = header_size(version) - 1;\n \tint i = 0;\n \tfor (i = 0; i < st->merged->stack_len; i++) {\n@@ -1368,7 +1367,7 @@ static int stack_check_addition(struct reftable_stack *st,\n \tint len = 0;\n \tint i = 0;\n \n-\tif (st->config.skip_name_check)\n+\tif (st->opts.skip_name_check)\n \t\treturn 0;\n \n \terr = reftable_block_source_from_file(&src, new_tab_name);\n@@ -1500,11 +1499,11 @@ int reftable_stack_clean(struct reftable_stack *st)\n int reftable_stack_print_directory(const char *stackdir, uint32_t hash_id)\n {\n \tstruct reftable_stack *stack = NULL;\n-\tstruct reftable_write_options cfg = { .hash_id = hash_id };\n+\tstruct reftable_write_options opts = { .hash_id = hash_id };\n \tstruct reftable_merged_table *merged = NULL;\n \tstruct reftable_table table = { NULL };\n \n-\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tint err = reftable_new_stack(&stack, stackdir, opts);\n \tif (err < 0)\n \t\tgoto done;\n \ndiff --git a/reftable/stack.h b/reftable/stack.h\nindex d43efa4760..97d7ebc043 100644\n--- a/reftable/stack.h\n+++ b/reftable/stack.h\n@@ -20,7 +20,7 @@ struct reftable_stack {\n \n \tchar *reftable_dir;\n \n-\tstruct reftable_write_options config;\n+\tstruct reftable_write_options opts;\n \n \tstruct reftable_reader **readers;\n \tsize_t readers_len;\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 1df3ffce52..3316d55f19 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -150,7 +150,7 @@ static void test_reftable_stack_add_one(void)\n \tchar *dir = get_tmp_dir(__LINE__);\n \tstruct strbuf scratch = STRBUF_INIT;\n \tint mask = umask(002);\n-\tstruct reftable_write_options cfg = {\n+\tstruct reftable_write_options opts = {\n \t\t.default_permissions = 0660,\n \t};\n \tstruct reftable_stack *st = NULL;\n@@ -163,7 +163,7 @@ static void test_reftable_stack_add_one(void)\n \t};\n \tstruct reftable_ref_record dest = { NULL };\n \tstruct stat stat_result = { 0 };\n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, &write_test_ref, &ref);\n@@ -186,7 +186,7 @@ static void test_reftable_stack_add_one(void)\n \tstrbuf_addstr(&scratch, \"/tables.list\");\n \terr = stat(scratch.buf, &stat_result);\n \tEXPECT(!err);\n-\tEXPECT((stat_result.st_mode & 0777) == cfg.default_permissions);\n+\tEXPECT((stat_result.st_mode & 0777) == opts.default_permissions);\n \n \tstrbuf_reset(&scratch);\n \tstrbuf_addstr(&scratch, dir);\n@@ -195,7 +195,7 @@ static void test_reftable_stack_add_one(void)\n \tstrbuf_addstr(&scratch, st->readers[0]->name);\n \terr = stat(scratch.buf, &stat_result);\n \tEXPECT(!err);\n-\tEXPECT((stat_result.st_mode & 0777) == cfg.default_permissions);\n+\tEXPECT((stat_result.st_mode & 0777) == opts.default_permissions);\n #else\n \t(void) stat_result;\n #endif\n@@ -209,7 +209,7 @@ static void test_reftable_stack_add_one(void)\n \n static void test_reftable_stack_uptodate(void)\n {\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st1 = NULL;\n \tstruct reftable_stack *st2 = NULL;\n \tchar *dir = get_tmp_dir(__LINE__);\n@@ -232,10 +232,10 @@ static void test_reftable_stack_uptodate(void)\n \t/* simulate multi-process access to the same stack\n \t by creating two stacks for the same directory.\n \t */\n-\terr = reftable_new_stack(&st1, dir, cfg);\n+\terr = reftable_new_stack(&st1, dir, opts);\n \tEXPECT_ERR(err);\n \n-\terr = reftable_new_stack(&st2, dir, cfg);\n+\terr = reftable_new_stack(&st2, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st1, &write_test_ref, &ref1);\n@@ -257,8 +257,7 @@ static void test_reftable_stack_uptodate(void)\n static void test_reftable_stack_transaction_api(void)\n {\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n \tstruct reftable_addition *add = NULL;\n@@ -271,8 +270,7 @@ static void test_reftable_stack_transaction_api(void)\n \t};\n \tstruct reftable_ref_record dest = { NULL };\n \n-\n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \treftable_addition_destroy(add);\n@@ -301,12 +299,12 @@ static void test_reftable_stack_transaction_api(void)\n static void test_reftable_stack_transaction_api_performs_auto_compaction(void)\n {\n \tchar *dir = get_tmp_dir(__LINE__);\n-\tstruct reftable_write_options cfg = {0};\n+\tstruct reftable_write_options opts = {0};\n \tstruct reftable_addition *add = NULL;\n \tstruct reftable_stack *st = NULL;\n \tint i, n = 20, err;\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i <= n; i++) {\n@@ -325,7 +323,7 @@ static void test_reftable_stack_transaction_api_performs_auto_compaction(void)\n \t\t * we can ensure that we indeed honor this setting and have\n \t\t * better control over when exactly auto compaction runs.\n \t\t */\n-\t\tst->config.disable_auto_compact = i != n;\n+\t\tst->opts.disable_auto_compact = i != n;\n \n \t\terr = reftable_stack_new_addition(&add, st);\n \t\tEXPECT_ERR(err);\n@@ -361,13 +359,13 @@ static void test_reftable_stack_auto_compaction_fails_gracefully(void)\n \t\t.value_type = REFTABLE_REF_VAL1,\n \t\t.value.val1 = {0x01},\n \t};\n-\tstruct reftable_write_options cfg = {0};\n+\tstruct reftable_write_options opts = {0};\n \tstruct reftable_stack *st;\n \tstruct strbuf table_path = STRBUF_INIT;\n \tchar *dir = get_tmp_dir(__LINE__);\n \tint err;\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, write_test_ref, &ref);\n@@ -442,8 +440,7 @@ static int write_error(struct reftable_writer *wr, void *arg)\n static void test_reftable_stack_update_index_check(void)\n {\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n \tstruct reftable_ref_record ref1 = {\n@@ -459,7 +456,7 @@ static void test_reftable_stack_update_index_check(void)\n \t\t.value.symref = \"master\",\n \t};\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, &write_test_ref, &ref1);\n@@ -474,12 +471,11 @@ static void test_reftable_stack_update_index_check(void)\n static void test_reftable_stack_lock_failure(void)\n {\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err, i;\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \tfor (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) {\n \t\terr = reftable_stack_add(st, &write_error, &i);\n@@ -494,7 +490,7 @@ static void test_reftable_stack_add(void)\n {\n \tint i = 0;\n \tint err = 0;\n-\tstruct reftable_write_options cfg = {\n+\tstruct reftable_write_options opts = {\n \t\t.exact_log_message = 1,\n \t\t.default_permissions = 0660,\n \t\t.disable_auto_compact = 1,\n@@ -507,7 +503,7 @@ static void test_reftable_stack_add(void)\n \tstruct stat stat_result;\n \tint N = ARRAY_SIZE(refs);\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i < N; i++) {\n@@ -566,7 +562,7 @@ static void test_reftable_stack_add(void)\n \tstrbuf_addstr(&path, \"/tables.list\");\n \terr = stat(path.buf, &stat_result);\n \tEXPECT(!err);\n-\tEXPECT((stat_result.st_mode & 0777) == cfg.default_permissions);\n+\tEXPECT((stat_result.st_mode & 0777) == opts.default_permissions);\n \n \tstrbuf_reset(&path);\n \tstrbuf_addstr(&path, dir);\n@@ -575,7 +571,7 @@ static void test_reftable_stack_add(void)\n \tstrbuf_addstr(&path, st->readers[0]->name);\n \terr = stat(path.buf, &stat_result);\n \tEXPECT(!err);\n-\tEXPECT((stat_result.st_mode & 0777) == cfg.default_permissions);\n+\tEXPECT((stat_result.st_mode & 0777) == opts.default_permissions);\n #else\n \t(void) stat_result;\n #endif\n@@ -593,7 +589,7 @@ static void test_reftable_stack_add(void)\n static void test_reftable_stack_log_normalize(void)\n {\n \tint err = 0;\n-\tstruct reftable_write_options cfg = {\n+\tstruct reftable_write_options opts = {\n \t\t0,\n \t};\n \tstruct reftable_stack *st = NULL;\n@@ -617,7 +613,7 @@ static void test_reftable_stack_log_normalize(void)\n \t\t.update_index = 1,\n \t};\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \tinput.value.update.message = \"one\\ntwo\";\n@@ -650,8 +646,7 @@ static void test_reftable_stack_tombstone(void)\n {\n \tint i = 0;\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n \tstruct reftable_ref_record refs[2] = { { NULL } };\n@@ -660,8 +655,7 @@ static void test_reftable_stack_tombstone(void)\n \tstruct reftable_ref_record dest = { NULL };\n \tstruct reftable_log_record log_dest = { NULL };\n \n-\n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \t/* even entries add the refs, odd entries delete them. */\n@@ -729,8 +723,7 @@ static void test_reftable_stack_tombstone(void)\n static void test_reftable_stack_hash_id(void)\n {\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n \n@@ -740,24 +733,24 @@ static void test_reftable_stack_hash_id(void)\n \t\t.value.symref = \"target\",\n \t\t.update_index = 1,\n \t};\n-\tstruct reftable_write_options cfg32 = { .hash_id = GIT_SHA256_FORMAT_ID };\n+\tstruct reftable_write_options opts32 = { .hash_id = GIT_SHA256_FORMAT_ID };\n \tstruct reftable_stack *st32 = NULL;\n-\tstruct reftable_write_options cfg_default = { 0 };\n+\tstruct reftable_write_options opts_default = { 0 };\n \tstruct reftable_stack *st_default = NULL;\n \tstruct reftable_ref_record dest = { NULL };\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, &write_test_ref, &ref);\n \tEXPECT_ERR(err);\n \n \t/* can't read it with the wrong hash ID. */\n-\terr = reftable_new_stack(&st32, dir, cfg32);\n+\terr = reftable_new_stack(&st32, dir, opts32);\n \tEXPECT(err == REFTABLE_FORMAT_ERROR);\n \n-\t/* check that we can read it back with default config too. */\n-\terr = reftable_new_stack(&st_default, dir, cfg_default);\n+\t/* check that we can read it back with default opts too. */\n+\terr = reftable_new_stack(&st_default, dir, opts_default);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n@@ -790,8 +783,7 @@ static void test_suggest_compaction_segment_nothing(void)\n static void test_reflog_expire(void)\n {\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tstruct reftable_log_record logs[20] = { { NULL } };\n \tint N = ARRAY_SIZE(logs) - 1;\n@@ -802,8 +794,7 @@ static void test_reflog_expire(void)\n \t};\n \tstruct reftable_log_record log = { NULL };\n \n-\n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 1; i <= N; i++) {\n@@ -866,21 +857,19 @@ static int write_nothing(struct reftable_writer *wr, void *arg)\n \n static void test_empty_add(void)\n {\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n \tstruct reftable_stack *st2 = NULL;\n \n-\n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, &write_nothing, NULL);\n \tEXPECT_ERR(err);\n \n-\terr = reftable_new_stack(&st2, dir, cfg);\n+\terr = reftable_new_stack(&st2, dir, opts);\n \tEXPECT_ERR(err);\n \tclear_dir(dir);\n \treftable_stack_destroy(st);\n@@ -899,16 +888,15 @@ static int fastlog2(uint64_t sz)\n \n static void test_reftable_stack_auto_compaction(void)\n {\n-\tstruct reftable_write_options cfg = {\n+\tstruct reftable_write_options opts = {\n \t\t.disable_auto_compact = 1,\n \t};\n \tstruct reftable_stack *st = NULL;\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n \tint err, i;\n \tint N = 100;\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i < N; i++) {\n@@ -938,13 +926,13 @@ static void test_reftable_stack_auto_compaction(void)\n \n static void test_reftable_stack_add_performs_auto_compaction(void)\n {\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tstruct strbuf refname = STRBUF_INIT;\n \tchar *dir = get_tmp_dir(__LINE__);\n \tint err, i, n = 20;\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i <= n; i++) {\n@@ -959,7 +947,7 @@ static void test_reftable_stack_add_performs_auto_compaction(void)\n \t\t * we can ensure that we indeed honor this setting and have\n \t\t * better control over when exactly auto compaction runs.\n \t\t */\n-\t\tst->config.disable_auto_compact = i != n;\n+\t\tst->opts.disable_auto_compact = i != n;\n \n \t\tstrbuf_reset(&refname);\n \t\tstrbuf_addf(&refname, \"branch-%04d\", i);\n@@ -986,14 +974,13 @@ static void test_reftable_stack_add_performs_auto_compaction(void)\n \n static void test_reftable_stack_compaction_concurrent(void)\n {\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st1 = NULL, *st2 = NULL;\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n \tint err, i;\n \tint N = 3;\n \n-\terr = reftable_new_stack(&st1, dir, cfg);\n+\terr = reftable_new_stack(&st1, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i < N; i++) {\n@@ -1010,7 +997,7 @@ static void test_reftable_stack_compaction_concurrent(void)\n \t\tEXPECT_ERR(err);\n \t}\n \n-\terr = reftable_new_stack(&st2, dir, cfg);\n+\terr = reftable_new_stack(&st2, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_compact_all(st1, NULL);\n@@ -1036,14 +1023,13 @@ static void unclean_stack_close(struct reftable_stack *st)\n \n static void test_reftable_stack_compaction_concurrent_clean(void)\n {\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st1 = NULL, *st2 = NULL, *st3 = NULL;\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n \tint err, i;\n \tint N = 3;\n \n-\terr = reftable_new_stack(&st1, dir, cfg);\n+\terr = reftable_new_stack(&st1, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i < N; i++) {\n@@ -1060,7 +1046,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void)\n \t\tEXPECT_ERR(err);\n \t}\n \n-\terr = reftable_new_stack(&st2, dir, cfg);\n+\terr = reftable_new_stack(&st2, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_compact_all(st1, NULL);\n@@ -1069,7 +1055,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void)\n \tunclean_stack_close(st1);\n \tunclean_stack_close(st2);\n \n-\terr = reftable_new_stack(&st3, dir, cfg);\n+\terr = reftable_new_stack(&st3, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_clean(st3);\n-- \n2.45.0\n\n"},{"id":"494428","messageId":"e6f8fc09c282e620a331a737c30a190107c76996.1715336798.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715336797.git.ps@pks.im","subject":"[PATCH v2 02/11] reftable: consistently pass write opts as value","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T10:29:25Z","receivedAt":"2024-05-10T10:29:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We sometimes pass the refatble write options as value and sometimes as a\npointer. This is quite confusing and makes the reader wonder whether the\noptions get modified sometimes.\n\nIn fact, `reftable_new_writer()` does cause the caller-provided options\nto get updated when some values aren't set up. This is quite unexpected,\nbut didn't cause any harm until now.\n\nRefactor the code to consistently pass the options as a value so that\nthey are local to the subsystem they are being passed into so that we\ncan avoid weirdness like this.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged_test.c | 6 +++---\n reftable/readwrite_test.c | 26 +++++++++++++-------------\n reftable/refname_test.c | 2 +-\n reftable/reftable-writer.h | 2 +-\n reftable/stack.c | 4 ++--\n reftable/writer.c | 12 +++++++-----\n 6 files changed, 27 insertions(+), 25 deletions(-)\n\ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nindex 530fc82d1c..4ac81de8d4 100644\n--- a/reftable/merged_test.c\n+++ b/reftable/merged_test.c\n@@ -42,7 +42,7 @@ static void write_test_table(struct strbuf *buf,\n \t\t}\n \t}\n \n-\tw = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, &opts);\n+\tw = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, opts);\n \treftable_writer_set_limits(w, min, max);\n \n \tfor (i = 0; i < n; i++) {\n@@ -70,7 +70,7 @@ static void write_test_log_table(struct strbuf *buf,\n \t\t.exact_log_message = 1,\n \t};\n \tstruct reftable_writer *w = NULL;\n-\tw = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, &opts);\n+\tw = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, opts);\n \treftable_writer_set_limits(w, update_index, update_index);\n \n \tfor (i = 0; i < n; i++) {\n@@ -403,7 +403,7 @@ static void test_default_write_opts(void)\n \tstruct reftable_write_options opts = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \n \tstruct reftable_ref_record rec = {\n \t\t.refname = \"master\",\ndiff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c\nindex a6dbd214c5..27631a041b 100644\n--- a/reftable/readwrite_test.c\n+++ b/reftable/readwrite_test.c\n@@ -51,7 +51,7 @@ static void write_table(char ***names, struct strbuf *buf, int N,\n \t\t.hash_id = hash_id,\n \t};\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, buf, opts);\n \tstruct reftable_ref_record ref = { NULL };\n \tint i = 0, n;\n \tstruct reftable_log_record log = { NULL };\n@@ -129,7 +129,7 @@ static void test_log_buffer_size(void)\n \t\t\t\t\t .message = \"commit: 9\\n\",\n \t\t\t\t } } };\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \n \t/* This tests buffer extension for log compression. Must use a random\n \t hash, to ensure that the compressed part is larger than the original.\n@@ -172,7 +172,7 @@ static void test_log_overflow(void)\n \t\t},\n \t};\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \n \tmemset(msg, 'x', sizeof(msg) - 1);\n \treftable_writer_set_limits(w, update_index, update_index);\n@@ -199,7 +199,7 @@ static void test_log_write_read(void)\n \tstruct reftable_block_source source = { NULL };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \tconst struct reftable_stats *stats = NULL;\n \treftable_writer_set_limits(w, 0, N);\n \tfor (i = 0; i < N; i++) {\n@@ -288,7 +288,7 @@ static void test_log_zlib_corruption(void)\n \tstruct reftable_block_source source = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \tconst struct reftable_stats *stats = NULL;\n \tchar message[100] = { 0 };\n \tint err, i, n;\n@@ -526,7 +526,7 @@ static void test_table_refs_for(int indexed)\n \n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \n \tstruct reftable_iterator it = { NULL };\n \tint j;\n@@ -619,7 +619,7 @@ static void test_write_empty_table(void)\n \tstruct reftable_write_options opts = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \tstruct reftable_block_source source = { NULL };\n \tstruct reftable_reader *rd = NULL;\n \tstruct reftable_ref_record rec = { NULL };\n@@ -657,7 +657,7 @@ static void test_write_object_id_min_length(void)\n \t};\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \tstruct reftable_ref_record ref = {\n \t\t.update_index = 1,\n \t\t.value_type = REFTABLE_REF_VAL1,\n@@ -692,7 +692,7 @@ static void test_write_object_id_length(void)\n \t};\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \tstruct reftable_ref_record ref = {\n \t\t.update_index = 1,\n \t\t.value_type = REFTABLE_REF_VAL1,\n@@ -726,7 +726,7 @@ static void test_write_empty_key(void)\n \tstruct reftable_write_options opts = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \tstruct reftable_ref_record ref = {\n \t\t.refname = \"\",\n \t\t.update_index = 1,\n@@ -749,7 +749,7 @@ static void test_write_key_order(void)\n \tstruct reftable_write_options opts = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \tstruct reftable_ref_record refs[2] = {\n \t\t{\n \t\t\t.refname = \"b\",\n@@ -792,7 +792,7 @@ static void test_write_multiple_indices(void)\n \tstruct reftable_reader *reader;\n \tint err, i;\n \n-\twriter = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, &opts);\n+\twriter = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, opts);\n \treftable_writer_set_limits(writer, 1, 1);\n \tfor (i = 0; i < 100; i++) {\n \t\tstruct reftable_ref_record ref = {\n@@ -869,7 +869,7 @@ static void test_write_multi_level_index(void)\n \tstruct reftable_reader *reader;\n \tint err;\n \n-\twriter = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, &opts);\n+\twriter = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, opts);\n \treftable_writer_set_limits(writer, 1, 1);\n \tfor (size_t i = 0; i < 200; i++) {\n \t\tstruct reftable_ref_record ref = {\ndiff --git a/reftable/refname_test.c b/reftable/refname_test.c\nindex b9cc62554e..3468253be7 100644\n--- a/reftable/refname_test.c\n+++ b/reftable/refname_test.c\n@@ -30,7 +30,7 @@ static void test_conflict(void)\n \tstruct reftable_write_options opts = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w =\n-\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n \tstruct reftable_ref_record rec = {\n \t\t.refname = \"a/b\",\n \t\t.value_type = REFTABLE_REF_SYMREF,\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 155bf0bbe2..44cb986465 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -92,7 +92,7 @@ struct reftable_stats {\n struct reftable_writer *\n reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\t int (*flush_func)(void *),\n-\t\t void *writer_arg, struct reftable_write_options *opts);\n+\t\t void *writer_arg, struct reftable_write_options opts);\n \n /* Set the range of update indices for the records we will add. When writing a\n table into a stack, the min should be at least\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 3979657793..7b4fff7c9e 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -767,7 +767,7 @@ int reftable_addition_add(struct reftable_addition *add,\n \ttab_fd = get_tempfile_fd(tab_file);\n \n \twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, &tab_fd,\n-\t\t\t\t &add->stack->opts);\n+\t\t\t\t add->stack->opts);\n \terr = write_table(wr, arg);\n \tif (err < 0)\n \t\tgoto done;\n@@ -861,7 +861,7 @@ static int stack_compact_locked(struct reftable_stack *st,\n \t}\n \n \twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush,\n-\t\t\t\t &tab_fd, &st->opts);\n+\t\t\t\t &tab_fd, st->opts);\n \terr = stack_write_compact(st, wr, first, last, config);\n \tif (err < 0)\n \t\tgoto done;\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 1d9ff0fbfa..ad2f2e6c65 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -122,20 +122,22 @@ static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n struct reftable_writer *\n reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\t int (*flush_func)(void *),\n-\t\t void *writer_arg, struct reftable_write_options *opts)\n+\t\t void *writer_arg, struct reftable_write_options opts)\n {\n \tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n \tstrbuf_init(&wp->block_writer_data.last_key, 0);\n-\toptions_set_defaults(opts);\n-\tif (opts->block_size >= (1 << 24)) {\n+\n+\toptions_set_defaults(&opts);\n+\tif (opts.block_size >= (1 << 24)) {\n \t\t/* TODO - error return? */\n \t\tabort();\n \t}\n+\n \twp->last_key = reftable_empty_strbuf;\n-\tREFTABLE_CALLOC_ARRAY(wp->block, opts->block_size);\n+\tREFTABLE_CALLOC_ARRAY(wp->block, opts.block_size);\n \twp->write = writer_func;\n \twp->write_arg = writer_arg;\n-\twp->opts = *opts;\n+\twp->opts = opts;\n \twp->flush = flush_func;\n \twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n \n-- \n2.45.0\n\n"},{"id":"494429","messageId":"aa2903e3e5ac76f82252e12919a8a09178e267ee.1715336798.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715336797.git.ps@pks.im","subject":"[PATCH v2 03/11] reftable/writer: drop static variable used to initialize strbuf","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T10:29:30Z","receivedAt":"2024-05-10T10:29:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have a static variable in the reftable writer code that is merely\nused to initialize the `last_key` of the writer. Convert the code to\ninstead use `strbuf_init()` and drop the variable.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/writer.c | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex ad2f2e6c65..7df6e53699 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -117,15 +117,12 @@ static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n \tw->block_writer->restart_interval = w->opts.restart_interval;\n }\n \n-static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n-\n struct reftable_writer *\n reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\t int (*flush_func)(void *),\n \t\t void *writer_arg, struct reftable_write_options opts)\n {\n \tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n-\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n \n \toptions_set_defaults(&opts);\n \tif (opts.block_size >= (1 << 24)) {\n@@ -133,7 +130,8 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\tabort();\n \t}\n \n-\twp->last_key = reftable_empty_strbuf;\n+\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n+\tstrbuf_init(&wp->last_key, 0);\n \tREFTABLE_CALLOC_ARRAY(wp->block, opts.block_size);\n \twp->write = writer_func;\n \twp->write_arg = writer_arg;\n-- \n2.45.0\n\n"},{"id":"494430","messageId":"5e7cbb7b193c578f7c946a5077a79421b0ac57f2.1715336798.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715336797.git.ps@pks.im","subject":"[PATCH v2 04/11] reftable/writer: improve error when passed an invalid block size","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T10:29:35Z","receivedAt":"2024-05-10T10:29:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable format only supports block sizes up to 16MB. When the\nwriter is being passed a value bigger than that it simply calls\nabort(3P), which isn't all that helpful due to the lack of a proper\nerror message.\n\nImprove this by calling `BUG()` instead.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/writer.c | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 7df6e53699..374b7d15ed 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -125,10 +125,8 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n \n \toptions_set_defaults(&opts);\n-\tif (opts.block_size >= (1 << 24)) {\n-\t\t/* TODO - error return? */\n-\t\tabort();\n-\t}\n+\tif (opts.block_size >= (1 << 24))\n+\t\tBUG(\"configured block size exceeds 16MB\");\n \n \tstrbuf_init(&wp->block_writer_data.last_key, 0);\n \tstrbuf_init(&wp->last_key, 0);\n-- \n2.45.0\n\n"},{"id":"494431","messageId":"ed1c150d909da836a760bc0699f585722d5988e9.1715336798.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715336797.git.ps@pks.im","subject":"[PATCH v2 05/11] reftable/dump: support dumping a table's block structure","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T10:29:40Z","receivedAt":"2024-05-10T10:29:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're about to introduce new configs that will allow users to have more\ncontrol over how exactly reftables are written. To verify that these\nconfigs are effective we will need to take a peak into the actual blocks\nwritten by the reftable backend.\n\nIntroduce a new mode to the dumping logic that prints out the block\nstructure. This logic can be invoked via `test-tool dump-reftables -b`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/dump.c | 8 ++-\n reftable/reader.c | 63 ++++++++++++++++++\n reftable/reftable-reader.h | 2 +\n t/t0613-reftable-write-options.sh | 102 ++++++++++++++++++++++++++++++\n 4 files changed, 174 insertions(+), 1 deletion(-)\n create mode 100755 t/t0613-reftable-write-options.sh\n\ndiff --git a/reftable/dump.c b/reftable/dump.c\nindex 9c770a10cc..24476cc2a9 100644\n--- a/reftable/dump.c\n+++ b/reftable/dump.c\n@@ -48,6 +48,7 @@ static void print_help(void)\n \tprintf(\"usage: dump [-cst] arg\\n\\n\"\n \t \"options: \\n\"\n \t \" -c compact\\n\"\n+\t \" -b dump blocks\\n\"\n \t \" -t dump table\\n\"\n \t \" -s dump stack\\n\"\n \t \" -6 sha256 hash format\\n\"\n@@ -58,6 +59,7 @@ static void print_help(void)\n int reftable_dump_main(int argc, char *const *argv)\n {\n \tint err = 0;\n+\tint opt_dump_blocks = 0;\n \tint opt_dump_table = 0;\n \tint opt_dump_stack = 0;\n \tint opt_compact = 0;\n@@ -67,6 +69,8 @@ int reftable_dump_main(int argc, char *const *argv)\n \tfor (; argc > 1; argv++, argc--)\n \t\tif (*argv[1] != '-')\n \t\t\tbreak;\n+\t\telse if (!strcmp(\"-b\", argv[1]))\n+\t\t\topt_dump_blocks = 1;\n \t\telse if (!strcmp(\"-t\", argv[1]))\n \t\t\topt_dump_table = 1;\n \t\telse if (!strcmp(\"-6\", argv[1]))\n@@ -88,7 +92,9 @@ int reftable_dump_main(int argc, char *const *argv)\n \n \targ = argv[1];\n \n-\tif (opt_dump_table) {\n+\tif (opt_dump_blocks) {\n+\t\terr = reftable_reader_print_blocks(arg);\n+\t} else if (opt_dump_table) {\n \t\terr = reftable_reader_print_file(arg);\n \t} else if (opt_dump_stack) {\n \t\terr = reftable_stack_print_directory(arg, opt_hash_id);\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 481dff10d4..f23c8523db 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -856,3 +856,66 @@ int reftable_reader_print_file(const char *tablename)\n \treftable_reader_free(r);\n \treturn err;\n }\n+\n+int reftable_reader_print_blocks(const char *tablename)\n+{\n+\tstruct {\n+\t\tconst char *name;\n+\t\tint type;\n+\t} sections[] = {\n+\t\t{\n+\t\t\t.name = \"ref\",\n+\t\t\t.type = BLOCK_TYPE_REF,\n+\t\t},\n+\t\t{\n+\t\t\t.name = \"obj\",\n+\t\t\t.type = BLOCK_TYPE_OBJ,\n+\t\t},\n+\t\t{\n+\t\t\t.name = \"log\",\n+\t\t\t.type = BLOCK_TYPE_LOG,\n+\t\t},\n+\t};\n+\tstruct reftable_block_source src = { 0 };\n+\tstruct table_iter ti = TABLE_ITER_INIT;\n+\tstruct reftable_reader *r = NULL;\n+\tsize_t i;\n+\tint err;\n+\n+\terr = reftable_block_source_from_file(&src, tablename);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&r, &src, tablename);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tprintf(\"header:\\n\");\n+\tprintf(\" block_size: %d\\n\", r->block_size);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(sections); i++) {\n+\t\terr = reader_start(r, &ti, sections[i].type, 0);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t\tif (err > 0)\n+\t\t\tcontinue;\n+\n+\t\tprintf(\"%s:\\n\", sections[i].name);\n+\n+\t\twhile (1) {\n+\t\t\tprintf(\" - length: %u\\n\", ti.br.block_len);\n+\t\t\tprintf(\" restarts: %u\\n\", ti.br.restart_count);\n+\n+\t\t\terr = table_iter_next_block(&ti);\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t\tif (err > 0)\n+\t\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+done:\n+\treftable_reader_free(r);\n+\ttable_iter_close(&ti);\n+\treturn err;\n+}\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nindex 4a4bc2fdf8..4a04857773 100644\n--- a/reftable/reftable-reader.h\n+++ b/reftable/reftable-reader.h\n@@ -97,5 +97,7 @@ void reftable_table_from_reader(struct reftable_table *tab,\n \n /* print table onto stdout for debugging. */\n int reftable_reader_print_file(const char *tablename);\n+/* print blocks onto stdout for debugging. */\n+int reftable_reader_print_blocks(const char *tablename);\n \n #endif\ndiff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh\nnew file mode 100755\nindex 0000000000..462980c37c\n--- /dev/null\n+++ b/t/t0613-reftable-write-options.sh\n@@ -0,0 +1,102 @@\n+#!/bin/sh\n+\n+test_description='reftable write options'\n+\n+GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+# Disable auto-compaction for all tests as we explicitly control repacking of\n+# refs.\n+GIT_TEST_REFTABLE_AUTOCOMPACTION=false\n+export GIT_TEST_REFTABLE_AUTOCOMPACTION\n+# Block sizes depend on the hash function, so we force SHA1 here.\n+GIT_TEST_DEFAULT_HASH=sha1\n+export GIT_TEST_DEFAULT_HASH\n+# Block sizes also depend on the actual refs we write, so we force \"master\" to\n+# be the default initial branch name.\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'default write options' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tgit pack-refs &&\n+\t\tcat >expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 4096\n+\t\tref:\n+\t\t - length: 129\n+\t\t restarts: 2\n+\t\tlog:\n+\t\t - length: 262\n+\t\t restarts: 2\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'disabled reflog writes no log blocks' '\n+\ttest_config_global core.logAllRefUpdates false &&\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tgit pack-refs &&\n+\t\tcat >expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 4096\n+\t\tref:\n+\t\t - length: 129\n+\t\t restarts: 2\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'many refs results in multiple blocks' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tfor i in $(test_seq 200)\n+\t\tdo\n+\t\t\tprintf \"update refs/heads/branch-%d HEAD\\n\" \"$i\" ||\n+\t\t\treturn 1\n+\t\tdone >input &&\n+\t\tgit update-ref --stdin expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 4096\n+\t\tref:\n+\t\t - length: 4049\n+\t\t restarts: 11\n+\t\t - length: 1136\n+\t\t restarts: 3\n+\t\tlog:\n+\t\t - length: 4041\n+\t\t restarts: 4\n+\t\t - length: 4015\n+\t\t restarts: 3\n+\t\t - length: 4014\n+\t\t restarts: 3\n+\t\t - length: 4012\n+\t\t restarts: 3\n+\t\t - length: 3289\n+\t\t restarts: 3\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_done\n-- \n2.45.0\n\n"},{"id":"494432","messageId":"be5bdc6dc13510890dbc9ed06e2de6dd866c29e5.1715336798.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715336797.git.ps@pks.im","subject":"[PATCH v2 06/11] refs/reftable: allow configuring block size","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T10:29:45Z","receivedAt":"2024-05-10T10:29:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Add a new option `reftable.blockSize` that allows the user to control\nthe block size used by the reftable library.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config.txt | 2 +\n Documentation/config/reftable.txt | 14 ++++++\n refs/reftable-backend.c | 31 ++++++++++++-\n t/t0613-reftable-write-options.sh | 72 +++++++++++++++++++++++++++++++\n 4 files changed, 118 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/config/reftable.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 70b448b132..fa1469e5e7 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -497,6 +497,8 @@ include::config/rebase.txt[]\n \n include::config/receive.txt[]\n \n+include::config/reftable.txt[]\n+\n include::config/remote.txt[]\n \n include::config/remotes.txt[]\ndiff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt\nnew file mode 100644\nindex 0000000000..fa7c4be014\n--- /dev/null\n+++ b/Documentation/config/reftable.txt\n@@ -0,0 +1,14 @@\n+reftable.blockSize::\n+\tThe size in bytes used by the reftable backend when writing blocks.\n+\tThe block size is determined by the writer, and does not have to be a\n+\tpower of 2. The block size must be larger than the longest reference\n+\tname or log entry used in the repository, as references cannot span\n+\tblocks.\n++\n+Powers of two that are friendly to the virtual memory system or\n+filesystem (such as 4kB or 8kB) are recommended. Larger sizes (64kB) can\n+yield better compression, with a possible increased cost incurred by\n+readers during access.\n++\n+The largest block size is `16777215` bytes (15.99 MiB). The default value is\n+`4096` bytes (4kB). A value of `0` will use the default value.\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 1cda48c504..bd9999cefc 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1,6 +1,7 @@\n #include \"../git-compat-util.h\"\n #include \"../abspath.h\"\n #include \"../chdir-notify.h\"\n+#include \"../config.h\"\n #include \"../environment.h\"\n #include \"../gettext.h\"\n #include \"../hash.h\"\n@@ -230,6 +231,22 @@ static int read_ref_without_reload(struct reftable_stack *stack,\n \treturn ret;\n }\n \n+static int reftable_be_config(const char *var, const char *value,\n+\t\t\t const struct config_context *ctx,\n+\t\t\t void *_opts)\n+{\n+\tstruct reftable_write_options *opts = _opts;\n+\n+\tif (!strcmp(var, \"reftable.blocksize\")) {\n+\t\tunsigned long block_size = git_config_ulong(var, value, ctx->kvi);\n+\t\tif (block_size > 16777215)\n+\t\t\tdie(\"reftable block size cannot exceed 16MB\");\n+\t\topts->block_size = block_size;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static struct ref_store *reftable_be_init(struct repository *repo,\n \t\t\t\t\t const char *gitdir,\n \t\t\t\t\t unsigned int store_flags)\n@@ -245,12 +262,24 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \tbase_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);\n \tstrmap_init(&refs->worktree_stacks);\n \trefs->store_flags = store_flags;\n-\trefs->write_options.block_size = 4096;\n+\n \trefs->write_options.hash_id = repo->hash_algo->format_id;\n \trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n \trefs->write_options.disable_auto_compact =\n \t\t!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1);\n \n+\tgit_config(reftable_be_config, &refs->write_options);\n+\n+\t/*\n+\t * It is somewhat unfortunate that we have to mirror the default block\n+\t * size of the reftable library here. But given that the write options\n+\t * wouldn't be updated by the library here, and given that we require\n+\t * the proper block size to trim reflog message so that they fit, we\n+\t * must set up a proper value here.\n+\t */\n+\tif (!refs->write_options.block_size)\n+\t\trefs->write_options.block_size = 4096;\n+\n \t/*\n \t * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.\n \t * This stack contains both the shared and the main worktree refs.\ndiff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh\nindex 462980c37c..8bdbc6ec70 100755\n--- a/t/t0613-reftable-write-options.sh\n+++ b/t/t0613-reftable-write-options.sh\n@@ -99,4 +99,76 @@ test_expect_success 'many refs results in multiple blocks' '\n \t)\n '\n \n+test_expect_success 'tiny block size leads to error' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: unable to compact stack: entry too large\n+\t\tEOF\n+\t\ttest_must_fail git -c reftable.blockSize=50 pack-refs 2>err &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'small block size leads to multiple ref blocks' '\n+\ttest_config_global core.logAllRefUpdates false &&\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\t\tgit -c reftable.blockSize=100 pack-refs &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 100\n+\t\tref:\n+\t\t - length: 53\n+\t\t restarts: 1\n+\t\t - length: 74\n+\t\t restarts: 1\n+\t\t - length: 38\n+\t\t restarts: 1\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'small block size fails with large reflog message' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit A &&\n+\t\tperl -e \"print \\\"a\\\" x 500\" >logmsg &&\n+\t\tcat >expect <<-EOF &&\n+\t\tfatal: update_ref failed for ref ${SQ}refs/heads/logme${SQ}: reftable: transaction failure: entry too large\n+\t\tEOF\n+\t\ttest_must_fail git -c reftable.blockSize=100 \\\n+\t\t\tupdate-ref -m \"$(cat logmsg)\" refs/heads/logme HEAD 2>err &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'block size exceeding maximum supported size' '\n+\ttest_config_global core.logAllRefUpdates false &&\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\t\tcat >expect <<-EOF &&\n+\t\tfatal: reftable block size cannot exceed 16MB\n+\t\tEOF\n+\t\ttest_must_fail git -c reftable.blockSize=16777216 pack-refs 2>err &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n-- \n2.45.0\n\n"},{"id":"494433","messageId":"05e8d1df2d9921426b803d71b22ba3dba188836c.1715336798.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715336797.git.ps@pks.im","subject":"[PATCH v2 07/11] reftable: use `uint16_t` to track restart interval","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T10:29:51Z","receivedAt":"2024-05-10T10:29:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The restart interval can at most be `UINT16_MAX` as specified in the\ntechnical documentation of the reftable format. Furthermore, it cannot\never be negative. Regardless of that we use an `int` to track the\nrestart interval.\n\nChange the type to use an `uint16_t` instead.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/block.h | 2 +-\n reftable/reftable-writer.h | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/reftable/block.h b/reftable/block.h\nindex ea4384a7e2..cd5577105d 100644\n--- a/reftable/block.h\n+++ b/reftable/block.h\n@@ -25,7 +25,7 @@ struct block_writer {\n \tuint32_t header_off;\n \n \t/* How often to restart keys. */\n-\tint restart_interval;\n+\tuint16_t restart_interval;\n \tint hash_size;\n \n \t/* Offset of next uint8_t to write. */\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 44cb986465..4cd8ebe6c7 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -28,7 +28,7 @@ struct reftable_write_options {\n \tunsigned skip_index_objects : 1;\n \n \t/* how often to write complete keys in each block. */\n-\tint restart_interval;\n+\tuint16_t restart_interval;\n \n \t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n \t * Defaults to SHA1 if unset\n-- \n2.45.0\n\n"},{"id":"494434","messageId":"bc0bf65553c8dd89bf5fcaa592fc3427507f1993.1715336798.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715336797.git.ps@pks.im","subject":"[PATCH v2 08/11] refs/reftable: allow configuring restart interval","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T10:29:55Z","receivedAt":"2024-05-10T10:30:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Add a new option `reftable.restartInterval` that allows the user to\ncontrol the restart interval when writing reftable records used by the\nreftable library.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config/reftable.txt | 19 ++++++++++++++\n refs/reftable-backend.c | 5 ++++\n t/t0613-reftable-write-options.sh | 43 +++++++++++++++++++++++++++++++\n 3 files changed, 67 insertions(+)\n\ndiff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt\nindex fa7c4be014..16b915c75e 100644\n--- a/Documentation/config/reftable.txt\n+++ b/Documentation/config/reftable.txt\n@@ -12,3 +12,22 @@ readers during access.\n +\n The largest block size is `16777215` bytes (15.99 MiB). The default value is\n `4096` bytes (4kB). A value of `0` will use the default value.\n+\n+reftable.restartInterval::\n+\tThe interval at which to create restart points. The reftable backend\n+\tdetermines the restart points at file creation. The process is\n+\tarbitrary, but every 16 or 64 records is recommended. Every 16 may be\n+\tmore suitable for smaller block sizes (4k or 8k), every 64 for larger\n+\tblock sizes (64k).\n++\n+More frequent restart points reduces prefix compression and increases\n+space consumed by the restart table, both of which increase file size.\n++\n+Less frequent restart points makes prefix compression more effective,\n+decreasing overall file size, with increased penalties for readers\n+walking through more records after the binary search step.\n++\n+A maximum of `65535` restart points per block is supported.\n++\n+The default value is to create restart points every 16 records. A value of `0`\n+will use the default value.\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex bd9999cefc..9972dfc1a3 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -242,6 +242,11 @@ static int reftable_be_config(const char *var, const char *value,\n \t\tif (block_size > 16777215)\n \t\t\tdie(\"reftable block size cannot exceed 16MB\");\n \t\topts->block_size = block_size;\n+\t} else if (!strcmp(var, \"reftable.restartinterval\")) {\n+\t\tunsigned long restart_interval = git_config_ulong(var, value, ctx->kvi);\n+\t\tif (restart_interval > UINT16_MAX)\n+\t\t\tdie(\"reftable block size cannot exceed %u\", (unsigned)UINT16_MAX);\n+\t\topts->restart_interval = restart_interval;\n \t}\n \n \treturn 0;\ndiff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh\nindex 8bdbc6ec70..e0a5b26f58 100755\n--- a/t/t0613-reftable-write-options.sh\n+++ b/t/t0613-reftable-write-options.sh\n@@ -171,4 +171,47 @@ test_expect_success 'block size exceeding maximum supported size' '\n \t)\n '\n \n+test_expect_success 'restart interval at every single record' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tfor i in $(test_seq 10)\n+\t\tdo\n+\t\t\tprintf \"update refs/heads/branch-%d HEAD\\n\" \"$i\" ||\n+\t\t\treturn 1\n+\t\tdone >input &&\n+\t\tgit update-ref --stdin expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 4096\n+\t\tref:\n+\t\t - length: 566\n+\t\t restarts: 13\n+\t\tlog:\n+\t\t - length: 1393\n+\t\t restarts: 12\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'restart interval exceeding maximum supported interval' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tcat >expect <<-EOF &&\n+\t\tfatal: reftable block size cannot exceed 65535\n+\t\tEOF\n+\t\ttest_must_fail git -c reftable.restartInterval=65536 pack-refs 2>err &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n-- \n2.45.0\n\n"},{"id":"494435","messageId":"6bc240fd0cb99d888e5d19423a53a475b6a455ca.1715336798.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715336797.git.ps@pks.im","subject":"[PATCH v2 09/11] refs/reftable: allow disabling writing the object index","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T10:30:00Z","receivedAt":"2024-05-10T10:30:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Besides the expected \"ref\" and \"log\" records, the reftable library also\nwrites \"obj\" records. These are basically a reverse mapping of object\nIDs to their respective ref records so that it becomes efficient to\nfigure out which references point to a specific object. The motivation\nfor this data structure is the \"uploadpack.allowTipSHA1InWant\" config,\nwhich allows a client to fetch any object by its hash that has a ref\npointing to it.\n\nThis reverse index is not used by Git at all though, and the expectation\nis that most hosters nowadays use \"uploadpack.allowAnySHA1InWant\". It\nmay thus be preferable for many users to disable writing these optional\nobject indices altogether to safe some precious disk space.\n\nAdd a new config \"reftable.indexObjects\" that allows the user to disable\nthe object index altogether.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config/reftable.txt | 6 +++\n refs/reftable-backend.c | 2 +\n t/t0613-reftable-write-options.sh | 69 +++++++++++++++++++++++++++++++\n 3 files changed, 77 insertions(+)\n\ndiff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt\nindex 16b915c75e..6e4466f3c5 100644\n--- a/Documentation/config/reftable.txt\n+++ b/Documentation/config/reftable.txt\n@@ -31,3 +31,9 @@ A maximum of `65535` restart points per block is supported.\n +\n The default value is to create restart points every 16 records. A value of `0`\n will use the default value.\n+\n+reftable.indexObjects::\n+\tWhether the reftable backend shall write object blocks. Object blocks\n+\tare a reverse mapping of object ID to the references pointing to them.\n++\n+The default value is `true`.\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 9972dfc1a3..63b75f770d 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -247,6 +247,8 @@ static int reftable_be_config(const char *var, const char *value,\n \t\tif (restart_interval > UINT16_MAX)\n \t\t\tdie(\"reftable block size cannot exceed %u\", (unsigned)UINT16_MAX);\n \t\topts->restart_interval = restart_interval;\n+\t} else if (!strcmp(var, \"reftable.indexobjects\")) {\n+\t\topts->skip_index_objects = !git_config_bool(var, value);\n \t}\n \n \treturn 0;\ndiff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh\nindex e0a5b26f58..e2708e11d5 100755\n--- a/t/t0613-reftable-write-options.sh\n+++ b/t/t0613-reftable-write-options.sh\n@@ -214,4 +214,73 @@ test_expect_success 'restart interval exceeding maximum supported interval' '\n \t)\n '\n \n+test_expect_success 'object index gets written by default with ref index' '\n+\ttest_config_global core.logAllRefUpdates false &&\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tfor i in $(test_seq 5)\n+\t\tdo\n+\t\t\tprintf \"update refs/heads/branch-%d HEAD\\n\" \"$i\" ||\n+\t\t\treturn 1\n+\t\tdone >input &&\n+\t\tgit update-ref --stdin expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 100\n+\t\tref:\n+\t\t - length: 53\n+\t\t restarts: 1\n+\t\t - length: 95\n+\t\t restarts: 1\n+\t\t - length: 71\n+\t\t restarts: 1\n+\t\t - length: 80\n+\t\t restarts: 1\n+\t\tobj:\n+\t\t - length: 11\n+\t\t restarts: 1\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'object index can be disabled' '\n+\ttest_config_global core.logAllRefUpdates false &&\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tfor i in $(test_seq 5)\n+\t\tdo\n+\t\t\tprintf \"update refs/heads/branch-%d HEAD\\n\" \"$i\" ||\n+\t\t\treturn 1\n+\t\tdone >input &&\n+\t\tgit update-ref --stdin expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 100\n+\t\tref:\n+\t\t - length: 53\n+\t\t restarts: 1\n+\t\t - length: 95\n+\t\t restarts: 1\n+\t\t - length: 71\n+\t\t restarts: 1\n+\t\t - length: 80\n+\t\t restarts: 1\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.45.0\n\n"},{"id":"494436","messageId":"9d4c1f034038df2ae232b6665a0d9d7ee5833c5f.1715336798.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715336797.git.ps@pks.im","subject":"[PATCH v2 10/11] reftable: make the compaction factor configurable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T10:30:05Z","receivedAt":"2024-05-10T10:30:10Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When auto-compacting, the reftable library packs references such that\nthe sizes of the tables form a geometric sequence. The factor for this\ngeometric sequence is hardcoded to 2 right now. We're about to expose\nthis as a config option though, so let's expose the factor via write\noptions.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/constants.h | 1 +\n reftable/reftable-writer.h | 6 ++++++\n reftable/stack.c | 14 ++++++++++----\n reftable/stack.h | 3 ++-\n reftable/stack_test.c | 4 ++--\n 5 files changed, 21 insertions(+), 7 deletions(-)\n\ndiff --git a/reftable/constants.h b/reftable/constants.h\nindex 5eee72c4c1..f6beb843eb 100644\n--- a/reftable/constants.h\n+++ b/reftable/constants.h\n@@ -17,5 +17,6 @@ license that can be found in the LICENSE file or at\n \n #define MAX_RESTARTS ((1 << 16) - 1)\n #define DEFAULT_BLOCK_SIZE 4096\n+#define DEFAULT_GEOMETRIC_FACTOR 2\n \n #endif\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 4cd8ebe6c7..155457b042 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -49,6 +49,12 @@ struct reftable_write_options {\n \n \t/* boolean: Prevent auto-compaction of tables. */\n \tunsigned disable_auto_compact : 1;\n+\n+\t/*\n+\t * Geometric sequence factor used by auto-compaction to decide which\n+\t * tables to compact. Defaults to 2 if unset.\n+\t */\n+\tuint8_t auto_compaction_factor;\n };\n \n /* reftable_block_stats holds statistics for a single block type */\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 7b4fff7c9e..762954b181 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -10,6 +10,7 @@ license that can be found in the LICENSE file or at\n \n #include \"../write-or-die.h\"\n #include \"system.h\"\n+#include \"constants.h\"\n #include \"merged.h\"\n #include \"reader.h\"\n #include \"refname.h\"\n@@ -1215,12 +1216,16 @@ static int segment_size(struct segment *s)\n \treturn s->end - s->start;\n }\n \n-struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n+struct segment suggest_compaction_segment(uint64_t *sizes, size_t n,\n+\t\t\t\t\t uint8_t factor)\n {\n \tstruct segment seg = { 0 };\n \tuint64_t bytes;\n \tsize_t i;\n \n+\tif (!factor)\n+\t\tfactor = DEFAULT_GEOMETRIC_FACTOR;\n+\n \t/*\n \t * If there are no tables or only a single one then we don't have to\n \t * compact anything. The sequence is geometric by definition already.\n@@ -1252,7 +1257,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n \t * \t64, 32, 16, 8, 4, 3, 1\n \t */\n \tfor (i = n - 1; i > 0; i--) {\n-\t\tif (sizes[i - 1] < sizes[i] * 2) {\n+\t\tif (sizes[i - 1] < sizes[i] * factor) {\n \t\t\tseg.end = i + 1;\n \t\t\tbytes = sizes[i];\n \t\t\tbreak;\n@@ -1278,7 +1283,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n \t\tuint64_t curr = bytes;\n \t\tbytes += sizes[i - 1];\n \n-\t\tif (sizes[i - 1] < curr * 2) {\n+\t\tif (sizes[i - 1] < curr * factor) {\n \t\t\tseg.start = i - 1;\n \t\t\tseg.bytes = bytes;\n \t\t}\n@@ -1304,7 +1309,8 @@ int reftable_stack_auto_compact(struct reftable_stack *st)\n {\n \tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n \tstruct segment seg =\n-\t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n+\t\tsuggest_compaction_segment(sizes, st->merged->stack_len,\n+\t\t\t\t\t st->opts.auto_compaction_factor);\n \treftable_free(sizes);\n \tif (segment_size(&seg) > 0)\n \t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\ndiff --git a/reftable/stack.h b/reftable/stack.h\nindex 97d7ebc043..5b45cff4f7 100644\n--- a/reftable/stack.h\n+++ b/reftable/stack.h\n@@ -35,6 +35,7 @@ struct segment {\n \tuint64_t bytes;\n };\n \n-struct segment suggest_compaction_segment(uint64_t *sizes, size_t n);\n+struct segment suggest_compaction_segment(uint64_t *sizes, size_t n,\n+\t\t\t\t\t uint8_t factor);\n \n #endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 3316d55f19..f6c11ef18d 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -767,7 +767,7 @@ static void test_suggest_compaction_segment(void)\n {\n \tuint64_t sizes[] = { 512, 64, 17, 16, 9, 9, 9, 16, 2, 16 };\n \tstruct segment min =\n-\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes), 2);\n \tEXPECT(min.start == 1);\n \tEXPECT(min.end == 10);\n }\n@@ -776,7 +776,7 @@ static void test_suggest_compaction_segment_nothing(void)\n {\n \tuint64_t sizes[] = { 64, 32, 16, 8, 4, 2 };\n \tstruct segment result =\n-\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes), 2);\n \tEXPECT(result.start == result.end);\n }\n \n-- \n2.45.0\n\n"},{"id":"494437","messageId":"e1282e53fb182dcb23255c360da38adb531be297.1715336798.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715336797.git.ps@pks.im","subject":"[PATCH v2 11/11] refs/reftable: allow configuring geometric factor","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-10T10:30:11Z","receivedAt":"2024-05-10T10:30:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Allow configuring the geometric factor used by the auto-compaction\nalgorithm whenever a new table is appended to the stack of tables.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config/reftable.txt | 10 ++++++++++\n refs/reftable-backend.c | 5 +++++\n 2 files changed, 15 insertions(+)\n\ndiff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt\nindex 6e4466f3c5..f928d1029b 100644\n--- a/Documentation/config/reftable.txt\n+++ b/Documentation/config/reftable.txt\n@@ -37,3 +37,13 @@ reftable.indexObjects::\n \tare a reverse mapping of object ID to the references pointing to them.\n +\n The default value is `true`.\n+\n+reftable.geometricFactor::\n+\tWhenever the reftable backend appends a new table to the stack, it\n+\tperforms auto compaction to ensure that there is only a handful of\n+\ttables. The backend does this by ensuring that tables form a geometric\n+\tsequence regarding the respective sizes of each table.\n++\n+By default, the geometric sequence uses a factor of 2, meaning that for any\n+table, the next-biggest table must at least be twice as big. A maximum factor\n+of 256 is supported.\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 63b75f770d..c4bd0fe1f9 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -249,6 +249,11 @@ static int reftable_be_config(const char *var, const char *value,\n \t\topts->restart_interval = restart_interval;\n \t} else if (!strcmp(var, \"reftable.indexobjects\")) {\n \t\topts->skip_index_objects = !git_config_bool(var, value);\n+\t} else if (!strcmp(var, \"reftable.geometricfactor\")) {\n+\t\tunsigned long factor = git_config_ulong(var, value, ctx->kvi);\n+\t\tif (factor > UINT8_MAX)\n+\t\t\tdie(\"reftable geometric factor cannot exceed %u\", (unsigned)UINT8_MAX);\n+\t\topts->auto_compaction_factor = factor;\n \t}\n \n \treturn 0;\n-- \n2.45.0\n\n"},{"id":"494462","messageId":"CAOLa=ZTJq40qhXLR=W0DSKKd+rehU-CzBafrkxP_nZH7xGJyWQ@mail.gmail.com","threadId":"61406","inReplyTo":"cover.1715336797.git.ps@pks.im","subject":"Re: [PATCH v2 00/11] reftable: expose write options as config","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-05-10T11:43:36Z","receivedAt":"2024-05-10T11:43:39Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Hi,\n>\n> this is the second version of my patch series that exposes various\n> options of the reftable writer via Git configuration.\n>\n> Changes compared to v1:\n>\n> - Drop unneeded return statements.\n>\n> - Move default geometric factor into \"constants.h\".\n>\n> - Fix a typo in a commit message.\n>\n> Thanks!\n>\n> Patrick\n>\n> Patrick Steinhardt (11):\n> reftable: consistently refer to `reftable_write_options` as `opts`\n> reftable: consistently pass write opts as value\n> reftable/writer: drop static variable used to initialize strbuf\n> reftable/writer: improve error when passed an invalid block size\n> reftable/dump: support dumping a table's block structure\n> refs/reftable: allow configuring block size\n> reftable: use `uint16_t` to track restart interval\n> refs/reftable: allow configuring restart interval\n> refs/reftable: allow disabling writing the object index\n> reftable: make the compaction factor configurable\n> refs/reftable: allow configuring geometric factor\n>\n> Documentation/config.txt | 2 +\n> Documentation/config/reftable.txt | 49 +++++\n> refs/reftable-backend.c | 43 ++++-\n> reftable/block.h | 2 +-\n> reftable/constants.h | 1 +\n> reftable/dump.c | 12 +-\n> reftable/merged_test.c | 6 +-\n> reftable/reader.c | 63 +++++++\n> reftable/readwrite_test.c | 26 +--\n> reftable/refname_test.c | 2 +-\n> reftable/reftable-reader.h | 2 +\n> reftable/reftable-stack.h | 2 +-\n> reftable/reftable-writer.h | 10 +-\n> reftable/stack.c | 57 +++---\n> reftable/stack.h | 5 +-\n> reftable/stack_test.c | 118 ++++++------\n> reftable/writer.c | 20 +--\n> t/t0613-reftable-write-options.sh | 286 ++++++++++++++++++++++++++++++\n> 18 files changed, 576 insertions(+), 130 deletions(-)\n> create mode 100644 Documentation/config/reftable.txt\n> create mode 100755 t/t0613-reftable-write-options.sh\n>\n> Range-diff against v1:\n> 1: 47cee6e25e = 1: 7efa566306 reftable: consistently refer to `reftable_write_options` as `opts`\n> 2: d8a0764e87 = 2: e6f8fc09c2 reftable: consistently pass write opts as value\n> 3: c040f81fba = 3: aa2903e3e5 reftable/writer: drop static variable used to initialize strbuf\n> 4: ef79bb1b7b = 4: 5e7cbb7b19 reftable/writer: improve error when passed an invalid block size\n> 5: 4d4407d4a4 = 5: ed1c150d90 reftable/dump: support dumping a table's block structure\n> 6: b4e4db5735 ! 6: be5bdc6dc1 refs/reftable: allow configuring block size\n> @@ refs/reftable-backend.c: static int read_ref_without_reload(struct reftable_stac\n> +\t\tif (block_size > 16777215)\n> +\t\t\tdie(\"reftable block size cannot exceed 16MB\");\n> +\t\topts->block_size = block_size;\n> -+\t\treturn 0;\n> +\t}\n> +\n> +\treturn 0;\n> 7: 79d9e07ca9 = 7: 05e8d1df2d reftable: use `uint16_t` to track restart interval\n> 8: 653ec4dfa5 ! 8: bc0bf65553 refs/reftable: allow configuring restart interval\n> @@ Documentation/config/reftable.txt: readers during access.\n>\n> ## refs/reftable-backend.c ##\n> @@ refs/reftable-backend.c: static int reftable_be_config(const char *var, const char *value,\n> + \t\tif (block_size > 16777215)\n> \t\t\tdie(\"reftable block size cannot exceed 16MB\");\n> \t\topts->block_size = block_size;\n> - \t\treturn 0;\n> +\t} else if (!strcmp(var, \"reftable.restartinterval\")) {\n> +\t\tunsigned long restart_interval = git_config_ulong(var, value, ctx->kvi);\n> +\t\tif (restart_interval > UINT16_MAX)\n> +\t\t\tdie(\"reftable block size cannot exceed %u\", (unsigned)UINT16_MAX);\n> +\t\topts->restart_interval = restart_interval;\n> -+\t\treturn 0;\n> \t}\n>\n> \treturn 0;\n> 9: 6f2c481acc ! 9: 6bc240fd0c refs/reftable: allow disabling writing the object index\n> @@ Documentation/config/reftable.txt: A maximum of `65535` restart points per block\n>\n> ## refs/reftable-backend.c ##\n> @@ refs/reftable-backend.c: static int reftable_be_config(const char *var, const char *value,\n> + \t\tif (restart_interval > UINT16_MAX)\n> \t\t\tdie(\"reftable block size cannot exceed %u\", (unsigned)UINT16_MAX);\n> \t\topts->restart_interval = restart_interval;\n> - \t\treturn 0;\n> +\t} else if (!strcmp(var, \"reftable.indexobjects\")) {\n> +\t\topts->skip_index_objects = !git_config_bool(var, value);\n> -+\t\treturn 0;\n> \t}\n>\n> \treturn 0;\n> 10: 30e2e33479 ! 10: 9d4c1f0340 reftable: make the compaction factor configurable\n> @@ Commit message\n>\n> Signed-off-by: Patrick Steinhardt \n>\n> + ## reftable/constants.h ##\n> +@@ reftable/constants.h: license that can be found in the LICENSE file or at\n> +\n> + #define MAX_RESTARTS ((1 << 16) - 1)\n> + #define DEFAULT_BLOCK_SIZE 4096\n> ++#define DEFAULT_GEOMETRIC_FACTOR 2\n> +\n> + #endif\n> +\n> ## reftable/reftable-writer.h ##\n> @@ reftable/reftable-writer.h: struct reftable_write_options {\n>\n> @@ reftable/reftable-writer.h: struct reftable_write_options {\n> /* reftable_block_stats holds statistics for a single block type */\n>\n> ## reftable/stack.c ##\n> +@@ reftable/stack.c: license that can be found in the LICENSE file or at\n> +\n> + #include \"../write-or-die.h\"\n> + #include \"system.h\"\n> ++#include \"constants.h\"\n> + #include \"merged.h\"\n> + #include \"reader.h\"\n> + #include \"refname.h\"\n> @@ reftable/stack.c: static int segment_size(struct segment *s)\n> \treturn s->end - s->start;\n> }\n> @@ reftable/stack.c: static int segment_size(struct segment *s)\n> \tsize_t i;\n>\n> +\tif (!factor)\n> -+\t\tfactor = 2;\n> ++\t\tfactor = DEFAULT_GEOMETRIC_FACTOR;\n> +\n> \t/*\n> \t * If there are no tables or only a single one then we don't have to\n> 11: 861f2e72d9 ! 11: e1282e53fb refs/reftable: allow configuring geometric factor\n> @@ Documentation/config/reftable.txt: reftable.indexObjects::\n> The default value is `true`.\n> +\n> +reftable.geometricFactor::\n> -+\tWhenever the reftable backend appends a new table to the table it\n> ++\tWhenever the reftable backend appends a new table to the stack, it\n> +\tperforms auto compaction to ensure that there is only a handful of\n> +\ttables. The backend does this by ensuring that tables form a geometric\n> +\tsequence regarding the respective sizes of each table.\n> @@ Documentation/config/reftable.txt: reftable.indexObjects::\n>\n> ## refs/reftable-backend.c ##\n> @@ refs/reftable-backend.c: static int reftable_be_config(const char *var, const char *value,\n> + \t\topts->restart_interval = restart_interval;\n> \t} else if (!strcmp(var, \"reftable.indexobjects\")) {\n> \t\topts->skip_index_objects = !git_config_bool(var, value);\n> - \t\treturn 0;\n> +\t} else if (!strcmp(var, \"reftable.geometricfactor\")) {\n> +\t\tunsigned long factor = git_config_ulong(var, value, ctx->kvi);\n> +\t\tif (factor > UINT8_MAX)\n>\n> base-commit: d4cc1ec35f3bcce816b69986ca41943f6ce21377\n> --\n> 2.45.0\n\nThe range diff looks good to me, thanks for the quick iteration :)\n"},{"id":"494509","messageId":"xmqq34qp4bxu.fsf@gitster.g","threadId":"61406","inReplyTo":"7efa56630681b6aca64c04fd19fb020ffbb76152.1715336798.git.ps@pks.im","subject":"Re: [PATCH v2 01/11] reftable: consistently refer to `reftable_write_options` as `opts`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-10T21:03:09Z","receivedAt":"2024-05-10T21:03:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n> -\t\t struct reftable_write_options config);\n> +\t\t struct reftable_write_options opts);\n\nPassing struct by value is somewhat unusual. As long as the\nstructure does not contain any pointer to something else, it is not\ntoo bad as structure assignment would also work well, though. Not a\nfault of this patch, and this series is not a place to change\nanything about it.\n\n> -\tif (config.hash_id == 0) {\n> -\t\tconfig.hash_id = GIT_SHA1_FORMAT_ID;\n> -\t}\n> +\tif (opts.hash_id == 0)\n> +\t\topts.hash_id = GIT_SHA1_FORMAT_ID;\n\nNice attention to detail.\n\n"},{"id":"494510","messageId":"xmqqy18h2wys.fsf@gitster.g","threadId":"61406","inReplyTo":"e6f8fc09c282e620a331a737c30a190107c76996.1715336798.git.ps@pks.im","subject":"Re: [PATCH v2 02/11] reftable: consistently pass write opts as value","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-10T21:11:55Z","receivedAt":"2024-05-10T21:12:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> We sometimes pass the refatble write options as value and sometimes as a\n> pointer. This is quite confusing and makes the reader wonder whether the\n> options get modified sometimes.\n>\n> In fact, `reftable_new_writer()` does cause the caller-provided options\n> to get updated when some values aren't set up. This is quite unexpected,\n> but didn't cause any harm until now.\n>\n> Refactor the code to consistently pass the options as a value so that\n> they are local to the subsystem they are being passed into so that we\n> can avoid weirdness like this.\n\nTurning pass-by-reference to pass-by-value of a large structure is a\nrather huge hammer to ensure that the structure is not modified\n(qualifying the pointer with \"const\" is). Consistency is good, but\nI am not sure offhand if this is making things consistent in a good\nway.\n"},{"id":"494511","messageId":"xmqqttj52wma.fsf@gitster.g","threadId":"61406","inReplyTo":"aa2903e3e5ac76f82252e12919a8a09178e267ee.1715336798.git.ps@pks.im","subject":"Re: [PATCH v2 03/11] reftable/writer: drop static variable used to initialize strbuf","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-10T21:19:25Z","receivedAt":"2024-05-10T21:19:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> We have a static variable in the reftable writer code that is merely\n> used to initialize the `last_key` of the writer. Convert the code to\n> instead use `strbuf_init()` and drop the variable.\n\nNice. There is no guarantee that a structure assignment of a\njust-initialized empty strbuf to another will stay to be a trouble\nfree operation, and using an explicit _init() call is the right\nthing to do.\n\n"},{"id":"494512","messageId":"xmqqo79d2wc7.fsf@gitster.g","threadId":"61406","inReplyTo":"5e7cbb7b193c578f7c946a5077a79421b0ac57f2.1715336798.git.ps@pks.im","subject":"Re: [PATCH v2 04/11] reftable/writer: improve error when passed an invalid block size","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-10T21:25:28Z","receivedAt":"2024-05-10T21:25:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> The reftable format only supports block sizes up to 16MB. When the\n> writer is being passed a value bigger than that it simply calls\n> abort(3P), which isn't all that helpful due to the lack of a proper\n> error message.\n>\n> Improve this by calling `BUG()` instead.\n\nAs a \"git\" person, I do not mind this at all.\n\nBut doesn't the reftable/ library codebase want to avoid things like\nBUG() that are very much tied to our codebase, for the same reason\nas it avoids things like xmalloc(), xcalloc(), and ALLOC_GROW()?\n\nWe may have crossed the bridge long time ago, though. We see a\nhandful calls to BUG() already inside reftable/ directory.\n"},{"id":"494516","messageId":"xmqq34qp2uud.fsf@gitster.g","threadId":"61406","inReplyTo":"bc0bf65553c8dd89bf5fcaa592fc3427507f1993.1715336798.git.ps@pks.im","subject":"Re: [PATCH v2 08/11] refs/reftable: allow configuring restart interval","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-10T21:57:46Z","receivedAt":"2024-05-10T21:58:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> +\n> +reftable.restartInterval::\n> +\tThe interval at which to create restart points. The reftable backend\n> +\tdetermines the restart points at file creation. The process is\n> +\tarbitrary, but every 16 or 64 records is recommended. Every 16 may be\n\nIt is unclear what exactly \"The process is arbitrary, but\" wants to\nsay, especially the use of the noun \"process\". The process the user\nuses to choose the inteval value is? The default value chosen by us\nwas arbitrary and out of thin air?\n\nJust striking the whole sentence (or removing up to \", but\" part and\nstarting the sentence with \"Every 16 or 64\") may make the resulting\nparagraph easier to follow, I suspect.\n\n> +\t} else if (!strcmp(var, \"reftable.restartinterval\")) {\n> +\t\tunsigned long restart_interval = git_config_ulong(var, value, ctx->kvi);\n> +\t\tif (restart_interval > UINT16_MAX)\n> +\t\t\tdie(\"reftable block size cannot exceed %u\", (unsigned)UINT16_MAX);\n\nOK.\n\n"},{"id":"494520","messageId":"xmqqy18h1fm4.fsf@gitster.g","threadId":"61406","inReplyTo":"9d4c1f034038df2ae232b6665a0d9d7ee5833c5f.1715336798.git.ps@pks.im","subject":"Re: [PATCH v2 10/11] reftable: make the compaction factor configurable","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-10T22:12:03Z","receivedAt":"2024-05-10T22:12:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> When auto-compacting, the reftable library packs references such that\n> the sizes of the tables form a geometric sequence. The factor for this\n> geometric sequence is hardcoded to 2 right now. We're about to expose\n> this as a config option though, so let's expose the factor via write\n> options.\n\nHmph. It is unclear if having this as uint8_t gives us a useful\nenhancement, but perhaps in the future hosters may find a more\naggressive geometric sequence is better for their workload or\nsomething and raise it to 3 or 4? I was actually wondering if a\nbase smaller than 2 (e.g. fibonacci) may work better.\n\nAnyway, making it configurable is a good first step. Allowing a bit\nfiner grained setting than just integral values can be done later if\nit proves necessary.\n"},{"id":"494568","messageId":"ZkHHCInClkoNBgVt@tanuki","threadId":"61406","inReplyTo":"xmqqy18h2wys.fsf@gitster.g","subject":"Re: [PATCH v2 02/11] reftable: consistently pass write opts as value","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-13T07:53:44Z","receivedAt":"2024-05-13T07:53:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, May 10, 2024 at 02:11:55PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > We sometimes pass the refatble write options as value and sometimes as a\n> > pointer. This is quite confusing and makes the reader wonder whether the\n> > options get modified sometimes.\n> >\n> > In fact, `reftable_new_writer()` does cause the caller-provided options\n> > to get updated when some values aren't set up. This is quite unexpected,\n> > but didn't cause any harm until now.\n> >\n> > Refactor the code to consistently pass the options as a value so that\n> > they are local to the subsystem they are being passed into so that we\n> > can avoid weirdness like this.\n> \n> Turning pass-by-reference to pass-by-value of a large structure is a\n> rather huge hammer to ensure that the structure is not modified\n> (qualifying the pointer with \"const\" is). Consistency is good, but\n> I am not sure offhand if this is making things consistent in a good\n> way.\n\nYeah, I've also been a bit \"meh\" about this. I'm happy to convert this\nto all be const pointers instead.\n\nPatrick\n"},{"id":"494569","messageId":"ZkHHEG27YA4EhjU1@tanuki","threadId":"61406","inReplyTo":"xmqqo79d2wc7.fsf@gitster.g","subject":"Re: [PATCH v2 04/11] reftable/writer: improve error when passed an invalid block size","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-13T07:53:52Z","receivedAt":"2024-05-13T07:53:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, May 10, 2024 at 02:25:28PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > The reftable format only supports block sizes up to 16MB. When the\n> > writer is being passed a value bigger than that it simply calls\n> > abort(3P), which isn't all that helpful due to the lack of a proper\n> > error message.\n> >\n> > Improve this by calling `BUG()` instead.\n> \n> As a \"git\" person, I do not mind this at all.\n> \n> But doesn't the reftable/ library codebase want to avoid things like\n> BUG() that are very much tied to our codebase, for the same reason\n> as it avoids things like xmalloc(), xcalloc(), and ALLOC_GROW()?\n> \n> We may have crossed the bridge long time ago, though. We see a\n> handful calls to BUG() already inside reftable/ directory.\n\nExactly. No matter what, once there will be a second user of the\nreftable library we will have to figure out a maintainable way to ensure\nthat the library can be used by other projects, too. And that will\nrequire some larger refactorings anyway.\n\nI think initially, the intent was to have a \"system.h\" header that\ncontains a bunch of wrappers that bridge the gap between reftables and\nGit. I feel like this abstraction does not make any sense though in its\ncurrent form as it is simply being included by the reftable code, which\nthen uses the Git functions directly.\n\nI think eventually, we will have to adapt this such that the Git\nincludes do not leak into the reftable code at all. Instead, we should\nhave a shim \"system.c\" that carries the Git-specific includes and then\nimplements a couple of wrapper functions. \"system.h\" would then only be\ncarrying function declarations of those wrappers.\n\nThat's a larger topic though, and I think that tackling this now would\nbe premature without any potential users of the reftable library.\n\nPatrick\n"},{"id":"494570","messageId":"ZkHHGtbI2oyxNlhV@tanuki","threadId":"61406","inReplyTo":"xmqq34qp2uud.fsf@gitster.g","subject":"Re: [PATCH v2 08/11] refs/reftable: allow configuring restart interval","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-13T07:54:02Z","receivedAt":"2024-05-13T07:54:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, May 10, 2024 at 02:57:46PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > +\n> > +reftable.restartInterval::\n> > +\tThe interval at which to create restart points. The reftable backend\n> > +\tdetermines the restart points at file creation. The process is\n> > +\tarbitrary, but every 16 or 64 records is recommended. Every 16 may be\n> \n> It is unclear what exactly \"The process is arbitrary, but\" wants to\n> say, especially the use of the noun \"process\". The process the user\n> uses to choose the inteval value is? The default value chosen by us\n> was arbitrary and out of thin air?\n\nThe latter is what I wanted to say, but I agree that it's hard to parse.\nAnd honestly, I don't even know how arbitrary it is, so I should\nprobably not claim something like this in the first place.\n\n> Just striking the whole sentence (or removing up to \", but\" part and\n> starting the sentence with \"Every 16 or 64\") may make the resulting\n> paragraph easier to follow, I suspect.\n\nWill do.\n\nPatrick\n"},{"id":"494571","messageId":"ZkHHIgyeWq9FCYuq@tanuki","threadId":"61406","inReplyTo":"xmqqy18h1fm4.fsf@gitster.g","subject":"Re: [PATCH v2 10/11] reftable: make the compaction factor configurable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-13T07:54:10Z","receivedAt":"2024-05-13T07:54:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, May 10, 2024 at 03:12:03PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > When auto-compacting, the reftable library packs references such that\n> > the sizes of the tables form a geometric sequence. The factor for this\n> > geometric sequence is hardcoded to 2 right now. We're about to expose\n> > this as a config option though, so let's expose the factor via write\n> > options.\n> \n> Hmph. It is unclear if having this as uint8_t gives us a useful\n> enhancement, but perhaps in the future hosters may find a more\n> aggressive geometric sequence is better for their workload or\n> something and raise it to 3 or 4? I was actually wondering if a\n> base smaller than 2 (e.g. fibonacci) may work better.\n> \n> Anyway, making it configurable is a good first step. Allowing a bit\n> finer grained setting than just integral values can be done later if\n> it proves necessary.\n\nThat's a fair point indeed. I had similar issues with git-repack(1)'s\n`--geometric=` option, where you can also only pick integers. There's\nalso a similar discussion in the patch series by Taylor [1], where I\nproposed to maybe introduce floats into git-config(1).\n\nSo this may be good enough for now, and when we gain the ability to\nparse floats we may convert this to accept floats, as well. An\nalternative would be to convert this to percent, where the default value\nwould be 200. That should give sufficient flexibility without having to\nintroduce floats.\n\nPatrick\n\n[1]: https://lore.kernel.org/git/Zjk2UIV3kEwZUDW+@nand.local/\n"},{"id":"494573","messageId":"cover.1715587849.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1714630191.git.ps@pks.im","subject":"[PATCH v3 00/11] reftable: expose write options as config","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-13T08:17:49Z","receivedAt":"2024-05-13T08:17:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the third version of my patch series that exposes several write\noptions of the reftable library via Git configs.\n\nChanges compared to v2:\n\n - Adapted patch 2 such that we now pass options as const pointers\n instead of by value.\n\n - Removed a confusing sentence in the documentation of the restart\n points in patch 8.\n\nOther than that I decided to rebase this on top of the current \"master\"\nbranch at 0f3415f1f8 (The second batch, 2024-05-08). This is because the\nrevamped patch 2 would cause new conflicts with 485c63cf5c (reftable:\nremove name checks, 2024-04-08) that didn't exist in v2 of this patch\nseries yet. Rebasing thus seemed like the more reasonable option.\n\nPatrick\n\nPatrick Steinhardt (11):\n reftable: consistently refer to `reftable_write_options` as `opts`\n reftable: pass opts as constant pointer\n reftable/writer: drop static variable used to initialize strbuf\n reftable/writer: improve error when passed an invalid block size\n reftable/dump: support dumping a table's block structure\n refs/reftable: allow configuring block size\n reftable: use `uint16_t` to track restart interval\n refs/reftable: allow configuring restart interval\n refs/reftable: allow disabling writing the object index\n reftable: make the compaction factor configurable\n refs/reftable: allow configuring geometric factor\n\n Documentation/config.txt | 2 +\n Documentation/config/reftable.txt | 48 +++++\n refs/reftable-backend.c | 49 ++++-\n reftable/block.h | 2 +-\n reftable/constants.h | 1 +\n reftable/dump.c | 12 +-\n reftable/reader.c | 63 +++++++\n reftable/reftable-reader.h | 2 +\n reftable/reftable-stack.h | 2 +-\n reftable/reftable-writer.h | 10 +-\n reftable/stack.c | 58 +++---\n reftable/stack.h | 5 +-\n reftable/stack_test.c | 118 ++++++------\n reftable/writer.c | 23 +--\n t/t0613-reftable-write-options.sh | 286 ++++++++++++++++++++++++++++++\n 15 files changed, 566 insertions(+), 115 deletions(-)\n create mode 100644 Documentation/config/reftable.txt\n create mode 100755 t/t0613-reftable-write-options.sh\n\nRange-diff against v2:\n 1: 7efa566306 ! 1: 71f4e31cf7 reftable: consistently refer to `reftable_write_options` as `opts`\n @@ reftable/stack.c: static uint64_t *stack_table_sizes_for_compaction(struct refta\n \tint overhead = header_size(version) - 1;\n \tint i = 0;\n \tfor (i = 0; i < st->merged->stack_len; i++) {\n -@@ reftable/stack.c: static int stack_check_addition(struct reftable_stack *st,\n - \tint len = 0;\n - \tint i = 0;\n - \n --\tif (st->config.skip_name_check)\n -+\tif (st->opts.skip_name_check)\n - \t\treturn 0;\n - \n - \terr = reftable_block_source_from_file(&src, new_tab_name);\n @@ reftable/stack.c: int reftable_stack_clean(struct reftable_stack *st)\n int reftable_stack_print_directory(const char *stackdir, uint32_t hash_id)\n {\n 2: e6f8fc09c2 ! 2: f1c9914a77 reftable: consistently pass write opts as value\n @@ Metadata\n Author: Patrick Steinhardt \n \n ## Commit message ##\n - reftable: consistently pass write opts as value\n + reftable: pass opts as constant pointer\n \n We sometimes pass the refatble write options as value and sometimes as a\n pointer. This is quite confusing and makes the reader wonder whether the\n @@ Commit message\n to get updated when some values aren't set up. This is quite unexpected,\n but didn't cause any harm until now.\n \n - Refactor the code to consistently pass the options as a value so that\n - they are local to the subsystem they are being passed into so that we\n - can avoid weirdness like this.\n + Adapt the code so that we do not modify the caller-provided values\n + anymore. While at it, refactor the code to code to consistently pass the\n + options as a constant pointer to clarify that the caller-provided opts\n + will not ever get modified.\n \n Signed-off-by: Patrick Steinhardt \n \n - ## reftable/merged_test.c ##\n -@@ reftable/merged_test.c: static void write_test_table(struct strbuf *buf,\n + ## refs/reftable-backend.c ##\n +@@ refs/reftable-backend.c: static struct reftable_stack *stack_for(struct reftable_ref_store *store,\n + \t\t\t\t store->base.repo->commondir, wtname_buf.buf);\n + \n + \t\t\tstore->err = reftable_new_stack(&stack, wt_dir.buf,\n +-\t\t\t\t\t\t\tstore->write_options);\n ++\t\t\t\t\t\t\t&store->write_options);\n + \t\t\tassert(store->err != REFTABLE_API_ERROR);\n + \t\t\tstrmap_put(&store->worktree_stacks, wtname_buf.buf, stack);\n \t\t}\n +@@ refs/reftable-backend.c: static struct ref_store *reftable_be_init(struct repository *repo,\n \t}\n + \tstrbuf_addstr(&path, \"/reftable\");\n + \trefs->err = reftable_new_stack(&refs->main_stack, path.buf,\n +-\t\t\t\t refs->write_options);\n ++\t\t\t\t &refs->write_options);\n + \tif (refs->err)\n + \t\tgoto done;\n \n --\tw = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, &opts);\n -+\tw = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, opts);\n - \treftable_writer_set_limits(w, min, max);\n - \n - \tfor (i = 0; i < n; i++) {\n -@@ reftable/merged_test.c: static void write_test_log_table(struct strbuf *buf,\n - \t\t.exact_log_message = 1,\n - \t};\n - \tstruct reftable_writer *w = NULL;\n --\tw = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, &opts);\n -+\tw = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, opts);\n - \treftable_writer_set_limits(w, update_index, update_index);\n - \n - \tfor (i = 0; i < n; i++) {\n -@@ reftable/merged_test.c: static void test_default_write_opts(void)\n - \tstruct reftable_write_options opts = { 0 };\n - \tstruct strbuf buf = STRBUF_INIT;\n - \tstruct reftable_writer *w =\n --\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n -+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n +@@ refs/reftable-backend.c: static struct ref_store *reftable_be_init(struct repository *repo,\n + \t\tstrbuf_addf(&path, \"%s/reftable\", gitdir);\n \n - \tstruct reftable_ref_record rec = {\n - \t\t.refname = \"master\",\n + \t\trefs->err = reftable_new_stack(&refs->worktree_stack, path.buf,\n +-\t\t\t\t\t refs->write_options);\n ++\t\t\t\t\t &refs->write_options);\n + \t\tif (refs->err)\n + \t\t\tgoto done;\n + \t}\n \n - ## reftable/readwrite_test.c ##\n -@@ reftable/readwrite_test.c: static void write_table(char ***names, struct strbuf *buf, int N,\n - \t\t.hash_id = hash_id,\n - \t};\n - \tstruct reftable_writer *w =\n --\t\treftable_new_writer(&strbuf_add_void, &noop_flush, buf, &opts);\n -+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, buf, opts);\n - \tstruct reftable_ref_record ref = { NULL };\n - \tint i = 0, n;\n - \tstruct reftable_log_record log = { NULL };\n -@@ reftable/readwrite_test.c: static void test_log_buffer_size(void)\n - \t\t\t\t\t .message = \"commit: 9\\n\",\n - \t\t\t\t } } };\n - \tstruct reftable_writer *w =\n --\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n -+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n - \n - \t/* This tests buffer extension for log compression. Must use a random\n - \t hash, to ensure that the compressed part is larger than the original.\n -@@ reftable/readwrite_test.c: static void test_log_overflow(void)\n - \t\t},\n - \t};\n - \tstruct reftable_writer *w =\n --\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n -+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n - \n - \tmemset(msg, 'x', sizeof(msg) - 1);\n - \treftable_writer_set_limits(w, update_index, update_index);\n -@@ reftable/readwrite_test.c: static void test_log_write_read(void)\n - \tstruct reftable_block_source source = { NULL };\n - \tstruct strbuf buf = STRBUF_INIT;\n - \tstruct reftable_writer *w =\n --\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n -+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n - \tconst struct reftable_stats *stats = NULL;\n - \treftable_writer_set_limits(w, 0, N);\n - \tfor (i = 0; i < N; i++) {\n -@@ reftable/readwrite_test.c: static void test_log_zlib_corruption(void)\n - \tstruct reftable_block_source source = { 0 };\n - \tstruct strbuf buf = STRBUF_INIT;\n - \tstruct reftable_writer *w =\n --\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n -+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n - \tconst struct reftable_stats *stats = NULL;\n - \tchar message[100] = { 0 };\n - \tint err, i, n;\n -@@ reftable/readwrite_test.c: static void test_table_refs_for(int indexed)\n - \n - \tstruct strbuf buf = STRBUF_INIT;\n - \tstruct reftable_writer *w =\n --\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n -+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n - \n - \tstruct reftable_iterator it = { NULL };\n - \tint j;\n -@@ reftable/readwrite_test.c: static void test_write_empty_table(void)\n + ## reftable/dump.c ##\n +@@ reftable/dump.c: static int compact_stack(const char *stackdir)\n + \tstruct reftable_stack *stack = NULL;\n \tstruct reftable_write_options opts = { 0 };\n - \tstruct strbuf buf = STRBUF_INIT;\n - \tstruct reftable_writer *w =\n --\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n -+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n - \tstruct reftable_block_source source = { NULL };\n - \tstruct reftable_reader *rd = NULL;\n - \tstruct reftable_ref_record rec = { NULL };\n -@@ reftable/readwrite_test.c: static void test_write_object_id_min_length(void)\n - \t};\n - \tstruct strbuf buf = STRBUF_INIT;\n - \tstruct reftable_writer *w =\n --\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n -+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n - \tstruct reftable_ref_record ref = {\n - \t\t.update_index = 1,\n - \t\t.value_type = REFTABLE_REF_VAL1,\n -@@ reftable/readwrite_test.c: static void test_write_object_id_length(void)\n - \t};\n - \tstruct strbuf buf = STRBUF_INIT;\n - \tstruct reftable_writer *w =\n --\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n -+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n - \tstruct reftable_ref_record ref = {\n - \t\t.update_index = 1,\n - \t\t.value_type = REFTABLE_REF_VAL1,\n -@@ reftable/readwrite_test.c: static void test_write_empty_key(void)\n - \tstruct reftable_write_options opts = { 0 };\n - \tstruct strbuf buf = STRBUF_INIT;\n - \tstruct reftable_writer *w =\n --\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n -+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n - \tstruct reftable_ref_record ref = {\n - \t\t.refname = \"\",\n - \t\t.update_index = 1,\n -@@ reftable/readwrite_test.c: static void test_write_key_order(void)\n - \tstruct reftable_write_options opts = { 0 };\n - \tstruct strbuf buf = STRBUF_INIT;\n - \tstruct reftable_writer *w =\n --\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n -+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n - \tstruct reftable_ref_record refs[2] = {\n - \t\t{\n - \t\t\t.refname = \"b\",\n -@@ reftable/readwrite_test.c: static void test_write_multiple_indices(void)\n - \tstruct reftable_reader *reader;\n - \tint err, i;\n \n --\twriter = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, &opts);\n -+\twriter = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, opts);\n - \treftable_writer_set_limits(writer, 1, 1);\n - \tfor (i = 0; i < 100; i++) {\n - \t\tstruct reftable_ref_record ref = {\n -@@ reftable/readwrite_test.c: static void test_write_multi_level_index(void)\n - \tstruct reftable_reader *reader;\n - \tint err;\n +-\tint err = reftable_new_stack(&stack, stackdir, opts);\n ++\tint err = reftable_new_stack(&stack, stackdir, &opts);\n + \tif (err < 0)\n + \t\tgoto done;\n \n --\twriter = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, &opts);\n -+\twriter = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, opts);\n - \treftable_writer_set_limits(writer, 1, 1);\n - \tfor (size_t i = 0; i < 200; i++) {\n - \t\tstruct reftable_ref_record ref = {\n \n - ## reftable/refname_test.c ##\n -@@ reftable/refname_test.c: static void test_conflict(void)\n - \tstruct reftable_write_options opts = { 0 };\n - \tstruct strbuf buf = STRBUF_INIT;\n - \tstruct reftable_writer *w =\n --\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts);\n -+\t\treftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts);\n - \tstruct reftable_ref_record rec = {\n - \t\t.refname = \"a/b\",\n - \t\t.value_type = REFTABLE_REF_SYMREF,\n + ## reftable/reftable-stack.h ##\n +@@ reftable/reftable-stack.h: struct reftable_stack;\n + * stored in 'dir'. Typically, this should be .git/reftables.\n + */\n + int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n +-\t\t struct reftable_write_options opts);\n ++\t\t const struct reftable_write_options *opts);\n + \n + /* returns the update_index at which a next table should be written. */\n + uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\n \n ## reftable/reftable-writer.h ##\n @@ reftable/reftable-writer.h: struct reftable_stats {\n @@ reftable/reftable-writer.h: struct reftable_stats {\n reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\t int (*flush_func)(void *),\n -\t\t void *writer_arg, struct reftable_write_options *opts);\n -+\t\t void *writer_arg, struct reftable_write_options opts);\n ++\t\t void *writer_arg, const struct reftable_write_options *opts);\n \n /* Set the range of update indices for the records we will add. When writing a\n table into a stack, the min should be at least\n \n ## reftable/stack.c ##\n -@@ reftable/stack.c: int reftable_addition_add(struct reftable_addition *add,\n - \ttab_fd = get_tempfile_fd(tab_file);\n +@@ reftable/stack.c: static int reftable_fd_flush(void *arg)\n + }\n + \n + int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n +-\t\t struct reftable_write_options opts)\n ++\t\t const struct reftable_write_options *_opts)\n + {\n + \tstruct reftable_stack *p = reftable_calloc(1, sizeof(*p));\n + \tstruct strbuf list_file_name = STRBUF_INIT;\n ++\tstruct reftable_write_options opts = {0};\n + \tint err = 0;\n \n - \twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, &tab_fd,\n --\t\t\t\t &add->stack->opts);\n -+\t\t\t\t add->stack->opts);\n - \terr = write_table(wr, arg);\n ++\tif (_opts)\n ++\t\topts = *_opts;\n + \tif (opts.hash_id == 0)\n + \t\topts.hash_id = GIT_SHA1_FORMAT_ID;\n + \n +@@ reftable/stack.c: int reftable_stack_print_directory(const char *stackdir, uint32_t hash_id)\n + \tstruct reftable_merged_table *merged = NULL;\n + \tstruct reftable_table table = { NULL };\n + \n +-\tint err = reftable_new_stack(&stack, stackdir, opts);\n ++\tint err = reftable_new_stack(&stack, stackdir, &opts);\n \tif (err < 0)\n \t\tgoto done;\n -@@ reftable/stack.c: static int stack_compact_locked(struct reftable_stack *st,\n + \n +\n + ## reftable/stack_test.c ##\n +@@ reftable/stack_test.c: static void test_reftable_stack_add_one(void)\n + \t};\n + \tstruct reftable_ref_record dest = { NULL };\n + \tstruct stat stat_result = { 0 };\n +-\terr = reftable_new_stack(&st, dir, opts);\n ++\terr = reftable_new_stack(&st, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \terr = reftable_stack_add(st, &write_test_ref, &ref);\n +@@ reftable/stack_test.c: static void test_reftable_stack_uptodate(void)\n + \t/* simulate multi-process access to the same stack\n + \t by creating two stacks for the same directory.\n + \t */\n +-\terr = reftable_new_stack(&st1, dir, opts);\n ++\terr = reftable_new_stack(&st1, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n +-\terr = reftable_new_stack(&st2, dir, opts);\n ++\terr = reftable_new_stack(&st2, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \terr = reftable_stack_add(st1, &write_test_ref, &ref1);\n +@@ reftable/stack_test.c: static void test_reftable_stack_transaction_api(void)\n + \t};\n + \tstruct reftable_ref_record dest = { NULL };\n + \n +-\terr = reftable_new_stack(&st, dir, opts);\n ++\terr = reftable_new_stack(&st, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \treftable_addition_destroy(add);\n +@@ reftable/stack_test.c: static void test_reftable_stack_transaction_api_performs_auto_compaction(void)\n + \tstruct reftable_stack *st = NULL;\n + \tint i, n = 20, err;\n + \n +-\terr = reftable_new_stack(&st, dir, opts);\n ++\terr = reftable_new_stack(&st, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \tfor (i = 0; i <= n; i++) {\n +@@ reftable/stack_test.c: static void test_reftable_stack_auto_compaction_fails_gracefully(void)\n + \tchar *dir = get_tmp_dir(__LINE__);\n + \tint err;\n + \n +-\terr = reftable_new_stack(&st, dir, opts);\n ++\terr = reftable_new_stack(&st, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \terr = reftable_stack_add(st, write_test_ref, &ref);\n +@@ reftable/stack_test.c: static void test_reftable_stack_update_index_check(void)\n + \t\t.value.symref = \"master\",\n + \t};\n + \n +-\terr = reftable_new_stack(&st, dir, opts);\n ++\terr = reftable_new_stack(&st, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \terr = reftable_stack_add(st, &write_test_ref, &ref1);\n +@@ reftable/stack_test.c: static void test_reftable_stack_lock_failure(void)\n + \tstruct reftable_stack *st = NULL;\n + \tint err, i;\n + \n +-\terr = reftable_new_stack(&st, dir, opts);\n ++\terr = reftable_new_stack(&st, dir, &opts);\n + \tEXPECT_ERR(err);\n + \tfor (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) {\n + \t\terr = reftable_stack_add(st, &write_error, &i);\n +@@ reftable/stack_test.c: static void test_reftable_stack_add(void)\n + \tstruct stat stat_result;\n + \tint N = ARRAY_SIZE(refs);\n + \n +-\terr = reftable_new_stack(&st, dir, opts);\n ++\terr = reftable_new_stack(&st, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \tfor (i = 0; i < N; i++) {\n +@@ reftable/stack_test.c: static void test_reftable_stack_log_normalize(void)\n + \t\t.update_index = 1,\n + \t};\n + \n +-\terr = reftable_new_stack(&st, dir, opts);\n ++\terr = reftable_new_stack(&st, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \tinput.value.update.message = \"one\\ntwo\";\n +@@ reftable/stack_test.c: static void test_reftable_stack_tombstone(void)\n + \tstruct reftable_ref_record dest = { NULL };\n + \tstruct reftable_log_record log_dest = { NULL };\n + \n +-\terr = reftable_new_stack(&st, dir, opts);\n ++\terr = reftable_new_stack(&st, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \t/* even entries add the refs, odd entries delete them. */\n +@@ reftable/stack_test.c: static void test_reftable_stack_hash_id(void)\n + \tstruct reftable_stack *st_default = NULL;\n + \tstruct reftable_ref_record dest = { NULL };\n + \n +-\terr = reftable_new_stack(&st, dir, opts);\n ++\terr = reftable_new_stack(&st, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \terr = reftable_stack_add(st, &write_test_ref, &ref);\n + \tEXPECT_ERR(err);\n + \n + \t/* can't read it with the wrong hash ID. */\n +-\terr = reftable_new_stack(&st32, dir, opts32);\n ++\terr = reftable_new_stack(&st32, dir, &opts32);\n + \tEXPECT(err == REFTABLE_FORMAT_ERROR);\n + \n + \t/* check that we can read it back with default opts too. */\n +-\terr = reftable_new_stack(&st_default, dir, opts_default);\n ++\terr = reftable_new_stack(&st_default, dir, &opts_default);\n + \tEXPECT_ERR(err);\n + \n + \terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n +@@ reftable/stack_test.c: static void test_reflog_expire(void)\n + \t};\n + \tstruct reftable_log_record log = { NULL };\n + \n +-\terr = reftable_new_stack(&st, dir, opts);\n ++\terr = reftable_new_stack(&st, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \tfor (i = 1; i <= N; i++) {\n +@@ reftable/stack_test.c: static void test_empty_add(void)\n + \tchar *dir = get_tmp_dir(__LINE__);\n + \tstruct reftable_stack *st2 = NULL;\n + \n +-\terr = reftable_new_stack(&st, dir, opts);\n ++\terr = reftable_new_stack(&st, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \terr = reftable_stack_add(st, &write_nothing, NULL);\n + \tEXPECT_ERR(err);\n + \n +-\terr = reftable_new_stack(&st2, dir, opts);\n ++\terr = reftable_new_stack(&st2, dir, &opts);\n + \tEXPECT_ERR(err);\n + \tclear_dir(dir);\n + \treftable_stack_destroy(st);\n +@@ reftable/stack_test.c: static void test_reftable_stack_auto_compaction(void)\n + \tint err, i;\n + \tint N = 100;\n + \n +-\terr = reftable_new_stack(&st, dir, opts);\n ++\terr = reftable_new_stack(&st, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \tfor (i = 0; i < N; i++) {\n +@@ reftable/stack_test.c: static void test_reftable_stack_add_performs_auto_compaction(void)\n + \tchar *dir = get_tmp_dir(__LINE__);\n + \tint err, i, n = 20;\n + \n +-\terr = reftable_new_stack(&st, dir, opts);\n ++\terr = reftable_new_stack(&st, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \tfor (i = 0; i <= n; i++) {\n +@@ reftable/stack_test.c: static void test_reftable_stack_compaction_concurrent(void)\n + \tint err, i;\n + \tint N = 3;\n + \n +-\terr = reftable_new_stack(&st1, dir, opts);\n ++\terr = reftable_new_stack(&st1, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \tfor (i = 0; i < N; i++) {\n +@@ reftable/stack_test.c: static void test_reftable_stack_compaction_concurrent(void)\n + \t\tEXPECT_ERR(err);\n \t}\n \n - \twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush,\n --\t\t\t\t &tab_fd, &st->opts);\n -+\t\t\t\t &tab_fd, st->opts);\n - \terr = stack_write_compact(st, wr, first, last, config);\n - \tif (err < 0)\n - \t\tgoto done;\n +-\terr = reftable_new_stack(&st2, dir, opts);\n ++\terr = reftable_new_stack(&st2, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \terr = reftable_stack_compact_all(st1, NULL);\n +@@ reftable/stack_test.c: static void test_reftable_stack_compaction_concurrent_clean(void)\n + \tint err, i;\n + \tint N = 3;\n + \n +-\terr = reftable_new_stack(&st1, dir, opts);\n ++\terr = reftable_new_stack(&st1, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \tfor (i = 0; i < N; i++) {\n +@@ reftable/stack_test.c: static void test_reftable_stack_compaction_concurrent_clean(void)\n + \t\tEXPECT_ERR(err);\n + \t}\n + \n +-\terr = reftable_new_stack(&st2, dir, opts);\n ++\terr = reftable_new_stack(&st2, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \terr = reftable_stack_compact_all(st1, NULL);\n +@@ reftable/stack_test.c: static void test_reftable_stack_compaction_concurrent_clean(void)\n + \tunclean_stack_close(st1);\n + \tunclean_stack_close(st2);\n + \n +-\terr = reftable_new_stack(&st3, dir, opts);\n ++\terr = reftable_new_stack(&st3, dir, &opts);\n + \tEXPECT_ERR(err);\n + \n + \terr = reftable_stack_clean(st3);\n \n ## reftable/writer.c ##\n @@ reftable/writer.c: static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n @@ reftable/writer.c: static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\t int (*flush_func)(void *),\n -\t\t void *writer_arg, struct reftable_write_options *opts)\n -+\t\t void *writer_arg, struct reftable_write_options opts)\n ++\t\t void *writer_arg, const struct reftable_write_options *_opts)\n {\n \tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n - \tstrbuf_init(&wp->block_writer_data.last_key, 0);\n +-\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n -\toptions_set_defaults(opts);\n -\tif (opts->block_size >= (1 << 24)) {\n ++\tstruct reftable_write_options opts = {0};\n +\n ++\tif (_opts)\n ++\t\topts = *_opts;\n +\toptions_set_defaults(&opts);\n +\tif (opts.block_size >= (1 << 24)) {\n \t\t/* TODO - error return? */\n \t\tabort();\n \t}\n +\n ++\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n \twp->last_key = reftable_empty_strbuf;\n -\tREFTABLE_CALLOC_ARRAY(wp->block, opts->block_size);\n +\tREFTABLE_CALLOC_ARRAY(wp->block, opts.block_size);\n 3: aa2903e3e5 ! 3: ef14bf7195 reftable/writer: drop static variable used to initialize strbuf\n @@ reftable/writer.c: static void writer_reinit_block_writer(struct reftable_writer\n struct reftable_writer *\n reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\t int (*flush_func)(void *),\n - \t\t void *writer_arg, struct reftable_write_options opts)\n - {\n - \tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n --\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n - \n - \toptions_set_defaults(&opts);\n - \tif (opts.block_size >= (1 << 24)) {\n @@ reftable/writer.c: reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n - \t\tabort();\n \t}\n \n + \tstrbuf_init(&wp->block_writer_data.last_key, 0);\n -\twp->last_key = reftable_empty_strbuf;\n -+\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n +\tstrbuf_init(&wp->last_key, 0);\n \tREFTABLE_CALLOC_ARRAY(wp->block, opts.block_size);\n \twp->write = writer_func;\n 4: 5e7cbb7b19 ! 4: 8ec26646f2 reftable/writer: improve error when passed an invalid block size\n @@ Commit message\n \n ## reftable/writer.c ##\n @@ reftable/writer.c: reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n - \tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n - \n + \tif (_opts)\n + \t\topts = *_opts;\n \toptions_set_defaults(&opts);\n -\tif (opts.block_size >= (1 << 24)) {\n -\t\t/* TODO - error return? */\n 5: ed1c150d90 = 5: c4377180ef reftable/dump: support dumping a table's block structure\n 6: be5bdc6dc1 = 6: 70720af4d3 refs/reftable: allow configuring block size\n 7: 05e8d1df2d = 7: b3fe81b7b7 reftable: use `uint16_t` to track restart interval\n 8: bc0bf65553 ! 8: 2b15795707 refs/reftable: allow configuring restart interval\n @@ Documentation/config/reftable.txt: readers during access.\n +\n +reftable.restartInterval::\n +\tThe interval at which to create restart points. The reftable backend\n -+\tdetermines the restart points at file creation. The process is\n -+\tarbitrary, but every 16 or 64 records is recommended. Every 16 may be\n ++\tdetermines the restart points at file creation. Every 16 may be\n +\tmore suitable for smaller block sizes (4k or 8k), every 64 for larger\n +\tblock sizes (64k).\n ++\n 9: 6bc240fd0c = 9: b128d584a5 refs/reftable: allow disabling writing the object index\n10: 9d4c1f0340 ! 10: fb1ca02e77 reftable: make the compaction factor configurable\n @@ reftable/stack.c: license that can be found in the LICENSE file or at\n +#include \"constants.h\"\n #include \"merged.h\"\n #include \"reader.h\"\n - #include \"refname.h\"\n + #include \"reftable-error.h\"\n @@ reftable/stack.c: static int segment_size(struct segment *s)\n \treturn s->end - s->start;\n }\n11: e1282e53fb = 11: d341741eb0 refs/reftable: allow configuring geometric factor\n-- \n2.45.GIT\n\n"},{"id":"494574","messageId":"71f4e31cf7c03b7679fc7d5c018063b938a6ee35.1715587849.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715587849.git.ps@pks.im","subject":"[PATCH v3 01/11] reftable: consistently refer to `reftable_write_options` as `opts`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-13T08:17:54Z","receivedAt":"2024-05-13T08:17:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Throughout the reftable library the `reftable_write_options` are\nsometimes referred to as `cfg` and sometimes as `opts`. Unify these to\nconsistently use `opts` to avoid confusion.\n\nWhile at it, touch up the coding style a bit by removing unneeded braces\naround one-line statements and newlines between variable declarations.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/dump.c | 4 +-\n reftable/reftable-stack.h | 2 +-\n reftable/stack.c | 41 +++++++-------\n reftable/stack.h | 2 +-\n reftable/stack_test.c | 114 +++++++++++++++++---------------------\n 5 files changed, 74 insertions(+), 89 deletions(-)\n\ndiff --git a/reftable/dump.c b/reftable/dump.c\nindex 26e0393c7d..9c770a10cc 100644\n--- a/reftable/dump.c\n+++ b/reftable/dump.c\n@@ -27,9 +27,9 @@ license that can be found in the LICENSE file or at\n static int compact_stack(const char *stackdir)\n {\n \tstruct reftable_stack *stack = NULL;\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \n-\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tint err = reftable_new_stack(&stack, stackdir, opts);\n \tif (err < 0)\n \t\tgoto done;\n \ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nindex 1b602dda58..9c8e4eef49 100644\n--- a/reftable/reftable-stack.h\n+++ b/reftable/reftable-stack.h\n@@ -29,7 +29,7 @@ struct reftable_stack;\n * stored in 'dir'. Typically, this should be .git/reftables.\n */\n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n-\t\t struct reftable_write_options config);\n+\t\t struct reftable_write_options opts);\n \n /* returns the update_index at which a next table should be written. */\n uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex a59ebe038d..54e7473f3a 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -54,15 +54,14 @@ static int reftable_fd_flush(void *arg)\n }\n \n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n-\t\t struct reftable_write_options config)\n+\t\t struct reftable_write_options opts)\n {\n \tstruct reftable_stack *p = reftable_calloc(1, sizeof(*p));\n \tstruct strbuf list_file_name = STRBUF_INIT;\n \tint err = 0;\n \n-\tif (config.hash_id == 0) {\n-\t\tconfig.hash_id = GIT_SHA1_FORMAT_ID;\n-\t}\n+\tif (opts.hash_id == 0)\n+\t\topts.hash_id = GIT_SHA1_FORMAT_ID;\n \n \t*dest = NULL;\n \n@@ -73,7 +72,7 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \tp->list_file = strbuf_detach(&list_file_name, NULL);\n \tp->list_fd = -1;\n \tp->reftable_dir = xstrdup(dir);\n-\tp->config = config;\n+\tp->opts = opts;\n \n \terr = reftable_stack_reload_maybe_reuse(p, 1);\n \tif (err < 0) {\n@@ -255,7 +254,7 @@ static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n \n \t/* success! */\n \terr = reftable_new_merged_table(&new_merged, new_tables,\n-\t\t\t\t\tnew_readers_len, st->config.hash_id);\n+\t\t\t\t\tnew_readers_len, st->opts.hash_id);\n \tif (err < 0)\n \t\tgoto done;\n \n@@ -578,8 +577,8 @@ static int reftable_stack_init_addition(struct reftable_addition *add,\n \t\t}\n \t\tgoto done;\n \t}\n-\tif (st->config.default_permissions) {\n-\t\tif (chmod(add->lock_file->filename.buf, st->config.default_permissions) < 0) {\n+\tif (st->opts.default_permissions) {\n+\t\tif (chmod(add->lock_file->filename.buf, st->opts.default_permissions) < 0) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n \t\t}\n@@ -678,7 +677,7 @@ int reftable_addition_commit(struct reftable_addition *add)\n \tif (err)\n \t\tgoto done;\n \n-\tif (!add->stack->config.disable_auto_compact) {\n+\tif (!add->stack->opts.disable_auto_compact) {\n \t\t/*\n \t\t * Auto-compact the stack to keep the number of tables in\n \t\t * control. It is possible that a concurrent writer is already\n@@ -756,9 +755,9 @@ int reftable_addition_add(struct reftable_addition *add,\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n-\tif (add->stack->config.default_permissions) {\n+\tif (add->stack->opts.default_permissions) {\n \t\tif (chmod(get_tempfile_path(tab_file),\n-\t\t\t add->stack->config.default_permissions)) {\n+\t\t\t add->stack->opts.default_permissions)) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n \t\t}\n@@ -766,7 +765,7 @@ int reftable_addition_add(struct reftable_addition *add,\n \ttab_fd = get_tempfile_fd(tab_file);\n \n \twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, &tab_fd,\n-\t\t\t\t &add->stack->config);\n+\t\t\t\t &add->stack->opts);\n \terr = write_table(wr, arg);\n \tif (err < 0)\n \t\tgoto done;\n@@ -849,14 +848,14 @@ static int stack_compact_locked(struct reftable_stack *st,\n \t}\n \ttab_fd = get_tempfile_fd(tab_file);\n \n-\tif (st->config.default_permissions &&\n-\t chmod(get_tempfile_path(tab_file), st->config.default_permissions) < 0) {\n+\tif (st->opts.default_permissions &&\n+\t chmod(get_tempfile_path(tab_file), st->opts.default_permissions) < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n \n \twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush,\n-\t\t\t\t &tab_fd, &st->config);\n+\t\t\t\t &tab_fd, &st->opts);\n \terr = stack_write_compact(st, wr, first, last, config);\n \tif (err < 0)\n \t\tgoto done;\n@@ -904,7 +903,7 @@ static int stack_write_compact(struct reftable_stack *st,\n \t\t\t\t st->readers[last]->max_update_index);\n \n \terr = reftable_new_merged_table(&mt, subtabs, subtabs_len,\n-\t\t\t\t\tst->config.hash_id);\n+\t\t\t\t\tst->opts.hash_id);\n \tif (err < 0) {\n \t\treftable_free(subtabs);\n \t\tgoto done;\n@@ -1094,9 +1093,9 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\tgoto done;\n \t}\n \n-\tif (st->config.default_permissions) {\n+\tif (st->opts.default_permissions) {\n \t\tif (chmod(get_lock_file_path(&tables_list_lock),\n-\t\t\t st->config.default_permissions) < 0) {\n+\t\t\t st->opts.default_permissions) < 0) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n \t\t}\n@@ -1286,7 +1285,7 @@ static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n {\n \tuint64_t *sizes =\n \t\treftable_calloc(st->merged->stack_len, sizeof(*sizes));\n-\tint version = (st->config.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2;\n+\tint version = (st->opts.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2;\n \tint overhead = header_size(version) - 1;\n \tint i = 0;\n \tfor (i = 0; i < st->merged->stack_len; i++) {\n@@ -1435,11 +1434,11 @@ int reftable_stack_clean(struct reftable_stack *st)\n int reftable_stack_print_directory(const char *stackdir, uint32_t hash_id)\n {\n \tstruct reftable_stack *stack = NULL;\n-\tstruct reftable_write_options cfg = { .hash_id = hash_id };\n+\tstruct reftable_write_options opts = { .hash_id = hash_id };\n \tstruct reftable_merged_table *merged = NULL;\n \tstruct reftable_table table = { NULL };\n \n-\tint err = reftable_new_stack(&stack, stackdir, cfg);\n+\tint err = reftable_new_stack(&stack, stackdir, opts);\n \tif (err < 0)\n \t\tgoto done;\n \ndiff --git a/reftable/stack.h b/reftable/stack.h\nindex d43efa4760..97d7ebc043 100644\n--- a/reftable/stack.h\n+++ b/reftable/stack.h\n@@ -20,7 +20,7 @@ struct reftable_stack {\n \n \tchar *reftable_dir;\n \n-\tstruct reftable_write_options config;\n+\tstruct reftable_write_options opts;\n \n \tstruct reftable_reader **readers;\n \tsize_t readers_len;\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 7889f818d1..e17ad4dc62 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -150,7 +150,7 @@ static void test_reftable_stack_add_one(void)\n \tchar *dir = get_tmp_dir(__LINE__);\n \tstruct strbuf scratch = STRBUF_INIT;\n \tint mask = umask(002);\n-\tstruct reftable_write_options cfg = {\n+\tstruct reftable_write_options opts = {\n \t\t.default_permissions = 0660,\n \t};\n \tstruct reftable_stack *st = NULL;\n@@ -163,7 +163,7 @@ static void test_reftable_stack_add_one(void)\n \t};\n \tstruct reftable_ref_record dest = { NULL };\n \tstruct stat stat_result = { 0 };\n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, &write_test_ref, &ref);\n@@ -186,7 +186,7 @@ static void test_reftable_stack_add_one(void)\n \tstrbuf_addstr(&scratch, \"/tables.list\");\n \terr = stat(scratch.buf, &stat_result);\n \tEXPECT(!err);\n-\tEXPECT((stat_result.st_mode & 0777) == cfg.default_permissions);\n+\tEXPECT((stat_result.st_mode & 0777) == opts.default_permissions);\n \n \tstrbuf_reset(&scratch);\n \tstrbuf_addstr(&scratch, dir);\n@@ -195,7 +195,7 @@ static void test_reftable_stack_add_one(void)\n \tstrbuf_addstr(&scratch, st->readers[0]->name);\n \terr = stat(scratch.buf, &stat_result);\n \tEXPECT(!err);\n-\tEXPECT((stat_result.st_mode & 0777) == cfg.default_permissions);\n+\tEXPECT((stat_result.st_mode & 0777) == opts.default_permissions);\n #else\n \t(void) stat_result;\n #endif\n@@ -209,7 +209,7 @@ static void test_reftable_stack_add_one(void)\n \n static void test_reftable_stack_uptodate(void)\n {\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st1 = NULL;\n \tstruct reftable_stack *st2 = NULL;\n \tchar *dir = get_tmp_dir(__LINE__);\n@@ -232,10 +232,10 @@ static void test_reftable_stack_uptodate(void)\n \t/* simulate multi-process access to the same stack\n \t by creating two stacks for the same directory.\n \t */\n-\terr = reftable_new_stack(&st1, dir, cfg);\n+\terr = reftable_new_stack(&st1, dir, opts);\n \tEXPECT_ERR(err);\n \n-\terr = reftable_new_stack(&st2, dir, cfg);\n+\terr = reftable_new_stack(&st2, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st1, &write_test_ref, &ref1);\n@@ -257,8 +257,7 @@ static void test_reftable_stack_uptodate(void)\n static void test_reftable_stack_transaction_api(void)\n {\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n \tstruct reftable_addition *add = NULL;\n@@ -271,8 +270,7 @@ static void test_reftable_stack_transaction_api(void)\n \t};\n \tstruct reftable_ref_record dest = { NULL };\n \n-\n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \treftable_addition_destroy(add);\n@@ -301,12 +299,12 @@ static void test_reftable_stack_transaction_api(void)\n static void test_reftable_stack_transaction_api_performs_auto_compaction(void)\n {\n \tchar *dir = get_tmp_dir(__LINE__);\n-\tstruct reftable_write_options cfg = {0};\n+\tstruct reftable_write_options opts = {0};\n \tstruct reftable_addition *add = NULL;\n \tstruct reftable_stack *st = NULL;\n \tint i, n = 20, err;\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i <= n; i++) {\n@@ -325,7 +323,7 @@ static void test_reftable_stack_transaction_api_performs_auto_compaction(void)\n \t\t * we can ensure that we indeed honor this setting and have\n \t\t * better control over when exactly auto compaction runs.\n \t\t */\n-\t\tst->config.disable_auto_compact = i != n;\n+\t\tst->opts.disable_auto_compact = i != n;\n \n \t\terr = reftable_stack_new_addition(&add, st);\n \t\tEXPECT_ERR(err);\n@@ -361,13 +359,13 @@ static void test_reftable_stack_auto_compaction_fails_gracefully(void)\n \t\t.value_type = REFTABLE_REF_VAL1,\n \t\t.value.val1 = {0x01},\n \t};\n-\tstruct reftable_write_options cfg = {0};\n+\tstruct reftable_write_options opts = {0};\n \tstruct reftable_stack *st;\n \tstruct strbuf table_path = STRBUF_INIT;\n \tchar *dir = get_tmp_dir(__LINE__);\n \tint err;\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, write_test_ref, &ref);\n@@ -404,8 +402,7 @@ static int write_error(struct reftable_writer *wr, void *arg)\n static void test_reftable_stack_update_index_check(void)\n {\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n \tstruct reftable_ref_record ref1 = {\n@@ -421,7 +418,7 @@ static void test_reftable_stack_update_index_check(void)\n \t\t.value.symref = \"master\",\n \t};\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, &write_test_ref, &ref1);\n@@ -436,12 +433,11 @@ static void test_reftable_stack_update_index_check(void)\n static void test_reftable_stack_lock_failure(void)\n {\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err, i;\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \tfor (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) {\n \t\terr = reftable_stack_add(st, &write_error, &i);\n@@ -456,7 +452,7 @@ static void test_reftable_stack_add(void)\n {\n \tint i = 0;\n \tint err = 0;\n-\tstruct reftable_write_options cfg = {\n+\tstruct reftable_write_options opts = {\n \t\t.exact_log_message = 1,\n \t\t.default_permissions = 0660,\n \t\t.disable_auto_compact = 1,\n@@ -469,7 +465,7 @@ static void test_reftable_stack_add(void)\n \tstruct stat stat_result;\n \tint N = ARRAY_SIZE(refs);\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i < N; i++) {\n@@ -528,7 +524,7 @@ static void test_reftable_stack_add(void)\n \tstrbuf_addstr(&path, \"/tables.list\");\n \terr = stat(path.buf, &stat_result);\n \tEXPECT(!err);\n-\tEXPECT((stat_result.st_mode & 0777) == cfg.default_permissions);\n+\tEXPECT((stat_result.st_mode & 0777) == opts.default_permissions);\n \n \tstrbuf_reset(&path);\n \tstrbuf_addstr(&path, dir);\n@@ -537,7 +533,7 @@ static void test_reftable_stack_add(void)\n \tstrbuf_addstr(&path, st->readers[0]->name);\n \terr = stat(path.buf, &stat_result);\n \tEXPECT(!err);\n-\tEXPECT((stat_result.st_mode & 0777) == cfg.default_permissions);\n+\tEXPECT((stat_result.st_mode & 0777) == opts.default_permissions);\n #else\n \t(void) stat_result;\n #endif\n@@ -555,7 +551,7 @@ static void test_reftable_stack_add(void)\n static void test_reftable_stack_log_normalize(void)\n {\n \tint err = 0;\n-\tstruct reftable_write_options cfg = {\n+\tstruct reftable_write_options opts = {\n \t\t0,\n \t};\n \tstruct reftable_stack *st = NULL;\n@@ -579,7 +575,7 @@ static void test_reftable_stack_log_normalize(void)\n \t\t.update_index = 1,\n \t};\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \tinput.value.update.message = \"one\\ntwo\";\n@@ -612,8 +608,7 @@ static void test_reftable_stack_tombstone(void)\n {\n \tint i = 0;\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n \tstruct reftable_ref_record refs[2] = { { NULL } };\n@@ -622,8 +617,7 @@ static void test_reftable_stack_tombstone(void)\n \tstruct reftable_ref_record dest = { NULL };\n \tstruct reftable_log_record log_dest = { NULL };\n \n-\n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \t/* even entries add the refs, odd entries delete them. */\n@@ -691,8 +685,7 @@ static void test_reftable_stack_tombstone(void)\n static void test_reftable_stack_hash_id(void)\n {\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n \n@@ -702,24 +695,24 @@ static void test_reftable_stack_hash_id(void)\n \t\t.value.symref = \"target\",\n \t\t.update_index = 1,\n \t};\n-\tstruct reftable_write_options cfg32 = { .hash_id = GIT_SHA256_FORMAT_ID };\n+\tstruct reftable_write_options opts32 = { .hash_id = GIT_SHA256_FORMAT_ID };\n \tstruct reftable_stack *st32 = NULL;\n-\tstruct reftable_write_options cfg_default = { 0 };\n+\tstruct reftable_write_options opts_default = { 0 };\n \tstruct reftable_stack *st_default = NULL;\n \tstruct reftable_ref_record dest = { NULL };\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, &write_test_ref, &ref);\n \tEXPECT_ERR(err);\n \n \t/* can't read it with the wrong hash ID. */\n-\terr = reftable_new_stack(&st32, dir, cfg32);\n+\terr = reftable_new_stack(&st32, dir, opts32);\n \tEXPECT(err == REFTABLE_FORMAT_ERROR);\n \n-\t/* check that we can read it back with default config too. */\n-\terr = reftable_new_stack(&st_default, dir, cfg_default);\n+\t/* check that we can read it back with default opts too. */\n+\terr = reftable_new_stack(&st_default, dir, opts_default);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n@@ -752,8 +745,7 @@ static void test_suggest_compaction_segment_nothing(void)\n static void test_reflog_expire(void)\n {\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tstruct reftable_log_record logs[20] = { { NULL } };\n \tint N = ARRAY_SIZE(logs) - 1;\n@@ -764,8 +756,7 @@ static void test_reflog_expire(void)\n \t};\n \tstruct reftable_log_record log = { NULL };\n \n-\n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 1; i <= N; i++) {\n@@ -828,21 +819,19 @@ static int write_nothing(struct reftable_writer *wr, void *arg)\n \n static void test_empty_add(void)\n {\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tint err;\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n \tstruct reftable_stack *st2 = NULL;\n \n-\n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, &write_nothing, NULL);\n \tEXPECT_ERR(err);\n \n-\terr = reftable_new_stack(&st2, dir, cfg);\n+\terr = reftable_new_stack(&st2, dir, opts);\n \tEXPECT_ERR(err);\n \tclear_dir(dir);\n \treftable_stack_destroy(st);\n@@ -861,16 +850,15 @@ static int fastlog2(uint64_t sz)\n \n static void test_reftable_stack_auto_compaction(void)\n {\n-\tstruct reftable_write_options cfg = {\n+\tstruct reftable_write_options opts = {\n \t\t.disable_auto_compact = 1,\n \t};\n \tstruct reftable_stack *st = NULL;\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n \tint err, i;\n \tint N = 100;\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i < N; i++) {\n@@ -900,13 +888,13 @@ static void test_reftable_stack_auto_compaction(void)\n \n static void test_reftable_stack_add_performs_auto_compaction(void)\n {\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st = NULL;\n \tstruct strbuf refname = STRBUF_INIT;\n \tchar *dir = get_tmp_dir(__LINE__);\n \tint err, i, n = 20;\n \n-\terr = reftable_new_stack(&st, dir, cfg);\n+\terr = reftable_new_stack(&st, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i <= n; i++) {\n@@ -921,7 +909,7 @@ static void test_reftable_stack_add_performs_auto_compaction(void)\n \t\t * we can ensure that we indeed honor this setting and have\n \t\t * better control over when exactly auto compaction runs.\n \t\t */\n-\t\tst->config.disable_auto_compact = i != n;\n+\t\tst->opts.disable_auto_compact = i != n;\n \n \t\tstrbuf_reset(&refname);\n \t\tstrbuf_addf(&refname, \"branch-%04d\", i);\n@@ -948,14 +936,13 @@ static void test_reftable_stack_add_performs_auto_compaction(void)\n \n static void test_reftable_stack_compaction_concurrent(void)\n {\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st1 = NULL, *st2 = NULL;\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n \tint err, i;\n \tint N = 3;\n \n-\terr = reftable_new_stack(&st1, dir, cfg);\n+\terr = reftable_new_stack(&st1, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i < N; i++) {\n@@ -972,7 +959,7 @@ static void test_reftable_stack_compaction_concurrent(void)\n \t\tEXPECT_ERR(err);\n \t}\n \n-\terr = reftable_new_stack(&st2, dir, cfg);\n+\terr = reftable_new_stack(&st2, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_compact_all(st1, NULL);\n@@ -998,14 +985,13 @@ static void unclean_stack_close(struct reftable_stack *st)\n \n static void test_reftable_stack_compaction_concurrent_clean(void)\n {\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options opts = { 0 };\n \tstruct reftable_stack *st1 = NULL, *st2 = NULL, *st3 = NULL;\n \tchar *dir = get_tmp_dir(__LINE__);\n-\n \tint err, i;\n \tint N = 3;\n \n-\terr = reftable_new_stack(&st1, dir, cfg);\n+\terr = reftable_new_stack(&st1, dir, opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i < N; i++) {\n@@ -1022,7 +1008,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void)\n \t\tEXPECT_ERR(err);\n \t}\n \n-\terr = reftable_new_stack(&st2, dir, cfg);\n+\terr = reftable_new_stack(&st2, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_compact_all(st1, NULL);\n@@ -1031,7 +1017,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void)\n \tunclean_stack_close(st1);\n \tunclean_stack_close(st2);\n \n-\terr = reftable_new_stack(&st3, dir, cfg);\n+\terr = reftable_new_stack(&st3, dir, opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_clean(st3);\n-- \n2.45.GIT\n\n"},{"id":"494575","messageId":"f1c9914a77ab9bfe27a62e324cc3eb388f4118d9.1715587849.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715587849.git.ps@pks.im","subject":"[PATCH v3 02/11] reftable: pass opts as constant pointer","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-13T08:17:59Z","receivedAt":"2024-05-13T08:18:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We sometimes pass the refatble write options as value and sometimes as a\npointer. This is quite confusing and makes the reader wonder whether the\noptions get modified sometimes.\n\nIn fact, `reftable_new_writer()` does cause the caller-provided options\nto get updated when some values aren't set up. This is quite unexpected,\nbut didn't cause any harm until now.\n\nAdapt the code so that we do not modify the caller-provided values\nanymore. While at it, refactor the code to code to consistently pass the\noptions as a constant pointer to clarify that the caller-provided opts\nwill not ever get modified.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 6 ++---\n reftable/dump.c | 2 +-\n reftable/reftable-stack.h | 2 +-\n reftable/reftable-writer.h | 2 +-\n reftable/stack.c | 7 ++++--\n reftable/stack_test.c | 48 +++++++++++++++++++-------------------\n reftable/writer.c | 17 +++++++++-----\n 7 files changed, 46 insertions(+), 38 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 010ef811b6..f8f930380d 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -129,7 +129,7 @@ static struct reftable_stack *stack_for(struct reftable_ref_store *store,\n \t\t\t\t store->base.repo->commondir, wtname_buf.buf);\n \n \t\t\tstore->err = reftable_new_stack(&stack, wt_dir.buf,\n-\t\t\t\t\t\t\tstore->write_options);\n+\t\t\t\t\t\t\t&store->write_options);\n \t\t\tassert(store->err != REFTABLE_API_ERROR);\n \t\t\tstrmap_put(&store->worktree_stacks, wtname_buf.buf, stack);\n \t\t}\n@@ -263,7 +263,7 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \t}\n \tstrbuf_addstr(&path, \"/reftable\");\n \trefs->err = reftable_new_stack(&refs->main_stack, path.buf,\n-\t\t\t\t refs->write_options);\n+\t\t\t\t &refs->write_options);\n \tif (refs->err)\n \t\tgoto done;\n \n@@ -280,7 +280,7 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \t\tstrbuf_addf(&path, \"%s/reftable\", gitdir);\n \n \t\trefs->err = reftable_new_stack(&refs->worktree_stack, path.buf,\n-\t\t\t\t\t refs->write_options);\n+\t\t\t\t\t &refs->write_options);\n \t\tif (refs->err)\n \t\t\tgoto done;\n \t}\ndiff --git a/reftable/dump.c b/reftable/dump.c\nindex 9c770a10cc..586f3eb288 100644\n--- a/reftable/dump.c\n+++ b/reftable/dump.c\n@@ -29,7 +29,7 @@ static int compact_stack(const char *stackdir)\n \tstruct reftable_stack *stack = NULL;\n \tstruct reftable_write_options opts = { 0 };\n \n-\tint err = reftable_new_stack(&stack, stackdir, opts);\n+\tint err = reftable_new_stack(&stack, stackdir, &opts);\n \tif (err < 0)\n \t\tgoto done;\n \ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nindex 9c8e4eef49..c15632c401 100644\n--- a/reftable/reftable-stack.h\n+++ b/reftable/reftable-stack.h\n@@ -29,7 +29,7 @@ struct reftable_stack;\n * stored in 'dir'. Typically, this should be .git/reftables.\n */\n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n-\t\t struct reftable_write_options opts);\n+\t\t const struct reftable_write_options *opts);\n \n /* returns the update_index at which a next table should be written. */\n uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex b601a69a40..03df3a4963 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -88,7 +88,7 @@ struct reftable_stats {\n struct reftable_writer *\n reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\t int (*flush_func)(void *),\n-\t\t void *writer_arg, struct reftable_write_options *opts);\n+\t\t void *writer_arg, const struct reftable_write_options *opts);\n \n /* Set the range of update indices for the records we will add. When writing a\n table into a stack, the min should be at least\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 54e7473f3a..d2e68be7e8 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -54,12 +54,15 @@ static int reftable_fd_flush(void *arg)\n }\n \n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n-\t\t struct reftable_write_options opts)\n+\t\t const struct reftable_write_options *_opts)\n {\n \tstruct reftable_stack *p = reftable_calloc(1, sizeof(*p));\n \tstruct strbuf list_file_name = STRBUF_INIT;\n+\tstruct reftable_write_options opts = {0};\n \tint err = 0;\n \n+\tif (_opts)\n+\t\topts = *_opts;\n \tif (opts.hash_id == 0)\n \t\topts.hash_id = GIT_SHA1_FORMAT_ID;\n \n@@ -1438,7 +1441,7 @@ int reftable_stack_print_directory(const char *stackdir, uint32_t hash_id)\n \tstruct reftable_merged_table *merged = NULL;\n \tstruct reftable_table table = { NULL };\n \n-\tint err = reftable_new_stack(&stack, stackdir, opts);\n+\tint err = reftable_new_stack(&stack, stackdir, &opts);\n \tif (err < 0)\n \t\tgoto done;\n \ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex e17ad4dc62..d15f11d712 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -163,7 +163,7 @@ static void test_reftable_stack_add_one(void)\n \t};\n \tstruct reftable_ref_record dest = { NULL };\n \tstruct stat stat_result = { 0 };\n-\terr = reftable_new_stack(&st, dir, opts);\n+\terr = reftable_new_stack(&st, dir, &opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, &write_test_ref, &ref);\n@@ -232,10 +232,10 @@ static void test_reftable_stack_uptodate(void)\n \t/* simulate multi-process access to the same stack\n \t by creating two stacks for the same directory.\n \t */\n-\terr = reftable_new_stack(&st1, dir, opts);\n+\terr = reftable_new_stack(&st1, dir, &opts);\n \tEXPECT_ERR(err);\n \n-\terr = reftable_new_stack(&st2, dir, opts);\n+\terr = reftable_new_stack(&st2, dir, &opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st1, &write_test_ref, &ref1);\n@@ -270,7 +270,7 @@ static void test_reftable_stack_transaction_api(void)\n \t};\n \tstruct reftable_ref_record dest = { NULL };\n \n-\terr = reftable_new_stack(&st, dir, opts);\n+\terr = reftable_new_stack(&st, dir, &opts);\n \tEXPECT_ERR(err);\n \n \treftable_addition_destroy(add);\n@@ -304,7 +304,7 @@ static void test_reftable_stack_transaction_api_performs_auto_compaction(void)\n \tstruct reftable_stack *st = NULL;\n \tint i, n = 20, err;\n \n-\terr = reftable_new_stack(&st, dir, opts);\n+\terr = reftable_new_stack(&st, dir, &opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i <= n; i++) {\n@@ -365,7 +365,7 @@ static void test_reftable_stack_auto_compaction_fails_gracefully(void)\n \tchar *dir = get_tmp_dir(__LINE__);\n \tint err;\n \n-\terr = reftable_new_stack(&st, dir, opts);\n+\terr = reftable_new_stack(&st, dir, &opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, write_test_ref, &ref);\n@@ -418,7 +418,7 @@ static void test_reftable_stack_update_index_check(void)\n \t\t.value.symref = \"master\",\n \t};\n \n-\terr = reftable_new_stack(&st, dir, opts);\n+\terr = reftable_new_stack(&st, dir, &opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, &write_test_ref, &ref1);\n@@ -437,7 +437,7 @@ static void test_reftable_stack_lock_failure(void)\n \tstruct reftable_stack *st = NULL;\n \tint err, i;\n \n-\terr = reftable_new_stack(&st, dir, opts);\n+\terr = reftable_new_stack(&st, dir, &opts);\n \tEXPECT_ERR(err);\n \tfor (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) {\n \t\terr = reftable_stack_add(st, &write_error, &i);\n@@ -465,7 +465,7 @@ static void test_reftable_stack_add(void)\n \tstruct stat stat_result;\n \tint N = ARRAY_SIZE(refs);\n \n-\terr = reftable_new_stack(&st, dir, opts);\n+\terr = reftable_new_stack(&st, dir, &opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i < N; i++) {\n@@ -575,7 +575,7 @@ static void test_reftable_stack_log_normalize(void)\n \t\t.update_index = 1,\n \t};\n \n-\terr = reftable_new_stack(&st, dir, opts);\n+\terr = reftable_new_stack(&st, dir, &opts);\n \tEXPECT_ERR(err);\n \n \tinput.value.update.message = \"one\\ntwo\";\n@@ -617,7 +617,7 @@ static void test_reftable_stack_tombstone(void)\n \tstruct reftable_ref_record dest = { NULL };\n \tstruct reftable_log_record log_dest = { NULL };\n \n-\terr = reftable_new_stack(&st, dir, opts);\n+\terr = reftable_new_stack(&st, dir, &opts);\n \tEXPECT_ERR(err);\n \n \t/* even entries add the refs, odd entries delete them. */\n@@ -701,18 +701,18 @@ static void test_reftable_stack_hash_id(void)\n \tstruct reftable_stack *st_default = NULL;\n \tstruct reftable_ref_record dest = { NULL };\n \n-\terr = reftable_new_stack(&st, dir, opts);\n+\terr = reftable_new_stack(&st, dir, &opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, &write_test_ref, &ref);\n \tEXPECT_ERR(err);\n \n \t/* can't read it with the wrong hash ID. */\n-\terr = reftable_new_stack(&st32, dir, opts32);\n+\terr = reftable_new_stack(&st32, dir, &opts32);\n \tEXPECT(err == REFTABLE_FORMAT_ERROR);\n \n \t/* check that we can read it back with default opts too. */\n-\terr = reftable_new_stack(&st_default, dir, opts_default);\n+\terr = reftable_new_stack(&st_default, dir, &opts_default);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_read_ref(st_default, \"master\", &dest);\n@@ -756,7 +756,7 @@ static void test_reflog_expire(void)\n \t};\n \tstruct reftable_log_record log = { NULL };\n \n-\terr = reftable_new_stack(&st, dir, opts);\n+\terr = reftable_new_stack(&st, dir, &opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 1; i <= N; i++) {\n@@ -825,13 +825,13 @@ static void test_empty_add(void)\n \tchar *dir = get_tmp_dir(__LINE__);\n \tstruct reftable_stack *st2 = NULL;\n \n-\terr = reftable_new_stack(&st, dir, opts);\n+\terr = reftable_new_stack(&st, dir, &opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_add(st, &write_nothing, NULL);\n \tEXPECT_ERR(err);\n \n-\terr = reftable_new_stack(&st2, dir, opts);\n+\terr = reftable_new_stack(&st2, dir, &opts);\n \tEXPECT_ERR(err);\n \tclear_dir(dir);\n \treftable_stack_destroy(st);\n@@ -858,7 +858,7 @@ static void test_reftable_stack_auto_compaction(void)\n \tint err, i;\n \tint N = 100;\n \n-\terr = reftable_new_stack(&st, dir, opts);\n+\terr = reftable_new_stack(&st, dir, &opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i < N; i++) {\n@@ -894,7 +894,7 @@ static void test_reftable_stack_add_performs_auto_compaction(void)\n \tchar *dir = get_tmp_dir(__LINE__);\n \tint err, i, n = 20;\n \n-\terr = reftable_new_stack(&st, dir, opts);\n+\terr = reftable_new_stack(&st, dir, &opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i <= n; i++) {\n@@ -942,7 +942,7 @@ static void test_reftable_stack_compaction_concurrent(void)\n \tint err, i;\n \tint N = 3;\n \n-\terr = reftable_new_stack(&st1, dir, opts);\n+\terr = reftable_new_stack(&st1, dir, &opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i < N; i++) {\n@@ -959,7 +959,7 @@ static void test_reftable_stack_compaction_concurrent(void)\n \t\tEXPECT_ERR(err);\n \t}\n \n-\terr = reftable_new_stack(&st2, dir, opts);\n+\terr = reftable_new_stack(&st2, dir, &opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_compact_all(st1, NULL);\n@@ -991,7 +991,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void)\n \tint err, i;\n \tint N = 3;\n \n-\terr = reftable_new_stack(&st1, dir, opts);\n+\terr = reftable_new_stack(&st1, dir, &opts);\n \tEXPECT_ERR(err);\n \n \tfor (i = 0; i < N; i++) {\n@@ -1008,7 +1008,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void)\n \t\tEXPECT_ERR(err);\n \t}\n \n-\terr = reftable_new_stack(&st2, dir, opts);\n+\terr = reftable_new_stack(&st2, dir, &opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_compact_all(st1, NULL);\n@@ -1017,7 +1017,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void)\n \tunclean_stack_close(st1);\n \tunclean_stack_close(st2);\n \n-\terr = reftable_new_stack(&st3, dir, opts);\n+\terr = reftable_new_stack(&st3, dir, &opts);\n \tEXPECT_ERR(err);\n \n \terr = reftable_stack_clean(st3);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 10eccaaa07..4cc6e2ebd8 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -122,20 +122,25 @@ static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n struct reftable_writer *\n reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\t int (*flush_func)(void *),\n-\t\t void *writer_arg, struct reftable_write_options *opts)\n+\t\t void *writer_arg, const struct reftable_write_options *_opts)\n {\n \tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n-\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n-\toptions_set_defaults(opts);\n-\tif (opts->block_size >= (1 << 24)) {\n+\tstruct reftable_write_options opts = {0};\n+\n+\tif (_opts)\n+\t\topts = *_opts;\n+\toptions_set_defaults(&opts);\n+\tif (opts.block_size >= (1 << 24)) {\n \t\t/* TODO - error return? */\n \t\tabort();\n \t}\n+\n+\tstrbuf_init(&wp->block_writer_data.last_key, 0);\n \twp->last_key = reftable_empty_strbuf;\n-\tREFTABLE_CALLOC_ARRAY(wp->block, opts->block_size);\n+\tREFTABLE_CALLOC_ARRAY(wp->block, opts.block_size);\n \twp->write = writer_func;\n \twp->write_arg = writer_arg;\n-\twp->opts = *opts;\n+\twp->opts = opts;\n \twp->flush = flush_func;\n \twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n \n-- \n2.45.GIT\n\n"},{"id":"494576","messageId":"ef14bf7195a492872fa069254f9c029c203a85bd.1715587849.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715587849.git.ps@pks.im","subject":"[PATCH v3 03/11] reftable/writer: drop static variable used to initialize strbuf","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-13T08:18:04Z","receivedAt":"2024-05-13T08:18:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have a static variable in the reftable writer code that is merely\nused to initialize the `last_key` of the writer. Convert the code to\ninstead use `strbuf_init()` and drop the variable.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/writer.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 4cc6e2ebd8..a043025b01 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -117,8 +117,6 @@ static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n \tw->block_writer->restart_interval = w->opts.restart_interval;\n }\n \n-static struct strbuf reftable_empty_strbuf = STRBUF_INIT;\n-\n struct reftable_writer *\n reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\t int (*flush_func)(void *),\n@@ -136,7 +134,7 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t}\n \n \tstrbuf_init(&wp->block_writer_data.last_key, 0);\n-\twp->last_key = reftable_empty_strbuf;\n+\tstrbuf_init(&wp->last_key, 0);\n \tREFTABLE_CALLOC_ARRAY(wp->block, opts.block_size);\n \twp->write = writer_func;\n \twp->write_arg = writer_arg;\n-- \n2.45.GIT\n\n"},{"id":"494577","messageId":"8ec26646f2653aa77076ca6498d7618e7c7dffa9.1715587849.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715587849.git.ps@pks.im","subject":"[PATCH v3 04/11] reftable/writer: improve error when passed an invalid block size","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-13T08:18:09Z","receivedAt":"2024-05-13T08:18:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable format only supports block sizes up to 16MB. When the\nwriter is being passed a value bigger than that it simply calls\nabort(3P), which isn't all that helpful due to the lack of a proper\nerror message.\n\nImprove this by calling `BUG()` instead.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/writer.c | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex a043025b01..45b3e9ce1f 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -128,10 +128,8 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \tif (_opts)\n \t\topts = *_opts;\n \toptions_set_defaults(&opts);\n-\tif (opts.block_size >= (1 << 24)) {\n-\t\t/* TODO - error return? */\n-\t\tabort();\n-\t}\n+\tif (opts.block_size >= (1 << 24))\n+\t\tBUG(\"configured block size exceeds 16MB\");\n \n \tstrbuf_init(&wp->block_writer_data.last_key, 0);\n \tstrbuf_init(&wp->last_key, 0);\n-- \n2.45.GIT\n\n"},{"id":"494578","messageId":"c4377180ef9c92a28fa71b1cde90d80d901c8710.1715587849.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715587849.git.ps@pks.im","subject":"[PATCH v3 05/11] reftable/dump: support dumping a table's block structure","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-13T08:18:13Z","receivedAt":"2024-05-13T08:18:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're about to introduce new configs that will allow users to have more\ncontrol over how exactly reftables are written. To verify that these\nconfigs are effective we will need to take a peak into the actual blocks\nwritten by the reftable backend.\n\nIntroduce a new mode to the dumping logic that prints out the block\nstructure. This logic can be invoked via `test-tool dump-reftables -b`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/dump.c | 8 ++-\n reftable/reader.c | 63 ++++++++++++++++++\n reftable/reftable-reader.h | 2 +\n t/t0613-reftable-write-options.sh | 102 ++++++++++++++++++++++++++++++\n 4 files changed, 174 insertions(+), 1 deletion(-)\n create mode 100755 t/t0613-reftable-write-options.sh\n\ndiff --git a/reftable/dump.c b/reftable/dump.c\nindex 586f3eb288..41abbb8ecf 100644\n--- a/reftable/dump.c\n+++ b/reftable/dump.c\n@@ -48,6 +48,7 @@ static void print_help(void)\n \tprintf(\"usage: dump [-cst] arg\\n\\n\"\n \t \"options: \\n\"\n \t \" -c compact\\n\"\n+\t \" -b dump blocks\\n\"\n \t \" -t dump table\\n\"\n \t \" -s dump stack\\n\"\n \t \" -6 sha256 hash format\\n\"\n@@ -58,6 +59,7 @@ static void print_help(void)\n int reftable_dump_main(int argc, char *const *argv)\n {\n \tint err = 0;\n+\tint opt_dump_blocks = 0;\n \tint opt_dump_table = 0;\n \tint opt_dump_stack = 0;\n \tint opt_compact = 0;\n@@ -67,6 +69,8 @@ int reftable_dump_main(int argc, char *const *argv)\n \tfor (; argc > 1; argv++, argc--)\n \t\tif (*argv[1] != '-')\n \t\t\tbreak;\n+\t\telse if (!strcmp(\"-b\", argv[1]))\n+\t\t\topt_dump_blocks = 1;\n \t\telse if (!strcmp(\"-t\", argv[1]))\n \t\t\topt_dump_table = 1;\n \t\telse if (!strcmp(\"-6\", argv[1]))\n@@ -88,7 +92,9 @@ int reftable_dump_main(int argc, char *const *argv)\n \n \targ = argv[1];\n \n-\tif (opt_dump_table) {\n+\tif (opt_dump_blocks) {\n+\t\terr = reftable_reader_print_blocks(arg);\n+\t} else if (opt_dump_table) {\n \t\terr = reftable_reader_print_file(arg);\n \t} else if (opt_dump_stack) {\n \t\terr = reftable_stack_print_directory(arg, opt_hash_id);\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 481dff10d4..f23c8523db 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -856,3 +856,66 @@ int reftable_reader_print_file(const char *tablename)\n \treftable_reader_free(r);\n \treturn err;\n }\n+\n+int reftable_reader_print_blocks(const char *tablename)\n+{\n+\tstruct {\n+\t\tconst char *name;\n+\t\tint type;\n+\t} sections[] = {\n+\t\t{\n+\t\t\t.name = \"ref\",\n+\t\t\t.type = BLOCK_TYPE_REF,\n+\t\t},\n+\t\t{\n+\t\t\t.name = \"obj\",\n+\t\t\t.type = BLOCK_TYPE_OBJ,\n+\t\t},\n+\t\t{\n+\t\t\t.name = \"log\",\n+\t\t\t.type = BLOCK_TYPE_LOG,\n+\t\t},\n+\t};\n+\tstruct reftable_block_source src = { 0 };\n+\tstruct table_iter ti = TABLE_ITER_INIT;\n+\tstruct reftable_reader *r = NULL;\n+\tsize_t i;\n+\tint err;\n+\n+\terr = reftable_block_source_from_file(&src, tablename);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\terr = reftable_new_reader(&r, &src, tablename);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n+\tprintf(\"header:\\n\");\n+\tprintf(\" block_size: %d\\n\", r->block_size);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(sections); i++) {\n+\t\terr = reader_start(r, &ti, sections[i].type, 0);\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t\tif (err > 0)\n+\t\t\tcontinue;\n+\n+\t\tprintf(\"%s:\\n\", sections[i].name);\n+\n+\t\twhile (1) {\n+\t\t\tprintf(\" - length: %u\\n\", ti.br.block_len);\n+\t\t\tprintf(\" restarts: %u\\n\", ti.br.restart_count);\n+\n+\t\t\terr = table_iter_next_block(&ti);\n+\t\t\tif (err < 0)\n+\t\t\t\tgoto done;\n+\t\t\tif (err > 0)\n+\t\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+done:\n+\treftable_reader_free(r);\n+\ttable_iter_close(&ti);\n+\treturn err;\n+}\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nindex 4a4bc2fdf8..4a04857773 100644\n--- a/reftable/reftable-reader.h\n+++ b/reftable/reftable-reader.h\n@@ -97,5 +97,7 @@ void reftable_table_from_reader(struct reftable_table *tab,\n \n /* print table onto stdout for debugging. */\n int reftable_reader_print_file(const char *tablename);\n+/* print blocks onto stdout for debugging. */\n+int reftable_reader_print_blocks(const char *tablename);\n \n #endif\ndiff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh\nnew file mode 100755\nindex 0000000000..462980c37c\n--- /dev/null\n+++ b/t/t0613-reftable-write-options.sh\n@@ -0,0 +1,102 @@\n+#!/bin/sh\n+\n+test_description='reftable write options'\n+\n+GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+# Disable auto-compaction for all tests as we explicitly control repacking of\n+# refs.\n+GIT_TEST_REFTABLE_AUTOCOMPACTION=false\n+export GIT_TEST_REFTABLE_AUTOCOMPACTION\n+# Block sizes depend on the hash function, so we force SHA1 here.\n+GIT_TEST_DEFAULT_HASH=sha1\n+export GIT_TEST_DEFAULT_HASH\n+# Block sizes also depend on the actual refs we write, so we force \"master\" to\n+# be the default initial branch name.\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'default write options' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tgit pack-refs &&\n+\t\tcat >expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 4096\n+\t\tref:\n+\t\t - length: 129\n+\t\t restarts: 2\n+\t\tlog:\n+\t\t - length: 262\n+\t\t restarts: 2\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'disabled reflog writes no log blocks' '\n+\ttest_config_global core.logAllRefUpdates false &&\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tgit pack-refs &&\n+\t\tcat >expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 4096\n+\t\tref:\n+\t\t - length: 129\n+\t\t restarts: 2\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'many refs results in multiple blocks' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tfor i in $(test_seq 200)\n+\t\tdo\n+\t\t\tprintf \"update refs/heads/branch-%d HEAD\\n\" \"$i\" ||\n+\t\t\treturn 1\n+\t\tdone >input &&\n+\t\tgit update-ref --stdin expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 4096\n+\t\tref:\n+\t\t - length: 4049\n+\t\t restarts: 11\n+\t\t - length: 1136\n+\t\t restarts: 3\n+\t\tlog:\n+\t\t - length: 4041\n+\t\t restarts: 4\n+\t\t - length: 4015\n+\t\t restarts: 3\n+\t\t - length: 4014\n+\t\t restarts: 3\n+\t\t - length: 4012\n+\t\t restarts: 3\n+\t\t - length: 3289\n+\t\t restarts: 3\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_done\n-- \n2.45.GIT\n\n"},{"id":"494579","messageId":"70720af4d3476e30eba2a85c2691d664e02ade31.1715587849.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715587849.git.ps@pks.im","subject":"[PATCH v3 06/11] refs/reftable: allow configuring block size","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-13T08:18:19Z","receivedAt":"2024-05-13T08:18:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Add a new option `reftable.blockSize` that allows the user to control\nthe block size used by the reftable library.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config.txt | 2 +\n Documentation/config/reftable.txt | 14 ++++++\n refs/reftable-backend.c | 31 ++++++++++++-\n t/t0613-reftable-write-options.sh | 72 +++++++++++++++++++++++++++++++\n 4 files changed, 118 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/config/reftable.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 6f649c997c..cbf0b99c44 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -498,6 +498,8 @@ include::config/rebase.txt[]\n \n include::config/receive.txt[]\n \n+include::config/reftable.txt[]\n+\n include::config/remote.txt[]\n \n include::config/remotes.txt[]\ndiff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt\nnew file mode 100644\nindex 0000000000..fa7c4be014\n--- /dev/null\n+++ b/Documentation/config/reftable.txt\n@@ -0,0 +1,14 @@\n+reftable.blockSize::\n+\tThe size in bytes used by the reftable backend when writing blocks.\n+\tThe block size is determined by the writer, and does not have to be a\n+\tpower of 2. The block size must be larger than the longest reference\n+\tname or log entry used in the repository, as references cannot span\n+\tblocks.\n++\n+Powers of two that are friendly to the virtual memory system or\n+filesystem (such as 4kB or 8kB) are recommended. Larger sizes (64kB) can\n+yield better compression, with a possible increased cost incurred by\n+readers during access.\n++\n+The largest block size is `16777215` bytes (15.99 MiB). The default value is\n+`4096` bytes (4kB). A value of `0` will use the default value.\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex f8f930380d..8d0ae9e285 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1,6 +1,7 @@\n #include \"../git-compat-util.h\"\n #include \"../abspath.h\"\n #include \"../chdir-notify.h\"\n+#include \"../config.h\"\n #include \"../environment.h\"\n #include \"../gettext.h\"\n #include \"../hash.h\"\n@@ -228,6 +229,22 @@ static int read_ref_without_reload(struct reftable_stack *stack,\n \treturn ret;\n }\n \n+static int reftable_be_config(const char *var, const char *value,\n+\t\t\t const struct config_context *ctx,\n+\t\t\t void *_opts)\n+{\n+\tstruct reftable_write_options *opts = _opts;\n+\n+\tif (!strcmp(var, \"reftable.blocksize\")) {\n+\t\tunsigned long block_size = git_config_ulong(var, value, ctx->kvi);\n+\t\tif (block_size > 16777215)\n+\t\t\tdie(\"reftable block size cannot exceed 16MB\");\n+\t\topts->block_size = block_size;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static struct ref_store *reftable_be_init(struct repository *repo,\n \t\t\t\t\t const char *gitdir,\n \t\t\t\t\t unsigned int store_flags)\n@@ -243,12 +260,24 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \tbase_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);\n \tstrmap_init(&refs->worktree_stacks);\n \trefs->store_flags = store_flags;\n-\trefs->write_options.block_size = 4096;\n+\n \trefs->write_options.hash_id = repo->hash_algo->format_id;\n \trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n \trefs->write_options.disable_auto_compact =\n \t\t!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1);\n \n+\tgit_config(reftable_be_config, &refs->write_options);\n+\n+\t/*\n+\t * It is somewhat unfortunate that we have to mirror the default block\n+\t * size of the reftable library here. But given that the write options\n+\t * wouldn't be updated by the library here, and given that we require\n+\t * the proper block size to trim reflog message so that they fit, we\n+\t * must set up a proper value here.\n+\t */\n+\tif (!refs->write_options.block_size)\n+\t\trefs->write_options.block_size = 4096;\n+\n \t/*\n \t * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.\n \t * This stack contains both the shared and the main worktree refs.\ndiff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh\nindex 462980c37c..8bdbc6ec70 100755\n--- a/t/t0613-reftable-write-options.sh\n+++ b/t/t0613-reftable-write-options.sh\n@@ -99,4 +99,76 @@ test_expect_success 'many refs results in multiple blocks' '\n \t)\n '\n \n+test_expect_success 'tiny block size leads to error' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tcat >expect <<-EOF &&\n+\t\terror: unable to compact stack: entry too large\n+\t\tEOF\n+\t\ttest_must_fail git -c reftable.blockSize=50 pack-refs 2>err &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'small block size leads to multiple ref blocks' '\n+\ttest_config_global core.logAllRefUpdates false &&\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\t\tgit -c reftable.blockSize=100 pack-refs &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 100\n+\t\tref:\n+\t\t - length: 53\n+\t\t restarts: 1\n+\t\t - length: 74\n+\t\t restarts: 1\n+\t\t - length: 38\n+\t\t restarts: 1\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'small block size fails with large reflog message' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit A &&\n+\t\tperl -e \"print \\\"a\\\" x 500\" >logmsg &&\n+\t\tcat >expect <<-EOF &&\n+\t\tfatal: update_ref failed for ref ${SQ}refs/heads/logme${SQ}: reftable: transaction failure: entry too large\n+\t\tEOF\n+\t\ttest_must_fail git -c reftable.blockSize=100 \\\n+\t\t\tupdate-ref -m \"$(cat logmsg)\" refs/heads/logme HEAD 2>err &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n+test_expect_success 'block size exceeding maximum supported size' '\n+\ttest_config_global core.logAllRefUpdates false &&\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\t\tcat >expect <<-EOF &&\n+\t\tfatal: reftable block size cannot exceed 16MB\n+\t\tEOF\n+\t\ttest_must_fail git -c reftable.blockSize=16777216 pack-refs 2>err &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n-- \n2.45.GIT\n\n"},{"id":"494580","messageId":"b3fe81b7b7f0175fa456142f9c8c502bf7f0e5fb.1715587849.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715587849.git.ps@pks.im","subject":"[PATCH v3 07/11] reftable: use `uint16_t` to track restart interval","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-13T08:18:23Z","receivedAt":"2024-05-13T08:18:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The restart interval can at most be `UINT16_MAX` as specified in the\ntechnical documentation of the reftable format. Furthermore, it cannot\never be negative. Regardless of that we use an `int` to track the\nrestart interval.\n\nChange the type to use an `uint16_t` instead.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/block.h | 2 +-\n reftable/reftable-writer.h | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/reftable/block.h b/reftable/block.h\nindex e91f3d2790..1c8f25ee6e 100644\n--- a/reftable/block.h\n+++ b/reftable/block.h\n@@ -29,7 +29,7 @@ struct block_writer {\n \tuint32_t header_off;\n \n \t/* How often to restart keys. */\n-\tint restart_interval;\n+\tuint16_t restart_interval;\n \tint hash_size;\n \n \t/* Offset of next uint8_t to write. */\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 03df3a4963..94804eaa68 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -28,7 +28,7 @@ struct reftable_write_options {\n \tunsigned skip_index_objects : 1;\n \n \t/* how often to write complete keys in each block. */\n-\tint restart_interval;\n+\tuint16_t restart_interval;\n \n \t/* 4-byte identifier (\"sha1\", \"s256\") of the hash.\n \t * Defaults to SHA1 if unset\n-- \n2.45.GIT\n\n"},{"id":"494581","messageId":"2b1579570798a325f8a5c97438b1fdeef65aa049.1715587849.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715587849.git.ps@pks.im","subject":"[PATCH v3 08/11] refs/reftable: allow configuring restart interval","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-13T08:18:28Z","receivedAt":"2024-05-13T08:18:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Add a new option `reftable.restartInterval` that allows the user to\ncontrol the restart interval when writing reftable records used by the\nreftable library.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config/reftable.txt | 18 +++++++++++++\n refs/reftable-backend.c | 5 ++++\n t/t0613-reftable-write-options.sh | 43 +++++++++++++++++++++++++++++++\n 3 files changed, 66 insertions(+)\n\ndiff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt\nindex fa7c4be014..2374be71d7 100644\n--- a/Documentation/config/reftable.txt\n+++ b/Documentation/config/reftable.txt\n@@ -12,3 +12,21 @@ readers during access.\n +\n The largest block size is `16777215` bytes (15.99 MiB). The default value is\n `4096` bytes (4kB). A value of `0` will use the default value.\n+\n+reftable.restartInterval::\n+\tThe interval at which to create restart points. The reftable backend\n+\tdetermines the restart points at file creation. Every 16 may be\n+\tmore suitable for smaller block sizes (4k or 8k), every 64 for larger\n+\tblock sizes (64k).\n++\n+More frequent restart points reduces prefix compression and increases\n+space consumed by the restart table, both of which increase file size.\n++\n+Less frequent restart points makes prefix compression more effective,\n+decreasing overall file size, with increased penalties for readers\n+walking through more records after the binary search step.\n++\n+A maximum of `65535` restart points per block is supported.\n++\n+The default value is to create restart points every 16 records. A value of `0`\n+will use the default value.\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 8d0ae9e285..a2880aabce 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -240,6 +240,11 @@ static int reftable_be_config(const char *var, const char *value,\n \t\tif (block_size > 16777215)\n \t\t\tdie(\"reftable block size cannot exceed 16MB\");\n \t\topts->block_size = block_size;\n+\t} else if (!strcmp(var, \"reftable.restartinterval\")) {\n+\t\tunsigned long restart_interval = git_config_ulong(var, value, ctx->kvi);\n+\t\tif (restart_interval > UINT16_MAX)\n+\t\t\tdie(\"reftable block size cannot exceed %u\", (unsigned)UINT16_MAX);\n+\t\topts->restart_interval = restart_interval;\n \t}\n \n \treturn 0;\ndiff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh\nindex 8bdbc6ec70..e0a5b26f58 100755\n--- a/t/t0613-reftable-write-options.sh\n+++ b/t/t0613-reftable-write-options.sh\n@@ -171,4 +171,47 @@ test_expect_success 'block size exceeding maximum supported size' '\n \t)\n '\n \n+test_expect_success 'restart interval at every single record' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tfor i in $(test_seq 10)\n+\t\tdo\n+\t\t\tprintf \"update refs/heads/branch-%d HEAD\\n\" \"$i\" ||\n+\t\t\treturn 1\n+\t\tdone >input &&\n+\t\tgit update-ref --stdin expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 4096\n+\t\tref:\n+\t\t - length: 566\n+\t\t restarts: 13\n+\t\tlog:\n+\t\t - length: 1393\n+\t\t restarts: 12\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'restart interval exceeding maximum supported interval' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tcat >expect <<-EOF &&\n+\t\tfatal: reftable block size cannot exceed 65535\n+\t\tEOF\n+\t\ttest_must_fail git -c reftable.restartInterval=65536 pack-refs 2>err &&\n+\t\ttest_cmp expect err\n+\t)\n+'\n+\n test_done\n-- \n2.45.GIT\n\n"},{"id":"494582","messageId":"b128d584a5aa3de73a5594e7996264beab424f68.1715587849.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715587849.git.ps@pks.im","subject":"[PATCH v3 09/11] refs/reftable: allow disabling writing the object index","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-13T08:18:33Z","receivedAt":"2024-05-13T08:18:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Besides the expected \"ref\" and \"log\" records, the reftable library also\nwrites \"obj\" records. These are basically a reverse mapping of object\nIDs to their respective ref records so that it becomes efficient to\nfigure out which references point to a specific object. The motivation\nfor this data structure is the \"uploadpack.allowTipSHA1InWant\" config,\nwhich allows a client to fetch any object by its hash that has a ref\npointing to it.\n\nThis reverse index is not used by Git at all though, and the expectation\nis that most hosters nowadays use \"uploadpack.allowAnySHA1InWant\". It\nmay thus be preferable for many users to disable writing these optional\nobject indices altogether to safe some precious disk space.\n\nAdd a new config \"reftable.indexObjects\" that allows the user to disable\nthe object index altogether.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config/reftable.txt | 6 +++\n refs/reftable-backend.c | 2 +\n t/t0613-reftable-write-options.sh | 69 +++++++++++++++++++++++++++++++\n 3 files changed, 77 insertions(+)\n\ndiff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt\nindex 2374be71d7..68083876fa 100644\n--- a/Documentation/config/reftable.txt\n+++ b/Documentation/config/reftable.txt\n@@ -30,3 +30,9 @@ A maximum of `65535` restart points per block is supported.\n +\n The default value is to create restart points every 16 records. A value of `0`\n will use the default value.\n+\n+reftable.indexObjects::\n+\tWhether the reftable backend shall write object blocks. Object blocks\n+\tare a reverse mapping of object ID to the references pointing to them.\n++\n+The default value is `true`.\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex a2880aabce..5ffb36770a 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -245,6 +245,8 @@ static int reftable_be_config(const char *var, const char *value,\n \t\tif (restart_interval > UINT16_MAX)\n \t\t\tdie(\"reftable block size cannot exceed %u\", (unsigned)UINT16_MAX);\n \t\topts->restart_interval = restart_interval;\n+\t} else if (!strcmp(var, \"reftable.indexobjects\")) {\n+\t\topts->skip_index_objects = !git_config_bool(var, value);\n \t}\n \n \treturn 0;\ndiff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh\nindex e0a5b26f58..e2708e11d5 100755\n--- a/t/t0613-reftable-write-options.sh\n+++ b/t/t0613-reftable-write-options.sh\n@@ -214,4 +214,73 @@ test_expect_success 'restart interval exceeding maximum supported interval' '\n \t)\n '\n \n+test_expect_success 'object index gets written by default with ref index' '\n+\ttest_config_global core.logAllRefUpdates false &&\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tfor i in $(test_seq 5)\n+\t\tdo\n+\t\t\tprintf \"update refs/heads/branch-%d HEAD\\n\" \"$i\" ||\n+\t\t\treturn 1\n+\t\tdone >input &&\n+\t\tgit update-ref --stdin expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 100\n+\t\tref:\n+\t\t - length: 53\n+\t\t restarts: 1\n+\t\t - length: 95\n+\t\t restarts: 1\n+\t\t - length: 71\n+\t\t restarts: 1\n+\t\t - length: 80\n+\t\t restarts: 1\n+\t\tobj:\n+\t\t - length: 11\n+\t\t restarts: 1\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'object index can be disabled' '\n+\ttest_config_global core.logAllRefUpdates false &&\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit initial &&\n+\t\tfor i in $(test_seq 5)\n+\t\tdo\n+\t\t\tprintf \"update refs/heads/branch-%d HEAD\\n\" \"$i\" ||\n+\t\t\treturn 1\n+\t\tdone >input &&\n+\t\tgit update-ref --stdin expect <<-EOF &&\n+\t\theader:\n+\t\t block_size: 100\n+\t\tref:\n+\t\t - length: 53\n+\t\t restarts: 1\n+\t\t - length: 95\n+\t\t restarts: 1\n+\t\t - length: 71\n+\t\t restarts: 1\n+\t\t - length: 80\n+\t\t restarts: 1\n+\t\tEOF\n+\t\ttest-tool dump-reftable -b .git/reftable/*.ref >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.45.GIT\n\n"},{"id":"494583","messageId":"fb1ca02e7762f1670883e89b2f20be224f9e9706.1715587849.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715587849.git.ps@pks.im","subject":"[PATCH v3 10/11] reftable: make the compaction factor configurable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-13T08:18:38Z","receivedAt":"2024-05-13T08:18:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When auto-compacting, the reftable library packs references such that\nthe sizes of the tables form a geometric sequence. The factor for this\ngeometric sequence is hardcoded to 2 right now. We're about to expose\nthis as a config option though, so let's expose the factor via write\noptions.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/constants.h | 1 +\n reftable/reftable-writer.h | 6 ++++++\n reftable/stack.c | 14 ++++++++++----\n reftable/stack.h | 3 ++-\n reftable/stack_test.c | 4 ++--\n 5 files changed, 21 insertions(+), 7 deletions(-)\n\ndiff --git a/reftable/constants.h b/reftable/constants.h\nindex 5eee72c4c1..f6beb843eb 100644\n--- a/reftable/constants.h\n+++ b/reftable/constants.h\n@@ -17,5 +17,6 @@ license that can be found in the LICENSE file or at\n \n #define MAX_RESTARTS ((1 << 16) - 1)\n #define DEFAULT_BLOCK_SIZE 4096\n+#define DEFAULT_GEOMETRIC_FACTOR 2\n \n #endif\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 94804eaa68..189b1f4144 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -45,6 +45,12 @@ struct reftable_write_options {\n \n \t/* boolean: Prevent auto-compaction of tables. */\n \tunsigned disable_auto_compact : 1;\n+\n+\t/*\n+\t * Geometric sequence factor used by auto-compaction to decide which\n+\t * tables to compact. Defaults to 2 if unset.\n+\t */\n+\tuint8_t auto_compaction_factor;\n };\n \n /* reftable_block_stats holds statistics for a single block type */\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex d2e68be7e8..0ebe69e81d 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -10,6 +10,7 @@ license that can be found in the LICENSE file or at\n \n #include \"../write-or-die.h\"\n #include \"system.h\"\n+#include \"constants.h\"\n #include \"merged.h\"\n #include \"reader.h\"\n #include \"reftable-error.h\"\n@@ -1212,12 +1213,16 @@ static int segment_size(struct segment *s)\n \treturn s->end - s->start;\n }\n \n-struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n+struct segment suggest_compaction_segment(uint64_t *sizes, size_t n,\n+\t\t\t\t\t uint8_t factor)\n {\n \tstruct segment seg = { 0 };\n \tuint64_t bytes;\n \tsize_t i;\n \n+\tif (!factor)\n+\t\tfactor = DEFAULT_GEOMETRIC_FACTOR;\n+\n \t/*\n \t * If there are no tables or only a single one then we don't have to\n \t * compact anything. The sequence is geometric by definition already.\n@@ -1249,7 +1254,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n \t * \t64, 32, 16, 8, 4, 3, 1\n \t */\n \tfor (i = n - 1; i > 0; i--) {\n-\t\tif (sizes[i - 1] < sizes[i] * 2) {\n+\t\tif (sizes[i - 1] < sizes[i] * factor) {\n \t\t\tseg.end = i + 1;\n \t\t\tbytes = sizes[i];\n \t\t\tbreak;\n@@ -1275,7 +1280,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n \t\tuint64_t curr = bytes;\n \t\tbytes += sizes[i - 1];\n \n-\t\tif (sizes[i - 1] < curr * 2) {\n+\t\tif (sizes[i - 1] < curr * factor) {\n \t\t\tseg.start = i - 1;\n \t\t\tseg.bytes = bytes;\n \t\t}\n@@ -1301,7 +1306,8 @@ int reftable_stack_auto_compact(struct reftable_stack *st)\n {\n \tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n \tstruct segment seg =\n-\t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n+\t\tsuggest_compaction_segment(sizes, st->merged->stack_len,\n+\t\t\t\t\t st->opts.auto_compaction_factor);\n \treftable_free(sizes);\n \tif (segment_size(&seg) > 0)\n \t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\ndiff --git a/reftable/stack.h b/reftable/stack.h\nindex 97d7ebc043..5b45cff4f7 100644\n--- a/reftable/stack.h\n+++ b/reftable/stack.h\n@@ -35,6 +35,7 @@ struct segment {\n \tuint64_t bytes;\n };\n \n-struct segment suggest_compaction_segment(uint64_t *sizes, size_t n);\n+struct segment suggest_compaction_segment(uint64_t *sizes, size_t n,\n+\t\t\t\t\t uint8_t factor);\n \n #endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex d15f11d712..0f7b1453e6 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -729,7 +729,7 @@ static void test_suggest_compaction_segment(void)\n {\n \tuint64_t sizes[] = { 512, 64, 17, 16, 9, 9, 9, 16, 2, 16 };\n \tstruct segment min =\n-\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes), 2);\n \tEXPECT(min.start == 1);\n \tEXPECT(min.end == 10);\n }\n@@ -738,7 +738,7 @@ static void test_suggest_compaction_segment_nothing(void)\n {\n \tuint64_t sizes[] = { 64, 32, 16, 8, 4, 2 };\n \tstruct segment result =\n-\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n+\t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes), 2);\n \tEXPECT(result.start == result.end);\n }\n \n-- \n2.45.GIT\n\n"},{"id":"494584","messageId":"d341741eb079633c6020020166b3fcfe667a6ac7.1715587849.git.ps@pks.im","threadId":"61406","inReplyTo":"cover.1715587849.git.ps@pks.im","subject":"[PATCH v3 11/11] refs/reftable: allow configuring geometric factor","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-13T08:18:43Z","receivedAt":"2024-05-13T08:18:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Allow configuring the geometric factor used by the auto-compaction\nalgorithm whenever a new table is appended to the stack of tables.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config/reftable.txt | 10 ++++++++++\n refs/reftable-backend.c | 5 +++++\n 2 files changed, 15 insertions(+)\n\ndiff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt\nindex 68083876fa..0515727977 100644\n--- a/Documentation/config/reftable.txt\n+++ b/Documentation/config/reftable.txt\n@@ -36,3 +36,13 @@ reftable.indexObjects::\n \tare a reverse mapping of object ID to the references pointing to them.\n +\n The default value is `true`.\n+\n+reftable.geometricFactor::\n+\tWhenever the reftable backend appends a new table to the stack, it\n+\tperforms auto compaction to ensure that there is only a handful of\n+\ttables. The backend does this by ensuring that tables form a geometric\n+\tsequence regarding the respective sizes of each table.\n++\n+By default, the geometric sequence uses a factor of 2, meaning that for any\n+table, the next-biggest table must at least be twice as big. A maximum factor\n+of 256 is supported.\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 5ffb36770a..da620fd598 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -247,6 +247,11 @@ static int reftable_be_config(const char *var, const char *value,\n \t\topts->restart_interval = restart_interval;\n \t} else if (!strcmp(var, \"reftable.indexobjects\")) {\n \t\topts->skip_index_objects = !git_config_bool(var, value);\n+\t} else if (!strcmp(var, \"reftable.geometricfactor\")) {\n+\t\tunsigned long factor = git_config_ulong(var, value, ctx->kvi);\n+\t\tif (factor > UINT8_MAX)\n+\t\t\tdie(\"reftable geometric factor cannot exceed %u\", (unsigned)UINT8_MAX);\n+\t\topts->auto_compaction_factor = factor;\n \t}\n \n \treturn 0;\n-- \n2.45.GIT\n\n"},{"id":"494638","messageId":"xmqqmsotheb5.fsf@gitster.g","threadId":"61406","inReplyTo":"ZkHHIgyeWq9FCYuq@tanuki","subject":"Re: [PATCH v2 10/11] reftable: make the compaction factor configurable","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-13T16:22:38Z","receivedAt":"2024-05-13T16:22:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> So this may be good enough for now, and when we gain the ability to\n> parse floats we may convert this to accept floats, as well. An\n> alternative would be to convert this to percent, where the default value\n> would be 200. That should give sufficient flexibility without having to\n> introduce floats.\n\nThere already is an established way to specify float with arbitrary\nprecision on the command line if we limit the value to 0..1, by the\nway.\n\nhttps://lore.kernel.org/git/Pine.LNX.4.58.0505191516350.2322@ppc970.osdl.org/\n\nIt is amusing to revisit an ancient discussion thread. I can see,\non the same thread in the discussion following that message, that\nback in May 2005, we were discussing \"intent to add\" that did not\nget invented until mid 2008 already ;-).\n"},{"id":"494677","messageId":"xmqq34qle3l5.fsf@gitster.g","threadId":"61406","inReplyTo":"ed1c150d909da836a760bc0699f585722d5988e9.1715336798.git.ps@pks.im","subject":"Re: [PATCH v2 05/11] reftable/dump: support dumping a table's block structure","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-13T22:42:30Z","receivedAt":"2024-05-13T22:42:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> We're about to introduce new configs that will allow users to have more\n> control over how exactly reftables are written. To verify that these\n> configs are effective we will need to take a peak into the actual blocks\n> written by the reftable backend.\n>\n> Introduce a new mode to the dumping logic that prints out the block\n> structure. This logic can be invoked via `test-tool dump-reftables -b`.\n\nThis step somehow looks familiar. Perhaps that is because I read it\nmore carefully than other steps during the last round.\n\nLooking good to me.\n"},{"id":"494678","messageId":"xmqqv83hcp0i.fsf@gitster.g","threadId":"61406","inReplyTo":"05e8d1df2d9921426b803d71b22ba3dba188836c.1715336798.git.ps@pks.im","subject":"Re: [PATCH v2 07/11] reftable: use `uint16_t` to track restart interval","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-13T22:42:37Z","receivedAt":"2024-05-13T22:42:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> The restart interval can at most be `UINT16_MAX` as specified in the\n> technical documentation of the reftable format. Furthermore, it cannot\n> ever be negative. Regardless of that we use an `int` to track the\n> restart interval.\n>\n> Change the type to use an `uint16_t` instead.\n\nNot wrong per-se, but this one is more or less a Meh, as we know we\ndo not work on 16-bit platforms anyway.\n"},{"id":"494697","messageId":"ZkLujyOtvKBf2LOM@tanuki","threadId":"61406","inReplyTo":"xmqqmsotheb5.fsf@gitster.g","subject":"Re: [PATCH v2 10/11] reftable: make the compaction factor configurable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-14T04:54:39Z","receivedAt":"2024-05-14T04:54:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 13, 2024 at 09:22:38AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > So this may be good enough for now, and when we gain the ability to\n> > parse floats we may convert this to accept floats, as well. An\n> > alternative would be to convert this to percent, where the default value\n> > would be 200. That should give sufficient flexibility without having to\n> > introduce floats.\n> \n> There already is an established way to specify float with arbitrary\n> precision on the command line if we limit the value to 0..1, by the\n> way.\n> \n> https://lore.kernel.org/git/Pine.LNX.4.58.0505191516350.2322@ppc970.osdl.org/\n\nThe problem is that we don't want to limit the value to 0..1, but to\n1..n. 1 really is the lowest semi-sensible number you can pick and means\nthat all tables should have the same size. And in practice, there is no\nupper limit, even though it's probably not all that reasonable to pick\nanything beyond 10.\n\nIn any case, I'd propose to keep this as-is for now. The simple reason\nis that we have preexisting commands (`git repack -g`) and in-flight\nseries (pseudo-merge bitmaps) that also use plain integers to represent\nthe geometric factor. So I'm aiming for consistency. From thereon we can\nthen iterate and design a solution that works for all of them, e.g. by\nallowing for floats outside of the 0..1 range in whatever form.\n\n> It is amusing to revisit an ancient discussion thread. I can see,\n> on the same thread in the discussion following that message, that\n> back in May 2005, we were discussing \"intent to add\" that did not\n> get invented until mid 2008 already ;-).\n\n:)\n\nPatrick\n"},{"id":"494698","messageId":"ZkLulczNVfU3zCcE@tanuki","threadId":"61406","inReplyTo":"xmqqv83hcp0i.fsf@gitster.g","subject":"Re: [PATCH v2 07/11] reftable: use `uint16_t` to track restart interval","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-14T04:54:45Z","receivedAt":"2024-05-14T04:54:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 13, 2024 at 03:42:37PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > The restart interval can at most be `UINT16_MAX` as specified in the\n> > technical documentation of the reftable format. Furthermore, it cannot\n> > ever be negative. Regardless of that we use an `int` to track the\n> > restart interval.\n> >\n> > Change the type to use an `uint16_t` instead.\n> \n> Not wrong per-se, but this one is more or less a Meh, as we know we\n> do not work on 16-bit platforms anyway.\n\nYeah, my intent isn't really \"platform portability\". It's rather that I\nthink that an `uint16_t` documents the accepted range of values much\nbetter than `int` does.\n\nPatrick\n"},{"id":"494936","messageId":"CAOLa=ZTGdM8u7PyBGis7XZvrrWLLpQXMuTdAQjYGy+0JdR7CRQ@mail.gmail.com","threadId":"61406","inReplyTo":"f1c9914a77ab9bfe27a62e324cc3eb388f4118d9.1715587849.git.ps@pks.im","subject":"Re: [PATCH v3 02/11] reftable: pass opts as constant pointer","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-05-17T08:02:28Z","receivedAt":"2024-05-17T08:02:30Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> We sometimes pass the refatble write options as value and sometimes as a\n> pointer. This is quite confusing and makes the reader wonder whether the\n> options get modified sometimes.\n>\n> In fact, `reftable_new_writer()` does cause the caller-provided options\n> to get updated when some values aren't set up. This is quite unexpected,\n> but didn't cause any harm until now.\n>\n> Adapt the code so that we do not modify the caller-provided values\n> anymore. While at it, refactor the code to code to consistently pass the\n> options as a constant pointer to clarify that the caller-provided opts\n> will not ever get modified.\n>\n\nSo from v2, we changed this from passing the value to passing the\naddress. Mostly, we're trying to stay consistent and have to pick\neither, so this makes more sense since we're passing around big structs.\n"},{"id":"494937","messageId":"CAOLa=ZSkuQ07BbJx-eSaui3WSZZTup54WnPkMSXnWwwnM=LATA@mail.gmail.com","threadId":"61406","inReplyTo":"70720af4d3476e30eba2a85c2691d664e02ade31.1715587849.git.ps@pks.im","subject":"Re: [PATCH v3 06/11] refs/reftable: allow configuring block size","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-05-17T08:09:27Z","receivedAt":"2024-05-17T08:09:29Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Add a new option `reftable.blockSize` that allows the user to control\n> the block size used by the reftable library.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> Documentation/config.txt | 2 +\n> Documentation/config/reftable.txt | 14 ++++++\n> refs/reftable-backend.c | 31 ++++++++++++-\n> t/t0613-reftable-write-options.sh | 72 +++++++++++++++++++++++++++++++\n> 4 files changed, 118 insertions(+), 1 deletion(-)\n> create mode 100644 Documentation/config/reftable.txt\n>\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index 6f649c997c..cbf0b99c44 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -498,6 +498,8 @@ include::config/rebase.txt[]\n>\n> include::config/receive.txt[]\n>\n> +include::config/reftable.txt[]\n> +\n> include::config/remote.txt[]\n>\n> include::config/remotes.txt[]\n> diff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt\n> new file mode 100644\n> index 0000000000..fa7c4be014\n> --- /dev/null\n> +++ b/Documentation/config/reftable.txt\n> @@ -0,0 +1,14 @@\n> +reftable.blockSize::\n> +\tThe size in bytes used by the reftable backend when writing blocks.\n> +\tThe block size is determined by the writer, and does not have to be a\n> +\tpower of 2. The block size must be larger than the longest reference\n> +\tname or log entry used in the repository, as references cannot span\n\nNit: s/as references cannot/as neither can/\n"},{"id":"494939","messageId":"CAOLa=ZQE2mFrDNGyNEwYXneW3Vw6wqg160WOpFL7+NfSfOfDhg@mail.gmail.com","threadId":"61406","inReplyTo":"cover.1715587849.git.ps@pks.im","subject":"Re: [PATCH v3 00/11] reftable: expose write options as config","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-05-17T08:14:19Z","receivedAt":"2024-05-17T08:14:20Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Hi,\n>\n> this is the third version of my patch series that exposes several write\n> options of the reftable library via Git configs.\n>\n> Changes compared to v2:\n>\n> - Adapted patch 2 such that we now pass options as const pointers\n> instead of by value.\n>\n> - Removed a confusing sentence in the documentation of the restart\n> points in patch 8.\n>\n> Other than that I decided to rebase this on top of the current \"master\"\n> branch at 0f3415f1f8 (The second batch, 2024-05-08). This is because the\n> revamped patch 2 would cause new conflicts with 485c63cf5c (reftable:\n> remove name checks, 2024-04-08) that didn't exist in v2 of this patch\n> series yet. Rebasing thus seemed like the more reasonable option.\n>\n\nI did go through the patches and only had a small nit, but not worth a\nre-roll.\n\nI was also wondering what happens if users tweak these values when a\nrepository already contains reftables with different values. Seems like\nit'll use the new configuration during new table creation and also\nduring autocompaction. Which makes sense.\n\nThanks\nKarthik\n"},{"id":"494959","messageId":"ZkcUyISlLsGqtOmC@tanuki","threadId":"61406","inReplyTo":"CAOLa=ZQE2mFrDNGyNEwYXneW3Vw6wqg160WOpFL7+NfSfOfDhg@mail.gmail.com","subject":"Re: [PATCH v3 00/11] reftable: expose write options as config","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-17T08:26:48Z","receivedAt":"2024-05-17T08:26:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, May 17, 2024 at 10:14:19AM +0200, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> \n> > Hi,\n> >\n> > this is the third version of my patch series that exposes several write\n> > options of the reftable library via Git configs.\n> >\n> > Changes compared to v2:\n> >\n> > - Adapted patch 2 such that we now pass options as const pointers\n> > instead of by value.\n> >\n> > - Removed a confusing sentence in the documentation of the restart\n> > points in patch 8.\n> >\n> > Other than that I decided to rebase this on top of the current \"master\"\n> > branch at 0f3415f1f8 (The second batch, 2024-05-08). This is because the\n> > revamped patch 2 would cause new conflicts with 485c63cf5c (reftable:\n> > remove name checks, 2024-04-08) that didn't exist in v2 of this patch\n> > series yet. Rebasing thus seemed like the more reasonable option.\n> >\n> \n> I did go through the patches and only had a small nit, but not worth a\n> re-roll.\n\nThanks!\n\n> I was also wondering what happens if users tweak these values when a\n> repository already contains reftables with different values. Seems like\n> it'll use the new configuration during new table creation and also\n> during autocompaction. Which makes sense.\n\nYup. It should be fine to change the values at will and for different\ntables to be written with different configs.\n\nPatrick\n"},{"id":"495252","messageId":"emcnb7fqqihc6mdjusk7wp7fr5zoxllpectcbtbhazbay273j3@us352qejrtnr","threadId":"61406","inReplyTo":"f1c9914a77ab9bfe27a62e324cc3eb388f4118d9.1715587849.git.ps@pks.im","subject":"Re: [PATCH v3 02/11] reftable: pass opts as constant pointer","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-05-21T23:22:43Z","receivedAt":"2024-05-21T23:22:45Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/05/13 10:17AM, Patrick Steinhardt wrote:\n> We sometimes pass the refatble write options as value and sometimes as a\n\ns/refatble/reftable\n\n> pointer. This is quite confusing and makes the reader wonder whether the\n> options get modified sometimes.\n> \n> In fact, `reftable_new_writer()` does cause the caller-provided options\n> to get updated when some values aren't set up. This is quite unexpected,\n> but didn't cause any harm until now.\n> \n> Adapt the code so that we do not modify the caller-provided values\n> anymore. While at it, refactor the code to code to consistently pass the\n> options as a constant pointer to clarify that the caller-provided opts\n> will not ever get modified.\n\nDoesn't really matter, but would it be more accurate to say \"pointer to\na constant type\"?\n\nOverall, I like this change. Improves consistency and readability :)\n\n-Justin\n"},{"id":"495253","messageId":"t7vrnm4kgyu5kl6evrfm6pvhwffwi5tku45tjds7mz7rpip2hz@77zv3givo7pp","threadId":"61406","inReplyTo":"c4377180ef9c92a28fa71b1cde90d80d901c8710.1715587849.git.ps@pks.im","subject":"Re: [PATCH v3 05/11] reftable/dump: support dumping a table's block structure","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-05-21T23:35:13Z","receivedAt":"2024-05-21T23:35:15Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/05/13 10:18AM, Patrick Steinhardt wrote:\n> +int reftable_reader_print_blocks(const char *tablename)\n> +{\n> +\tstruct {\n> +\t\tconst char *name;\n> +\t\tint type;\n> +\t} sections[] = {\n> +\t\t{\n> +\t\t\t.name = \"ref\",\n> +\t\t\t.type = BLOCK_TYPE_REF,\n> +\t\t},\n> +\t\t{\n> +\t\t\t.name = \"obj\",\n> +\t\t\t.type = BLOCK_TYPE_OBJ,\n> +\t\t},\n> +\t\t{\n> +\t\t\t.name = \"log\",\n> +\t\t\t.type = BLOCK_TYPE_LOG,\n> +\t\t},\n> +\t};\n\nI noticed that we are not including all the block types. Would we ever\nwant to also be able to dump index blocks? Or would they not be useful\nin this context?\n\n-Justin\n"},{"id":"495256","messageId":"2zq4lspmqkcw6ssaxgc5txaubivwvghwgf4kmsmk3nfsizbmnf@twznyofqazml","threadId":"61406","inReplyTo":"2b1579570798a325f8a5c97438b1fdeef65aa049.1715587849.git.ps@pks.im","subject":"Re: [PATCH v3 08/11] refs/reftable: allow configuring restart interval","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-05-21T23:50:58Z","receivedAt":"2024-05-21T23:51:00Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/05/13 10:18AM, Patrick Steinhardt wrote:\n> +reftable.restartInterval::\n> +\tThe interval at which to create restart points. The reftable backend\n> +\tdetermines the restart points at file creation. Every 16 may be\n> +\tmore suitable for smaller block sizes (4k or 8k), every 64 for larger\n> +\tblock sizes (64k).\n> ++\n> +More frequent restart points reduces prefix compression and increases\n> +space consumed by the restart table, both of which increase file size.\n> ++\n> +Less frequent restart points makes prefix compression more effective,\n> +decreasing overall file size, with increased penalties for readers\n> +walking through more records after the binary search step.\n> ++\n> +A maximum of `65535` restart points per block is supported.\n> ++\n> +The default value is to create restart points every 16 records. A value of `0`\n> +will use the default value.\n\nOut of curiousity, if for some reason we didn't want any prefix\ncompression, would the best way to do this be via setting the restart\ninterval to 1? I guess this means the number of references would also be\nlimited by the maximum number of restart points.\n\n-Justin\n"},{"id":"495257","messageId":"akuvjnvz43fzcggytbkyjhq6htljucucmexecel3h4eooqrcca@j5umyot7clwe","threadId":"61406","inReplyTo":"cover.1715587849.git.ps@pks.im","subject":"Re: [PATCH v3 00/11] reftable: expose write options as config","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-05-21T23:54:33Z","receivedAt":"2024-05-21T23:54:35Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/05/13 10:17AM, Patrick Steinhardt wrote:\n> Hi,\n> \n> this is the third version of my patch series that exposes several write\n> options of the reftable library via Git configs.\n> \n> Changes compared to v2:\n> \n> - Adapted patch 2 such that we now pass options as const pointers\n> instead of by value.\n> \n> - Removed a confusing sentence in the documentation of the restart\n> points in patch 8.\n> \n> Other than that I decided to rebase this on top of the current \"master\"\n> branch at 0f3415f1f8 (The second batch, 2024-05-08). This is because the\n> revamped patch 2 would cause new conflicts with 485c63cf5c (reftable:\n> remove name checks, 2024-04-08) that didn't exist in v2 of this patch\n> series yet. Rebasing thus seemed like the more reasonable option.\n\nThanks Patrick! I reviewed this version and left a couple\ncomments/questions, but nothing that would neccessitate a reroll. :)\n\n-Justin\n"},{"id":"495266","messageId":"Zk2cdW5r1Hv6bCj5@tanuki","threadId":"61406","inReplyTo":"emcnb7fqqihc6mdjusk7wp7fr5zoxllpectcbtbhazbay273j3@us352qejrtnr","subject":"Re: [PATCH v3 02/11] reftable: pass opts as constant pointer","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-22T07:19:17Z","receivedAt":"2024-05-22T07:19:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, May 21, 2024 at 06:22:43PM -0500, Justin Tobler wrote:\n> On 24/05/13 10:17AM, Patrick Steinhardt wrote:\n> > We sometimes pass the refatble write options as value and sometimes as a\n> \n> s/refatble/reftable\n> \n> > pointer. This is quite confusing and makes the reader wonder whether the\n> > options get modified sometimes.\n> > \n> > In fact, `reftable_new_writer()` does cause the caller-provided options\n> > to get updated when some values aren't set up. This is quite unexpected,\n> > but didn't cause any harm until now.\n> > \n> > Adapt the code so that we do not modify the caller-provided values\n> > anymore. While at it, refactor the code to code to consistently pass the\n> > options as a constant pointer to clarify that the caller-provided opts\n> > will not ever get modified.\n> \n> Doesn't really matter, but would it be more accurate to say \"pointer to\n> a constant type\"?\n> \n> Overall, I like this change. Improves consistency and readability :)\n\nTrue. As you mentioned, I'll not reroll this series just for these two\nsmall issues though.\n\nPatrick\n"},{"id":"495267","messageId":"Zk2chD_uAScoQJB4@tanuki","threadId":"61406","inReplyTo":"t7vrnm4kgyu5kl6evrfm6pvhwffwi5tku45tjds7mz7rpip2hz@77zv3givo7pp","subject":"Re: [PATCH v3 05/11] reftable/dump: support dumping a table's block structure","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-22T07:19:32Z","receivedAt":"2024-05-22T07:19:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, May 21, 2024 at 06:35:13PM -0500, Justin Tobler wrote:\n> On 24/05/13 10:18AM, Patrick Steinhardt wrote:\n> > +int reftable_reader_print_blocks(const char *tablename)\n> > +{\n> > +\tstruct {\n> > +\t\tconst char *name;\n> > +\t\tint type;\n> > +\t} sections[] = {\n> > +\t\t{\n> > +\t\t\t.name = \"ref\",\n> > +\t\t\t.type = BLOCK_TYPE_REF,\n> > +\t\t},\n> > +\t\t{\n> > +\t\t\t.name = \"obj\",\n> > +\t\t\t.type = BLOCK_TYPE_OBJ,\n> > +\t\t},\n> > +\t\t{\n> > +\t\t\t.name = \"log\",\n> > +\t\t\t.type = BLOCK_TYPE_LOG,\n> > +\t\t},\n> > +\t};\n> \n> I noticed that we are not including all the block types. Would we ever\n> want to also be able to dump index blocks? Or would they not be useful\n> in this context?\n\nMaybe. It wasn't really necessary to include index blocks in this\ncontext as I was already able to extract all relevant information\nwithout them. So I decided to skip them. We may iterate on this in the\nfuture as required.\n\nPatrick\n"},{"id":"495268","messageId":"Zk2ciqJje6cH11tz@tanuki","threadId":"61406","inReplyTo":"2zq4lspmqkcw6ssaxgc5txaubivwvghwgf4kmsmk3nfsizbmnf@twznyofqazml","subject":"Re: [PATCH v3 08/11] refs/reftable: allow configuring restart interval","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-22T07:19:38Z","receivedAt":"2024-05-22T07:19:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, May 21, 2024 at 06:50:58PM -0500, Justin Tobler wrote:\n> On 24/05/13 10:18AM, Patrick Steinhardt wrote:\n> > +reftable.restartInterval::\n> > +\tThe interval at which to create restart points. The reftable backend\n> > +\tdetermines the restart points at file creation. Every 16 may be\n> > +\tmore suitable for smaller block sizes (4k or 8k), every 64 for larger\n> > +\tblock sizes (64k).\n> > ++\n> > +More frequent restart points reduces prefix compression and increases\n> > +space consumed by the restart table, both of which increase file size.\n> > ++\n> > +Less frequent restart points makes prefix compression more effective,\n> > +decreasing overall file size, with increased penalties for readers\n> > +walking through more records after the binary search step.\n> > ++\n> > +A maximum of `65535` restart points per block is supported.\n> > ++\n> > +The default value is to create restart points every 16 records. A value of `0`\n> > +will use the default value.\n> \n> Out of curiousity, if for some reason we didn't want any prefix\n> compression, would the best way to do this be via setting the restart\n> interval to 1? I guess this means the number of references would also be\n> limited by the maximum number of restart points.\n\nYup, exactly.\n\nPatrick\n"},{"id":"495269","messageId":"Zk2ckCckDT8WhI9k@tanuki","threadId":"61406","inReplyTo":"akuvjnvz43fzcggytbkyjhq6htljucucmexecel3h4eooqrcca@j5umyot7clwe","subject":"Re: [PATCH v3 00/11] reftable: expose write options as config","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-22T07:19:44Z","receivedAt":"2024-05-22T07:19:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, May 21, 2024 at 06:54:33PM -0500, Justin Tobler wrote:\n> On 24/05/13 10:17AM, Patrick Steinhardt wrote:\n> > Hi,\n> > \n> > this is the third version of my patch series that exposes several write\n> > options of the reftable library via Git configs.\n> > \n> > Changes compared to v2:\n> > \n> > - Adapted patch 2 such that we now pass options as const pointers\n> > instead of by value.\n> > \n> > - Removed a confusing sentence in the documentation of the restart\n> > points in patch 8.\n> > \n> > Other than that I decided to rebase this on top of the current \"master\"\n> > branch at 0f3415f1f8 (The second batch, 2024-05-08). This is because the\n> > revamped patch 2 would cause new conflicts with 485c63cf5c (reftable:\n> > remove name checks, 2024-04-08) that didn't exist in v2 of this patch\n> > series yet. Rebasing thus seemed like the more reasonable option.\n> \n> Thanks Patrick! I reviewed this version and left a couple\n> comments/questions, but nothing that would neccessitate a reroll. :)\n\nThanks for your review! As mentioned, I agree that there is no strong\nmotivator to reroll this series as the only changes would be a typo fix\nin the second patch.\n\nPatrick\n"}]}