{"thread":{"id":"61055","subject":"[PATCH] reftable/stack: use geometric table compaction","startedAt":"2024-03-05T20:03:50Z","lastAt":"2024-04-08T19:12:22Z","messageCount":52,"participants":["Justin Tobler via GitGitGadget","Patrick Steinhardt","Justin Tobler","Karthik Nayak","Junio C Hamano","Han-Wen Nienhuys"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"489982","messageId":"pull.1683.git.1709669025722.gitgitgadget@gmail.com","threadId":"61055","inReplyTo":null,"subject":"[PATCH] reftable/stack: use geometric table compaction","fromName":"Justin Tobler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-03-05T20:03:45Z","receivedAt":"2024-03-05T20:03:50Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"From: Justin Tobler \n\nTo reduce the number of on-disk reftables, compaction is performed.\nContiguous tables with the same binary log value of size are grouped\ninto segments. The segment that has both the lowest binary log value and\ncontains more than one table is set as the starting point when\nidentifying the compaction segment.\n\nSince segments containing a single table are not initially considered\nfor compaction, if the table appended to the list does not match the\nprevious table log value, no compaction occurs for the new table. It is\ntherefore possible for unbounded growth of the table list. This can be\ndemonstrated by repeating the following sequence:\n\ngit branch -f foo\ngit branch -d foo\n\nEach operation results in a new table being written with no compaction\noccurring until a separate operation produces a table matching the\nprevious table log value.\n\nTo avoid unbounded growth of the table list, walk through each table and\nevaluate if it needs to be included in the compaction segment to restore\na geometric sequence.\n\nSome tests in `t0610-reftable-basics.sh` assert the on-disk state of\ntables and are therefore updated to specify the correct new table count.\nSince compaction is more aggressive in ensuring tables maintain a\ngeometric sequence, the expected table count is reduced in these tests.\nIn `reftable/stack_test.c` tests related to `sizes_to_segments()` are\nremoved because the function is no longer needed. Also, the\n`test_suggest_compaction_segment()` test is updated to better showcase\nand reflect the new geometric compaction behavior.\n\nSigned-off-by: Justin Tobler \n---\n reftable/stack: use geometric table compaction\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1683%2Fjltobler%2Fjt%2Freftable-geometric-compaction-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1683/jltobler/jt/reftable-geometric-compaction-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1683\n\n reftable/stack.c | 106 +++++++++++++++----------------------\n reftable/stack.h | 3 --\n reftable/stack_test.c | 66 +++++------------------\n t/t0610-reftable-basics.sh | 24 ++++-----\n 4 files changed, 70 insertions(+), 129 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex b64e55648aa..e4ea8753977 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1214,75 +1214,57 @@ static int segment_size(struct segment *s)\n \treturn s->end - s->start;\n }\n \n-int fastlog2(uint64_t sz)\n-{\n-\tint l = 0;\n-\tif (sz == 0)\n-\t\treturn 0;\n-\tfor (; sz; sz /= 2) {\n-\t\tl++;\n-\t}\n-\treturn l - 1;\n-}\n-\n-struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n)\n-{\n-\tstruct segment *segs = reftable_calloc(n, sizeof(*segs));\n-\tstruct segment cur = { 0 };\n-\tsize_t next = 0, i;\n-\n-\tif (n == 0) {\n-\t\t*seglen = 0;\n-\t\treturn segs;\n-\t}\n-\tfor (i = 0; i < n; i++) {\n-\t\tint log = fastlog2(sizes[i]);\n-\t\tif (cur.log != log && cur.bytes > 0) {\n-\t\t\tstruct segment fresh = {\n-\t\t\t\t.start = i,\n-\t\t\t};\n-\n-\t\t\tsegs[next++] = cur;\n-\t\t\tcur = fresh;\n-\t\t}\n-\n-\t\tcur.log = log;\n-\t\tcur.end = i + 1;\n-\t\tcur.bytes += sizes[i];\n-\t}\n-\tsegs[next++] = cur;\n-\t*seglen = next;\n-\treturn segs;\n-}\n-\n struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n {\n-\tstruct segment min_seg = {\n-\t\t.log = 64,\n-\t};\n-\tstruct segment *segs;\n-\tsize_t seglen = 0, i;\n-\n-\tsegs = sizes_to_segments(&seglen, sizes, n);\n-\tfor (i = 0; i < seglen; i++) {\n-\t\tif (segment_size(&segs[i]) == 1)\n-\t\t\tcontinue;\n+\tstruct segment seg = { 0 };\n+\tuint64_t bytes;\n+\tsize_t i;\n \n-\t\tif (segs[i].log < min_seg.log)\n-\t\t\tmin_seg = segs[i];\n-\t}\n+\t/*\n+\t * If there are no tables or only a single one then we don't have to\n+\t * compact anything. The sequence is geometric by definition already.\n+\t */\n+\tif (n <= 1)\n+\t\treturn seg;\n \n-\twhile (min_seg.start > 0) {\n-\t\tsize_t prev = min_seg.start - 1;\n-\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev]))\n+\t/*\n+\t * Find the ending table of the compaction segment needed to restore the\n+\t * geometric sequence.\n+\t *\n+\t * To do so, we iterate backwards starting from the most recent table\n+\t * until a valid segment end is found. If the preceding table is smaller\n+\t * than the current table multiplied by the geometric factor (2), the\n+\t * current table is set as the compaction segment end.\n+\t */\n+\tfor (i = n - 1; i > 0; i--) {\n+\t\tif (sizes[i - 1] < sizes[i] * 2) {\n+\t\t\tseg.end = i;\n+\t\t\tbytes = sizes[i];\n \t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Find the starting table of the compaction segment by iterating\n+\t * through the remaing tables and keeping track of the accumulated size\n+\t * of all tables seen from the segment end table.\n+\t *\n+\t * Note that we keep iterating even after we have found the first\n+\t * first starting point. This is because there may be tables in the\n+\t * stack preceding that first starting point which violate the geometric\n+\t * sequence.\n+\t */\n+\tfor (; i > 0; i--) {\n+\t\tuint64_t curr = bytes;\n+\t\tbytes += sizes[i - 1];\n \n-\t\tmin_seg.start = prev;\n-\t\tmin_seg.bytes += sizes[prev];\n+\t\tif (sizes[i - 1] < curr * 2) {\n+\t\t\tseg.start = i - 1;\n+\t\t\tseg.bytes = bytes;\n+\t\t}\n \t}\n \n-\treftable_free(segs);\n-\treturn min_seg;\n+\treturn seg;\n }\n \n static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n@@ -1305,7 +1287,7 @@ int reftable_stack_auto_compact(struct reftable_stack *st)\n \t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n \treftable_free(sizes);\n \tif (segment_size(&seg) > 0)\n-\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n+\t\treturn stack_compact_range_stats(st, seg.start, seg.end,\n \t\t\t\t\t\t NULL);\n \n \treturn 0;\ndiff --git a/reftable/stack.h b/reftable/stack.h\nindex d919455669e..656f896cc28 100644\n--- a/reftable/stack.h\n+++ b/reftable/stack.h\n@@ -33,12 +33,9 @@ int read_lines(const char *filename, char ***lines);\n \n struct segment {\n \tsize_t start, end;\n-\tint log;\n \tuint64_t bytes;\n };\n \n-int fastlog2(uint64_t sz);\n-struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n);\n struct segment suggest_compaction_segment(uint64_t *sizes, size_t n);\n \n #endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 509f4866236..85600a9573e 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -720,59 +720,14 @@ static void test_reftable_stack_hash_id(void)\n \tclear_dir(dir);\n }\n \n-static void test_log2(void)\n-{\n-\tEXPECT(1 == fastlog2(3));\n-\tEXPECT(2 == fastlog2(4));\n-\tEXPECT(2 == fastlog2(5));\n-}\n-\n-static void test_sizes_to_segments(void)\n-{\n-\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n-\t/* .................0 1 2 3 4 5 */\n-\n-\tsize_t seglen = 0;\n-\tstruct segment *segs =\n-\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n-\tEXPECT(segs[2].log == 3);\n-\tEXPECT(segs[2].start == 5);\n-\tEXPECT(segs[2].end == 6);\n-\n-\tEXPECT(segs[1].log == 2);\n-\tEXPECT(segs[1].start == 2);\n-\tEXPECT(segs[1].end == 5);\n-\treftable_free(segs);\n-}\n-\n-static void test_sizes_to_segments_empty(void)\n-{\n-\tsize_t seglen = 0;\n-\tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n-\tEXPECT(seglen == 0);\n-\treftable_free(segs);\n-}\n-\n-static void test_sizes_to_segments_all_equal(void)\n-{\n-\tuint64_t sizes[] = { 5, 5 };\n-\tsize_t seglen = 0;\n-\tstruct segment *segs =\n-\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n-\tEXPECT(seglen == 1);\n-\tEXPECT(segs[0].start == 0);\n-\tEXPECT(segs[0].end == 2);\n-\treftable_free(segs);\n-}\n-\n static void test_suggest_compaction_segment(void)\n {\n-\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n+\tuint64_t sizes[] = { 512, 64, 17, 16, 9, 9, 9, 16, 2, 16 };\n \t/* .................0 1 2 3 4 5 6 */\n \tstruct segment min =\n \t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n-\tEXPECT(min.start == 2);\n-\tEXPECT(min.end == 7);\n+\tEXPECT(min.start == 1);\n+\tEXPECT(min.end == 9);\n }\n \n static void test_suggest_compaction_segment_nothing(void)\n@@ -884,6 +839,17 @@ static void test_empty_add(void)\n \treftable_stack_destroy(st2);\n }\n \n+static int fastlog2(uint64_t sz)\n+{\n+\tint l = 0;\n+\tif (sz == 0)\n+\t\treturn 0;\n+\tfor (; sz; sz /= 2) {\n+\t\tl++;\n+\t}\n+\treturn l - 1;\n+}\n+\n static void test_reftable_stack_auto_compaction(void)\n {\n \tstruct reftable_write_options cfg = { 0 };\n@@ -1072,7 +1038,6 @@ static void test_reftable_stack_compaction_concurrent_clean(void)\n int stack_test_main(int argc, const char *argv[])\n {\n \tRUN_TEST(test_empty_add);\n-\tRUN_TEST(test_log2);\n \tRUN_TEST(test_names_equal);\n \tRUN_TEST(test_parse_names);\n \tRUN_TEST(test_read_file);\n@@ -1092,9 +1057,6 @@ int stack_test_main(int argc, const char *argv[])\n \tRUN_TEST(test_reftable_stack_update_index_check);\n \tRUN_TEST(test_reftable_stack_uptodate);\n \tRUN_TEST(test_reftable_stack_validate_refname);\n-\tRUN_TEST(test_sizes_to_segments);\n-\tRUN_TEST(test_sizes_to_segments_all_equal);\n-\tRUN_TEST(test_sizes_to_segments_empty);\n \tRUN_TEST(test_suggest_compaction_segment);\n \tRUN_TEST(test_suggest_compaction_segment_nothing);\n \treturn 0;\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex 6a131e40b81..a3b1a04123e 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -293,7 +293,7 @@ test_expect_success 'ref transaction: writes cause auto-compaction' '\n \ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n \ttest_commit -C repo --no-tag A &&\n-\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n \ttest_commit -C repo --no-tag B &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n@@ -324,7 +324,7 @@ test_expect_success 'ref transaction: writes are synced' '\n \t\tgit -C repo -c core.fsync=reference \\\n \t\t-c core.fsyncMethod=fsync update-ref refs/heads/branch HEAD &&\n \tcheck_fsync_events trace2.txt <<-EOF\n-\t\"name\":\"hardware-flush\",\"count\":2\n+\t\"name\":\"hardware-flush\",\"count\":4\n \tEOF\n '\n \n@@ -334,8 +334,8 @@ test_expect_success 'pack-refs: compacts tables' '\n \n \ttest_commit -C repo A &&\n \tls -1 repo/.git/reftable >table-files &&\n-\ttest_line_count = 4 table-files &&\n-\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 3 table-files &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n \n \tgit -C repo pack-refs &&\n \tls -1 repo/.git/reftable >table-files &&\n@@ -367,7 +367,7 @@ do\n \t\t\tumask $umask &&\n \t\t\tgit init --shared=true repo &&\n \t\t\ttest_commit -C repo A &&\n-\t\t\ttest_line_count = 3 repo/.git/reftable/tables.list\n+\t\t\ttest_line_count = 2 repo/.git/reftable/tables.list\n \t\t) &&\n \t\tgit -C repo pack-refs &&\n \t\ttest_expect_perms \"-rw-rw-r--\" repo/.git/reftable/tables.list &&\n@@ -737,10 +737,10 @@ test_expect_success 'worktree: pack-refs in main repo packs main refs' '\n \ttest_commit -C repo A &&\n \tgit -C repo worktree add ../worktree &&\n \n-\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \tgit -C repo pack-refs &&\n-\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n '\n \n@@ -750,11 +750,11 @@ test_expect_success 'worktree: pack-refs in worktree packs worktree refs' '\n \ttest_commit -C repo A &&\n \tgit -C repo worktree add ../worktree &&\n \n-\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \tgit -C worktree pack-refs &&\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 4 repo/.git/reftable/tables.list\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n '\n \n test_expect_success 'worktree: creating shared ref updates main stack' '\n@@ -770,7 +770,7 @@ test_expect_success 'worktree: creating shared ref updates main stack' '\n \n \tgit -C worktree update-ref refs/heads/shared HEAD &&\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 2 repo/.git/reftable/tables.list\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n '\n \n test_expect_success 'worktree: creating per-worktree ref updates worktree stack' '\n\nbase-commit: b387623c12f3f4a376e4d35a610fd3e55d7ea907\n-- \ngitgitgadget\n"},{"id":"490080","messageId":"ZehhzdOYYvreRIfq@tanuki","threadId":"61055","inReplyTo":"pull.1683.git.1709669025722.gitgitgadget@gmail.com","subject":"Re: [PATCH] reftable/stack: use geometric table compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-03-06T12:30:05Z","receivedAt":"2024-03-06T12:30:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Mar 05, 2024 at 08:03:45PM +0000, Justin Tobler via GitGitGadget wrote:\n> From: Justin Tobler \n> \n> To reduce the number of on-disk reftables, compaction is performed.\n> Contiguous tables with the same binary log value of size are grouped\n> into segments. The segment that has both the lowest binary log value and\n> contains more than one table is set as the starting point when\n> identifying the compaction segment.\n> \n> Since segments containing a single table are not initially considered\n> for compaction, if the table appended to the list does not match the\n> previous table log value, no compaction occurs for the new table. It is\n> therefore possible for unbounded growth of the table list. This can be\n> demonstrated by repeating the following sequence:\n> \n> git branch -f foo\n> git branch -d foo\n> \n> Each operation results in a new table being written with no compaction\n> occurring until a separate operation produces a table matching the\n> previous table log value.\n> \n> To avoid unbounded growth of the table list, walk through each table and\n> evaluate if it needs to be included in the compaction segment to restore\n> a geometric sequence.\n\nI think the description of what exactly changes could use some more\nexplanation and some arguments why the new behaviour is okay, too. It's\nquite a large rewrite of the compaction logic, so pinpointing exactly\nhow these are different would go a long way.\n\n> Some tests in `t0610-reftable-basics.sh` assert the on-disk state of\n> tables and are therefore updated to specify the correct new table count.\n> Since compaction is more aggressive in ensuring tables maintain a\n> geometric sequence, the expected table count is reduced in these tests.\n> In `reftable/stack_test.c` tests related to `sizes_to_segments()` are\n> removed because the function is no longer needed. Also, the\n> `test_suggest_compaction_segment()` test is updated to better showcase\n> and reflect the new geometric compaction behavior.\n> \n> Signed-off-by: Justin Tobler \n> ---\n> reftable/stack: use geometric table compaction\n> \n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1683%2Fjltobler%2Fjt%2Freftable-geometric-compaction-v1\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1683/jltobler/jt/reftable-geometric-compaction-v1\n> Pull-Request: https://github.com/gitgitgadget/git/pull/1683\n> \n> reftable/stack.c | 106 +++++++++++++++----------------------\n> reftable/stack.h | 3 --\n> reftable/stack_test.c | 66 +++++------------------\n> t/t0610-reftable-basics.sh | 24 ++++-----\n> 4 files changed, 70 insertions(+), 129 deletions(-)\n> \n> diff --git a/reftable/stack.c b/reftable/stack.c\n> index b64e55648aa..e4ea8753977 100644\n> --- a/reftable/stack.c\n> +++ b/reftable/stack.c\n> @@ -1214,75 +1214,57 @@ static int segment_size(struct segment *s)\n> \treturn s->end - s->start;\n> }\n> \n> -int fastlog2(uint64_t sz)\n> -{\n> -\tint l = 0;\n> -\tif (sz == 0)\n> -\t\treturn 0;\n> -\tfor (; sz; sz /= 2) {\n> -\t\tl++;\n> -\t}\n> -\treturn l - 1;\n> -}\n> -\n> -struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n)\n> -{\n> -\tstruct segment *segs = reftable_calloc(n, sizeof(*segs));\n> -\tstruct segment cur = { 0 };\n> -\tsize_t next = 0, i;\n> -\n> -\tif (n == 0) {\n> -\t\t*seglen = 0;\n> -\t\treturn segs;\n> -\t}\n> -\tfor (i = 0; i < n; i++) {\n> -\t\tint log = fastlog2(sizes[i]);\n> -\t\tif (cur.log != log && cur.bytes > 0) {\n> -\t\t\tstruct segment fresh = {\n> -\t\t\t\t.start = i,\n> -\t\t\t};\n> -\n> -\t\t\tsegs[next++] = cur;\n> -\t\t\tcur = fresh;\n> -\t\t}\n> -\n> -\t\tcur.log = log;\n> -\t\tcur.end = i + 1;\n> -\t\tcur.bytes += sizes[i];\n> -\t}\n> -\tsegs[next++] = cur;\n> -\t*seglen = next;\n> -\treturn segs;\n> -}\n> -\n> struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n> {\n> -\tstruct segment min_seg = {\n> -\t\t.log = 64,\n> -\t};\n> -\tstruct segment *segs;\n> -\tsize_t seglen = 0, i;\n> -\n> -\tsegs = sizes_to_segments(&seglen, sizes, n);\n> -\tfor (i = 0; i < seglen; i++) {\n> -\t\tif (segment_size(&segs[i]) == 1)\n> -\t\t\tcontinue;\n> +\tstruct segment seg = { 0 };\n> +\tuint64_t bytes;\n> +\tsize_t i;\n> \n> -\t\tif (segs[i].log < min_seg.log)\n> -\t\t\tmin_seg = segs[i];\n> -\t}\n> +\t/*\n> +\t * If there are no tables or only a single one then we don't have to\n> +\t * compact anything. The sequence is geometric by definition already.\n> +\t */\n> +\tif (n <= 1)\n> +\t\treturn seg;\n> \n> -\twhile (min_seg.start > 0) {\n> -\t\tsize_t prev = min_seg.start - 1;\n> -\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev]))\n> +\t/*\n> +\t * Find the ending table of the compaction segment needed to restore the\n> +\t * geometric sequence.\n> +\t *\n> +\t * To do so, we iterate backwards starting from the most recent table\n> +\t * until a valid segment end is found. If the preceding table is smaller\n> +\t * than the current table multiplied by the geometric factor (2), the\n> +\t * current table is set as the compaction segment end.\n> +\t */\n> +\tfor (i = n - 1; i > 0; i--) {\n> +\t\tif (sizes[i - 1] < sizes[i] * 2) {\n> +\t\t\tseg.end = i;\n> +\t\t\tbytes = sizes[i];\n> \t\t\tbreak;\n> +\t\t}\n> +\t}\n\nI was briefly wondering whether we have to compare the _sum_ of all\npreceding table sizes to the next size here. Otherwise it may happen\nthat compaction will lead to a new table that is immediately violating\nthe geometric sequence again.\n\nBut I think due to properties of the geometric sequence, the sum of all\nentries preceding the current value cannot be greater than the value\nitself. So this should be fine.\n\nThis might be worth a comment.\n\n> +\n> +\t/*\n> +\t * Find the starting table of the compaction segment by iterating\n> +\t * through the remaing tables and keeping track of the accumulated size\n\ns/remaing/remaining/\n\n> +\t * of all tables seen from the segment end table.\n> +\t *\n> +\t * Note that we keep iterating even after we have found the first\n> +\t * first starting point. This is because there may be tables in the\n\nNit: s/first//, duplicate word.\n\n> +\t * stack preceding that first starting point which violate the geometric\n> +\t * sequence.\n> +\t */\n> +\tfor (; i > 0; i--) {\n> +\t\tuint64_t curr = bytes;\n> +\t\tbytes += sizes[i - 1];\n> \n> -\t\tmin_seg.start = prev;\n> -\t\tmin_seg.bytes += sizes[prev];\n> +\t\tif (sizes[i - 1] < curr * 2) {\n> +\t\t\tseg.start = i - 1;\n> +\t\t\tseg.bytes = bytes;\n> +\t\t}\n> \t}\n\nOverall I really like the rewritten algorithm, it's a ton easier to\nunderstand compared to the preceding code.\n\nOne thing I'd suggest doing though is to provide a benchmark of how the\nnew compaction strategy compares to the old one. A comparatively easy\nway to do this is to write N refs sequentially -- with a big enough N\n(e.g. 1 million), compaction time will eventually become an important\nfactor. So something like the following (untested):\n\nhyperfine \\\n --prepare \"rm -rf repo && git init --ref-format=reftable repo && git -C repo commit --allow-empty --message msg\" \\\n 'for ((i = 0 ; i < 1000000; i++ )); do git -C repo update-ref refs/heads/branch-$i HEAD'\n\n> \n> -\treftable_free(segs);\n> -\treturn min_seg;\n> +\treturn seg;\n> }\n> \n> static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n[snip]\n> @@ -737,10 +737,10 @@ test_expect_success 'worktree: pack-refs in main repo packs main refs' '\n> \ttest_commit -C repo A &&\n> \tgit -C repo worktree add ../worktree &&\n> \n> -\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n> -\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n> +\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n> +\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n> \tgit -C repo pack-refs &&\n> -\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n> +\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n> \ttest_line_count = 1 repo/.git/reftable/tables.list\n> '\n\nThis test needs updating as git-pack-refs(1) has become a no-op here.\n\n> @@ -750,11 +750,11 @@ test_expect_success 'worktree: pack-refs in worktree packs worktree refs' '\n> \ttest_commit -C repo A &&\n> \tgit -C repo worktree add ../worktree &&\n> \n> -\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n> -\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n> +\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n> +\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n> \tgit -C worktree pack-refs &&\n> \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n> -\ttest_line_count = 4 repo/.git/reftable/tables.list\n> +\ttest_line_count = 1 repo/.git/reftable/tables.list\n> '\n\nSame.\n\n> test_expect_success 'worktree: creating shared ref updates main stack' '\n> @@ -770,7 +770,7 @@ test_expect_success 'worktree: creating shared ref updates main stack' '\n> \n> \tgit -C worktree update-ref refs/heads/shared HEAD &&\n> \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n> -\ttest_line_count = 2 repo/.git/reftable/tables.list\n> +\ttest_line_count = 1 repo/.git/reftable/tables.list\n> '\n\nSame.\n\nOne thing missing is a test that demonstrates the previously-broken\nbehaviour.\n\nPatrick\n\n> test_expect_success 'worktree: creating per-worktree ref updates worktree stack' '\n> \n> base-commit: b387623c12f3f4a376e4d35a610fd3e55d7ea907\n> -- \n> gitgitgadget\n> \n"},{"id":"490082","messageId":"ZehjoXhH6deTJmSm@tanuki","threadId":"61055","inReplyTo":"pull.1683.git.1709669025722.gitgitgadget@gmail.com","subject":"Re: [PATCH] reftable/stack: use geometric table compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-03-06T12:37:53Z","receivedAt":"2024-03-06T12:37:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Mar 05, 2024 at 08:03:45PM +0000, Justin Tobler via GitGitGadget wrote:\n> From: Justin Tobler \n> @@ -1305,7 +1287,7 @@ int reftable_stack_auto_compact(struct reftable_stack *st)\n> \t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n> \treftable_free(sizes);\n> \tif (segment_size(&seg) > 0)\n> -\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n> +\t\treturn stack_compact_range_stats(st, seg.start, seg.end,\n> \t\t\t\t\t\t NULL);\n> \n> \treturn 0;\n\nOne more thing: I think it would make sense to move the refactoring\nwhere you change whether the end segment index is inclusive or exclusive\ninto a separate patch so that it's easier to reason about. Also, the\nfact that no tests would require changes would further stress the point\nthat this is a mere refactoring without unintended side effects.\n\nPatrick\n"},{"id":"491181","messageId":"pull.1683.v2.git.1711060819.gitgitgadget@gmail.com","threadId":"61055","inReplyTo":"pull.1683.git.1709669025722.gitgitgadget@gmail.com","subject":"[PATCH v2 0/3] reftable/stack: use geometric table compaction","fromName":"Justin Tobler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-03-21T22:40:16Z","receivedAt":"2024-03-21T22:40:23Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Hello again,\n\nThis is the second version my patch series that refactors the reftable\ncompaction strategy to instead follow a geometric sequence. Changes compared\nto v1:\n\n * Added GIT_TEST_REFTABLE_NO_AUTOCOMPACTION environment variable to disable\n reftable compaction when testing.\n * Refactored worktree tests in t0610-reftable-basics.sh to properly assert\n git-pack-refs(1) works as expected.\n * Added test to validate that alternating table sizes are compacted.\n * Added benchmark to compare compaction strategies.\n * Moved change that made compaction segment end inclusive to its own\n commit.\n * Added additional explanation in commits and comments and fixed typos.\n\nThanks for taking a look!\n\nJustin\n\nJustin Tobler (3):\n reftable/stack: add env to disable autocompaction\n reftable/stack: use geometric table compaction\n reftable/segment: make segment end inclusive\n\n reftable/stack.c | 113 ++++++++++++++++---------------------\n reftable/stack.h | 3 -\n reftable/stack_test.c | 66 +++++-----------------\n reftable/system.h | 1 +\n t/t0610-reftable-basics.sh | 43 +++++++++-----\n 5 files changed, 94 insertions(+), 132 deletions(-)\n\n\nbase-commit: 3bd955d26919e149552f34aacf8a4e6368c26cec\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1683%2Fjltobler%2Fjt%2Freftable-geometric-compaction-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1683/jltobler/jt/reftable-geometric-compaction-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1683\n\nRange-diff vs v1:\n\n -: ----------- > 1: cb6b152e5c8 reftable/stack: add env to disable autocompaction\n 1: 7a518853a10 ! 2: def70084523 reftable/stack: use geometric table compaction\n @@ Commit message\n occurring until a separate operation produces a table matching the\n previous table log value.\n \n - To avoid unbounded growth of the table list, walk through each table and\n - evaluate if it needs to be included in the compaction segment to restore\n - a geometric sequence.\n + Instead, to avoid unbounded growth of the table list, the compaction\n + strategy is updated to ensure tables follow a geometric sequence after\n + each operation. This is done by walking the table list in reverse index\n + order to identify the compaction segment start and end. The compaction\n + segment end is found by identifying the first table which has a\n + preceding table size less than twice the current table. Next, the\n + compaction segment start is found iterating through the remaining tables\n + in the list checking if the previous table size is less than twice the\n + cumulative of tables from the segment end. This ensures the correct\n + segment start is found and that the newly compacted table does not\n + violate the geometric sequence.\n +\n + When creating 10 thousand references, the new strategy has no\n + performance impact:\n +\n + Benchmark 1: update-ref: create refs sequentially (revision = HEAD~)\n + Time (mean ± σ): 26.516 s ± 0.047 s [User: 17.864 s, System: 8.491 s]\n + Range (min … max): 26.447 s … 26.569 s 10 runs\n +\n + Benchmark 2: update-ref: create refs sequentially (revision = HEAD)\n + Time (mean ± σ): 26.417 s ± 0.028 s [User: 17.738 s, System: 8.500 s]\n + Range (min … max): 26.366 s … 26.444 s 10 runs\n +\n + Summary\n + update-ref: create refs sequentially (revision = HEAD) ran\n + 1.00 ± 0.00 times faster than update-ref: create refs sequentially (revision = HEAD~)\n \n Some tests in `t0610-reftable-basics.sh` assert the on-disk state of\n tables and are therefore updated to specify the correct new table count.\n @@ reftable/stack.c: static int segment_size(struct segment *s)\n +\t * until a valid segment end is found. If the preceding table is smaller\n +\t * than the current table multiplied by the geometric factor (2), the\n +\t * current table is set as the compaction segment end.\n ++\t *\n ++\t * Tables after the ending point are not added to the byte count because\n ++\t * they are already valid members of the geometric sequence. Due to the\n ++\t * properties of a geometric sequence, it is not possible for the sum of\n ++\t * these tables to exceed the value of the ending point table.\n +\t */\n +\tfor (i = n - 1; i > 0; i--) {\n +\t\tif (sizes[i - 1] < sizes[i] * 2) {\n -+\t\t\tseg.end = i;\n ++\t\t\tseg.end = i + 1;\n +\t\t\tbytes = sizes[i];\n \t\t\tbreak;\n +\t\t}\n @@ reftable/stack.c: static int segment_size(struct segment *s)\n +\n +\t/*\n +\t * Find the starting table of the compaction segment by iterating\n -+\t * through the remaing tables and keeping track of the accumulated size\n -+\t * of all tables seen from the segment end table.\n ++\t * through the remaining tables and keeping track of the accumulated\n ++\t * size of all tables seen from the segment end table.\n +\t *\n +\t * Note that we keep iterating even after we have found the first\n -+\t * first starting point. This is because there may be tables in the\n -+\t * stack preceding that first starting point which violate the geometric\n ++\t * starting point. This is because there may be tables in the stack\n ++\t * preceding that first starting point which violate the geometric\n +\t * sequence.\n +\t */\n +\tfor (; i > 0; i--) {\n @@ reftable/stack.c: static int segment_size(struct segment *s)\n }\n \n static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n -@@ reftable/stack.c: int reftable_stack_auto_compact(struct reftable_stack *st)\n - \t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n - \treftable_free(sizes);\n - \tif (segment_size(&seg) > 0)\n --\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n -+\t\treturn stack_compact_range_stats(st, seg.start, seg.end,\n - \t\t\t\t\t\t NULL);\n - \n - \treturn 0;\n \n ## reftable/stack.h ##\n @@ reftable/stack.h: int read_lines(const char *filename, char ***lines);\n @@ reftable/stack_test.c: static void test_reftable_stack_hash_id(void)\n -\tEXPECT(min.start == 2);\n -\tEXPECT(min.end == 7);\n +\tEXPECT(min.start == 1);\n -+\tEXPECT(min.end == 9);\n ++\tEXPECT(min.end == 10);\n }\n \n static void test_suggest_compaction_segment_nothing(void)\n @@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: writes cause a\n \n \ttest_commit -C repo --no-tag B &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n + '\n + \n ++test_expect_success 'ref transaction: alternating table sizes are compacted' '\n ++\ttest_when_finished \"rm -rf repo\" &&\n ++\tgit init repo &&\n ++\ttest_commit -C repo A &&\n ++\tfor i in $(test_seq 20)\n ++\tdo\n ++\t\tgit -C repo branch -f foo &&\n ++\t\tgit -C repo branch -d foo || return 1\n ++\tdone &&\n ++\ttest_line_count = 2 repo/.git/reftable/tables.list\n ++'\n ++\n + check_fsync_events () {\n + \tlocal trace=\"$1\" &&\n + \tshift &&\n @@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: writes are synced' '\n \t\tgit -C repo -c core.fsync=reference \\\n \t\t-c core.fsyncMethod=fsync update-ref refs/heads/branch HEAD &&\n @@ t/t0610-reftable-basics.sh: do\n \t\tgit -C repo pack-refs &&\n \t\ttest_expect_perms \"-rw-rw-r--\" repo/.git/reftable/tables.list &&\n @@ t/t0610-reftable-basics.sh: test_expect_success 'worktree: pack-refs in main repo packs main refs' '\n + \ttest_when_finished \"rm -rf repo worktree\" &&\n + \tgit init repo &&\n \ttest_commit -C repo A &&\n - \tgit -C repo worktree add ../worktree &&\n +-\tgit -C repo worktree add ../worktree &&\n ++\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo worktree add ../worktree &&\n ++\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C worktree update-ref refs/worktree/per-worktree HEAD &&\n \n -\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n -\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n -+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n -+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n ++\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n ++\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n \tgit -C repo pack-refs &&\n -\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n -+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n ++\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n '\n \n @@ t/t0610-reftable-basics.sh: test_expect_success 'worktree: pack-refs in worktree packs worktree refs' '\n + \ttest_when_finished \"rm -rf repo worktree\" &&\n + \tgit init repo &&\n \ttest_commit -C repo A &&\n - \tgit -C repo worktree add ../worktree &&\n +-\tgit -C repo worktree add ../worktree &&\n ++\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo worktree add ../worktree &&\n ++\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C worktree update-ref refs/worktree/per-worktree HEAD &&\n \n -\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n -\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n -+\ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n -+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n ++\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n ++\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n \tgit -C worktree pack-refs &&\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n -\ttest_line_count = 4 repo/.git/reftable/tables.list\n -+\ttest_line_count = 1 repo/.git/reftable/tables.list\n ++\ttest_line_count = 3 repo/.git/reftable/tables.list\n '\n \n test_expect_success 'worktree: creating shared ref updates main stack' '\n + \ttest_when_finished \"rm -rf repo worktree\" &&\n + \tgit init repo &&\n + \ttest_commit -C repo A &&\n ++\ttest_commit -C repo B &&\n + \n + \tgit -C repo worktree add ../worktree &&\n + \tgit -C repo pack-refs &&\n @@ t/t0610-reftable-basics.sh: test_expect_success 'worktree: creating shared ref updates main stack' '\n + \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n + \ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n - \tgit -C worktree update-ref refs/heads/shared HEAD &&\n +-\tgit -C worktree update-ref refs/heads/shared HEAD &&\n ++\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C worktree update-ref refs/heads/shared HEAD &&\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n --\ttest_line_count = 2 repo/.git/reftable/tables.list\n -+\ttest_line_count = 1 repo/.git/reftable/tables.list\n + \ttest_line_count = 2 repo/.git/reftable/tables.list\n '\n - \n - test_expect_success 'worktree: creating per-worktree ref updates worktree stack' '\n -: ----------- > 3: a23e3fc6972 reftable/segment: make segment end inclusive\n\n-- \ngitgitgadget\n"},{"id":"491182","messageId":"cb6b152e5c86ab14165e072ac20dfa865b5f3aaa.1711060820.git.gitgitgadget@gmail.com","threadId":"61055","inReplyTo":"pull.1683.v2.git.1711060819.gitgitgadget@gmail.com","subject":"[PATCH v2 1/3] reftable/stack: add env to disable autocompaction","fromName":"Justin Tobler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-03-21T22:40:17Z","receivedAt":"2024-03-21T22:40:24Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"From: Justin Tobler \n\nIn future tests it will be neccesary to create repositories with a set\nnumber of tables. To make this easier, introduce the\n`GIT_TEST_REFTABLE_NO_AUTOCOMPACTION` environment variable that, when\nset, disables autocompaction of reftables.\n\nSigned-off-by: Justin Tobler \n---\n reftable/stack.c | 2 +-\n reftable/system.h | 1 +\n 2 files changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex b64e55648aa..2370d93d13b 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -681,7 +681,7 @@ int reftable_addition_commit(struct reftable_addition *add)\n \tif (err)\n \t\tgoto done;\n \n-\tif (!add->stack->disable_auto_compact)\n+\tif (!add->stack->disable_auto_compact && !git_env_bool(\"GIT_TEST_REFTABLE_NO_AUTOCOMPACTION\", 0))\n \t\terr = reftable_stack_auto_compact(add->stack);\n \n done:\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 6b74a815143..ec08b728177 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -15,6 +15,7 @@ license that can be found in the LICENSE file or at\n #include \"strbuf.h\"\n #include \"hash-ll.h\" /* hash ID, sizes.*/\n #include \"dir.h\" /* remove_dir_recursively, for tests.*/\n+#include \"parse.h\"\n \n int hash_size(uint32_t id);\n \n-- \ngitgitgadget\n\n"},{"id":"491184","messageId":"def7008452303f71c1fa469609bc199c629a19ec.1711060820.git.gitgitgadget@gmail.com","threadId":"61055","inReplyTo":"pull.1683.v2.git.1711060819.gitgitgadget@gmail.com","subject":"[PATCH v2 2/3] reftable/stack: use geometric table compaction","fromName":"Justin Tobler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-03-21T22:40:18Z","receivedAt":"2024-03-21T22:40:26Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"From: Justin Tobler \n\nTo reduce the number of on-disk reftables, compaction is performed.\nContiguous tables with the same binary log value of size are grouped\ninto segments. The segment that has both the lowest binary log value and\ncontains more than one table is set as the starting point when\nidentifying the compaction segment.\n\nSince segments containing a single table are not initially considered\nfor compaction, if the table appended to the list does not match the\nprevious table log value, no compaction occurs for the new table. It is\ntherefore possible for unbounded growth of the table list. This can be\ndemonstrated by repeating the following sequence:\n\ngit branch -f foo\ngit branch -d foo\n\nEach operation results in a new table being written with no compaction\noccurring until a separate operation produces a table matching the\nprevious table log value.\n\nInstead, to avoid unbounded growth of the table list, the compaction\nstrategy is updated to ensure tables follow a geometric sequence after\neach operation. This is done by walking the table list in reverse index\norder to identify the compaction segment start and end. The compaction\nsegment end is found by identifying the first table which has a\npreceding table size less than twice the current table. Next, the\ncompaction segment start is found iterating through the remaining tables\nin the list checking if the previous table size is less than twice the\ncumulative of tables from the segment end. This ensures the correct\nsegment start is found and that the newly compacted table does not\nviolate the geometric sequence.\n\nWhen creating 10 thousand references, the new strategy has no\nperformance impact:\n\nBenchmark 1: update-ref: create refs sequentially (revision = HEAD~)\n Time (mean ± σ): 26.516 s ± 0.047 s [User: 17.864 s, System: 8.491 s]\n Range (min … max): 26.447 s … 26.569 s 10 runs\n\nBenchmark 2: update-ref: create refs sequentially (revision = HEAD)\n Time (mean ± σ): 26.417 s ± 0.028 s [User: 17.738 s, System: 8.500 s]\n Range (min … max): 26.366 s … 26.444 s 10 runs\n\nSummary\n update-ref: create refs sequentially (revision = HEAD) ran\n 1.00 ± 0.00 times faster than update-ref: create refs sequentially (revision = HEAD~)\n\nSome tests in `t0610-reftable-basics.sh` assert the on-disk state of\ntables and are therefore updated to specify the correct new table count.\nSince compaction is more aggressive in ensuring tables maintain a\ngeometric sequence, the expected table count is reduced in these tests.\nIn `reftable/stack_test.c` tests related to `sizes_to_segments()` are\nremoved because the function is no longer needed. Also, the\n`test_suggest_compaction_segment()` test is updated to better showcase\nand reflect the new geometric compaction behavior.\n\nSigned-off-by: Justin Tobler \n---\n reftable/stack.c | 109 ++++++++++++++++---------------------\n reftable/stack.h | 3 -\n reftable/stack_test.c | 66 +++++-----------------\n t/t0610-reftable-basics.sh | 43 ++++++++++-----\n 4 files changed, 91 insertions(+), 130 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 2370d93d13b..ef55dc75cde 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1214,75 +1214,62 @@ static int segment_size(struct segment *s)\n \treturn s->end - s->start;\n }\n \n-int fastlog2(uint64_t sz)\n-{\n-\tint l = 0;\n-\tif (sz == 0)\n-\t\treturn 0;\n-\tfor (; sz; sz /= 2) {\n-\t\tl++;\n-\t}\n-\treturn l - 1;\n-}\n-\n-struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n)\n-{\n-\tstruct segment *segs = reftable_calloc(n, sizeof(*segs));\n-\tstruct segment cur = { 0 };\n-\tsize_t next = 0, i;\n-\n-\tif (n == 0) {\n-\t\t*seglen = 0;\n-\t\treturn segs;\n-\t}\n-\tfor (i = 0; i < n; i++) {\n-\t\tint log = fastlog2(sizes[i]);\n-\t\tif (cur.log != log && cur.bytes > 0) {\n-\t\t\tstruct segment fresh = {\n-\t\t\t\t.start = i,\n-\t\t\t};\n-\n-\t\t\tsegs[next++] = cur;\n-\t\t\tcur = fresh;\n-\t\t}\n-\n-\t\tcur.log = log;\n-\t\tcur.end = i + 1;\n-\t\tcur.bytes += sizes[i];\n-\t}\n-\tsegs[next++] = cur;\n-\t*seglen = next;\n-\treturn segs;\n-}\n-\n struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n {\n-\tstruct segment min_seg = {\n-\t\t.log = 64,\n-\t};\n-\tstruct segment *segs;\n-\tsize_t seglen = 0, i;\n-\n-\tsegs = sizes_to_segments(&seglen, sizes, n);\n-\tfor (i = 0; i < seglen; i++) {\n-\t\tif (segment_size(&segs[i]) == 1)\n-\t\t\tcontinue;\n+\tstruct segment seg = { 0 };\n+\tuint64_t bytes;\n+\tsize_t i;\n \n-\t\tif (segs[i].log < min_seg.log)\n-\t\t\tmin_seg = segs[i];\n-\t}\n+\t/*\n+\t * If there are no tables or only a single one then we don't have to\n+\t * compact anything. The sequence is geometric by definition already.\n+\t */\n+\tif (n <= 1)\n+\t\treturn seg;\n \n-\twhile (min_seg.start > 0) {\n-\t\tsize_t prev = min_seg.start - 1;\n-\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev]))\n+\t/*\n+\t * Find the ending table of the compaction segment needed to restore the\n+\t * geometric sequence.\n+\t *\n+\t * To do so, we iterate backwards starting from the most recent table\n+\t * until a valid segment end is found. If the preceding table is smaller\n+\t * than the current table multiplied by the geometric factor (2), the\n+\t * current table is set as the compaction segment end.\n+\t *\n+\t * Tables after the ending point are not added to the byte count because\n+\t * they are already valid members of the geometric sequence. Due to the\n+\t * properties of a geometric sequence, it is not possible for the sum of\n+\t * these tables to exceed the value of the ending point table.\n+\t */\n+\tfor (i = n - 1; i > 0; i--) {\n+\t\tif (sizes[i - 1] < sizes[i] * 2) {\n+\t\t\tseg.end = i + 1;\n+\t\t\tbytes = sizes[i];\n \t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Find the starting table of the compaction segment by iterating\n+\t * through the remaining tables and keeping track of the accumulated\n+\t * size of all tables seen from the segment end table.\n+\t *\n+\t * Note that we keep iterating even after we have found the first\n+\t * starting point. This is because there may be tables in the stack\n+\t * preceding that first starting point which violate the geometric\n+\t * sequence.\n+\t */\n+\tfor (; i > 0; i--) {\n+\t\tuint64_t curr = bytes;\n+\t\tbytes += sizes[i - 1];\n \n-\t\tmin_seg.start = prev;\n-\t\tmin_seg.bytes += sizes[prev];\n+\t\tif (sizes[i - 1] < curr * 2) {\n+\t\t\tseg.start = i - 1;\n+\t\t\tseg.bytes = bytes;\n+\t\t}\n \t}\n \n-\treftable_free(segs);\n-\treturn min_seg;\n+\treturn seg;\n }\n \n static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\ndiff --git a/reftable/stack.h b/reftable/stack.h\nindex d919455669e..656f896cc28 100644\n--- a/reftable/stack.h\n+++ b/reftable/stack.h\n@@ -33,12 +33,9 @@ int read_lines(const char *filename, char ***lines);\n \n struct segment {\n \tsize_t start, end;\n-\tint log;\n \tuint64_t bytes;\n };\n \n-int fastlog2(uint64_t sz);\n-struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n);\n struct segment suggest_compaction_segment(uint64_t *sizes, size_t n);\n \n #endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 509f4866236..e5f6ff5c9e4 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -720,59 +720,14 @@ static void test_reftable_stack_hash_id(void)\n \tclear_dir(dir);\n }\n \n-static void test_log2(void)\n-{\n-\tEXPECT(1 == fastlog2(3));\n-\tEXPECT(2 == fastlog2(4));\n-\tEXPECT(2 == fastlog2(5));\n-}\n-\n-static void test_sizes_to_segments(void)\n-{\n-\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n-\t/* .................0 1 2 3 4 5 */\n-\n-\tsize_t seglen = 0;\n-\tstruct segment *segs =\n-\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n-\tEXPECT(segs[2].log == 3);\n-\tEXPECT(segs[2].start == 5);\n-\tEXPECT(segs[2].end == 6);\n-\n-\tEXPECT(segs[1].log == 2);\n-\tEXPECT(segs[1].start == 2);\n-\tEXPECT(segs[1].end == 5);\n-\treftable_free(segs);\n-}\n-\n-static void test_sizes_to_segments_empty(void)\n-{\n-\tsize_t seglen = 0;\n-\tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n-\tEXPECT(seglen == 0);\n-\treftable_free(segs);\n-}\n-\n-static void test_sizes_to_segments_all_equal(void)\n-{\n-\tuint64_t sizes[] = { 5, 5 };\n-\tsize_t seglen = 0;\n-\tstruct segment *segs =\n-\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n-\tEXPECT(seglen == 1);\n-\tEXPECT(segs[0].start == 0);\n-\tEXPECT(segs[0].end == 2);\n-\treftable_free(segs);\n-}\n-\n static void test_suggest_compaction_segment(void)\n {\n-\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n+\tuint64_t sizes[] = { 512, 64, 17, 16, 9, 9, 9, 16, 2, 16 };\n \t/* .................0 1 2 3 4 5 6 */\n \tstruct segment min =\n \t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n-\tEXPECT(min.start == 2);\n-\tEXPECT(min.end == 7);\n+\tEXPECT(min.start == 1);\n+\tEXPECT(min.end == 10);\n }\n \n static void test_suggest_compaction_segment_nothing(void)\n@@ -884,6 +839,17 @@ static void test_empty_add(void)\n \treftable_stack_destroy(st2);\n }\n \n+static int fastlog2(uint64_t sz)\n+{\n+\tint l = 0;\n+\tif (sz == 0)\n+\t\treturn 0;\n+\tfor (; sz; sz /= 2) {\n+\t\tl++;\n+\t}\n+\treturn l - 1;\n+}\n+\n static void test_reftable_stack_auto_compaction(void)\n {\n \tstruct reftable_write_options cfg = { 0 };\n@@ -1072,7 +1038,6 @@ static void test_reftable_stack_compaction_concurrent_clean(void)\n int stack_test_main(int argc, const char *argv[])\n {\n \tRUN_TEST(test_empty_add);\n-\tRUN_TEST(test_log2);\n \tRUN_TEST(test_names_equal);\n \tRUN_TEST(test_parse_names);\n \tRUN_TEST(test_read_file);\n@@ -1092,9 +1057,6 @@ int stack_test_main(int argc, const char *argv[])\n \tRUN_TEST(test_reftable_stack_update_index_check);\n \tRUN_TEST(test_reftable_stack_uptodate);\n \tRUN_TEST(test_reftable_stack_validate_refname);\n-\tRUN_TEST(test_sizes_to_segments);\n-\tRUN_TEST(test_sizes_to_segments_all_equal);\n-\tRUN_TEST(test_sizes_to_segments_empty);\n \tRUN_TEST(test_suggest_compaction_segment);\n \tRUN_TEST(test_suggest_compaction_segment_nothing);\n \treturn 0;\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex 686781192eb..e6c3f94d874 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -293,12 +293,24 @@ test_expect_success 'ref transaction: writes cause auto-compaction' '\n \ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n \ttest_commit -C repo --no-tag A &&\n-\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n \ttest_commit -C repo --no-tag B &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n '\n \n+test_expect_success 'ref transaction: alternating table sizes are compacted' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tfor i in $(test_seq 20)\n+\tdo\n+\t\tgit -C repo branch -f foo &&\n+\t\tgit -C repo branch -d foo || return 1\n+\tdone &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list\n+'\n+\n check_fsync_events () {\n \tlocal trace=\"$1\" &&\n \tshift &&\n@@ -324,7 +336,7 @@ test_expect_success 'ref transaction: writes are synced' '\n \t\tgit -C repo -c core.fsync=reference \\\n \t\t-c core.fsyncMethod=fsync update-ref refs/heads/branch HEAD &&\n \tcheck_fsync_events trace2.txt <<-EOF\n-\t\"name\":\"hardware-flush\",\"count\":2\n+\t\"name\":\"hardware-flush\",\"count\":4\n \tEOF\n '\n \n@@ -346,8 +358,8 @@ test_expect_success 'pack-refs: compacts tables' '\n \n \ttest_commit -C repo A &&\n \tls -1 repo/.git/reftable >table-files &&\n-\ttest_line_count = 4 table-files &&\n-\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 3 table-files &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n \n \tgit -C repo pack-refs &&\n \tls -1 repo/.git/reftable >table-files &&\n@@ -379,7 +391,7 @@ do\n \t\t\tumask $umask &&\n \t\t\tgit init --shared=true repo &&\n \t\t\ttest_commit -C repo A &&\n-\t\t\ttest_line_count = 3 repo/.git/reftable/tables.list\n+\t\t\ttest_line_count = 2 repo/.git/reftable/tables.list\n \t\t) &&\n \t\tgit -C repo pack-refs &&\n \t\ttest_expect_perms \"-rw-rw-r--\" repo/.git/reftable/tables.list &&\n@@ -747,12 +759,13 @@ test_expect_success 'worktree: pack-refs in main repo packs main refs' '\n \ttest_when_finished \"rm -rf repo worktree\" &&\n \tgit init repo &&\n \ttest_commit -C repo A &&\n-\tgit -C repo worktree add ../worktree &&\n+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo worktree add ../worktree &&\n+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C worktree update-ref refs/worktree/per-worktree HEAD &&\n \n-\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n \tgit -C repo pack-refs &&\n-\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n '\n \n@@ -760,19 +773,21 @@ test_expect_success 'worktree: pack-refs in worktree packs worktree refs' '\n \ttest_when_finished \"rm -rf repo worktree\" &&\n \tgit init repo &&\n \ttest_commit -C repo A &&\n-\tgit -C repo worktree add ../worktree &&\n+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo worktree add ../worktree &&\n+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C worktree update-ref refs/worktree/per-worktree HEAD &&\n \n-\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n \tgit -C worktree pack-refs &&\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 4 repo/.git/reftable/tables.list\n+\ttest_line_count = 3 repo/.git/reftable/tables.list\n '\n \n test_expect_success 'worktree: creating shared ref updates main stack' '\n \ttest_when_finished \"rm -rf repo worktree\" &&\n \tgit init repo &&\n \ttest_commit -C repo A &&\n+\ttest_commit -C repo B &&\n \n \tgit -C repo worktree add ../worktree &&\n \tgit -C repo pack-refs &&\n@@ -780,7 +795,7 @@ test_expect_success 'worktree: creating shared ref updates main stack' '\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n-\tgit -C worktree update-ref refs/heads/shared HEAD &&\n+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C worktree update-ref refs/heads/shared HEAD &&\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 2 repo/.git/reftable/tables.list\n '\n-- \ngitgitgadget\n\n"},{"id":"491183","messageId":"a23e3fc6972ff055f8c55722da1f2691b7004998.1711060820.git.gitgitgadget@gmail.com","threadId":"61055","inReplyTo":"pull.1683.v2.git.1711060819.gitgitgadget@gmail.com","subject":"[PATCH v2 3/3] reftable/segment: make segment end inclusive","fromName":"Justin Tobler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-03-21T22:40:19Z","receivedAt":"2024-03-21T22:40:27Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"From: Justin Tobler \n\nFor a reftable segment, the start of the range is inclusive and the end\nis exclusive. In practice we increment the end when creating the\ncompaction segment only to decrement the segment end when using it.\n\nSimplify by making the segment end inclusive. The corresponding test,\n`test_suggest_compaction_segment()`, is updated to show that the segment\nend is now inclusive.\n\nSigned-off-by: Justin Tobler \n---\n reftable/stack.c | 4 ++--\n reftable/stack_test.c | 2 +-\n 2 files changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex ef55dc75cde..0c16ae9b12d 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1243,7 +1243,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n \t */\n \tfor (i = n - 1; i > 0; i--) {\n \t\tif (sizes[i - 1] < sizes[i] * 2) {\n-\t\t\tseg.end = i + 1;\n+\t\t\tseg.end = i;\n \t\t\tbytes = sizes[i];\n \t\t\tbreak;\n \t\t}\n@@ -1292,7 +1292,7 @@ int reftable_stack_auto_compact(struct reftable_stack *st)\n \t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n \treftable_free(sizes);\n \tif (segment_size(&seg) > 0)\n-\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n+\t\treturn stack_compact_range_stats(st, seg.start, seg.end,\n \t\t\t\t\t\t NULL);\n \n \treturn 0;\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex e5f6ff5c9e4..85600a9573e 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -727,7 +727,7 @@ static void test_suggest_compaction_segment(void)\n \tstruct segment min =\n \t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n \tEXPECT(min.start == 1);\n-\tEXPECT(min.end == 10);\n+\tEXPECT(min.end == 9);\n }\n \n static void test_suggest_compaction_segment_nothing(void)\n-- \ngitgitgadget\n"},{"id":"491185","messageId":"g55luyh5diq6w4d5jigkhrwnxfu4ekz6nb2s7sjug2agjoexd5@63cyzht4ihcx","threadId":"61055","inReplyTo":"ZehjoXhH6deTJmSm@tanuki","subject":"Re: [PATCH] reftable/stack: use geometric table compaction","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-03-21T22:48:37Z","receivedAt":"2024-03-21T22:49:28Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/03/06 01:37PM, Patrick Steinhardt wrote:\n> On Tue, Mar 05, 2024 at 08:03:45PM +0000, Justin Tobler via GitGitGadget wrote:\n> > From: Justin Tobler \n> > @@ -1305,7 +1287,7 @@ int reftable_stack_auto_compact(struct reftable_stack *st)\n> > \t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n> > \treftable_free(sizes);\n> > \tif (segment_size(&seg) > 0)\n> > -\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n> > +\t\treturn stack_compact_range_stats(st, seg.start, seg.end,\n> > \t\t\t\t\t\t NULL);\n> > \n> > \treturn 0;\n> \n> One more thing: I think it would make sense to move the refactoring\n> where you change whether the end segment index is inclusive or exclusive\n> into a separate patch so that it's easier to reason about. Also, the\n> fact that no tests would require changes would further stress the point\n> that this is a mere refactoring without unintended side effects.\n\nThe `test_suggest_compaction_segment()` in `stack_test.c` does have to\nbe updated to reflect the segment end now being inclusive. But other\nthan that, no tests have to be updated.\n\nThanks Patrick for all the great feedback! I've updated per your\ncomments in V2 of the patch series.\n\n-Justin\n"},{"id":"491199","messageId":"Zfzd8YdAwzYRYkA3@tanuki","threadId":"61055","inReplyTo":"pull.1683.v2.git.1711060819.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/3] reftable/stack: use geometric table compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-03-22T01:25:05Z","receivedAt":"2024-03-22T01:25:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Mar 21, 2024 at 10:40:16PM +0000, Justin Tobler via GitGitGadget wrote:\n> Hello again,\n> \n> This is the second version my patch series that refactors the reftable\n> compaction strategy to instead follow a geometric sequence. Changes compared\n> to v1:\n> \n> * Added GIT_TEST_REFTABLE_NO_AUTOCOMPACTION environment variable to disable\n> reftable compaction when testing.\n> * Refactored worktree tests in t0610-reftable-basics.sh to properly assert\n> git-pack-refs(1) works as expected.\n> * Added test to validate that alternating table sizes are compacted.\n> * Added benchmark to compare compaction strategies.\n> * Moved change that made compaction segment end inclusive to its own\n> commit.\n> * Added additional explanation in commits and comments and fixed typos.\n> \n> Thanks for taking a look!\n\nCc'ing Han-Wen and Josh for additional input. From my point of view the\nnew algorithm is simpler to understand and less fragile, but I do wonder\nwhether there is anything that we're missing.\n\nPatrick\n"},{"id":"491200","messageId":"Zfzd-FcBNW2eHjEe@tanuki","threadId":"61055","inReplyTo":"cb6b152e5c86ab14165e072ac20dfa865b5f3aaa.1711060820.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/3] reftable/stack: add env to disable autocompaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-03-22T01:25:12Z","receivedAt":"2024-03-22T01:25:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Mar 21, 2024 at 10:40:17PM +0000, Justin Tobler via GitGitGadget wrote:\n> From: Justin Tobler \n> \n> In future tests it will be neccesary to create repositories with a set\n> number of tables. To make this easier, introduce the\n> `GIT_TEST_REFTABLE_NO_AUTOCOMPACTION` environment variable that, when\n> set, disables autocompaction of reftables.\n> \n> Signed-off-by: Justin Tobler \n\nMight be worth it to demonstrate in a test what this does, also to make\nsure that it actually works as expected and doesn't regress. Even though\nit may be a bit overboarding to add tests for test-only functionality.\nDunno... ultimately it wouldn't hurt though, I guess?\n\nPatrick\n\n> ---\n> reftable/stack.c | 2 +-\n> reftable/system.h | 1 +\n> 2 files changed, 2 insertions(+), 1 deletion(-)\n> \n> diff --git a/reftable/stack.c b/reftable/stack.c\n> index b64e55648aa..2370d93d13b 100644\n> --- a/reftable/stack.c\n> +++ b/reftable/stack.c\n> @@ -681,7 +681,7 @@ int reftable_addition_commit(struct reftable_addition *add)\n> \tif (err)\n> \t\tgoto done;\n> \n> -\tif (!add->stack->disable_auto_compact)\n> +\tif (!add->stack->disable_auto_compact && !git_env_bool(\"GIT_TEST_REFTABLE_NO_AUTOCOMPACTION\", 0))\n> \t\terr = reftable_stack_auto_compact(add->stack);\n> \n> done:\n> diff --git a/reftable/system.h b/reftable/system.h\n> index 6b74a815143..ec08b728177 100644\n> --- a/reftable/system.h\n> +++ b/reftable/system.h\n> @@ -15,6 +15,7 @@ license that can be found in the LICENSE file or at\n> #include \"strbuf.h\"\n> #include \"hash-ll.h\" /* hash ID, sizes.*/\n> #include \"dir.h\" /* remove_dir_recursively, for tests.*/\n> +#include \"parse.h\"\n> \n> int hash_size(uint32_t id);\n> \n> -- \n> gitgitgadget\n> \n"},{"id":"491201","messageId":"Zfzd_yxeXWWTJdyP@tanuki","threadId":"61055","inReplyTo":"def7008452303f71c1fa469609bc199c629a19ec.1711060820.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/3] reftable/stack: use geometric table compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-03-22T01:25:19Z","receivedAt":"2024-03-22T01:25:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Mar 21, 2024 at 10:40:18PM +0000, Justin Tobler via GitGitGadget wrote:\n> From: Justin Tobler \n> \n> To reduce the number of on-disk reftables, compaction is performed.\n> Contiguous tables with the same binary log value of size are grouped\n> into segments. The segment that has both the lowest binary log value and\n> contains more than one table is set as the starting point when\n> identifying the compaction segment.\n> \n> Since segments containing a single table are not initially considered\n> for compaction, if the table appended to the list does not match the\n> previous table log value, no compaction occurs for the new table. It is\n> therefore possible for unbounded growth of the table list. This can be\n> demonstrated by repeating the following sequence:\n> \n> git branch -f foo\n> git branch -d foo\n> \n> Each operation results in a new table being written with no compaction\n> occurring until a separate operation produces a table matching the\n> previous table log value.\n> \n> Instead, to avoid unbounded growth of the table list, the compaction\n> strategy is updated to ensure tables follow a geometric sequence after\n> each operation. This is done by walking the table list in reverse index\n> order to identify the compaction segment start and end. The compaction\n> segment end is found by identifying the first table which has a\n> preceding table size less than twice the current table. Next, the\n> compaction segment start is found iterating through the remaining tables\n> in the list checking if the previous table size is less than twice the\n> cumulative of tables from the segment end. This ensures the correct\n> segment start is found and that the newly compacted table does not\n> violate the geometric sequence.\n\nI don't think we need to go into so much detail how exactly the\nalgorithm is working -- these kind of comments should ideally exist in\nthe code. What would be more interesting to explain here is _why_ we\nchose the new algorithm over the old one instead of just trying to fix\nthe issue.\n\nOther than that this patch LGTM.\n\nPatrick\n\n> When creating 10 thousand references, the new strategy has no\n> performance impact:\n> \n> Benchmark 1: update-ref: create refs sequentially (revision = HEAD~)\n> Time (mean ± σ): 26.516 s ± 0.047 s [User: 17.864 s, System: 8.491 s]\n> Range (min … max): 26.447 s … 26.569 s 10 runs\n> \n> Benchmark 2: update-ref: create refs sequentially (revision = HEAD)\n> Time (mean ± σ): 26.417 s ± 0.028 s [User: 17.738 s, System: 8.500 s]\n> Range (min … max): 26.366 s … 26.444 s 10 runs\n> \n> Summary\n> update-ref: create refs sequentially (revision = HEAD) ran\n> 1.00 ± 0.00 times faster than update-ref: create refs sequentially (revision = HEAD~)\n> \n> Some tests in `t0610-reftable-basics.sh` assert the on-disk state of\n> tables and are therefore updated to specify the correct new table count.\n> Since compaction is more aggressive in ensuring tables maintain a\n> geometric sequence, the expected table count is reduced in these tests.\n> In `reftable/stack_test.c` tests related to `sizes_to_segments()` are\n> removed because the function is no longer needed. Also, the\n> `test_suggest_compaction_segment()` test is updated to better showcase\n> and reflect the new geometric compaction behavior.\n> \n> Signed-off-by: Justin Tobler \n> ---\n> reftable/stack.c | 109 ++++++++++++++++---------------------\n> reftable/stack.h | 3 -\n> reftable/stack_test.c | 66 +++++-----------------\n> t/t0610-reftable-basics.sh | 43 ++++++++++-----\n> 4 files changed, 91 insertions(+), 130 deletions(-)\n> \n> diff --git a/reftable/stack.c b/reftable/stack.c\n> index 2370d93d13b..ef55dc75cde 100644\n> --- a/reftable/stack.c\n> +++ b/reftable/stack.c\n> @@ -1214,75 +1214,62 @@ static int segment_size(struct segment *s)\n> \treturn s->end - s->start;\n> }\n> \n> -int fastlog2(uint64_t sz)\n> -{\n> -\tint l = 0;\n> -\tif (sz == 0)\n> -\t\treturn 0;\n> -\tfor (; sz; sz /= 2) {\n> -\t\tl++;\n> -\t}\n> -\treturn l - 1;\n> -}\n> -\n> -struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n)\n> -{\n> -\tstruct segment *segs = reftable_calloc(n, sizeof(*segs));\n> -\tstruct segment cur = { 0 };\n> -\tsize_t next = 0, i;\n> -\n> -\tif (n == 0) {\n> -\t\t*seglen = 0;\n> -\t\treturn segs;\n> -\t}\n> -\tfor (i = 0; i < n; i++) {\n> -\t\tint log = fastlog2(sizes[i]);\n> -\t\tif (cur.log != log && cur.bytes > 0) {\n> -\t\t\tstruct segment fresh = {\n> -\t\t\t\t.start = i,\n> -\t\t\t};\n> -\n> -\t\t\tsegs[next++] = cur;\n> -\t\t\tcur = fresh;\n> -\t\t}\n> -\n> -\t\tcur.log = log;\n> -\t\tcur.end = i + 1;\n> -\t\tcur.bytes += sizes[i];\n> -\t}\n> -\tsegs[next++] = cur;\n> -\t*seglen = next;\n> -\treturn segs;\n> -}\n> -\n> struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n> {\n> -\tstruct segment min_seg = {\n> -\t\t.log = 64,\n> -\t};\n> -\tstruct segment *segs;\n> -\tsize_t seglen = 0, i;\n> -\n> -\tsegs = sizes_to_segments(&seglen, sizes, n);\n> -\tfor (i = 0; i < seglen; i++) {\n> -\t\tif (segment_size(&segs[i]) == 1)\n> -\t\t\tcontinue;\n> +\tstruct segment seg = { 0 };\n> +\tuint64_t bytes;\n> +\tsize_t i;\n> \n> -\t\tif (segs[i].log < min_seg.log)\n> -\t\t\tmin_seg = segs[i];\n> -\t}\n> +\t/*\n> +\t * If there are no tables or only a single one then we don't have to\n> +\t * compact anything. The sequence is geometric by definition already.\n> +\t */\n> +\tif (n <= 1)\n> +\t\treturn seg;\n> \n> -\twhile (min_seg.start > 0) {\n> -\t\tsize_t prev = min_seg.start - 1;\n> -\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev]))\n> +\t/*\n> +\t * Find the ending table of the compaction segment needed to restore the\n> +\t * geometric sequence.\n> +\t *\n> +\t * To do so, we iterate backwards starting from the most recent table\n> +\t * until a valid segment end is found. If the preceding table is smaller\n> +\t * than the current table multiplied by the geometric factor (2), the\n> +\t * current table is set as the compaction segment end.\n> +\t *\n> +\t * Tables after the ending point are not added to the byte count because\n> +\t * they are already valid members of the geometric sequence. Due to the\n> +\t * properties of a geometric sequence, it is not possible for the sum of\n> +\t * these tables to exceed the value of the ending point table.\n> +\t */\n> +\tfor (i = n - 1; i > 0; i--) {\n> +\t\tif (sizes[i - 1] < sizes[i] * 2) {\n> +\t\t\tseg.end = i + 1;\n> +\t\t\tbytes = sizes[i];\n> \t\t\tbreak;\n> +\t\t}\n> +\t}\n> +\n> +\t/*\n> +\t * Find the starting table of the compaction segment by iterating\n> +\t * through the remaining tables and keeping track of the accumulated\n> +\t * size of all tables seen from the segment end table.\n> +\t *\n> +\t * Note that we keep iterating even after we have found the first\n> +\t * starting point. This is because there may be tables in the stack\n> +\t * preceding that first starting point which violate the geometric\n> +\t * sequence.\n> +\t */\n> +\tfor (; i > 0; i--) {\n> +\t\tuint64_t curr = bytes;\n> +\t\tbytes += sizes[i - 1];\n> \n> -\t\tmin_seg.start = prev;\n> -\t\tmin_seg.bytes += sizes[prev];\n> +\t\tif (sizes[i - 1] < curr * 2) {\n> +\t\t\tseg.start = i - 1;\n> +\t\t\tseg.bytes = bytes;\n> +\t\t}\n> \t}\n> \n> -\treftable_free(segs);\n> -\treturn min_seg;\n> +\treturn seg;\n> }\n> \n> static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n> diff --git a/reftable/stack.h b/reftable/stack.h\n> index d919455669e..656f896cc28 100644\n> --- a/reftable/stack.h\n> +++ b/reftable/stack.h\n> @@ -33,12 +33,9 @@ int read_lines(const char *filename, char ***lines);\n> \n> struct segment {\n> \tsize_t start, end;\n> -\tint log;\n> \tuint64_t bytes;\n> };\n> \n> -int fastlog2(uint64_t sz);\n> -struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n);\n> struct segment suggest_compaction_segment(uint64_t *sizes, size_t n);\n> \n> #endif\n> diff --git a/reftable/stack_test.c b/reftable/stack_test.c\n> index 509f4866236..e5f6ff5c9e4 100644\n> --- a/reftable/stack_test.c\n> +++ b/reftable/stack_test.c\n> @@ -720,59 +720,14 @@ static void test_reftable_stack_hash_id(void)\n> \tclear_dir(dir);\n> }\n> \n> -static void test_log2(void)\n> -{\n> -\tEXPECT(1 == fastlog2(3));\n> -\tEXPECT(2 == fastlog2(4));\n> -\tEXPECT(2 == fastlog2(5));\n> -}\n> -\n> -static void test_sizes_to_segments(void)\n> -{\n> -\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n> -\t/* .................0 1 2 3 4 5 */\n> -\n> -\tsize_t seglen = 0;\n> -\tstruct segment *segs =\n> -\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n> -\tEXPECT(segs[2].log == 3);\n> -\tEXPECT(segs[2].start == 5);\n> -\tEXPECT(segs[2].end == 6);\n> -\n> -\tEXPECT(segs[1].log == 2);\n> -\tEXPECT(segs[1].start == 2);\n> -\tEXPECT(segs[1].end == 5);\n> -\treftable_free(segs);\n> -}\n> -\n> -static void test_sizes_to_segments_empty(void)\n> -{\n> -\tsize_t seglen = 0;\n> -\tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n> -\tEXPECT(seglen == 0);\n> -\treftable_free(segs);\n> -}\n> -\n> -static void test_sizes_to_segments_all_equal(void)\n> -{\n> -\tuint64_t sizes[] = { 5, 5 };\n> -\tsize_t seglen = 0;\n> -\tstruct segment *segs =\n> -\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n> -\tEXPECT(seglen == 1);\n> -\tEXPECT(segs[0].start == 0);\n> -\tEXPECT(segs[0].end == 2);\n> -\treftable_free(segs);\n> -}\n> -\n> static void test_suggest_compaction_segment(void)\n> {\n> -\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n> +\tuint64_t sizes[] = { 512, 64, 17, 16, 9, 9, 9, 16, 2, 16 };\n> \t/* .................0 1 2 3 4 5 6 */\n> \tstruct segment min =\n> \t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n> -\tEXPECT(min.start == 2);\n> -\tEXPECT(min.end == 7);\n> +\tEXPECT(min.start == 1);\n> +\tEXPECT(min.end == 10);\n> }\n> \n> static void test_suggest_compaction_segment_nothing(void)\n> @@ -884,6 +839,17 @@ static void test_empty_add(void)\n> \treftable_stack_destroy(st2);\n> }\n> \n> +static int fastlog2(uint64_t sz)\n> +{\n> +\tint l = 0;\n> +\tif (sz == 0)\n> +\t\treturn 0;\n> +\tfor (; sz; sz /= 2) {\n> +\t\tl++;\n> +\t}\n> +\treturn l - 1;\n> +}\n> +\n> static void test_reftable_stack_auto_compaction(void)\n> {\n> \tstruct reftable_write_options cfg = { 0 };\n> @@ -1072,7 +1038,6 @@ static void test_reftable_stack_compaction_concurrent_clean(void)\n> int stack_test_main(int argc, const char *argv[])\n> {\n> \tRUN_TEST(test_empty_add);\n> -\tRUN_TEST(test_log2);\n> \tRUN_TEST(test_names_equal);\n> \tRUN_TEST(test_parse_names);\n> \tRUN_TEST(test_read_file);\n> @@ -1092,9 +1057,6 @@ int stack_test_main(int argc, const char *argv[])\n> \tRUN_TEST(test_reftable_stack_update_index_check);\n> \tRUN_TEST(test_reftable_stack_uptodate);\n> \tRUN_TEST(test_reftable_stack_validate_refname);\n> -\tRUN_TEST(test_sizes_to_segments);\n> -\tRUN_TEST(test_sizes_to_segments_all_equal);\n> -\tRUN_TEST(test_sizes_to_segments_empty);\n> \tRUN_TEST(test_suggest_compaction_segment);\n> \tRUN_TEST(test_suggest_compaction_segment_nothing);\n> \treturn 0;\n> diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\n> index 686781192eb..e6c3f94d874 100755\n> --- a/t/t0610-reftable-basics.sh\n> +++ b/t/t0610-reftable-basics.sh\n> @@ -293,12 +293,24 @@ test_expect_success 'ref transaction: writes cause auto-compaction' '\n> \ttest_line_count = 1 repo/.git/reftable/tables.list &&\n> \n> \ttest_commit -C repo --no-tag A &&\n> -\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n> +\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n> \n> \ttest_commit -C repo --no-tag B &&\n> \ttest_line_count = 1 repo/.git/reftable/tables.list\n> '\n> \n> +test_expect_success 'ref transaction: alternating table sizes are compacted' '\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\tgit init repo &&\n> +\ttest_commit -C repo A &&\n> +\tfor i in $(test_seq 20)\n> +\tdo\n> +\t\tgit -C repo branch -f foo &&\n> +\t\tgit -C repo branch -d foo || return 1\n> +\tdone &&\n> +\ttest_line_count = 2 repo/.git/reftable/tables.list\n> +'\n> +\n> check_fsync_events () {\n> \tlocal trace=\"$1\" &&\n> \tshift &&\n> @@ -324,7 +336,7 @@ test_expect_success 'ref transaction: writes are synced' '\n> \t\tgit -C repo -c core.fsync=reference \\\n> \t\t-c core.fsyncMethod=fsync update-ref refs/heads/branch HEAD &&\n> \tcheck_fsync_events trace2.txt <<-EOF\n> -\t\"name\":\"hardware-flush\",\"count\":2\n> +\t\"name\":\"hardware-flush\",\"count\":4\n> \tEOF\n> '\n> \n> @@ -346,8 +358,8 @@ test_expect_success 'pack-refs: compacts tables' '\n> \n> \ttest_commit -C repo A &&\n> \tls -1 repo/.git/reftable >table-files &&\n> -\ttest_line_count = 4 table-files &&\n> -\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n> +\ttest_line_count = 3 table-files &&\n> +\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n> \n> \tgit -C repo pack-refs &&\n> \tls -1 repo/.git/reftable >table-files &&\n> @@ -379,7 +391,7 @@ do\n> \t\t\tumask $umask &&\n> \t\t\tgit init --shared=true repo &&\n> \t\t\ttest_commit -C repo A &&\n> -\t\t\ttest_line_count = 3 repo/.git/reftable/tables.list\n> +\t\t\ttest_line_count = 2 repo/.git/reftable/tables.list\n> \t\t) &&\n> \t\tgit -C repo pack-refs &&\n> \t\ttest_expect_perms \"-rw-rw-r--\" repo/.git/reftable/tables.list &&\n> @@ -747,12 +759,13 @@ test_expect_success 'worktree: pack-refs in main repo packs main refs' '\n> \ttest_when_finished \"rm -rf repo worktree\" &&\n> \tgit init repo &&\n> \ttest_commit -C repo A &&\n> -\tgit -C repo worktree add ../worktree &&\n> +\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo worktree add ../worktree &&\n> +\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C worktree update-ref refs/worktree/per-worktree HEAD &&\n> \n> -\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n> -\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n> +\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n> +\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n> \tgit -C repo pack-refs &&\n> -\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n> +\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n> \ttest_line_count = 1 repo/.git/reftable/tables.list\n> '\n> \n> @@ -760,19 +773,21 @@ test_expect_success 'worktree: pack-refs in worktree packs worktree refs' '\n> \ttest_when_finished \"rm -rf repo worktree\" &&\n> \tgit init repo &&\n> \ttest_commit -C repo A &&\n> -\tgit -C repo worktree add ../worktree &&\n> +\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo worktree add ../worktree &&\n> +\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C worktree update-ref refs/worktree/per-worktree HEAD &&\n> \n> -\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n> -\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n> +\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n> +\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n> \tgit -C worktree pack-refs &&\n> \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n> -\ttest_line_count = 4 repo/.git/reftable/tables.list\n> +\ttest_line_count = 3 repo/.git/reftable/tables.list\n> '\n> \n> test_expect_success 'worktree: creating shared ref updates main stack' '\n> \ttest_when_finished \"rm -rf repo worktree\" &&\n> \tgit init repo &&\n> \ttest_commit -C repo A &&\n> +\ttest_commit -C repo B &&\n> \n> \tgit -C repo worktree add ../worktree &&\n> \tgit -C repo pack-refs &&\n> @@ -780,7 +795,7 @@ test_expect_success 'worktree: creating shared ref updates main stack' '\n> \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n> \ttest_line_count = 1 repo/.git/reftable/tables.list &&\n> \n> -\tgit -C worktree update-ref refs/heads/shared HEAD &&\n> +\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C worktree update-ref refs/heads/shared HEAD &&\n> \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n> \ttest_line_count = 2 repo/.git/reftable/tables.list\n> '\n> -- \n> gitgitgadget\n> \n"},{"id":"491693","messageId":"CAOLa=ZQFiBKWs1qT=MyJhBKgn8MJBL-5G6X7EjeXkKwNOaCC4w@mail.gmail.com","threadId":"61055","inReplyTo":"def7008452303f71c1fa469609bc199c629a19ec.1711060820.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/3] reftable/stack: use geometric table compaction","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-03-27T13:24:23Z","receivedAt":"2024-03-27T13:24:27Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"\"Justin Tobler via GitGitGadget\" writes:\n\n> From: Justin Tobler \n>\n> To reduce the number of on-disk reftables, compaction is performed.\n> Contiguous tables with the same binary log value of size are grouped\n> into segments. The segment that has both the lowest binary log value and\n> contains more than one table is set as the starting point when\n> identifying the compaction segment.\n>\n> Since segments containing a single table are not initially considered\n> for compaction, if the table appended to the list does not match the\n> previous table log value, no compaction occurs for the new table. It is\n> therefore possible for unbounded growth of the table list. This can be\n> demonstrated by repeating the following sequence:\n>\n\nNit: A numerical example would really help make this simpler to understand.\n\n> +\t/*\n> +\t * Find the ending table of the compaction segment needed to restore the\n> +\t * geometric sequence.\n> +\t *\n> +\t * To do so, we iterate backwards starting from the most recent table\n> +\t * until a valid segment end is found. If the preceding table is smaller\n> +\t * than the current table multiplied by the geometric factor (2), the\n> +\t * current table is set as the compaction segment end.\n> +\t *\n> +\t * Tables after the ending point are not added to the byte count because\n> +\t * they are already valid members of the geometric sequence. Due to the\n> +\t * properties of a geometric sequence, it is not possible for the sum of\n> +\t * these tables to exceed the value of the ending point table.\n> +\t */\n> +\tfor (i = n - 1; i > 0; i--) {\n> +\t\tif (sizes[i - 1] < sizes[i] * 2) {\n> +\t\t\tseg.end = i + 1;\n> +\t\t\tbytes = sizes[i];\n> \t\t\tbreak;\n> +\t\t}\n> +\t}\n> +\n> +\t/*\n> +\t * Find the starting table of the compaction segment by iterating\n> +\t * through the remaining tables and keeping track of the accumulated\n> +\t * size of all tables seen from the segment end table.\n> +\t *\n\nNit: we need the accumulated sum because the tables from the end of the\nsegment will be recursively merged backwards. This might be worthwhile\nto add here.\n\n\n> static void test_suggest_compaction_segment(void)\n> {\n> -\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n> +\tuint64_t sizes[] = { 512, 64, 17, 16, 9, 9, 9, 16, 2, 16 };\n> \t/* .................0 1 2 3 4 5 6 */\n\nNit: since we're here, maybe worthwhile cleaning up this comment. Not\nsure what it actually is for.\n"},{"id":"491807","messageId":"pull.1683.v3.git.1711685809.gitgitgadget@gmail.com","threadId":"61055","inReplyTo":"pull.1683.v2.git.1711060819.gitgitgadget@gmail.com","subject":"[PATCH v3 0/3] reftable/stack: use geometric table compaction","fromName":"Justin Tobler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-03-29T04:16:46Z","receivedAt":"2024-03-29T04:16:54Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Hello again,\n\nThis is the third version my patch series that refactors the reftable\ncompaction strategy to instead follow a geometric sequence. Changes compared\nto v2:\n\n * Added test to validate the GIT_TEST_REFTABLE_NO_AUTOCOMPACTION\n environment variable works as expected.\n * Added additional clarifying comments and examples to explain how the new\n compaction strategy works.\n * Removed outdated comment from stack_test.c test\n\nThanks for taking a look!\n\n-Justin\n\nJustin Tobler (3):\n reftable/stack: add env to disable autocompaction\n reftable/stack: use geometric table compaction\n reftable/stack: make segment end inclusive\n\n reftable/stack.c | 124 ++++++++++++++++++-------------------\n reftable/stack.h | 3 -\n reftable/stack_test.c | 67 +++++---------------\n reftable/system.h | 1 +\n t/t0610-reftable-basics.sh | 58 ++++++++++++-----\n 5 files changed, 120 insertions(+), 133 deletions(-)\n\n\nbase-commit: c75fd8d8150afdf836b63a8e0534d9b9e3e111ba\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1683%2Fjltobler%2Fjt%2Freftable-geometric-compaction-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1683/jltobler/jt/reftable-geometric-compaction-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1683\n\nRange-diff vs v2:\n\n 1: cb6b152e5c8 ! 1: 2fdd8ea1133 reftable/stack: add env to disable autocompaction\n @@ reftable/stack.c: int reftable_addition_commit(struct reftable_addition *add)\n \n ## reftable/system.h ##\n @@ reftable/system.h: license that can be found in the LICENSE file or at\n - #include \"strbuf.h\"\n + #include \"tempfile.h\"\n #include \"hash-ll.h\" /* hash ID, sizes.*/\n #include \"dir.h\" /* remove_dir_recursively, for tests.*/\n +#include \"parse.h\"\n \n int hash_size(uint32_t id);\n \n +\n + ## t/t0610-reftable-basics.sh ##\n +@@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: writes cause auto-compaction' '\n + \ttest_line_count = 1 repo/.git/reftable/tables.list\n + '\n + \n ++test_expect_success 'ref transaction: environment variable disables auto-compaction' '\n ++\ttest_when_finished \"rm -rf repo\" &&\n ++\n ++\tgit init repo &&\n ++\ttest_commit -C repo A &&\n ++\tfor i in $(test_seq 20)\n ++\tdo\n ++\t\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo update-ref branch-$i HEAD || return 1\n ++\tdone &&\n ++\ttest_line_count = 23 repo/.git/reftable/tables.list &&\n ++\n ++\tgit -C repo update-ref foo HEAD &&\n ++\ttest_line_count = 1 repo/.git/reftable/tables.list\n ++'\n ++\n + check_fsync_events () {\n + \tlocal trace=\"$1\" &&\n + \tshift &&\n 2: def70084523 ! 2: 7e62c2286ae reftable/stack: use geometric table compaction\n @@ Commit message\n \n Instead, to avoid unbounded growth of the table list, the compaction\n strategy is updated to ensure tables follow a geometric sequence after\n - each operation. This is done by walking the table list in reverse index\n - order to identify the compaction segment start and end. The compaction\n - segment end is found by identifying the first table which has a\n - preceding table size less than twice the current table. Next, the\n - compaction segment start is found iterating through the remaining tables\n - in the list checking if the previous table size is less than twice the\n - cumulative of tables from the segment end. This ensures the correct\n - segment start is found and that the newly compacted table does not\n - violate the geometric sequence.\n + each operation by individually evaluating each table in reverse index\n + order. This strategy results in a much simpler and more robust algorithm\n + compared to the previous one while also maintaining a minimal ordered\n + set of tables on-disk.\n \n When creating 10 thousand references, the new strategy has no\n performance impact:\n @@ reftable/stack.c: static int segment_size(struct segment *s)\n +\t * they are already valid members of the geometric sequence. Due to the\n +\t * properties of a geometric sequence, it is not possible for the sum of\n +\t * these tables to exceed the value of the ending point table.\n ++\t *\n ++\t * Example table size sequence requiring no compaction:\n ++\t * \t64, 32, 16, 8, 4, 2, 1\n ++\t *\n ++\t * Example compaction segment end set to table with size 3:\n ++\t * \t64, 32, 16, 8, 4, 3, 1\n +\t */\n +\tfor (i = n - 1; i > 0; i--) {\n +\t\tif (sizes[i - 1] < sizes[i] * 2) {\n @@ reftable/stack.c: static int segment_size(struct segment *s)\n \t\t\tbreak;\n +\t\t}\n +\t}\n -+\n + \n +-\t\tmin_seg.start = prev;\n +-\t\tmin_seg.bytes += sizes[prev];\n +\t/*\n +\t * Find the starting table of the compaction segment by iterating\n +\t * through the remaining tables and keeping track of the accumulated\n -+\t * size of all tables seen from the segment end table.\n ++\t * size of all tables seen from the segment end table. The previous\n ++\t * table is compared to the accumulated size because the tables from the\n ++\t * segment end are merged backwards recursively.\n +\t *\n +\t * Note that we keep iterating even after we have found the first\n +\t * starting point. This is because there may be tables in the stack\n +\t * preceding that first starting point which violate the geometric\n +\t * sequence.\n ++\t *\n ++\t * Example compaction segment start set to table with size 32:\n ++\t * \t128, 32, 16, 8, 4, 3, 1\n +\t */\n +\tfor (; i > 0; i--) {\n +\t\tuint64_t curr = bytes;\n +\t\tbytes += sizes[i - 1];\n - \n --\t\tmin_seg.start = prev;\n --\t\tmin_seg.bytes += sizes[prev];\n ++\n +\t\tif (sizes[i - 1] < curr * 2) {\n +\t\t\tseg.start = i - 1;\n +\t\t\tseg.bytes = bytes;\n @@ reftable/stack_test.c: static void test_reftable_stack_hash_id(void)\n static void test_suggest_compaction_segment(void)\n {\n -\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n +-\t/* .................0 1 2 3 4 5 6 */\n +\tuint64_t sizes[] = { 512, 64, 17, 16, 9, 9, 9, 16, 2, 16 };\n - \t/* .................0 1 2 3 4 5 6 */\n \tstruct segment min =\n \t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n -\tEXPECT(min.start == 2);\n @@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: writes cause a\n \n \ttest_commit -C repo --no-tag B &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n +@@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: environment variable disables auto-compact\n + \tdo\n + \t\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo update-ref branch-$i HEAD || return 1\n + \tdone &&\n +-\ttest_line_count = 23 repo/.git/reftable/tables.list &&\n ++\ttest_line_count = 22 repo/.git/reftable/tables.list &&\n + \n + \tgit -C repo update-ref foo HEAD &&\n + \ttest_line_count = 1 repo/.git/reftable/tables.list\n '\n \n +test_expect_success 'ref transaction: alternating table sizes are compacted' '\n 3: a23e3fc6972 ! 3: 9a33914c852 reftable/segment: make segment end inclusive\n @@ Metadata\n Author: Justin Tobler \n \n ## Commit message ##\n - reftable/segment: make segment end inclusive\n + reftable/stack: make segment end inclusive\n \n For a reftable segment, the start of the range is inclusive and the end\n is exclusive. In practice we increment the end when creating the\n\n-- \ngitgitgadget\n"},{"id":"491806","messageId":"2fdd8ea11331da13ec56d511fe4cadd47544419f.1711685809.git.gitgitgadget@gmail.com","threadId":"61055","inReplyTo":"pull.1683.v3.git.1711685809.gitgitgadget@gmail.com","subject":"[PATCH v3 1/3] reftable/stack: add env to disable autocompaction","fromName":"Justin Tobler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-03-29T04:16:47Z","receivedAt":"2024-03-29T04:16:55Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"From: Justin Tobler \n\nIn future tests it will be neccesary to create repositories with a set\nnumber of tables. To make this easier, introduce the\n`GIT_TEST_REFTABLE_NO_AUTOCOMPACTION` environment variable that, when\nset, disables autocompaction of reftables.\n\nSigned-off-by: Justin Tobler \n---\n reftable/stack.c | 2 +-\n reftable/system.h | 1 +\n t/t0610-reftable-basics.sh | 15 +++++++++++++++\n 3 files changed, 17 insertions(+), 1 deletion(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 1ecf1b9751c..07262beaaf7 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -681,7 +681,7 @@ int reftable_addition_commit(struct reftable_addition *add)\n \tif (err)\n \t\tgoto done;\n \n-\tif (!add->stack->disable_auto_compact)\n+\tif (!add->stack->disable_auto_compact && !git_env_bool(\"GIT_TEST_REFTABLE_NO_AUTOCOMPACTION\", 0))\n \t\terr = reftable_stack_auto_compact(add->stack);\n \n done:\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 5d8b6dede50..05b7c8554af 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -17,6 +17,7 @@ license that can be found in the LICENSE file or at\n #include \"tempfile.h\"\n #include \"hash-ll.h\" /* hash ID, sizes.*/\n #include \"dir.h\" /* remove_dir_recursively, for tests.*/\n+#include \"parse.h\"\n \n int hash_size(uint32_t id);\n \ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex 686781192eb..434044078ed 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -299,6 +299,21 @@ test_expect_success 'ref transaction: writes cause auto-compaction' '\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n '\n \n+test_expect_success 'ref transaction: environment variable disables auto-compaction' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tfor i in $(test_seq 20)\n+\tdo\n+\t\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo update-ref branch-$i HEAD || return 1\n+\tdone &&\n+\ttest_line_count = 23 repo/.git/reftable/tables.list &&\n+\n+\tgit -C repo update-ref foo HEAD &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list\n+'\n+\n check_fsync_events () {\n \tlocal trace=\"$1\" &&\n \tshift &&\n-- \ngitgitgadget\n\n"},{"id":"491808","messageId":"9a33914c852a0487dbd90c83f53fa0e36414fda1.1711685809.git.gitgitgadget@gmail.com","threadId":"61055","inReplyTo":"pull.1683.v3.git.1711685809.gitgitgadget@gmail.com","subject":"[PATCH v3 3/3] reftable/stack: make segment end inclusive","fromName":"Justin Tobler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-03-29T04:16:49Z","receivedAt":"2024-03-29T04:16:55Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"From: Justin Tobler \n\nFor a reftable segment, the start of the range is inclusive and the end\nis exclusive. In practice we increment the end when creating the\ncompaction segment only to decrement the segment end when using it.\n\nSimplify by making the segment end inclusive. The corresponding test,\n`test_suggest_compaction_segment()`, is updated to show that the segment\nend is now inclusive.\n\nSigned-off-by: Justin Tobler \n---\n reftable/stack.c | 4 ++--\n reftable/stack_test.c | 2 +-\n 2 files changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex e7b9a1de5a4..0973c47dd92 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1237,7 +1237,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n \t */\n \tfor (i = n - 1; i > 0; i--) {\n \t\tif (sizes[i - 1] < sizes[i] * 2) {\n-\t\t\tseg.end = i + 1;\n+\t\t\tseg.end = i;\n \t\t\tbytes = sizes[i];\n \t\t\tbreak;\n \t\t}\n@@ -1291,7 +1291,7 @@ int reftable_stack_auto_compact(struct reftable_stack *st)\n \t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n \treftable_free(sizes);\n \tif (segment_size(&seg) > 0)\n-\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n+\t\treturn stack_compact_range_stats(st, seg.start, seg.end,\n \t\t\t\t\t\t NULL);\n \n \treturn 0;\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 21541742fe5..4d7305623a0 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -723,7 +723,7 @@ static void test_suggest_compaction_segment(void)\n \tstruct segment min =\n \t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n \tEXPECT(min.start == 1);\n-\tEXPECT(min.end == 10);\n+\tEXPECT(min.end == 9);\n }\n \n static void test_suggest_compaction_segment_nothing(void)\n-- \ngitgitgadget\n"},{"id":"491809","messageId":"7e62c2286ae9c63368f1508f065b33422ca24bc8.1711685809.git.gitgitgadget@gmail.com","threadId":"61055","inReplyTo":"pull.1683.v3.git.1711685809.gitgitgadget@gmail.com","subject":"[PATCH v3 2/3] reftable/stack: use geometric table compaction","fromName":"Justin Tobler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-03-29T04:16:48Z","receivedAt":"2024-03-29T04:16:55Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"From: Justin Tobler \n\nTo reduce the number of on-disk reftables, compaction is performed.\nContiguous tables with the same binary log value of size are grouped\ninto segments. The segment that has both the lowest binary log value and\ncontains more than one table is set as the starting point when\nidentifying the compaction segment.\n\nSince segments containing a single table are not initially considered\nfor compaction, if the table appended to the list does not match the\nprevious table log value, no compaction occurs for the new table. It is\ntherefore possible for unbounded growth of the table list. This can be\ndemonstrated by repeating the following sequence:\n\ngit branch -f foo\ngit branch -d foo\n\nEach operation results in a new table being written with no compaction\noccurring until a separate operation produces a table matching the\nprevious table log value.\n\nInstead, to avoid unbounded growth of the table list, the compaction\nstrategy is updated to ensure tables follow a geometric sequence after\neach operation by individually evaluating each table in reverse index\norder. This strategy results in a much simpler and more robust algorithm\ncompared to the previous one while also maintaining a minimal ordered\nset of tables on-disk.\n\nWhen creating 10 thousand references, the new strategy has no\nperformance impact:\n\nBenchmark 1: update-ref: create refs sequentially (revision = HEAD~)\n Time (mean ± σ): 26.516 s ± 0.047 s [User: 17.864 s, System: 8.491 s]\n Range (min … max): 26.447 s … 26.569 s 10 runs\n\nBenchmark 2: update-ref: create refs sequentially (revision = HEAD)\n Time (mean ± σ): 26.417 s ± 0.028 s [User: 17.738 s, System: 8.500 s]\n Range (min … max): 26.366 s … 26.444 s 10 runs\n\nSummary\n update-ref: create refs sequentially (revision = HEAD) ran\n 1.00 ± 0.00 times faster than update-ref: create refs sequentially (revision = HEAD~)\n\nSome tests in `t0610-reftable-basics.sh` assert the on-disk state of\ntables and are therefore updated to specify the correct new table count.\nSince compaction is more aggressive in ensuring tables maintain a\ngeometric sequence, the expected table count is reduced in these tests.\nIn `reftable/stack_test.c` tests related to `sizes_to_segments()` are\nremoved because the function is no longer needed. Also, the\n`test_suggest_compaction_segment()` test is updated to better showcase\nand reflect the new geometric compaction behavior.\n\nSigned-off-by: Justin Tobler \n---\n reftable/stack.c | 120 ++++++++++++++++++-------------------\n reftable/stack.h | 3 -\n reftable/stack_test.c | 67 +++++----------------\n t/t0610-reftable-basics.sh | 45 +++++++++-----\n 4 files changed, 103 insertions(+), 132 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 07262beaaf7..e7b9a1de5a4 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1202,75 +1202,73 @@ static int segment_size(struct segment *s)\n \treturn s->end - s->start;\n }\n \n-int fastlog2(uint64_t sz)\n-{\n-\tint l = 0;\n-\tif (sz == 0)\n-\t\treturn 0;\n-\tfor (; sz; sz /= 2) {\n-\t\tl++;\n-\t}\n-\treturn l - 1;\n-}\n-\n-struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n)\n-{\n-\tstruct segment *segs = reftable_calloc(n, sizeof(*segs));\n-\tstruct segment cur = { 0 };\n-\tsize_t next = 0, i;\n-\n-\tif (n == 0) {\n-\t\t*seglen = 0;\n-\t\treturn segs;\n-\t}\n-\tfor (i = 0; i < n; i++) {\n-\t\tint log = fastlog2(sizes[i]);\n-\t\tif (cur.log != log && cur.bytes > 0) {\n-\t\t\tstruct segment fresh = {\n-\t\t\t\t.start = i,\n-\t\t\t};\n-\n-\t\t\tsegs[next++] = cur;\n-\t\t\tcur = fresh;\n-\t\t}\n-\n-\t\tcur.log = log;\n-\t\tcur.end = i + 1;\n-\t\tcur.bytes += sizes[i];\n-\t}\n-\tsegs[next++] = cur;\n-\t*seglen = next;\n-\treturn segs;\n-}\n-\n struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n {\n-\tstruct segment min_seg = {\n-\t\t.log = 64,\n-\t};\n-\tstruct segment *segs;\n-\tsize_t seglen = 0, i;\n-\n-\tsegs = sizes_to_segments(&seglen, sizes, n);\n-\tfor (i = 0; i < seglen; i++) {\n-\t\tif (segment_size(&segs[i]) == 1)\n-\t\t\tcontinue;\n+\tstruct segment seg = { 0 };\n+\tuint64_t bytes;\n+\tsize_t i;\n \n-\t\tif (segs[i].log < min_seg.log)\n-\t\t\tmin_seg = segs[i];\n-\t}\n+\t/*\n+\t * If there are no tables or only a single one then we don't have to\n+\t * compact anything. The sequence is geometric by definition already.\n+\t */\n+\tif (n <= 1)\n+\t\treturn seg;\n \n-\twhile (min_seg.start > 0) {\n-\t\tsize_t prev = min_seg.start - 1;\n-\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev]))\n+\t/*\n+\t * Find the ending table of the compaction segment needed to restore the\n+\t * geometric sequence.\n+\t *\n+\t * To do so, we iterate backwards starting from the most recent table\n+\t * until a valid segment end is found. If the preceding table is smaller\n+\t * than the current table multiplied by the geometric factor (2), the\n+\t * current table is set as the compaction segment end.\n+\t *\n+\t * Tables after the ending point are not added to the byte count because\n+\t * they are already valid members of the geometric sequence. Due to the\n+\t * properties of a geometric sequence, it is not possible for the sum of\n+\t * these tables to exceed the value of the ending point table.\n+\t *\n+\t * Example table size sequence requiring no compaction:\n+\t * \t64, 32, 16, 8, 4, 2, 1\n+\t *\n+\t * Example compaction segment end set to table with size 3:\n+\t * \t64, 32, 16, 8, 4, 3, 1\n+\t */\n+\tfor (i = n - 1; i > 0; i--) {\n+\t\tif (sizes[i - 1] < sizes[i] * 2) {\n+\t\t\tseg.end = i + 1;\n+\t\t\tbytes = sizes[i];\n \t\t\tbreak;\n+\t\t}\n+\t}\n \n-\t\tmin_seg.start = prev;\n-\t\tmin_seg.bytes += sizes[prev];\n+\t/*\n+\t * Find the starting table of the compaction segment by iterating\n+\t * through the remaining tables and keeping track of the accumulated\n+\t * size of all tables seen from the segment end table. The previous\n+\t * table is compared to the accumulated size because the tables from the\n+\t * segment end are merged backwards recursively.\n+\t *\n+\t * Note that we keep iterating even after we have found the first\n+\t * starting point. This is because there may be tables in the stack\n+\t * preceding that first starting point which violate the geometric\n+\t * sequence.\n+\t *\n+\t * Example compaction segment start set to table with size 32:\n+\t * \t128, 32, 16, 8, 4, 3, 1\n+\t */\n+\tfor (; i > 0; i--) {\n+\t\tuint64_t curr = bytes;\n+\t\tbytes += sizes[i - 1];\n+\n+\t\tif (sizes[i - 1] < curr * 2) {\n+\t\t\tseg.start = i - 1;\n+\t\t\tseg.bytes = bytes;\n+\t\t}\n \t}\n \n-\treftable_free(segs);\n-\treturn min_seg;\n+\treturn seg;\n }\n \n static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\ndiff --git a/reftable/stack.h b/reftable/stack.h\nindex d919455669e..656f896cc28 100644\n--- a/reftable/stack.h\n+++ b/reftable/stack.h\n@@ -33,12 +33,9 @@ int read_lines(const char *filename, char ***lines);\n \n struct segment {\n \tsize_t start, end;\n-\tint log;\n \tuint64_t bytes;\n };\n \n-int fastlog2(uint64_t sz);\n-struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n);\n struct segment suggest_compaction_segment(uint64_t *sizes, size_t n);\n \n #endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 7336757cf53..21541742fe5 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -717,59 +717,13 @@ static void test_reftable_stack_hash_id(void)\n \tclear_dir(dir);\n }\n \n-static void test_log2(void)\n-{\n-\tEXPECT(1 == fastlog2(3));\n-\tEXPECT(2 == fastlog2(4));\n-\tEXPECT(2 == fastlog2(5));\n-}\n-\n-static void test_sizes_to_segments(void)\n-{\n-\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n-\t/* .................0 1 2 3 4 5 */\n-\n-\tsize_t seglen = 0;\n-\tstruct segment *segs =\n-\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n-\tEXPECT(segs[2].log == 3);\n-\tEXPECT(segs[2].start == 5);\n-\tEXPECT(segs[2].end == 6);\n-\n-\tEXPECT(segs[1].log == 2);\n-\tEXPECT(segs[1].start == 2);\n-\tEXPECT(segs[1].end == 5);\n-\treftable_free(segs);\n-}\n-\n-static void test_sizes_to_segments_empty(void)\n-{\n-\tsize_t seglen = 0;\n-\tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n-\tEXPECT(seglen == 0);\n-\treftable_free(segs);\n-}\n-\n-static void test_sizes_to_segments_all_equal(void)\n-{\n-\tuint64_t sizes[] = { 5, 5 };\n-\tsize_t seglen = 0;\n-\tstruct segment *segs =\n-\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n-\tEXPECT(seglen == 1);\n-\tEXPECT(segs[0].start == 0);\n-\tEXPECT(segs[0].end == 2);\n-\treftable_free(segs);\n-}\n-\n static void test_suggest_compaction_segment(void)\n {\n-\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n-\t/* .................0 1 2 3 4 5 6 */\n+\tuint64_t sizes[] = { 512, 64, 17, 16, 9, 9, 9, 16, 2, 16 };\n \tstruct segment min =\n \t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n-\tEXPECT(min.start == 2);\n-\tEXPECT(min.end == 7);\n+\tEXPECT(min.start == 1);\n+\tEXPECT(min.end == 10);\n }\n \n static void test_suggest_compaction_segment_nothing(void)\n@@ -880,6 +834,17 @@ static void test_empty_add(void)\n \treftable_stack_destroy(st2);\n }\n \n+static int fastlog2(uint64_t sz)\n+{\n+\tint l = 0;\n+\tif (sz == 0)\n+\t\treturn 0;\n+\tfor (; sz; sz /= 2) {\n+\t\tl++;\n+\t}\n+\treturn l - 1;\n+}\n+\n static void test_reftable_stack_auto_compaction(void)\n {\n \tstruct reftable_write_options cfg = { 0 };\n@@ -1068,7 +1033,6 @@ static void test_reftable_stack_compaction_concurrent_clean(void)\n int stack_test_main(int argc, const char *argv[])\n {\n \tRUN_TEST(test_empty_add);\n-\tRUN_TEST(test_log2);\n \tRUN_TEST(test_names_equal);\n \tRUN_TEST(test_parse_names);\n \tRUN_TEST(test_read_file);\n@@ -1088,9 +1052,6 @@ int stack_test_main(int argc, const char *argv[])\n \tRUN_TEST(test_reftable_stack_update_index_check);\n \tRUN_TEST(test_reftable_stack_uptodate);\n \tRUN_TEST(test_reftable_stack_validate_refname);\n-\tRUN_TEST(test_sizes_to_segments);\n-\tRUN_TEST(test_sizes_to_segments_all_equal);\n-\tRUN_TEST(test_sizes_to_segments_empty);\n \tRUN_TEST(test_suggest_compaction_segment);\n \tRUN_TEST(test_suggest_compaction_segment_nothing);\n \treturn 0;\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex 434044078ed..b95626549e7 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -293,7 +293,7 @@ test_expect_success 'ref transaction: writes cause auto-compaction' '\n \ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n \ttest_commit -C repo --no-tag A &&\n-\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n \ttest_commit -C repo --no-tag B &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n@@ -308,12 +308,24 @@ test_expect_success 'ref transaction: environment variable disables auto-compact\n \tdo\n \t\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo update-ref branch-$i HEAD || return 1\n \tdone &&\n-\ttest_line_count = 23 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 22 repo/.git/reftable/tables.list &&\n \n \tgit -C repo update-ref foo HEAD &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n '\n \n+test_expect_success 'ref transaction: alternating table sizes are compacted' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tfor i in $(test_seq 20)\n+\tdo\n+\t\tgit -C repo branch -f foo &&\n+\t\tgit -C repo branch -d foo || return 1\n+\tdone &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list\n+'\n+\n check_fsync_events () {\n \tlocal trace=\"$1\" &&\n \tshift &&\n@@ -339,7 +351,7 @@ test_expect_success 'ref transaction: writes are synced' '\n \t\tgit -C repo -c core.fsync=reference \\\n \t\t-c core.fsyncMethod=fsync update-ref refs/heads/branch HEAD &&\n \tcheck_fsync_events trace2.txt <<-EOF\n-\t\"name\":\"hardware-flush\",\"count\":2\n+\t\"name\":\"hardware-flush\",\"count\":4\n \tEOF\n '\n \n@@ -361,8 +373,8 @@ test_expect_success 'pack-refs: compacts tables' '\n \n \ttest_commit -C repo A &&\n \tls -1 repo/.git/reftable >table-files &&\n-\ttest_line_count = 4 table-files &&\n-\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 3 table-files &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n \n \tgit -C repo pack-refs &&\n \tls -1 repo/.git/reftable >table-files &&\n@@ -394,7 +406,7 @@ do\n \t\t\tumask $umask &&\n \t\t\tgit init --shared=true repo &&\n \t\t\ttest_commit -C repo A &&\n-\t\t\ttest_line_count = 3 repo/.git/reftable/tables.list\n+\t\t\ttest_line_count = 2 repo/.git/reftable/tables.list\n \t\t) &&\n \t\tgit -C repo pack-refs &&\n \t\ttest_expect_perms \"-rw-rw-r--\" repo/.git/reftable/tables.list &&\n@@ -762,12 +774,13 @@ test_expect_success 'worktree: pack-refs in main repo packs main refs' '\n \ttest_when_finished \"rm -rf repo worktree\" &&\n \tgit init repo &&\n \ttest_commit -C repo A &&\n-\tgit -C repo worktree add ../worktree &&\n+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo worktree add ../worktree &&\n+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C worktree update-ref refs/worktree/per-worktree HEAD &&\n \n-\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n \tgit -C repo pack-refs &&\n-\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n '\n \n@@ -775,19 +788,21 @@ test_expect_success 'worktree: pack-refs in worktree packs worktree refs' '\n \ttest_when_finished \"rm -rf repo worktree\" &&\n \tgit init repo &&\n \ttest_commit -C repo A &&\n-\tgit -C repo worktree add ../worktree &&\n+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo worktree add ../worktree &&\n+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C worktree update-ref refs/worktree/per-worktree HEAD &&\n \n-\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n \tgit -C worktree pack-refs &&\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 4 repo/.git/reftable/tables.list\n+\ttest_line_count = 3 repo/.git/reftable/tables.list\n '\n \n test_expect_success 'worktree: creating shared ref updates main stack' '\n \ttest_when_finished \"rm -rf repo worktree\" &&\n \tgit init repo &&\n \ttest_commit -C repo A &&\n+\ttest_commit -C repo B &&\n \n \tgit -C repo worktree add ../worktree &&\n \tgit -C repo pack-refs &&\n@@ -795,7 +810,7 @@ test_expect_success 'worktree: creating shared ref updates main stack' '\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n-\tgit -C worktree update-ref refs/heads/shared HEAD &&\n+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C worktree update-ref refs/heads/shared HEAD &&\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 2 repo/.git/reftable/tables.list\n '\n-- \ngitgitgadget\n\n"},{"id":"491839","messageId":"xmqqzfugyjx9.fsf@gitster.g","threadId":"61055","inReplyTo":"2fdd8ea11331da13ec56d511fe4cadd47544419f.1711685809.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/3] reftable/stack: add env to disable autocompaction","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-03-29T18:25:38Z","receivedAt":"2024-03-29T18:25:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Justin Tobler via GitGitGadget\" writes:\n\n> From: Justin Tobler \n>\n> In future tests it will be neccesary to create repositories with a set\n> number of tables. To make this easier, introduce the\n> `GIT_TEST_REFTABLE_NO_AUTOCOMPACTION` environment variable that, when\n> set, disables autocompaction of reftables.\n\n\"when set\" -> \"when set to true\"?\n\n> Signed-off-by: Justin Tobler \n> ---\n> reftable/stack.c | 2 +-\n> reftable/system.h | 1 +\n> t/t0610-reftable-basics.sh | 15 +++++++++++++++\n> 3 files changed, 17 insertions(+), 1 deletion(-)\n>\n> diff --git a/reftable/stack.c b/reftable/stack.c\n> index 1ecf1b9751c..07262beaaf7 100644\n> --- a/reftable/stack.c\n> +++ b/reftable/stack.c\n> @@ -681,7 +681,7 @@ int reftable_addition_commit(struct reftable_addition *add)\n> \tif (err)\n> \t\tgoto done;\n> \n> -\tif (!add->stack->disable_auto_compact)\n> +\tif (!add->stack->disable_auto_compact && !git_env_bool(\"GIT_TEST_REFTABLE_NO_AUTOCOMPACTION\", 0))\n> \t\terr = reftable_stack_auto_compact(add->stack);\n> \n> done:\n> diff --git a/reftable/system.h b/reftable/system.h\n> index 5d8b6dede50..05b7c8554af 100644\n> --- a/reftable/system.h\n> +++ b/reftable/system.h\n> @@ -17,6 +17,7 @@ license that can be found in the LICENSE file or at\n> #include \"tempfile.h\"\n> #include \"hash-ll.h\" /* hash ID, sizes.*/\n> #include \"dir.h\" /* remove_dir_recursively, for tests.*/\n> +#include \"parse.h\"\n> \n> int hash_size(uint32_t id);\n> \n> diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\n> index 686781192eb..434044078ed 100755\n> --- a/t/t0610-reftable-basics.sh\n> +++ b/t/t0610-reftable-basics.sh\n> @@ -299,6 +299,21 @@ test_expect_success 'ref transaction: writes cause auto-compaction' '\n> \ttest_line_count = 1 repo/.git/reftable/tables.list\n> '\n> \n> +test_expect_success 'ref transaction: environment variable disables auto-compaction' '\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\n> +\tgit init repo &&\n> +\ttest_commit -C repo A &&\n> +\tfor i in $(test_seq 20)\n> +\tdo\n> +\t\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo update-ref branch-$i HEAD || return 1\n> +\tdone &&\n> +\ttest_line_count = 23 repo/.git/reftable/tables.list &&\n\nI am not sure if it is a sensible assumption that init + test_commit\n(which itself is opaque) will create exactly 3 tables forever, even\nif it may happen to be true right now. Shouldn't you be counting\nthe lines before entering the for loop and adding 20 to that number\nto set the expectation?\n\n> +\tgit -C repo update-ref foo HEAD &&\n> +\ttest_line_count = 1 repo/.git/reftable/tables.list\n> +'\n> +\n> check_fsync_events () {\n> \tlocal trace=\"$1\" &&\n> \tshift &&\n"},{"id":"491841","messageId":"xmqqsf08yjf2.fsf@gitster.g","threadId":"61055","inReplyTo":"9a33914c852a0487dbd90c83f53fa0e36414fda1.1711685809.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 3/3] reftable/stack: make segment end inclusive","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-03-29T18:36:33Z","receivedAt":"2024-03-29T18:36:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Justin Tobler via GitGitGadget\" writes:\n\n> From: Justin Tobler \n>\n> For a reftable segment, the start of the range is inclusive and the end\n> is exclusive. In practice we increment the end when creating the\n> compaction segment only to decrement the segment end when using it.\n>\n> Simplify by making the segment end inclusive. The corresponding test,\n> `test_suggest_compaction_segment()`, is updated to show that the segment\n> end is now inclusive.\n>\n> Signed-off-by: Justin Tobler \n> ---\n> reftable/stack.c | 4 ++--\n> reftable/stack_test.c | 2 +-\n> 2 files changed, 3 insertions(+), 3 deletions(-)\n\nI'd defer it to Patrick (and Han-Wen, if he wants to comment on it),\nbut isn't it a natural expectation shared among CS folks that it is\nthe most usual way to express a range to use inclusive lower-end and\nexclusive upper-end? \n\nAfter all, that is how an array works, i.e. msg[n] is NULL and\nbeyond the end where n == strlen(msg).\n\nSo, I dunno.\n\n> diff --git a/reftable/stack.c b/reftable/stack.c\n> index e7b9a1de5a4..0973c47dd92 100644\n> --- a/reftable/stack.c\n> +++ b/reftable/stack.c\n> @@ -1237,7 +1237,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n> \t */\n> \tfor (i = n - 1; i > 0; i--) {\n> \t\tif (sizes[i - 1] < sizes[i] * 2) {\n> -\t\t\tseg.end = i + 1;\n> +\t\t\tseg.end = i;\n> \t\t\tbytes = sizes[i];\n> \t\t\tbreak;\n> \t\t}\n\n\n\n> @@ -1291,7 +1291,7 @@ int reftable_stack_auto_compact(struct reftable_stack *st)\n> \t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n> \treftable_free(sizes);\n> \tif (segment_size(&seg) > 0)\n> -\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n> +\t\treturn stack_compact_range_stats(st, seg.start, seg.end,\n> \t\t\t\t\t\t NULL);\n> \n> \treturn 0;\n> diff --git a/reftable/stack_test.c b/reftable/stack_test.c\n> index 21541742fe5..4d7305623a0 100644\n> --- a/reftable/stack_test.c\n> +++ b/reftable/stack_test.c\n> @@ -723,7 +723,7 @@ static void test_suggest_compaction_segment(void)\n> \tstruct segment min =\n> \t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n> \tEXPECT(min.start == 1);\n> -\tEXPECT(min.end == 10);\n> +\tEXPECT(min.end == 9);\n> }\n> \n> static void test_suggest_compaction_segment_nothing(void)\n"},{"id":"491861","messageId":"xmqqttkovh1j.fsf@gitster.g","threadId":"61055","inReplyTo":"2fdd8ea11331da13ec56d511fe4cadd47544419f.1711685809.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/3] reftable/stack: add env to disable autocompaction","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-03-29T21:56:08Z","receivedAt":"2024-03-29T21:56:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Justin Tobler via GitGitGadget\" writes:\n\n> -\tif (!add->stack->disable_auto_compact)\n> +\tif (!add->stack->disable_auto_compact && !git_env_bool(\"GIT_TEST_REFTABLE_NO_AUTOCOMPACTION\", 0))\n\nFold the line after \" &&\", i.e.\n\n\tif (!add->stack->disable_auto_compact &&\n\t !git_env_bool(\"GIT_TEST_REFTABLE_NO_AUTOCOMPACTION\", 0))\n\n> diff --git a/reftable/system.h b/reftable/system.h\n> index 5d8b6dede50..05b7c8554af 100644\n> --- a/reftable/system.h\n> +++ b/reftable/system.h\n> @@ -17,6 +17,7 @@ license that can be found in the LICENSE file or at\n> #include \"tempfile.h\"\n> #include \"hash-ll.h\" /* hash ID, sizes.*/\n> #include \"dir.h\" /* remove_dir_recursively, for tests.*/\n> +#include \"parse.h\"\n> \n> int hash_size(uint32_t id);\n> \n> diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\n> index 686781192eb..434044078ed 100755\n> --- a/t/t0610-reftable-basics.sh\n> +++ b/t/t0610-reftable-basics.sh\n> @@ -299,6 +299,21 @@ test_expect_success 'ref transaction: writes cause auto-compaction' '\n> \ttest_line_count = 1 repo/.git/reftable/tables.list\n> '\n> \n> +test_expect_success 'ref transaction: environment variable disables auto-compaction' '\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\n> +\tgit init repo &&\n> +\ttest_commit -C repo A &&\n> +\tfor i in $(test_seq 20)\n> +\tdo\n> +\t\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo update-ref branch-$i HEAD || return 1\n\nFold the line before \"git\", i.e.\n\n\t\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true \\\n\t\tgit -C repo update-ref branch-$i HEAD || return 1\n\n> +\tdone &&\n> +\ttest_line_count = 23 repo/.git/reftable/tables.list &&\n> +\n> +\tgit -C repo update-ref foo HEAD &&\n> +\ttest_line_count = 1 repo/.git/reftable/tables.list\n> +'\n> +\n> check_fsync_events () {\n> \tlocal trace=\"$1\" &&\n> \tshift &&\n"},{"id":"492028","messageId":"ZguyfEN4dLGU_YkS@tanuki","threadId":"61055","inReplyTo":"xmqqsf08yjf2.fsf@gitster.g","subject":"Re: [PATCH v3 3/3] reftable/stack: make segment end inclusive","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-02T07:23:40Z","receivedAt":"2024-04-02T07:23:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Mar 29, 2024 at 11:36:33AM -0700, Junio C Hamano wrote:\n> \"Justin Tobler via GitGitGadget\" writes:\n> \n> > From: Justin Tobler \n> >\n> > For a reftable segment, the start of the range is inclusive and the end\n> > is exclusive. In practice we increment the end when creating the\n> > compaction segment only to decrement the segment end when using it.\n> >\n> > Simplify by making the segment end inclusive. The corresponding test,\n> > `test_suggest_compaction_segment()`, is updated to show that the segment\n> > end is now inclusive.\n> >\n> > Signed-off-by: Justin Tobler \n> > ---\n> > reftable/stack.c | 4 ++--\n> > reftable/stack_test.c | 2 +-\n> > 2 files changed, 3 insertions(+), 3 deletions(-)\n> \n> I'd defer it to Patrick (and Han-Wen, if he wants to comment on it),\n> but isn't it a natural expectation shared among CS folks that it is\n> the most usual way to express a range to use inclusive lower-end and\n> exclusive upper-end? \n> \n> After all, that is how an array works, i.e. msg[n] is NULL and\n> beyond the end where n == strlen(msg).\n> \n> So, I dunno.\n\nI don't really have a strong opinion here, to be honest. I think the\nprevious way to handle this was fine, the new way is okay, too. Which\nmay indicate that we can just drop this patch to avoid needless churn\nunless somebody feels strongly about this.\n\nPatrick\n\n> > diff --git a/reftable/stack.c b/reftable/stack.c\n> > index e7b9a1de5a4..0973c47dd92 100644\n> > --- a/reftable/stack.c\n> > +++ b/reftable/stack.c\n> > @@ -1237,7 +1237,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n> > \t */\n> > \tfor (i = n - 1; i > 0; i--) {\n> > \t\tif (sizes[i - 1] < sizes[i] * 2) {\n> > -\t\t\tseg.end = i + 1;\n> > +\t\t\tseg.end = i;\n> > \t\t\tbytes = sizes[i];\n> > \t\t\tbreak;\n> > \t\t}\n> \n> \n> \n> > @@ -1291,7 +1291,7 @@ int reftable_stack_auto_compact(struct reftable_stack *st)\n> > \t\tsuggest_compaction_segment(sizes, st->merged->stack_len);\n> > \treftable_free(sizes);\n> > \tif (segment_size(&seg) > 0)\n> > -\t\treturn stack_compact_range_stats(st, seg.start, seg.end - 1,\n> > +\t\treturn stack_compact_range_stats(st, seg.start, seg.end,\n> > \t\t\t\t\t\t NULL);\n> > \n> > \treturn 0;\n> > diff --git a/reftable/stack_test.c b/reftable/stack_test.c\n> > index 21541742fe5..4d7305623a0 100644\n> > --- a/reftable/stack_test.c\n> > +++ b/reftable/stack_test.c\n> > @@ -723,7 +723,7 @@ static void test_suggest_compaction_segment(void)\n> > \tstruct segment min =\n> > \t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n> > \tEXPECT(min.start == 1);\n> > -\tEXPECT(min.end == 10);\n> > +\tEXPECT(min.end == 9);\n> > }\n> > \n> > static void test_suggest_compaction_segment_nothing(void)\n"},{"id":"492029","messageId":"ZguygYtc7plWTIA2@tanuki","threadId":"61055","inReplyTo":"2fdd8ea11331da13ec56d511fe4cadd47544419f.1711685809.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/3] reftable/stack: add env to disable autocompaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-02T07:23:45Z","receivedAt":"2024-04-02T07:23:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Mar 29, 2024 at 04:16:47AM +0000, Justin Tobler via GitGitGadget wrote:\n> From: Justin Tobler \n> \n> In future tests it will be neccesary to create repositories with a set\n> number of tables. To make this easier, introduce the\n> `GIT_TEST_REFTABLE_NO_AUTOCOMPACTION` environment variable that, when\n> set, disables autocompaction of reftables.\n> \n> Signed-off-by: Justin Tobler \n> ---\n> reftable/stack.c | 2 +-\n> reftable/system.h | 1 +\n> t/t0610-reftable-basics.sh | 15 +++++++++++++++\n> 3 files changed, 17 insertions(+), 1 deletion(-)\n> \n> diff --git a/reftable/stack.c b/reftable/stack.c\n> index 1ecf1b9751c..07262beaaf7 100644\n> --- a/reftable/stack.c\n> +++ b/reftable/stack.c\n> @@ -681,7 +681,7 @@ int reftable_addition_commit(struct reftable_addition *add)\n> \tif (err)\n> \t\tgoto done;\n> \n> -\tif (!add->stack->disable_auto_compact)\n> +\tif (!add->stack->disable_auto_compact && !git_env_bool(\"GIT_TEST_REFTABLE_NO_AUTOCOMPACTION\", 0))\n> \t\terr = reftable_stack_auto_compact(add->stack);\n\nThe double-negation in `GIT_TEST_REFTABLE_NO_AUTOCOMPACTION=false` may be\nsomewhat hard to parse. Should we rename this to\n`GIT_TEST_REFTABLE_AUTO_COMPACTION` with a default value of `1`?\n\nPatrick\n\n> done:\n> diff --git a/reftable/system.h b/reftable/system.h\n> index 5d8b6dede50..05b7c8554af 100644\n> --- a/reftable/system.h\n> +++ b/reftable/system.h\n> @@ -17,6 +17,7 @@ license that can be found in the LICENSE file or at\n> #include \"tempfile.h\"\n> #include \"hash-ll.h\" /* hash ID, sizes.*/\n> #include \"dir.h\" /* remove_dir_recursively, for tests.*/\n> +#include \"parse.h\"\n> \n> int hash_size(uint32_t id);\n> \n> diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\n> index 686781192eb..434044078ed 100755\n> --- a/t/t0610-reftable-basics.sh\n> +++ b/t/t0610-reftable-basics.sh\n> @@ -299,6 +299,21 @@ test_expect_success 'ref transaction: writes cause auto-compaction' '\n> \ttest_line_count = 1 repo/.git/reftable/tables.list\n> '\n> \n> +test_expect_success 'ref transaction: environment variable disables auto-compaction' '\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\n> +\tgit init repo &&\n> +\ttest_commit -C repo A &&\n> +\tfor i in $(test_seq 20)\n> +\tdo\n> +\t\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo update-ref branch-$i HEAD || return 1\n> +\tdone &&\n> +\ttest_line_count = 23 repo/.git/reftable/tables.list &&\n> +\n> +\tgit -C repo update-ref foo HEAD &&\n> +\ttest_line_count = 1 repo/.git/reftable/tables.list\n> +'\n> +\n> check_fsync_events () {\n> \tlocal trace=\"$1\" &&\n> \tshift &&\n> -- \n> gitgitgadget\n> \n"},{"id":"492030","messageId":"Zguyhhyot0ZphACv@tanuki","threadId":"61055","inReplyTo":"7e62c2286ae9c63368f1508f065b33422ca24bc8.1711685809.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 2/3] reftable/stack: use geometric table compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-02T07:23:50Z","receivedAt":"2024-04-02T07:23:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Mar 29, 2024 at 04:16:48AM +0000, Justin Tobler via GitGitGadget wrote:\n> From: Justin Tobler \n> \n> To reduce the number of on-disk reftables, compaction is performed.\n> Contiguous tables with the same binary log value of size are grouped\n> into segments. The segment that has both the lowest binary log value and\n> contains more than one table is set as the starting point when\n> identifying the compaction segment.\n> \n> Since segments containing a single table are not initially considered\n> for compaction, if the table appended to the list does not match the\n> previous table log value, no compaction occurs for the new table. It is\n> therefore possible for unbounded growth of the table list. This can be\n> demonstrated by repeating the following sequence:\n> \n> git branch -f foo\n> git branch -d foo\n> \n> Each operation results in a new table being written with no compaction\n> occurring until a separate operation produces a table matching the\n> previous table log value.\n> \n> Instead, to avoid unbounded growth of the table list, the compaction\n> strategy is updated to ensure tables follow a geometric sequence after\n> each operation by individually evaluating each table in reverse index\n> order. This strategy results in a much simpler and more robust algorithm\n> compared to the previous one while also maintaining a minimal ordered\n> set of tables on-disk.\n> \n> When creating 10 thousand references, the new strategy has no\n> performance impact:\n> \n> Benchmark 1: update-ref: create refs sequentially (revision = HEAD~)\n> Time (mean ± σ): 26.516 s ± 0.047 s [User: 17.864 s, System: 8.491 s]\n> Range (min … max): 26.447 s … 26.569 s 10 runs\n> \n> Benchmark 2: update-ref: create refs sequentially (revision = HEAD)\n> Time (mean ± σ): 26.417 s ± 0.028 s [User: 17.738 s, System: 8.500 s]\n> Range (min … max): 26.366 s … 26.444 s 10 runs\n> \n> Summary\n> update-ref: create refs sequentially (revision = HEAD) ran\n> 1.00 ± 0.00 times faster than update-ref: create refs sequentially (revision = HEAD~)\n> \n> Some tests in `t0610-reftable-basics.sh` assert the on-disk state of\n> tables and are therefore updated to specify the correct new table count.\n> Since compaction is more aggressive in ensuring tables maintain a\n> geometric sequence, the expected table count is reduced in these tests.\n> In `reftable/stack_test.c` tests related to `sizes_to_segments()` are\n> removed because the function is no longer needed. Also, the\n> `test_suggest_compaction_segment()` test is updated to better showcase\n> and reflect the new geometric compaction behavior.\n> \n> Signed-off-by: Justin Tobler \n> ---\n> reftable/stack.c | 120 ++++++++++++++++++-------------------\n> reftable/stack.h | 3 -\n> reftable/stack_test.c | 67 +++++----------------\n> t/t0610-reftable-basics.sh | 45 +++++++++-----\n> 4 files changed, 103 insertions(+), 132 deletions(-)\n> \n> diff --git a/reftable/stack.c b/reftable/stack.c\n> index 07262beaaf7..e7b9a1de5a4 100644\n> --- a/reftable/stack.c\n> +++ b/reftable/stack.c\n> @@ -1202,75 +1202,73 @@ static int segment_size(struct segment *s)\n> \treturn s->end - s->start;\n> }\n> \n> -int fastlog2(uint64_t sz)\n> -{\n> -\tint l = 0;\n> -\tif (sz == 0)\n> -\t\treturn 0;\n> -\tfor (; sz; sz /= 2) {\n> -\t\tl++;\n> -\t}\n> -\treturn l - 1;\n> -}\n> -\n> -struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n)\n> -{\n> -\tstruct segment *segs = reftable_calloc(n, sizeof(*segs));\n> -\tstruct segment cur = { 0 };\n> -\tsize_t next = 0, i;\n> -\n> -\tif (n == 0) {\n> -\t\t*seglen = 0;\n> -\t\treturn segs;\n> -\t}\n> -\tfor (i = 0; i < n; i++) {\n> -\t\tint log = fastlog2(sizes[i]);\n> -\t\tif (cur.log != log && cur.bytes > 0) {\n> -\t\t\tstruct segment fresh = {\n> -\t\t\t\t.start = i,\n> -\t\t\t};\n> -\n> -\t\t\tsegs[next++] = cur;\n> -\t\t\tcur = fresh;\n> -\t\t}\n> -\n> -\t\tcur.log = log;\n> -\t\tcur.end = i + 1;\n> -\t\tcur.bytes += sizes[i];\n> -\t}\n> -\tsegs[next++] = cur;\n> -\t*seglen = next;\n> -\treturn segs;\n> -}\n> -\n> struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n> {\n> -\tstruct segment min_seg = {\n> -\t\t.log = 64,\n> -\t};\n> -\tstruct segment *segs;\n> -\tsize_t seglen = 0, i;\n> -\n> -\tsegs = sizes_to_segments(&seglen, sizes, n);\n> -\tfor (i = 0; i < seglen; i++) {\n> -\t\tif (segment_size(&segs[i]) == 1)\n> -\t\t\tcontinue;\n> +\tstruct segment seg = { 0 };\n> +\tuint64_t bytes;\n> +\tsize_t i;\n> \n> -\t\tif (segs[i].log < min_seg.log)\n> -\t\t\tmin_seg = segs[i];\n> -\t}\n> +\t/*\n> +\t * If there are no tables or only a single one then we don't have to\n> +\t * compact anything. The sequence is geometric by definition already.\n> +\t */\n> +\tif (n <= 1)\n> +\t\treturn seg;\n> \n> -\twhile (min_seg.start > 0) {\n> -\t\tsize_t prev = min_seg.start - 1;\n> -\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev]))\n> +\t/*\n> +\t * Find the ending table of the compaction segment needed to restore the\n> +\t * geometric sequence.\n> +\t *\n> +\t * To do so, we iterate backwards starting from the most recent table\n> +\t * until a valid segment end is found. If the preceding table is smaller\n> +\t * than the current table multiplied by the geometric factor (2), the\n> +\t * current table is set as the compaction segment end.\n> +\t *\n> +\t * Tables after the ending point are not added to the byte count because\n> +\t * they are already valid members of the geometric sequence. Due to the\n> +\t * properties of a geometric sequence, it is not possible for the sum of\n> +\t * these tables to exceed the value of the ending point table.\n> +\t *\n> +\t * Example table size sequence requiring no compaction:\n> +\t * \t64, 32, 16, 8, 4, 2, 1\n> +\t *\n> +\t * Example compaction segment end set to table with size 3:\n> +\t * \t64, 32, 16, 8, 4, 3, 1\n> +\t */\n> +\tfor (i = n - 1; i > 0; i--) {\n> +\t\tif (sizes[i - 1] < sizes[i] * 2) {\n> +\t\t\tseg.end = i + 1;\n> +\t\t\tbytes = sizes[i];\n> \t\t\tbreak;\n> +\t\t}\n> +\t}\n> \n> -\t\tmin_seg.start = prev;\n> -\t\tmin_seg.bytes += sizes[prev];\n> +\t/*\n> +\t * Find the starting table of the compaction segment by iterating\n> +\t * through the remaining tables and keeping track of the accumulated\n> +\t * size of all tables seen from the segment end table. The previous\n> +\t * table is compared to the accumulated size because the tables from the\n> +\t * segment end are merged backwards recursively.\n> +\t *\n> +\t * Note that we keep iterating even after we have found the first\n> +\t * starting point. This is because there may be tables in the stack\n> +\t * preceding that first starting point which violate the geometric\n> +\t * sequence.\n> +\t *\n> +\t * Example compaction segment start set to table with size 32:\n> +\t * \t128, 32, 16, 8, 4, 3, 1\n> +\t */\n> +\tfor (; i > 0; i--) {\n> +\t\tuint64_t curr = bytes;\n> +\t\tbytes += sizes[i - 1];\n> +\n> +\t\tif (sizes[i - 1] < curr * 2) {\n> +\t\t\tseg.start = i - 1;\n> +\t\t\tseg.bytes = bytes;\n> +\t\t}\n> \t}\n> \n> -\treftable_free(segs);\n> -\treturn min_seg;\n> +\treturn seg;\n> }\n> \n> static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n> diff --git a/reftable/stack.h b/reftable/stack.h\n> index d919455669e..656f896cc28 100644\n> --- a/reftable/stack.h\n> +++ b/reftable/stack.h\n> @@ -33,12 +33,9 @@ int read_lines(const char *filename, char ***lines);\n> \n> struct segment {\n> \tsize_t start, end;\n> -\tint log;\n> \tuint64_t bytes;\n> };\n> \n> -int fastlog2(uint64_t sz);\n> -struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n);\n> struct segment suggest_compaction_segment(uint64_t *sizes, size_t n);\n> \n> #endif\n> diff --git a/reftable/stack_test.c b/reftable/stack_test.c\n> index 7336757cf53..21541742fe5 100644\n> --- a/reftable/stack_test.c\n> +++ b/reftable/stack_test.c\n> @@ -717,59 +717,13 @@ static void test_reftable_stack_hash_id(void)\n> \tclear_dir(dir);\n> }\n> \n> -static void test_log2(void)\n> -{\n> -\tEXPECT(1 == fastlog2(3));\n> -\tEXPECT(2 == fastlog2(4));\n> -\tEXPECT(2 == fastlog2(5));\n> -}\n> -\n> -static void test_sizes_to_segments(void)\n> -{\n> -\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n> -\t/* .................0 1 2 3 4 5 */\n> -\n> -\tsize_t seglen = 0;\n> -\tstruct segment *segs =\n> -\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n> -\tEXPECT(segs[2].log == 3);\n> -\tEXPECT(segs[2].start == 5);\n> -\tEXPECT(segs[2].end == 6);\n> -\n> -\tEXPECT(segs[1].log == 2);\n> -\tEXPECT(segs[1].start == 2);\n> -\tEXPECT(segs[1].end == 5);\n> -\treftable_free(segs);\n> -}\n> -\n> -static void test_sizes_to_segments_empty(void)\n> -{\n> -\tsize_t seglen = 0;\n> -\tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n> -\tEXPECT(seglen == 0);\n> -\treftable_free(segs);\n> -}\n> -\n> -static void test_sizes_to_segments_all_equal(void)\n> -{\n> -\tuint64_t sizes[] = { 5, 5 };\n> -\tsize_t seglen = 0;\n> -\tstruct segment *segs =\n> -\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n> -\tEXPECT(seglen == 1);\n> -\tEXPECT(segs[0].start == 0);\n> -\tEXPECT(segs[0].end == 2);\n> -\treftable_free(segs);\n> -}\n> -\n> static void test_suggest_compaction_segment(void)\n> {\n> -\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n> -\t/* .................0 1 2 3 4 5 6 */\n> +\tuint64_t sizes[] = { 512, 64, 17, 16, 9, 9, 9, 16, 2, 16 };\n> \tstruct segment min =\n> \t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n> -\tEXPECT(min.start == 2);\n> -\tEXPECT(min.end == 7);\n> +\tEXPECT(min.start == 1);\n> +\tEXPECT(min.end == 10);\n> }\n> \n> static void test_suggest_compaction_segment_nothing(void)\n> @@ -880,6 +834,17 @@ static void test_empty_add(void)\n> \treftable_stack_destroy(st2);\n> }\n> \n> +static int fastlog2(uint64_t sz)\n> +{\n> +\tint l = 0;\n> +\tif (sz == 0)\n> +\t\treturn 0;\n> +\tfor (; sz; sz /= 2) {\n> +\t\tl++;\n> +\t}\n\nNit: we could drop the curly braces while at it.\n\n> +\treturn l - 1;\n> +}\n> +\n> static void test_reftable_stack_auto_compaction(void)\n> {\n> \tstruct reftable_write_options cfg = { 0 };\n> @@ -1068,7 +1033,6 @@ static void test_reftable_stack_compaction_concurrent_clean(void)\n> int stack_test_main(int argc, const char *argv[])\n> {\n> \tRUN_TEST(test_empty_add);\n> -\tRUN_TEST(test_log2);\n> \tRUN_TEST(test_names_equal);\n> \tRUN_TEST(test_parse_names);\n> \tRUN_TEST(test_read_file);\n> @@ -1088,9 +1052,6 @@ int stack_test_main(int argc, const char *argv[])\n> \tRUN_TEST(test_reftable_stack_update_index_check);\n> \tRUN_TEST(test_reftable_stack_uptodate);\n> \tRUN_TEST(test_reftable_stack_validate_refname);\n> -\tRUN_TEST(test_sizes_to_segments);\n> -\tRUN_TEST(test_sizes_to_segments_all_equal);\n> -\tRUN_TEST(test_sizes_to_segments_empty);\n> \tRUN_TEST(test_suggest_compaction_segment);\n> \tRUN_TEST(test_suggest_compaction_segment_nothing);\n> \treturn 0;\n> diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\n> index 434044078ed..b95626549e7 100755\n> --- a/t/t0610-reftable-basics.sh\n> +++ b/t/t0610-reftable-basics.sh\n> @@ -293,7 +293,7 @@ test_expect_success 'ref transaction: writes cause auto-compaction' '\n> \ttest_line_count = 1 repo/.git/reftable/tables.list &&\n> \n> \ttest_commit -C repo --no-tag A &&\n> -\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n> +\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n> \n> \ttest_commit -C repo --no-tag B &&\n> \ttest_line_count = 1 repo/.git/reftable/tables.list\n> @@ -308,12 +308,24 @@ test_expect_success 'ref transaction: environment variable disables auto-compact\n> \tdo\n> \t\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo update-ref branch-$i HEAD || return 1\n> \tdone &&\n> -\ttest_line_count = 23 repo/.git/reftable/tables.list &&\n> +\ttest_line_count = 22 repo/.git/reftable/tables.list &&\n> \n> \tgit -C repo update-ref foo HEAD &&\n> \ttest_line_count = 1 repo/.git/reftable/tables.list\n> '\n> \n> +test_expect_success 'ref transaction: alternating table sizes are compacted' '\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\tgit init repo &&\n> +\ttest_commit -C repo A &&\n> +\tfor i in $(test_seq 20)\n\nNit: we could reduce the number of iterations here so that we don't have\nto spawn 40 Git commands. Using something like 10 or even 5 iterations\nshould be sufficient to demonstrate the problem, no?\n\nPatrick\n"},{"id":"492073","messageId":"xmqqfrw365ls.fsf@gitster.g","threadId":"61055","inReplyTo":"ZguygYtc7plWTIA2@tanuki","subject":"Re: [PATCH v3 1/3] reftable/stack: add env to disable autocompaction","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-02T17:23:43Z","receivedAt":"2024-04-02T17:23:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> -\tif (!add->stack->disable_auto_compact)\n>> +\tif (!add->stack->disable_auto_compact && !git_env_bool(\"GIT_TEST_REFTABLE_NO_AUTOCOMPACTION\", 0))\n>> \t\terr = reftable_stack_auto_compact(add->stack);\n>\n> The double-negation in `GIT_TEST_REFTABLE_NO_AUTOCOMPACTION=false` may be\n> somewhat hard to parse. Should we rename this to\n> `GIT_TEST_REFTABLE_AUTO_COMPACTION` with a default value of `1`?\n\nSounds like a nice improvement. Also the overlong line should be\nfolded.\n\nThanks.\n"},{"id":"492124","messageId":"pull.1683.v4.git.1712103636.gitgitgadget@gmail.com","threadId":"61055","inReplyTo":"pull.1683.v3.git.1711685809.gitgitgadget@gmail.com","subject":"[PATCH v4 0/2] reftable/stack: use geometric table compaction","fromName":"Justin Tobler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-04-03T00:20:34Z","receivedAt":"2024-04-03T00:20:40Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Hello again,\n\nThis is the fourth version my patch series that refactors the reftable\ncompaction strategy to instead follow a geometric sequence. Changes compared\nto v3:\n\n * Changed env name from GIT_TEST_REFTABLE_NO_AUTOCOMPACTION to\n GIT_TEST_REFTABLE_AUTOCOMPACTION and set the default to false. This\n should hopefully be a bit more intuitive since it avoids the double\n negative.\n * Updated the corresponding env var test in t0610-reftable-basics.sh to\n assert on the number of tables added and be overall less fragile.\n * Folded lines that were too long.\n * Updated some comments in stack.c to more accurately explain that table\n segment end is exclusive.\n * Dropped reftable/stack: make segment end inclusive commit to keep segment\n end exclusive and better follow expectations.\n\nThanks for taking a look!\n\n-Justin\n\nJustin Tobler (2):\n reftable/stack: add env to disable autocompaction\n reftable/stack: use geometric table compaction\n\n reftable/stack.c | 126 +++++++++++++++++++------------------\n reftable/stack.h | 3 -\n reftable/stack_test.c | 66 ++++---------------\n reftable/system.h | 1 +\n t/t0610-reftable-basics.sh | 65 +++++++++++++++----\n 5 files changed, 132 insertions(+), 129 deletions(-)\n\n\nbase-commit: c75fd8d8150afdf836b63a8e0534d9b9e3e111ba\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1683%2Fjltobler%2Fjt%2Freftable-geometric-compaction-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1683/jltobler/jt/reftable-geometric-compaction-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1683\n\nRange-diff vs v3:\n\n 1: 2fdd8ea1133 ! 1: 2a0421e5f20 reftable/stack: add env to disable autocompaction\n @@ Commit message\n \n In future tests it will be neccesary to create repositories with a set\n number of tables. To make this easier, introduce the\n - `GIT_TEST_REFTABLE_NO_AUTOCOMPACTION` environment variable that, when\n - set, disables autocompaction of reftables.\n + `GIT_TEST_REFTABLE_AUTOCOMPACTION` environment variable that, when set\n + to false, disables autocompaction of reftables.\n \n Signed-off-by: Justin Tobler \n \n @@ reftable/stack.c: int reftable_addition_commit(struct reftable_addition *add)\n \t\tgoto done;\n \n -\tif (!add->stack->disable_auto_compact)\n -+\tif (!add->stack->disable_auto_compact && !git_env_bool(\"GIT_TEST_REFTABLE_NO_AUTOCOMPACTION\", 0))\n ++\tif (!add->stack->disable_auto_compact &&\n ++\t git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1))\n \t\terr = reftable_stack_auto_compact(add->stack);\n \n done:\n @@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: writes cause a\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n '\n \n -+test_expect_success 'ref transaction: environment variable disables auto-compaction' '\n ++test_expect_success 'ref transaction: env var disables compaction' '\n +\ttest_when_finished \"rm -rf repo\" &&\n +\n +\tgit init repo &&\n +\ttest_commit -C repo A &&\n -+\tfor i in $(test_seq 20)\n ++\n ++\tstart=$(wc -l 0; i--) {\n @@ reftable/stack_test.c: static void test_empty_add(void)\n +\tint l = 0;\n +\tif (sz == 0)\n +\t\treturn 0;\n -+\tfor (; sz; sz /= 2) {\n ++\tfor (; sz; sz /= 2)\n +\t\tl++;\n -+\t}\n +\treturn l - 1;\n +}\n +\n @@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: writes cause a\n \n \ttest_commit -C repo --no-tag B &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n -@@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: environment variable disables auto-compact\n - \tdo\n - \t\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo update-ref branch-$i HEAD || return 1\n - \tdone &&\n --\ttest_line_count = 23 repo/.git/reftable/tables.list &&\n -+\ttest_line_count = 22 repo/.git/reftable/tables.list &&\n - \n - \tgit -C repo update-ref foo HEAD &&\n - \ttest_line_count = 1 repo/.git/reftable/tables.list\n +@@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: env var disables compaction' '\n + \ttest_line_count -lt $expected repo/.git/reftable/tables.list\n '\n \n +test_expect_success 'ref transaction: alternating table sizes are compacted' '\n +\ttest_when_finished \"rm -rf repo\" &&\n ++\n +\tgit init repo &&\n +\ttest_commit -C repo A &&\n -+\tfor i in $(test_seq 20)\n ++\tfor i in $(test_seq 5)\n +\tdo\n +\t\tgit -C repo branch -f foo &&\n +\t\tgit -C repo branch -d foo || return 1\n @@ t/t0610-reftable-basics.sh: test_expect_success 'worktree: pack-refs in main rep\n \ttest_when_finished \"rm -rf repo worktree\" &&\n \tgit init repo &&\n \ttest_commit -C repo A &&\n --\tgit -C repo worktree add ../worktree &&\n -+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo worktree add ../worktree &&\n -+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C worktree update-ref refs/worktree/per-worktree HEAD &&\n ++\n ++\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n + \tgit -C repo worktree add ../worktree &&\n ++\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n ++\tgit -C worktree update-ref refs/worktree/per-worktree HEAD &&\n \n -\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n -\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n @@ t/t0610-reftable-basics.sh: test_expect_success 'worktree: pack-refs in worktree\n \ttest_when_finished \"rm -rf repo worktree\" &&\n \tgit init repo &&\n \ttest_commit -C repo A &&\n --\tgit -C repo worktree add ../worktree &&\n -+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo worktree add ../worktree &&\n -+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C worktree update-ref refs/worktree/per-worktree HEAD &&\n ++\n ++\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n + \tgit -C repo worktree add ../worktree &&\n ++\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n ++\tgit -C worktree update-ref refs/worktree/per-worktree HEAD &&\n \n -\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n -\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n @@ t/t0610-reftable-basics.sh: test_expect_success 'worktree: pack-refs in worktree\n '\n \n test_expect_success 'worktree: creating shared ref updates main stack' '\n - \ttest_when_finished \"rm -rf repo worktree\" &&\n - \tgit init repo &&\n - \ttest_commit -C repo A &&\n -+\ttest_commit -C repo B &&\n - \n - \tgit -C repo worktree add ../worktree &&\n - \tgit -C repo pack-refs &&\n @@ t/t0610-reftable-basics.sh: test_expect_success 'worktree: creating shared ref updates main stack' '\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n --\tgit -C worktree update-ref refs/heads/shared HEAD &&\n -+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C worktree update-ref refs/heads/shared HEAD &&\n ++\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n + \tgit -C worktree update-ref refs/heads/shared HEAD &&\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 2 repo/.git/reftable/tables.list\n - '\n 3: 9a33914c852 < -: ----------- reftable/stack: make segment end inclusive\n\n-- \ngitgitgadget\n"},{"id":"492123","messageId":"2a0421e5f20b7d43619be20e8f500fedd55f2829.1712103636.git.gitgitgadget@gmail.com","threadId":"61055","inReplyTo":"pull.1683.v4.git.1712103636.gitgitgadget@gmail.com","subject":"[PATCH v4 1/2] reftable/stack: add env to disable autocompaction","fromName":"Justin Tobler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-04-03T00:20:35Z","receivedAt":"2024-04-03T00:20:41Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"From: Justin Tobler \n\nIn future tests it will be neccesary to create repositories with a set\nnumber of tables. To make this easier, introduce the\n`GIT_TEST_REFTABLE_AUTOCOMPACTION` environment variable that, when set\nto false, disables autocompaction of reftables.\n\nSigned-off-by: Justin Tobler \n---\n reftable/stack.c | 3 ++-\n reftable/system.h | 1 +\n t/t0610-reftable-basics.sh | 21 +++++++++++++++++++++\n 3 files changed, 24 insertions(+), 1 deletion(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 1ecf1b9751c..4c373fb0ee2 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -681,7 +681,8 @@ int reftable_addition_commit(struct reftable_addition *add)\n \tif (err)\n \t\tgoto done;\n \n-\tif (!add->stack->disable_auto_compact)\n+\tif (!add->stack->disable_auto_compact &&\n+\t git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1))\n \t\terr = reftable_stack_auto_compact(add->stack);\n \n done:\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 5d8b6dede50..05b7c8554af 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -17,6 +17,7 @@ license that can be found in the LICENSE file or at\n #include \"tempfile.h\"\n #include \"hash-ll.h\" /* hash ID, sizes.*/\n #include \"dir.h\" /* remove_dir_recursively, for tests.*/\n+#include \"parse.h\"\n \n int hash_size(uint32_t id);\n \ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex 686781192eb..444f7497907 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -299,6 +299,27 @@ test_expect_success 'ref transaction: writes cause auto-compaction' '\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n '\n \n+test_expect_success 'ref transaction: env var disables compaction' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tstart=$(wc -l \n\nTo reduce the number of on-disk reftables, compaction is performed.\nContiguous tables with the same binary log value of size are grouped\ninto segments. The segment that has both the lowest binary log value and\ncontains more than one table is set as the starting point when\nidentifying the compaction segment.\n\nSince segments containing a single table are not initially considered\nfor compaction, if the table appended to the list does not match the\nprevious table log value, no compaction occurs for the new table. It is\ntherefore possible for unbounded growth of the table list. This can be\ndemonstrated by repeating the following sequence:\n\ngit branch -f foo\ngit branch -d foo\n\nEach operation results in a new table being written with no compaction\noccurring until a separate operation produces a table matching the\nprevious table log value.\n\nInstead, to avoid unbounded growth of the table list, the compaction\nstrategy is updated to ensure tables follow a geometric sequence after\neach operation by individually evaluating each table in reverse index\norder. This strategy results in a much simpler and more robust algorithm\ncompared to the previous one while also maintaining a minimal ordered\nset of tables on-disk.\n\nWhen creating 10 thousand references, the new strategy has no\nperformance impact:\n\nBenchmark 1: update-ref: create refs sequentially (revision = HEAD~)\n Time (mean ± σ): 26.516 s ± 0.047 s [User: 17.864 s, System: 8.491 s]\n Range (min … max): 26.447 s … 26.569 s 10 runs\n\nBenchmark 2: update-ref: create refs sequentially (revision = HEAD)\n Time (mean ± σ): 26.417 s ± 0.028 s [User: 17.738 s, System: 8.500 s]\n Range (min … max): 26.366 s … 26.444 s 10 runs\n\nSummary\n update-ref: create refs sequentially (revision = HEAD) ran\n 1.00 ± 0.00 times faster than update-ref: create refs sequentially (revision = HEAD~)\n\nSome tests in `t0610-reftable-basics.sh` assert the on-disk state of\ntables and are therefore updated to specify the correct new table count.\nSince compaction is more aggressive in ensuring tables maintain a\ngeometric sequence, the expected table count is reduced in these tests.\nIn `reftable/stack_test.c` tests related to `sizes_to_segments()` are\nremoved because the function is no longer needed. Also, the\n`test_suggest_compaction_segment()` test is updated to better showcase\nand reflect the new geometric compaction behavior.\n\nSigned-off-by: Justin Tobler \n---\n reftable/stack.c | 123 +++++++++++++++++++------------------\n reftable/stack.h | 3 -\n reftable/stack_test.c | 66 ++++----------------\n t/t0610-reftable-basics.sh | 44 +++++++++----\n 4 files changed, 108 insertions(+), 128 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 4c373fb0ee2..5f087cbd4bc 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1203,75 +1203,76 @@ static int segment_size(struct segment *s)\n \treturn s->end - s->start;\n }\n \n-int fastlog2(uint64_t sz)\n-{\n-\tint l = 0;\n-\tif (sz == 0)\n-\t\treturn 0;\n-\tfor (; sz; sz /= 2) {\n-\t\tl++;\n-\t}\n-\treturn l - 1;\n-}\n-\n-struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n)\n-{\n-\tstruct segment *segs = reftable_calloc(n, sizeof(*segs));\n-\tstruct segment cur = { 0 };\n-\tsize_t next = 0, i;\n-\n-\tif (n == 0) {\n-\t\t*seglen = 0;\n-\t\treturn segs;\n-\t}\n-\tfor (i = 0; i < n; i++) {\n-\t\tint log = fastlog2(sizes[i]);\n-\t\tif (cur.log != log && cur.bytes > 0) {\n-\t\t\tstruct segment fresh = {\n-\t\t\t\t.start = i,\n-\t\t\t};\n-\n-\t\t\tsegs[next++] = cur;\n-\t\t\tcur = fresh;\n-\t\t}\n-\n-\t\tcur.log = log;\n-\t\tcur.end = i + 1;\n-\t\tcur.bytes += sizes[i];\n-\t}\n-\tsegs[next++] = cur;\n-\t*seglen = next;\n-\treturn segs;\n-}\n-\n struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n {\n-\tstruct segment min_seg = {\n-\t\t.log = 64,\n-\t};\n-\tstruct segment *segs;\n-\tsize_t seglen = 0, i;\n-\n-\tsegs = sizes_to_segments(&seglen, sizes, n);\n-\tfor (i = 0; i < seglen; i++) {\n-\t\tif (segment_size(&segs[i]) == 1)\n-\t\t\tcontinue;\n+\tstruct segment seg = { 0 };\n+\tuint64_t bytes;\n+\tsize_t i;\n \n-\t\tif (segs[i].log < min_seg.log)\n-\t\t\tmin_seg = segs[i];\n-\t}\n+\t/*\n+\t * If there are no tables or only a single one then we don't have to\n+\t * compact anything. The sequence is geometric by definition already.\n+\t */\n+\tif (n <= 1)\n+\t\treturn seg;\n \n-\twhile (min_seg.start > 0) {\n-\t\tsize_t prev = min_seg.start - 1;\n-\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev]))\n+\t/*\n+\t * Find the ending table of the compaction segment needed to restore the\n+\t * geometric sequence. Note that the segment end is exclusive.\n+\t *\n+\t * To do so, we iterate backwards starting from the most recent table\n+\t * until a valid segment end is found. If the preceding table is smaller\n+\t * than the current table multiplied by the geometric factor (2), the\n+\t * compaction segment end has been identified.\n+\t *\n+\t * Tables after the ending point are not added to the byte count because\n+\t * they are already valid members of the geometric sequence. Due to the\n+\t * properties of a geometric sequence, it is not possible for the sum of\n+\t * these tables to exceed the value of the ending point table.\n+\t *\n+\t * Example table size sequence requiring no compaction:\n+\t * \t64, 32, 16, 8, 4, 2, 1\n+\t *\n+\t * Example table size sequence where compaction segment end is set to\n+\t * the last table. Since the segment end is exclusive, the last table is\n+\t * excluded during subsequent compaction and the table with size 3 is\n+\t * the final table included:\n+\t * \t64, 32, 16, 8, 4, 3, 1\n+\t */\n+\tfor (i = n - 1; i > 0; i--) {\n+\t\tif (sizes[i - 1] < sizes[i] * 2) {\n+\t\t\tseg.end = i + 1;\n+\t\t\tbytes = sizes[i];\n \t\t\tbreak;\n+\t\t}\n+\t}\n \n-\t\tmin_seg.start = prev;\n-\t\tmin_seg.bytes += sizes[prev];\n+\t/*\n+\t * Find the starting table of the compaction segment by iterating\n+\t * through the remaining tables and keeping track of the accumulated\n+\t * size of all tables seen from the segment end table. The previous\n+\t * table is compared to the accumulated size because the tables from the\n+\t * segment end are merged backwards recursively.\n+\t *\n+\t * Note that we keep iterating even after we have found the first\n+\t * starting point. This is because there may be tables in the stack\n+\t * preceding that first starting point which violate the geometric\n+\t * sequence.\n+\t *\n+\t * Example compaction segment start set to table with size 32:\n+\t * \t128, 32, 16, 8, 4, 3, 1\n+\t */\n+\tfor (; i > 0; i--) {\n+\t\tuint64_t curr = bytes;\n+\t\tbytes += sizes[i - 1];\n+\n+\t\tif (sizes[i - 1] < curr * 2) {\n+\t\t\tseg.start = i - 1;\n+\t\t\tseg.bytes = bytes;\n+\t\t}\n \t}\n \n-\treftable_free(segs);\n-\treturn min_seg;\n+\treturn seg;\n }\n \n static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\ndiff --git a/reftable/stack.h b/reftable/stack.h\nindex d919455669e..656f896cc28 100644\n--- a/reftable/stack.h\n+++ b/reftable/stack.h\n@@ -33,12 +33,9 @@ int read_lines(const char *filename, char ***lines);\n \n struct segment {\n \tsize_t start, end;\n-\tint log;\n \tuint64_t bytes;\n };\n \n-int fastlog2(uint64_t sz);\n-struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n);\n struct segment suggest_compaction_segment(uint64_t *sizes, size_t n);\n \n #endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 7336757cf53..a23cee22f4d 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -717,59 +717,13 @@ static void test_reftable_stack_hash_id(void)\n \tclear_dir(dir);\n }\n \n-static void test_log2(void)\n-{\n-\tEXPECT(1 == fastlog2(3));\n-\tEXPECT(2 == fastlog2(4));\n-\tEXPECT(2 == fastlog2(5));\n-}\n-\n-static void test_sizes_to_segments(void)\n-{\n-\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n-\t/* .................0 1 2 3 4 5 */\n-\n-\tsize_t seglen = 0;\n-\tstruct segment *segs =\n-\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n-\tEXPECT(segs[2].log == 3);\n-\tEXPECT(segs[2].start == 5);\n-\tEXPECT(segs[2].end == 6);\n-\n-\tEXPECT(segs[1].log == 2);\n-\tEXPECT(segs[1].start == 2);\n-\tEXPECT(segs[1].end == 5);\n-\treftable_free(segs);\n-}\n-\n-static void test_sizes_to_segments_empty(void)\n-{\n-\tsize_t seglen = 0;\n-\tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n-\tEXPECT(seglen == 0);\n-\treftable_free(segs);\n-}\n-\n-static void test_sizes_to_segments_all_equal(void)\n-{\n-\tuint64_t sizes[] = { 5, 5 };\n-\tsize_t seglen = 0;\n-\tstruct segment *segs =\n-\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n-\tEXPECT(seglen == 1);\n-\tEXPECT(segs[0].start == 0);\n-\tEXPECT(segs[0].end == 2);\n-\treftable_free(segs);\n-}\n-\n static void test_suggest_compaction_segment(void)\n {\n-\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n-\t/* .................0 1 2 3 4 5 6 */\n+\tuint64_t sizes[] = { 512, 64, 17, 16, 9, 9, 9, 16, 2, 16 };\n \tstruct segment min =\n \t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n-\tEXPECT(min.start == 2);\n-\tEXPECT(min.end == 7);\n+\tEXPECT(min.start == 1);\n+\tEXPECT(min.end == 10);\n }\n \n static void test_suggest_compaction_segment_nothing(void)\n@@ -880,6 +834,16 @@ static void test_empty_add(void)\n \treftable_stack_destroy(st2);\n }\n \n+static int fastlog2(uint64_t sz)\n+{\n+\tint l = 0;\n+\tif (sz == 0)\n+\t\treturn 0;\n+\tfor (; sz; sz /= 2)\n+\t\tl++;\n+\treturn l - 1;\n+}\n+\n static void test_reftable_stack_auto_compaction(void)\n {\n \tstruct reftable_write_options cfg = { 0 };\n@@ -1068,7 +1032,6 @@ static void test_reftable_stack_compaction_concurrent_clean(void)\n int stack_test_main(int argc, const char *argv[])\n {\n \tRUN_TEST(test_empty_add);\n-\tRUN_TEST(test_log2);\n \tRUN_TEST(test_names_equal);\n \tRUN_TEST(test_parse_names);\n \tRUN_TEST(test_read_file);\n@@ -1088,9 +1051,6 @@ int stack_test_main(int argc, const char *argv[])\n \tRUN_TEST(test_reftable_stack_update_index_check);\n \tRUN_TEST(test_reftable_stack_uptodate);\n \tRUN_TEST(test_reftable_stack_validate_refname);\n-\tRUN_TEST(test_sizes_to_segments);\n-\tRUN_TEST(test_sizes_to_segments_all_equal);\n-\tRUN_TEST(test_sizes_to_segments_empty);\n \tRUN_TEST(test_suggest_compaction_segment);\n \tRUN_TEST(test_suggest_compaction_segment_nothing);\n \treturn 0;\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex 444f7497907..853f0a051d8 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -293,7 +293,7 @@ test_expect_success 'ref transaction: writes cause auto-compaction' '\n \ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n \ttest_commit -C repo --no-tag A &&\n-\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n \ttest_commit -C repo --no-tag B &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n@@ -320,6 +320,19 @@ test_expect_success 'ref transaction: env var disables compaction' '\n \ttest_line_count -lt $expected repo/.git/reftable/tables.list\n '\n \n+test_expect_success 'ref transaction: alternating table sizes are compacted' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tfor i in $(test_seq 5)\n+\tdo\n+\t\tgit -C repo branch -f foo &&\n+\t\tgit -C repo branch -d foo || return 1\n+\tdone &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list\n+'\n+\n check_fsync_events () {\n \tlocal trace=\"$1\" &&\n \tshift &&\n@@ -345,7 +358,7 @@ test_expect_success 'ref transaction: writes are synced' '\n \t\tgit -C repo -c core.fsync=reference \\\n \t\t-c core.fsyncMethod=fsync update-ref refs/heads/branch HEAD &&\n \tcheck_fsync_events trace2.txt <<-EOF\n-\t\"name\":\"hardware-flush\",\"count\":2\n+\t\"name\":\"hardware-flush\",\"count\":4\n \tEOF\n '\n \n@@ -367,8 +380,8 @@ test_expect_success 'pack-refs: compacts tables' '\n \n \ttest_commit -C repo A &&\n \tls -1 repo/.git/reftable >table-files &&\n-\ttest_line_count = 4 table-files &&\n-\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 3 table-files &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n \n \tgit -C repo pack-refs &&\n \tls -1 repo/.git/reftable >table-files &&\n@@ -400,7 +413,7 @@ do\n \t\t\tumask $umask &&\n \t\t\tgit init --shared=true repo &&\n \t\t\ttest_commit -C repo A &&\n-\t\t\ttest_line_count = 3 repo/.git/reftable/tables.list\n+\t\t\ttest_line_count = 2 repo/.git/reftable/tables.list\n \t\t) &&\n \t\tgit -C repo pack-refs &&\n \t\ttest_expect_perms \"-rw-rw-r--\" repo/.git/reftable/tables.list &&\n@@ -768,12 +781,16 @@ test_expect_success 'worktree: pack-refs in main repo packs main refs' '\n \ttest_when_finished \"rm -rf repo worktree\" &&\n \tgit init repo &&\n \ttest_commit -C repo A &&\n+\n+\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n \tgit -C repo worktree add ../worktree &&\n+\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n+\tgit -C worktree update-ref refs/worktree/per-worktree HEAD &&\n \n-\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n \tgit -C repo pack-refs &&\n-\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n '\n \n@@ -781,13 +798,17 @@ test_expect_success 'worktree: pack-refs in worktree packs worktree refs' '\n \ttest_when_finished \"rm -rf repo worktree\" &&\n \tgit init repo &&\n \ttest_commit -C repo A &&\n+\n+\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n \tgit -C repo worktree add ../worktree &&\n+\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n+\tgit -C worktree update-ref refs/worktree/per-worktree HEAD &&\n \n-\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n \tgit -C worktree pack-refs &&\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 4 repo/.git/reftable/tables.list\n+\ttest_line_count = 3 repo/.git/reftable/tables.list\n '\n \n test_expect_success 'worktree: creating shared ref updates main stack' '\n@@ -801,6 +822,7 @@ test_expect_success 'worktree: creating shared ref updates main stack' '\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n+\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n \tgit -C worktree update-ref refs/heads/shared HEAD &&\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 2 repo/.git/reftable/tables.list\n-- \ngitgitgadget\n"},{"id":"492131","messageId":"ZgzfV5aI8UshiHzX@tanuki","threadId":"61055","inReplyTo":"pull.1683.v4.git.1712103636.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 0/2] reftable/stack: use geometric table compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-03T04:47:19Z","receivedAt":"2024-04-03T04:47:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Apr 03, 2024 at 12:20:34AM +0000, Justin Tobler via GitGitGadget wrote:\n> Hello again,\n> \n> This is the fourth version my patch series that refactors the reftable\n> compaction strategy to instead follow a geometric sequence. Changes compared\n> to v3:\n> \n> * Changed env name from GIT_TEST_REFTABLE_NO_AUTOCOMPACTION to\n> GIT_TEST_REFTABLE_AUTOCOMPACTION and set the default to false. This\n\nYou probably mean true here, not false :)\n\n> should hopefully be a bit more intuitive since it avoids the double\n> negative.\n> * Updated the corresponding env var test in t0610-reftable-basics.sh to\n> assert on the number of tables added and be overall less fragile.\n> * Folded lines that were too long.\n> * Updated some comments in stack.c to more accurately explain that table\n> segment end is exclusive.\n> * Dropped reftable/stack: make segment end inclusive commit to keep segment\n> end exclusive and better follow expectations.\n> \n> Thanks for taking a look!\n\nThis version looks good to me, thanks!\n\nPatrick\n\n> -Justin\n> \n> Justin Tobler (2):\n> reftable/stack: add env to disable autocompaction\n> reftable/stack: use geometric table compaction\n> \n> reftable/stack.c | 126 +++++++++++++++++++------------------\n> reftable/stack.h | 3 -\n> reftable/stack_test.c | 66 ++++---------------\n> reftable/system.h | 1 +\n> t/t0610-reftable-basics.sh | 65 +++++++++++++++----\n> 5 files changed, 132 insertions(+), 129 deletions(-)\n> \n> \n> base-commit: c75fd8d8150afdf836b63a8e0534d9b9e3e111ba\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1683%2Fjltobler%2Fjt%2Freftable-geometric-compaction-v4\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1683/jltobler/jt/reftable-geometric-compaction-v4\n> Pull-Request: https://github.com/gitgitgadget/git/pull/1683\n> \n> Range-diff vs v3:\n> \n> 1: 2fdd8ea1133 ! 1: 2a0421e5f20 reftable/stack: add env to disable autocompaction\n> @@ Commit message\n> \n> In future tests it will be neccesary to create repositories with a set\n> number of tables. To make this easier, introduce the\n> - `GIT_TEST_REFTABLE_NO_AUTOCOMPACTION` environment variable that, when\n> - set, disables autocompaction of reftables.\n> + `GIT_TEST_REFTABLE_AUTOCOMPACTION` environment variable that, when set\n> + to false, disables autocompaction of reftables.\n> \n> Signed-off-by: Justin Tobler \n> \n> @@ reftable/stack.c: int reftable_addition_commit(struct reftable_addition *add)\n> \t\tgoto done;\n> \n> -\tif (!add->stack->disable_auto_compact)\n> -+\tif (!add->stack->disable_auto_compact && !git_env_bool(\"GIT_TEST_REFTABLE_NO_AUTOCOMPACTION\", 0))\n> ++\tif (!add->stack->disable_auto_compact &&\n> ++\t git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1))\n> \t\terr = reftable_stack_auto_compact(add->stack);\n> \n> done:\n> @@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: writes cause a\n> \ttest_line_count = 1 repo/.git/reftable/tables.list\n> '\n> \n> -+test_expect_success 'ref transaction: environment variable disables auto-compaction' '\n> ++test_expect_success 'ref transaction: env var disables compaction' '\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\n> +\tgit init repo &&\n> +\ttest_commit -C repo A &&\n> -+\tfor i in $(test_seq 20)\n> ++\n> ++\tstart=$(wc -l ++\titerations=5 &&\n> ++\texpected=$((start + iterations)) &&\n> ++\n> ++\tfor i in $(test_seq $iterations)\n> +\tdo\n> -+\t\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo update-ref branch-$i HEAD || return 1\n> ++\t\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n> ++\t\tgit -C repo update-ref branch-$i HEAD || return 1\n> +\tdone &&\n> -+\ttest_line_count = 23 repo/.git/reftable/tables.list &&\n> ++\ttest_line_count = $expected repo/.git/reftable/tables.list &&\n> +\n> +\tgit -C repo update-ref foo HEAD &&\n> -+\ttest_line_count = 1 repo/.git/reftable/tables.list\n> ++\ttest_line_count -lt $expected repo/.git/reftable/tables.list\n> +'\n> +\n> check_fsync_events () {\n> 2: 7e62c2286ae ! 2: e0f4d0dbcc1 reftable/stack: use geometric table compaction\n> @@ reftable/stack.c: static int segment_size(struct segment *s)\n> -\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev]))\n> +\t/*\n> +\t * Find the ending table of the compaction segment needed to restore the\n> -+\t * geometric sequence.\n> ++\t * geometric sequence. Note that the segment end is exclusive.\n> +\t *\n> +\t * To do so, we iterate backwards starting from the most recent table\n> +\t * until a valid segment end is found. If the preceding table is smaller\n> +\t * than the current table multiplied by the geometric factor (2), the\n> -+\t * current table is set as the compaction segment end.\n> ++\t * compaction segment end has been identified.\n> +\t *\n> +\t * Tables after the ending point are not added to the byte count because\n> +\t * they are already valid members of the geometric sequence. Due to the\n> @@ reftable/stack.c: static int segment_size(struct segment *s)\n> +\t * Example table size sequence requiring no compaction:\n> +\t * \t64, 32, 16, 8, 4, 2, 1\n> +\t *\n> -+\t * Example compaction segment end set to table with size 3:\n> ++\t * Example table size sequence where compaction segment end is set to\n> ++\t * the last table. Since the segment end is exclusive, the last table is\n> ++\t * excluded during subsequent compaction and the table with size 3 is\n> ++\t * the final table included:\n> +\t * \t64, 32, 16, 8, 4, 3, 1\n> +\t */\n> +\tfor (i = n - 1; i > 0; i--) {\n> @@ reftable/stack_test.c: static void test_empty_add(void)\n> +\tint l = 0;\n> +\tif (sz == 0)\n> +\t\treturn 0;\n> -+\tfor (; sz; sz /= 2) {\n> ++\tfor (; sz; sz /= 2)\n> +\t\tl++;\n> -+\t}\n> +\treturn l - 1;\n> +}\n> +\n> @@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: writes cause a\n> \n> \ttest_commit -C repo --no-tag B &&\n> \ttest_line_count = 1 repo/.git/reftable/tables.list\n> -@@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: environment variable disables auto-compact\n> - \tdo\n> - \t\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo update-ref branch-$i HEAD || return 1\n> - \tdone &&\n> --\ttest_line_count = 23 repo/.git/reftable/tables.list &&\n> -+\ttest_line_count = 22 repo/.git/reftable/tables.list &&\n> - \n> - \tgit -C repo update-ref foo HEAD &&\n> - \ttest_line_count = 1 repo/.git/reftable/tables.list\n> +@@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: env var disables compaction' '\n> + \ttest_line_count -lt $expected repo/.git/reftable/tables.list\n> '\n> \n> +test_expect_success 'ref transaction: alternating table sizes are compacted' '\n> +\ttest_when_finished \"rm -rf repo\" &&\n> ++\n> +\tgit init repo &&\n> +\ttest_commit -C repo A &&\n> -+\tfor i in $(test_seq 20)\n> ++\tfor i in $(test_seq 5)\n> +\tdo\n> +\t\tgit -C repo branch -f foo &&\n> +\t\tgit -C repo branch -d foo || return 1\n> @@ t/t0610-reftable-basics.sh: test_expect_success 'worktree: pack-refs in main rep\n> \ttest_when_finished \"rm -rf repo worktree\" &&\n> \tgit init repo &&\n> \ttest_commit -C repo A &&\n> --\tgit -C repo worktree add ../worktree &&\n> -+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo worktree add ../worktree &&\n> -+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C worktree update-ref refs/worktree/per-worktree HEAD &&\n> ++\n> ++\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n> + \tgit -C repo worktree add ../worktree &&\n> ++\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n> ++\tgit -C worktree update-ref refs/worktree/per-worktree HEAD &&\n> \n> -\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n> -\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n> @@ t/t0610-reftable-basics.sh: test_expect_success 'worktree: pack-refs in worktree\n> \ttest_when_finished \"rm -rf repo worktree\" &&\n> \tgit init repo &&\n> \ttest_commit -C repo A &&\n> --\tgit -C repo worktree add ../worktree &&\n> -+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C repo worktree add ../worktree &&\n> -+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C worktree update-ref refs/worktree/per-worktree HEAD &&\n> ++\n> ++\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n> + \tgit -C repo worktree add ../worktree &&\n> ++\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n> ++\tgit -C worktree update-ref refs/worktree/per-worktree HEAD &&\n> \n> -\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n> -\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n> @@ t/t0610-reftable-basics.sh: test_expect_success 'worktree: pack-refs in worktree\n> '\n> \n> test_expect_success 'worktree: creating shared ref updates main stack' '\n> - \ttest_when_finished \"rm -rf repo worktree\" &&\n> - \tgit init repo &&\n> - \ttest_commit -C repo A &&\n> -+\ttest_commit -C repo B &&\n> - \n> - \tgit -C repo worktree add ../worktree &&\n> - \tgit -C repo pack-refs &&\n> @@ t/t0610-reftable-basics.sh: test_expect_success 'worktree: creating shared ref updates main stack' '\n> \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n> \ttest_line_count = 1 repo/.git/reftable/tables.list &&\n> \n> --\tgit -C worktree update-ref refs/heads/shared HEAD &&\n> -+\tGIT_TEST_REFTABLE_NO_AUTOCOMPACTION=true git -C worktree update-ref refs/heads/shared HEAD &&\n> ++\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n> + \tgit -C worktree update-ref refs/heads/shared HEAD &&\n> \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n> \ttest_line_count = 2 repo/.git/reftable/tables.list\n> - '\n> 3: 9a33914c852 < -: ----------- reftable/stack: make segment end inclusive\n> \n> -- \n> gitgitgadget\n"},{"id":"492149","messageId":"CAOw_e7az+S4nkG1HKDcqXkK0BOhs6cBpkb=QEGBoZW8MSEEvGg@mail.gmail.com","threadId":"61055","inReplyTo":"Zfzd8YdAwzYRYkA3@tanuki","subject":"Re: [PATCH v2 0/3] reftable/stack: use geometric table compaction","fromName":"Han-Wen Nienhuys","fromEmail":"hanwenn@gmail.com","sentAt":"2024-04-03T10:13:42Z","receivedAt":"2024-04-03T10:13:53Z","isPatch":true,"sender":{"key":"hanwenn@gmail.com","avatar":"https://gravatar.com/avatar/058832acb8d613baeb6ce9d21b009d9772424a309b9a521330423daef909a27a?d=mp&s=160"},"body":"On Fri, Mar 22, 2024 at 10:51 AM Patrick Steinhardt wrote:\n> > Thanks for taking a look!\n>\n> Cc'ing Han-Wen and Josh for additional input. From my point of view the\n> new algorithm is simpler to understand and less fragile, but I do wonder\n> whether there is anything that we're missing.\n\nGood spotting. I hadn't thought about alternating tables.\n\nI have one minor criticism:\n\nEnvironment variables are untyped global variables without any form of\ndata protection, so I find them unsavoury, and have tried to avoid\nthem throughout. (The whole reftable library only looks at $TMPDIR in\ntests). They're also accessible to end users, so it can become a\nfeature that can inadvertently become a maintenance burden.\n\nFor testing, there is a stack->disable_auto_compact.\n\nIf you want to keep that style, I would elevate disable_auto_compact\ninto reftable_write_options to make it API surface. This will let you\nuse it in tests written in C, which can be unittests and therefore\nmore precise and fine-grained. They also run more quickly, and are\neasier to instrument with asan/valgrind/etc. The test for tables with\nalternating sizes can be easily written in C.\n\nIf you really need it, you could initialize disable_auto_compact from\nthe environment, but I would suggest avoiding it if possible.\n\n-- \nHan-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen\n"},{"id":"492150","messageId":"Zg0tEa7DDEuub09Q@tanuki","threadId":"61055","inReplyTo":"CAOw_e7az+S4nkG1HKDcqXkK0BOhs6cBpkb=QEGBoZW8MSEEvGg@mail.gmail.com","subject":"Re: [PATCH v2 0/3] reftable/stack: use geometric table compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-03T10:18:57Z","receivedAt":"2024-04-03T10:19:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Apr 03, 2024 at 12:13:42PM +0200, Han-Wen Nienhuys wrote:\n> On Fri, Mar 22, 2024 at 10:51 AM Patrick Steinhardt wrote:\n> > > Thanks for taking a look!\n> >\n> > Cc'ing Han-Wen and Josh for additional input. From my point of view the\n> > new algorithm is simpler to understand and less fragile, but I do wonder\n> > whether there is anything that we're missing.\n> \n> Good spotting. I hadn't thought about alternating tables.\n> \n> I have one minor criticism:\n> \n> Environment variables are untyped global variables without any form of\n> data protection, so I find them unsavoury, and have tried to avoid\n> them throughout. (The whole reftable library only looks at $TMPDIR in\n> tests). They're also accessible to end users, so it can become a\n> feature that can inadvertently become a maintenance burden.\n> \n> For testing, there is a stack->disable_auto_compact.\n> \n> If you want to keep that style, I would elevate disable_auto_compact\n> into reftable_write_options to make it API surface. This will let you\n> use it in tests written in C, which can be unittests and therefore\n> more precise and fine-grained. They also run more quickly, and are\n> easier to instrument with asan/valgrind/etc. The test for tables with\n> alternating sizes can be easily written in C.\n> \n> If you really need it, you could initialize disable_auto_compact from\n> the environment, but I would suggest avoiding it if possible.\n\nThat's actually a good point. I think keeping this as an environment\nvariable isn't too bad as a stop-gap measure for now, and it should be\nobvious to users that it's not for general use due to the `GIT_TEST`\nprefix.\n\nBut I'm definitely supportive of lifting it out of the reftable library\nand into the reftable backend so that it is specific to Git, not to the\nreftable library.\n\nPatrick\n"},{"id":"492153","messageId":"CAOLa=ZR0odWdoVBDTN+0j5d4f=s1z5zGcTNK+2tBgDX7RjXX2Q@mail.gmail.com","threadId":"61055","inReplyTo":"pull.1683.v4.git.1712103636.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 0/2] reftable/stack: use geometric table compaction","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-04-03T11:12:14Z","receivedAt":"2024-04-03T11:12:16Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Hello,\n\n\"Justin Tobler via GitGitGadget\" writes:\n> Hello again,\n>\n> This is the fourth version my patch series that refactors the reftable\n> compaction strategy to instead follow a geometric sequence. Changes compared\n> to v3:\n>\n> * Changed env name from GIT_TEST_REFTABLE_NO_AUTOCOMPACTION to\n> GIT_TEST_REFTABLE_AUTOCOMPACTION and set the default to false. This\n> should hopefully be a bit more intuitive since it avoids the double\n> negative.\n> * Updated the corresponding env var test in t0610-reftable-basics.sh to\n> assert on the number of tables added and be overall less fragile.\n> * Folded lines that were too long.\n> * Updated some comments in stack.c to more accurately explain that table\n> segment end is exclusive.\n> * Dropped reftable/stack: make segment end inclusive commit to keep segment\n> end exclusive and better follow expectations.\n>\n> Thanks for taking a look!\n>\n> -Justin\n>\n\nJust a note that this doesn't merge nicely with nice because of\nconflicts with a2f711ade0c4816a59155d72559cbc4759cd4699.\n"},{"id":"492163","messageId":"b24jjzw72rlcpctteokr5yfbuwuy2cc3qvibzhuju4gbj63lfa@gbtsmvufyuhd","threadId":"61055","inReplyTo":"Zg0tEa7DDEuub09Q@tanuki","subject":"Re: [PATCH v2 0/3] reftable/stack: use geometric table compaction","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-04-03T15:14:43Z","receivedAt":"2024-04-03T15:15:41Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/04/03 12:18PM, Patrick Steinhardt wrote:\n> On Wed, Apr 03, 2024 at 12:13:42PM +0200, Han-Wen Nienhuys wrote:\n> > On Fri, Mar 22, 2024 at 10:51 AM Patrick Steinhardt wrote:\n> > > > Thanks for taking a look!\n> > >\n> > > Cc'ing Han-Wen and Josh for additional input. From my point of view the\n> > > new algorithm is simpler to understand and less fragile, but I do wonder\n> > > whether there is anything that we're missing.\n> > \n> > Good spotting. I hadn't thought about alternating tables.\n> > \n> > I have one minor criticism:\n> > \n> > Environment variables are untyped global variables without any form of\n> > data protection, so I find them unsavoury, and have tried to avoid\n> > them throughout. (The whole reftable library only looks at $TMPDIR in\n> > tests). They're also accessible to end users, so it can become a\n> > feature that can inadvertently become a maintenance burden.\n> > \n> > For testing, there is a stack->disable_auto_compact.\n> > \n> > If you want to keep that style, I would elevate disable_auto_compact\n> > into reftable_write_options to make it API surface. This will let you\n> > use it in tests written in C, which can be unittests and therefore\n> > more precise and fine-grained. They also run more quickly, and are\n> > easier to instrument with asan/valgrind/etc. The test for tables with\n> > alternating sizes can be easily written in C.\n> > \n> > If you really need it, you could initialize disable_auto_compact from\n> > the environment, but I would suggest avoiding it if possible.\n> \n> That's actually a good point. I think keeping this as an environment\n> variable isn't too bad as a stop-gap measure for now, and it should be\n> obvious to users that it's not for general use due to the `GIT_TEST`\n> prefix.\n> \n> But I'm definitely supportive of lifting it out of the reftable library\n> and into the reftable backend so that it is specific to Git, not to the\n> reftable library.\n\nMoving the env out of the reftable library seems reasonable to me. I'll\nmake this change as part of the next version of this series.\n\n-Justin\n\n"},{"id":"492172","messageId":"xmqqo7aq8kmc.fsf@gitster.g","threadId":"61055","inReplyTo":"Zg0tEa7DDEuub09Q@tanuki","subject":"Re: [PATCH v2 0/3] reftable/stack: use geometric table compaction","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-03T16:40:59Z","receivedAt":"2024-04-03T16:41:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> But I'm definitely supportive of lifting it out of the reftable library\n> and into the reftable backend so that it is specific to Git, not to the\n> reftable library.\n\nAbsolutely. Thanks for bringing up a good point and a nice\nsolution. I do think it makes sense to handle the environment\nvariable on the Git side.\n"},{"id":"492174","messageId":"xmqqfrw28jwt.fsf@gitster.g","threadId":"61055","inReplyTo":"CAOLa=ZR0odWdoVBDTN+0j5d4f=s1z5zGcTNK+2tBgDX7RjXX2Q@mail.gmail.com","subject":"Re: [PATCH v4 0/2] reftable/stack: use geometric table compaction","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-03T16:56:18Z","receivedAt":"2024-04-03T16:56:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> Just a note that this doesn't merge nicely with nice because of\n> conflicts with a2f711ade0c4816a59155d72559cbc4759cd4699.\n\nTrue. I've been resolving the conflict already, so there is not\nmuch new to see here ;-)\n\nIt seems that the plan is to update the section that conflicts even\nfurther to avoid the access to the environment variable, so I'll\nhave to update my rerere database yet another time.\n\nThanks.\n"},{"id":"492183","messageId":"xmqq5xwy6z1b.fsf@gitster.g","threadId":"61055","inReplyTo":"pull.1683.v2.git.1711060819.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/3] reftable/stack: use geometric table compaction","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-03T19:12:32Z","receivedAt":"2024-04-03T19:12:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Justin Tobler via GitGitGadget\" writes:\n\n> This is the second version my patch series that refactors the reftable\n> compaction strategy to instead follow a geometric sequence. Changes compared\n> to v1:\n>\n> * Added GIT_TEST_REFTABLE_NO_AUTOCOMPACTION environment variable to disable\n> reftable compaction when testing.\n> * Refactored worktree tests in t0610-reftable-basics.sh to properly assert\n> git-pack-refs(1) works as expected.\n> * Added test to validate that alternating table sizes are compacted.\n> * Added benchmark to compare compaction strategies.\n> * Moved change that made compaction segment end inclusive to its own\n> commit.\n> * Added additional explanation in commits and comments and fixed typos.\n\nHas anybody took a look at recent failures with this series present\nin 'seen' [*1*] and without [*2*] in osx-reftable jobs for t0610?\n\n\n\n*1* https://github.com/git/git/actions/runs/8543205866/job/23406512990\n*2* https://github.com/git/git/actions/runs/8543840764/job/23408543876\n"},{"id":"492184","messageId":"Zg2uS5JXjQZsVZhJ@framework","threadId":"61055","inReplyTo":"xmqq5xwy6z1b.fsf@gitster.g","subject":"Re: [PATCH v2 0/3] reftable/stack: use geometric table compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-03T19:30:19Z","receivedAt":"2024-04-03T19:30:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Apr 03, 2024 at 12:12:32PM -0700, Junio C Hamano wrote:\n> \"Justin Tobler via GitGitGadget\" writes:\n> \n> > This is the second version my patch series that refactors the reftable\n> > compaction strategy to instead follow a geometric sequence. Changes compared\n> > to v1:\n> >\n> > * Added GIT_TEST_REFTABLE_NO_AUTOCOMPACTION environment variable to disable\n> > reftable compaction when testing.\n> > * Refactored worktree tests in t0610-reftable-basics.sh to properly assert\n> > git-pack-refs(1) works as expected.\n> > * Added test to validate that alternating table sizes are compacted.\n> > * Added benchmark to compare compaction strategies.\n> > * Moved change that made compaction segment end inclusive to its own\n> > commit.\n> > * Added additional explanation in commits and comments and fixed typos.\n> \n> Has anybody took a look at recent failures with this series present\n> in 'seen' [*1*] and without [*2*] in osx-reftable jobs for t0610?\n> \n> *1* https://github.com/git/git/actions/runs/8543205866/job/23406512990\n> *2* https://github.com/git/git/actions/runs/8543840764/job/23408543876\n\nI noticed that both `seen` and `next` started to fail in the GitLab\nmirror today. Unless somebody else beats me to it I'll investigate\ntomorrow what causes these.\n\nPatrick\n"},{"id":"492199","messageId":"Zg472uByuW6domfh@tanuki","threadId":"61055","inReplyTo":"Zg2uS5JXjQZsVZhJ@framework","subject":"Re: [PATCH v2 0/3] reftable/stack: use geometric table compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-04T05:34:18Z","receivedAt":"2024-04-04T05:34:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Apr 03, 2024 at 09:30:19PM +0200, Patrick Steinhardt wrote:\n> On Wed, Apr 03, 2024 at 12:12:32PM -0700, Junio C Hamano wrote:\n> > \"Justin Tobler via GitGitGadget\" writes:\n> > \n> > > This is the second version my patch series that refactors the reftable\n> > > compaction strategy to instead follow a geometric sequence. Changes compared\n> > > to v1:\n> > >\n> > > * Added GIT_TEST_REFTABLE_NO_AUTOCOMPACTION environment variable to disable\n> > > reftable compaction when testing.\n> > > * Refactored worktree tests in t0610-reftable-basics.sh to properly assert\n> > > git-pack-refs(1) works as expected.\n> > > * Added test to validate that alternating table sizes are compacted.\n> > > * Added benchmark to compare compaction strategies.\n> > > * Moved change that made compaction segment end inclusive to its own\n> > > commit.\n> > > * Added additional explanation in commits and comments and fixed typos.\n> > \n> > Has anybody took a look at recent failures with this series present\n> > in 'seen' [*1*] and without [*2*] in osx-reftable jobs for t0610?\n> > \n> > *1* https://github.com/git/git/actions/runs/8543205866/job/23406512990\n> > *2* https://github.com/git/git/actions/runs/8543840764/job/23408543876\n> \n> I noticed that both `seen` and `next` started to fail in the GitLab\n> mirror today. Unless somebody else beats me to it I'll investigate\n> tomorrow what causes these.\n\nThings work on GitLab CI again, all pipelines are green there now. Which\nprobably also is because you have evicted this series from \"seen\". On\nGitHub most of the failures I see are still related to the regression in\nlibcurl.\n\nBut your first link definitely is specific to the changes in this patch\nseries and comes from a bad interaction with \"ps/pack-refs-auto\". That\nseries added a few tests where the exact number of tables that exist is\nnow different.\n\nJustin wanted to make that series a dependency anyway, so I assume that\nhe'll then address those issues.\n\nPatrick\n"},{"id":"492255","messageId":"pull.1683.v5.git.1712255369.gitgitgadget@gmail.com","threadId":"61055","inReplyTo":"pull.1683.v4.git.1712103636.gitgitgadget@gmail.com","subject":"[PATCH v5 0/3] reftable/stack: use geometric table compaction","fromName":"Justin Tobler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-04-04T18:29:26Z","receivedAt":"2024-04-04T18:29:33Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Hello again,\n\nThis is the fifth version my patch series that refactors the reftable\ncompaction strategy to instead follow a geometric sequence. Changes compared\nto v4:\n\n * To fix some failing tests and conflicts, this patch series now depends on\n the ps/pack-refs-auto series which is currently in next.\n * Lifted the GIT_TEST_REFTABLE_AUTOCOMPACTION env out of the reftable\n library and into the reftable backend code.\n\nThanks for taking a look!\n\n-Justin\n\nJustin Tobler (3):\n reftable/stack: allow disabling of auto-compaction\n reftable/stack: add env to disable autocompaction\n reftable/stack: use geometric table compaction\n\n refs/reftable-backend.c | 4 ++\n reftable/reftable-writer.h | 3 +\n reftable/stack.c | 125 +++++++++++++++++++------------------\n reftable/stack.h | 4 --\n reftable/stack_test.c | 77 ++++++-----------------\n t/t0610-reftable-basics.sh | 71 ++++++++++++++++-----\n 6 files changed, 146 insertions(+), 138 deletions(-)\n\n\nbase-commit: 4b32163adf4863c6df3bb6b43540fa2ca3494e28\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1683%2Fjltobler%2Fjt%2Freftable-geometric-compaction-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1683/jltobler/jt/reftable-geometric-compaction-v5\nPull-Request: https://github.com/gitgitgadget/git/pull/1683\n\nRange-diff vs v4:\n\n -: ----------- > 1: a7011dbc6aa reftable/stack: allow disabling of auto-compaction\n 1: 2a0421e5f20 ! 2: 7c4fe0e9ec5 reftable/stack: add env to disable autocompaction\n @@ Commit message\n \n Signed-off-by: Justin Tobler \n \n - ## reftable/stack.c ##\n -@@ reftable/stack.c: int reftable_addition_commit(struct reftable_addition *add)\n - \tif (err)\n - \t\tgoto done;\n - \n --\tif (!add->stack->disable_auto_compact)\n -+\tif (!add->stack->disable_auto_compact &&\n -+\t git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1))\n - \t\terr = reftable_stack_auto_compact(add->stack);\n - \n - done:\n -\n - ## reftable/system.h ##\n -@@ reftable/system.h: license that can be found in the LICENSE file or at\n - #include \"tempfile.h\"\n - #include \"hash-ll.h\" /* hash ID, sizes.*/\n - #include \"dir.h\" /* remove_dir_recursively, for tests.*/\n + ## refs/reftable-backend.c ##\n +@@\n + #include \"../reftable/reftable-merged.h\"\n + #include \"../setup.h\"\n + #include \"../strmap.h\"\n +#include \"parse.h\"\n + #include \"refs-internal.h\"\n \n - int hash_size(uint32_t id);\n + /*\n +@@ refs/reftable-backend.c: static struct ref_store *reftable_be_init(struct repository *repo,\n + \trefs->write_options.hash_id = repo->hash_algo->format_id;\n + \trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n \n ++\tif (!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1))\n ++\t\trefs->write_options.disable_auto_compact = 1;\n ++\n + \t/*\n + \t * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.\n + \t * This stack contains both the shared and the main worktree refs.\n \n ## t/t0610-reftable-basics.sh ##\n @@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: writes cause auto-compaction' '\n 2: e0f4d0dbcc1 ! 3: 8f124acf0f8 reftable/stack: use geometric table compaction\n @@ reftable/stack_test.c: static void test_empty_add(void)\n +\n static void test_reftable_stack_auto_compaction(void)\n {\n - \tstruct reftable_write_options cfg = { 0 };\n + \tstruct reftable_write_options cfg = {\n @@ reftable/stack_test.c: static void test_reftable_stack_compaction_concurrent_clean(void)\n int stack_test_main(int argc, const char *argv[])\n {\n @@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: writes are syn\n \tEOF\n '\n \n +@@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: fails gracefully when auto compaction fail\n + \t\t\tdone ||\n + \t\t\texit 1\n + \t\tdone &&\n +-\t\ttest_line_count = 13 .git/reftable/tables.list\n ++\t\ttest_line_count = 10 .git/reftable/tables.list\n + \t)\n + '\n + \n @@ t/t0610-reftable-basics.sh: test_expect_success 'pack-refs: compacts tables' '\n \n \ttest_commit -C repo A &&\n @@ t/t0610-reftable-basics.sh: test_expect_success 'pack-refs: compacts tables' '\n \n \tgit -C repo pack-refs &&\n \tls -1 repo/.git/reftable >table-files &&\n +@@ t/t0610-reftable-basics.sh: test_expect_success \"$command: auto compaction\" '\n + \t\t# The tables should have been auto-compacted, and thus auto\n + \t\t# compaction should not have to do anything.\n + \t\tls -1 .git/reftable >tables-expect &&\n +-\t\ttest_line_count = 4 tables-expect &&\n ++\t\ttest_line_count = 3 tables-expect &&\n + \t\tgit $command --auto &&\n + \t\tls -1 .git/reftable >tables-actual &&\n + \t\ttest_cmp tables-expect tables-actual &&\n +@@ t/t0610-reftable-basics.sh: test_expect_success \"$command: auto compaction\" '\n + \t\tgit branch B &&\n + \t\tgit branch C &&\n + \t\trm .git/reftable/*.lock &&\n +-\t\ttest_line_count = 5 .git/reftable/tables.list &&\n ++\t\ttest_line_count = 4 .git/reftable/tables.list &&\n + \n + \t\tgit $command --auto &&\n + \t\ttest_line_count = 1 .git/reftable/tables.list\n @@ t/t0610-reftable-basics.sh: do\n \t\t\tumask $umask &&\n \t\t\tgit init --shared=true repo &&\n\n-- \ngitgitgadget\n"},{"id":"492256","messageId":"a7011dbc6aa5cb256957bda5456ca989ce75e4a5.1712255369.git.gitgitgadget@gmail.com","threadId":"61055","inReplyTo":"pull.1683.v5.git.1712255369.gitgitgadget@gmail.com","subject":"[PATCH v5 1/3] reftable/stack: allow disabling of auto-compaction","fromName":"Justin Tobler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-04-04T18:29:27Z","receivedAt":"2024-04-04T18:29:34Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"From: Justin Tobler \n\nMove the `disable_auto_compact` option into `reftable_write_options` to\nallow a stack to be configured with auto-compaction disabled. In a\nsubsequent commit, this is used to disable auto-compaction when a\nspecific environment variable is set.\n\nSigned-off-by: Justin Tobler \n---\n reftable/reftable-writer.h | 3 +++\n reftable/stack.c | 2 +-\n reftable/stack.h | 1 -\n reftable/stack_test.c | 11 ++++++-----\n 4 files changed, 10 insertions(+), 7 deletions(-)\n\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 7c7cae5f99b..155bf0bbe2a 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -46,6 +46,9 @@ struct reftable_write_options {\n \t * is a single line, and add '\\n' if missing.\n \t */\n \tunsigned exact_log_message : 1;\n+\n+\t/* boolean: Prevent auto-compaction of tables. */\n+\tunsigned disable_auto_compact : 1;\n };\n \n /* reftable_block_stats holds statistics for a single block type */\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex dde50b61d69..1a7cdad12c9 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -680,7 +680,7 @@ int reftable_addition_commit(struct reftable_addition *add)\n \tif (err)\n \t\tgoto done;\n \n-\tif (!add->stack->disable_auto_compact) {\n+\tif (!add->stack->config.disable_auto_compact) {\n \t\t/*\n \t\t * Auto-compact the stack to keep the number of tables in\n \t\t * control. It is possible that a concurrent writer is already\ndiff --git a/reftable/stack.h b/reftable/stack.h\nindex d919455669e..c862053025f 100644\n--- a/reftable/stack.h\n+++ b/reftable/stack.h\n@@ -19,7 +19,6 @@ struct reftable_stack {\n \tint list_fd;\n \n \tchar *reftable_dir;\n-\tint disable_auto_compact;\n \n \tstruct reftable_write_options config;\n \ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 351e35bd86d..4fec823f14f 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -325,7 +325,7 @@ static void test_reftable_stack_transaction_api_performs_auto_compaction(void)\n \t\t * we can ensure that we indeed honor this setting and have\n \t\t * better control over when exactly auto compaction runs.\n \t\t */\n-\t\tst->disable_auto_compact = i != n;\n+\t\tst->config.disable_auto_compact = i != n;\n \n \t\terr = reftable_stack_new_addition(&add, st);\n \t\tEXPECT_ERR(err);\n@@ -497,6 +497,7 @@ static void test_reftable_stack_add(void)\n \tstruct reftable_write_options cfg = {\n \t\t.exact_log_message = 1,\n \t\t.default_permissions = 0660,\n+\t\t.disable_auto_compact = 1,\n \t};\n \tstruct reftable_stack *st = NULL;\n \tchar *dir = get_tmp_dir(__LINE__);\n@@ -508,7 +509,6 @@ static void test_reftable_stack_add(void)\n \n \terr = reftable_new_stack(&st, dir, cfg);\n \tEXPECT_ERR(err);\n-\tst->disable_auto_compact = 1;\n \n \tfor (i = 0; i < N; i++) {\n \t\tchar buf[256];\n@@ -935,7 +935,9 @@ static void test_empty_add(void)\n \n static void test_reftable_stack_auto_compaction(void)\n {\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options cfg = {\n+\t\t.disable_auto_compact = 1,\n+\t};\n \tstruct reftable_stack *st = NULL;\n \tchar *dir = get_tmp_dir(__LINE__);\n \n@@ -945,7 +947,6 @@ static void test_reftable_stack_auto_compaction(void)\n \terr = reftable_new_stack(&st, dir, cfg);\n \tEXPECT_ERR(err);\n \n-\tst->disable_auto_compact = 1; /* call manually below for coverage. */\n \tfor (i = 0; i < N; i++) {\n \t\tchar name[100];\n \t\tstruct reftable_ref_record ref = {\n@@ -994,7 +995,7 @@ static void test_reftable_stack_add_performs_auto_compaction(void)\n \t\t * we can ensure that we indeed honor this setting and have\n \t\t * better control over when exactly auto compaction runs.\n \t\t */\n-\t\tst->disable_auto_compact = i != n;\n+\t\tst->config.disable_auto_compact = i != n;\n \n \t\tstrbuf_reset(&refname);\n \t\tstrbuf_addf(&refname, \"branch-%04d\", i);\n-- \ngitgitgadget\n\n"},{"id":"492257","messageId":"7c4fe0e9ec597203ee37d2c2503be319e87ff5ee.1712255369.git.gitgitgadget@gmail.com","threadId":"61055","inReplyTo":"pull.1683.v5.git.1712255369.gitgitgadget@gmail.com","subject":"[PATCH v5 2/3] reftable/stack: add env to disable autocompaction","fromName":"Justin Tobler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-04-04T18:29:28Z","receivedAt":"2024-04-04T18:29:35Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"From: Justin Tobler \n\nIn future tests it will be neccesary to create repositories with a set\nnumber of tables. To make this easier, introduce the\n`GIT_TEST_REFTABLE_AUTOCOMPACTION` environment variable that, when set\nto false, disables autocompaction of reftables.\n\nSigned-off-by: Justin Tobler \n---\n refs/reftable-backend.c | 4 ++++\n t/t0610-reftable-basics.sh | 21 +++++++++++++++++++++\n 2 files changed, 25 insertions(+)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 0bed6d2ab48..6b6191f89dd 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -18,6 +18,7 @@\n #include \"../reftable/reftable-merged.h\"\n #include \"../setup.h\"\n #include \"../strmap.h\"\n+#include \"parse.h\"\n #include \"refs-internal.h\"\n \n /*\n@@ -248,6 +249,9 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \trefs->write_options.hash_id = repo->hash_algo->format_id;\n \trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n \n+\tif (!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1))\n+\t\trefs->write_options.disable_auto_compact = 1;\n+\n \t/*\n \t * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.\n \t * This stack contains both the shared and the main worktree refs.\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex 931d888bbbc..c9e10b34684 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -299,6 +299,27 @@ test_expect_success 'ref transaction: writes cause auto-compaction' '\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n '\n \n+test_expect_success 'ref transaction: env var disables compaction' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tstart=$(wc -l \n\nTo reduce the number of on-disk reftables, compaction is performed.\nContiguous tables with the same binary log value of size are grouped\ninto segments. The segment that has both the lowest binary log value and\ncontains more than one table is set as the starting point when\nidentifying the compaction segment.\n\nSince segments containing a single table are not initially considered\nfor compaction, if the table appended to the list does not match the\nprevious table log value, no compaction occurs for the new table. It is\ntherefore possible for unbounded growth of the table list. This can be\ndemonstrated by repeating the following sequence:\n\ngit branch -f foo\ngit branch -d foo\n\nEach operation results in a new table being written with no compaction\noccurring until a separate operation produces a table matching the\nprevious table log value.\n\nInstead, to avoid unbounded growth of the table list, the compaction\nstrategy is updated to ensure tables follow a geometric sequence after\neach operation by individually evaluating each table in reverse index\norder. This strategy results in a much simpler and more robust algorithm\ncompared to the previous one while also maintaining a minimal ordered\nset of tables on-disk.\n\nWhen creating 10 thousand references, the new strategy has no\nperformance impact:\n\nBenchmark 1: update-ref: create refs sequentially (revision = HEAD~)\n Time (mean ± σ): 26.516 s ± 0.047 s [User: 17.864 s, System: 8.491 s]\n Range (min … max): 26.447 s … 26.569 s 10 runs\n\nBenchmark 2: update-ref: create refs sequentially (revision = HEAD)\n Time (mean ± σ): 26.417 s ± 0.028 s [User: 17.738 s, System: 8.500 s]\n Range (min … max): 26.366 s … 26.444 s 10 runs\n\nSummary\n update-ref: create refs sequentially (revision = HEAD) ran\n 1.00 ± 0.00 times faster than update-ref: create refs sequentially (revision = HEAD~)\n\nSome tests in `t0610-reftable-basics.sh` assert the on-disk state of\ntables and are therefore updated to specify the correct new table count.\nSince compaction is more aggressive in ensuring tables maintain a\ngeometric sequence, the expected table count is reduced in these tests.\nIn `reftable/stack_test.c` tests related to `sizes_to_segments()` are\nremoved because the function is no longer needed. Also, the\n`test_suggest_compaction_segment()` test is updated to better showcase\nand reflect the new geometric compaction behavior.\n\nSigned-off-by: Justin Tobler \n---\n reftable/stack.c | 123 +++++++++++++++++++------------------\n reftable/stack.h | 3 -\n reftable/stack_test.c | 66 ++++----------------\n t/t0610-reftable-basics.sh | 50 ++++++++++-----\n 4 files changed, 111 insertions(+), 131 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 1a7cdad12c9..80266bcbab1 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1216,75 +1216,76 @@ static int segment_size(struct segment *s)\n \treturn s->end - s->start;\n }\n \n-int fastlog2(uint64_t sz)\n-{\n-\tint l = 0;\n-\tif (sz == 0)\n-\t\treturn 0;\n-\tfor (; sz; sz /= 2) {\n-\t\tl++;\n-\t}\n-\treturn l - 1;\n-}\n-\n-struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n)\n-{\n-\tstruct segment *segs = reftable_calloc(n, sizeof(*segs));\n-\tstruct segment cur = { 0 };\n-\tsize_t next = 0, i;\n-\n-\tif (n == 0) {\n-\t\t*seglen = 0;\n-\t\treturn segs;\n-\t}\n-\tfor (i = 0; i < n; i++) {\n-\t\tint log = fastlog2(sizes[i]);\n-\t\tif (cur.log != log && cur.bytes > 0) {\n-\t\t\tstruct segment fresh = {\n-\t\t\t\t.start = i,\n-\t\t\t};\n-\n-\t\t\tsegs[next++] = cur;\n-\t\t\tcur = fresh;\n-\t\t}\n-\n-\t\tcur.log = log;\n-\t\tcur.end = i + 1;\n-\t\tcur.bytes += sizes[i];\n-\t}\n-\tsegs[next++] = cur;\n-\t*seglen = next;\n-\treturn segs;\n-}\n-\n struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n {\n-\tstruct segment min_seg = {\n-\t\t.log = 64,\n-\t};\n-\tstruct segment *segs;\n-\tsize_t seglen = 0, i;\n-\n-\tsegs = sizes_to_segments(&seglen, sizes, n);\n-\tfor (i = 0; i < seglen; i++) {\n-\t\tif (segment_size(&segs[i]) == 1)\n-\t\t\tcontinue;\n+\tstruct segment seg = { 0 };\n+\tuint64_t bytes;\n+\tsize_t i;\n \n-\t\tif (segs[i].log < min_seg.log)\n-\t\t\tmin_seg = segs[i];\n-\t}\n+\t/*\n+\t * If there are no tables or only a single one then we don't have to\n+\t * compact anything. The sequence is geometric by definition already.\n+\t */\n+\tif (n <= 1)\n+\t\treturn seg;\n \n-\twhile (min_seg.start > 0) {\n-\t\tsize_t prev = min_seg.start - 1;\n-\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev]))\n+\t/*\n+\t * Find the ending table of the compaction segment needed to restore the\n+\t * geometric sequence. Note that the segment end is exclusive.\n+\t *\n+\t * To do so, we iterate backwards starting from the most recent table\n+\t * until a valid segment end is found. If the preceding table is smaller\n+\t * than the current table multiplied by the geometric factor (2), the\n+\t * compaction segment end has been identified.\n+\t *\n+\t * Tables after the ending point are not added to the byte count because\n+\t * they are already valid members of the geometric sequence. Due to the\n+\t * properties of a geometric sequence, it is not possible for the sum of\n+\t * these tables to exceed the value of the ending point table.\n+\t *\n+\t * Example table size sequence requiring no compaction:\n+\t * \t64, 32, 16, 8, 4, 2, 1\n+\t *\n+\t * Example table size sequence where compaction segment end is set to\n+\t * the last table. Since the segment end is exclusive, the last table is\n+\t * excluded during subsequent compaction and the table with size 3 is\n+\t * the final table included:\n+\t * \t64, 32, 16, 8, 4, 3, 1\n+\t */\n+\tfor (i = n - 1; i > 0; i--) {\n+\t\tif (sizes[i - 1] < sizes[i] * 2) {\n+\t\t\tseg.end = i + 1;\n+\t\t\tbytes = sizes[i];\n \t\t\tbreak;\n+\t\t}\n+\t}\n \n-\t\tmin_seg.start = prev;\n-\t\tmin_seg.bytes += sizes[prev];\n+\t/*\n+\t * Find the starting table of the compaction segment by iterating\n+\t * through the remaining tables and keeping track of the accumulated\n+\t * size of all tables seen from the segment end table. The previous\n+\t * table is compared to the accumulated size because the tables from the\n+\t * segment end are merged backwards recursively.\n+\t *\n+\t * Note that we keep iterating even after we have found the first\n+\t * starting point. This is because there may be tables in the stack\n+\t * preceding that first starting point which violate the geometric\n+\t * sequence.\n+\t *\n+\t * Example compaction segment start set to table with size 32:\n+\t * \t128, 32, 16, 8, 4, 3, 1\n+\t */\n+\tfor (; i > 0; i--) {\n+\t\tuint64_t curr = bytes;\n+\t\tbytes += sizes[i - 1];\n+\n+\t\tif (sizes[i - 1] < curr * 2) {\n+\t\t\tseg.start = i - 1;\n+\t\t\tseg.bytes = bytes;\n+\t\t}\n \t}\n \n-\treftable_free(segs);\n-\treturn min_seg;\n+\treturn seg;\n }\n \n static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\ndiff --git a/reftable/stack.h b/reftable/stack.h\nindex c862053025f..d43efa47607 100644\n--- a/reftable/stack.h\n+++ b/reftable/stack.h\n@@ -32,12 +32,9 @@ int read_lines(const char *filename, char ***lines);\n \n struct segment {\n \tsize_t start, end;\n-\tint log;\n \tuint64_t bytes;\n };\n \n-int fastlog2(uint64_t sz);\n-struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n);\n struct segment suggest_compaction_segment(uint64_t *sizes, size_t n);\n \n #endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 4fec823f14f..1df3ffce526 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -770,59 +770,13 @@ static void test_reftable_stack_hash_id(void)\n \tclear_dir(dir);\n }\n \n-static void test_log2(void)\n-{\n-\tEXPECT(1 == fastlog2(3));\n-\tEXPECT(2 == fastlog2(4));\n-\tEXPECT(2 == fastlog2(5));\n-}\n-\n-static void test_sizes_to_segments(void)\n-{\n-\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n-\t/* .................0 1 2 3 4 5 */\n-\n-\tsize_t seglen = 0;\n-\tstruct segment *segs =\n-\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n-\tEXPECT(segs[2].log == 3);\n-\tEXPECT(segs[2].start == 5);\n-\tEXPECT(segs[2].end == 6);\n-\n-\tEXPECT(segs[1].log == 2);\n-\tEXPECT(segs[1].start == 2);\n-\tEXPECT(segs[1].end == 5);\n-\treftable_free(segs);\n-}\n-\n-static void test_sizes_to_segments_empty(void)\n-{\n-\tsize_t seglen = 0;\n-\tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n-\tEXPECT(seglen == 0);\n-\treftable_free(segs);\n-}\n-\n-static void test_sizes_to_segments_all_equal(void)\n-{\n-\tuint64_t sizes[] = { 5, 5 };\n-\tsize_t seglen = 0;\n-\tstruct segment *segs =\n-\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n-\tEXPECT(seglen == 1);\n-\tEXPECT(segs[0].start == 0);\n-\tEXPECT(segs[0].end == 2);\n-\treftable_free(segs);\n-}\n-\n static void test_suggest_compaction_segment(void)\n {\n-\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n-\t/* .................0 1 2 3 4 5 6 */\n+\tuint64_t sizes[] = { 512, 64, 17, 16, 9, 9, 9, 16, 2, 16 };\n \tstruct segment min =\n \t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n-\tEXPECT(min.start == 2);\n-\tEXPECT(min.end == 7);\n+\tEXPECT(min.start == 1);\n+\tEXPECT(min.end == 10);\n }\n \n static void test_suggest_compaction_segment_nothing(void)\n@@ -933,6 +887,16 @@ static void test_empty_add(void)\n \treftable_stack_destroy(st2);\n }\n \n+static int fastlog2(uint64_t sz)\n+{\n+\tint l = 0;\n+\tif (sz == 0)\n+\t\treturn 0;\n+\tfor (; sz; sz /= 2)\n+\t\tl++;\n+\treturn l - 1;\n+}\n+\n static void test_reftable_stack_auto_compaction(void)\n {\n \tstruct reftable_write_options cfg = {\n@@ -1122,7 +1086,6 @@ static void test_reftable_stack_compaction_concurrent_clean(void)\n int stack_test_main(int argc, const char *argv[])\n {\n \tRUN_TEST(test_empty_add);\n-\tRUN_TEST(test_log2);\n \tRUN_TEST(test_names_equal);\n \tRUN_TEST(test_parse_names);\n \tRUN_TEST(test_read_file);\n@@ -1143,9 +1106,6 @@ int stack_test_main(int argc, const char *argv[])\n \tRUN_TEST(test_reftable_stack_update_index_check);\n \tRUN_TEST(test_reftable_stack_uptodate);\n \tRUN_TEST(test_reftable_stack_validate_refname);\n-\tRUN_TEST(test_sizes_to_segments);\n-\tRUN_TEST(test_sizes_to_segments_all_equal);\n-\tRUN_TEST(test_sizes_to_segments_empty);\n \tRUN_TEST(test_suggest_compaction_segment);\n \tRUN_TEST(test_suggest_compaction_segment_nothing);\n \treturn 0;\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex c9e10b34684..8eec093788d 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -293,7 +293,7 @@ test_expect_success 'ref transaction: writes cause auto-compaction' '\n \ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n \ttest_commit -C repo --no-tag A &&\n-\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n \ttest_commit -C repo --no-tag B &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n@@ -320,6 +320,19 @@ test_expect_success 'ref transaction: env var disables compaction' '\n \ttest_line_count -lt $expected repo/.git/reftable/tables.list\n '\n \n+test_expect_success 'ref transaction: alternating table sizes are compacted' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tfor i in $(test_seq 5)\n+\tdo\n+\t\tgit -C repo branch -f foo &&\n+\t\tgit -C repo branch -d foo || return 1\n+\tdone &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list\n+'\n+\n check_fsync_events () {\n \tlocal trace=\"$1\" &&\n \tshift &&\n@@ -345,7 +358,7 @@ test_expect_success 'ref transaction: writes are synced' '\n \t\tgit -C repo -c core.fsync=reference \\\n \t\t-c core.fsyncMethod=fsync update-ref refs/heads/branch HEAD &&\n \tcheck_fsync_events trace2.txt <<-EOF\n-\t\"name\":\"hardware-flush\",\"count\":2\n+\t\"name\":\"hardware-flush\",\"count\":4\n \tEOF\n '\n \n@@ -377,7 +390,7 @@ test_expect_success 'ref transaction: fails gracefully when auto compaction fail\n \t\t\tdone ||\n \t\t\texit 1\n \t\tdone &&\n-\t\ttest_line_count = 13 .git/reftable/tables.list\n+\t\ttest_line_count = 10 .git/reftable/tables.list\n \t)\n '\n \n@@ -387,8 +400,8 @@ test_expect_success 'pack-refs: compacts tables' '\n \n \ttest_commit -C repo A &&\n \tls -1 repo/.git/reftable >table-files &&\n-\ttest_line_count = 4 table-files &&\n-\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 3 table-files &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n \n \tgit -C repo pack-refs &&\n \tls -1 repo/.git/reftable >table-files &&\n@@ -429,7 +442,7 @@ test_expect_success \"$command: auto compaction\" '\n \t\t# The tables should have been auto-compacted, and thus auto\n \t\t# compaction should not have to do anything.\n \t\tls -1 .git/reftable >tables-expect &&\n-\t\ttest_line_count = 4 tables-expect &&\n+\t\ttest_line_count = 3 tables-expect &&\n \t\tgit $command --auto &&\n \t\tls -1 .git/reftable >tables-actual &&\n \t\ttest_cmp tables-expect tables-actual &&\n@@ -447,7 +460,7 @@ test_expect_success \"$command: auto compaction\" '\n \t\tgit branch B &&\n \t\tgit branch C &&\n \t\trm .git/reftable/*.lock &&\n-\t\ttest_line_count = 5 .git/reftable/tables.list &&\n+\t\ttest_line_count = 4 .git/reftable/tables.list &&\n \n \t\tgit $command --auto &&\n \t\ttest_line_count = 1 .git/reftable/tables.list\n@@ -479,7 +492,7 @@ do\n \t\t\tumask $umask &&\n \t\t\tgit init --shared=true repo &&\n \t\t\ttest_commit -C repo A &&\n-\t\t\ttest_line_count = 3 repo/.git/reftable/tables.list\n+\t\t\ttest_line_count = 2 repo/.git/reftable/tables.list\n \t\t) &&\n \t\tgit -C repo pack-refs &&\n \t\ttest_expect_perms \"-rw-rw-r--\" repo/.git/reftable/tables.list &&\n@@ -847,12 +860,16 @@ test_expect_success 'worktree: pack-refs in main repo packs main refs' '\n \ttest_when_finished \"rm -rf repo worktree\" &&\n \tgit init repo &&\n \ttest_commit -C repo A &&\n+\n+\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n \tgit -C repo worktree add ../worktree &&\n+\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n+\tgit -C worktree update-ref refs/worktree/per-worktree HEAD &&\n \n-\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n \tgit -C repo pack-refs &&\n-\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n '\n \n@@ -860,13 +877,17 @@ test_expect_success 'worktree: pack-refs in worktree packs worktree refs' '\n \ttest_when_finished \"rm -rf repo worktree\" &&\n \tgit init repo &&\n \ttest_commit -C repo A &&\n+\n+\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n \tgit -C repo worktree add ../worktree &&\n+\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n+\tgit -C worktree update-ref refs/worktree/per-worktree HEAD &&\n \n-\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n \tgit -C worktree pack-refs &&\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 4 repo/.git/reftable/tables.list\n+\ttest_line_count = 3 repo/.git/reftable/tables.list\n '\n \n test_expect_success 'worktree: creating shared ref updates main stack' '\n@@ -880,6 +901,7 @@ test_expect_success 'worktree: creating shared ref updates main stack' '\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n+\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n \tgit -C worktree update-ref refs/heads/shared HEAD &&\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 2 repo/.git/reftable/tables.list\n-- \ngitgitgadget\n"},{"id":"492258","messageId":"u47s7kwjrbcdseqb72nuplhs3tuaac5bl5dk6qqu7z3ptma5gn@qlrggnkip3yu","threadId":"61055","inReplyTo":"Zg472uByuW6domfh@tanuki","subject":"Re: [PATCH v2 0/3] reftable/stack: use geometric table compaction","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-04-04T18:28:39Z","receivedAt":"2024-04-04T18:29:38Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/04/04 07:34AM, Patrick Steinhardt wrote:\n> On Wed, Apr 03, 2024 at 09:30:19PM +0200, Patrick Steinhardt wrote:\n> > On Wed, Apr 03, 2024 at 12:12:32PM -0700, Junio C Hamano wrote:\n> > > \"Justin Tobler via GitGitGadget\" writes:\n> > > \n> > > > This is the second version my patch series that refactors the reftable\n> > > > compaction strategy to instead follow a geometric sequence. Changes compared\n> > > > to v1:\n> > > >\n> > > > * Added GIT_TEST_REFTABLE_NO_AUTOCOMPACTION environment variable to disable\n> > > > reftable compaction when testing.\n> > > > * Refactored worktree tests in t0610-reftable-basics.sh to properly assert\n> > > > git-pack-refs(1) works as expected.\n> > > > * Added test to validate that alternating table sizes are compacted.\n> > > > * Added benchmark to compare compaction strategies.\n> > > > * Moved change that made compaction segment end inclusive to its own\n> > > > commit.\n> > > > * Added additional explanation in commits and comments and fixed typos.\n> > > \n> > > Has anybody took a look at recent failures with this series present\n> > > in 'seen' [*1*] and without [*2*] in osx-reftable jobs for t0610?\n> > > \n> > > *1* https://github.com/git/git/actions/runs/8543205866/job/23406512990\n> > > *2* https://github.com/git/git/actions/runs/8543840764/job/23408543876\n> > \n> > I noticed that both `seen` and `next` started to fail in the GitLab\n> > mirror today. Unless somebody else beats me to it I'll investigate\n> > tomorrow what causes these.\n> \n> Things work on GitLab CI again, all pipelines are green there now. Which\n> probably also is because you have evicted this series from \"seen\". On\n> GitHub most of the failures I see are still related to the regression in\n> libcurl.\n> \n> But your first link definitely is specific to the changes in this patch\n> series and comes from a bad interaction with \"ps/pack-refs-auto\". That\n> series added a few tests where the exact number of tables that exist is\n> now different.\n> \n> Justin wanted to make that series a dependency anyway, so I assume that\n> he'll then address those issues.\n\nYes, I've made this series depend on \"ps/pack-refs-auto\" and have\nupdated the conflicting tests in the next version :)\n\n-Justin\n\n"},{"id":"492488","messageId":"ZhOKxLCWxp-lOyVh@tanuki","threadId":"61055","inReplyTo":"pull.1683.v5.git.1712255369.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 0/3] reftable/stack: use geometric table compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T06:12:20Z","receivedAt":"2024-04-08T06:12:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Apr 04, 2024 at 06:29:26PM +0000, Justin Tobler via GitGitGadget wrote:\n> Hello again,\n> \n> This is the fifth version my patch series that refactors the reftable\n> compaction strategy to instead follow a geometric sequence. Changes compared\n> to v4:\n> \n> * To fix some failing tests and conflicts, this patch series now depends on\n> the ps/pack-refs-auto series which is currently in next.\n> * Lifted the GIT_TEST_REFTABLE_AUTOCOMPACTION env out of the reftable\n> library and into the reftable backend code.\n> \n> Thanks for taking a look!\n> \n> -Justin\n\nI've added two additional nits which you may or may not want to address.\nBut overall this patch series looks good to me. Thanks!\n\nPatrick\n\n> Justin Tobler (3):\n> reftable/stack: allow disabling of auto-compaction\n> reftable/stack: add env to disable autocompaction\n> reftable/stack: use geometric table compaction\n> \n> refs/reftable-backend.c | 4 ++\n> reftable/reftable-writer.h | 3 +\n> reftable/stack.c | 125 +++++++++++++++++++------------------\n> reftable/stack.h | 4 --\n> reftable/stack_test.c | 77 ++++++-----------------\n> t/t0610-reftable-basics.sh | 71 ++++++++++++++++-----\n> 6 files changed, 146 insertions(+), 138 deletions(-)\n> \n> \n> base-commit: 4b32163adf4863c6df3bb6b43540fa2ca3494e28\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1683%2Fjltobler%2Fjt%2Freftable-geometric-compaction-v5\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1683/jltobler/jt/reftable-geometric-compaction-v5\n> Pull-Request: https://github.com/gitgitgadget/git/pull/1683\n> \n> Range-diff vs v4:\n> \n> -: ----------- > 1: a7011dbc6aa reftable/stack: allow disabling of auto-compaction\n> 1: 2a0421e5f20 ! 2: 7c4fe0e9ec5 reftable/stack: add env to disable autocompaction\n> @@ Commit message\n> \n> Signed-off-by: Justin Tobler \n> \n> - ## reftable/stack.c ##\n> -@@ reftable/stack.c: int reftable_addition_commit(struct reftable_addition *add)\n> - \tif (err)\n> - \t\tgoto done;\n> - \n> --\tif (!add->stack->disable_auto_compact)\n> -+\tif (!add->stack->disable_auto_compact &&\n> -+\t git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1))\n> - \t\terr = reftable_stack_auto_compact(add->stack);\n> - \n> - done:\n> -\n> - ## reftable/system.h ##\n> -@@ reftable/system.h: license that can be found in the LICENSE file or at\n> - #include \"tempfile.h\"\n> - #include \"hash-ll.h\" /* hash ID, sizes.*/\n> - #include \"dir.h\" /* remove_dir_recursively, for tests.*/\n> + ## refs/reftable-backend.c ##\n> +@@\n> + #include \"../reftable/reftable-merged.h\"\n> + #include \"../setup.h\"\n> + #include \"../strmap.h\"\n> +#include \"parse.h\"\n> + #include \"refs-internal.h\"\n> \n> - int hash_size(uint32_t id);\n> + /*\n> +@@ refs/reftable-backend.c: static struct ref_store *reftable_be_init(struct repository *repo,\n> + \trefs->write_options.hash_id = repo->hash_algo->format_id;\n> + \trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n> \n> ++\tif (!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1))\n> ++\t\trefs->write_options.disable_auto_compact = 1;\n> ++\n> + \t/*\n> + \t * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.\n> + \t * This stack contains both the shared and the main worktree refs.\n> \n> ## t/t0610-reftable-basics.sh ##\n> @@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: writes cause auto-compaction' '\n> 2: e0f4d0dbcc1 ! 3: 8f124acf0f8 reftable/stack: use geometric table compaction\n> @@ reftable/stack_test.c: static void test_empty_add(void)\n> +\n> static void test_reftable_stack_auto_compaction(void)\n> {\n> - \tstruct reftable_write_options cfg = { 0 };\n> + \tstruct reftable_write_options cfg = {\n> @@ reftable/stack_test.c: static void test_reftable_stack_compaction_concurrent_clean(void)\n> int stack_test_main(int argc, const char *argv[])\n> {\n> @@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: writes are syn\n> \tEOF\n> '\n> \n> +@@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: fails gracefully when auto compaction fail\n> + \t\t\tdone ||\n> + \t\t\texit 1\n> + \t\tdone &&\n> +-\t\ttest_line_count = 13 .git/reftable/tables.list\n> ++\t\ttest_line_count = 10 .git/reftable/tables.list\n> + \t)\n> + '\n> + \n> @@ t/t0610-reftable-basics.sh: test_expect_success 'pack-refs: compacts tables' '\n> \n> \ttest_commit -C repo A &&\n> @@ t/t0610-reftable-basics.sh: test_expect_success 'pack-refs: compacts tables' '\n> \n> \tgit -C repo pack-refs &&\n> \tls -1 repo/.git/reftable >table-files &&\n> +@@ t/t0610-reftable-basics.sh: test_expect_success \"$command: auto compaction\" '\n> + \t\t# The tables should have been auto-compacted, and thus auto\n> + \t\t# compaction should not have to do anything.\n> + \t\tls -1 .git/reftable >tables-expect &&\n> +-\t\ttest_line_count = 4 tables-expect &&\n> ++\t\ttest_line_count = 3 tables-expect &&\n> + \t\tgit $command --auto &&\n> + \t\tls -1 .git/reftable >tables-actual &&\n> + \t\ttest_cmp tables-expect tables-actual &&\n> +@@ t/t0610-reftable-basics.sh: test_expect_success \"$command: auto compaction\" '\n> + \t\tgit branch B &&\n> + \t\tgit branch C &&\n> + \t\trm .git/reftable/*.lock &&\n> +-\t\ttest_line_count = 5 .git/reftable/tables.list &&\n> ++\t\ttest_line_count = 4 .git/reftable/tables.list &&\n> + \n> + \t\tgit $command --auto &&\n> + \t\ttest_line_count = 1 .git/reftable/tables.list\n> @@ t/t0610-reftable-basics.sh: do\n> \t\t\tumask $umask &&\n> \t\t\tgit init --shared=true repo &&\n> \n> -- \n> gitgitgadget\n"},{"id":"492489","messageId":"ZhOKy5W5e11ce2Rd@tanuki","threadId":"61055","inReplyTo":"a7011dbc6aa5cb256957bda5456ca989ce75e4a5.1712255369.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 1/3] reftable/stack: allow disabling of auto-compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T06:12:27Z","receivedAt":"2024-04-08T06:12:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Apr 04, 2024 at 06:29:27PM +0000, Justin Tobler via GitGitGadget wrote:\n> From: Justin Tobler \n> \n> Move the `disable_auto_compact` option into `reftable_write_options` to\n> allow a stack to be configured with auto-compaction disabled. In a\n> subsequent commit, this is used to disable auto-compaction when a\n> specific environment variable is set.\n\nThis patch looks good to me. I think the commit subject and message\ncould use a bit of polishing though: we do not add a new way to disable\nauto-compaction as that already exists. The important bit though is that\nthis toggle is purely internal right now and thus cannot be accessed by\nlibrary users.\n\nI'd thus propose something along the following lines:\n\n ```\n reftable/stack: expose option to disable auto-compaction\n\n While the reftable stack already has a bit controls whether or not\n to run auto-compation, this bit is not accessible to users of the\n library. There are usecases though where a caller may want to have\n more control over auto-compaction.\n\n Move the `disable_auto_compact` option into `reftable_write_options`\n to allow external callers to disable auto-compaction. This will be\n used in a subsequent commit.\n ```\n\nPatrick\n\n> Signed-off-by: Justin Tobler \n> ---\n> reftable/reftable-writer.h | 3 +++\n> reftable/stack.c | 2 +-\n> reftable/stack.h | 1 -\n> reftable/stack_test.c | 11 ++++++-----\n> 4 files changed, 10 insertions(+), 7 deletions(-)\n> \n> diff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\n> index 7c7cae5f99b..155bf0bbe2a 100644\n> --- a/reftable/reftable-writer.h\n> +++ b/reftable/reftable-writer.h\n> @@ -46,6 +46,9 @@ struct reftable_write_options {\n> \t * is a single line, and add '\\n' if missing.\n> \t */\n> \tunsigned exact_log_message : 1;\n> +\n> +\t/* boolean: Prevent auto-compaction of tables. */\n> +\tunsigned disable_auto_compact : 1;\n> };\n> \n> /* reftable_block_stats holds statistics for a single block type */\n> diff --git a/reftable/stack.c b/reftable/stack.c\n> index dde50b61d69..1a7cdad12c9 100644\n> --- a/reftable/stack.c\n> +++ b/reftable/stack.c\n> @@ -680,7 +680,7 @@ int reftable_addition_commit(struct reftable_addition *add)\n> \tif (err)\n> \t\tgoto done;\n> \n> -\tif (!add->stack->disable_auto_compact) {\n> +\tif (!add->stack->config.disable_auto_compact) {\n> \t\t/*\n> \t\t * Auto-compact the stack to keep the number of tables in\n> \t\t * control. It is possible that a concurrent writer is already\n> diff --git a/reftable/stack.h b/reftable/stack.h\n> index d919455669e..c862053025f 100644\n> --- a/reftable/stack.h\n> +++ b/reftable/stack.h\n> @@ -19,7 +19,6 @@ struct reftable_stack {\n> \tint list_fd;\n> \n> \tchar *reftable_dir;\n> -\tint disable_auto_compact;\n> \n> \tstruct reftable_write_options config;\n> \n> diff --git a/reftable/stack_test.c b/reftable/stack_test.c\n> index 351e35bd86d..4fec823f14f 100644\n> --- a/reftable/stack_test.c\n> +++ b/reftable/stack_test.c\n> @@ -325,7 +325,7 @@ static void test_reftable_stack_transaction_api_performs_auto_compaction(void)\n> \t\t * we can ensure that we indeed honor this setting and have\n> \t\t * better control over when exactly auto compaction runs.\n> \t\t */\n> -\t\tst->disable_auto_compact = i != n;\n> +\t\tst->config.disable_auto_compact = i != n;\n> \n> \t\terr = reftable_stack_new_addition(&add, st);\n> \t\tEXPECT_ERR(err);\n> @@ -497,6 +497,7 @@ static void test_reftable_stack_add(void)\n> \tstruct reftable_write_options cfg = {\n> \t\t.exact_log_message = 1,\n> \t\t.default_permissions = 0660,\n> +\t\t.disable_auto_compact = 1,\n> \t};\n> \tstruct reftable_stack *st = NULL;\n> \tchar *dir = get_tmp_dir(__LINE__);\n> @@ -508,7 +509,6 @@ static void test_reftable_stack_add(void)\n> \n> \terr = reftable_new_stack(&st, dir, cfg);\n> \tEXPECT_ERR(err);\n> -\tst->disable_auto_compact = 1;\n> \n> \tfor (i = 0; i < N; i++) {\n> \t\tchar buf[256];\n> @@ -935,7 +935,9 @@ static void test_empty_add(void)\n> \n> static void test_reftable_stack_auto_compaction(void)\n> {\n> -\tstruct reftable_write_options cfg = { 0 };\n> +\tstruct reftable_write_options cfg = {\n> +\t\t.disable_auto_compact = 1,\n> +\t};\n> \tstruct reftable_stack *st = NULL;\n> \tchar *dir = get_tmp_dir(__LINE__);\n> \n> @@ -945,7 +947,6 @@ static void test_reftable_stack_auto_compaction(void)\n> \terr = reftable_new_stack(&st, dir, cfg);\n> \tEXPECT_ERR(err);\n> \n> -\tst->disable_auto_compact = 1; /* call manually below for coverage. */\n> \tfor (i = 0; i < N; i++) {\n> \t\tchar name[100];\n> \t\tstruct reftable_ref_record ref = {\n> @@ -994,7 +995,7 @@ static void test_reftable_stack_add_performs_auto_compaction(void)\n> \t\t * we can ensure that we indeed honor this setting and have\n> \t\t * better control over when exactly auto compaction runs.\n> \t\t */\n> -\t\tst->disable_auto_compact = i != n;\n> +\t\tst->config.disable_auto_compact = i != n;\n> \n> \t\tstrbuf_reset(&refname);\n> \t\tstrbuf_addf(&refname, \"branch-%04d\", i);\n> -- \n> gitgitgadget\n> \n"},{"id":"492490","messageId":"ZhOK262a5WCxS9MN@tanuki","threadId":"61055","inReplyTo":"7c4fe0e9ec597203ee37d2c2503be319e87ff5ee.1712255369.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 2/3] reftable/stack: add env to disable autocompaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T06:12:43Z","receivedAt":"2024-04-08T06:12:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Apr 04, 2024 at 06:29:28PM +0000, Justin Tobler via GitGitGadget wrote:\n> From: Justin Tobler \n> \n> In future tests it will be neccesary to create repositories with a set\n> number of tables. To make this easier, introduce the\n> `GIT_TEST_REFTABLE_AUTOCOMPACTION` environment variable that, when set\n> to false, disables autocompaction of reftables.\n> \n> Signed-off-by: Justin Tobler \n> ---\n> refs/reftable-backend.c | 4 ++++\n> t/t0610-reftable-basics.sh | 21 +++++++++++++++++++++\n> 2 files changed, 25 insertions(+)\n> \n> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> index 0bed6d2ab48..6b6191f89dd 100644\n> --- a/refs/reftable-backend.c\n> +++ b/refs/reftable-backend.c\n> @@ -18,6 +18,7 @@\n> #include \"../reftable/reftable-merged.h\"\n> #include \"../setup.h\"\n> #include \"../strmap.h\"\n> +#include \"parse.h\"\n> #include \"refs-internal.h\"\n> \n> /*\n> @@ -248,6 +249,9 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n> \trefs->write_options.hash_id = repo->hash_algo->format_id;\n> \trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n> \n> +\tif (!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1))\n> +\t\trefs->write_options.disable_auto_compact = 1;\n> +\n\nThis could be simplified to:\n\n ```\n refs->write_options.disable_auto_compact =\n !git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1);\n ```\n\nPatrick\n\n> \t/*\n> \t * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.\n> \t * This stack contains both the shared and the main worktree refs.\n> diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\n> index 931d888bbbc..c9e10b34684 100755\n> --- a/t/t0610-reftable-basics.sh\n> +++ b/t/t0610-reftable-basics.sh\n> @@ -299,6 +299,27 @@ test_expect_success 'ref transaction: writes cause auto-compaction' '\n> \ttest_line_count = 1 repo/.git/reftable/tables.list\n> '\n> \n> +test_expect_success 'ref transaction: env var disables compaction' '\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\n> +\tgit init repo &&\n> +\ttest_commit -C repo A &&\n> +\n> +\tstart=$(wc -l +\titerations=5 &&\n> +\texpected=$((start + iterations)) &&\n> +\n> +\tfor i in $(test_seq $iterations)\n> +\tdo\n> +\t\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n> +\t\tgit -C repo update-ref branch-$i HEAD || return 1\n> +\tdone &&\n> +\ttest_line_count = $expected repo/.git/reftable/tables.list &&\n> +\n> +\tgit -C repo update-ref foo HEAD &&\n> +\ttest_line_count -lt $expected repo/.git/reftable/tables.list\n> +'\n> +\n> check_fsync_events () {\n> \tlocal trace=\"$1\" &&\n> \tshift &&\n> -- \n> gitgitgadget\n> \n"},{"id":"492550","messageId":"pull.1683.v6.git.1712593016.gitgitgadget@gmail.com","threadId":"61055","inReplyTo":"pull.1683.v5.git.1712255369.gitgitgadget@gmail.com","subject":"[PATCH v6 0/3] reftable/stack: use geometric table compaction","fromName":"Justin Tobler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-04-08T16:16:52Z","receivedAt":"2024-04-08T16:17:00Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"Hello again,\n\nThis is the sixth version my patch series that refactors the reftable\ncompaction strategy to instead follow a geometric sequence. Changes compared\nto v5:\n\n * Reworded commit message to more clearly explain that the already existing\n configuration to disable auto-compaction is being exposed to callers of\n the library.\n * Simplified expression to set the disable_auto_compact configuration.\n\nThanks for taking a look!\n\n-Justin\n\nJustin Tobler (3):\n reftable/stack: expose option to disable auto-compaction\n reftable/stack: add env to disable autocompaction\n reftable/stack: use geometric table compaction\n\n refs/reftable-backend.c | 3 +\n reftable/reftable-writer.h | 3 +\n reftable/stack.c | 125 +++++++++++++++++++------------------\n reftable/stack.h | 4 --\n reftable/stack_test.c | 77 ++++++-----------------\n t/t0610-reftable-basics.sh | 71 ++++++++++++++++-----\n 6 files changed, 145 insertions(+), 138 deletions(-)\n\n\nbase-commit: 4b32163adf4863c6df3bb6b43540fa2ca3494e28\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1683%2Fjltobler%2Fjt%2Freftable-geometric-compaction-v6\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1683/jltobler/jt/reftable-geometric-compaction-v6\nPull-Request: https://github.com/gitgitgadget/git/pull/1683\n\nRange-diff vs v5:\n\n 1: a7011dbc6aa ! 1: 9c8f6b336ec reftable/stack: allow disabling of auto-compaction\n @@ Metadata\n Author: Justin Tobler \n \n ## Commit message ##\n - reftable/stack: allow disabling of auto-compaction\n + reftable/stack: expose option to disable auto-compaction\n +\n + The reftable stack already has a variable to configure whether or not to\n + run auto-compaction, but it is inaccessible to users of the library.\n + There exist use cases where a caller may want to have more control over\n + auto-compaction.\n \n Move the `disable_auto_compact` option into `reftable_write_options` to\n - allow a stack to be configured with auto-compaction disabled. In a\n - subsequent commit, this is used to disable auto-compaction when a\n - specific environment variable is set.\n + allow external callers to disable auto-compaction. This will be used in\n + a subsequent commit.\n \n Signed-off-by: Justin Tobler \n \n 2: 7c4fe0e9ec5 ! 2: c7bc7346540 reftable/stack: add env to disable autocompaction\n @@ refs/reftable-backend.c\n \n /*\n @@ refs/reftable-backend.c: static struct ref_store *reftable_be_init(struct repository *repo,\n + \trefs->write_options.block_size = 4096;\n \trefs->write_options.hash_id = repo->hash_algo->format_id;\n \trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n ++\trefs->write_options.disable_auto_compact =\n ++\t\t!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1);\n \n -+\tif (!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1))\n -+\t\trefs->write_options.disable_auto_compact = 1;\n -+\n \t/*\n \t * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.\n - \t * This stack contains both the shared and the main worktree refs.\n \n ## t/t0610-reftable-basics.sh ##\n @@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: writes cause auto-compaction' '\n 3: 8f124acf0f8 = 3: d75494a88b0 reftable/stack: use geometric table compaction\n\n-- \ngitgitgadget\n"},{"id":"492552","messageId":"9c8f6b336ecea50b1c0476cd274481560aa79740.1712593016.git.gitgitgadget@gmail.com","threadId":"61055","inReplyTo":"pull.1683.v6.git.1712593016.gitgitgadget@gmail.com","subject":"[PATCH v6 1/3] reftable/stack: expose option to disable auto-compaction","fromName":"Justin Tobler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-04-08T16:16:53Z","receivedAt":"2024-04-08T16:17:01Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"From: Justin Tobler \n\nThe reftable stack already has a variable to configure whether or not to\nrun auto-compaction, but it is inaccessible to users of the library.\nThere exist use cases where a caller may want to have more control over\nauto-compaction.\n\nMove the `disable_auto_compact` option into `reftable_write_options` to\nallow external callers to disable auto-compaction. This will be used in\na subsequent commit.\n\nSigned-off-by: Justin Tobler \n---\n reftable/reftable-writer.h | 3 +++\n reftable/stack.c | 2 +-\n reftable/stack.h | 1 -\n reftable/stack_test.c | 11 ++++++-----\n 4 files changed, 10 insertions(+), 7 deletions(-)\n\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 7c7cae5f99b..155bf0bbe2a 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -46,6 +46,9 @@ struct reftable_write_options {\n \t * is a single line, and add '\\n' if missing.\n \t */\n \tunsigned exact_log_message : 1;\n+\n+\t/* boolean: Prevent auto-compaction of tables. */\n+\tunsigned disable_auto_compact : 1;\n };\n \n /* reftable_block_stats holds statistics for a single block type */\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex dde50b61d69..1a7cdad12c9 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -680,7 +680,7 @@ int reftable_addition_commit(struct reftable_addition *add)\n \tif (err)\n \t\tgoto done;\n \n-\tif (!add->stack->disable_auto_compact) {\n+\tif (!add->stack->config.disable_auto_compact) {\n \t\t/*\n \t\t * Auto-compact the stack to keep the number of tables in\n \t\t * control. It is possible that a concurrent writer is already\ndiff --git a/reftable/stack.h b/reftable/stack.h\nindex d919455669e..c862053025f 100644\n--- a/reftable/stack.h\n+++ b/reftable/stack.h\n@@ -19,7 +19,6 @@ struct reftable_stack {\n \tint list_fd;\n \n \tchar *reftable_dir;\n-\tint disable_auto_compact;\n \n \tstruct reftable_write_options config;\n \ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 351e35bd86d..4fec823f14f 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -325,7 +325,7 @@ static void test_reftable_stack_transaction_api_performs_auto_compaction(void)\n \t\t * we can ensure that we indeed honor this setting and have\n \t\t * better control over when exactly auto compaction runs.\n \t\t */\n-\t\tst->disable_auto_compact = i != n;\n+\t\tst->config.disable_auto_compact = i != n;\n \n \t\terr = reftable_stack_new_addition(&add, st);\n \t\tEXPECT_ERR(err);\n@@ -497,6 +497,7 @@ static void test_reftable_stack_add(void)\n \tstruct reftable_write_options cfg = {\n \t\t.exact_log_message = 1,\n \t\t.default_permissions = 0660,\n+\t\t.disable_auto_compact = 1,\n \t};\n \tstruct reftable_stack *st = NULL;\n \tchar *dir = get_tmp_dir(__LINE__);\n@@ -508,7 +509,6 @@ static void test_reftable_stack_add(void)\n \n \terr = reftable_new_stack(&st, dir, cfg);\n \tEXPECT_ERR(err);\n-\tst->disable_auto_compact = 1;\n \n \tfor (i = 0; i < N; i++) {\n \t\tchar buf[256];\n@@ -935,7 +935,9 @@ static void test_empty_add(void)\n \n static void test_reftable_stack_auto_compaction(void)\n {\n-\tstruct reftable_write_options cfg = { 0 };\n+\tstruct reftable_write_options cfg = {\n+\t\t.disable_auto_compact = 1,\n+\t};\n \tstruct reftable_stack *st = NULL;\n \tchar *dir = get_tmp_dir(__LINE__);\n \n@@ -945,7 +947,6 @@ static void test_reftable_stack_auto_compaction(void)\n \terr = reftable_new_stack(&st, dir, cfg);\n \tEXPECT_ERR(err);\n \n-\tst->disable_auto_compact = 1; /* call manually below for coverage. */\n \tfor (i = 0; i < N; i++) {\n \t\tchar name[100];\n \t\tstruct reftable_ref_record ref = {\n@@ -994,7 +995,7 @@ static void test_reftable_stack_add_performs_auto_compaction(void)\n \t\t * we can ensure that we indeed honor this setting and have\n \t\t * better control over when exactly auto compaction runs.\n \t\t */\n-\t\tst->disable_auto_compact = i != n;\n+\t\tst->config.disable_auto_compact = i != n;\n \n \t\tstrbuf_reset(&refname);\n \t\tstrbuf_addf(&refname, \"branch-%04d\", i);\n-- \ngitgitgadget\n\n"},{"id":"492551","messageId":"c7bc734654009a5275ac0bf5297782a846f6ad01.1712593016.git.gitgitgadget@gmail.com","threadId":"61055","inReplyTo":"pull.1683.v6.git.1712593016.gitgitgadget@gmail.com","subject":"[PATCH v6 2/3] reftable/stack: add env to disable autocompaction","fromName":"Justin Tobler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-04-08T16:16:54Z","receivedAt":"2024-04-08T16:17:02Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"From: Justin Tobler \n\nIn future tests it will be neccesary to create repositories with a set\nnumber of tables. To make this easier, introduce the\n`GIT_TEST_REFTABLE_AUTOCOMPACTION` environment variable that, when set\nto false, disables autocompaction of reftables.\n\nSigned-off-by: Justin Tobler \n---\n refs/reftable-backend.c | 3 +++\n t/t0610-reftable-basics.sh | 21 +++++++++++++++++++++\n 2 files changed, 24 insertions(+)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 0bed6d2ab48..1cda48c5046 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -18,6 +18,7 @@\n #include \"../reftable/reftable-merged.h\"\n #include \"../setup.h\"\n #include \"../strmap.h\"\n+#include \"parse.h\"\n #include \"refs-internal.h\"\n \n /*\n@@ -247,6 +248,8 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \trefs->write_options.block_size = 4096;\n \trefs->write_options.hash_id = repo->hash_algo->format_id;\n \trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n+\trefs->write_options.disable_auto_compact =\n+\t\t!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1);\n \n \t/*\n \t * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex 931d888bbbc..c9e10b34684 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -299,6 +299,27 @@ test_expect_success 'ref transaction: writes cause auto-compaction' '\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n '\n \n+test_expect_success 'ref transaction: env var disables compaction' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\n+\tstart=$(wc -l \n\nTo reduce the number of on-disk reftables, compaction is performed.\nContiguous tables with the same binary log value of size are grouped\ninto segments. The segment that has both the lowest binary log value and\ncontains more than one table is set as the starting point when\nidentifying the compaction segment.\n\nSince segments containing a single table are not initially considered\nfor compaction, if the table appended to the list does not match the\nprevious table log value, no compaction occurs for the new table. It is\ntherefore possible for unbounded growth of the table list. This can be\ndemonstrated by repeating the following sequence:\n\ngit branch -f foo\ngit branch -d foo\n\nEach operation results in a new table being written with no compaction\noccurring until a separate operation produces a table matching the\nprevious table log value.\n\nInstead, to avoid unbounded growth of the table list, the compaction\nstrategy is updated to ensure tables follow a geometric sequence after\neach operation by individually evaluating each table in reverse index\norder. This strategy results in a much simpler and more robust algorithm\ncompared to the previous one while also maintaining a minimal ordered\nset of tables on-disk.\n\nWhen creating 10 thousand references, the new strategy has no\nperformance impact:\n\nBenchmark 1: update-ref: create refs sequentially (revision = HEAD~)\n Time (mean ± σ): 26.516 s ± 0.047 s [User: 17.864 s, System: 8.491 s]\n Range (min … max): 26.447 s … 26.569 s 10 runs\n\nBenchmark 2: update-ref: create refs sequentially (revision = HEAD)\n Time (mean ± σ): 26.417 s ± 0.028 s [User: 17.738 s, System: 8.500 s]\n Range (min … max): 26.366 s … 26.444 s 10 runs\n\nSummary\n update-ref: create refs sequentially (revision = HEAD) ran\n 1.00 ± 0.00 times faster than update-ref: create refs sequentially (revision = HEAD~)\n\nSome tests in `t0610-reftable-basics.sh` assert the on-disk state of\ntables and are therefore updated to specify the correct new table count.\nSince compaction is more aggressive in ensuring tables maintain a\ngeometric sequence, the expected table count is reduced in these tests.\nIn `reftable/stack_test.c` tests related to `sizes_to_segments()` are\nremoved because the function is no longer needed. Also, the\n`test_suggest_compaction_segment()` test is updated to better showcase\nand reflect the new geometric compaction behavior.\n\nSigned-off-by: Justin Tobler \n---\n reftable/stack.c | 123 +++++++++++++++++++------------------\n reftable/stack.h | 3 -\n reftable/stack_test.c | 66 ++++----------------\n t/t0610-reftable-basics.sh | 50 ++++++++++-----\n 4 files changed, 111 insertions(+), 131 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 1a7cdad12c9..80266bcbab1 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1216,75 +1216,76 @@ static int segment_size(struct segment *s)\n \treturn s->end - s->start;\n }\n \n-int fastlog2(uint64_t sz)\n-{\n-\tint l = 0;\n-\tif (sz == 0)\n-\t\treturn 0;\n-\tfor (; sz; sz /= 2) {\n-\t\tl++;\n-\t}\n-\treturn l - 1;\n-}\n-\n-struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n)\n-{\n-\tstruct segment *segs = reftable_calloc(n, sizeof(*segs));\n-\tstruct segment cur = { 0 };\n-\tsize_t next = 0, i;\n-\n-\tif (n == 0) {\n-\t\t*seglen = 0;\n-\t\treturn segs;\n-\t}\n-\tfor (i = 0; i < n; i++) {\n-\t\tint log = fastlog2(sizes[i]);\n-\t\tif (cur.log != log && cur.bytes > 0) {\n-\t\t\tstruct segment fresh = {\n-\t\t\t\t.start = i,\n-\t\t\t};\n-\n-\t\t\tsegs[next++] = cur;\n-\t\t\tcur = fresh;\n-\t\t}\n-\n-\t\tcur.log = log;\n-\t\tcur.end = i + 1;\n-\t\tcur.bytes += sizes[i];\n-\t}\n-\tsegs[next++] = cur;\n-\t*seglen = next;\n-\treturn segs;\n-}\n-\n struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n {\n-\tstruct segment min_seg = {\n-\t\t.log = 64,\n-\t};\n-\tstruct segment *segs;\n-\tsize_t seglen = 0, i;\n-\n-\tsegs = sizes_to_segments(&seglen, sizes, n);\n-\tfor (i = 0; i < seglen; i++) {\n-\t\tif (segment_size(&segs[i]) == 1)\n-\t\t\tcontinue;\n+\tstruct segment seg = { 0 };\n+\tuint64_t bytes;\n+\tsize_t i;\n \n-\t\tif (segs[i].log < min_seg.log)\n-\t\t\tmin_seg = segs[i];\n-\t}\n+\t/*\n+\t * If there are no tables or only a single one then we don't have to\n+\t * compact anything. The sequence is geometric by definition already.\n+\t */\n+\tif (n <= 1)\n+\t\treturn seg;\n \n-\twhile (min_seg.start > 0) {\n-\t\tsize_t prev = min_seg.start - 1;\n-\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev]))\n+\t/*\n+\t * Find the ending table of the compaction segment needed to restore the\n+\t * geometric sequence. Note that the segment end is exclusive.\n+\t *\n+\t * To do so, we iterate backwards starting from the most recent table\n+\t * until a valid segment end is found. If the preceding table is smaller\n+\t * than the current table multiplied by the geometric factor (2), the\n+\t * compaction segment end has been identified.\n+\t *\n+\t * Tables after the ending point are not added to the byte count because\n+\t * they are already valid members of the geometric sequence. Due to the\n+\t * properties of a geometric sequence, it is not possible for the sum of\n+\t * these tables to exceed the value of the ending point table.\n+\t *\n+\t * Example table size sequence requiring no compaction:\n+\t * \t64, 32, 16, 8, 4, 2, 1\n+\t *\n+\t * Example table size sequence where compaction segment end is set to\n+\t * the last table. Since the segment end is exclusive, the last table is\n+\t * excluded during subsequent compaction and the table with size 3 is\n+\t * the final table included:\n+\t * \t64, 32, 16, 8, 4, 3, 1\n+\t */\n+\tfor (i = n - 1; i > 0; i--) {\n+\t\tif (sizes[i - 1] < sizes[i] * 2) {\n+\t\t\tseg.end = i + 1;\n+\t\t\tbytes = sizes[i];\n \t\t\tbreak;\n+\t\t}\n+\t}\n \n-\t\tmin_seg.start = prev;\n-\t\tmin_seg.bytes += sizes[prev];\n+\t/*\n+\t * Find the starting table of the compaction segment by iterating\n+\t * through the remaining tables and keeping track of the accumulated\n+\t * size of all tables seen from the segment end table. The previous\n+\t * table is compared to the accumulated size because the tables from the\n+\t * segment end are merged backwards recursively.\n+\t *\n+\t * Note that we keep iterating even after we have found the first\n+\t * starting point. This is because there may be tables in the stack\n+\t * preceding that first starting point which violate the geometric\n+\t * sequence.\n+\t *\n+\t * Example compaction segment start set to table with size 32:\n+\t * \t128, 32, 16, 8, 4, 3, 1\n+\t */\n+\tfor (; i > 0; i--) {\n+\t\tuint64_t curr = bytes;\n+\t\tbytes += sizes[i - 1];\n+\n+\t\tif (sizes[i - 1] < curr * 2) {\n+\t\t\tseg.start = i - 1;\n+\t\t\tseg.bytes = bytes;\n+\t\t}\n \t}\n \n-\treftable_free(segs);\n-\treturn min_seg;\n+\treturn seg;\n }\n \n static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\ndiff --git a/reftable/stack.h b/reftable/stack.h\nindex c862053025f..d43efa47607 100644\n--- a/reftable/stack.h\n+++ b/reftable/stack.h\n@@ -32,12 +32,9 @@ int read_lines(const char *filename, char ***lines);\n \n struct segment {\n \tsize_t start, end;\n-\tint log;\n \tuint64_t bytes;\n };\n \n-int fastlog2(uint64_t sz);\n-struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n);\n struct segment suggest_compaction_segment(uint64_t *sizes, size_t n);\n \n #endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 4fec823f14f..1df3ffce526 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -770,59 +770,13 @@ static void test_reftable_stack_hash_id(void)\n \tclear_dir(dir);\n }\n \n-static void test_log2(void)\n-{\n-\tEXPECT(1 == fastlog2(3));\n-\tEXPECT(2 == fastlog2(4));\n-\tEXPECT(2 == fastlog2(5));\n-}\n-\n-static void test_sizes_to_segments(void)\n-{\n-\tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n-\t/* .................0 1 2 3 4 5 */\n-\n-\tsize_t seglen = 0;\n-\tstruct segment *segs =\n-\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n-\tEXPECT(segs[2].log == 3);\n-\tEXPECT(segs[2].start == 5);\n-\tEXPECT(segs[2].end == 6);\n-\n-\tEXPECT(segs[1].log == 2);\n-\tEXPECT(segs[1].start == 2);\n-\tEXPECT(segs[1].end == 5);\n-\treftable_free(segs);\n-}\n-\n-static void test_sizes_to_segments_empty(void)\n-{\n-\tsize_t seglen = 0;\n-\tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n-\tEXPECT(seglen == 0);\n-\treftable_free(segs);\n-}\n-\n-static void test_sizes_to_segments_all_equal(void)\n-{\n-\tuint64_t sizes[] = { 5, 5 };\n-\tsize_t seglen = 0;\n-\tstruct segment *segs =\n-\t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n-\tEXPECT(seglen == 1);\n-\tEXPECT(segs[0].start == 0);\n-\tEXPECT(segs[0].end == 2);\n-\treftable_free(segs);\n-}\n-\n static void test_suggest_compaction_segment(void)\n {\n-\tuint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };\n-\t/* .................0 1 2 3 4 5 6 */\n+\tuint64_t sizes[] = { 512, 64, 17, 16, 9, 9, 9, 16, 2, 16 };\n \tstruct segment min =\n \t\tsuggest_compaction_segment(sizes, ARRAY_SIZE(sizes));\n-\tEXPECT(min.start == 2);\n-\tEXPECT(min.end == 7);\n+\tEXPECT(min.start == 1);\n+\tEXPECT(min.end == 10);\n }\n \n static void test_suggest_compaction_segment_nothing(void)\n@@ -933,6 +887,16 @@ static void test_empty_add(void)\n \treftable_stack_destroy(st2);\n }\n \n+static int fastlog2(uint64_t sz)\n+{\n+\tint l = 0;\n+\tif (sz == 0)\n+\t\treturn 0;\n+\tfor (; sz; sz /= 2)\n+\t\tl++;\n+\treturn l - 1;\n+}\n+\n static void test_reftable_stack_auto_compaction(void)\n {\n \tstruct reftable_write_options cfg = {\n@@ -1122,7 +1086,6 @@ static void test_reftable_stack_compaction_concurrent_clean(void)\n int stack_test_main(int argc, const char *argv[])\n {\n \tRUN_TEST(test_empty_add);\n-\tRUN_TEST(test_log2);\n \tRUN_TEST(test_names_equal);\n \tRUN_TEST(test_parse_names);\n \tRUN_TEST(test_read_file);\n@@ -1143,9 +1106,6 @@ int stack_test_main(int argc, const char *argv[])\n \tRUN_TEST(test_reftable_stack_update_index_check);\n \tRUN_TEST(test_reftable_stack_uptodate);\n \tRUN_TEST(test_reftable_stack_validate_refname);\n-\tRUN_TEST(test_sizes_to_segments);\n-\tRUN_TEST(test_sizes_to_segments_all_equal);\n-\tRUN_TEST(test_sizes_to_segments_empty);\n \tRUN_TEST(test_suggest_compaction_segment);\n \tRUN_TEST(test_suggest_compaction_segment_nothing);\n \treturn 0;\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex c9e10b34684..8eec093788d 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -293,7 +293,7 @@ test_expect_success 'ref transaction: writes cause auto-compaction' '\n \ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n \ttest_commit -C repo --no-tag A &&\n-\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n \ttest_commit -C repo --no-tag B &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n@@ -320,6 +320,19 @@ test_expect_success 'ref transaction: env var disables compaction' '\n \ttest_line_count -lt $expected repo/.git/reftable/tables.list\n '\n \n+test_expect_success 'ref transaction: alternating table sizes are compacted' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\n+\tgit init repo &&\n+\ttest_commit -C repo A &&\n+\tfor i in $(test_seq 5)\n+\tdo\n+\t\tgit -C repo branch -f foo &&\n+\t\tgit -C repo branch -d foo || return 1\n+\tdone &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list\n+'\n+\n check_fsync_events () {\n \tlocal trace=\"$1\" &&\n \tshift &&\n@@ -345,7 +358,7 @@ test_expect_success 'ref transaction: writes are synced' '\n \t\tgit -C repo -c core.fsync=reference \\\n \t\t-c core.fsyncMethod=fsync update-ref refs/heads/branch HEAD &&\n \tcheck_fsync_events trace2.txt <<-EOF\n-\t\"name\":\"hardware-flush\",\"count\":2\n+\t\"name\":\"hardware-flush\",\"count\":4\n \tEOF\n '\n \n@@ -377,7 +390,7 @@ test_expect_success 'ref transaction: fails gracefully when auto compaction fail\n \t\t\tdone ||\n \t\t\texit 1\n \t\tdone &&\n-\t\ttest_line_count = 13 .git/reftable/tables.list\n+\t\ttest_line_count = 10 .git/reftable/tables.list\n \t)\n '\n \n@@ -387,8 +400,8 @@ test_expect_success 'pack-refs: compacts tables' '\n \n \ttest_commit -C repo A &&\n \tls -1 repo/.git/reftable >table-files &&\n-\ttest_line_count = 4 table-files &&\n-\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 3 table-files &&\n+\ttest_line_count = 2 repo/.git/reftable/tables.list &&\n \n \tgit -C repo pack-refs &&\n \tls -1 repo/.git/reftable >table-files &&\n@@ -429,7 +442,7 @@ test_expect_success \"$command: auto compaction\" '\n \t\t# The tables should have been auto-compacted, and thus auto\n \t\t# compaction should not have to do anything.\n \t\tls -1 .git/reftable >tables-expect &&\n-\t\ttest_line_count = 4 tables-expect &&\n+\t\ttest_line_count = 3 tables-expect &&\n \t\tgit $command --auto &&\n \t\tls -1 .git/reftable >tables-actual &&\n \t\ttest_cmp tables-expect tables-actual &&\n@@ -447,7 +460,7 @@ test_expect_success \"$command: auto compaction\" '\n \t\tgit branch B &&\n \t\tgit branch C &&\n \t\trm .git/reftable/*.lock &&\n-\t\ttest_line_count = 5 .git/reftable/tables.list &&\n+\t\ttest_line_count = 4 .git/reftable/tables.list &&\n \n \t\tgit $command --auto &&\n \t\ttest_line_count = 1 .git/reftable/tables.list\n@@ -479,7 +492,7 @@ do\n \t\t\tumask $umask &&\n \t\t\tgit init --shared=true repo &&\n \t\t\ttest_commit -C repo A &&\n-\t\t\ttest_line_count = 3 repo/.git/reftable/tables.list\n+\t\t\ttest_line_count = 2 repo/.git/reftable/tables.list\n \t\t) &&\n \t\tgit -C repo pack-refs &&\n \t\ttest_expect_perms \"-rw-rw-r--\" repo/.git/reftable/tables.list &&\n@@ -847,12 +860,16 @@ test_expect_success 'worktree: pack-refs in main repo packs main refs' '\n \ttest_when_finished \"rm -rf repo worktree\" &&\n \tgit init repo &&\n \ttest_commit -C repo A &&\n+\n+\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n \tgit -C repo worktree add ../worktree &&\n+\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n+\tgit -C worktree update-ref refs/worktree/per-worktree HEAD &&\n \n-\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n \tgit -C repo pack-refs &&\n-\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list\n '\n \n@@ -860,13 +877,17 @@ test_expect_success 'worktree: pack-refs in worktree packs worktree refs' '\n \ttest_when_finished \"rm -rf repo worktree\" &&\n \tgit init repo &&\n \ttest_commit -C repo A &&\n+\n+\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n \tgit -C repo worktree add ../worktree &&\n+\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n+\tgit -C worktree update-ref refs/worktree/per-worktree HEAD &&\n \n-\ttest_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 4 repo/.git/reftable/tables.list &&\n+\ttest_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list &&\n+\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n \tgit -C worktree pack-refs &&\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n-\ttest_line_count = 4 repo/.git/reftable/tables.list\n+\ttest_line_count = 3 repo/.git/reftable/tables.list\n '\n \n test_expect_success 'worktree: creating shared ref updates main stack' '\n@@ -880,6 +901,7 @@ test_expect_success 'worktree: creating shared ref updates main stack' '\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 1 repo/.git/reftable/tables.list &&\n \n+\tGIT_TEST_REFTABLE_AUTOCOMPACTION=false \\\n \tgit -C worktree update-ref refs/heads/shared HEAD &&\n \ttest_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list &&\n \ttest_line_count = 2 repo/.git/reftable/tables.list\n-- \ngitgitgadget\n"},{"id":"492554","messageId":"n32hb7imvnr7jfl3nh27fhmcpstwm65oznxmqky3g4gcleu6ao@jiiq4avcqrpm","threadId":"61055","inReplyTo":"ZhOKxLCWxp-lOyVh@tanuki","subject":"Re: [PATCH v5 0/3] reftable/stack: use geometric table compaction","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-04-08T16:17:20Z","receivedAt":"2024-04-08T16:18:20Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/04/08 08:12AM, Patrick Steinhardt wrote:\n> On Thu, Apr 04, 2024 at 06:29:26PM +0000, Justin Tobler via GitGitGadget wrote:\n> > Hello again,\n> > \n> > This is the fifth version my patch series that refactors the reftable\n> > compaction strategy to instead follow a geometric sequence. Changes compared\n> > to v4:\n> > \n> > * To fix some failing tests and conflicts, this patch series now depends on\n> > the ps/pack-refs-auto series which is currently in next.\n> > * Lifted the GIT_TEST_REFTABLE_AUTOCOMPACTION env out of the reftable\n> > library and into the reftable backend code.\n> > \n> > Thanks for taking a look!\n> > \n> > -Justin\n> \n> I've added two additional nits which you may or may not want to address.\n> But overall this patch series looks good to me. Thanks!\n\nThanks Patrick! I've updated per your suggestions in the next patch\nseries version.\n\n-Justin\n"},{"id":"492555","messageId":"xmqqjzl7eskr.fsf@gitster.g","threadId":"61055","inReplyTo":"ZhOK262a5WCxS9MN@tanuki","subject":"Re: [PATCH v5 2/3] reftable/stack: add env to disable autocompaction","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-08T16:18:28Z","receivedAt":"2024-04-08T16:18:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> @@ -248,6 +249,9 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n>> \trefs->write_options.hash_id = repo->hash_algo->format_id;\n>> \trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n>> \n>> +\tif (!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1))\n>> +\t\trefs->write_options.disable_auto_compact = 1;\n>> +\n>\n> This could be simplified to:\n>\n> ```\n> refs->write_options.disable_auto_compact =\n> !git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1);\n> ```\n\nI presume that the .disable_auto_compact member is off initially,\ngiven that this is inside reftable_be_init(), but your rewrite makes\nit easier on readers, as they do not have to know what value the\nmember originally has at this point in the flow. So even though it\nreplaces two lines of code with another two lines of code, it does\ncount as a valuable simplification at the conceptual level.\n\nThanks.\n\n"},{"id":"492557","messageId":"ZhQZaI3JcShhYwXn@tanuki","threadId":"61055","inReplyTo":"pull.1683.v6.git.1712593016.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 0/3] reftable/stack: use geometric table compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T16:20:56Z","receivedAt":"2024-04-08T16:21:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Apr 08, 2024 at 04:16:52PM +0000, Justin Tobler via GitGitGadget wrote:\n> Hello again,\n> \n> This is the sixth version my patch series that refactors the reftable\n> compaction strategy to instead follow a geometric sequence. Changes compared\n> to v5:\n> \n> * Reworded commit message to more clearly explain that the already existing\n> configuration to disable auto-compaction is being exposed to callers of\n> the library.\n> * Simplified expression to set the disable_auto_compact configuration.\n> \n> Thanks for taking a look!\n\nThanks, this version looks good to me!\n\nPatrick\n\n> -Justin\n> \n> Justin Tobler (3):\n> reftable/stack: expose option to disable auto-compaction\n> reftable/stack: add env to disable autocompaction\n> reftable/stack: use geometric table compaction\n> \n> refs/reftable-backend.c | 3 +\n> reftable/reftable-writer.h | 3 +\n> reftable/stack.c | 125 +++++++++++++++++++------------------\n> reftable/stack.h | 4 --\n> reftable/stack_test.c | 77 ++++++-----------------\n> t/t0610-reftable-basics.sh | 71 ++++++++++++++++-----\n> 6 files changed, 145 insertions(+), 138 deletions(-)\n> \n> \n> base-commit: 4b32163adf4863c6df3bb6b43540fa2ca3494e28\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1683%2Fjltobler%2Fjt%2Freftable-geometric-compaction-v6\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1683/jltobler/jt/reftable-geometric-compaction-v6\n> Pull-Request: https://github.com/gitgitgadget/git/pull/1683\n> \n> Range-diff vs v5:\n> \n> 1: a7011dbc6aa ! 1: 9c8f6b336ec reftable/stack: allow disabling of auto-compaction\n> @@ Metadata\n> Author: Justin Tobler \n> \n> ## Commit message ##\n> - reftable/stack: allow disabling of auto-compaction\n> + reftable/stack: expose option to disable auto-compaction\n> +\n> + The reftable stack already has a variable to configure whether or not to\n> + run auto-compaction, but it is inaccessible to users of the library.\n> + There exist use cases where a caller may want to have more control over\n> + auto-compaction.\n> \n> Move the `disable_auto_compact` option into `reftable_write_options` to\n> - allow a stack to be configured with auto-compaction disabled. In a\n> - subsequent commit, this is used to disable auto-compaction when a\n> - specific environment variable is set.\n> + allow external callers to disable auto-compaction. This will be used in\n> + a subsequent commit.\n> \n> Signed-off-by: Justin Tobler \n> \n> 2: 7c4fe0e9ec5 ! 2: c7bc7346540 reftable/stack: add env to disable autocompaction\n> @@ refs/reftable-backend.c\n> \n> /*\n> @@ refs/reftable-backend.c: static struct ref_store *reftable_be_init(struct repository *repo,\n> + \trefs->write_options.block_size = 4096;\n> \trefs->write_options.hash_id = repo->hash_algo->format_id;\n> \trefs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);\n> ++\trefs->write_options.disable_auto_compact =\n> ++\t\t!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1);\n> \n> -+\tif (!git_env_bool(\"GIT_TEST_REFTABLE_AUTOCOMPACTION\", 1))\n> -+\t\trefs->write_options.disable_auto_compact = 1;\n> -+\n> \t/*\n> \t * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.\n> - \t * This stack contains both the shared and the main worktree refs.\n> \n> ## t/t0610-reftable-basics.sh ##\n> @@ t/t0610-reftable-basics.sh: test_expect_success 'ref transaction: writes cause auto-compaction' '\n> 3: 8f124acf0f8 = 3: d75494a88b0 reftable/stack: use geometric table compaction\n> \n> -- \n> gitgitgadget\n"},{"id":"492577","messageId":"xmqqmsq3bre4.fsf@gitster.g","threadId":"61055","inReplyTo":"ZhQZaI3JcShhYwXn@tanuki","subject":"Re: [PATCH v6 0/3] reftable/stack: use geometric table compaction","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-08T19:12:19Z","receivedAt":"2024-04-08T19:12:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Mon, Apr 08, 2024 at 04:16:52PM +0000, Justin Tobler via GitGitGadget wrote:\n>> Hello again,\n>> \n>> This is the sixth version my patch series that refactors the reftable\n>> compaction strategy to instead follow a geometric sequence. Changes compared\n>> to v5:\n>> \n>> * Reworded commit message to more clearly explain that the already existing\n>> configuration to disable auto-compaction is being exposed to callers of\n>> the library.\n>> * Simplified expression to set the disable_auto_compact configuration.\n>> \n>> Thanks for taking a look!\n>\n> Thanks, this version looks good to me!\n\nWill queue. I'll mark it for 'next' after I take a brief look for\nmyself.\n\nThanks, both. \n"}]}