{"thread":{"id":"60822","subject":"[PATCH 0/9] reftable: code style improvements","startedAt":"2024-01-31T08:01:06Z","lastAt":"2024-02-06T09:10:43Z","messageCount":40,"participants":["Patrick Steinhardt","Eric Sunshine","Junio C Hamano","Toon Claes","Karthik Nayak"],"isPatch":true,"patchVersion":1,"patchTotal":9},"messages":[{"id":"487630","messageId":"cover.1706687982.git.ps@pks.im","threadId":"60822","inReplyTo":null,"subject":"[PATCH 0/9] reftable: code style improvements","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-01-31T08:00:59Z","receivedAt":"2024-01-31T08:01:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series contains a more or less random assortments of code\nstyle improvements for the reftable library. The patch series is\nstructured as follows:\n\n - Patches 1-2 introduce macros to allocate and grow arrays. These are\n basically the same as our ALLOC_GROW and ALLOC_ARRAY macros, but\n specific to the reftable library.\n\n - Patches 3-6 refactor some code to use `size_t` to index into array\n slices instead of `int`.\n\n - Patches 7-9 are some more debatable cleanups that helped me\n personally to make sense of the code better.\n\nI've been a bit hesitant to send out these patches as they may be\nconsidered \"noise\", especially the last three ones. But the reftable\nlibrary feels quite alien in our codebase, which does increase the\nmental overhead at least for me when reading its ccode.\n\nI think it should be a goal of ours to align it closer to our normal\ncoding style, which will of course end up in quite a bit of churn over\ntime. A different approach would be to do these refactorings while\nalready touching the code anyway due to other reasons. But I found\nmyself doing the same refactorings over and over again in different\ncontexts for patch series I ultimately didn't end up sending, so I'd\nreally rather want to get those out of my way.\n\nAnyway, please let me know how you feel about patch series like these.\n\nPatrick\n\nPatrick Steinhardt (9):\n reftable: introduce macros to grow arrays\n reftable: introduce macros to allocate arrays\n reftable/stack: fix parameter validation when compacting range\n reftable/stack: index segments with `size_t`\n reftable/stack: use `size_t` to track stack slices during compaction\n reftable/stack: use `size_t` to track stack length\n reftable/merged: refactor seeking of records\n reftable/merged: refactor initialization of iterators\n reftable/record: improve semantics when initializing records\n\n reftable/basics.c | 15 ++--\n reftable/basics.h | 17 ++++-\n reftable/block.c | 25 +++---\n reftable/block_test.c | 2 +-\n reftable/blocksource.c | 4 +-\n reftable/iter.c | 3 +-\n reftable/merged.c | 100 +++++++++++-------------\n reftable/merged_test.c | 52 ++++++-------\n reftable/pq.c | 8 +-\n reftable/publicbasics.c | 3 +-\n reftable/reader.c | 12 ++-\n reftable/readwrite_test.c | 8 +-\n reftable/record.c | 43 +++--------\n reftable/record.h | 10 +--\n reftable/record_test.c | 8 +-\n reftable/refname.c | 4 +-\n reftable/reftable-merged.h | 2 +-\n reftable/stack.c | 151 +++++++++++++++++--------------------\n reftable/stack.h | 6 +-\n reftable/stack_test.c | 7 +-\n reftable/tree.c | 4 +-\n reftable/writer.c | 21 ++----\n 22 files changed, 221 insertions(+), 284 deletions(-)\n\n\nbase-commit: bc7ee2e5e16f0d1e710ef8fab3db59ab11f2bbe7\n-- \n2.43.GIT\n\n"},{"id":"487631","messageId":"0597e6944a1a65720d050f47bc82766d5bcf860b.1706687982.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706687982.git.ps@pks.im","subject":"[PATCH 1/9] reftable: introduce macros to grow arrays","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-01-31T08:01:03Z","receivedAt":"2024-01-31T08:01:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Throughout the reftable library we have many cases where we need to grow\narrays. In order to avoid too many reallocations, we roughly double the\ncapacity of the array on each iteration. The resulting code pattern is\nthus duplicated across many sites.\n\nWe have similar patterns in our main codebase, which is why we have\neventually introduced an `ALLOC_GROW()` macro to abstract it away and\navoid some code duplication. We cannot easily reuse this macro here\nthough because `ALLOC_GROW()` uses `REALLOC_ARRAY()`, which in turn will\ncall realloc(3P) to grow the array. The reftable code is structured as a\nlibrary though (even if the boundaries are fuzzy), and one property this\nbrings with it is that it is possible to plug in your own allocators. So\ninstead of using realloc(3P), we need to use `reftable_realloc()` that\nknows to use the user-provided implementation.\n\nSo let's introduce two new macros `REFTABLE_REALLOC_ARRAY()` and\n`REFTABLE_ALLOC_GROW()` that mirror what we do in our main codebase,\nwith two modifications:\n\n - They use `reftable_realloc()`, as explained above.\n\n - They use a different growth factor of `2 * cap + 1` instead of `(cap\n + 16) * 3 / 2`.\n\nThe second change is because we know a bit more about the allocation\npatterns in the reftable library. For In most cases, we end up only having a\nsingle item in the array, so the initial capacity that our global growth\nfactor uses (which is 24), significantly overallocates in a lot of code\npaths. This effect is indeed measurable:\n\n Benchmark 1: update-ref: create many refs (growth factor = 2 * cap + 1)\n Time (mean ± σ): 4.834 s ± 0.020 s [User: 2.219 s, System: 2.614 s]\n Range (min … max): 4.793 s … 4.871 s 20 runs\n\n Benchmark 2: update-ref: create many refs (growth factor = (cap + 16) * 3 + 2)\n Time (mean ± σ): 4.933 s ± 0.021 s [User: 2.325 s, System: 2.607 s]\n Range (min … max): 4.889 s … 4.962 s 20 runs\n\n Summary\n update-ref: create many refs (growth factor = 2 * cap + 1) ran\n 1.02 ± 0.01 times faster than update-ref: create many refs (growth factor = (cap + 16) * 3 + 2)\n\nConvert the reftable library to use these new macros.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 8 ++------\n reftable/basics.h | 11 +++++++++++\n reftable/block.c | 7 +------\n reftable/merged_test.c | 20 ++++++--------------\n reftable/pq.c | 8 ++------\n reftable/stack.c | 29 ++++++++++++-----------------\n reftable/writer.c | 14 ++------------\n 7 files changed, 36 insertions(+), 61 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex f761e48028..af9004cec2 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -89,17 +89,13 @@ void parse_names(char *buf, int size, char ***namesp)\n \t\t\tnext = end;\n \t\t}\n \t\tif (p < next) {\n-\t\t\tif (names_len == names_cap) {\n-\t\t\t\tnames_cap = 2 * names_cap + 1;\n-\t\t\t\tnames = reftable_realloc(\n-\t\t\t\t\tnames, names_cap * sizeof(*names));\n-\t\t\t}\n+\t\t\tREFTABLE_ALLOC_GROW(names, names_len + 1, names_cap);\n \t\t\tnames[names_len++] = xstrdup(p);\n \t\t}\n \t\tp = next + 1;\n \t}\n \n-\tnames = reftable_realloc(names, (names_len + 1) * sizeof(*names));\n+\tREFTABLE_REALLOC_ARRAY(names, names_len + 1);\n \tnames[names_len] = NULL;\n \t*namesp = names;\n }\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 096b36862b..2f855cd724 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -53,6 +53,17 @@ void *reftable_realloc(void *p, size_t sz);\n void reftable_free(void *p);\n void *reftable_calloc(size_t sz);\n \n+#define REFTABLE_REALLOC_ARRAY(x, alloc) (x) = reftable_realloc((x), st_mult(sizeof(*(x)), (alloc)))\n+#define REFTABLE_ALLOC_GROW(x, nr, alloc) \\\n+\tdo { \\\n+\t\tif ((nr) > alloc) { \\\n+\t\t\talloc = 2 * (alloc) + 1; \\\n+\t\t\tif (alloc < (nr)) \\\n+\t\t\t\talloc = (nr); \\\n+\t\t\tREFTABLE_REALLOC_ARRAY(x, alloc); \\\n+\t\t} \\\n+\t} while (0)\n+\n /* Find the longest shared prefix size of `a` and `b` */\n struct strbuf;\n int common_prefix_size(struct strbuf *a, struct strbuf *b);\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 1df3d8a0f0..6952d0facf 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -51,12 +51,7 @@ static int block_writer_register_restart(struct block_writer *w, int n,\n \tif (2 + 3 * rlen + n > w->block_size - w->next)\n \t\treturn -1;\n \tif (is_restart) {\n-\t\tif (w->restart_len == w->restart_cap) {\n-\t\t\tw->restart_cap = w->restart_cap * 2 + 1;\n-\t\t\tw->restarts = reftable_realloc(\n-\t\t\t\tw->restarts, sizeof(uint32_t) * w->restart_cap);\n-\t\t}\n-\n+\t\tREFTABLE_ALLOC_GROW(w->restarts, w->restart_len + 1, w->restart_cap);\n \t\tw->restarts[w->restart_len++] = w->next;\n \t}\n \ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nindex 46908f738f..e05351e035 100644\n--- a/reftable/merged_test.c\n+++ b/reftable/merged_test.c\n@@ -231,14 +231,10 @@ static void test_merged(void)\n \twhile (len < 100) { /* cap loops/recursion. */\n \t\tstruct reftable_ref_record ref = { NULL };\n \t\tint err = reftable_iterator_next_ref(&it, &ref);\n-\t\tif (err > 0) {\n+\t\tif (err > 0)\n \t\t\tbreak;\n-\t\t}\n-\t\tif (len == cap) {\n-\t\t\tcap = 2 * cap + 1;\n-\t\t\tout = reftable_realloc(\n-\t\t\t\tout, sizeof(struct reftable_ref_record) * cap);\n-\t\t}\n+\n+\t\tREFTABLE_ALLOC_GROW(out, len + 1, cap);\n \t\tout[len++] = ref;\n \t}\n \treftable_iterator_destroy(&it);\n@@ -368,14 +364,10 @@ static void test_merged_logs(void)\n \twhile (len < 100) { /* cap loops/recursion. */\n \t\tstruct reftable_log_record log = { NULL };\n \t\tint err = reftable_iterator_next_log(&it, &log);\n-\t\tif (err > 0) {\n+\t\tif (err > 0)\n \t\t\tbreak;\n-\t\t}\n-\t\tif (len == cap) {\n-\t\t\tcap = 2 * cap + 1;\n-\t\t\tout = reftable_realloc(\n-\t\t\t\tout, sizeof(struct reftable_log_record) * cap);\n-\t\t}\n+\n+\t\tREFTABLE_ALLOC_GROW(out, len + 1, cap);\n \t\tout[len++] = log;\n \t}\n \treftable_iterator_destroy(&it);\ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex dcefeb793a..2461daf5ff 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -75,13 +75,9 @@ void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry\n {\n \tint i = 0;\n \n-\tif (pq->len == pq->cap) {\n-\t\tpq->cap = 2 * pq->cap + 1;\n-\t\tpq->heap = reftable_realloc(pq->heap,\n-\t\t\t\t\t pq->cap * sizeof(struct pq_entry));\n-\t}\n-\n+\tREFTABLE_ALLOC_GROW(pq->heap, pq->len + 1, pq->cap);\n \tpq->heap[pq->len++] = *e;\n+\n \ti = pq->len - 1;\n \twhile (i > 0) {\n \t\tint j = (i - 1) / 2;\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex bf3869ce70..1dfab99e96 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -551,7 +551,7 @@ struct reftable_addition {\n \tstruct reftable_stack *stack;\n \n \tchar **new_tables;\n-\tint new_tables_len;\n+\tsize_t new_tables_len, new_tables_cap;\n \tuint64_t next_update_index;\n };\n \n@@ -602,8 +602,9 @@ static int reftable_stack_init_addition(struct reftable_addition *add,\n \n static void reftable_addition_close(struct reftable_addition *add)\n {\n-\tint i = 0;\n \tstruct strbuf nm = STRBUF_INIT;\n+\tsize_t i;\n+\n \tfor (i = 0; i < add->new_tables_len; i++) {\n \t\tstack_filename(&nm, add->stack, add->new_tables[i]);\n \t\tunlink(nm.buf);\n@@ -613,6 +614,7 @@ static void reftable_addition_close(struct reftable_addition *add)\n \treftable_free(add->new_tables);\n \tadd->new_tables = NULL;\n \tadd->new_tables_len = 0;\n+\tadd->new_tables_cap = 0;\n \n \tdelete_tempfile(&add->lock_file);\n \tstrbuf_release(&nm);\n@@ -631,8 +633,8 @@ int reftable_addition_commit(struct reftable_addition *add)\n {\n \tstruct strbuf table_list = STRBUF_INIT;\n \tint lock_file_fd = get_tempfile_fd(add->lock_file);\n-\tint i = 0;\n \tint err = 0;\n+\tsize_t i;\n \n \tif (add->new_tables_len == 0)\n \t\tgoto done;\n@@ -660,12 +662,12 @@ int reftable_addition_commit(struct reftable_addition *add)\n \t}\n \n \t/* success, no more state to clean up. */\n-\tfor (i = 0; i < add->new_tables_len; i++) {\n+\tfor (i = 0; i < add->new_tables_len; i++)\n \t\treftable_free(add->new_tables[i]);\n-\t}\n \treftable_free(add->new_tables);\n \tadd->new_tables = NULL;\n \tadd->new_tables_len = 0;\n+\tadd->new_tables_cap = 0;\n \n \terr = reftable_stack_reload_maybe_reuse(add->stack, 1);\n \tif (err)\n@@ -792,11 +794,9 @@ int reftable_addition_add(struct reftable_addition *add,\n \t\tgoto done;\n \t}\n \n-\tadd->new_tables = reftable_realloc(add->new_tables,\n-\t\t\t\t\t sizeof(*add->new_tables) *\n-\t\t\t\t\t\t (add->new_tables_len + 1));\n-\tadd->new_tables[add->new_tables_len] = strbuf_detach(&next_name, NULL);\n-\tadd->new_tables_len++;\n+\tREFTABLE_ALLOC_GROW(add->new_tables, add->new_tables_len + 1,\n+\t\t\t add->new_tables_cap);\n+\tadd->new_tables[add->new_tables_len++] = strbuf_detach(&next_name, NULL);\n done:\n \tif (tab_fd > 0) {\n \t\tclose(tab_fd);\n@@ -1367,17 +1367,12 @@ static int stack_check_addition(struct reftable_stack *st,\n \twhile (1) {\n \t\tstruct reftable_ref_record ref = { NULL };\n \t\terr = reftable_iterator_next_ref(&it, &ref);\n-\t\tif (err > 0) {\n+\t\tif (err > 0)\n \t\t\tbreak;\n-\t\t}\n \t\tif (err < 0)\n \t\t\tgoto done;\n \n-\t\tif (len >= cap) {\n-\t\t\tcap = 2 * cap + 1;\n-\t\t\trefs = reftable_realloc(refs, cap * sizeof(refs[0]));\n-\t\t}\n-\n+\t\tREFTABLE_ALLOC_GROW(refs, len + 1, cap);\n \t\trefs[len++] = ref;\n \t}\n \ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex ee4590e20f..4483bb21c3 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -200,12 +200,7 @@ static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n \t\treturn;\n \t}\n \n-\tif (key->offset_len == key->offset_cap) {\n-\t\tkey->offset_cap = 2 * key->offset_cap + 1;\n-\t\tkey->offsets = reftable_realloc(\n-\t\t\tkey->offsets, sizeof(uint64_t) * key->offset_cap);\n-\t}\n-\n+\tREFTABLE_ALLOC_GROW(key->offsets, key->offset_len + 1, key->offset_cap);\n \tkey->offsets[key->offset_len++] = off;\n }\n \n@@ -674,12 +669,7 @@ static int writer_flush_nonempty_block(struct reftable_writer *w)\n \tif (err < 0)\n \t\treturn err;\n \n-\tif (w->index_cap == w->index_len) {\n-\t\tw->index_cap = 2 * w->index_cap + 1;\n-\t\tw->index = reftable_realloc(\n-\t\t\tw->index,\n-\t\t\tsizeof(struct reftable_index_record) * w->index_cap);\n-\t}\n+\tREFTABLE_ALLOC_GROW(w->index, w->index_len + 1, w->index_cap);\n \n \tir.offset = w->next;\n \tstrbuf_reset(&ir.last_key);\n-- \n2.43.GIT\n\n"},{"id":"487632","messageId":"eee6580c84f733ac195585267570944da28d5b7d.1706687982.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706687982.git.ps@pks.im","subject":"[PATCH 2/9] reftable: introduce macros to allocate arrays","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-01-31T08:01:07Z","receivedAt":"2024-01-31T08:01:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Similar to the preceding commit, let's carry over macros to allocate\narrays with `REFTABLE_ALLOC_ARRAY()` and `REFTABLE_CALLOC_ARRAY()`. This\nrequires us to change the signature of `reftable_calloc()`, which only\ntakes a single argument right now and thus puts the burden on the caller\nto calculate the final array's size. This is a net improvement though as\nit means that we can now provide proper overflow checks when multiplying\nthe array size with the member size.\n\nConvert callsites of `reftable_calloc()` to the new signature, using the\nnew macros where possible.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 4 +++-\n reftable/block_test.c | 2 +-\n reftable/blocksource.c | 4 ++--\n reftable/iter.c | 3 +--\n reftable/merged.c | 4 ++--\n reftable/merged_test.c | 22 +++++++++++++---------\n reftable/publicbasics.c | 3 ++-\n reftable/reader.c | 8 +++-----\n reftable/readwrite_test.c | 8 +++++---\n reftable/record_test.c | 4 ++--\n reftable/refname.c | 4 ++--\n reftable/stack.c | 26 ++++++++++++--------------\n reftable/tree.c | 4 ++--\n reftable/writer.c | 7 +++----\n 14 files changed, 53 insertions(+), 50 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 2f855cd724..4c3ac963a3 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -51,8 +51,10 @@ int names_length(char **names);\n void *reftable_malloc(size_t sz);\n void *reftable_realloc(void *p, size_t sz);\n void reftable_free(void *p);\n-void *reftable_calloc(size_t sz);\n+void *reftable_calloc(size_t nelem, size_t elsize);\n \n+#define REFTABLE_ALLOC_ARRAY(x, alloc) (x) = reftable_malloc(st_mult(sizeof(*(x)), (alloc)))\n+#define REFTABLE_CALLOC_ARRAY(x, alloc) (x) = reftable_calloc((alloc), sizeof(*(x)))\n #define REFTABLE_REALLOC_ARRAY(x, alloc) (x) = reftable_realloc((x), st_mult(sizeof(*(x)), (alloc)))\n #define REFTABLE_ALLOC_GROW(x, nr, alloc) \\\n \tdo { \\\ndiff --git a/reftable/block_test.c b/reftable/block_test.c\nindex dedb05c7d8..e162c6e33f 100644\n--- a/reftable/block_test.c\n+++ b/reftable/block_test.c\n@@ -36,7 +36,7 @@ static void test_block_read_write(void)\n \tint j = 0;\n \tstruct strbuf want = STRBUF_INIT;\n \n-\tblock.data = reftable_calloc(block_size);\n+\tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock.source = malloc_block_source();\n \tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex 8c41e3c70f..eeed254ba9 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -29,7 +29,7 @@ static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n {\n \tstruct strbuf *b = v;\n \tassert(off + size <= b->len);\n-\tdest->data = reftable_calloc(size);\n+\tREFTABLE_CALLOC_ARRAY(dest->data, size);\n \tmemcpy(dest->data, b->buf + off, size);\n \tdest->len = size;\n \treturn size;\n@@ -132,7 +132,7 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,\n \t\treturn REFTABLE_IO_ERROR;\n \t}\n \n-\tp = reftable_calloc(sizeof(*p));\n+\tREFTABLE_CALLOC_ARRAY(p, 1);\n \tp->size = st.st_size;\n \tp->data = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tclose(fd);\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex a8d174c040..8b5ebf6183 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -160,8 +160,7 @@ int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n \t\t\t int oid_len, uint64_t *offsets, int offset_len)\n {\n \tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n-\tstruct indexed_table_ref_iter *itr =\n-\t\treftable_calloc(sizeof(struct indexed_table_ref_iter));\n+\tstruct indexed_table_ref_iter *itr = reftable_calloc(1, sizeof(*itr));\n \tint err = 0;\n \n \t*itr = empty;\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex c258ce953e..2031fd51b4 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -190,7 +190,7 @@ int reftable_new_merged_table(struct reftable_merged_table **dest,\n \t\t}\n \t}\n \n-\tm = reftable_calloc(sizeof(struct reftable_merged_table));\n+\tREFTABLE_CALLOC_ARRAY(m, 1);\n \tm->stack = stack;\n \tm->stack_len = n;\n \tm->min = first_min;\n@@ -240,7 +240,7 @@ static int merged_table_seek_record(struct reftable_merged_table *mt,\n \t\t\t\t struct reftable_record *rec)\n {\n \tstruct reftable_iterator *iters = reftable_calloc(\n-\t\tsizeof(struct reftable_iterator) * mt->stack_len);\n+\t\tmt->stack_len, sizeof(*iters));\n \tstruct merged_iter merged = {\n \t\t.stack = iters,\n \t\t.typ = reftable_record_type(rec),\ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nindex e05351e035..e233a9d581 100644\n--- a/reftable/merged_test.c\n+++ b/reftable/merged_test.c\n@@ -93,10 +93,12 @@ merged_table_from_records(struct reftable_ref_record **refs,\n \tint i = 0;\n \tstruct reftable_merged_table *mt = NULL;\n \tint err;\n-\tstruct reftable_table *tabs =\n-\t\treftable_calloc(n * sizeof(struct reftable_table));\n-\t*readers = reftable_calloc(n * sizeof(struct reftable_reader *));\n-\t*source = reftable_calloc(n * sizeof(**source));\n+\tstruct reftable_table *tabs;\n+\n+\tREFTABLE_CALLOC_ARRAY(tabs, n);\n+\tREFTABLE_CALLOC_ARRAY(*readers, n);\n+\tREFTABLE_CALLOC_ARRAY(*source, n);\n+\n \tfor (i = 0; i < n; i++) {\n \t\twrite_test_table(&buf[i], refs[i], sizes[i]);\n \t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n@@ -266,10 +268,12 @@ merged_table_from_log_records(struct reftable_log_record **logs,\n \tint i = 0;\n \tstruct reftable_merged_table *mt = NULL;\n \tint err;\n-\tstruct reftable_table *tabs =\n-\t\treftable_calloc(n * sizeof(struct reftable_table));\n-\t*readers = reftable_calloc(n * sizeof(struct reftable_reader *));\n-\t*source = reftable_calloc(n * sizeof(**source));\n+\tstruct reftable_table *tabs;\n+\n+\tREFTABLE_CALLOC_ARRAY(tabs, n);\n+\tREFTABLE_CALLOC_ARRAY(*readers, n);\n+\tREFTABLE_CALLOC_ARRAY(*source, n);\n+\n \tfor (i = 0; i < n; i++) {\n \t\twrite_test_log_table(&buf[i], logs[i], sizes[i], i + 1);\n \t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n@@ -412,7 +416,7 @@ static void test_default_write_opts(void)\n \t};\n \tint err;\n \tstruct reftable_block_source source = { NULL };\n-\tstruct reftable_table *tab = reftable_calloc(sizeof(*tab) * 1);\n+\tstruct reftable_table *tab = reftable_calloc(1, sizeof(*tab));\n \tuint32_t hash_id;\n \tstruct reftable_reader *rd = NULL;\n \tstruct reftable_merged_table *merged = NULL;\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\nindex bcb82530d6..44b84a125e 100644\n--- a/reftable/publicbasics.c\n+++ b/reftable/publicbasics.c\n@@ -37,8 +37,9 @@ void reftable_free(void *p)\n \t\tfree(p);\n }\n \n-void *reftable_calloc(size_t sz)\n+void *reftable_calloc(size_t nelem, size_t elsize)\n {\n+\tsize_t sz = st_mult(nelem, elsize);\n \tvoid *p = reftable_malloc(sz);\n \tmemset(p, 0, sz);\n \treturn p;\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 64dc366fb1..3e0de5e8ad 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -539,8 +539,7 @@ static int reader_seek_indexed(struct reftable_reader *r,\n \n \tif (err == 0) {\n \t\tstruct table_iter empty = TABLE_ITER_INIT;\n-\t\tstruct table_iter *malloced =\n-\t\t\treftable_calloc(sizeof(struct table_iter));\n+\t\tstruct table_iter *malloced = reftable_calloc(1, sizeof(*malloced));\n \t\t*malloced = empty;\n \t\ttable_iter_copy_from(malloced, &next);\n \t\titerator_from_table_iter(it, malloced);\n@@ -635,8 +634,7 @@ void reader_close(struct reftable_reader *r)\n int reftable_new_reader(struct reftable_reader **p,\n \t\t\tstruct reftable_block_source *src, char const *name)\n {\n-\tstruct reftable_reader *rd =\n-\t\treftable_calloc(sizeof(struct reftable_reader));\n+\tstruct reftable_reader *rd = reftable_calloc(1, sizeof(*rd));\n \tint err = init_reader(rd, src, name);\n \tif (err == 0) {\n \t\t*p = rd;\n@@ -711,7 +709,7 @@ static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n \t\t\t\t\t uint8_t *oid)\n {\n \tstruct table_iter ti_empty = TABLE_ITER_INIT;\n-\tstruct table_iter *ti = reftable_calloc(sizeof(struct table_iter));\n+\tstruct table_iter *ti = reftable_calloc(1, sizeof(*ti));\n \tstruct filtering_ref_iterator *filter = NULL;\n \tstruct filtering_ref_iterator empty = FILTERING_REF_ITERATOR_INIT;\n \tint oid_len = hash_size(r->hash_id);\ndiff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c\nindex b8a3224016..91ea7a10ef 100644\n--- a/reftable/readwrite_test.c\n+++ b/reftable/readwrite_test.c\n@@ -56,7 +56,9 @@ static void write_table(char ***names, struct strbuf *buf, int N,\n \tint i = 0, n;\n \tstruct reftable_log_record log = { NULL };\n \tconst struct reftable_stats *stats = NULL;\n-\t*names = reftable_calloc(sizeof(char *) * (N + 1));\n+\n+\tREFTABLE_CALLOC_ARRAY(*names, N + 1);\n+\n \treftable_writer_set_limits(w, update_index, update_index);\n \tfor (i = 0; i < N; i++) {\n \t\tchar name[100];\n@@ -188,7 +190,7 @@ static void test_log_overflow(void)\n static void test_log_write_read(void)\n {\n \tint N = 2;\n-\tchar **names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tchar **names = reftable_calloc(N + 1, sizeof(*names));\n \tint err;\n \tstruct reftable_write_options opts = {\n \t\t.block_size = 256,\n@@ -519,7 +521,7 @@ static void test_table_read_write_seek_index(void)\n static void test_table_refs_for(int indexed)\n {\n \tint N = 50;\n-\tchar **want_names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tchar **want_names = reftable_calloc(N + 1, sizeof(*want_names));\n \tint want_names_len = 0;\n \tuint8_t want_hash[GIT_SHA1_RAWSZ];\n \ndiff --git a/reftable/record_test.c b/reftable/record_test.c\nindex 2876db7d27..999366ad46 100644\n--- a/reftable/record_test.c\n+++ b/reftable/record_test.c\n@@ -231,8 +231,8 @@ static void test_reftable_log_record_roundtrip(void)\n \t\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n \t\t\t\t.value = {\n \t\t\t\t\t.update = {\n-\t\t\t\t\t\t.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n-\t\t\t\t\t\t.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t\t\t.new_hash = reftable_calloc(GIT_SHA1_RAWSZ, 1),\n+\t\t\t\t\t\t.old_hash = reftable_calloc(GIT_SHA1_RAWSZ, 1),\n \t\t\t\t\t\t.name = xstrdup(\"old name\"),\n \t\t\t\t\t\t.email = xstrdup(\"old@email\"),\n \t\t\t\t\t\t.message = xstrdup(\"old message\"),\ndiff --git a/reftable/refname.c b/reftable/refname.c\nindex 9573496932..7570e4acf9 100644\n--- a/reftable/refname.c\n+++ b/reftable/refname.c\n@@ -140,8 +140,8 @@ int validate_ref_record_addition(struct reftable_table tab,\n {\n \tstruct modification mod = {\n \t\t.tab = tab,\n-\t\t.add = reftable_calloc(sizeof(char *) * sz),\n-\t\t.del = reftable_calloc(sizeof(char *) * sz),\n+\t\t.add = reftable_calloc(sz, sizeof(*mod.add)),\n+\t\t.del = reftable_calloc(sz, sizeof(*mod.del)),\n \t};\n \tint i = 0;\n \tint err = 0;\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 1dfab99e96..d084823a92 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -50,8 +50,7 @@ static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz)\n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \t\t struct reftable_write_options config)\n {\n-\tstruct reftable_stack *p =\n-\t\treftable_calloc(sizeof(struct reftable_stack));\n+\tstruct reftable_stack *p = reftable_calloc(1, sizeof(*p));\n \tstruct strbuf list_file_name = STRBUF_INIT;\n \tint err = 0;\n \n@@ -114,7 +113,7 @@ int read_lines(const char *filename, char ***namesp)\n \tint err = 0;\n \tif (fd < 0) {\n \t\tif (errno == ENOENT) {\n-\t\t\t*namesp = reftable_calloc(sizeof(char *));\n+\t\t\tREFTABLE_CALLOC_ARRAY(*namesp, 1);\n \t\t\treturn 0;\n \t\t}\n \n@@ -191,8 +190,7 @@ void reftable_stack_destroy(struct reftable_stack *st)\n static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n \t\t\t\t\t\t int cur_len)\n {\n-\tstruct reftable_reader **cur =\n-\t\treftable_calloc(sizeof(struct reftable_reader *) * cur_len);\n+\tstruct reftable_reader **cur = reftable_calloc(cur_len, sizeof(*cur));\n \tint i = 0;\n \tfor (i = 0; i < cur_len; i++) {\n \t\tcur[i] = st->readers[i];\n@@ -208,9 +206,9 @@ static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n \tint err = 0;\n \tint names_len = names_length(names);\n \tstruct reftable_reader **new_readers =\n-\t\treftable_calloc(sizeof(struct reftable_reader *) * names_len);\n+\t\treftable_calloc(names_len, sizeof(*new_readers));\n \tstruct reftable_table *new_tables =\n-\t\treftable_calloc(sizeof(struct reftable_table) * names_len);\n+\t\treftable_calloc(names_len, sizeof(*new_tables));\n \tint new_readers_len = 0;\n \tstruct reftable_merged_table *new_merged = NULL;\n \tstruct strbuf table_path = STRBUF_INIT;\n@@ -344,7 +342,7 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\t\t\tgoto out;\n \t\t\t}\n \n-\t\t\tnames = reftable_calloc(sizeof(char *));\n+\t\t\tREFTABLE_CALLOC_ARRAY(names, 1);\n \t\t} else {\n \t\t\terr = fd_read_lines(fd, &names);\n \t\t\tif (err < 0)\n@@ -686,7 +684,7 @@ int reftable_stack_new_addition(struct reftable_addition **dest,\n {\n \tint err = 0;\n \tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n-\t*dest = reftable_calloc(sizeof(**dest));\n+\tREFTABLE_CALLOC_ARRAY(*dest, 1);\n \t**dest = empty;\n \terr = reftable_stack_init_addition(*dest, st);\n \tif (err) {\n@@ -871,7 +869,7 @@ static int stack_write_compact(struct reftable_stack *st,\n {\n \tint subtabs_len = last - first + 1;\n \tstruct reftable_table *subtabs = reftable_calloc(\n-\t\tsizeof(struct reftable_table) * (last - first + 1));\n+\t\tlast - first + 1, sizeof(*subtabs));\n \tstruct reftable_merged_table *mt = NULL;\n \tint err = 0;\n \tstruct reftable_iterator it = { NULL };\n@@ -979,9 +977,9 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n \tint compact_count = last - first + 1;\n \tchar **listp = NULL;\n \tchar **delete_on_success =\n-\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\t\treftable_calloc(compact_count + 1, sizeof(*delete_on_success));\n \tchar **subtable_locks =\n-\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\t\treftable_calloc(compact_count + 1, sizeof(*subtable_locks));\n \tint i = 0;\n \tint j = 0;\n \tint is_empty_table = 0;\n@@ -1204,7 +1202,7 @@ int fastlog2(uint64_t sz)\n \n struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n {\n-\tstruct segment *segs = reftable_calloc(sizeof(struct segment) * n);\n+\tstruct segment *segs = reftable_calloc(n, sizeof(*segs));\n \tint next = 0;\n \tstruct segment cur = { 0 };\n \tint i = 0;\n@@ -1268,7 +1266,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, int n)\n static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n {\n \tuint64_t *sizes =\n-\t\treftable_calloc(sizeof(uint64_t) * st->merged->stack_len);\n+\t\treftable_calloc(st->merged->stack_len, sizeof(*sizes));\n \tint version = (st->config.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2;\n \tint overhead = header_size(version) - 1;\n \tint i = 0;\ndiff --git a/reftable/tree.c b/reftable/tree.c\nindex a5bf880985..528f33ae38 100644\n--- a/reftable/tree.c\n+++ b/reftable/tree.c\n@@ -20,8 +20,8 @@ struct tree_node *tree_search(void *key, struct tree_node **rootp,\n \t\tif (!insert) {\n \t\t\treturn NULL;\n \t\t} else {\n-\t\t\tstruct tree_node *n =\n-\t\t\t\treftable_calloc(sizeof(struct tree_node));\n+\t\t\tstruct tree_node *n;\n+\t\t\tREFTABLE_CALLOC_ARRAY(n, 1);\n \t\t\tn->key = key;\n \t\t\t*rootp = n;\n \t\t\treturn *rootp;\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 4483bb21c3..47b3448132 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -49,7 +49,7 @@ static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n {\n \tint n = 0;\n \tif (w->pending_padding > 0) {\n-\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding);\n+\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding, sizeof(*zeroed));\n \t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n \t\tif (n < 0)\n \t\t\treturn n;\n@@ -123,8 +123,7 @@ struct reftable_writer *\n reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\t void *writer_arg, struct reftable_write_options *opts)\n {\n-\tstruct reftable_writer *wp =\n-\t\treftable_calloc(sizeof(struct reftable_writer));\n+\tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n \tstrbuf_init(&wp->block_writer_data.last_key, 0);\n \toptions_set_defaults(opts);\n \tif (opts->block_size >= (1 << 24)) {\n@@ -132,7 +131,7 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\tabort();\n \t}\n \twp->last_key = reftable_empty_strbuf;\n-\twp->block = reftable_calloc(opts->block_size);\n+\tREFTABLE_CALLOC_ARRAY(wp->block, opts->block_size);\n \twp->write = writer_func;\n \twp->write_arg = writer_arg;\n \twp->opts = *opts;\n-- \n2.43.GIT\n\n"},{"id":"487633","messageId":"e2d05f7c385fd9ec23b1dbb8cc273c97505444e4.1706687982.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706687982.git.ps@pks.im","subject":"[PATCH 3/9] reftable/stack: fix parameter validation when compacting range","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-01-31T08:01:11Z","receivedAt":"2024-01-31T08:01:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `stack_compact_range()` function receives a \"first\" and \"last\" index\nthat indicates which tables of the reftable stack should be compacted.\nNaturally, \"first\" must be smaller than \"last\" in order to identify a\nproper range of tables to compress, which we indeed also assert in the\nfunction. But the validations happens after we have already allocated\narrays with a size of `last - first + 1`, leading to an underflow and\nthus an invalid allocation size.\n\nFix this by reordering the array allocations to happen after we have\nvalidated parameters. While at it, convert the array allocations to use\nthe newly introduced macros.\n\nNote that the relevant variables pointing into arrays should also be\nconverted to use `size_t` instead of `int`. This is left for a later\ncommit in this series.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 24 +++++++++++++-----------\n 1 file changed, 13 insertions(+), 11 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex d084823a92..b6b24c90bf 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -966,6 +966,7 @@ static int stack_write_compact(struct reftable_stack *st,\n static int stack_compact_range(struct reftable_stack *st, int first, int last,\n \t\t\t struct reftable_log_expiry_config *expiry)\n {\n+\tchar **delete_on_success = NULL, **subtable_locks = NULL, **listp = NULL;\n \tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n \tstruct strbuf new_table_name = STRBUF_INIT;\n \tstruct strbuf lock_file_name = STRBUF_INIT;\n@@ -974,12 +975,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n \tint err = 0;\n \tint have_lock = 0;\n \tint lock_file_fd = -1;\n-\tint compact_count = last - first + 1;\n-\tchar **listp = NULL;\n-\tchar **delete_on_success =\n-\t\treftable_calloc(compact_count + 1, sizeof(*delete_on_success));\n-\tchar **subtable_locks =\n-\t\treftable_calloc(compact_count + 1, sizeof(*subtable_locks));\n+\tint compact_count;\n \tint i = 0;\n \tint j = 0;\n \tint is_empty_table = 0;\n@@ -989,6 +985,10 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n \t\tgoto done;\n \t}\n \n+\tcompact_count = last - first + 1;\n+\tREFTABLE_CALLOC_ARRAY(delete_on_success, compact_count + 1);\n+\tREFTABLE_CALLOC_ARRAY(subtable_locks, compact_count + 1);\n+\n \tst->stats.attempts++;\n \n \tstrbuf_reset(&lock_file_name);\n@@ -1146,12 +1146,14 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n done:\n \tfree_names(delete_on_success);\n \n-\tlistp = subtable_locks;\n-\twhile (*listp) {\n-\t\tunlink(*listp);\n-\t\tlistp++;\n+\tif (subtable_locks) {\n+\t\tlistp = subtable_locks;\n+\t\twhile (*listp) {\n+\t\t\tunlink(*listp);\n+\t\t\tlistp++;\n+\t\t}\n+\t\tfree_names(subtable_locks);\n \t}\n-\tfree_names(subtable_locks);\n \tif (lock_file_fd >= 0) {\n \t\tclose(lock_file_fd);\n \t\tlock_file_fd = -1;\n-- \n2.43.GIT\n\n"},{"id":"487634","messageId":"b8b2cce742dea7d720366ae086a80c976183e30e.1706687982.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706687982.git.ps@pks.im","subject":"[PATCH 4/9] reftable/stack: index segments with `size_t`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-01-31T08:01:15Z","receivedAt":"2024-01-31T08:01:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We use `int`s to index into arrays of segments and track the length of\nthem, which is considered to be a code smell in the Git project. Convert\nthe code to use `size_t` instead.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 25 +++++++++++--------------\n reftable/stack.h | 6 +++---\n reftable/stack_test.c | 7 +++----\n 3 files changed, 17 insertions(+), 21 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex b6b24c90bf..2be3d1e4ba 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1202,12 +1202,11 @@ int fastlog2(uint64_t sz)\n \treturn l - 1;\n }\n \n-struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n+struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n)\n {\n \tstruct segment *segs = reftable_calloc(n, sizeof(*segs));\n-\tint next = 0;\n \tstruct segment cur = { 0 };\n-\tint i = 0;\n+\tsize_t next = 0, i;\n \n \tif (n == 0) {\n \t\t*seglen = 0;\n@@ -1233,29 +1232,27 @@ struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n \treturn segs;\n }\n \n-struct segment suggest_compaction_segment(uint64_t *sizes, int n)\n+struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n {\n-\tint seglen = 0;\n-\tstruct segment *segs = sizes_to_segments(&seglen, sizes, n);\n \tstruct segment min_seg = {\n \t\t.log = 64,\n \t};\n-\tint i = 0;\n+\tstruct segment *segs;\n+\tsize_t seglen = 0, i;\n+\n+\tsegs = sizes_to_segments(&seglen, sizes, n);\n \tfor (i = 0; i < seglen; i++) {\n-\t\tif (segment_size(&segs[i]) == 1) {\n+\t\tif (segment_size(&segs[i]) == 1)\n \t\t\tcontinue;\n-\t\t}\n \n-\t\tif (segs[i].log < min_seg.log) {\n+\t\tif (segs[i].log < min_seg.log)\n \t\t\tmin_seg = segs[i];\n-\t\t}\n \t}\n \n \twhile (min_seg.start > 0) {\n-\t\tint prev = min_seg.start - 1;\n-\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev])) {\n+\t\tsize_t prev = min_seg.start - 1;\n+\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev]))\n \t\t\tbreak;\n-\t\t}\n \n \t\tmin_seg.start = prev;\n \t\tmin_seg.bytes += sizes[prev];\ndiff --git a/reftable/stack.h b/reftable/stack.h\nindex c1e3efa899..d919455669 100644\n--- a/reftable/stack.h\n+++ b/reftable/stack.h\n@@ -32,13 +32,13 @@ struct reftable_stack {\n int read_lines(const char *filename, char ***lines);\n \n struct segment {\n-\tint start, end;\n+\tsize_t start, end;\n \tint log;\n \tuint64_t bytes;\n };\n \n int fastlog2(uint64_t sz);\n-struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n);\n-struct segment suggest_compaction_segment(uint64_t *sizes, int n);\n+struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n);\n+struct segment suggest_compaction_segment(uint64_t *sizes, size_t n);\n \n #endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 289e902146..2d5b24e5c5 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -711,7 +711,7 @@ static void test_sizes_to_segments(void)\n \tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n \t/* .................0 1 2 3 4 5 */\n \n-\tint seglen = 0;\n+\tsize_t seglen = 0;\n \tstruct segment *segs =\n \t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n \tEXPECT(segs[2].log == 3);\n@@ -726,7 +726,7 @@ static void test_sizes_to_segments(void)\n \n static void test_sizes_to_segments_empty(void)\n {\n-\tint seglen = 0;\n+\tsize_t seglen = 0;\n \tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n \tEXPECT(seglen == 0);\n \treftable_free(segs);\n@@ -735,8 +735,7 @@ static void test_sizes_to_segments_empty(void)\n static void test_sizes_to_segments_all_equal(void)\n {\n \tuint64_t sizes[] = { 5, 5 };\n-\n-\tint seglen = 0;\n+\tsize_t seglen = 0;\n \tstruct segment *segs =\n \t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n \tEXPECT(seglen == 1);\n-- \n2.43.GIT\n\n"},{"id":"487635","messageId":"2f6a69aa14ddb865548bafa5f3005d0579ea09f9.1706687982.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706687982.git.ps@pks.im","subject":"[PATCH 5/9] reftable/stack: use `size_t` to track stack slices during compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-01-31T08:01:19Z","receivedAt":"2024-01-31T08:01:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We use `int`s to track reftable slices when compacting the reftable\nstack, which is considered to be a code smell in the Git project.\nConvert the code to use `size_t` instead.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 32 ++++++++++++++++----------------\n 1 file changed, 16 insertions(+), 16 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 2be3d1e4ba..c1f8cf1cef 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -24,7 +24,8 @@ static int stack_try_add(struct reftable_stack *st,\n \t\t\t\t\t void *arg),\n \t\t\t void *arg);\n static int stack_write_compact(struct reftable_stack *st,\n-\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_writer *wr,\n+\t\t\t size_t first, size_t last,\n \t\t\t struct reftable_log_expiry_config *config);\n static int stack_check_addition(struct reftable_stack *st,\n \t\t\t\tconst char *new_tab_name);\n@@ -820,7 +821,8 @@ uint64_t reftable_stack_next_update_index(struct reftable_stack *st)\n \treturn 1;\n }\n \n-static int stack_compact_locked(struct reftable_stack *st, int first, int last,\n+static int stack_compact_locked(struct reftable_stack *st,\n+\t\t\t\tsize_t first, size_t last,\n \t\t\t\tstruct strbuf *temp_tab,\n \t\t\t\tstruct reftable_log_expiry_config *config)\n {\n@@ -864,22 +866,21 @@ static int stack_compact_locked(struct reftable_stack *st, int first, int last,\n }\n \n static int stack_write_compact(struct reftable_stack *st,\n-\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_writer *wr,\n+\t\t\t size_t first, size_t last,\n \t\t\t struct reftable_log_expiry_config *config)\n {\n \tint subtabs_len = last - first + 1;\n \tstruct reftable_table *subtabs = reftable_calloc(\n \t\tlast - first + 1, sizeof(*subtabs));\n \tstruct reftable_merged_table *mt = NULL;\n-\tint err = 0;\n \tstruct reftable_iterator it = { NULL };\n \tstruct reftable_ref_record ref = { NULL };\n \tstruct reftable_log_record log = { NULL };\n-\n \tuint64_t entries = 0;\n+\tint err = 0;\n \n-\tint i = 0, j = 0;\n-\tfor (i = first, j = 0; i <= last; i++) {\n+\tfor (size_t i = first, j = 0; i <= last; i++) {\n \t\tstruct reftable_reader *t = st->readers[i];\n \t\treftable_table_from_reader(&subtabs[j++], t);\n \t\tst->stats.bytes += t->size;\n@@ -963,7 +964,8 @@ static int stack_write_compact(struct reftable_stack *st,\n }\n \n /* < 0: error. 0 == OK, > 0 attempt failed; could retry. */\n-static int stack_compact_range(struct reftable_stack *st, int first, int last,\n+static int stack_compact_range(struct reftable_stack *st,\n+\t\t\t size_t first, size_t last,\n \t\t\t struct reftable_log_expiry_config *expiry)\n {\n \tchar **delete_on_success = NULL, **subtable_locks = NULL, **listp = NULL;\n@@ -972,12 +974,10 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n \tstruct strbuf lock_file_name = STRBUF_INIT;\n \tstruct strbuf ref_list_contents = STRBUF_INIT;\n \tstruct strbuf new_table_path = STRBUF_INIT;\n+\tsize_t i, j, compact_count;\n \tint err = 0;\n \tint have_lock = 0;\n \tint lock_file_fd = -1;\n-\tint compact_count;\n-\tint i = 0;\n-\tint j = 0;\n \tint is_empty_table = 0;\n \n \tif (first > last || (!expiry && first == last)) {\n@@ -1172,17 +1172,17 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n int reftable_stack_compact_all(struct reftable_stack *st,\n \t\t\t struct reftable_log_expiry_config *config)\n {\n-\treturn stack_compact_range(st, 0, st->merged->stack_len - 1, config);\n+\treturn stack_compact_range(st, 0, st->merged->stack_len ?\n+\t\t\tst->merged->stack_len - 1 : 0, config);\n }\n \n-static int stack_compact_range_stats(struct reftable_stack *st, int first,\n-\t\t\t\t int last,\n+static int stack_compact_range_stats(struct reftable_stack *st,\n+\t\t\t\t size_t first, size_t last,\n \t\t\t\t struct reftable_log_expiry_config *config)\n {\n \tint err = stack_compact_range(st, first, last, config);\n-\tif (err > 0) {\n+\tif (err > 0)\n \t\tst->stats.failures++;\n-\t}\n \treturn err;\n }\n \n-- \n2.43.GIT\n\n"},{"id":"487636","messageId":"1ee9a4477f1bf49258a4d5e9bc446c6c7dae85fb.1706687982.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706687982.git.ps@pks.im","subject":"[PATCH 6/9] reftable/stack: use `size_t` to track stack length","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-01-31T08:01:23Z","receivedAt":"2024-01-31T08:01:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"While the stack length is already stored as `size_t`, we frequently use\n`int`s to refer to those stacks throughout the reftable library. Convert\nthose cases to use `size_t` instead to make things consistent.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 7 +++----\n reftable/basics.h | 2 +-\n reftable/merged.c | 11 +++++------\n reftable/merged_test.c | 14 ++++++--------\n reftable/reftable-merged.h | 2 +-\n reftable/stack.c | 21 ++++++++++-----------\n 6 files changed, 26 insertions(+), 31 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex af9004cec2..0785aff941 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -64,12 +64,11 @@ void free_names(char **a)\n \treftable_free(a);\n }\n \n-int names_length(char **names)\n+size_t names_length(char **names)\n {\n \tchar **p = names;\n-\tfor (; *p; p++) {\n-\t\t/* empty */\n-\t}\n+\twhile (*p)\n+\t\tp++;\n \treturn p - names;\n }\n \ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 4c3ac963a3..91f3533efe 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -44,7 +44,7 @@ void parse_names(char *buf, int size, char ***namesp);\n int names_equal(char **a, char **b);\n \n /* returns the array size of a NULL-terminated array of strings. */\n-int names_length(char **names);\n+size_t names_length(char **names);\n \n /* Allocation routines; they invoke the functions set through\n * reftable_set_alloc() */\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 2031fd51b4..e2c6253324 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -45,11 +45,10 @@ static int merged_iter_init(struct merged_iter *mi)\n static void merged_iter_close(void *p)\n {\n \tstruct merged_iter *mi = p;\n-\tint i = 0;\n+\n \tmerged_iter_pqueue_release(&mi->pq);\n-\tfor (i = 0; i < mi->stack_len; i++) {\n+\tfor (size_t i = 0; i < mi->stack_len; i++)\n \t\treftable_iterator_destroy(&mi->stack[i]);\n-\t}\n \treftable_free(mi->stack);\n \tstrbuf_release(&mi->key);\n \tstrbuf_release(&mi->entry_key);\n@@ -168,14 +167,14 @@ static void iterator_from_merged_iter(struct reftable_iterator *it,\n }\n \n int reftable_new_merged_table(struct reftable_merged_table **dest,\n-\t\t\t struct reftable_table *stack, int n,\n+\t\t\t struct reftable_table *stack, size_t n,\n \t\t\t uint32_t hash_id)\n {\n \tstruct reftable_merged_table *m = NULL;\n \tuint64_t last_max = 0;\n \tuint64_t first_min = 0;\n-\tint i = 0;\n-\tfor (i = 0; i < n; i++) {\n+\n+\tfor (size_t i = 0; i < n; i++) {\n \t\tuint64_t min = reftable_table_min_update_index(&stack[i]);\n \t\tuint64_t max = reftable_table_max_update_index(&stack[i]);\n \ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nindex e233a9d581..442917cc83 100644\n--- a/reftable/merged_test.c\n+++ b/reftable/merged_test.c\n@@ -88,18 +88,17 @@ static struct reftable_merged_table *\n merged_table_from_records(struct reftable_ref_record **refs,\n \t\t\t struct reftable_block_source **source,\n \t\t\t struct reftable_reader ***readers, int *sizes,\n-\t\t\t struct strbuf *buf, int n)\n+\t\t\t struct strbuf *buf, size_t n)\n {\n-\tint i = 0;\n \tstruct reftable_merged_table *mt = NULL;\n-\tint err;\n \tstruct reftable_table *tabs;\n+\tint err;\n \n \tREFTABLE_CALLOC_ARRAY(tabs, n);\n \tREFTABLE_CALLOC_ARRAY(*readers, n);\n \tREFTABLE_CALLOC_ARRAY(*source, n);\n \n-\tfor (i = 0; i < n; i++) {\n+\tfor (size_t i = 0; i < n; i++) {\n \t\twrite_test_table(&buf[i], refs[i], sizes[i]);\n \t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n \n@@ -263,18 +262,17 @@ static struct reftable_merged_table *\n merged_table_from_log_records(struct reftable_log_record **logs,\n \t\t\t struct reftable_block_source **source,\n \t\t\t struct reftable_reader ***readers, int *sizes,\n-\t\t\t struct strbuf *buf, int n)\n+\t\t\t struct strbuf *buf, size_t n)\n {\n-\tint i = 0;\n \tstruct reftable_merged_table *mt = NULL;\n-\tint err;\n \tstruct reftable_table *tabs;\n+\tint err;\n \n \tREFTABLE_CALLOC_ARRAY(tabs, n);\n \tREFTABLE_CALLOC_ARRAY(*readers, n);\n \tREFTABLE_CALLOC_ARRAY(*source, n);\n \n-\tfor (i = 0; i < n; i++) {\n+\tfor (size_t i = 0; i < n; i++) {\n \t\twrite_test_log_table(&buf[i], logs[i], sizes[i], i + 1);\n \t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n \ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nindex 1a6d16915a..c91a2d83a2 100644\n--- a/reftable/reftable-merged.h\n+++ b/reftable/reftable-merged.h\n@@ -33,7 +33,7 @@ struct reftable_table;\n the stack array.\n */\n int reftable_new_merged_table(struct reftable_merged_table **dest,\n-\t\t\t struct reftable_table *stack, int n,\n+\t\t\t struct reftable_table *stack, size_t n,\n \t\t\t uint32_t hash_id);\n \n /* returns an iterator positioned just before 'name' */\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex c1f8cf1cef..079ba7fde8 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -202,18 +202,18 @@ static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n \t\t\t\t int reuse_open)\n {\n-\tint cur_len = !st->merged ? 0 : st->merged->stack_len;\n+\tsize_t cur_len = !st->merged ? 0 : st->merged->stack_len;\n \tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n-\tint err = 0;\n-\tint names_len = names_length(names);\n+\tsize_t names_len = names_length(names);\n \tstruct reftable_reader **new_readers =\n \t\treftable_calloc(names_len, sizeof(*new_readers));\n \tstruct reftable_table *new_tables =\n \t\treftable_calloc(names_len, sizeof(*new_tables));\n-\tint new_readers_len = 0;\n+\tsize_t new_readers_len = 0;\n \tstruct reftable_merged_table *new_merged = NULL;\n \tstruct strbuf table_path = STRBUF_INIT;\n-\tint i;\n+\tint err = 0;\n+\tsize_t i;\n \n \twhile (*names) {\n \t\tstruct reftable_reader *rd = NULL;\n@@ -221,11 +221,10 @@ static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n \n \t\t/* this is linear; we assume compaction keeps the number of\n \t\t tables under control so this is not quadratic. */\n-\t\tint j = 0;\n-\t\tfor (j = 0; reuse_open && j < cur_len; j++) {\n-\t\t\tif (cur[j] && 0 == strcmp(cur[j]->name, name)) {\n-\t\t\t\trd = cur[j];\n-\t\t\t\tcur[j] = NULL;\n+\t\tfor (i = 0; reuse_open && i < cur_len; i++) {\n+\t\t\tif (cur[i] && 0 == strcmp(cur[i]->name, name)) {\n+\t\t\t\trd = cur[i];\n+\t\t\t\tcur[i] = NULL;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t}\n@@ -870,7 +869,7 @@ static int stack_write_compact(struct reftable_stack *st,\n \t\t\t size_t first, size_t last,\n \t\t\t struct reftable_log_expiry_config *config)\n {\n-\tint subtabs_len = last - first + 1;\n+\tsize_t subtabs_len = last - first + 1;\n \tstruct reftable_table *subtabs = reftable_calloc(\n \t\tlast - first + 1, sizeof(*subtabs));\n \tstruct reftable_merged_table *mt = NULL;\n-- \n2.43.GIT\n\n"},{"id":"487637","messageId":"00aeaeee63dc1371d3338b72f779c6f165bd14f7.1706687982.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706687982.git.ps@pks.im","subject":"[PATCH 7/9] reftable/merged: refactor seeking of records","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-01-31T08:01:28Z","receivedAt":"2024-01-31T08:01:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The code to seek reftable records in the merged table code is quite hard\nto read and does not conform to our coding style in multiple ways:\n\n - We have multiple exit paths where we release resources even though\n that is not really necessary\n\n - We use a scoped error variable `e` which is hard to reason about.\n This variable is not required at all.\n\n - We allocate memory in the variable declarations, which is easy to\n miss.\n\nRefactor the function so that it becomes more maintainable in the\nfuture.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 54 ++++++++++++++++++-----------------------------\n 1 file changed, 21 insertions(+), 33 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex e2c6253324..0abcda26e8 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -238,50 +238,38 @@ static int merged_table_seek_record(struct reftable_merged_table *mt,\n \t\t\t\t struct reftable_iterator *it,\n \t\t\t\t struct reftable_record *rec)\n {\n-\tstruct reftable_iterator *iters = reftable_calloc(\n-\t\tmt->stack_len, sizeof(*iters));\n \tstruct merged_iter merged = {\n-\t\t.stack = iters,\n \t\t.typ = reftable_record_type(rec),\n \t\t.hash_id = mt->hash_id,\n \t\t.suppress_deletions = mt->suppress_deletions,\n \t\t.key = STRBUF_INIT,\n \t\t.entry_key = STRBUF_INIT,\n \t};\n-\tint n = 0;\n-\tint err = 0;\n-\tint i = 0;\n-\tfor (i = 0; i < mt->stack_len && err == 0; i++) {\n-\t\tint e = reftable_table_seek_record(&mt->stack[i], &iters[n],\n-\t\t\t\t\t\t rec);\n-\t\tif (e < 0) {\n-\t\t\terr = e;\n-\t\t}\n-\t\tif (e == 0) {\n-\t\t\tn++;\n-\t\t}\n-\t}\n-\tif (err < 0) {\n-\t\tint i = 0;\n-\t\tfor (i = 0; i < n; i++) {\n-\t\t\treftable_iterator_destroy(&iters[i]);\n-\t\t}\n-\t\treftable_free(iters);\n-\t\treturn err;\n+\tstruct merged_iter *p;\n+\tint err;\n+\n+\tREFTABLE_CALLOC_ARRAY(merged.stack, mt->stack_len);\n+\tfor (size_t i = 0; i < mt->stack_len; i++) {\n+\t\terr = reftable_table_seek_record(&mt->stack[i],\n+\t\t\t\t\t\t &merged.stack[merged.stack_len], rec);\n+\t\tif (err < 0)\n+\t\t\tgoto out;\n+\t\tif (!err)\n+\t\t\tmerged.stack_len++;\n \t}\n \n-\tmerged.stack_len = n;\n \terr = merged_iter_init(&merged);\n-\tif (err < 0) {\n+\tif (err < 0)\n+\t\tgoto out;\n+\n+\tp = reftable_malloc(sizeof(struct merged_iter));\n+\t*p = merged;\n+\titerator_from_merged_iter(it, p);\n+\n+out:\n+\tif (err < 0)\n \t\tmerged_iter_close(&merged);\n-\t\treturn err;\n-\t} else {\n-\t\tstruct merged_iter *p =\n-\t\t\treftable_malloc(sizeof(struct merged_iter));\n-\t\t*p = merged;\n-\t\titerator_from_merged_iter(it, p);\n-\t}\n-\treturn 0;\n+\treturn err;\n }\n \n int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n-- \n2.43.GIT\n\n"},{"id":"487638","messageId":"31864740e996368f7b87cdd47e828502fe0e80ad.1706687982.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706687982.git.ps@pks.im","subject":"[PATCH 8/9] reftable/merged: refactor initialization of iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-01-31T08:01:32Z","receivedAt":"2024-01-31T08:01:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor the initialization of the merged iterator to fit our code style\nbetter. This refactoring prepares the code for a refactoring of how\nrecords are being initialized.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 27 +++++++++++++--------------\n 1 file changed, 13 insertions(+), 14 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 0abcda26e8..0e60e2a39b 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -19,24 +19,23 @@ license that can be found in the LICENSE file or at\n \n static int merged_iter_init(struct merged_iter *mi)\n {\n-\tint i = 0;\n-\tfor (i = 0; i < mi->stack_len; i++) {\n-\t\tstruct reftable_record rec = reftable_new_record(mi->typ);\n-\t\tint err = iterator_next(&mi->stack[i], &rec);\n-\t\tif (err < 0) {\n+\tfor (size_t i = 0; i < mi->stack_len; i++) {\n+\t\tstruct pq_entry e = {\n+\t\t\t.rec = reftable_new_record(mi->typ),\n+\t\t\t.index = i,\n+\t\t};\n+\t\tint err;\n+\n+\t\terr = iterator_next(&mi->stack[i], &e.rec);\n+\t\tif (err < 0)\n \t\t\treturn err;\n-\t\t}\n-\n \t\tif (err > 0) {\n \t\t\treftable_iterator_destroy(&mi->stack[i]);\n-\t\t\treftable_record_release(&rec);\n-\t\t} else {\n-\t\t\tstruct pq_entry e = {\n-\t\t\t\t.rec = rec,\n-\t\t\t\t.index = i,\n-\t\t\t};\n-\t\t\tmerged_iter_pqueue_add(&mi->pq, &e);\n+\t\t\treftable_record_release(&e.rec);\n+\t\t\tcontinue;\n \t\t}\n+\n+\t\tmerged_iter_pqueue_add(&mi->pq, &e);\n \t}\n \n \treturn 0;\n-- \n2.43.GIT\n\n"},{"id":"487639","messageId":"fd8a1ce99d10ac63f495e1723b143235fcf318e1.1706687982.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706687982.git.ps@pks.im","subject":"[PATCH 9/9] reftable/record: improve semantics when initializing records","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-01-31T08:01:36Z","receivedAt":"2024-01-31T08:01:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"According to our usual coding style, the `reftable_new_record()`\nfunction would indicate that it is allocating a new record. This is not\nthe case though as the function merely initializes records without\nallocating any memory.\n\nReplace `reftable_new_record()` with a new `reftable_record_init()`\nfunction that takes a record pointer as input and initializes it\naccordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/block.c | 18 +++++++++---------\n reftable/merged.c | 8 +++++---\n reftable/reader.c | 4 ++--\n reftable/record.c | 43 ++++++++++--------------------------------\n reftable/record.h | 10 +++++-----\n reftable/record_test.c | 4 ++--\n 6 files changed, 33 insertions(+), 54 deletions(-)\n\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 6952d0facf..9ad220747e 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -380,23 +380,23 @@ int block_reader_seek(struct block_reader *br, struct block_iter *it,\n \t\t.key = *want,\n \t\t.r = br,\n \t};\n-\tstruct reftable_record rec = reftable_new_record(block_reader_type(br));\n-\tint err = 0;\n \tstruct block_iter next = BLOCK_ITER_INIT;\n+\tstruct reftable_record rec;\n+\tint err = 0, i;\n \n-\tint i = binsearch(br->restart_count, &restart_key_less, &args);\n \tif (args.error) {\n \t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n \t}\n \n-\tit->br = br;\n-\tif (i > 0) {\n-\t\ti--;\n-\t\tit->next_off = block_reader_restart_offset(br, i);\n-\t} else {\n+\ti = binsearch(br->restart_count, &restart_key_less, &args);\n+\tif (i > 0)\n+\t\tit->next_off = block_reader_restart_offset(br, i - 1);\n+\telse\n \t\tit->next_off = br->header_off + 4;\n-\t}\n+\tit->br = br;\n+\n+\treftable_record_init(&rec, block_reader_type(br));\n \n \t/* We're looking for the last entry less/equal than the wanted key, so\n \t we have to go one entry too far and then back up.\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 0e60e2a39b..a0f222e07b 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -21,11 +21,11 @@ static int merged_iter_init(struct merged_iter *mi)\n {\n \tfor (size_t i = 0; i < mi->stack_len; i++) {\n \t\tstruct pq_entry e = {\n-\t\t\t.rec = reftable_new_record(mi->typ),\n \t\t\t.index = i,\n \t\t};\n \t\tint err;\n \n+\t\treftable_record_init(&e.rec, mi->typ);\n \t\terr = iterator_next(&mi->stack[i], &e.rec);\n \t\tif (err < 0)\n \t\t\treturn err;\n@@ -57,10 +57,12 @@ static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,\n \t\t\t\t\t size_t idx)\n {\n \tstruct pq_entry e = {\n-\t\t.rec = reftable_new_record(mi->typ),\n \t\t.index = idx,\n \t};\n-\tint err = iterator_next(&mi->stack[idx], &e.rec);\n+\tint err;\n+\n+\treftable_record_init(&e.rec, mi->typ);\n+\terr = iterator_next(&mi->stack[idx], &e.rec);\n \tif (err < 0)\n \t\treturn err;\n \ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 3e0de5e8ad..5e6c8f30a1 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -444,13 +444,13 @@ static int reader_start(struct reftable_reader *r, struct table_iter *ti,\n static int reader_seek_linear(struct table_iter *ti,\n \t\t\t struct reftable_record *want)\n {\n-\tstruct reftable_record rec =\n-\t\treftable_new_record(reftable_record_type(want));\n \tstruct strbuf want_key = STRBUF_INIT;\n \tstruct strbuf got_key = STRBUF_INIT;\n \tstruct table_iter next = TABLE_ITER_INIT;\n+\tstruct reftable_record rec;\n \tint err = -1;\n \n+\treftable_record_init(&rec, reftable_record_type(want));\n \treftable_record_key(want, &want_key);\n \n \twhile (1) {\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 5c3fbb7b2a..8b84b8157f 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -1257,45 +1257,22 @@ reftable_record_vtable(struct reftable_record *rec)\n \tabort();\n }\n \n-struct reftable_record reftable_new_record(uint8_t typ)\n+void reftable_record_init(struct reftable_record *rec, uint8_t typ)\n {\n-\tstruct reftable_record clean = {\n-\t\t.type = typ,\n-\t};\n+\tmemset(rec, 0, sizeof(*rec));\n+\trec->type = typ;\n \n-\t/* the following is involved, but the naive solution (just return\n-\t * `clean` as is, except for BLOCK_TYPE_INDEX), returns a garbage\n-\t * clean.u.obj.offsets pointer on Windows VS CI. Go figure.\n-\t */\n \tswitch (typ) {\n-\tcase BLOCK_TYPE_OBJ:\n-\t{\n-\t\tstruct reftable_obj_record obj = { 0 };\n-\t\tclean.u.obj = obj;\n-\t\tbreak;\n-\t}\n-\tcase BLOCK_TYPE_INDEX:\n-\t{\n-\t\tstruct reftable_index_record idx = {\n-\t\t\t.last_key = STRBUF_INIT,\n-\t\t};\n-\t\tclean.u.idx = idx;\n-\t\tbreak;\n-\t}\n \tcase BLOCK_TYPE_REF:\n-\t{\n-\t\tstruct reftable_ref_record ref = { 0 };\n-\t\tclean.u.ref = ref;\n-\t\tbreak;\n-\t}\n \tcase BLOCK_TYPE_LOG:\n-\t{\n-\t\tstruct reftable_log_record log = { 0 };\n-\t\tclean.u.log = log;\n-\t\tbreak;\n-\t}\n+\tcase BLOCK_TYPE_OBJ:\n+\t\treturn;\n+\tcase BLOCK_TYPE_INDEX:\n+\t\tstrbuf_init(&rec->u.idx.last_key, 0);\n+\t\treturn;\n+\tdefault:\n+\t\tBUG(\"unhandled record type\");\n \t}\n-\treturn clean;\n }\n \n void reftable_record_print(struct reftable_record *rec, int hash_size)\ndiff --git a/reftable/record.h b/reftable/record.h\nindex fd80cd451d..e64ed30c80 100644\n--- a/reftable/record.h\n+++ b/reftable/record.h\n@@ -69,9 +69,6 @@ struct reftable_record_vtable {\n /* returns true for recognized block types. Block start with the block type. */\n int reftable_is_block_type(uint8_t typ);\n \n-/* return an initialized record for the given type */\n-struct reftable_record reftable_new_record(uint8_t typ);\n-\n /* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns\n * number of bytes written. */\n int reftable_encode_key(int *is_restart, struct string_view dest,\n@@ -100,8 +97,8 @@ struct reftable_obj_record {\n /* record is a generic wrapper for different types of records. It is normally\n * created on the stack, or embedded within another struct. If the type is\n * known, a fresh instance can be initialized explicitly. Otherwise, use\n- * reftable_new_record() to initialize generically (as the index_record is not\n- * valid as 0-initialized structure)\n+ * `reftable_record_init()` to initialize generically (as the index_record is\n+ * not valid as 0-initialized structure)\n */\n struct reftable_record {\n \tuint8_t type;\n@@ -113,6 +110,9 @@ struct reftable_record {\n \t} u;\n };\n \n+/* Initialize the reftable record for the given type */\n+void reftable_record_init(struct reftable_record *rec, uint8_t typ);\n+\n /* see struct record_vtable */\n int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);\n void reftable_record_print(struct reftable_record *rec, int hash_size);\ndiff --git a/reftable/record_test.c b/reftable/record_test.c\nindex 999366ad46..a86cff5526 100644\n--- a/reftable/record_test.c\n+++ b/reftable/record_test.c\n@@ -16,11 +16,11 @@\n \n static void test_copy(struct reftable_record *rec)\n {\n-\tstruct reftable_record copy = { 0 };\n+\tstruct reftable_record copy;\n \tuint8_t typ;\n \n \ttyp = reftable_record_type(rec);\n-\tcopy = reftable_new_record(typ);\n+\treftable_record_init(©, typ);\n \treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n \t/* do it twice to catch memory leaks */\n \treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n-- \n2.43.GIT\n\n"},{"id":"487652","messageId":"CAPig+cQ1gKt2jgmtxJErCCE_C-_Ne7VnqxWj4CtOPWE-Xf6_iw@mail.gmail.com","threadId":"60822","inReplyTo":"0597e6944a1a65720d050f47bc82766d5bcf860b.1706687982.git.ps@pks.im","subject":"Re: [PATCH 1/9] reftable: introduce macros to grow arrays","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2024-01-31T17:44:44Z","receivedAt":"2024-01-31T17:44:56Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Wed, Jan 31, 2024 at 3:01 AM Patrick Steinhardt wrote:\n> Throughout the reftable library we have many cases where we need to grow\n> arrays. In order to avoid too many reallocations, we roughly double the\n> capacity of the array on each iteration. The resulting code pattern is\n> thus duplicated across many sites.\n>\n> We have similar patterns in our main codebase, which is why we have\n> eventually introduced an `ALLOC_GROW()` macro to abstract it away and\n> avoid some code duplication. We cannot easily reuse this macro here\n> though because `ALLOC_GROW()` uses `REALLOC_ARRAY()`, which in turn will\n> call realloc(3P) to grow the array. The reftable code is structured as a\n> library though (even if the boundaries are fuzzy), and one property this\n> brings with it is that it is possible to plug in your own allocators. So\n> instead of using realloc(3P), we need to use `reftable_realloc()` that\n> knows to use the user-provided implementation.\n>\n> So let's introduce two new macros `REFTABLE_REALLOC_ARRAY()` and\n> `REFTABLE_ALLOC_GROW()` that mirror what we do in our main codebase,\n> with two modifications:\n>\n> - They use `reftable_realloc()`, as explained above.\n>\n> - They use a different growth factor of `2 * cap + 1` instead of `(cap\n> + 16) * 3 / 2`.\n>\n> The second change is because we know a bit more about the allocation\n> patterns in the reftable library. For In most cases, we end up only having a\n\ns/For In/In/\n\n> single item in the array, so the initial capacity that our global growth\n> factor uses (which is 24), significantly overallocates in a lot of code\n\nPerhaps? s/overallocates/over-allocates/\n\n> paths. This effect is indeed measurable:\n> [...]\n>\n> Convert the reftable library to use these new macros.\n>\n> Signed-off-by: Patrick Steinhardt \n\n(Neither above comment is worth a reroll.)\n"},{"id":"487653","messageId":"CAPig+cTZcxiZUPgjqEzMNG72X4fwnNT_DwjTj5cQh5zEpiN2Hw@mail.gmail.com","threadId":"60822","inReplyTo":"00aeaeee63dc1371d3338b72f779c6f165bd14f7.1706687982.git.ps@pks.im","subject":"Re: [PATCH 7/9] reftable/merged: refactor seeking of records","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2024-01-31T17:55:22Z","receivedAt":"2024-01-31T17:55:34Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Wed, Jan 31, 2024 at 3:02 AM Patrick Steinhardt wrote:\n> The code to seek reftable records in the merged table code is quite hard\n> to read and does not conform to our coding style in multiple ways:\n>\n> - We have multiple exit paths where we release resources even though\n> that is not really necessary\n\ns/$/./\n\n> - We use a scoped error variable `e` which is hard to reason about.\n> This variable is not required at all.\n>\n> - We allocate memory in the variable declarations, which is easy to\n> miss.\n>\n> Refactor the function so that it becomes more maintainable in the\n> future.\n>\n> Signed-off-by: Patrick Steinhardt \n"},{"id":"487665","messageId":"xmqqmssl44wo.fsf@gitster.g","threadId":"60822","inReplyTo":"0597e6944a1a65720d050f47bc82766d5bcf860b.1706687982.git.ps@pks.im","subject":"Re: [PATCH 1/9] reftable: introduce macros to grow arrays","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-01-31T20:35:51Z","receivedAt":"2024-01-31T20:35:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> patterns in the reftable library. For In most cases, we end up only having a\n> single item in the array, so the initial capacity that our global growth\n> factor uses (which is 24), significantly overallocates in a lot of code\n> paths. \n\nYou need to know not just that you very often initially have only\none but you rarely grow it beyond 3, or something like that to\nexplain \"significantly overallocates\", though.\n\n> This effect is indeed measurable:\n\nAnd measuring is very good, but I somehow expected that you would\nmeasure not the time (if you often under-allocate and end up\nreallocating too many times, it might consume more time, though) but\nthe peak memory usage. I cannot quite tell what to think of that 2%\ntime difference.\n\n> Convert the reftable library to use these new macros.\n\nIn any case, the conversion shortens the code and is a good thing.\nI wish we had a way to tell these macros that we are actually using\nthe same single allocator (which we are doing in our code when\nlinking the reftable thing to us anyway), which would have made this\neven simpler because you did not have to introduce separate macros..\n\n> Signed-off-by: Patrick Steinhardt \n> ---\n> reftable/basics.c | 8 ++------\n> reftable/basics.h | 11 +++++++++++\n> reftable/block.c | 7 +------\n> reftable/merged_test.c | 20 ++++++--------------\n> reftable/pq.c | 8 ++------\n> reftable/stack.c | 29 ++++++++++++-----------------\n> reftable/writer.c | 14 ++------------\n> 7 files changed, 36 insertions(+), 61 deletions(-)\n>\n> diff --git a/reftable/basics.c b/reftable/basics.c\n> index f761e48028..af9004cec2 100644\n> --- a/reftable/basics.c\n> +++ b/reftable/basics.c\n> @@ -89,17 +89,13 @@ void parse_names(char *buf, int size, char ***namesp)\n> \t\t\tnext = end;\n> \t\t}\n> \t\tif (p < next) {\n> -\t\t\tif (names_len == names_cap) {\n> -\t\t\t\tnames_cap = 2 * names_cap + 1;\n> -\t\t\t\tnames = reftable_realloc(\n> -\t\t\t\t\tnames, names_cap * sizeof(*names));\n> -\t\t\t}\n> +\t\t\tREFTABLE_ALLOC_GROW(names, names_len + 1, names_cap);\n> \t\t\tnames[names_len++] = xstrdup(p);\n> \t\t}\n> \t\tp = next + 1;\n> \t}\n> \n> -\tnames = reftable_realloc(names, (names_len + 1) * sizeof(*names));\n> +\tREFTABLE_REALLOC_ARRAY(names, names_len + 1);\n> \tnames[names_len] = NULL;\n> \t*namesp = names;\n> }\n> diff --git a/reftable/basics.h b/reftable/basics.h\n> index 096b36862b..2f855cd724 100644\n> --- a/reftable/basics.h\n> +++ b/reftable/basics.h\n> @@ -53,6 +53,17 @@ void *reftable_realloc(void *p, size_t sz);\n> void reftable_free(void *p);\n> void *reftable_calloc(size_t sz);\n> \n> +#define REFTABLE_REALLOC_ARRAY(x, alloc) (x) = reftable_realloc((x), st_mult(sizeof(*(x)), (alloc)))\n> +#define REFTABLE_ALLOC_GROW(x, nr, alloc) \\\n> +\tdo { \\\n> +\t\tif ((nr) > alloc) { \\\n> +\t\t\talloc = 2 * (alloc) + 1; \\\n> +\t\t\tif (alloc < (nr)) \\\n> +\t\t\t\talloc = (nr); \\\n> +\t\t\tREFTABLE_REALLOC_ARRAY(x, alloc); \\\n> +\t\t} \\\n> +\t} while (0)\n> +\n> /* Find the longest shared prefix size of `a` and `b` */\n> struct strbuf;\n> int common_prefix_size(struct strbuf *a, struct strbuf *b);\n> diff --git a/reftable/block.c b/reftable/block.c\n> index 1df3d8a0f0..6952d0facf 100644\n> --- a/reftable/block.c\n> +++ b/reftable/block.c\n> @@ -51,12 +51,7 @@ static int block_writer_register_restart(struct block_writer *w, int n,\n> \tif (2 + 3 * rlen + n > w->block_size - w->next)\n> \t\treturn -1;\n> \tif (is_restart) {\n> -\t\tif (w->restart_len == w->restart_cap) {\n> -\t\t\tw->restart_cap = w->restart_cap * 2 + 1;\n> -\t\t\tw->restarts = reftable_realloc(\n> -\t\t\t\tw->restarts, sizeof(uint32_t) * w->restart_cap);\n> -\t\t}\n> -\n> +\t\tREFTABLE_ALLOC_GROW(w->restarts, w->restart_len + 1, w->restart_cap);\n> \t\tw->restarts[w->restart_len++] = w->next;\n> \t}\n> \n> diff --git a/reftable/merged_test.c b/reftable/merged_test.c\n> index 46908f738f..e05351e035 100644\n> --- a/reftable/merged_test.c\n> +++ b/reftable/merged_test.c\n> @@ -231,14 +231,10 @@ static void test_merged(void)\n> \twhile (len < 100) { /* cap loops/recursion. */\n> \t\tstruct reftable_ref_record ref = { NULL };\n> \t\tint err = reftable_iterator_next_ref(&it, &ref);\n> -\t\tif (err > 0) {\n> +\t\tif (err > 0)\n> \t\t\tbreak;\n> -\t\t}\n> -\t\tif (len == cap) {\n> -\t\t\tcap = 2 * cap + 1;\n> -\t\t\tout = reftable_realloc(\n> -\t\t\t\tout, sizeof(struct reftable_ref_record) * cap);\n> -\t\t}\n> +\n> +\t\tREFTABLE_ALLOC_GROW(out, len + 1, cap);\n> \t\tout[len++] = ref;\n> \t}\n> \treftable_iterator_destroy(&it);\n> @@ -368,14 +364,10 @@ static void test_merged_logs(void)\n> \twhile (len < 100) { /* cap loops/recursion. */\n> \t\tstruct reftable_log_record log = { NULL };\n> \t\tint err = reftable_iterator_next_log(&it, &log);\n> -\t\tif (err > 0) {\n> +\t\tif (err > 0)\n> \t\t\tbreak;\n> -\t\t}\n> -\t\tif (len == cap) {\n> -\t\t\tcap = 2 * cap + 1;\n> -\t\t\tout = reftable_realloc(\n> -\t\t\t\tout, sizeof(struct reftable_log_record) * cap);\n> -\t\t}\n> +\n> +\t\tREFTABLE_ALLOC_GROW(out, len + 1, cap);\n> \t\tout[len++] = log;\n> \t}\n> \treftable_iterator_destroy(&it);\n> diff --git a/reftable/pq.c b/reftable/pq.c\n> index dcefeb793a..2461daf5ff 100644\n> --- a/reftable/pq.c\n> +++ b/reftable/pq.c\n> @@ -75,13 +75,9 @@ void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry\n> {\n> \tint i = 0;\n> \n> -\tif (pq->len == pq->cap) {\n> -\t\tpq->cap = 2 * pq->cap + 1;\n> -\t\tpq->heap = reftable_realloc(pq->heap,\n> -\t\t\t\t\t pq->cap * sizeof(struct pq_entry));\n> -\t}\n> -\n> +\tREFTABLE_ALLOC_GROW(pq->heap, pq->len + 1, pq->cap);\n> \tpq->heap[pq->len++] = *e;\n> +\n> \ti = pq->len - 1;\n> \twhile (i > 0) {\n> \t\tint j = (i - 1) / 2;\n> diff --git a/reftable/stack.c b/reftable/stack.c\n> index bf3869ce70..1dfab99e96 100644\n> --- a/reftable/stack.c\n> +++ b/reftable/stack.c\n> @@ -551,7 +551,7 @@ struct reftable_addition {\n> \tstruct reftable_stack *stack;\n> \n> \tchar **new_tables;\n> -\tint new_tables_len;\n> +\tsize_t new_tables_len, new_tables_cap;\n> \tuint64_t next_update_index;\n> };\n> \n> @@ -602,8 +602,9 @@ static int reftable_stack_init_addition(struct reftable_addition *add,\n> \n> static void reftable_addition_close(struct reftable_addition *add)\n> {\n> -\tint i = 0;\n> \tstruct strbuf nm = STRBUF_INIT;\n> +\tsize_t i;\n> +\n> \tfor (i = 0; i < add->new_tables_len; i++) {\n> \t\tstack_filename(&nm, add->stack, add->new_tables[i]);\n> \t\tunlink(nm.buf);\n> @@ -613,6 +614,7 @@ static void reftable_addition_close(struct reftable_addition *add)\n> \treftable_free(add->new_tables);\n> \tadd->new_tables = NULL;\n> \tadd->new_tables_len = 0;\n> +\tadd->new_tables_cap = 0;\n> \n> \tdelete_tempfile(&add->lock_file);\n> \tstrbuf_release(&nm);\n> @@ -631,8 +633,8 @@ int reftable_addition_commit(struct reftable_addition *add)\n> {\n> \tstruct strbuf table_list = STRBUF_INIT;\n> \tint lock_file_fd = get_tempfile_fd(add->lock_file);\n> -\tint i = 0;\n> \tint err = 0;\n> +\tsize_t i;\n> \n> \tif (add->new_tables_len == 0)\n> \t\tgoto done;\n> @@ -660,12 +662,12 @@ int reftable_addition_commit(struct reftable_addition *add)\n> \t}\n> \n> \t/* success, no more state to clean up. */\n> -\tfor (i = 0; i < add->new_tables_len; i++) {\n> +\tfor (i = 0; i < add->new_tables_len; i++)\n> \t\treftable_free(add->new_tables[i]);\n> -\t}\n> \treftable_free(add->new_tables);\n> \tadd->new_tables = NULL;\n> \tadd->new_tables_len = 0;\n> +\tadd->new_tables_cap = 0;\n> \n> \terr = reftable_stack_reload_maybe_reuse(add->stack, 1);\n> \tif (err)\n> @@ -792,11 +794,9 @@ int reftable_addition_add(struct reftable_addition *add,\n> \t\tgoto done;\n> \t}\n> \n> -\tadd->new_tables = reftable_realloc(add->new_tables,\n> -\t\t\t\t\t sizeof(*add->new_tables) *\n> -\t\t\t\t\t\t (add->new_tables_len + 1));\n> -\tadd->new_tables[add->new_tables_len] = strbuf_detach(&next_name, NULL);\n> -\tadd->new_tables_len++;\n> +\tREFTABLE_ALLOC_GROW(add->new_tables, add->new_tables_len + 1,\n> +\t\t\t add->new_tables_cap);\n> +\tadd->new_tables[add->new_tables_len++] = strbuf_detach(&next_name, NULL);\n> done:\n> \tif (tab_fd > 0) {\n> \t\tclose(tab_fd);\n> @@ -1367,17 +1367,12 @@ static int stack_check_addition(struct reftable_stack *st,\n> \twhile (1) {\n> \t\tstruct reftable_ref_record ref = { NULL };\n> \t\terr = reftable_iterator_next_ref(&it, &ref);\n> -\t\tif (err > 0) {\n> +\t\tif (err > 0)\n> \t\t\tbreak;\n> -\t\t}\n> \t\tif (err < 0)\n> \t\t\tgoto done;\n> \n> -\t\tif (len >= cap) {\n> -\t\t\tcap = 2 * cap + 1;\n> -\t\t\trefs = reftable_realloc(refs, cap * sizeof(refs[0]));\n> -\t\t}\n> -\n> +\t\tREFTABLE_ALLOC_GROW(refs, len + 1, cap);\n> \t\trefs[len++] = ref;\n> \t}\n> \n> diff --git a/reftable/writer.c b/reftable/writer.c\n> index ee4590e20f..4483bb21c3 100644\n> --- a/reftable/writer.c\n> +++ b/reftable/writer.c\n> @@ -200,12 +200,7 @@ static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n> \t\treturn;\n> \t}\n> \n> -\tif (key->offset_len == key->offset_cap) {\n> -\t\tkey->offset_cap = 2 * key->offset_cap + 1;\n> -\t\tkey->offsets = reftable_realloc(\n> -\t\t\tkey->offsets, sizeof(uint64_t) * key->offset_cap);\n> -\t}\n> -\n> +\tREFTABLE_ALLOC_GROW(key->offsets, key->offset_len + 1, key->offset_cap);\n> \tkey->offsets[key->offset_len++] = off;\n> }\n> \n> @@ -674,12 +669,7 @@ static int writer_flush_nonempty_block(struct reftable_writer *w)\n> \tif (err < 0)\n> \t\treturn err;\n> \n> -\tif (w->index_cap == w->index_len) {\n> -\t\tw->index_cap = 2 * w->index_cap + 1;\n> -\t\tw->index = reftable_realloc(\n> -\t\t\tw->index,\n> -\t\t\tsizeof(struct reftable_index_record) * w->index_cap);\n> -\t}\n> +\tREFTABLE_ALLOC_GROW(w->index, w->index_len + 1, w->index_cap);\n> \n> \tir.offset = w->next;\n> \tstrbuf_reset(&ir.last_key);\n"},{"id":"487698","messageId":"ZbtIQ3Hk6Mgvkv4j@tanuki","threadId":"60822","inReplyTo":"xmqqmssl44wo.fsf@gitster.g","subject":"Re: [PATCH 1/9] reftable: introduce macros to grow arrays","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-01T07:29:07Z","receivedAt":"2024-02-01T07:29:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Jan 31, 2024 at 12:35:51PM -0800, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > patterns in the reftable library. For In most cases, we end up only having a\n> > single item in the array, so the initial capacity that our global growth\n> > factor uses (which is 24), significantly overallocates in a lot of code\n> > paths. \n> \n> You need to know not just that you very often initially have only\n> one but you rarely grow it beyond 3, or something like that to\n> explain \"significantly overallocates\", though.\n\nTrue.\n\n> > This effect is indeed measurable:\n> \n> And measuring is very good, but I somehow expected that you would\n> measure not the time (if you often under-allocate and end up\n> reallocating too many times, it might consume more time, though) but\n> the peak memory usage. I cannot quite tell what to think of that 2%\n> time difference.\n\nVery good point indeed. I don't think peak memory usage is really all\nthat helpful either because the problem is not that we are allocating\narrays that we keep around all the time, but many small arrays which are\nshort lived. So what is telling is the total number of bytes we end up\nallocating:\n\n Before change:\n\n HEAP SUMMARY:\n in use at exit: 671,983 bytes in 152 blocks\n total heap usage: 3,843,446 allocs, 3,843,294 frees, 223,761,402 bytes allocated\n\n Growth factor (alloc * 2 + 1):\n\n HEAP SUMMARY:\n in use at exit: 671,983 bytes in 152 blocks\n total heap usage: 3,843,446 allocs, 3,843,294 frees, 223,761,410 bytes allocated\n\n Growth factor (alloc + 16) * 2 / 3:\n\n HEAP SUMMARY:\n in use at exit: 671,983 bytes in 152 blocks\n total heap usage: 3,833,673 allocs, 3,833,521 frees, 4,728,251,742 bytes allocated\n\nAllocating 21 times as many bytes with our default growth factor should\nbe a much more compelling argument why we don't actually want to use it\ncompared to the 2% speedup.\n\nIt's somewhat amazing though that this huge difference only makes for a\n2% speedup.\n\n> > Convert the reftable library to use these new macros.\n> \n> In any case, the conversion shortens the code and is a good thing.\n> I wish we had a way to tell these macros that we are actually using\n> the same single allocator (which we are doing in our code when\n> linking the reftable thing to us anyway), which would have made this\n> even simpler because you did not have to introduce separate macros..\n\nYeah, I wasn't a huge fan of this, either. I initially just wanted to\nreuse our usual macros, but when I noticed the resulting difference in\nallocated bytes I already had two arguments against this: the fact that\nwe have pluggable allocators in the reftable library and the growth\nfactor. While we could make our macros more flexible so that they can\naccommodate for both, I don't think that the result would be pretty.\n\nSo at that point I decided to just duplicate the code. It still ends up\nremoving a lot of code duplication in the reftable library itself, so I\ndon't think it is too bad.\n\nPatrick\n"},{"id":"487699","messageId":"cover.1706772591.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706687982.git.ps@pks.im","subject":"[PATCH v2 0/9] reftable: code style improvements","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-01T07:32:50Z","receivedAt":"2024-02-01T07:32:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the second version of my patch series that aims to improve the\ncode style of the reftable library.\n\nThe only changes compared to v1 are improvements for the commit\nmessages. The edit for commit 1 should hopefully make it a much more\ncompelling argument why we don't want to use the default growth factor\nused by our own `ALLOC_GROW()` macro.\n\nThanks!\n\nPatrick\n\nPatrick Steinhardt (9):\n reftable: introduce macros to grow arrays\n reftable: introduce macros to allocate arrays\n reftable/stack: fix parameter validation when compacting range\n reftable/stack: index segments with `size_t`\n reftable/stack: use `size_t` to track stack slices during compaction\n reftable/stack: use `size_t` to track stack length\n reftable/merged: refactor seeking of records\n reftable/merged: refactor initialization of iterators\n reftable/record: improve semantics when initializing records\n\n reftable/basics.c | 15 ++--\n reftable/basics.h | 17 ++++-\n reftable/block.c | 25 +++---\n reftable/block_test.c | 2 +-\n reftable/blocksource.c | 4 +-\n reftable/iter.c | 3 +-\n reftable/merged.c | 100 +++++++++++-------------\n reftable/merged_test.c | 52 ++++++-------\n reftable/pq.c | 8 +-\n reftable/publicbasics.c | 3 +-\n reftable/reader.c | 12 ++-\n reftable/readwrite_test.c | 8 +-\n reftable/record.c | 43 +++--------\n reftable/record.h | 10 +--\n reftable/record_test.c | 8 +-\n reftable/refname.c | 4 +-\n reftable/reftable-merged.h | 2 +-\n reftable/stack.c | 151 +++++++++++++++++--------------------\n reftable/stack.h | 6 +-\n reftable/stack_test.c | 7 +-\n reftable/tree.c | 4 +-\n reftable/writer.c | 21 ++----\n 22 files changed, 221 insertions(+), 284 deletions(-)\n\nRange-diff against v1:\n 1: 0597e6944a ! 1: 12bd721ddf reftable: introduce macros to grow arrays\n @@ Commit message\n Throughout the reftable library we have many cases where we need to grow\n arrays. In order to avoid too many reallocations, we roughly double the\n capacity of the array on each iteration. The resulting code pattern is\n - thus duplicated across many sites.\n + duplicated across many sites.\n \n We have similar patterns in our main codebase, which is why we have\n eventually introduced an `ALLOC_GROW()` macro to abstract it away and\n @@ Commit message\n + 16) * 3 / 2`.\n \n The second change is because we know a bit more about the allocation\n - patterns in the reftable library. For In most cases, we end up only having a\n - single item in the array, so the initial capacity that our global growth\n - factor uses (which is 24), significantly overallocates in a lot of code\n - paths. This effect is indeed measurable:\n + patterns in the reftable library. In most cases, we end up only having a\n + handful of items in the array and don't end up growing them. The initial\n + capacity that our normal growth factor uses (which is 24) would thus end\n + up over-allocating in a lot of code paths. This effect is measurable:\n \n - Benchmark 1: update-ref: create many refs (growth factor = 2 * cap + 1)\n - Time (mean ± σ): 4.834 s ± 0.020 s [User: 2.219 s, System: 2.614 s]\n - Range (min … max): 4.793 s … 4.871 s 20 runs\n + - Before change:\n \n - Benchmark 2: update-ref: create many refs (growth factor = (cap + 16) * 3 + 2)\n - Time (mean ± σ): 4.933 s ± 0.021 s [User: 2.325 s, System: 2.607 s]\n - Range (min … max): 4.889 s … 4.962 s 20 runs\n + HEAP SUMMARY:\n + in use at exit: 671,983 bytes in 152 blocks\n + total heap usage: 3,843,446 allocs, 3,843,294 frees, 223,761,402 bytes allocated\n \n - Summary\n - update-ref: create many refs (growth factor = 2 * cap + 1) ran\n - 1.02 ± 0.01 times faster than update-ref: create many refs (growth factor = (cap + 16) * 3 + 2)\n + - After change with a growth factor of `(2 * alloc + 1)`:\n +\n + HEAP SUMMARY:\n + in use at exit: 671,983 bytes in 152 blocks\n + total heap usage: 3,843,446 allocs, 3,843,294 frees, 223,761,410 bytes allocated\n +\n + - After change with a growth factor of `(alloc + 16)* 2 / 3`:\n +\n + HEAP SUMMARY:\n + in use at exit: 671,983 bytes in 152 blocks\n + total heap usage: 3,833,673 allocs, 3,833,521 frees, 4,728,251,742 bytes allocated\n +\n + While the total heap usage is roughly the same, we do end up allocating\n + significantly more bytes with our usual growth factor (in fact, roughly\n + 21 times as many).\n \n Convert the reftable library to use these new macros.\n \n 2: eee6580c84 = 2: 2dde581a02 reftable: introduce macros to allocate arrays\n 3: e2d05f7c38 = 3: f134702dc5 reftable/stack: fix parameter validation when compacting range\n 4: b8b2cce742 = 4: 50dac904e8 reftable/stack: index segments with `size_t`\n 5: 2f6a69aa14 = 5: a5ffbf09dd reftable/stack: use `size_t` to track stack slices during compaction\n 6: 1ee9a4477f = 6: 55605fb53b reftable/stack: use `size_t` to track stack length\n 7: 00aeaeee63 ! 7: 80cf2fd272 reftable/merged: refactor seeking of records\n @@ Commit message\n to read and does not conform to our coding style in multiple ways:\n \n - We have multiple exit paths where we release resources even though\n - that is not really necessary\n + that is not really necessary.\n \n - We use a scoped error variable `e` which is hard to reason about.\n This variable is not required at all.\n 8: 31864740e9 = 8: 8c1be2b159 reftable/merged: refactor initialization of iterators\n 9: fd8a1ce99d = 9: c39d7e30e7 reftable/record: improve semantics when initializing records\n\nbase-commit: bc7ee2e5e16f0d1e710ef8fab3db59ab11f2bbe7\n-- \n2.43.GIT\n\n"},{"id":"487700","messageId":"12bd721ddff7020eb9e9ebd4e797d50193250bc0.1706772591.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706772591.git.ps@pks.im","subject":"[PATCH v2 1/9] reftable: introduce macros to grow arrays","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-01T07:32:54Z","receivedAt":"2024-02-01T07:32:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Throughout the reftable library we have many cases where we need to grow\narrays. In order to avoid too many reallocations, we roughly double the\ncapacity of the array on each iteration. The resulting code pattern is\nduplicated across many sites.\n\nWe have similar patterns in our main codebase, which is why we have\neventually introduced an `ALLOC_GROW()` macro to abstract it away and\navoid some code duplication. We cannot easily reuse this macro here\nthough because `ALLOC_GROW()` uses `REALLOC_ARRAY()`, which in turn will\ncall realloc(3P) to grow the array. The reftable code is structured as a\nlibrary though (even if the boundaries are fuzzy), and one property this\nbrings with it is that it is possible to plug in your own allocators. So\ninstead of using realloc(3P), we need to use `reftable_realloc()` that\nknows to use the user-provided implementation.\n\nSo let's introduce two new macros `REFTABLE_REALLOC_ARRAY()` and\n`REFTABLE_ALLOC_GROW()` that mirror what we do in our main codebase,\nwith two modifications:\n\n - They use `reftable_realloc()`, as explained above.\n\n - They use a different growth factor of `2 * cap + 1` instead of `(cap\n + 16) * 3 / 2`.\n\nThe second change is because we know a bit more about the allocation\npatterns in the reftable library. In most cases, we end up only having a\nhandful of items in the array and don't end up growing them. The initial\ncapacity that our normal growth factor uses (which is 24) would thus end\nup over-allocating in a lot of code paths. This effect is measurable:\n\n - Before change:\n\n HEAP SUMMARY:\n in use at exit: 671,983 bytes in 152 blocks\n total heap usage: 3,843,446 allocs, 3,843,294 frees, 223,761,402 bytes allocated\n\n - After change with a growth factor of `(2 * alloc + 1)`:\n\n HEAP SUMMARY:\n in use at exit: 671,983 bytes in 152 blocks\n total heap usage: 3,843,446 allocs, 3,843,294 frees, 223,761,410 bytes allocated\n\n - After change with a growth factor of `(alloc + 16)* 2 / 3`:\n\n HEAP SUMMARY:\n in use at exit: 671,983 bytes in 152 blocks\n total heap usage: 3,833,673 allocs, 3,833,521 frees, 4,728,251,742 bytes allocated\n\nWhile the total heap usage is roughly the same, we do end up allocating\nsignificantly more bytes with our usual growth factor (in fact, roughly\n21 times as many).\n\nConvert the reftable library to use these new macros.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 8 ++------\n reftable/basics.h | 11 +++++++++++\n reftable/block.c | 7 +------\n reftable/merged_test.c | 20 ++++++--------------\n reftable/pq.c | 8 ++------\n reftable/stack.c | 29 ++++++++++++-----------------\n reftable/writer.c | 14 ++------------\n 7 files changed, 36 insertions(+), 61 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex f761e48028..af9004cec2 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -89,17 +89,13 @@ void parse_names(char *buf, int size, char ***namesp)\n \t\t\tnext = end;\n \t\t}\n \t\tif (p < next) {\n-\t\t\tif (names_len == names_cap) {\n-\t\t\t\tnames_cap = 2 * names_cap + 1;\n-\t\t\t\tnames = reftable_realloc(\n-\t\t\t\t\tnames, names_cap * sizeof(*names));\n-\t\t\t}\n+\t\t\tREFTABLE_ALLOC_GROW(names, names_len + 1, names_cap);\n \t\t\tnames[names_len++] = xstrdup(p);\n \t\t}\n \t\tp = next + 1;\n \t}\n \n-\tnames = reftable_realloc(names, (names_len + 1) * sizeof(*names));\n+\tREFTABLE_REALLOC_ARRAY(names, names_len + 1);\n \tnames[names_len] = NULL;\n \t*namesp = names;\n }\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 096b36862b..2f855cd724 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -53,6 +53,17 @@ void *reftable_realloc(void *p, size_t sz);\n void reftable_free(void *p);\n void *reftable_calloc(size_t sz);\n \n+#define REFTABLE_REALLOC_ARRAY(x, alloc) (x) = reftable_realloc((x), st_mult(sizeof(*(x)), (alloc)))\n+#define REFTABLE_ALLOC_GROW(x, nr, alloc) \\\n+\tdo { \\\n+\t\tif ((nr) > alloc) { \\\n+\t\t\talloc = 2 * (alloc) + 1; \\\n+\t\t\tif (alloc < (nr)) \\\n+\t\t\t\talloc = (nr); \\\n+\t\t\tREFTABLE_REALLOC_ARRAY(x, alloc); \\\n+\t\t} \\\n+\t} while (0)\n+\n /* Find the longest shared prefix size of `a` and `b` */\n struct strbuf;\n int common_prefix_size(struct strbuf *a, struct strbuf *b);\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 1df3d8a0f0..6952d0facf 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -51,12 +51,7 @@ static int block_writer_register_restart(struct block_writer *w, int n,\n \tif (2 + 3 * rlen + n > w->block_size - w->next)\n \t\treturn -1;\n \tif (is_restart) {\n-\t\tif (w->restart_len == w->restart_cap) {\n-\t\t\tw->restart_cap = w->restart_cap * 2 + 1;\n-\t\t\tw->restarts = reftable_realloc(\n-\t\t\t\tw->restarts, sizeof(uint32_t) * w->restart_cap);\n-\t\t}\n-\n+\t\tREFTABLE_ALLOC_GROW(w->restarts, w->restart_len + 1, w->restart_cap);\n \t\tw->restarts[w->restart_len++] = w->next;\n \t}\n \ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nindex 46908f738f..e05351e035 100644\n--- a/reftable/merged_test.c\n+++ b/reftable/merged_test.c\n@@ -231,14 +231,10 @@ static void test_merged(void)\n \twhile (len < 100) { /* cap loops/recursion. */\n \t\tstruct reftable_ref_record ref = { NULL };\n \t\tint err = reftable_iterator_next_ref(&it, &ref);\n-\t\tif (err > 0) {\n+\t\tif (err > 0)\n \t\t\tbreak;\n-\t\t}\n-\t\tif (len == cap) {\n-\t\t\tcap = 2 * cap + 1;\n-\t\t\tout = reftable_realloc(\n-\t\t\t\tout, sizeof(struct reftable_ref_record) * cap);\n-\t\t}\n+\n+\t\tREFTABLE_ALLOC_GROW(out, len + 1, cap);\n \t\tout[len++] = ref;\n \t}\n \treftable_iterator_destroy(&it);\n@@ -368,14 +364,10 @@ static void test_merged_logs(void)\n \twhile (len < 100) { /* cap loops/recursion. */\n \t\tstruct reftable_log_record log = { NULL };\n \t\tint err = reftable_iterator_next_log(&it, &log);\n-\t\tif (err > 0) {\n+\t\tif (err > 0)\n \t\t\tbreak;\n-\t\t}\n-\t\tif (len == cap) {\n-\t\t\tcap = 2 * cap + 1;\n-\t\t\tout = reftable_realloc(\n-\t\t\t\tout, sizeof(struct reftable_log_record) * cap);\n-\t\t}\n+\n+\t\tREFTABLE_ALLOC_GROW(out, len + 1, cap);\n \t\tout[len++] = log;\n \t}\n \treftable_iterator_destroy(&it);\ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex dcefeb793a..2461daf5ff 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -75,13 +75,9 @@ void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry\n {\n \tint i = 0;\n \n-\tif (pq->len == pq->cap) {\n-\t\tpq->cap = 2 * pq->cap + 1;\n-\t\tpq->heap = reftable_realloc(pq->heap,\n-\t\t\t\t\t pq->cap * sizeof(struct pq_entry));\n-\t}\n-\n+\tREFTABLE_ALLOC_GROW(pq->heap, pq->len + 1, pq->cap);\n \tpq->heap[pq->len++] = *e;\n+\n \ti = pq->len - 1;\n \twhile (i > 0) {\n \t\tint j = (i - 1) / 2;\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex bf3869ce70..1dfab99e96 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -551,7 +551,7 @@ struct reftable_addition {\n \tstruct reftable_stack *stack;\n \n \tchar **new_tables;\n-\tint new_tables_len;\n+\tsize_t new_tables_len, new_tables_cap;\n \tuint64_t next_update_index;\n };\n \n@@ -602,8 +602,9 @@ static int reftable_stack_init_addition(struct reftable_addition *add,\n \n static void reftable_addition_close(struct reftable_addition *add)\n {\n-\tint i = 0;\n \tstruct strbuf nm = STRBUF_INIT;\n+\tsize_t i;\n+\n \tfor (i = 0; i < add->new_tables_len; i++) {\n \t\tstack_filename(&nm, add->stack, add->new_tables[i]);\n \t\tunlink(nm.buf);\n@@ -613,6 +614,7 @@ static void reftable_addition_close(struct reftable_addition *add)\n \treftable_free(add->new_tables);\n \tadd->new_tables = NULL;\n \tadd->new_tables_len = 0;\n+\tadd->new_tables_cap = 0;\n \n \tdelete_tempfile(&add->lock_file);\n \tstrbuf_release(&nm);\n@@ -631,8 +633,8 @@ int reftable_addition_commit(struct reftable_addition *add)\n {\n \tstruct strbuf table_list = STRBUF_INIT;\n \tint lock_file_fd = get_tempfile_fd(add->lock_file);\n-\tint i = 0;\n \tint err = 0;\n+\tsize_t i;\n \n \tif (add->new_tables_len == 0)\n \t\tgoto done;\n@@ -660,12 +662,12 @@ int reftable_addition_commit(struct reftable_addition *add)\n \t}\n \n \t/* success, no more state to clean up. */\n-\tfor (i = 0; i < add->new_tables_len; i++) {\n+\tfor (i = 0; i < add->new_tables_len; i++)\n \t\treftable_free(add->new_tables[i]);\n-\t}\n \treftable_free(add->new_tables);\n \tadd->new_tables = NULL;\n \tadd->new_tables_len = 0;\n+\tadd->new_tables_cap = 0;\n \n \terr = reftable_stack_reload_maybe_reuse(add->stack, 1);\n \tif (err)\n@@ -792,11 +794,9 @@ int reftable_addition_add(struct reftable_addition *add,\n \t\tgoto done;\n \t}\n \n-\tadd->new_tables = reftable_realloc(add->new_tables,\n-\t\t\t\t\t sizeof(*add->new_tables) *\n-\t\t\t\t\t\t (add->new_tables_len + 1));\n-\tadd->new_tables[add->new_tables_len] = strbuf_detach(&next_name, NULL);\n-\tadd->new_tables_len++;\n+\tREFTABLE_ALLOC_GROW(add->new_tables, add->new_tables_len + 1,\n+\t\t\t add->new_tables_cap);\n+\tadd->new_tables[add->new_tables_len++] = strbuf_detach(&next_name, NULL);\n done:\n \tif (tab_fd > 0) {\n \t\tclose(tab_fd);\n@@ -1367,17 +1367,12 @@ static int stack_check_addition(struct reftable_stack *st,\n \twhile (1) {\n \t\tstruct reftable_ref_record ref = { NULL };\n \t\terr = reftable_iterator_next_ref(&it, &ref);\n-\t\tif (err > 0) {\n+\t\tif (err > 0)\n \t\t\tbreak;\n-\t\t}\n \t\tif (err < 0)\n \t\t\tgoto done;\n \n-\t\tif (len >= cap) {\n-\t\t\tcap = 2 * cap + 1;\n-\t\t\trefs = reftable_realloc(refs, cap * sizeof(refs[0]));\n-\t\t}\n-\n+\t\tREFTABLE_ALLOC_GROW(refs, len + 1, cap);\n \t\trefs[len++] = ref;\n \t}\n \ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex ee4590e20f..4483bb21c3 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -200,12 +200,7 @@ static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n \t\treturn;\n \t}\n \n-\tif (key->offset_len == key->offset_cap) {\n-\t\tkey->offset_cap = 2 * key->offset_cap + 1;\n-\t\tkey->offsets = reftable_realloc(\n-\t\t\tkey->offsets, sizeof(uint64_t) * key->offset_cap);\n-\t}\n-\n+\tREFTABLE_ALLOC_GROW(key->offsets, key->offset_len + 1, key->offset_cap);\n \tkey->offsets[key->offset_len++] = off;\n }\n \n@@ -674,12 +669,7 @@ static int writer_flush_nonempty_block(struct reftable_writer *w)\n \tif (err < 0)\n \t\treturn err;\n \n-\tif (w->index_cap == w->index_len) {\n-\t\tw->index_cap = 2 * w->index_cap + 1;\n-\t\tw->index = reftable_realloc(\n-\t\t\tw->index,\n-\t\t\tsizeof(struct reftable_index_record) * w->index_cap);\n-\t}\n+\tREFTABLE_ALLOC_GROW(w->index, w->index_len + 1, w->index_cap);\n \n \tir.offset = w->next;\n \tstrbuf_reset(&ir.last_key);\n-- \n2.43.GIT\n\n"},{"id":"487701","messageId":"2dde581a0256e4634ca4a64f313a98204763906a.1706772591.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706772591.git.ps@pks.im","subject":"[PATCH v2 2/9] reftable: introduce macros to allocate arrays","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-01T07:32:58Z","receivedAt":"2024-02-01T07:33:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Similar to the preceding commit, let's carry over macros to allocate\narrays with `REFTABLE_ALLOC_ARRAY()` and `REFTABLE_CALLOC_ARRAY()`. This\nrequires us to change the signature of `reftable_calloc()`, which only\ntakes a single argument right now and thus puts the burden on the caller\nto calculate the final array's size. This is a net improvement though as\nit means that we can now provide proper overflow checks when multiplying\nthe array size with the member size.\n\nConvert callsites of `reftable_calloc()` to the new signature, using the\nnew macros where possible.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 4 +++-\n reftable/block_test.c | 2 +-\n reftable/blocksource.c | 4 ++--\n reftable/iter.c | 3 +--\n reftable/merged.c | 4 ++--\n reftable/merged_test.c | 22 +++++++++++++---------\n reftable/publicbasics.c | 3 ++-\n reftable/reader.c | 8 +++-----\n reftable/readwrite_test.c | 8 +++++---\n reftable/record_test.c | 4 ++--\n reftable/refname.c | 4 ++--\n reftable/stack.c | 26 ++++++++++++--------------\n reftable/tree.c | 4 ++--\n reftable/writer.c | 7 +++----\n 14 files changed, 53 insertions(+), 50 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 2f855cd724..4c3ac963a3 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -51,8 +51,10 @@ int names_length(char **names);\n void *reftable_malloc(size_t sz);\n void *reftable_realloc(void *p, size_t sz);\n void reftable_free(void *p);\n-void *reftable_calloc(size_t sz);\n+void *reftable_calloc(size_t nelem, size_t elsize);\n \n+#define REFTABLE_ALLOC_ARRAY(x, alloc) (x) = reftable_malloc(st_mult(sizeof(*(x)), (alloc)))\n+#define REFTABLE_CALLOC_ARRAY(x, alloc) (x) = reftable_calloc((alloc), sizeof(*(x)))\n #define REFTABLE_REALLOC_ARRAY(x, alloc) (x) = reftable_realloc((x), st_mult(sizeof(*(x)), (alloc)))\n #define REFTABLE_ALLOC_GROW(x, nr, alloc) \\\n \tdo { \\\ndiff --git a/reftable/block_test.c b/reftable/block_test.c\nindex dedb05c7d8..e162c6e33f 100644\n--- a/reftable/block_test.c\n+++ b/reftable/block_test.c\n@@ -36,7 +36,7 @@ static void test_block_read_write(void)\n \tint j = 0;\n \tstruct strbuf want = STRBUF_INIT;\n \n-\tblock.data = reftable_calloc(block_size);\n+\tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock.source = malloc_block_source();\n \tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex 8c41e3c70f..eeed254ba9 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -29,7 +29,7 @@ static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n {\n \tstruct strbuf *b = v;\n \tassert(off + size <= b->len);\n-\tdest->data = reftable_calloc(size);\n+\tREFTABLE_CALLOC_ARRAY(dest->data, size);\n \tmemcpy(dest->data, b->buf + off, size);\n \tdest->len = size;\n \treturn size;\n@@ -132,7 +132,7 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,\n \t\treturn REFTABLE_IO_ERROR;\n \t}\n \n-\tp = reftable_calloc(sizeof(*p));\n+\tREFTABLE_CALLOC_ARRAY(p, 1);\n \tp->size = st.st_size;\n \tp->data = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tclose(fd);\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex a8d174c040..8b5ebf6183 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -160,8 +160,7 @@ int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n \t\t\t int oid_len, uint64_t *offsets, int offset_len)\n {\n \tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n-\tstruct indexed_table_ref_iter *itr =\n-\t\treftable_calloc(sizeof(struct indexed_table_ref_iter));\n+\tstruct indexed_table_ref_iter *itr = reftable_calloc(1, sizeof(*itr));\n \tint err = 0;\n \n \t*itr = empty;\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex c258ce953e..2031fd51b4 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -190,7 +190,7 @@ int reftable_new_merged_table(struct reftable_merged_table **dest,\n \t\t}\n \t}\n \n-\tm = reftable_calloc(sizeof(struct reftable_merged_table));\n+\tREFTABLE_CALLOC_ARRAY(m, 1);\n \tm->stack = stack;\n \tm->stack_len = n;\n \tm->min = first_min;\n@@ -240,7 +240,7 @@ static int merged_table_seek_record(struct reftable_merged_table *mt,\n \t\t\t\t struct reftable_record *rec)\n {\n \tstruct reftable_iterator *iters = reftable_calloc(\n-\t\tsizeof(struct reftable_iterator) * mt->stack_len);\n+\t\tmt->stack_len, sizeof(*iters));\n \tstruct merged_iter merged = {\n \t\t.stack = iters,\n \t\t.typ = reftable_record_type(rec),\ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nindex e05351e035..e233a9d581 100644\n--- a/reftable/merged_test.c\n+++ b/reftable/merged_test.c\n@@ -93,10 +93,12 @@ merged_table_from_records(struct reftable_ref_record **refs,\n \tint i = 0;\n \tstruct reftable_merged_table *mt = NULL;\n \tint err;\n-\tstruct reftable_table *tabs =\n-\t\treftable_calloc(n * sizeof(struct reftable_table));\n-\t*readers = reftable_calloc(n * sizeof(struct reftable_reader *));\n-\t*source = reftable_calloc(n * sizeof(**source));\n+\tstruct reftable_table *tabs;\n+\n+\tREFTABLE_CALLOC_ARRAY(tabs, n);\n+\tREFTABLE_CALLOC_ARRAY(*readers, n);\n+\tREFTABLE_CALLOC_ARRAY(*source, n);\n+\n \tfor (i = 0; i < n; i++) {\n \t\twrite_test_table(&buf[i], refs[i], sizes[i]);\n \t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n@@ -266,10 +268,12 @@ merged_table_from_log_records(struct reftable_log_record **logs,\n \tint i = 0;\n \tstruct reftable_merged_table *mt = NULL;\n \tint err;\n-\tstruct reftable_table *tabs =\n-\t\treftable_calloc(n * sizeof(struct reftable_table));\n-\t*readers = reftable_calloc(n * sizeof(struct reftable_reader *));\n-\t*source = reftable_calloc(n * sizeof(**source));\n+\tstruct reftable_table *tabs;\n+\n+\tREFTABLE_CALLOC_ARRAY(tabs, n);\n+\tREFTABLE_CALLOC_ARRAY(*readers, n);\n+\tREFTABLE_CALLOC_ARRAY(*source, n);\n+\n \tfor (i = 0; i < n; i++) {\n \t\twrite_test_log_table(&buf[i], logs[i], sizes[i], i + 1);\n \t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n@@ -412,7 +416,7 @@ static void test_default_write_opts(void)\n \t};\n \tint err;\n \tstruct reftable_block_source source = { NULL };\n-\tstruct reftable_table *tab = reftable_calloc(sizeof(*tab) * 1);\n+\tstruct reftable_table *tab = reftable_calloc(1, sizeof(*tab));\n \tuint32_t hash_id;\n \tstruct reftable_reader *rd = NULL;\n \tstruct reftable_merged_table *merged = NULL;\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\nindex bcb82530d6..44b84a125e 100644\n--- a/reftable/publicbasics.c\n+++ b/reftable/publicbasics.c\n@@ -37,8 +37,9 @@ void reftable_free(void *p)\n \t\tfree(p);\n }\n \n-void *reftable_calloc(size_t sz)\n+void *reftable_calloc(size_t nelem, size_t elsize)\n {\n+\tsize_t sz = st_mult(nelem, elsize);\n \tvoid *p = reftable_malloc(sz);\n \tmemset(p, 0, sz);\n \treturn p;\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 64dc366fb1..3e0de5e8ad 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -539,8 +539,7 @@ static int reader_seek_indexed(struct reftable_reader *r,\n \n \tif (err == 0) {\n \t\tstruct table_iter empty = TABLE_ITER_INIT;\n-\t\tstruct table_iter *malloced =\n-\t\t\treftable_calloc(sizeof(struct table_iter));\n+\t\tstruct table_iter *malloced = reftable_calloc(1, sizeof(*malloced));\n \t\t*malloced = empty;\n \t\ttable_iter_copy_from(malloced, &next);\n \t\titerator_from_table_iter(it, malloced);\n@@ -635,8 +634,7 @@ void reader_close(struct reftable_reader *r)\n int reftable_new_reader(struct reftable_reader **p,\n \t\t\tstruct reftable_block_source *src, char const *name)\n {\n-\tstruct reftable_reader *rd =\n-\t\treftable_calloc(sizeof(struct reftable_reader));\n+\tstruct reftable_reader *rd = reftable_calloc(1, sizeof(*rd));\n \tint err = init_reader(rd, src, name);\n \tif (err == 0) {\n \t\t*p = rd;\n@@ -711,7 +709,7 @@ static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n \t\t\t\t\t uint8_t *oid)\n {\n \tstruct table_iter ti_empty = TABLE_ITER_INIT;\n-\tstruct table_iter *ti = reftable_calloc(sizeof(struct table_iter));\n+\tstruct table_iter *ti = reftable_calloc(1, sizeof(*ti));\n \tstruct filtering_ref_iterator *filter = NULL;\n \tstruct filtering_ref_iterator empty = FILTERING_REF_ITERATOR_INIT;\n \tint oid_len = hash_size(r->hash_id);\ndiff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c\nindex b8a3224016..91ea7a10ef 100644\n--- a/reftable/readwrite_test.c\n+++ b/reftable/readwrite_test.c\n@@ -56,7 +56,9 @@ static void write_table(char ***names, struct strbuf *buf, int N,\n \tint i = 0, n;\n \tstruct reftable_log_record log = { NULL };\n \tconst struct reftable_stats *stats = NULL;\n-\t*names = reftable_calloc(sizeof(char *) * (N + 1));\n+\n+\tREFTABLE_CALLOC_ARRAY(*names, N + 1);\n+\n \treftable_writer_set_limits(w, update_index, update_index);\n \tfor (i = 0; i < N; i++) {\n \t\tchar name[100];\n@@ -188,7 +190,7 @@ static void test_log_overflow(void)\n static void test_log_write_read(void)\n {\n \tint N = 2;\n-\tchar **names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tchar **names = reftable_calloc(N + 1, sizeof(*names));\n \tint err;\n \tstruct reftable_write_options opts = {\n \t\t.block_size = 256,\n@@ -519,7 +521,7 @@ static void test_table_read_write_seek_index(void)\n static void test_table_refs_for(int indexed)\n {\n \tint N = 50;\n-\tchar **want_names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tchar **want_names = reftable_calloc(N + 1, sizeof(*want_names));\n \tint want_names_len = 0;\n \tuint8_t want_hash[GIT_SHA1_RAWSZ];\n \ndiff --git a/reftable/record_test.c b/reftable/record_test.c\nindex 2876db7d27..999366ad46 100644\n--- a/reftable/record_test.c\n+++ b/reftable/record_test.c\n@@ -231,8 +231,8 @@ static void test_reftable_log_record_roundtrip(void)\n \t\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n \t\t\t\t.value = {\n \t\t\t\t\t.update = {\n-\t\t\t\t\t\t.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n-\t\t\t\t\t\t.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t\t\t.new_hash = reftable_calloc(GIT_SHA1_RAWSZ, 1),\n+\t\t\t\t\t\t.old_hash = reftable_calloc(GIT_SHA1_RAWSZ, 1),\n \t\t\t\t\t\t.name = xstrdup(\"old name\"),\n \t\t\t\t\t\t.email = xstrdup(\"old@email\"),\n \t\t\t\t\t\t.message = xstrdup(\"old message\"),\ndiff --git a/reftable/refname.c b/reftable/refname.c\nindex 9573496932..7570e4acf9 100644\n--- a/reftable/refname.c\n+++ b/reftable/refname.c\n@@ -140,8 +140,8 @@ int validate_ref_record_addition(struct reftable_table tab,\n {\n \tstruct modification mod = {\n \t\t.tab = tab,\n-\t\t.add = reftable_calloc(sizeof(char *) * sz),\n-\t\t.del = reftable_calloc(sizeof(char *) * sz),\n+\t\t.add = reftable_calloc(sz, sizeof(*mod.add)),\n+\t\t.del = reftable_calloc(sz, sizeof(*mod.del)),\n \t};\n \tint i = 0;\n \tint err = 0;\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 1dfab99e96..d084823a92 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -50,8 +50,7 @@ static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz)\n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \t\t struct reftable_write_options config)\n {\n-\tstruct reftable_stack *p =\n-\t\treftable_calloc(sizeof(struct reftable_stack));\n+\tstruct reftable_stack *p = reftable_calloc(1, sizeof(*p));\n \tstruct strbuf list_file_name = STRBUF_INIT;\n \tint err = 0;\n \n@@ -114,7 +113,7 @@ int read_lines(const char *filename, char ***namesp)\n \tint err = 0;\n \tif (fd < 0) {\n \t\tif (errno == ENOENT) {\n-\t\t\t*namesp = reftable_calloc(sizeof(char *));\n+\t\t\tREFTABLE_CALLOC_ARRAY(*namesp, 1);\n \t\t\treturn 0;\n \t\t}\n \n@@ -191,8 +190,7 @@ void reftable_stack_destroy(struct reftable_stack *st)\n static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n \t\t\t\t\t\t int cur_len)\n {\n-\tstruct reftable_reader **cur =\n-\t\treftable_calloc(sizeof(struct reftable_reader *) * cur_len);\n+\tstruct reftable_reader **cur = reftable_calloc(cur_len, sizeof(*cur));\n \tint i = 0;\n \tfor (i = 0; i < cur_len; i++) {\n \t\tcur[i] = st->readers[i];\n@@ -208,9 +206,9 @@ static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n \tint err = 0;\n \tint names_len = names_length(names);\n \tstruct reftable_reader **new_readers =\n-\t\treftable_calloc(sizeof(struct reftable_reader *) * names_len);\n+\t\treftable_calloc(names_len, sizeof(*new_readers));\n \tstruct reftable_table *new_tables =\n-\t\treftable_calloc(sizeof(struct reftable_table) * names_len);\n+\t\treftable_calloc(names_len, sizeof(*new_tables));\n \tint new_readers_len = 0;\n \tstruct reftable_merged_table *new_merged = NULL;\n \tstruct strbuf table_path = STRBUF_INIT;\n@@ -344,7 +342,7 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\t\t\tgoto out;\n \t\t\t}\n \n-\t\t\tnames = reftable_calloc(sizeof(char *));\n+\t\t\tREFTABLE_CALLOC_ARRAY(names, 1);\n \t\t} else {\n \t\t\terr = fd_read_lines(fd, &names);\n \t\t\tif (err < 0)\n@@ -686,7 +684,7 @@ int reftable_stack_new_addition(struct reftable_addition **dest,\n {\n \tint err = 0;\n \tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n-\t*dest = reftable_calloc(sizeof(**dest));\n+\tREFTABLE_CALLOC_ARRAY(*dest, 1);\n \t**dest = empty;\n \terr = reftable_stack_init_addition(*dest, st);\n \tif (err) {\n@@ -871,7 +869,7 @@ static int stack_write_compact(struct reftable_stack *st,\n {\n \tint subtabs_len = last - first + 1;\n \tstruct reftable_table *subtabs = reftable_calloc(\n-\t\tsizeof(struct reftable_table) * (last - first + 1));\n+\t\tlast - first + 1, sizeof(*subtabs));\n \tstruct reftable_merged_table *mt = NULL;\n \tint err = 0;\n \tstruct reftable_iterator it = { NULL };\n@@ -979,9 +977,9 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n \tint compact_count = last - first + 1;\n \tchar **listp = NULL;\n \tchar **delete_on_success =\n-\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\t\treftable_calloc(compact_count + 1, sizeof(*delete_on_success));\n \tchar **subtable_locks =\n-\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\t\treftable_calloc(compact_count + 1, sizeof(*subtable_locks));\n \tint i = 0;\n \tint j = 0;\n \tint is_empty_table = 0;\n@@ -1204,7 +1202,7 @@ int fastlog2(uint64_t sz)\n \n struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n {\n-\tstruct segment *segs = reftable_calloc(sizeof(struct segment) * n);\n+\tstruct segment *segs = reftable_calloc(n, sizeof(*segs));\n \tint next = 0;\n \tstruct segment cur = { 0 };\n \tint i = 0;\n@@ -1268,7 +1266,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, int n)\n static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n {\n \tuint64_t *sizes =\n-\t\treftable_calloc(sizeof(uint64_t) * st->merged->stack_len);\n+\t\treftable_calloc(st->merged->stack_len, sizeof(*sizes));\n \tint version = (st->config.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2;\n \tint overhead = header_size(version) - 1;\n \tint i = 0;\ndiff --git a/reftable/tree.c b/reftable/tree.c\nindex a5bf880985..528f33ae38 100644\n--- a/reftable/tree.c\n+++ b/reftable/tree.c\n@@ -20,8 +20,8 @@ struct tree_node *tree_search(void *key, struct tree_node **rootp,\n \t\tif (!insert) {\n \t\t\treturn NULL;\n \t\t} else {\n-\t\t\tstruct tree_node *n =\n-\t\t\t\treftable_calloc(sizeof(struct tree_node));\n+\t\t\tstruct tree_node *n;\n+\t\t\tREFTABLE_CALLOC_ARRAY(n, 1);\n \t\t\tn->key = key;\n \t\t\t*rootp = n;\n \t\t\treturn *rootp;\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 4483bb21c3..47b3448132 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -49,7 +49,7 @@ static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n {\n \tint n = 0;\n \tif (w->pending_padding > 0) {\n-\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding);\n+\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding, sizeof(*zeroed));\n \t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n \t\tif (n < 0)\n \t\t\treturn n;\n@@ -123,8 +123,7 @@ struct reftable_writer *\n reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\t void *writer_arg, struct reftable_write_options *opts)\n {\n-\tstruct reftable_writer *wp =\n-\t\treftable_calloc(sizeof(struct reftable_writer));\n+\tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n \tstrbuf_init(&wp->block_writer_data.last_key, 0);\n \toptions_set_defaults(opts);\n \tif (opts->block_size >= (1 << 24)) {\n@@ -132,7 +131,7 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\tabort();\n \t}\n \twp->last_key = reftable_empty_strbuf;\n-\twp->block = reftable_calloc(opts->block_size);\n+\tREFTABLE_CALLOC_ARRAY(wp->block, opts->block_size);\n \twp->write = writer_func;\n \twp->write_arg = writer_arg;\n \twp->opts = *opts;\n-- \n2.43.GIT\n\n"},{"id":"487702","messageId":"f134702dc5f656942baafbd80af46ad928ee1449.1706772591.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706772591.git.ps@pks.im","subject":"[PATCH v2 3/9] reftable/stack: fix parameter validation when compacting range","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-01T07:33:02Z","receivedAt":"2024-02-01T07:33:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `stack_compact_range()` function receives a \"first\" and \"last\" index\nthat indicates which tables of the reftable stack should be compacted.\nNaturally, \"first\" must be smaller than \"last\" in order to identify a\nproper range of tables to compress, which we indeed also assert in the\nfunction. But the validations happens after we have already allocated\narrays with a size of `last - first + 1`, leading to an underflow and\nthus an invalid allocation size.\n\nFix this by reordering the array allocations to happen after we have\nvalidated parameters. While at it, convert the array allocations to use\nthe newly introduced macros.\n\nNote that the relevant variables pointing into arrays should also be\nconverted to use `size_t` instead of `int`. This is left for a later\ncommit in this series.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 24 +++++++++++++-----------\n 1 file changed, 13 insertions(+), 11 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex d084823a92..b6b24c90bf 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -966,6 +966,7 @@ static int stack_write_compact(struct reftable_stack *st,\n static int stack_compact_range(struct reftable_stack *st, int first, int last,\n \t\t\t struct reftable_log_expiry_config *expiry)\n {\n+\tchar **delete_on_success = NULL, **subtable_locks = NULL, **listp = NULL;\n \tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n \tstruct strbuf new_table_name = STRBUF_INIT;\n \tstruct strbuf lock_file_name = STRBUF_INIT;\n@@ -974,12 +975,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n \tint err = 0;\n \tint have_lock = 0;\n \tint lock_file_fd = -1;\n-\tint compact_count = last - first + 1;\n-\tchar **listp = NULL;\n-\tchar **delete_on_success =\n-\t\treftable_calloc(compact_count + 1, sizeof(*delete_on_success));\n-\tchar **subtable_locks =\n-\t\treftable_calloc(compact_count + 1, sizeof(*subtable_locks));\n+\tint compact_count;\n \tint i = 0;\n \tint j = 0;\n \tint is_empty_table = 0;\n@@ -989,6 +985,10 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n \t\tgoto done;\n \t}\n \n+\tcompact_count = last - first + 1;\n+\tREFTABLE_CALLOC_ARRAY(delete_on_success, compact_count + 1);\n+\tREFTABLE_CALLOC_ARRAY(subtable_locks, compact_count + 1);\n+\n \tst->stats.attempts++;\n \n \tstrbuf_reset(&lock_file_name);\n@@ -1146,12 +1146,14 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n done:\n \tfree_names(delete_on_success);\n \n-\tlistp = subtable_locks;\n-\twhile (*listp) {\n-\t\tunlink(*listp);\n-\t\tlistp++;\n+\tif (subtable_locks) {\n+\t\tlistp = subtable_locks;\n+\t\twhile (*listp) {\n+\t\t\tunlink(*listp);\n+\t\t\tlistp++;\n+\t\t}\n+\t\tfree_names(subtable_locks);\n \t}\n-\tfree_names(subtable_locks);\n \tif (lock_file_fd >= 0) {\n \t\tclose(lock_file_fd);\n \t\tlock_file_fd = -1;\n-- \n2.43.GIT\n\n"},{"id":"487703","messageId":"50dac904e86830aadcd4d3890ef6a0567d501bcf.1706772591.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706772591.git.ps@pks.im","subject":"[PATCH v2 4/9] reftable/stack: index segments with `size_t`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-01T07:33:07Z","receivedAt":"2024-02-01T07:33:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We use `int`s to index into arrays of segments and track the length of\nthem, which is considered to be a code smell in the Git project. Convert\nthe code to use `size_t` instead.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 25 +++++++++++--------------\n reftable/stack.h | 6 +++---\n reftable/stack_test.c | 7 +++----\n 3 files changed, 17 insertions(+), 21 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex b6b24c90bf..2be3d1e4ba 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1202,12 +1202,11 @@ int fastlog2(uint64_t sz)\n \treturn l - 1;\n }\n \n-struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n+struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n)\n {\n \tstruct segment *segs = reftable_calloc(n, sizeof(*segs));\n-\tint next = 0;\n \tstruct segment cur = { 0 };\n-\tint i = 0;\n+\tsize_t next = 0, i;\n \n \tif (n == 0) {\n \t\t*seglen = 0;\n@@ -1233,29 +1232,27 @@ struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n \treturn segs;\n }\n \n-struct segment suggest_compaction_segment(uint64_t *sizes, int n)\n+struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n {\n-\tint seglen = 0;\n-\tstruct segment *segs = sizes_to_segments(&seglen, sizes, n);\n \tstruct segment min_seg = {\n \t\t.log = 64,\n \t};\n-\tint i = 0;\n+\tstruct segment *segs;\n+\tsize_t seglen = 0, i;\n+\n+\tsegs = sizes_to_segments(&seglen, sizes, n);\n \tfor (i = 0; i < seglen; i++) {\n-\t\tif (segment_size(&segs[i]) == 1) {\n+\t\tif (segment_size(&segs[i]) == 1)\n \t\t\tcontinue;\n-\t\t}\n \n-\t\tif (segs[i].log < min_seg.log) {\n+\t\tif (segs[i].log < min_seg.log)\n \t\t\tmin_seg = segs[i];\n-\t\t}\n \t}\n \n \twhile (min_seg.start > 0) {\n-\t\tint prev = min_seg.start - 1;\n-\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev])) {\n+\t\tsize_t prev = min_seg.start - 1;\n+\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev]))\n \t\t\tbreak;\n-\t\t}\n \n \t\tmin_seg.start = prev;\n \t\tmin_seg.bytes += sizes[prev];\ndiff --git a/reftable/stack.h b/reftable/stack.h\nindex c1e3efa899..d919455669 100644\n--- a/reftable/stack.h\n+++ b/reftable/stack.h\n@@ -32,13 +32,13 @@ struct reftable_stack {\n int read_lines(const char *filename, char ***lines);\n \n struct segment {\n-\tint start, end;\n+\tsize_t start, end;\n \tint log;\n \tuint64_t bytes;\n };\n \n int fastlog2(uint64_t sz);\n-struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n);\n-struct segment suggest_compaction_segment(uint64_t *sizes, int n);\n+struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n);\n+struct segment suggest_compaction_segment(uint64_t *sizes, size_t n);\n \n #endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 289e902146..2d5b24e5c5 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -711,7 +711,7 @@ static void test_sizes_to_segments(void)\n \tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n \t/* .................0 1 2 3 4 5 */\n \n-\tint seglen = 0;\n+\tsize_t seglen = 0;\n \tstruct segment *segs =\n \t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n \tEXPECT(segs[2].log == 3);\n@@ -726,7 +726,7 @@ static void test_sizes_to_segments(void)\n \n static void test_sizes_to_segments_empty(void)\n {\n-\tint seglen = 0;\n+\tsize_t seglen = 0;\n \tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n \tEXPECT(seglen == 0);\n \treftable_free(segs);\n@@ -735,8 +735,7 @@ static void test_sizes_to_segments_empty(void)\n static void test_sizes_to_segments_all_equal(void)\n {\n \tuint64_t sizes[] = { 5, 5 };\n-\n-\tint seglen = 0;\n+\tsize_t seglen = 0;\n \tstruct segment *segs =\n \t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n \tEXPECT(seglen == 1);\n-- \n2.43.GIT\n\n"},{"id":"487704","messageId":"a5ffbf09dde32fcc8b2b9198875a82813b3de0ed.1706772591.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706772591.git.ps@pks.im","subject":"[PATCH v2 5/9] reftable/stack: use `size_t` to track stack slices during compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-01T07:33:11Z","receivedAt":"2024-02-01T07:33:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We use `int`s to track reftable slices when compacting the reftable\nstack, which is considered to be a code smell in the Git project.\nConvert the code to use `size_t` instead.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 32 ++++++++++++++++----------------\n 1 file changed, 16 insertions(+), 16 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 2be3d1e4ba..c1f8cf1cef 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -24,7 +24,8 @@ static int stack_try_add(struct reftable_stack *st,\n \t\t\t\t\t void *arg),\n \t\t\t void *arg);\n static int stack_write_compact(struct reftable_stack *st,\n-\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_writer *wr,\n+\t\t\t size_t first, size_t last,\n \t\t\t struct reftable_log_expiry_config *config);\n static int stack_check_addition(struct reftable_stack *st,\n \t\t\t\tconst char *new_tab_name);\n@@ -820,7 +821,8 @@ uint64_t reftable_stack_next_update_index(struct reftable_stack *st)\n \treturn 1;\n }\n \n-static int stack_compact_locked(struct reftable_stack *st, int first, int last,\n+static int stack_compact_locked(struct reftable_stack *st,\n+\t\t\t\tsize_t first, size_t last,\n \t\t\t\tstruct strbuf *temp_tab,\n \t\t\t\tstruct reftable_log_expiry_config *config)\n {\n@@ -864,22 +866,21 @@ static int stack_compact_locked(struct reftable_stack *st, int first, int last,\n }\n \n static int stack_write_compact(struct reftable_stack *st,\n-\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_writer *wr,\n+\t\t\t size_t first, size_t last,\n \t\t\t struct reftable_log_expiry_config *config)\n {\n \tint subtabs_len = last - first + 1;\n \tstruct reftable_table *subtabs = reftable_calloc(\n \t\tlast - first + 1, sizeof(*subtabs));\n \tstruct reftable_merged_table *mt = NULL;\n-\tint err = 0;\n \tstruct reftable_iterator it = { NULL };\n \tstruct reftable_ref_record ref = { NULL };\n \tstruct reftable_log_record log = { NULL };\n-\n \tuint64_t entries = 0;\n+\tint err = 0;\n \n-\tint i = 0, j = 0;\n-\tfor (i = first, j = 0; i <= last; i++) {\n+\tfor (size_t i = first, j = 0; i <= last; i++) {\n \t\tstruct reftable_reader *t = st->readers[i];\n \t\treftable_table_from_reader(&subtabs[j++], t);\n \t\tst->stats.bytes += t->size;\n@@ -963,7 +964,8 @@ static int stack_write_compact(struct reftable_stack *st,\n }\n \n /* < 0: error. 0 == OK, > 0 attempt failed; could retry. */\n-static int stack_compact_range(struct reftable_stack *st, int first, int last,\n+static int stack_compact_range(struct reftable_stack *st,\n+\t\t\t size_t first, size_t last,\n \t\t\t struct reftable_log_expiry_config *expiry)\n {\n \tchar **delete_on_success = NULL, **subtable_locks = NULL, **listp = NULL;\n@@ -972,12 +974,10 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n \tstruct strbuf lock_file_name = STRBUF_INIT;\n \tstruct strbuf ref_list_contents = STRBUF_INIT;\n \tstruct strbuf new_table_path = STRBUF_INIT;\n+\tsize_t i, j, compact_count;\n \tint err = 0;\n \tint have_lock = 0;\n \tint lock_file_fd = -1;\n-\tint compact_count;\n-\tint i = 0;\n-\tint j = 0;\n \tint is_empty_table = 0;\n \n \tif (first > last || (!expiry && first == last)) {\n@@ -1172,17 +1172,17 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n int reftable_stack_compact_all(struct reftable_stack *st,\n \t\t\t struct reftable_log_expiry_config *config)\n {\n-\treturn stack_compact_range(st, 0, st->merged->stack_len - 1, config);\n+\treturn stack_compact_range(st, 0, st->merged->stack_len ?\n+\t\t\tst->merged->stack_len - 1 : 0, config);\n }\n \n-static int stack_compact_range_stats(struct reftable_stack *st, int first,\n-\t\t\t\t int last,\n+static int stack_compact_range_stats(struct reftable_stack *st,\n+\t\t\t\t size_t first, size_t last,\n \t\t\t\t struct reftable_log_expiry_config *config)\n {\n \tint err = stack_compact_range(st, first, last, config);\n-\tif (err > 0) {\n+\tif (err > 0)\n \t\tst->stats.failures++;\n-\t}\n \treturn err;\n }\n \n-- \n2.43.GIT\n\n"},{"id":"487705","messageId":"55605fb53bfc0c81b9f2beec9c279df2634ebe4e.1706772591.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706772591.git.ps@pks.im","subject":"[PATCH v2 6/9] reftable/stack: use `size_t` to track stack length","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-01T07:33:15Z","receivedAt":"2024-02-01T07:33:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"While the stack length is already stored as `size_t`, we frequently use\n`int`s to refer to those stacks throughout the reftable library. Convert\nthose cases to use `size_t` instead to make things consistent.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 7 +++----\n reftable/basics.h | 2 +-\n reftable/merged.c | 11 +++++------\n reftable/merged_test.c | 14 ++++++--------\n reftable/reftable-merged.h | 2 +-\n reftable/stack.c | 21 ++++++++++-----------\n 6 files changed, 26 insertions(+), 31 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex af9004cec2..0785aff941 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -64,12 +64,11 @@ void free_names(char **a)\n \treftable_free(a);\n }\n \n-int names_length(char **names)\n+size_t names_length(char **names)\n {\n \tchar **p = names;\n-\tfor (; *p; p++) {\n-\t\t/* empty */\n-\t}\n+\twhile (*p)\n+\t\tp++;\n \treturn p - names;\n }\n \ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 4c3ac963a3..91f3533efe 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -44,7 +44,7 @@ void parse_names(char *buf, int size, char ***namesp);\n int names_equal(char **a, char **b);\n \n /* returns the array size of a NULL-terminated array of strings. */\n-int names_length(char **names);\n+size_t names_length(char **names);\n \n /* Allocation routines; they invoke the functions set through\n * reftable_set_alloc() */\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 2031fd51b4..e2c6253324 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -45,11 +45,10 @@ static int merged_iter_init(struct merged_iter *mi)\n static void merged_iter_close(void *p)\n {\n \tstruct merged_iter *mi = p;\n-\tint i = 0;\n+\n \tmerged_iter_pqueue_release(&mi->pq);\n-\tfor (i = 0; i < mi->stack_len; i++) {\n+\tfor (size_t i = 0; i < mi->stack_len; i++)\n \t\treftable_iterator_destroy(&mi->stack[i]);\n-\t}\n \treftable_free(mi->stack);\n \tstrbuf_release(&mi->key);\n \tstrbuf_release(&mi->entry_key);\n@@ -168,14 +167,14 @@ static void iterator_from_merged_iter(struct reftable_iterator *it,\n }\n \n int reftable_new_merged_table(struct reftable_merged_table **dest,\n-\t\t\t struct reftable_table *stack, int n,\n+\t\t\t struct reftable_table *stack, size_t n,\n \t\t\t uint32_t hash_id)\n {\n \tstruct reftable_merged_table *m = NULL;\n \tuint64_t last_max = 0;\n \tuint64_t first_min = 0;\n-\tint i = 0;\n-\tfor (i = 0; i < n; i++) {\n+\n+\tfor (size_t i = 0; i < n; i++) {\n \t\tuint64_t min = reftable_table_min_update_index(&stack[i]);\n \t\tuint64_t max = reftable_table_max_update_index(&stack[i]);\n \ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nindex e233a9d581..442917cc83 100644\n--- a/reftable/merged_test.c\n+++ b/reftable/merged_test.c\n@@ -88,18 +88,17 @@ static struct reftable_merged_table *\n merged_table_from_records(struct reftable_ref_record **refs,\n \t\t\t struct reftable_block_source **source,\n \t\t\t struct reftable_reader ***readers, int *sizes,\n-\t\t\t struct strbuf *buf, int n)\n+\t\t\t struct strbuf *buf, size_t n)\n {\n-\tint i = 0;\n \tstruct reftable_merged_table *mt = NULL;\n-\tint err;\n \tstruct reftable_table *tabs;\n+\tint err;\n \n \tREFTABLE_CALLOC_ARRAY(tabs, n);\n \tREFTABLE_CALLOC_ARRAY(*readers, n);\n \tREFTABLE_CALLOC_ARRAY(*source, n);\n \n-\tfor (i = 0; i < n; i++) {\n+\tfor (size_t i = 0; i < n; i++) {\n \t\twrite_test_table(&buf[i], refs[i], sizes[i]);\n \t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n \n@@ -263,18 +262,17 @@ static struct reftable_merged_table *\n merged_table_from_log_records(struct reftable_log_record **logs,\n \t\t\t struct reftable_block_source **source,\n \t\t\t struct reftable_reader ***readers, int *sizes,\n-\t\t\t struct strbuf *buf, int n)\n+\t\t\t struct strbuf *buf, size_t n)\n {\n-\tint i = 0;\n \tstruct reftable_merged_table *mt = NULL;\n-\tint err;\n \tstruct reftable_table *tabs;\n+\tint err;\n \n \tREFTABLE_CALLOC_ARRAY(tabs, n);\n \tREFTABLE_CALLOC_ARRAY(*readers, n);\n \tREFTABLE_CALLOC_ARRAY(*source, n);\n \n-\tfor (i = 0; i < n; i++) {\n+\tfor (size_t i = 0; i < n; i++) {\n \t\twrite_test_log_table(&buf[i], logs[i], sizes[i], i + 1);\n \t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n \ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nindex 1a6d16915a..c91a2d83a2 100644\n--- a/reftable/reftable-merged.h\n+++ b/reftable/reftable-merged.h\n@@ -33,7 +33,7 @@ struct reftable_table;\n the stack array.\n */\n int reftable_new_merged_table(struct reftable_merged_table **dest,\n-\t\t\t struct reftable_table *stack, int n,\n+\t\t\t struct reftable_table *stack, size_t n,\n \t\t\t uint32_t hash_id);\n \n /* returns an iterator positioned just before 'name' */\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex c1f8cf1cef..079ba7fde8 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -202,18 +202,18 @@ static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n \t\t\t\t int reuse_open)\n {\n-\tint cur_len = !st->merged ? 0 : st->merged->stack_len;\n+\tsize_t cur_len = !st->merged ? 0 : st->merged->stack_len;\n \tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n-\tint err = 0;\n-\tint names_len = names_length(names);\n+\tsize_t names_len = names_length(names);\n \tstruct reftable_reader **new_readers =\n \t\treftable_calloc(names_len, sizeof(*new_readers));\n \tstruct reftable_table *new_tables =\n \t\treftable_calloc(names_len, sizeof(*new_tables));\n-\tint new_readers_len = 0;\n+\tsize_t new_readers_len = 0;\n \tstruct reftable_merged_table *new_merged = NULL;\n \tstruct strbuf table_path = STRBUF_INIT;\n-\tint i;\n+\tint err = 0;\n+\tsize_t i;\n \n \twhile (*names) {\n \t\tstruct reftable_reader *rd = NULL;\n@@ -221,11 +221,10 @@ static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n \n \t\t/* this is linear; we assume compaction keeps the number of\n \t\t tables under control so this is not quadratic. */\n-\t\tint j = 0;\n-\t\tfor (j = 0; reuse_open && j < cur_len; j++) {\n-\t\t\tif (cur[j] && 0 == strcmp(cur[j]->name, name)) {\n-\t\t\t\trd = cur[j];\n-\t\t\t\tcur[j] = NULL;\n+\t\tfor (i = 0; reuse_open && i < cur_len; i++) {\n+\t\t\tif (cur[i] && 0 == strcmp(cur[i]->name, name)) {\n+\t\t\t\trd = cur[i];\n+\t\t\t\tcur[i] = NULL;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t}\n@@ -870,7 +869,7 @@ static int stack_write_compact(struct reftable_stack *st,\n \t\t\t size_t first, size_t last,\n \t\t\t struct reftable_log_expiry_config *config)\n {\n-\tint subtabs_len = last - first + 1;\n+\tsize_t subtabs_len = last - first + 1;\n \tstruct reftable_table *subtabs = reftable_calloc(\n \t\tlast - first + 1, sizeof(*subtabs));\n \tstruct reftable_merged_table *mt = NULL;\n-- \n2.43.GIT\n\n"},{"id":"487706","messageId":"80cf2fd272577b13f4fd23f009d22294629f5f3b.1706772591.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706772591.git.ps@pks.im","subject":"[PATCH v2 7/9] reftable/merged: refactor seeking of records","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-01T07:33:19Z","receivedAt":"2024-02-01T07:33:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The code to seek reftable records in the merged table code is quite hard\nto read and does not conform to our coding style in multiple ways:\n\n - We have multiple exit paths where we release resources even though\n that is not really necessary.\n\n - We use a scoped error variable `e` which is hard to reason about.\n This variable is not required at all.\n\n - We allocate memory in the variable declarations, which is easy to\n miss.\n\nRefactor the function so that it becomes more maintainable in the\nfuture.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 54 ++++++++++++++++++-----------------------------\n 1 file changed, 21 insertions(+), 33 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex e2c6253324..0abcda26e8 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -238,50 +238,38 @@ static int merged_table_seek_record(struct reftable_merged_table *mt,\n \t\t\t\t struct reftable_iterator *it,\n \t\t\t\t struct reftable_record *rec)\n {\n-\tstruct reftable_iterator *iters = reftable_calloc(\n-\t\tmt->stack_len, sizeof(*iters));\n \tstruct merged_iter merged = {\n-\t\t.stack = iters,\n \t\t.typ = reftable_record_type(rec),\n \t\t.hash_id = mt->hash_id,\n \t\t.suppress_deletions = mt->suppress_deletions,\n \t\t.key = STRBUF_INIT,\n \t\t.entry_key = STRBUF_INIT,\n \t};\n-\tint n = 0;\n-\tint err = 0;\n-\tint i = 0;\n-\tfor (i = 0; i < mt->stack_len && err == 0; i++) {\n-\t\tint e = reftable_table_seek_record(&mt->stack[i], &iters[n],\n-\t\t\t\t\t\t rec);\n-\t\tif (e < 0) {\n-\t\t\terr = e;\n-\t\t}\n-\t\tif (e == 0) {\n-\t\t\tn++;\n-\t\t}\n-\t}\n-\tif (err < 0) {\n-\t\tint i = 0;\n-\t\tfor (i = 0; i < n; i++) {\n-\t\t\treftable_iterator_destroy(&iters[i]);\n-\t\t}\n-\t\treftable_free(iters);\n-\t\treturn err;\n+\tstruct merged_iter *p;\n+\tint err;\n+\n+\tREFTABLE_CALLOC_ARRAY(merged.stack, mt->stack_len);\n+\tfor (size_t i = 0; i < mt->stack_len; i++) {\n+\t\terr = reftable_table_seek_record(&mt->stack[i],\n+\t\t\t\t\t\t &merged.stack[merged.stack_len], rec);\n+\t\tif (err < 0)\n+\t\t\tgoto out;\n+\t\tif (!err)\n+\t\t\tmerged.stack_len++;\n \t}\n \n-\tmerged.stack_len = n;\n \terr = merged_iter_init(&merged);\n-\tif (err < 0) {\n+\tif (err < 0)\n+\t\tgoto out;\n+\n+\tp = reftable_malloc(sizeof(struct merged_iter));\n+\t*p = merged;\n+\titerator_from_merged_iter(it, p);\n+\n+out:\n+\tif (err < 0)\n \t\tmerged_iter_close(&merged);\n-\t\treturn err;\n-\t} else {\n-\t\tstruct merged_iter *p =\n-\t\t\treftable_malloc(sizeof(struct merged_iter));\n-\t\t*p = merged;\n-\t\titerator_from_merged_iter(it, p);\n-\t}\n-\treturn 0;\n+\treturn err;\n }\n \n int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n-- \n2.43.GIT\n\n"},{"id":"487707","messageId":"8c1be2b1599011f57479249c19f4e4742b7d8d8b.1706772591.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706772591.git.ps@pks.im","subject":"[PATCH v2 8/9] reftable/merged: refactor initialization of iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-01T07:33:23Z","receivedAt":"2024-02-01T07:33:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor the initialization of the merged iterator to fit our code style\nbetter. This refactoring prepares the code for a refactoring of how\nrecords are being initialized.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 27 +++++++++++++--------------\n 1 file changed, 13 insertions(+), 14 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 0abcda26e8..0e60e2a39b 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -19,24 +19,23 @@ license that can be found in the LICENSE file or at\n \n static int merged_iter_init(struct merged_iter *mi)\n {\n-\tint i = 0;\n-\tfor (i = 0; i < mi->stack_len; i++) {\n-\t\tstruct reftable_record rec = reftable_new_record(mi->typ);\n-\t\tint err = iterator_next(&mi->stack[i], &rec);\n-\t\tif (err < 0) {\n+\tfor (size_t i = 0; i < mi->stack_len; i++) {\n+\t\tstruct pq_entry e = {\n+\t\t\t.rec = reftable_new_record(mi->typ),\n+\t\t\t.index = i,\n+\t\t};\n+\t\tint err;\n+\n+\t\terr = iterator_next(&mi->stack[i], &e.rec);\n+\t\tif (err < 0)\n \t\t\treturn err;\n-\t\t}\n-\n \t\tif (err > 0) {\n \t\t\treftable_iterator_destroy(&mi->stack[i]);\n-\t\t\treftable_record_release(&rec);\n-\t\t} else {\n-\t\t\tstruct pq_entry e = {\n-\t\t\t\t.rec = rec,\n-\t\t\t\t.index = i,\n-\t\t\t};\n-\t\t\tmerged_iter_pqueue_add(&mi->pq, &e);\n+\t\t\treftable_record_release(&e.rec);\n+\t\t\tcontinue;\n \t\t}\n+\n+\t\tmerged_iter_pqueue_add(&mi->pq, &e);\n \t}\n \n \treturn 0;\n-- \n2.43.GIT\n\n"},{"id":"487708","messageId":"c39d7e30e77745fb6dcea3f287f38e4a4c041202.1706772591.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706772591.git.ps@pks.im","subject":"[PATCH v2 9/9] reftable/record: improve semantics when initializing records","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-01T07:33:27Z","receivedAt":"2024-02-01T07:33:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"According to our usual coding style, the `reftable_new_record()`\nfunction would indicate that it is allocating a new record. This is not\nthe case though as the function merely initializes records without\nallocating any memory.\n\nReplace `reftable_new_record()` with a new `reftable_record_init()`\nfunction that takes a record pointer as input and initializes it\naccordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/block.c | 18 +++++++++---------\n reftable/merged.c | 8 +++++---\n reftable/reader.c | 4 ++--\n reftable/record.c | 43 ++++++++++--------------------------------\n reftable/record.h | 10 +++++-----\n reftable/record_test.c | 4 ++--\n 6 files changed, 33 insertions(+), 54 deletions(-)\n\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 6952d0facf..9ad220747e 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -380,23 +380,23 @@ int block_reader_seek(struct block_reader *br, struct block_iter *it,\n \t\t.key = *want,\n \t\t.r = br,\n \t};\n-\tstruct reftable_record rec = reftable_new_record(block_reader_type(br));\n-\tint err = 0;\n \tstruct block_iter next = BLOCK_ITER_INIT;\n+\tstruct reftable_record rec;\n+\tint err = 0, i;\n \n-\tint i = binsearch(br->restart_count, &restart_key_less, &args);\n \tif (args.error) {\n \t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n \t}\n \n-\tit->br = br;\n-\tif (i > 0) {\n-\t\ti--;\n-\t\tit->next_off = block_reader_restart_offset(br, i);\n-\t} else {\n+\ti = binsearch(br->restart_count, &restart_key_less, &args);\n+\tif (i > 0)\n+\t\tit->next_off = block_reader_restart_offset(br, i - 1);\n+\telse\n \t\tit->next_off = br->header_off + 4;\n-\t}\n+\tit->br = br;\n+\n+\treftable_record_init(&rec, block_reader_type(br));\n \n \t/* We're looking for the last entry less/equal than the wanted key, so\n \t we have to go one entry too far and then back up.\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 0e60e2a39b..a0f222e07b 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -21,11 +21,11 @@ static int merged_iter_init(struct merged_iter *mi)\n {\n \tfor (size_t i = 0; i < mi->stack_len; i++) {\n \t\tstruct pq_entry e = {\n-\t\t\t.rec = reftable_new_record(mi->typ),\n \t\t\t.index = i,\n \t\t};\n \t\tint err;\n \n+\t\treftable_record_init(&e.rec, mi->typ);\n \t\terr = iterator_next(&mi->stack[i], &e.rec);\n \t\tif (err < 0)\n \t\t\treturn err;\n@@ -57,10 +57,12 @@ static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,\n \t\t\t\t\t size_t idx)\n {\n \tstruct pq_entry e = {\n-\t\t.rec = reftable_new_record(mi->typ),\n \t\t.index = idx,\n \t};\n-\tint err = iterator_next(&mi->stack[idx], &e.rec);\n+\tint err;\n+\n+\treftable_record_init(&e.rec, mi->typ);\n+\terr = iterator_next(&mi->stack[idx], &e.rec);\n \tif (err < 0)\n \t\treturn err;\n \ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 3e0de5e8ad..5e6c8f30a1 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -444,13 +444,13 @@ static int reader_start(struct reftable_reader *r, struct table_iter *ti,\n static int reader_seek_linear(struct table_iter *ti,\n \t\t\t struct reftable_record *want)\n {\n-\tstruct reftable_record rec =\n-\t\treftable_new_record(reftable_record_type(want));\n \tstruct strbuf want_key = STRBUF_INIT;\n \tstruct strbuf got_key = STRBUF_INIT;\n \tstruct table_iter next = TABLE_ITER_INIT;\n+\tstruct reftable_record rec;\n \tint err = -1;\n \n+\treftable_record_init(&rec, reftable_record_type(want));\n \treftable_record_key(want, &want_key);\n \n \twhile (1) {\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 5c3fbb7b2a..8b84b8157f 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -1257,45 +1257,22 @@ reftable_record_vtable(struct reftable_record *rec)\n \tabort();\n }\n \n-struct reftable_record reftable_new_record(uint8_t typ)\n+void reftable_record_init(struct reftable_record *rec, uint8_t typ)\n {\n-\tstruct reftable_record clean = {\n-\t\t.type = typ,\n-\t};\n+\tmemset(rec, 0, sizeof(*rec));\n+\trec->type = typ;\n \n-\t/* the following is involved, but the naive solution (just return\n-\t * `clean` as is, except for BLOCK_TYPE_INDEX), returns a garbage\n-\t * clean.u.obj.offsets pointer on Windows VS CI. Go figure.\n-\t */\n \tswitch (typ) {\n-\tcase BLOCK_TYPE_OBJ:\n-\t{\n-\t\tstruct reftable_obj_record obj = { 0 };\n-\t\tclean.u.obj = obj;\n-\t\tbreak;\n-\t}\n-\tcase BLOCK_TYPE_INDEX:\n-\t{\n-\t\tstruct reftable_index_record idx = {\n-\t\t\t.last_key = STRBUF_INIT,\n-\t\t};\n-\t\tclean.u.idx = idx;\n-\t\tbreak;\n-\t}\n \tcase BLOCK_TYPE_REF:\n-\t{\n-\t\tstruct reftable_ref_record ref = { 0 };\n-\t\tclean.u.ref = ref;\n-\t\tbreak;\n-\t}\n \tcase BLOCK_TYPE_LOG:\n-\t{\n-\t\tstruct reftable_log_record log = { 0 };\n-\t\tclean.u.log = log;\n-\t\tbreak;\n-\t}\n+\tcase BLOCK_TYPE_OBJ:\n+\t\treturn;\n+\tcase BLOCK_TYPE_INDEX:\n+\t\tstrbuf_init(&rec->u.idx.last_key, 0);\n+\t\treturn;\n+\tdefault:\n+\t\tBUG(\"unhandled record type\");\n \t}\n-\treturn clean;\n }\n \n void reftable_record_print(struct reftable_record *rec, int hash_size)\ndiff --git a/reftable/record.h b/reftable/record.h\nindex fd80cd451d..e64ed30c80 100644\n--- a/reftable/record.h\n+++ b/reftable/record.h\n@@ -69,9 +69,6 @@ struct reftable_record_vtable {\n /* returns true for recognized block types. Block start with the block type. */\n int reftable_is_block_type(uint8_t typ);\n \n-/* return an initialized record for the given type */\n-struct reftable_record reftable_new_record(uint8_t typ);\n-\n /* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns\n * number of bytes written. */\n int reftable_encode_key(int *is_restart, struct string_view dest,\n@@ -100,8 +97,8 @@ struct reftable_obj_record {\n /* record is a generic wrapper for different types of records. It is normally\n * created on the stack, or embedded within another struct. If the type is\n * known, a fresh instance can be initialized explicitly. Otherwise, use\n- * reftable_new_record() to initialize generically (as the index_record is not\n- * valid as 0-initialized structure)\n+ * `reftable_record_init()` to initialize generically (as the index_record is\n+ * not valid as 0-initialized structure)\n */\n struct reftable_record {\n \tuint8_t type;\n@@ -113,6 +110,9 @@ struct reftable_record {\n \t} u;\n };\n \n+/* Initialize the reftable record for the given type */\n+void reftable_record_init(struct reftable_record *rec, uint8_t typ);\n+\n /* see struct record_vtable */\n int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);\n void reftable_record_print(struct reftable_record *rec, int hash_size);\ndiff --git a/reftable/record_test.c b/reftable/record_test.c\nindex 999366ad46..a86cff5526 100644\n--- a/reftable/record_test.c\n+++ b/reftable/record_test.c\n@@ -16,11 +16,11 @@\n \n static void test_copy(struct reftable_record *rec)\n {\n-\tstruct reftable_record copy = { 0 };\n+\tstruct reftable_record copy;\n \tuint8_t typ;\n \n \ttyp = reftable_record_type(rec);\n-\tcopy = reftable_new_record(typ);\n+\treftable_record_init(©, typ);\n \treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n \t/* do it twice to catch memory leaks */\n \treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n-- \n2.43.GIT\n\n"},{"id":"487748","messageId":"87o7d0p38k.fsf@iotcl.com","threadId":"60822","inReplyTo":"f134702dc5f656942baafbd80af46ad928ee1449.1706772591.git.ps@pks.im","subject":"Re: [PATCH v2 3/9] reftable/stack: fix parameter validation when compacting range","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2024-02-01T16:15:14Z","receivedAt":"2024-02-01T16:23:23Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"\nPatrick Steinhardt writes:\n\n> diff --git a/reftable/stack.c b/reftable/stack.c\n> index d084823a92..b6b24c90bf 100644\n> --- a/reftable/stack.c\n> +++ b/reftable/stack.c\n> @@ -1146,12 +1146,14 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n> done:\n> \tfree_names(delete_on_success);\n>\n> -\tlistp = subtable_locks;\n> -\twhile (*listp) {\n> -\t\tunlink(*listp);\n> -\t\tlistp++;\n> +\tif (subtable_locks) {\n> +\t\tlistp = subtable_locks;\n> +\t\twhile (*listp) {\n> +\t\t\tunlink(*listp);\n> +\t\t\tlistp++;\n> +\t\t}\n> +\t\tfree_names(subtable_locks);\n> \t}\n> -\tfree_names(subtable_locks);\n> \tif (lock_file_fd >= 0) {\n> \t\tclose(lock_file_fd);\n> \t\tlock_file_fd = -1;\n\nTechnically, this change is not needed, because `free_names()` deals\nwith NULL pointers already.\n\n--\nToon\n"},{"id":"487749","messageId":"xmqqbk902llj.fsf@gitster.g","threadId":"60822","inReplyTo":"ZbtIQ3Hk6Mgvkv4j@tanuki","subject":"Re: [PATCH 1/9] reftable: introduce macros to grow arrays","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-01T16:30:32Z","receivedAt":"2024-02-01T16:30:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> Very good point indeed. I don't think peak memory usage is really all\n> that helpful either because the problem is not that we are allocating\n> arrays that we keep around all the time, but many small arrays which are\n> short lived. So what is telling is the total number of bytes we end up\n> allocating:\n\nTrue. sum of requested sizes to all the calls to malloc() and\nfriends to allocate, ignoring what gets free()d in between, is what\nwould show how much we try to consume.\n\n> Allocating 21 times as many bytes with our default growth factor should\n> be a much more compelling argument why we don't actually want to use it\n> compared to the 2% speedup.\n\n;-).\n\n"},{"id":"487790","messageId":"Zbx744Uln_UyyyYi@tanuki","threadId":"60822","inReplyTo":"87o7d0p38k.fsf@iotcl.com","subject":"Re: [PATCH v2 3/9] reftable/stack: fix parameter validation when compacting range","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-02T05:21:39Z","receivedAt":"2024-02-02T05:21:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Feb 01, 2024 at 05:15:14PM +0100, Toon Claes wrote:\n> \n> Patrick Steinhardt writes:\n> \n> > diff --git a/reftable/stack.c b/reftable/stack.c\n> > index d084823a92..b6b24c90bf 100644\n> > --- a/reftable/stack.c\n> > +++ b/reftable/stack.c\n> > @@ -1146,12 +1146,14 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n> > done:\n> > \tfree_names(delete_on_success);\n> >\n> > -\tlistp = subtable_locks;\n> > -\twhile (*listp) {\n> > -\t\tunlink(*listp);\n> > -\t\tlistp++;\n> > +\tif (subtable_locks) {\n> > +\t\tlistp = subtable_locks;\n> > +\t\twhile (*listp) {\n> > +\t\t\tunlink(*listp);\n> > +\t\t\tlistp++;\n> > +\t\t}\n> > +\t\tfree_names(subtable_locks);\n> > \t}\n> > -\tfree_names(subtable_locks);\n> > \tif (lock_file_fd >= 0) {\n> > \t\tclose(lock_file_fd);\n> > \t\tlock_file_fd = -1;\n> \n> Technically, this change is not needed, because `free_names()` deals\n> with NULL pointers already.\n\nIt actually is, but it's easy to miss. The reason why I've started to\nguard the code isn't `free_names()`. It's rather the `while (*listp)`,\nwhere we dereference `listp` unconditionally. Before these refactorings,\n`subtable_locks` would never be `NULL`. Now it can be though, and in\nthat case we would be dereferencing a `NULL` pointer.\n\nPatrick\n"},{"id":"487932","messageId":"CAOLa=ZSDz4PMqFGp3MHr7Ls2xOUs+NEnG-y09J9knNd_eGpZUQ@mail.gmail.com","threadId":"60822","inReplyTo":"2dde581a0256e4634ca4a64f313a98204763906a.1706772591.git.ps@pks.im","subject":"Re: [PATCH v2 2/9] reftable: introduce macros to allocate arrays","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-05T15:48:41Z","receivedAt":"2024-02-05T15:48:43Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Similar to the preceding commit, let's carry over macros to allocate\n> arrays with `REFTABLE_ALLOC_ARRAY()` and `REFTABLE_CALLOC_ARRAY()`. This\n> requires us to change the signature of `reftable_calloc()`, which only\n> takes a single argument right now and thus puts the burden on the caller\n> to calculate the final array's size. This is a net improvement though as\n> it means that we can now provide proper overflow checks when multiplying\n> the array size with the member size.\n>\n> Convert callsites of `reftable_calloc()` to the new signature, using the\n> new macros where possible.\n\nWhat about converting users of `reftable_malloc()` to use\n`REFTABLE_ALLOC_ARRAY()`. This means currently `REFTABLE_ALLOC_ARRAY()`\nis defined and never used in this patch series.\n"},{"id":"488027","messageId":"ZcHNPXf9ka_KhJxD@tanuki","threadId":"60822","inReplyTo":"CAOLa=ZSDz4PMqFGp3MHr7Ls2xOUs+NEnG-y09J9knNd_eGpZUQ@mail.gmail.com","subject":"Re: [PATCH v2 2/9] reftable: introduce macros to allocate arrays","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-06T06:10:05Z","receivedAt":"2024-02-06T06:10:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Feb 05, 2024 at 07:48:41AM -0800, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> \n> > Similar to the preceding commit, let's carry over macros to allocate\n> > arrays with `REFTABLE_ALLOC_ARRAY()` and `REFTABLE_CALLOC_ARRAY()`. This\n> > requires us to change the signature of `reftable_calloc()`, which only\n> > takes a single argument right now and thus puts the burden on the caller\n> > to calculate the final array's size. This is a net improvement though as\n> > it means that we can now provide proper overflow checks when multiplying\n> > the array size with the member size.\n> >\n> > Convert callsites of `reftable_calloc()` to the new signature, using the\n> > new macros where possible.\n> \n> What about converting users of `reftable_malloc()` to use\n> `REFTABLE_ALLOC_ARRAY()`. This means currently `REFTABLE_ALLOC_ARRAY()`\n> is defined and never used in this patch series.\n\nGood point, will do.\n\nPatrick\n"},{"id":"488028","messageId":"cover.1707200355.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1706687982.git.ps@pks.im","subject":"[PATCH v3 0/9] reftable: code style improvements","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-06T06:35:19Z","receivedAt":"2024-02-06T06:35:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the third version of my patch series that tries to align the\nreftable library's coding style to be closer to Git's own code style.\n\nThe only change compared to v2 is that I've now also converted some\ncalls to `reftable_malloc()` to use `REFTABLE_ALLOC_ARRAY`.\n\nThanks!\n\nPatrick\n\nPatrick Steinhardt (9):\n reftable: introduce macros to grow arrays\n reftable: introduce macros to allocate arrays\n reftable/stack: fix parameter validation when compacting range\n reftable/stack: index segments with `size_t`\n reftable/stack: use `size_t` to track stack slices during compaction\n reftable/stack: use `size_t` to track stack length\n reftable/merged: refactor seeking of records\n reftable/merged: refactor initialization of iterators\n reftable/record: improve semantics when initializing records\n\n reftable/basics.c | 15 ++--\n reftable/basics.h | 17 ++++-\n reftable/block.c | 35 ++++-----\n reftable/block_test.c | 2 +-\n reftable/blocksource.c | 4 +-\n reftable/iter.c | 3 +-\n reftable/merged.c | 100 +++++++++++-------------\n reftable/merged_test.c | 52 ++++++-------\n reftable/pq.c | 8 +-\n reftable/publicbasics.c | 3 +-\n reftable/reader.c | 12 ++-\n reftable/readwrite_test.c | 8 +-\n reftable/record.c | 57 +++++---------\n reftable/record.h | 10 +--\n reftable/record_test.c | 8 +-\n reftable/refname.c | 4 +-\n reftable/reftable-merged.h | 2 +-\n reftable/stack.c | 153 +++++++++++++++++--------------------\n reftable/stack.h | 6 +-\n reftable/stack_test.c | 7 +-\n reftable/tree.c | 4 +-\n reftable/writer.c | 21 ++---\n 22 files changed, 236 insertions(+), 295 deletions(-)\n\nRange-diff against v2:\n 1: 12bd721ddf = 1: 12bd721ddf reftable: introduce macros to grow arrays\n 2: 2dde581a02 ! 2: 95689ca7ce reftable: introduce macros to allocate arrays\n @@ Commit message\n it means that we can now provide proper overflow checks when multiplying\n the array size with the member size.\n \n - Convert callsites of `reftable_calloc()` to the new signature, using the\n - new macros where possible.\n + Convert callsites of `reftable_calloc()` to the new signature and start\n + using the new macros where possible.\n \n Signed-off-by: Patrick Steinhardt \n \n @@ reftable/basics.h: int names_length(char **names);\n #define REFTABLE_ALLOC_GROW(x, nr, alloc) \\\n \tdo { \\\n \n + ## reftable/block.c ##\n +@@ reftable/block.c: int block_writer_finish(struct block_writer *w)\n + \t\tint block_header_skip = 4 + w->header_off;\n + \t\tuLongf src_len = w->next - block_header_skip;\n + \t\tuLongf dest_cap = src_len * 1.001 + 12;\n ++\t\tuint8_t *compressed;\n ++\n ++\t\tREFTABLE_ALLOC_ARRAY(compressed, dest_cap);\n + \n +-\t\tuint8_t *compressed = reftable_malloc(dest_cap);\n + \t\twhile (1) {\n + \t\t\tuLongf out_dest_len = dest_cap;\n + \t\t\tint zresult = compress2(compressed, &out_dest_len,\n +@@ reftable/block.c: int block_reader_init(struct block_reader *br, struct reftable_block *block,\n + \t\tuLongf dst_len = sz - block_header_skip; /* total size of dest\n + \t\t\t\t\t\t\t buffer. */\n + \t\tuLongf src_len = block->len - block_header_skip;\n +-\t\t/* Log blocks specify the *uncompressed* size in their header.\n +-\t\t */\n +-\t\tuncompressed = reftable_malloc(sz);\n ++\n ++\t\t/* Log blocks specify the *uncompressed* size in their header. */\n ++\t\tREFTABLE_ALLOC_ARRAY(uncompressed, sz);\n + \n + \t\t/* Copy over the block header verbatim. It's not compressed. */\n + \t\tmemcpy(uncompressed, block->data, block_header_skip);\n +\n ## reftable/block_test.c ##\n @@ reftable/block_test.c: static void test_block_read_write(void)\n \tint j = 0;\n @@ reftable/readwrite_test.c: static void test_table_read_write_seek_index(void)\n \tuint8_t want_hash[GIT_SHA1_RAWSZ];\n \n \n + ## reftable/record.c ##\n +@@ reftable/record.c: static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n + \t\t(const struct reftable_obj_record *)src_rec;\n + \n + \treftable_obj_record_release(obj);\n +-\tobj->hash_prefix = reftable_malloc(src->hash_prefix_len);\n ++\n ++\tREFTABLE_ALLOC_ARRAY(obj->hash_prefix, src->hash_prefix_len);\n + \tobj->hash_prefix_len = src->hash_prefix_len;\n + \tif (src->hash_prefix_len)\n + \t\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n + \n +-\tobj->offsets = reftable_malloc(src->offset_len * sizeof(uint64_t));\n ++\tREFTABLE_ALLOC_ARRAY(obj->offsets, src->offset_len);\n + \tobj->offset_len = src->offset_len;\n + \tCOPY_ARRAY(obj->offsets, src->offsets, src->offset_len);\n + }\n +@@ reftable/record.c: static int reftable_obj_record_decode(void *rec, struct strbuf key,\n + \tint n = 0;\n + \tuint64_t last;\n + \tint j;\n +-\tr->hash_prefix = reftable_malloc(key.len);\n ++\n ++\tREFTABLE_ALLOC_ARRAY(r->hash_prefix, key.len);\n + \tmemcpy(r->hash_prefix, key.buf, key.len);\n + \tr->hash_prefix_len = key.len;\n + \n +@@ reftable/record.c: static int reftable_obj_record_decode(void *rec, struct strbuf key,\n + \tif (count == 0)\n + \t\treturn start.len - in.len;\n + \n +-\tr->offsets = reftable_malloc(count * sizeof(uint64_t));\n ++\tREFTABLE_ALLOC_ARRAY(r->offsets, count);\n + \tr->offset_len = count;\n + \n + \tn = get_var_int(&r->offsets[0], &in);\n +@@ reftable/record.c: static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n + \t\t}\n + \n + \t\tif (dst->value.update.new_hash) {\n +-\t\t\tdst->value.update.new_hash = reftable_malloc(hash_size);\n ++\t\t\tREFTABLE_ALLOC_ARRAY(dst->value.update.new_hash, hash_size);\n + \t\t\tmemcpy(dst->value.update.new_hash,\n + \t\t\t src->value.update.new_hash, hash_size);\n + \t\t}\n + \t\tif (dst->value.update.old_hash) {\n +-\t\t\tdst->value.update.old_hash = reftable_malloc(hash_size);\n ++\t\t\tREFTABLE_ALLOC_ARRAY(dst->value.update.old_hash, hash_size);\n + \t\t\tmemcpy(dst->value.update.old_hash,\n + \t\t\t src->value.update.old_hash, hash_size);\n + \t\t}\n +\n ## reftable/record_test.c ##\n @@ reftable/record_test.c: static void test_reftable_log_record_roundtrip(void)\n \t\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n @@ reftable/stack.c: static ssize_t reftable_fd_write(void *arg, const void *data,\n \tstruct strbuf list_file_name = STRBUF_INIT;\n \tint err = 0;\n \n +@@ reftable/stack.c: static int fd_read_lines(int fd, char ***namesp)\n + \t\tgoto done;\n + \t}\n + \n +-\tbuf = reftable_malloc(size + 1);\n ++\tREFTABLE_ALLOC_ARRAY(buf, size + 1);\n + \tif (read_in_full(fd, buf, size) != size) {\n + \t\terr = REFTABLE_IO_ERROR;\n + \t\tgoto done;\n @@ reftable/stack.c: int read_lines(const char *filename, char ***namesp)\n \tint err = 0;\n \tif (fd < 0) {\n 3: f134702dc5 = 3: f0e8f08884 reftable/stack: fix parameter validation when compacting range\n 4: 50dac904e8 = 4: 7bcfe7b305 reftable/stack: index segments with `size_t`\n 5: a5ffbf09dd = 5: a0867c0378 reftable/stack: use `size_t` to track stack slices during compaction\n 6: 55605fb53b = 6: 29c5a54ae8 reftable/stack: use `size_t` to track stack length\n 7: 80cf2fd272 = 7: 4605ad7247 reftable/merged: refactor seeking of records\n 8: 8c1be2b159 = 8: 8c35968ce8 reftable/merged: refactor initialization of iterators\n 9: c39d7e30e7 = 9: 5bb2858c13 reftable/record: improve semantics when initializing records\n-- \n2.43.GIT\n\n"},{"id":"488029","messageId":"12bd721ddff7020eb9e9ebd4e797d50193250bc0.1707200355.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1707200355.git.ps@pks.im","subject":"[PATCH v3 1/9] reftable: introduce macros to grow arrays","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-06T06:35:23Z","receivedAt":"2024-02-06T06:35:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Throughout the reftable library we have many cases where we need to grow\narrays. In order to avoid too many reallocations, we roughly double the\ncapacity of the array on each iteration. The resulting code pattern is\nduplicated across many sites.\n\nWe have similar patterns in our main codebase, which is why we have\neventually introduced an `ALLOC_GROW()` macro to abstract it away and\navoid some code duplication. We cannot easily reuse this macro here\nthough because `ALLOC_GROW()` uses `REALLOC_ARRAY()`, which in turn will\ncall realloc(3P) to grow the array. The reftable code is structured as a\nlibrary though (even if the boundaries are fuzzy), and one property this\nbrings with it is that it is possible to plug in your own allocators. So\ninstead of using realloc(3P), we need to use `reftable_realloc()` that\nknows to use the user-provided implementation.\n\nSo let's introduce two new macros `REFTABLE_REALLOC_ARRAY()` and\n`REFTABLE_ALLOC_GROW()` that mirror what we do in our main codebase,\nwith two modifications:\n\n - They use `reftable_realloc()`, as explained above.\n\n - They use a different growth factor of `2 * cap + 1` instead of `(cap\n + 16) * 3 / 2`.\n\nThe second change is because we know a bit more about the allocation\npatterns in the reftable library. In most cases, we end up only having a\nhandful of items in the array and don't end up growing them. The initial\ncapacity that our normal growth factor uses (which is 24) would thus end\nup over-allocating in a lot of code paths. This effect is measurable:\n\n - Before change:\n\n HEAP SUMMARY:\n in use at exit: 671,983 bytes in 152 blocks\n total heap usage: 3,843,446 allocs, 3,843,294 frees, 223,761,402 bytes allocated\n\n - After change with a growth factor of `(2 * alloc + 1)`:\n\n HEAP SUMMARY:\n in use at exit: 671,983 bytes in 152 blocks\n total heap usage: 3,843,446 allocs, 3,843,294 frees, 223,761,410 bytes allocated\n\n - After change with a growth factor of `(alloc + 16)* 2 / 3`:\n\n HEAP SUMMARY:\n in use at exit: 671,983 bytes in 152 blocks\n total heap usage: 3,833,673 allocs, 3,833,521 frees, 4,728,251,742 bytes allocated\n\nWhile the total heap usage is roughly the same, we do end up allocating\nsignificantly more bytes with our usual growth factor (in fact, roughly\n21 times as many).\n\nConvert the reftable library to use these new macros.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 8 ++------\n reftable/basics.h | 11 +++++++++++\n reftable/block.c | 7 +------\n reftable/merged_test.c | 20 ++++++--------------\n reftable/pq.c | 8 ++------\n reftable/stack.c | 29 ++++++++++++-----------------\n reftable/writer.c | 14 ++------------\n 7 files changed, 36 insertions(+), 61 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex f761e48028..af9004cec2 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -89,17 +89,13 @@ void parse_names(char *buf, int size, char ***namesp)\n \t\t\tnext = end;\n \t\t}\n \t\tif (p < next) {\n-\t\t\tif (names_len == names_cap) {\n-\t\t\t\tnames_cap = 2 * names_cap + 1;\n-\t\t\t\tnames = reftable_realloc(\n-\t\t\t\t\tnames, names_cap * sizeof(*names));\n-\t\t\t}\n+\t\t\tREFTABLE_ALLOC_GROW(names, names_len + 1, names_cap);\n \t\t\tnames[names_len++] = xstrdup(p);\n \t\t}\n \t\tp = next + 1;\n \t}\n \n-\tnames = reftable_realloc(names, (names_len + 1) * sizeof(*names));\n+\tREFTABLE_REALLOC_ARRAY(names, names_len + 1);\n \tnames[names_len] = NULL;\n \t*namesp = names;\n }\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 096b36862b..2f855cd724 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -53,6 +53,17 @@ void *reftable_realloc(void *p, size_t sz);\n void reftable_free(void *p);\n void *reftable_calloc(size_t sz);\n \n+#define REFTABLE_REALLOC_ARRAY(x, alloc) (x) = reftable_realloc((x), st_mult(sizeof(*(x)), (alloc)))\n+#define REFTABLE_ALLOC_GROW(x, nr, alloc) \\\n+\tdo { \\\n+\t\tif ((nr) > alloc) { \\\n+\t\t\talloc = 2 * (alloc) + 1; \\\n+\t\t\tif (alloc < (nr)) \\\n+\t\t\t\talloc = (nr); \\\n+\t\t\tREFTABLE_REALLOC_ARRAY(x, alloc); \\\n+\t\t} \\\n+\t} while (0)\n+\n /* Find the longest shared prefix size of `a` and `b` */\n struct strbuf;\n int common_prefix_size(struct strbuf *a, struct strbuf *b);\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 1df3d8a0f0..6952d0facf 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -51,12 +51,7 @@ static int block_writer_register_restart(struct block_writer *w, int n,\n \tif (2 + 3 * rlen + n > w->block_size - w->next)\n \t\treturn -1;\n \tif (is_restart) {\n-\t\tif (w->restart_len == w->restart_cap) {\n-\t\t\tw->restart_cap = w->restart_cap * 2 + 1;\n-\t\t\tw->restarts = reftable_realloc(\n-\t\t\t\tw->restarts, sizeof(uint32_t) * w->restart_cap);\n-\t\t}\n-\n+\t\tREFTABLE_ALLOC_GROW(w->restarts, w->restart_len + 1, w->restart_cap);\n \t\tw->restarts[w->restart_len++] = w->next;\n \t}\n \ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nindex 46908f738f..e05351e035 100644\n--- a/reftable/merged_test.c\n+++ b/reftable/merged_test.c\n@@ -231,14 +231,10 @@ static void test_merged(void)\n \twhile (len < 100) { /* cap loops/recursion. */\n \t\tstruct reftable_ref_record ref = { NULL };\n \t\tint err = reftable_iterator_next_ref(&it, &ref);\n-\t\tif (err > 0) {\n+\t\tif (err > 0)\n \t\t\tbreak;\n-\t\t}\n-\t\tif (len == cap) {\n-\t\t\tcap = 2 * cap + 1;\n-\t\t\tout = reftable_realloc(\n-\t\t\t\tout, sizeof(struct reftable_ref_record) * cap);\n-\t\t}\n+\n+\t\tREFTABLE_ALLOC_GROW(out, len + 1, cap);\n \t\tout[len++] = ref;\n \t}\n \treftable_iterator_destroy(&it);\n@@ -368,14 +364,10 @@ static void test_merged_logs(void)\n \twhile (len < 100) { /* cap loops/recursion. */\n \t\tstruct reftable_log_record log = { NULL };\n \t\tint err = reftable_iterator_next_log(&it, &log);\n-\t\tif (err > 0) {\n+\t\tif (err > 0)\n \t\t\tbreak;\n-\t\t}\n-\t\tif (len == cap) {\n-\t\t\tcap = 2 * cap + 1;\n-\t\t\tout = reftable_realloc(\n-\t\t\t\tout, sizeof(struct reftable_log_record) * cap);\n-\t\t}\n+\n+\t\tREFTABLE_ALLOC_GROW(out, len + 1, cap);\n \t\tout[len++] = log;\n \t}\n \treftable_iterator_destroy(&it);\ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex dcefeb793a..2461daf5ff 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -75,13 +75,9 @@ void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry\n {\n \tint i = 0;\n \n-\tif (pq->len == pq->cap) {\n-\t\tpq->cap = 2 * pq->cap + 1;\n-\t\tpq->heap = reftable_realloc(pq->heap,\n-\t\t\t\t\t pq->cap * sizeof(struct pq_entry));\n-\t}\n-\n+\tREFTABLE_ALLOC_GROW(pq->heap, pq->len + 1, pq->cap);\n \tpq->heap[pq->len++] = *e;\n+\n \ti = pq->len - 1;\n \twhile (i > 0) {\n \t\tint j = (i - 1) / 2;\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex bf3869ce70..1dfab99e96 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -551,7 +551,7 @@ struct reftable_addition {\n \tstruct reftable_stack *stack;\n \n \tchar **new_tables;\n-\tint new_tables_len;\n+\tsize_t new_tables_len, new_tables_cap;\n \tuint64_t next_update_index;\n };\n \n@@ -602,8 +602,9 @@ static int reftable_stack_init_addition(struct reftable_addition *add,\n \n static void reftable_addition_close(struct reftable_addition *add)\n {\n-\tint i = 0;\n \tstruct strbuf nm = STRBUF_INIT;\n+\tsize_t i;\n+\n \tfor (i = 0; i < add->new_tables_len; i++) {\n \t\tstack_filename(&nm, add->stack, add->new_tables[i]);\n \t\tunlink(nm.buf);\n@@ -613,6 +614,7 @@ static void reftable_addition_close(struct reftable_addition *add)\n \treftable_free(add->new_tables);\n \tadd->new_tables = NULL;\n \tadd->new_tables_len = 0;\n+\tadd->new_tables_cap = 0;\n \n \tdelete_tempfile(&add->lock_file);\n \tstrbuf_release(&nm);\n@@ -631,8 +633,8 @@ int reftable_addition_commit(struct reftable_addition *add)\n {\n \tstruct strbuf table_list = STRBUF_INIT;\n \tint lock_file_fd = get_tempfile_fd(add->lock_file);\n-\tint i = 0;\n \tint err = 0;\n+\tsize_t i;\n \n \tif (add->new_tables_len == 0)\n \t\tgoto done;\n@@ -660,12 +662,12 @@ int reftable_addition_commit(struct reftable_addition *add)\n \t}\n \n \t/* success, no more state to clean up. */\n-\tfor (i = 0; i < add->new_tables_len; i++) {\n+\tfor (i = 0; i < add->new_tables_len; i++)\n \t\treftable_free(add->new_tables[i]);\n-\t}\n \treftable_free(add->new_tables);\n \tadd->new_tables = NULL;\n \tadd->new_tables_len = 0;\n+\tadd->new_tables_cap = 0;\n \n \terr = reftable_stack_reload_maybe_reuse(add->stack, 1);\n \tif (err)\n@@ -792,11 +794,9 @@ int reftable_addition_add(struct reftable_addition *add,\n \t\tgoto done;\n \t}\n \n-\tadd->new_tables = reftable_realloc(add->new_tables,\n-\t\t\t\t\t sizeof(*add->new_tables) *\n-\t\t\t\t\t\t (add->new_tables_len + 1));\n-\tadd->new_tables[add->new_tables_len] = strbuf_detach(&next_name, NULL);\n-\tadd->new_tables_len++;\n+\tREFTABLE_ALLOC_GROW(add->new_tables, add->new_tables_len + 1,\n+\t\t\t add->new_tables_cap);\n+\tadd->new_tables[add->new_tables_len++] = strbuf_detach(&next_name, NULL);\n done:\n \tif (tab_fd > 0) {\n \t\tclose(tab_fd);\n@@ -1367,17 +1367,12 @@ static int stack_check_addition(struct reftable_stack *st,\n \twhile (1) {\n \t\tstruct reftable_ref_record ref = { NULL };\n \t\terr = reftable_iterator_next_ref(&it, &ref);\n-\t\tif (err > 0) {\n+\t\tif (err > 0)\n \t\t\tbreak;\n-\t\t}\n \t\tif (err < 0)\n \t\t\tgoto done;\n \n-\t\tif (len >= cap) {\n-\t\t\tcap = 2 * cap + 1;\n-\t\t\trefs = reftable_realloc(refs, cap * sizeof(refs[0]));\n-\t\t}\n-\n+\t\tREFTABLE_ALLOC_GROW(refs, len + 1, cap);\n \t\trefs[len++] = ref;\n \t}\n \ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex ee4590e20f..4483bb21c3 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -200,12 +200,7 @@ static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n \t\treturn;\n \t}\n \n-\tif (key->offset_len == key->offset_cap) {\n-\t\tkey->offset_cap = 2 * key->offset_cap + 1;\n-\t\tkey->offsets = reftable_realloc(\n-\t\t\tkey->offsets, sizeof(uint64_t) * key->offset_cap);\n-\t}\n-\n+\tREFTABLE_ALLOC_GROW(key->offsets, key->offset_len + 1, key->offset_cap);\n \tkey->offsets[key->offset_len++] = off;\n }\n \n@@ -674,12 +669,7 @@ static int writer_flush_nonempty_block(struct reftable_writer *w)\n \tif (err < 0)\n \t\treturn err;\n \n-\tif (w->index_cap == w->index_len) {\n-\t\tw->index_cap = 2 * w->index_cap + 1;\n-\t\tw->index = reftable_realloc(\n-\t\t\tw->index,\n-\t\t\tsizeof(struct reftable_index_record) * w->index_cap);\n-\t}\n+\tREFTABLE_ALLOC_GROW(w->index, w->index_len + 1, w->index_cap);\n \n \tir.offset = w->next;\n \tstrbuf_reset(&ir.last_key);\n-- \n2.43.GIT\n\n"},{"id":"488030","messageId":"95689ca7ce235a5717e44a0a6d5c495310babba2.1707200355.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1707200355.git.ps@pks.im","subject":"[PATCH v3 2/9] reftable: introduce macros to allocate arrays","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-06T06:35:27Z","receivedAt":"2024-02-06T06:35:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Similar to the preceding commit, let's carry over macros to allocate\narrays with `REFTABLE_ALLOC_ARRAY()` and `REFTABLE_CALLOC_ARRAY()`. This\nrequires us to change the signature of `reftable_calloc()`, which only\ntakes a single argument right now and thus puts the burden on the caller\nto calculate the final array's size. This is a net improvement though as\nit means that we can now provide proper overflow checks when multiplying\nthe array size with the member size.\n\nConvert callsites of `reftable_calloc()` to the new signature and start\nusing the new macros where possible.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 4 +++-\n reftable/block.c | 10 ++++++----\n reftable/block_test.c | 2 +-\n reftable/blocksource.c | 4 ++--\n reftable/iter.c | 3 +--\n reftable/merged.c | 4 ++--\n reftable/merged_test.c | 22 +++++++++++++---------\n reftable/publicbasics.c | 3 ++-\n reftable/reader.c | 8 +++-----\n reftable/readwrite_test.c | 8 +++++---\n reftable/record.c | 14 ++++++++------\n reftable/record_test.c | 4 ++--\n reftable/refname.c | 4 ++--\n reftable/stack.c | 28 +++++++++++++---------------\n reftable/tree.c | 4 ++--\n reftable/writer.c | 7 +++----\n 16 files changed, 68 insertions(+), 61 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 2f855cd724..4c3ac963a3 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -51,8 +51,10 @@ int names_length(char **names);\n void *reftable_malloc(size_t sz);\n void *reftable_realloc(void *p, size_t sz);\n void reftable_free(void *p);\n-void *reftable_calloc(size_t sz);\n+void *reftable_calloc(size_t nelem, size_t elsize);\n \n+#define REFTABLE_ALLOC_ARRAY(x, alloc) (x) = reftable_malloc(st_mult(sizeof(*(x)), (alloc)))\n+#define REFTABLE_CALLOC_ARRAY(x, alloc) (x) = reftable_calloc((alloc), sizeof(*(x)))\n #define REFTABLE_REALLOC_ARRAY(x, alloc) (x) = reftable_realloc((x), st_mult(sizeof(*(x)), (alloc)))\n #define REFTABLE_ALLOC_GROW(x, nr, alloc) \\\n \tdo { \\\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 6952d0facf..838759823a 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -143,8 +143,10 @@ int block_writer_finish(struct block_writer *w)\n \t\tint block_header_skip = 4 + w->header_off;\n \t\tuLongf src_len = w->next - block_header_skip;\n \t\tuLongf dest_cap = src_len * 1.001 + 12;\n+\t\tuint8_t *compressed;\n+\n+\t\tREFTABLE_ALLOC_ARRAY(compressed, dest_cap);\n \n-\t\tuint8_t *compressed = reftable_malloc(dest_cap);\n \t\twhile (1) {\n \t\t\tuLongf out_dest_len = dest_cap;\n \t\t\tint zresult = compress2(compressed, &out_dest_len,\n@@ -201,9 +203,9 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,\n \t\tuLongf dst_len = sz - block_header_skip; /* total size of dest\n \t\t\t\t\t\t\t buffer. */\n \t\tuLongf src_len = block->len - block_header_skip;\n-\t\t/* Log blocks specify the *uncompressed* size in their header.\n-\t\t */\n-\t\tuncompressed = reftable_malloc(sz);\n+\n+\t\t/* Log blocks specify the *uncompressed* size in their header. */\n+\t\tREFTABLE_ALLOC_ARRAY(uncompressed, sz);\n \n \t\t/* Copy over the block header verbatim. It's not compressed. */\n \t\tmemcpy(uncompressed, block->data, block_header_skip);\ndiff --git a/reftable/block_test.c b/reftable/block_test.c\nindex dedb05c7d8..e162c6e33f 100644\n--- a/reftable/block_test.c\n+++ b/reftable/block_test.c\n@@ -36,7 +36,7 @@ static void test_block_read_write(void)\n \tint j = 0;\n \tstruct strbuf want = STRBUF_INIT;\n \n-\tblock.data = reftable_calloc(block_size);\n+\tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock.source = malloc_block_source();\n \tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex 8c41e3c70f..eeed254ba9 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -29,7 +29,7 @@ static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n {\n \tstruct strbuf *b = v;\n \tassert(off + size <= b->len);\n-\tdest->data = reftable_calloc(size);\n+\tREFTABLE_CALLOC_ARRAY(dest->data, size);\n \tmemcpy(dest->data, b->buf + off, size);\n \tdest->len = size;\n \treturn size;\n@@ -132,7 +132,7 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,\n \t\treturn REFTABLE_IO_ERROR;\n \t}\n \n-\tp = reftable_calloc(sizeof(*p));\n+\tREFTABLE_CALLOC_ARRAY(p, 1);\n \tp->size = st.st_size;\n \tp->data = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tclose(fd);\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex a8d174c040..8b5ebf6183 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -160,8 +160,7 @@ int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n \t\t\t int oid_len, uint64_t *offsets, int offset_len)\n {\n \tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n-\tstruct indexed_table_ref_iter *itr =\n-\t\treftable_calloc(sizeof(struct indexed_table_ref_iter));\n+\tstruct indexed_table_ref_iter *itr = reftable_calloc(1, sizeof(*itr));\n \tint err = 0;\n \n \t*itr = empty;\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex c258ce953e..2031fd51b4 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -190,7 +190,7 @@ int reftable_new_merged_table(struct reftable_merged_table **dest,\n \t\t}\n \t}\n \n-\tm = reftable_calloc(sizeof(struct reftable_merged_table));\n+\tREFTABLE_CALLOC_ARRAY(m, 1);\n \tm->stack = stack;\n \tm->stack_len = n;\n \tm->min = first_min;\n@@ -240,7 +240,7 @@ static int merged_table_seek_record(struct reftable_merged_table *mt,\n \t\t\t\t struct reftable_record *rec)\n {\n \tstruct reftable_iterator *iters = reftable_calloc(\n-\t\tsizeof(struct reftable_iterator) * mt->stack_len);\n+\t\tmt->stack_len, sizeof(*iters));\n \tstruct merged_iter merged = {\n \t\t.stack = iters,\n \t\t.typ = reftable_record_type(rec),\ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nindex e05351e035..e233a9d581 100644\n--- a/reftable/merged_test.c\n+++ b/reftable/merged_test.c\n@@ -93,10 +93,12 @@ merged_table_from_records(struct reftable_ref_record **refs,\n \tint i = 0;\n \tstruct reftable_merged_table *mt = NULL;\n \tint err;\n-\tstruct reftable_table *tabs =\n-\t\treftable_calloc(n * sizeof(struct reftable_table));\n-\t*readers = reftable_calloc(n * sizeof(struct reftable_reader *));\n-\t*source = reftable_calloc(n * sizeof(**source));\n+\tstruct reftable_table *tabs;\n+\n+\tREFTABLE_CALLOC_ARRAY(tabs, n);\n+\tREFTABLE_CALLOC_ARRAY(*readers, n);\n+\tREFTABLE_CALLOC_ARRAY(*source, n);\n+\n \tfor (i = 0; i < n; i++) {\n \t\twrite_test_table(&buf[i], refs[i], sizes[i]);\n \t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n@@ -266,10 +268,12 @@ merged_table_from_log_records(struct reftable_log_record **logs,\n \tint i = 0;\n \tstruct reftable_merged_table *mt = NULL;\n \tint err;\n-\tstruct reftable_table *tabs =\n-\t\treftable_calloc(n * sizeof(struct reftable_table));\n-\t*readers = reftable_calloc(n * sizeof(struct reftable_reader *));\n-\t*source = reftable_calloc(n * sizeof(**source));\n+\tstruct reftable_table *tabs;\n+\n+\tREFTABLE_CALLOC_ARRAY(tabs, n);\n+\tREFTABLE_CALLOC_ARRAY(*readers, n);\n+\tREFTABLE_CALLOC_ARRAY(*source, n);\n+\n \tfor (i = 0; i < n; i++) {\n \t\twrite_test_log_table(&buf[i], logs[i], sizes[i], i + 1);\n \t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n@@ -412,7 +416,7 @@ static void test_default_write_opts(void)\n \t};\n \tint err;\n \tstruct reftable_block_source source = { NULL };\n-\tstruct reftable_table *tab = reftable_calloc(sizeof(*tab) * 1);\n+\tstruct reftable_table *tab = reftable_calloc(1, sizeof(*tab));\n \tuint32_t hash_id;\n \tstruct reftable_reader *rd = NULL;\n \tstruct reftable_merged_table *merged = NULL;\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\nindex bcb82530d6..44b84a125e 100644\n--- a/reftable/publicbasics.c\n+++ b/reftable/publicbasics.c\n@@ -37,8 +37,9 @@ void reftable_free(void *p)\n \t\tfree(p);\n }\n \n-void *reftable_calloc(size_t sz)\n+void *reftable_calloc(size_t nelem, size_t elsize)\n {\n+\tsize_t sz = st_mult(nelem, elsize);\n \tvoid *p = reftable_malloc(sz);\n \tmemset(p, 0, sz);\n \treturn p;\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 64dc366fb1..3e0de5e8ad 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -539,8 +539,7 @@ static int reader_seek_indexed(struct reftable_reader *r,\n \n \tif (err == 0) {\n \t\tstruct table_iter empty = TABLE_ITER_INIT;\n-\t\tstruct table_iter *malloced =\n-\t\t\treftable_calloc(sizeof(struct table_iter));\n+\t\tstruct table_iter *malloced = reftable_calloc(1, sizeof(*malloced));\n \t\t*malloced = empty;\n \t\ttable_iter_copy_from(malloced, &next);\n \t\titerator_from_table_iter(it, malloced);\n@@ -635,8 +634,7 @@ void reader_close(struct reftable_reader *r)\n int reftable_new_reader(struct reftable_reader **p,\n \t\t\tstruct reftable_block_source *src, char const *name)\n {\n-\tstruct reftable_reader *rd =\n-\t\treftable_calloc(sizeof(struct reftable_reader));\n+\tstruct reftable_reader *rd = reftable_calloc(1, sizeof(*rd));\n \tint err = init_reader(rd, src, name);\n \tif (err == 0) {\n \t\t*p = rd;\n@@ -711,7 +709,7 @@ static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n \t\t\t\t\t uint8_t *oid)\n {\n \tstruct table_iter ti_empty = TABLE_ITER_INIT;\n-\tstruct table_iter *ti = reftable_calloc(sizeof(struct table_iter));\n+\tstruct table_iter *ti = reftable_calloc(1, sizeof(*ti));\n \tstruct filtering_ref_iterator *filter = NULL;\n \tstruct filtering_ref_iterator empty = FILTERING_REF_ITERATOR_INIT;\n \tint oid_len = hash_size(r->hash_id);\ndiff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c\nindex b8a3224016..91ea7a10ef 100644\n--- a/reftable/readwrite_test.c\n+++ b/reftable/readwrite_test.c\n@@ -56,7 +56,9 @@ static void write_table(char ***names, struct strbuf *buf, int N,\n \tint i = 0, n;\n \tstruct reftable_log_record log = { NULL };\n \tconst struct reftable_stats *stats = NULL;\n-\t*names = reftable_calloc(sizeof(char *) * (N + 1));\n+\n+\tREFTABLE_CALLOC_ARRAY(*names, N + 1);\n+\n \treftable_writer_set_limits(w, update_index, update_index);\n \tfor (i = 0; i < N; i++) {\n \t\tchar name[100];\n@@ -188,7 +190,7 @@ static void test_log_overflow(void)\n static void test_log_write_read(void)\n {\n \tint N = 2;\n-\tchar **names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tchar **names = reftable_calloc(N + 1, sizeof(*names));\n \tint err;\n \tstruct reftable_write_options opts = {\n \t\t.block_size = 256,\n@@ -519,7 +521,7 @@ static void test_table_read_write_seek_index(void)\n static void test_table_refs_for(int indexed)\n {\n \tint N = 50;\n-\tchar **want_names = reftable_calloc(sizeof(char *) * (N + 1));\n+\tchar **want_names = reftable_calloc(N + 1, sizeof(*want_names));\n \tint want_names_len = 0;\n \tuint8_t want_hash[GIT_SHA1_RAWSZ];\n \ndiff --git a/reftable/record.c b/reftable/record.c\nindex 5c3fbb7b2a..2f3cd6b62c 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -497,12 +497,13 @@ static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n \t\t(const struct reftable_obj_record *)src_rec;\n \n \treftable_obj_record_release(obj);\n-\tobj->hash_prefix = reftable_malloc(src->hash_prefix_len);\n+\n+\tREFTABLE_ALLOC_ARRAY(obj->hash_prefix, src->hash_prefix_len);\n \tobj->hash_prefix_len = src->hash_prefix_len;\n \tif (src->hash_prefix_len)\n \t\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n \n-\tobj->offsets = reftable_malloc(src->offset_len * sizeof(uint64_t));\n+\tREFTABLE_ALLOC_ARRAY(obj->offsets, src->offset_len);\n \tobj->offset_len = src->offset_len;\n \tCOPY_ARRAY(obj->offsets, src->offsets, src->offset_len);\n }\n@@ -559,7 +560,8 @@ static int reftable_obj_record_decode(void *rec, struct strbuf key,\n \tint n = 0;\n \tuint64_t last;\n \tint j;\n-\tr->hash_prefix = reftable_malloc(key.len);\n+\n+\tREFTABLE_ALLOC_ARRAY(r->hash_prefix, key.len);\n \tmemcpy(r->hash_prefix, key.buf, key.len);\n \tr->hash_prefix_len = key.len;\n \n@@ -577,7 +579,7 @@ static int reftable_obj_record_decode(void *rec, struct strbuf key,\n \tif (count == 0)\n \t\treturn start.len - in.len;\n \n-\tr->offsets = reftable_malloc(count * sizeof(uint64_t));\n+\tREFTABLE_ALLOC_ARRAY(r->offsets, count);\n \tr->offset_len = count;\n \n \tn = get_var_int(&r->offsets[0], &in);\n@@ -715,12 +717,12 @@ static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n \t\t}\n \n \t\tif (dst->value.update.new_hash) {\n-\t\t\tdst->value.update.new_hash = reftable_malloc(hash_size);\n+\t\t\tREFTABLE_ALLOC_ARRAY(dst->value.update.new_hash, hash_size);\n \t\t\tmemcpy(dst->value.update.new_hash,\n \t\t\t src->value.update.new_hash, hash_size);\n \t\t}\n \t\tif (dst->value.update.old_hash) {\n-\t\t\tdst->value.update.old_hash = reftable_malloc(hash_size);\n+\t\t\tREFTABLE_ALLOC_ARRAY(dst->value.update.old_hash, hash_size);\n \t\t\tmemcpy(dst->value.update.old_hash,\n \t\t\t src->value.update.old_hash, hash_size);\n \t\t}\ndiff --git a/reftable/record_test.c b/reftable/record_test.c\nindex 2876db7d27..999366ad46 100644\n--- a/reftable/record_test.c\n+++ b/reftable/record_test.c\n@@ -231,8 +231,8 @@ static void test_reftable_log_record_roundtrip(void)\n \t\t\t\t.value_type = REFTABLE_LOG_UPDATE,\n \t\t\t\t.value = {\n \t\t\t\t\t.update = {\n-\t\t\t\t\t\t.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n-\t\t\t\t\t\t.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),\n+\t\t\t\t\t\t.new_hash = reftable_calloc(GIT_SHA1_RAWSZ, 1),\n+\t\t\t\t\t\t.old_hash = reftable_calloc(GIT_SHA1_RAWSZ, 1),\n \t\t\t\t\t\t.name = xstrdup(\"old name\"),\n \t\t\t\t\t\t.email = xstrdup(\"old@email\"),\n \t\t\t\t\t\t.message = xstrdup(\"old message\"),\ndiff --git a/reftable/refname.c b/reftable/refname.c\nindex 9573496932..7570e4acf9 100644\n--- a/reftable/refname.c\n+++ b/reftable/refname.c\n@@ -140,8 +140,8 @@ int validate_ref_record_addition(struct reftable_table tab,\n {\n \tstruct modification mod = {\n \t\t.tab = tab,\n-\t\t.add = reftable_calloc(sizeof(char *) * sz),\n-\t\t.del = reftable_calloc(sizeof(char *) * sz),\n+\t\t.add = reftable_calloc(sz, sizeof(*mod.add)),\n+\t\t.del = reftable_calloc(sz, sizeof(*mod.del)),\n \t};\n \tint i = 0;\n \tint err = 0;\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 1dfab99e96..a7b2c61026 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -50,8 +50,7 @@ static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz)\n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \t\t struct reftable_write_options config)\n {\n-\tstruct reftable_stack *p =\n-\t\treftable_calloc(sizeof(struct reftable_stack));\n+\tstruct reftable_stack *p = reftable_calloc(1, sizeof(*p));\n \tstruct strbuf list_file_name = STRBUF_INIT;\n \tint err = 0;\n \n@@ -94,7 +93,7 @@ static int fd_read_lines(int fd, char ***namesp)\n \t\tgoto done;\n \t}\n \n-\tbuf = reftable_malloc(size + 1);\n+\tREFTABLE_ALLOC_ARRAY(buf, size + 1);\n \tif (read_in_full(fd, buf, size) != size) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n@@ -114,7 +113,7 @@ int read_lines(const char *filename, char ***namesp)\n \tint err = 0;\n \tif (fd < 0) {\n \t\tif (errno == ENOENT) {\n-\t\t\t*namesp = reftable_calloc(sizeof(char *));\n+\t\t\tREFTABLE_CALLOC_ARRAY(*namesp, 1);\n \t\t\treturn 0;\n \t\t}\n \n@@ -191,8 +190,7 @@ void reftable_stack_destroy(struct reftable_stack *st)\n static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n \t\t\t\t\t\t int cur_len)\n {\n-\tstruct reftable_reader **cur =\n-\t\treftable_calloc(sizeof(struct reftable_reader *) * cur_len);\n+\tstruct reftable_reader **cur = reftable_calloc(cur_len, sizeof(*cur));\n \tint i = 0;\n \tfor (i = 0; i < cur_len; i++) {\n \t\tcur[i] = st->readers[i];\n@@ -208,9 +206,9 @@ static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n \tint err = 0;\n \tint names_len = names_length(names);\n \tstruct reftable_reader **new_readers =\n-\t\treftable_calloc(sizeof(struct reftable_reader *) * names_len);\n+\t\treftable_calloc(names_len, sizeof(*new_readers));\n \tstruct reftable_table *new_tables =\n-\t\treftable_calloc(sizeof(struct reftable_table) * names_len);\n+\t\treftable_calloc(names_len, sizeof(*new_tables));\n \tint new_readers_len = 0;\n \tstruct reftable_merged_table *new_merged = NULL;\n \tstruct strbuf table_path = STRBUF_INIT;\n@@ -344,7 +342,7 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\t\t\tgoto out;\n \t\t\t}\n \n-\t\t\tnames = reftable_calloc(sizeof(char *));\n+\t\t\tREFTABLE_CALLOC_ARRAY(names, 1);\n \t\t} else {\n \t\t\terr = fd_read_lines(fd, &names);\n \t\t\tif (err < 0)\n@@ -686,7 +684,7 @@ int reftable_stack_new_addition(struct reftable_addition **dest,\n {\n \tint err = 0;\n \tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n-\t*dest = reftable_calloc(sizeof(**dest));\n+\tREFTABLE_CALLOC_ARRAY(*dest, 1);\n \t**dest = empty;\n \terr = reftable_stack_init_addition(*dest, st);\n \tif (err) {\n@@ -871,7 +869,7 @@ static int stack_write_compact(struct reftable_stack *st,\n {\n \tint subtabs_len = last - first + 1;\n \tstruct reftable_table *subtabs = reftable_calloc(\n-\t\tsizeof(struct reftable_table) * (last - first + 1));\n+\t\tlast - first + 1, sizeof(*subtabs));\n \tstruct reftable_merged_table *mt = NULL;\n \tint err = 0;\n \tstruct reftable_iterator it = { NULL };\n@@ -979,9 +977,9 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n \tint compact_count = last - first + 1;\n \tchar **listp = NULL;\n \tchar **delete_on_success =\n-\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\t\treftable_calloc(compact_count + 1, sizeof(*delete_on_success));\n \tchar **subtable_locks =\n-\t\treftable_calloc(sizeof(char *) * (compact_count + 1));\n+\t\treftable_calloc(compact_count + 1, sizeof(*subtable_locks));\n \tint i = 0;\n \tint j = 0;\n \tint is_empty_table = 0;\n@@ -1204,7 +1202,7 @@ int fastlog2(uint64_t sz)\n \n struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n {\n-\tstruct segment *segs = reftable_calloc(sizeof(struct segment) * n);\n+\tstruct segment *segs = reftable_calloc(n, sizeof(*segs));\n \tint next = 0;\n \tstruct segment cur = { 0 };\n \tint i = 0;\n@@ -1268,7 +1266,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, int n)\n static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n {\n \tuint64_t *sizes =\n-\t\treftable_calloc(sizeof(uint64_t) * st->merged->stack_len);\n+\t\treftable_calloc(st->merged->stack_len, sizeof(*sizes));\n \tint version = (st->config.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2;\n \tint overhead = header_size(version) - 1;\n \tint i = 0;\ndiff --git a/reftable/tree.c b/reftable/tree.c\nindex a5bf880985..528f33ae38 100644\n--- a/reftable/tree.c\n+++ b/reftable/tree.c\n@@ -20,8 +20,8 @@ struct tree_node *tree_search(void *key, struct tree_node **rootp,\n \t\tif (!insert) {\n \t\t\treturn NULL;\n \t\t} else {\n-\t\t\tstruct tree_node *n =\n-\t\t\t\treftable_calloc(sizeof(struct tree_node));\n+\t\t\tstruct tree_node *n;\n+\t\t\tREFTABLE_CALLOC_ARRAY(n, 1);\n \t\t\tn->key = key;\n \t\t\t*rootp = n;\n \t\t\treturn *rootp;\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 4483bb21c3..47b3448132 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -49,7 +49,7 @@ static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n {\n \tint n = 0;\n \tif (w->pending_padding > 0) {\n-\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding);\n+\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding, sizeof(*zeroed));\n \t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n \t\tif (n < 0)\n \t\t\treturn n;\n@@ -123,8 +123,7 @@ struct reftable_writer *\n reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\t void *writer_arg, struct reftable_write_options *opts)\n {\n-\tstruct reftable_writer *wp =\n-\t\treftable_calloc(sizeof(struct reftable_writer));\n+\tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n \tstrbuf_init(&wp->block_writer_data.last_key, 0);\n \toptions_set_defaults(opts);\n \tif (opts->block_size >= (1 << 24)) {\n@@ -132,7 +131,7 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \t\tabort();\n \t}\n \twp->last_key = reftable_empty_strbuf;\n-\twp->block = reftable_calloc(opts->block_size);\n+\tREFTABLE_CALLOC_ARRAY(wp->block, opts->block_size);\n \twp->write = writer_func;\n \twp->write_arg = writer_arg;\n \twp->opts = *opts;\n-- \n2.43.GIT\n\n"},{"id":"488031","messageId":"f0e8f08884769fef048997ef6ba2c8dd1633a1c5.1707200355.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1707200355.git.ps@pks.im","subject":"[PATCH v3 3/9] reftable/stack: fix parameter validation when compacting range","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-06T06:35:31Z","receivedAt":"2024-02-06T06:35:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `stack_compact_range()` function receives a \"first\" and \"last\" index\nthat indicates which tables of the reftable stack should be compacted.\nNaturally, \"first\" must be smaller than \"last\" in order to identify a\nproper range of tables to compress, which we indeed also assert in the\nfunction. But the validations happens after we have already allocated\narrays with a size of `last - first + 1`, leading to an underflow and\nthus an invalid allocation size.\n\nFix this by reordering the array allocations to happen after we have\nvalidated parameters. While at it, convert the array allocations to use\nthe newly introduced macros.\n\nNote that the relevant variables pointing into arrays should also be\nconverted to use `size_t` instead of `int`. This is left for a later\ncommit in this series.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 24 +++++++++++++-----------\n 1 file changed, 13 insertions(+), 11 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex a7b2c61026..1de2f6751c 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -966,6 +966,7 @@ static int stack_write_compact(struct reftable_stack *st,\n static int stack_compact_range(struct reftable_stack *st, int first, int last,\n \t\t\t struct reftable_log_expiry_config *expiry)\n {\n+\tchar **delete_on_success = NULL, **subtable_locks = NULL, **listp = NULL;\n \tstruct strbuf temp_tab_file_name = STRBUF_INIT;\n \tstruct strbuf new_table_name = STRBUF_INIT;\n \tstruct strbuf lock_file_name = STRBUF_INIT;\n@@ -974,12 +975,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n \tint err = 0;\n \tint have_lock = 0;\n \tint lock_file_fd = -1;\n-\tint compact_count = last - first + 1;\n-\tchar **listp = NULL;\n-\tchar **delete_on_success =\n-\t\treftable_calloc(compact_count + 1, sizeof(*delete_on_success));\n-\tchar **subtable_locks =\n-\t\treftable_calloc(compact_count + 1, sizeof(*subtable_locks));\n+\tint compact_count;\n \tint i = 0;\n \tint j = 0;\n \tint is_empty_table = 0;\n@@ -989,6 +985,10 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n \t\tgoto done;\n \t}\n \n+\tcompact_count = last - first + 1;\n+\tREFTABLE_CALLOC_ARRAY(delete_on_success, compact_count + 1);\n+\tREFTABLE_CALLOC_ARRAY(subtable_locks, compact_count + 1);\n+\n \tst->stats.attempts++;\n \n \tstrbuf_reset(&lock_file_name);\n@@ -1146,12 +1146,14 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n done:\n \tfree_names(delete_on_success);\n \n-\tlistp = subtable_locks;\n-\twhile (*listp) {\n-\t\tunlink(*listp);\n-\t\tlistp++;\n+\tif (subtable_locks) {\n+\t\tlistp = subtable_locks;\n+\t\twhile (*listp) {\n+\t\t\tunlink(*listp);\n+\t\t\tlistp++;\n+\t\t}\n+\t\tfree_names(subtable_locks);\n \t}\n-\tfree_names(subtable_locks);\n \tif (lock_file_fd >= 0) {\n \t\tclose(lock_file_fd);\n \t\tlock_file_fd = -1;\n-- \n2.43.GIT\n\n"},{"id":"488032","messageId":"7bcfe7b305b800f0d7c20d8322ca8b68b4f9d656.1707200355.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1707200355.git.ps@pks.im","subject":"[PATCH v3 4/9] reftable/stack: index segments with `size_t`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-06T06:35:35Z","receivedAt":"2024-02-06T06:35:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We use `int`s to index into arrays of segments and track the length of\nthem, which is considered to be a code smell in the Git project. Convert\nthe code to use `size_t` instead.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 25 +++++++++++--------------\n reftable/stack.h | 6 +++---\n reftable/stack_test.c | 7 +++----\n 3 files changed, 17 insertions(+), 21 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 1de2f6751c..5da4ea8141 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1202,12 +1202,11 @@ int fastlog2(uint64_t sz)\n \treturn l - 1;\n }\n \n-struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n+struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n)\n {\n \tstruct segment *segs = reftable_calloc(n, sizeof(*segs));\n-\tint next = 0;\n \tstruct segment cur = { 0 };\n-\tint i = 0;\n+\tsize_t next = 0, i;\n \n \tif (n == 0) {\n \t\t*seglen = 0;\n@@ -1233,29 +1232,27 @@ struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n)\n \treturn segs;\n }\n \n-struct segment suggest_compaction_segment(uint64_t *sizes, int n)\n+struct segment suggest_compaction_segment(uint64_t *sizes, size_t n)\n {\n-\tint seglen = 0;\n-\tstruct segment *segs = sizes_to_segments(&seglen, sizes, n);\n \tstruct segment min_seg = {\n \t\t.log = 64,\n \t};\n-\tint i = 0;\n+\tstruct segment *segs;\n+\tsize_t seglen = 0, i;\n+\n+\tsegs = sizes_to_segments(&seglen, sizes, n);\n \tfor (i = 0; i < seglen; i++) {\n-\t\tif (segment_size(&segs[i]) == 1) {\n+\t\tif (segment_size(&segs[i]) == 1)\n \t\t\tcontinue;\n-\t\t}\n \n-\t\tif (segs[i].log < min_seg.log) {\n+\t\tif (segs[i].log < min_seg.log)\n \t\t\tmin_seg = segs[i];\n-\t\t}\n \t}\n \n \twhile (min_seg.start > 0) {\n-\t\tint prev = min_seg.start - 1;\n-\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev])) {\n+\t\tsize_t prev = min_seg.start - 1;\n+\t\tif (fastlog2(min_seg.bytes) < fastlog2(sizes[prev]))\n \t\t\tbreak;\n-\t\t}\n \n \t\tmin_seg.start = prev;\n \t\tmin_seg.bytes += sizes[prev];\ndiff --git a/reftable/stack.h b/reftable/stack.h\nindex c1e3efa899..d919455669 100644\n--- a/reftable/stack.h\n+++ b/reftable/stack.h\n@@ -32,13 +32,13 @@ struct reftable_stack {\n int read_lines(const char *filename, char ***lines);\n \n struct segment {\n-\tint start, end;\n+\tsize_t start, end;\n \tint log;\n \tuint64_t bytes;\n };\n \n int fastlog2(uint64_t sz);\n-struct segment *sizes_to_segments(int *seglen, uint64_t *sizes, int n);\n-struct segment suggest_compaction_segment(uint64_t *sizes, int n);\n+struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n);\n+struct segment suggest_compaction_segment(uint64_t *sizes, size_t n);\n \n #endif\ndiff --git a/reftable/stack_test.c b/reftable/stack_test.c\nindex 289e902146..2d5b24e5c5 100644\n--- a/reftable/stack_test.c\n+++ b/reftable/stack_test.c\n@@ -711,7 +711,7 @@ static void test_sizes_to_segments(void)\n \tuint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };\n \t/* .................0 1 2 3 4 5 */\n \n-\tint seglen = 0;\n+\tsize_t seglen = 0;\n \tstruct segment *segs =\n \t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n \tEXPECT(segs[2].log == 3);\n@@ -726,7 +726,7 @@ static void test_sizes_to_segments(void)\n \n static void test_sizes_to_segments_empty(void)\n {\n-\tint seglen = 0;\n+\tsize_t seglen = 0;\n \tstruct segment *segs = sizes_to_segments(&seglen, NULL, 0);\n \tEXPECT(seglen == 0);\n \treftable_free(segs);\n@@ -735,8 +735,7 @@ static void test_sizes_to_segments_empty(void)\n static void test_sizes_to_segments_all_equal(void)\n {\n \tuint64_t sizes[] = { 5, 5 };\n-\n-\tint seglen = 0;\n+\tsize_t seglen = 0;\n \tstruct segment *segs =\n \t\tsizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes));\n \tEXPECT(seglen == 1);\n-- \n2.43.GIT\n\n"},{"id":"488033","messageId":"a0867c037831b4a73e465e630ca6810663b34dac.1707200355.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1707200355.git.ps@pks.im","subject":"[PATCH v3 5/9] reftable/stack: use `size_t` to track stack slices during compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-06T06:35:41Z","receivedAt":"2024-02-06T06:35:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We use `int`s to track reftable slices when compacting the reftable\nstack, which is considered to be a code smell in the Git project.\nConvert the code to use `size_t` instead.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 32 ++++++++++++++++----------------\n 1 file changed, 16 insertions(+), 16 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 5da4ea8141..a86481a9a6 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -24,7 +24,8 @@ static int stack_try_add(struct reftable_stack *st,\n \t\t\t\t\t void *arg),\n \t\t\t void *arg);\n static int stack_write_compact(struct reftable_stack *st,\n-\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_writer *wr,\n+\t\t\t size_t first, size_t last,\n \t\t\t struct reftable_log_expiry_config *config);\n static int stack_check_addition(struct reftable_stack *st,\n \t\t\t\tconst char *new_tab_name);\n@@ -820,7 +821,8 @@ uint64_t reftable_stack_next_update_index(struct reftable_stack *st)\n \treturn 1;\n }\n \n-static int stack_compact_locked(struct reftable_stack *st, int first, int last,\n+static int stack_compact_locked(struct reftable_stack *st,\n+\t\t\t\tsize_t first, size_t last,\n \t\t\t\tstruct strbuf *temp_tab,\n \t\t\t\tstruct reftable_log_expiry_config *config)\n {\n@@ -864,22 +866,21 @@ static int stack_compact_locked(struct reftable_stack *st, int first, int last,\n }\n \n static int stack_write_compact(struct reftable_stack *st,\n-\t\t\t struct reftable_writer *wr, int first, int last,\n+\t\t\t struct reftable_writer *wr,\n+\t\t\t size_t first, size_t last,\n \t\t\t struct reftable_log_expiry_config *config)\n {\n \tint subtabs_len = last - first + 1;\n \tstruct reftable_table *subtabs = reftable_calloc(\n \t\tlast - first + 1, sizeof(*subtabs));\n \tstruct reftable_merged_table *mt = NULL;\n-\tint err = 0;\n \tstruct reftable_iterator it = { NULL };\n \tstruct reftable_ref_record ref = { NULL };\n \tstruct reftable_log_record log = { NULL };\n-\n \tuint64_t entries = 0;\n+\tint err = 0;\n \n-\tint i = 0, j = 0;\n-\tfor (i = first, j = 0; i <= last; i++) {\n+\tfor (size_t i = first, j = 0; i <= last; i++) {\n \t\tstruct reftable_reader *t = st->readers[i];\n \t\treftable_table_from_reader(&subtabs[j++], t);\n \t\tst->stats.bytes += t->size;\n@@ -963,7 +964,8 @@ static int stack_write_compact(struct reftable_stack *st,\n }\n \n /* < 0: error. 0 == OK, > 0 attempt failed; could retry. */\n-static int stack_compact_range(struct reftable_stack *st, int first, int last,\n+static int stack_compact_range(struct reftable_stack *st,\n+\t\t\t size_t first, size_t last,\n \t\t\t struct reftable_log_expiry_config *expiry)\n {\n \tchar **delete_on_success = NULL, **subtable_locks = NULL, **listp = NULL;\n@@ -972,12 +974,10 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n \tstruct strbuf lock_file_name = STRBUF_INIT;\n \tstruct strbuf ref_list_contents = STRBUF_INIT;\n \tstruct strbuf new_table_path = STRBUF_INIT;\n+\tsize_t i, j, compact_count;\n \tint err = 0;\n \tint have_lock = 0;\n \tint lock_file_fd = -1;\n-\tint compact_count;\n-\tint i = 0;\n-\tint j = 0;\n \tint is_empty_table = 0;\n \n \tif (first > last || (!expiry && first == last)) {\n@@ -1172,17 +1172,17 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,\n int reftable_stack_compact_all(struct reftable_stack *st,\n \t\t\t struct reftable_log_expiry_config *config)\n {\n-\treturn stack_compact_range(st, 0, st->merged->stack_len - 1, config);\n+\treturn stack_compact_range(st, 0, st->merged->stack_len ?\n+\t\t\tst->merged->stack_len - 1 : 0, config);\n }\n \n-static int stack_compact_range_stats(struct reftable_stack *st, int first,\n-\t\t\t\t int last,\n+static int stack_compact_range_stats(struct reftable_stack *st,\n+\t\t\t\t size_t first, size_t last,\n \t\t\t\t struct reftable_log_expiry_config *config)\n {\n \tint err = stack_compact_range(st, first, last, config);\n-\tif (err > 0) {\n+\tif (err > 0)\n \t\tst->stats.failures++;\n-\t}\n \treturn err;\n }\n \n-- \n2.43.GIT\n\n"},{"id":"488034","messageId":"29c5a54ae8754e106769d9ba50f0e4547a32d993.1707200355.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1707200355.git.ps@pks.im","subject":"[PATCH v3 6/9] reftable/stack: use `size_t` to track stack length","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-06T06:35:46Z","receivedAt":"2024-02-06T06:35:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"While the stack length is already stored as `size_t`, we frequently use\n`int`s to refer to those stacks throughout the reftable library. Convert\nthose cases to use `size_t` instead to make things consistent.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 7 +++----\n reftable/basics.h | 2 +-\n reftable/merged.c | 11 +++++------\n reftable/merged_test.c | 14 ++++++--------\n reftable/reftable-merged.h | 2 +-\n reftable/stack.c | 21 ++++++++++-----------\n 6 files changed, 26 insertions(+), 31 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex af9004cec2..0785aff941 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -64,12 +64,11 @@ void free_names(char **a)\n \treftable_free(a);\n }\n \n-int names_length(char **names)\n+size_t names_length(char **names)\n {\n \tchar **p = names;\n-\tfor (; *p; p++) {\n-\t\t/* empty */\n-\t}\n+\twhile (*p)\n+\t\tp++;\n \treturn p - names;\n }\n \ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 4c3ac963a3..91f3533efe 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -44,7 +44,7 @@ void parse_names(char *buf, int size, char ***namesp);\n int names_equal(char **a, char **b);\n \n /* returns the array size of a NULL-terminated array of strings. */\n-int names_length(char **names);\n+size_t names_length(char **names);\n \n /* Allocation routines; they invoke the functions set through\n * reftable_set_alloc() */\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 2031fd51b4..e2c6253324 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -45,11 +45,10 @@ static int merged_iter_init(struct merged_iter *mi)\n static void merged_iter_close(void *p)\n {\n \tstruct merged_iter *mi = p;\n-\tint i = 0;\n+\n \tmerged_iter_pqueue_release(&mi->pq);\n-\tfor (i = 0; i < mi->stack_len; i++) {\n+\tfor (size_t i = 0; i < mi->stack_len; i++)\n \t\treftable_iterator_destroy(&mi->stack[i]);\n-\t}\n \treftable_free(mi->stack);\n \tstrbuf_release(&mi->key);\n \tstrbuf_release(&mi->entry_key);\n@@ -168,14 +167,14 @@ static void iterator_from_merged_iter(struct reftable_iterator *it,\n }\n \n int reftable_new_merged_table(struct reftable_merged_table **dest,\n-\t\t\t struct reftable_table *stack, int n,\n+\t\t\t struct reftable_table *stack, size_t n,\n \t\t\t uint32_t hash_id)\n {\n \tstruct reftable_merged_table *m = NULL;\n \tuint64_t last_max = 0;\n \tuint64_t first_min = 0;\n-\tint i = 0;\n-\tfor (i = 0; i < n; i++) {\n+\n+\tfor (size_t i = 0; i < n; i++) {\n \t\tuint64_t min = reftable_table_min_update_index(&stack[i]);\n \t\tuint64_t max = reftable_table_max_update_index(&stack[i]);\n \ndiff --git a/reftable/merged_test.c b/reftable/merged_test.c\nindex e233a9d581..442917cc83 100644\n--- a/reftable/merged_test.c\n+++ b/reftable/merged_test.c\n@@ -88,18 +88,17 @@ static struct reftable_merged_table *\n merged_table_from_records(struct reftable_ref_record **refs,\n \t\t\t struct reftable_block_source **source,\n \t\t\t struct reftable_reader ***readers, int *sizes,\n-\t\t\t struct strbuf *buf, int n)\n+\t\t\t struct strbuf *buf, size_t n)\n {\n-\tint i = 0;\n \tstruct reftable_merged_table *mt = NULL;\n-\tint err;\n \tstruct reftable_table *tabs;\n+\tint err;\n \n \tREFTABLE_CALLOC_ARRAY(tabs, n);\n \tREFTABLE_CALLOC_ARRAY(*readers, n);\n \tREFTABLE_CALLOC_ARRAY(*source, n);\n \n-\tfor (i = 0; i < n; i++) {\n+\tfor (size_t i = 0; i < n; i++) {\n \t\twrite_test_table(&buf[i], refs[i], sizes[i]);\n \t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n \n@@ -263,18 +262,17 @@ static struct reftable_merged_table *\n merged_table_from_log_records(struct reftable_log_record **logs,\n \t\t\t struct reftable_block_source **source,\n \t\t\t struct reftable_reader ***readers, int *sizes,\n-\t\t\t struct strbuf *buf, int n)\n+\t\t\t struct strbuf *buf, size_t n)\n {\n-\tint i = 0;\n \tstruct reftable_merged_table *mt = NULL;\n-\tint err;\n \tstruct reftable_table *tabs;\n+\tint err;\n \n \tREFTABLE_CALLOC_ARRAY(tabs, n);\n \tREFTABLE_CALLOC_ARRAY(*readers, n);\n \tREFTABLE_CALLOC_ARRAY(*source, n);\n \n-\tfor (i = 0; i < n; i++) {\n+\tfor (size_t i = 0; i < n; i++) {\n \t\twrite_test_log_table(&buf[i], logs[i], sizes[i], i + 1);\n \t\tblock_source_from_strbuf(&(*source)[i], &buf[i]);\n \ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nindex 1a6d16915a..c91a2d83a2 100644\n--- a/reftable/reftable-merged.h\n+++ b/reftable/reftable-merged.h\n@@ -33,7 +33,7 @@ struct reftable_table;\n the stack array.\n */\n int reftable_new_merged_table(struct reftable_merged_table **dest,\n-\t\t\t struct reftable_table *stack, int n,\n+\t\t\t struct reftable_table *stack, size_t n,\n \t\t\t uint32_t hash_id);\n \n /* returns an iterator positioned just before 'name' */\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex a86481a9a6..bb684a3dc1 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -202,18 +202,18 @@ static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n \t\t\t\t int reuse_open)\n {\n-\tint cur_len = !st->merged ? 0 : st->merged->stack_len;\n+\tsize_t cur_len = !st->merged ? 0 : st->merged->stack_len;\n \tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n-\tint err = 0;\n-\tint names_len = names_length(names);\n+\tsize_t names_len = names_length(names);\n \tstruct reftable_reader **new_readers =\n \t\treftable_calloc(names_len, sizeof(*new_readers));\n \tstruct reftable_table *new_tables =\n \t\treftable_calloc(names_len, sizeof(*new_tables));\n-\tint new_readers_len = 0;\n+\tsize_t new_readers_len = 0;\n \tstruct reftable_merged_table *new_merged = NULL;\n \tstruct strbuf table_path = STRBUF_INIT;\n-\tint i;\n+\tint err = 0;\n+\tsize_t i;\n \n \twhile (*names) {\n \t\tstruct reftable_reader *rd = NULL;\n@@ -221,11 +221,10 @@ static int reftable_stack_reload_once(struct reftable_stack *st, char **names,\n \n \t\t/* this is linear; we assume compaction keeps the number of\n \t\t tables under control so this is not quadratic. */\n-\t\tint j = 0;\n-\t\tfor (j = 0; reuse_open && j < cur_len; j++) {\n-\t\t\tif (cur[j] && 0 == strcmp(cur[j]->name, name)) {\n-\t\t\t\trd = cur[j];\n-\t\t\t\tcur[j] = NULL;\n+\t\tfor (i = 0; reuse_open && i < cur_len; i++) {\n+\t\t\tif (cur[i] && 0 == strcmp(cur[i]->name, name)) {\n+\t\t\t\trd = cur[i];\n+\t\t\t\tcur[i] = NULL;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t}\n@@ -870,7 +869,7 @@ static int stack_write_compact(struct reftable_stack *st,\n \t\t\t size_t first, size_t last,\n \t\t\t struct reftable_log_expiry_config *config)\n {\n-\tint subtabs_len = last - first + 1;\n+\tsize_t subtabs_len = last - first + 1;\n \tstruct reftable_table *subtabs = reftable_calloc(\n \t\tlast - first + 1, sizeof(*subtabs));\n \tstruct reftable_merged_table *mt = NULL;\n-- \n2.43.GIT\n\n"},{"id":"488035","messageId":"4605ad724746db8c33710e54369105214481dbfb.1707200355.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1707200355.git.ps@pks.im","subject":"[PATCH v3 7/9] reftable/merged: refactor seeking of records","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-06T06:35:50Z","receivedAt":"2024-02-06T06:35:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The code to seek reftable records in the merged table code is quite hard\nto read and does not conform to our coding style in multiple ways:\n\n - We have multiple exit paths where we release resources even though\n that is not really necessary.\n\n - We use a scoped error variable `e` which is hard to reason about.\n This variable is not required at all.\n\n - We allocate memory in the variable declarations, which is easy to\n miss.\n\nRefactor the function so that it becomes more maintainable in the\nfuture.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 54 ++++++++++++++++++-----------------------------\n 1 file changed, 21 insertions(+), 33 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex e2c6253324..0abcda26e8 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -238,50 +238,38 @@ static int merged_table_seek_record(struct reftable_merged_table *mt,\n \t\t\t\t struct reftable_iterator *it,\n \t\t\t\t struct reftable_record *rec)\n {\n-\tstruct reftable_iterator *iters = reftable_calloc(\n-\t\tmt->stack_len, sizeof(*iters));\n \tstruct merged_iter merged = {\n-\t\t.stack = iters,\n \t\t.typ = reftable_record_type(rec),\n \t\t.hash_id = mt->hash_id,\n \t\t.suppress_deletions = mt->suppress_deletions,\n \t\t.key = STRBUF_INIT,\n \t\t.entry_key = STRBUF_INIT,\n \t};\n-\tint n = 0;\n-\tint err = 0;\n-\tint i = 0;\n-\tfor (i = 0; i < mt->stack_len && err == 0; i++) {\n-\t\tint e = reftable_table_seek_record(&mt->stack[i], &iters[n],\n-\t\t\t\t\t\t rec);\n-\t\tif (e < 0) {\n-\t\t\terr = e;\n-\t\t}\n-\t\tif (e == 0) {\n-\t\t\tn++;\n-\t\t}\n-\t}\n-\tif (err < 0) {\n-\t\tint i = 0;\n-\t\tfor (i = 0; i < n; i++) {\n-\t\t\treftable_iterator_destroy(&iters[i]);\n-\t\t}\n-\t\treftable_free(iters);\n-\t\treturn err;\n+\tstruct merged_iter *p;\n+\tint err;\n+\n+\tREFTABLE_CALLOC_ARRAY(merged.stack, mt->stack_len);\n+\tfor (size_t i = 0; i < mt->stack_len; i++) {\n+\t\terr = reftable_table_seek_record(&mt->stack[i],\n+\t\t\t\t\t\t &merged.stack[merged.stack_len], rec);\n+\t\tif (err < 0)\n+\t\t\tgoto out;\n+\t\tif (!err)\n+\t\t\tmerged.stack_len++;\n \t}\n \n-\tmerged.stack_len = n;\n \terr = merged_iter_init(&merged);\n-\tif (err < 0) {\n+\tif (err < 0)\n+\t\tgoto out;\n+\n+\tp = reftable_malloc(sizeof(struct merged_iter));\n+\t*p = merged;\n+\titerator_from_merged_iter(it, p);\n+\n+out:\n+\tif (err < 0)\n \t\tmerged_iter_close(&merged);\n-\t\treturn err;\n-\t} else {\n-\t\tstruct merged_iter *p =\n-\t\t\treftable_malloc(sizeof(struct merged_iter));\n-\t\t*p = merged;\n-\t\titerator_from_merged_iter(it, p);\n-\t}\n-\treturn 0;\n+\treturn err;\n }\n \n int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,\n-- \n2.43.GIT\n\n"},{"id":"488036","messageId":"8c35968ce82666e2fe0c055699e7bd1ac4de0de8.1707200355.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1707200355.git.ps@pks.im","subject":"[PATCH v3 8/9] reftable/merged: refactor initialization of iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-06T06:35:55Z","receivedAt":"2024-02-06T06:36:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor the initialization of the merged iterator to fit our code style\nbetter. This refactoring prepares the code for a refactoring of how\nrecords are being initialized.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 27 +++++++++++++--------------\n 1 file changed, 13 insertions(+), 14 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 0abcda26e8..0e60e2a39b 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -19,24 +19,23 @@ license that can be found in the LICENSE file or at\n \n static int merged_iter_init(struct merged_iter *mi)\n {\n-\tint i = 0;\n-\tfor (i = 0; i < mi->stack_len; i++) {\n-\t\tstruct reftable_record rec = reftable_new_record(mi->typ);\n-\t\tint err = iterator_next(&mi->stack[i], &rec);\n-\t\tif (err < 0) {\n+\tfor (size_t i = 0; i < mi->stack_len; i++) {\n+\t\tstruct pq_entry e = {\n+\t\t\t.rec = reftable_new_record(mi->typ),\n+\t\t\t.index = i,\n+\t\t};\n+\t\tint err;\n+\n+\t\terr = iterator_next(&mi->stack[i], &e.rec);\n+\t\tif (err < 0)\n \t\t\treturn err;\n-\t\t}\n-\n \t\tif (err > 0) {\n \t\t\treftable_iterator_destroy(&mi->stack[i]);\n-\t\t\treftable_record_release(&rec);\n-\t\t} else {\n-\t\t\tstruct pq_entry e = {\n-\t\t\t\t.rec = rec,\n-\t\t\t\t.index = i,\n-\t\t\t};\n-\t\t\tmerged_iter_pqueue_add(&mi->pq, &e);\n+\t\t\treftable_record_release(&e.rec);\n+\t\t\tcontinue;\n \t\t}\n+\n+\t\tmerged_iter_pqueue_add(&mi->pq, &e);\n \t}\n \n \treturn 0;\n-- \n2.43.GIT\n\n"},{"id":"488037","messageId":"5bb2858c1366367e293ca81838c08b643420026b.1707200355.git.ps@pks.im","threadId":"60822","inReplyTo":"cover.1707200355.git.ps@pks.im","subject":"[PATCH v3 9/9] reftable/record: improve semantics when initializing records","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-06T06:35:59Z","receivedAt":"2024-02-06T06:36:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"According to our usual coding style, the `reftable_new_record()`\nfunction would indicate that it is allocating a new record. This is not\nthe case though as the function merely initializes records without\nallocating any memory.\n\nReplace `reftable_new_record()` with a new `reftable_record_init()`\nfunction that takes a record pointer as input and initializes it\naccordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/block.c | 18 +++++++++---------\n reftable/merged.c | 8 +++++---\n reftable/reader.c | 4 ++--\n reftable/record.c | 43 ++++++++++--------------------------------\n reftable/record.h | 10 +++++-----\n reftable/record_test.c | 4 ++--\n 6 files changed, 33 insertions(+), 54 deletions(-)\n\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 838759823a..ce8a7d24f3 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -382,23 +382,23 @@ int block_reader_seek(struct block_reader *br, struct block_iter *it,\n \t\t.key = *want,\n \t\t.r = br,\n \t};\n-\tstruct reftable_record rec = reftable_new_record(block_reader_type(br));\n-\tint err = 0;\n \tstruct block_iter next = BLOCK_ITER_INIT;\n+\tstruct reftable_record rec;\n+\tint err = 0, i;\n \n-\tint i = binsearch(br->restart_count, &restart_key_less, &args);\n \tif (args.error) {\n \t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n \t}\n \n-\tit->br = br;\n-\tif (i > 0) {\n-\t\ti--;\n-\t\tit->next_off = block_reader_restart_offset(br, i);\n-\t} else {\n+\ti = binsearch(br->restart_count, &restart_key_less, &args);\n+\tif (i > 0)\n+\t\tit->next_off = block_reader_restart_offset(br, i - 1);\n+\telse\n \t\tit->next_off = br->header_off + 4;\n-\t}\n+\tit->br = br;\n+\n+\treftable_record_init(&rec, block_reader_type(br));\n \n \t/* We're looking for the last entry less/equal than the wanted key, so\n \t we have to go one entry too far and then back up.\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 0e60e2a39b..a0f222e07b 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -21,11 +21,11 @@ static int merged_iter_init(struct merged_iter *mi)\n {\n \tfor (size_t i = 0; i < mi->stack_len; i++) {\n \t\tstruct pq_entry e = {\n-\t\t\t.rec = reftable_new_record(mi->typ),\n \t\t\t.index = i,\n \t\t};\n \t\tint err;\n \n+\t\treftable_record_init(&e.rec, mi->typ);\n \t\terr = iterator_next(&mi->stack[i], &e.rec);\n \t\tif (err < 0)\n \t\t\treturn err;\n@@ -57,10 +57,12 @@ static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,\n \t\t\t\t\t size_t idx)\n {\n \tstruct pq_entry e = {\n-\t\t.rec = reftable_new_record(mi->typ),\n \t\t.index = idx,\n \t};\n-\tint err = iterator_next(&mi->stack[idx], &e.rec);\n+\tint err;\n+\n+\treftable_record_init(&e.rec, mi->typ);\n+\terr = iterator_next(&mi->stack[idx], &e.rec);\n \tif (err < 0)\n \t\treturn err;\n \ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 3e0de5e8ad..5e6c8f30a1 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -444,13 +444,13 @@ static int reader_start(struct reftable_reader *r, struct table_iter *ti,\n static int reader_seek_linear(struct table_iter *ti,\n \t\t\t struct reftable_record *want)\n {\n-\tstruct reftable_record rec =\n-\t\treftable_new_record(reftable_record_type(want));\n \tstruct strbuf want_key = STRBUF_INIT;\n \tstruct strbuf got_key = STRBUF_INIT;\n \tstruct table_iter next = TABLE_ITER_INIT;\n+\tstruct reftable_record rec;\n \tint err = -1;\n \n+\treftable_record_init(&rec, reftable_record_type(want));\n \treftable_record_key(want, &want_key);\n \n \twhile (1) {\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 2f3cd6b62c..26c5e43f9b 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -1259,45 +1259,22 @@ reftable_record_vtable(struct reftable_record *rec)\n \tabort();\n }\n \n-struct reftable_record reftable_new_record(uint8_t typ)\n+void reftable_record_init(struct reftable_record *rec, uint8_t typ)\n {\n-\tstruct reftable_record clean = {\n-\t\t.type = typ,\n-\t};\n+\tmemset(rec, 0, sizeof(*rec));\n+\trec->type = typ;\n \n-\t/* the following is involved, but the naive solution (just return\n-\t * `clean` as is, except for BLOCK_TYPE_INDEX), returns a garbage\n-\t * clean.u.obj.offsets pointer on Windows VS CI. Go figure.\n-\t */\n \tswitch (typ) {\n-\tcase BLOCK_TYPE_OBJ:\n-\t{\n-\t\tstruct reftable_obj_record obj = { 0 };\n-\t\tclean.u.obj = obj;\n-\t\tbreak;\n-\t}\n-\tcase BLOCK_TYPE_INDEX:\n-\t{\n-\t\tstruct reftable_index_record idx = {\n-\t\t\t.last_key = STRBUF_INIT,\n-\t\t};\n-\t\tclean.u.idx = idx;\n-\t\tbreak;\n-\t}\n \tcase BLOCK_TYPE_REF:\n-\t{\n-\t\tstruct reftable_ref_record ref = { 0 };\n-\t\tclean.u.ref = ref;\n-\t\tbreak;\n-\t}\n \tcase BLOCK_TYPE_LOG:\n-\t{\n-\t\tstruct reftable_log_record log = { 0 };\n-\t\tclean.u.log = log;\n-\t\tbreak;\n-\t}\n+\tcase BLOCK_TYPE_OBJ:\n+\t\treturn;\n+\tcase BLOCK_TYPE_INDEX:\n+\t\tstrbuf_init(&rec->u.idx.last_key, 0);\n+\t\treturn;\n+\tdefault:\n+\t\tBUG(\"unhandled record type\");\n \t}\n-\treturn clean;\n }\n \n void reftable_record_print(struct reftable_record *rec, int hash_size)\ndiff --git a/reftable/record.h b/reftable/record.h\nindex fd80cd451d..e64ed30c80 100644\n--- a/reftable/record.h\n+++ b/reftable/record.h\n@@ -69,9 +69,6 @@ struct reftable_record_vtable {\n /* returns true for recognized block types. Block start with the block type. */\n int reftable_is_block_type(uint8_t typ);\n \n-/* return an initialized record for the given type */\n-struct reftable_record reftable_new_record(uint8_t typ);\n-\n /* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns\n * number of bytes written. */\n int reftable_encode_key(int *is_restart, struct string_view dest,\n@@ -100,8 +97,8 @@ struct reftable_obj_record {\n /* record is a generic wrapper for different types of records. It is normally\n * created on the stack, or embedded within another struct. If the type is\n * known, a fresh instance can be initialized explicitly. Otherwise, use\n- * reftable_new_record() to initialize generically (as the index_record is not\n- * valid as 0-initialized structure)\n+ * `reftable_record_init()` to initialize generically (as the index_record is\n+ * not valid as 0-initialized structure)\n */\n struct reftable_record {\n \tuint8_t type;\n@@ -113,6 +110,9 @@ struct reftable_record {\n \t} u;\n };\n \n+/* Initialize the reftable record for the given type */\n+void reftable_record_init(struct reftable_record *rec, uint8_t typ);\n+\n /* see struct record_vtable */\n int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);\n void reftable_record_print(struct reftable_record *rec, int hash_size);\ndiff --git a/reftable/record_test.c b/reftable/record_test.c\nindex 999366ad46..a86cff5526 100644\n--- a/reftable/record_test.c\n+++ b/reftable/record_test.c\n@@ -16,11 +16,11 @@\n \n static void test_copy(struct reftable_record *rec)\n {\n-\tstruct reftable_record copy = { 0 };\n+\tstruct reftable_record copy;\n \tuint8_t typ;\n \n \ttyp = reftable_record_type(rec);\n-\tcopy = reftable_new_record(typ);\n+\treftable_record_init(©, typ);\n \treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n \t/* do it twice to catch memory leaks */\n \treftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ);\n-- \n2.43.GIT\n\n"},{"id":"488056","messageId":"CAOLa=ZRAZjEPYO3QdbqZrAxwmC0W5p9icg9tyA1yp68beSaYNQ@mail.gmail.com","threadId":"60822","inReplyTo":"cover.1707200355.git.ps@pks.im","subject":"Re: [PATCH v3 0/9] reftable: code style improvements","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-02-06T09:10:42Z","receivedAt":"2024-02-06T09:10:43Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Hello,\n\nPatrick Steinhardt writes:\n> Hi,\n>\n> this is the third version of my patch series that tries to align the\n> reftable library's coding style to be closer to Git's own code style.\n>\n> The only change compared to v2 is that I've now also converted some\n> calls to `reftable_malloc()` to use `REFTABLE_ALLOC_ARRAY`.\n>\n\nThe range-diff looks good to me, since this was the only change I\nrequested, the patch series looks good now.\n\nThanks!\n"}]}