{"thread":{"id":"62121","subject":"[PATCH 00/22] reftable: handle allocation errors","startedAt":"2024-09-16T12:28:20Z","lastAt":"2024-10-04T05:43:37Z","messageCount":151,"participants":["Patrick Steinhardt","Junio C Hamano","René Scharfe","Han-Wen Nienhuys","Eric Sunshine"],"isPatch":true,"patchVersion":1,"patchTotal":22},"messages":[{"id":"502869","messageId":"cover.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":null,"subject":"[PATCH 00/22] reftable: handle allocation errors","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:28:15Z","receivedAt":"2024-09-16T12:28:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthe reftable library provides pluggable allocators that can be switched\nout via `reftable_set_alloc()`. These pluggable allocators are not\nexpected to behave like `xmalloc()` and friends -- in other words, they\nare allowed to fail. This is done such that the reftable library can\ntruly behave like a library and let its callers handle such conditions.\n\nIt has been true since the inception of the reftable library that\nallocations may fail. But regardless of that we do not have any error\nhandling for this scenario at all. Consequently, we can easily cause a\nsegfault when running into an out-of-memory situation.\n\nWhile the easy solution would be to plug in `xmalloc()` and friends such\nthat we abort on these cases, that would contradict the very notion of\nwhat this library should be. Most importantly, I want to soonish revive\nthe effort to implement a reftable backend in libgit2, and thus I have\nto make sure that the reftable library (of which we are the canonical\nupstream now) can be used in such contexts.\n\nThis patch series here is the first one in a set of series that will\ndetangle the reftable library from the rest of the Git codebase again so\nthat it becomes a true standalone library that other projects can use.\n\nThe series is based on top of ed155187b4 (Sync with Git 2.46.1,\n2024-09-13) with the following two depedencies merged into it to avoid\nconflicts:\n\n - cp/unit-test-reftable-stack at 2b14ced370 (t-reftable-stack: add\n test for stack iterators, 2024-09-08).\n\n - ps/reftable-exclude at adf7a0ceef (refs/reftable: wire up support\n for exclude patterns, 2024-09-09).\n\nThanks!\n\nPatrick\n\nPatrick Steinhardt (22):\n reftable/error: introduce out-of-memory error code\n reftable/basics: merge \"publicbasics\" into \"basics\"\n reftable: introduce `reftable_strdup()`\n reftable/basics: handle allocation failures in `reftable_calloc()`\n reftable/basics: handle allocation failures in `parse_names()`\n reftable/record: handle allocation failures on copy\n reftable/record: handle allocation failures when decoding records\n reftable/writer: handle allocation failures in `writer_index_hash()`\n reftable/writer: handle allocation failures in `reftable_new_writer()`\n reftable/merged: handle allocation failures in\n `merged_table_init_iter()`\n reftable/reader: handle allocation failures for unindexed reader\n reftable/reader: handle allocation failures in `reader_init_iter()`\n reftable/stack: handle allocation failures on reload\n reftable/stack: handle allocation failures in `reftable_new_stack()`\n reftable/stack: handle allocation failures in `stack_compact_range()`\n reftable/stack: handle allocation failures in auto compaction\n reftable/iter: handle allocation failures when creating indexed table\n iter\n reftable/blocksource: handle allocation failures\n reftable/block: handle allocation failures\n reftable/pq: handle allocation failures when adding entries\n reftable/tree: handle allocation failures\n reftable: handle trivial allocation failures\n\n Makefile | 1 -\n refs/reftable-backend.c | 39 ++++--\n reftable/basics.c | 87 +++++++++++++-\n reftable/basics.h | 13 +-\n reftable/block.c | 23 +++-\n reftable/block.h | 4 +-\n reftable/blocksource.c | 25 +++-\n reftable/error.c | 2 +\n reftable/iter.c | 22 +++-\n reftable/iter.h | 2 +-\n reftable/merged.c | 84 ++++++++-----\n reftable/merged.h | 6 +-\n reftable/pq.c | 7 +-\n reftable/pq.h | 2 +-\n reftable/publicbasics.c | 66 -----------\n reftable/reader.c | 68 ++++++++---\n reftable/reader.h | 6 +-\n reftable/record.c | 164 +++++++++++++++++++-------\n reftable/record.h | 6 +-\n reftable/reftable-basics.h | 18 +++\n reftable/reftable-error.h | 3 +\n reftable/reftable-malloc.h | 18 ---\n reftable/reftable-merged.h | 8 +-\n reftable/reftable-reader.h | 8 +-\n reftable/reftable-stack.h | 8 +-\n reftable/reftable-writer.h | 12 +-\n reftable/stack.c | 177 ++++++++++++++++++++++------\n reftable/tree.c | 42 +++++--\n reftable/tree.h | 21 +++-\n reftable/writer.c | 150 +++++++++++++++--------\n t/helper/test-reftable.c | 10 +-\n t/unit-tests/lib-reftable.c | 8 +-\n t/unit-tests/t-reftable-basics.c | 11 +-\n t/unit-tests/t-reftable-block.c | 24 ++--\n t/unit-tests/t-reftable-merged.c | 16 ++-\n t/unit-tests/t-reftable-readwrite.c | 61 ++++++----\n t/unit-tests/t-reftable-stack.c | 4 +-\n t/unit-tests/t-reftable-tree.c | 10 +-\n 38 files changed, 848 insertions(+), 388 deletions(-)\n delete mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-basics.h\n delete mode 100644 reftable/reftable-malloc.h\n\n\nbase-commit: 1755182f5a2e12ff27c230f34f0040a8afee58e4\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502870","messageId":"8c99ecc3255b1b10cdec047da01804dc23a7d287.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 01/22] reftable/error: introduce out-of-memory error code","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:28:18Z","receivedAt":"2024-09-16T12:28:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library does not use the same memory allocation functions\nas the rest of the Git codebase. Instead, as the reftable library is\nsupposed to be usable as a standalone library without Git, it provides a\nset of pluggable memory allocators.\n\nCompared to `xmalloc()` and friends these allocators are _not_ expected\nto die when an allocation fails. This design choice is concious, as a\nlibrary should leave it to its caller to handle any kind of error. While\nit is very likely that the caller cannot really do much in the case of\nan out-of-memory situation anyway, we are not the ones to make that\ndecision.\n\nCuriously though, we never handle allocation errors even though memory\nallocation functions are allowed to fail. And as we do not plug in Git's\nmemory allocator via `reftable_set_alloc()` either the consequence is\nthat we'd instead segfault as soon as we run out of memory.\n\nWhile the easy fix would be to wire up `xmalloc()` and friends, it\nwould only fix the usage of the reftable library in Git itself. Other\nusers like libgit2, which is about to revive its efforts to land a\nbackend for reftables, wouldn't be able to benefit from this solution.\n\nInstead, we are about to do it the hard way: adapt all allocation sites\nto perform error checking. Introduce a new error code for out-of-memory\nerrors that we will wire up in subsequent steps.\n\nThis commit also serves as the motivator for all the remaining steps in\nthis series such that we do not have to repeat the same arguments in\nevery single subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/error.c | 2 ++\n reftable/reftable-error.h | 3 +++\n 2 files changed, 5 insertions(+)\n\ndiff --git a/reftable/error.c b/reftable/error.c\nindex a25f28a43eb..660d0296170 100644\n--- a/reftable/error.c\n+++ b/reftable/error.c\n@@ -35,6 +35,8 @@ const char *reftable_error_str(int err)\n \t\treturn \"entry too large\";\n \tcase REFTABLE_OUTDATED_ERROR:\n \t\treturn \"data concurrently modified\";\n+\tcase REFTABLE_OUT_OF_MEMORY_ERROR:\n+\t\treturn \"out of memory\";\n \tcase -1:\n \t\treturn \"general error\";\n \tdefault:\ndiff --git a/reftable/reftable-error.h b/reftable/reftable-error.h\nindex 6368cd9ed9d..f4048265629 100644\n--- a/reftable/reftable-error.h\n+++ b/reftable/reftable-error.h\n@@ -57,6 +57,9 @@ enum reftable_error {\n \n \t/* Trying to write out-of-date data. */\n \tREFTABLE_OUTDATED_ERROR = -12,\n+\n+\t/* An allocation has failed due to an out-of-memory situation. */\n+\tREFTABLE_OUT_OF_MEMORY_ERROR = -13,\n };\n \n /* convert the numeric error code to a string. The string should not be\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502871","messageId":"4dcdf1d48ecf9a05cfa643d5a6165240898811a7.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 02/22] reftable/basics: merge \"publicbasics\" into \"basics\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:28:21Z","receivedAt":"2024-09-16T12:28:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The split between \"basics\" and \"publicbasics\" is somewhat arbitrary and\nnot in line with how we typically structure code in the reftable\nlibrary. While we do indeed split up headers into a public and internal\npart, we don't do that for the compilation unit itself. Furthermore, the\ndeclarations for \"publicbasics.c\" are in \"reftable-malloc.h\", which\nisn't in line with our naming schema, either.\n\nFix these inconsistencies by:\n\n - Merging \"publicbasics.c\" into \"basics.c\".\n\n - Renaming \"reftable-malloc.h\" to \"reftable-basics.h\" as the public\n header.\n\nSigned-off-by: Patrick Steinhardt \n---\n Makefile | 1 -\n reftable/basics.c | 55 +++++++++++++++++++++++++++++++\n reftable/basics.h | 3 ++\n reftable/publicbasics.c | 66 --------------------------------------\n reftable/reftable-basics.h | 18 +++++++++++\n reftable/reftable-malloc.h | 18 -----------\n 6 files changed, 76 insertions(+), 85 deletions(-)\n delete mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-basics.h\n delete mode 100644 reftable/reftable-malloc.h\n\ndiff --git a/Makefile b/Makefile\nindex e3abf0ba831..39b10923d49 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2683,7 +2683,6 @@ REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n-REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/merged.o\n REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 0058619ca67..cf072935c88 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -7,6 +7,49 @@ license that can be found in the LICENSE file or at\n */\n \n #include \"basics.h\"\n+#include \"reftable-basics.h\"\n+\n+static void *(*reftable_malloc_ptr)(size_t sz);\n+static void *(*reftable_realloc_ptr)(void *, size_t);\n+static void (*reftable_free_ptr)(void *);\n+\n+void *reftable_malloc(size_t sz)\n+{\n+\tif (reftable_malloc_ptr)\n+\t\treturn (*reftable_malloc_ptr)(sz);\n+\treturn malloc(sz);\n+}\n+\n+void *reftable_realloc(void *p, size_t sz)\n+{\n+\tif (reftable_realloc_ptr)\n+\t\treturn (*reftable_realloc_ptr)(p, sz);\n+\treturn realloc(p, sz);\n+}\n+\n+void reftable_free(void *p)\n+{\n+\tif (reftable_free_ptr)\n+\t\treftable_free_ptr(p);\n+\telse\n+\t\tfree(p);\n+}\n+\n+void *reftable_calloc(size_t nelem, size_t elsize)\n+{\n+\tsize_t sz = st_mult(nelem, elsize);\n+\tvoid *p = reftable_malloc(sz);\n+\tmemset(p, 0, sz);\n+\treturn p;\n+}\n+\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n+{\n+\treftable_malloc_ptr = malloc;\n+\treftable_realloc_ptr = realloc;\n+\treftable_free_ptr = free;\n+}\n \n void put_be24(uint8_t *out, uint32_t i)\n {\n@@ -121,3 +164,15 @@ int common_prefix_size(struct strbuf *a, struct strbuf *b)\n \n \treturn p;\n }\n+\n+int hash_size(uint32_t id)\n+{\n+\tswitch (id) {\n+\tcase 0:\n+\tcase GIT_SHA1_FORMAT_ID:\n+\t\treturn GIT_SHA1_RAWSZ;\n+\tcase GIT_SHA256_FORMAT_ID:\n+\t\treturn GIT_SHA256_RAWSZ;\n+\t}\n+\tabort();\n+}\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex c8fec68d4e8..4e2e76014aa 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -14,6 +14,7 @@ license that can be found in the LICENSE file or at\n */\n \n #include \"system.h\"\n+#include \"reftable-basics.h\"\n \n /* Bigendian en/decoding of integers */\n \n@@ -71,4 +72,6 @@ void *reftable_calloc(size_t nelem, size_t elsize);\n struct strbuf;\n int common_prefix_size(struct strbuf *a, struct strbuf *b);\n \n+int hash_size(uint32_t id);\n+\n #endif\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\ndeleted file mode 100644\nindex 44b84a125e4..00000000000\n--- a/reftable/publicbasics.c\n+++ /dev/null\n@@ -1,66 +0,0 @@\n-/*\n-Copyright 2020 Google LLC\n-\n-Use of this source code is governed by a BSD-style\n-license that can be found in the LICENSE file or at\n-https://developers.google.com/open-source/licenses/bsd\n-*/\n-\n-#include \"system.h\"\n-#include \"reftable-malloc.h\"\n-\n-#include \"basics.h\"\n-\n-static void *(*reftable_malloc_ptr)(size_t sz);\n-static void *(*reftable_realloc_ptr)(void *, size_t);\n-static void (*reftable_free_ptr)(void *);\n-\n-void *reftable_malloc(size_t sz)\n-{\n-\tif (reftable_malloc_ptr)\n-\t\treturn (*reftable_malloc_ptr)(sz);\n-\treturn malloc(sz);\n-}\n-\n-void *reftable_realloc(void *p, size_t sz)\n-{\n-\tif (reftable_realloc_ptr)\n-\t\treturn (*reftable_realloc_ptr)(p, sz);\n-\treturn realloc(p, sz);\n-}\n-\n-void reftable_free(void *p)\n-{\n-\tif (reftable_free_ptr)\n-\t\treftable_free_ptr(p);\n-\telse\n-\t\tfree(p);\n-}\n-\n-void *reftable_calloc(size_t nelem, size_t elsize)\n-{\n-\tsize_t sz = st_mult(nelem, elsize);\n-\tvoid *p = reftable_malloc(sz);\n-\tmemset(p, 0, sz);\n-\treturn p;\n-}\n-\n-void reftable_set_alloc(void *(*malloc)(size_t),\n-\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n-{\n-\treftable_malloc_ptr = malloc;\n-\treftable_realloc_ptr = realloc;\n-\treftable_free_ptr = free;\n-}\n-\n-int hash_size(uint32_t id)\n-{\n-\tswitch (id) {\n-\tcase 0:\n-\tcase GIT_SHA1_FORMAT_ID:\n-\t\treturn GIT_SHA1_RAWSZ;\n-\tcase GIT_SHA256_FORMAT_ID:\n-\t\treturn GIT_SHA256_RAWSZ;\n-\t}\n-\tabort();\n-}\ndiff --git a/reftable/reftable-basics.h b/reftable/reftable-basics.h\nnew file mode 100644\nindex 00000000000..6e8e636b716\n--- /dev/null\n+++ b/reftable/reftable-basics.h\n@@ -0,0 +1,18 @@\n+/*\n+ * Copyright 2020 Google LLC\n+ *\n+ * Use of this source code is governed by a BSD-style\n+ * license that can be found in the LICENSE file or at\n+ * https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_BASICS_H\n+#define REFTABLE_BASICS_H\n+\n+#include \n+\n+/* Overrides the functions to use for memory management. */\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n+\n+#endif\ndiff --git a/reftable/reftable-malloc.h b/reftable/reftable-malloc.h\ndeleted file mode 100644\nindex 5f2185f1f34..00000000000\n--- a/reftable/reftable-malloc.h\n+++ /dev/null\n@@ -1,18 +0,0 @@\n-/*\n-Copyright 2020 Google LLC\n-\n-Use of this source code is governed by a BSD-style\n-license that can be found in the LICENSE file or at\n-https://developers.google.com/open-source/licenses/bsd\n-*/\n-\n-#ifndef REFTABLE_H\n-#define REFTABLE_H\n-\n-#include \n-\n-/* Overrides the functions to use for memory management. */\n-void reftable_set_alloc(void *(*malloc)(size_t),\n-\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n-\n-#endif\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502872","messageId":"21fa9b15d96809728455943877c1b32781364041.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 03/22] reftable: introduce `reftable_strdup()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:28:23Z","receivedAt":"2024-09-16T12:28:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library provides the ability to swap out allocators. There\nis a gap here though, because we continue to use `xstrdup()` even in the\ncase where all the other allocators have been swapped out.\n\nIntroduce `reftable_strdup()` that uses `reftable_malloc()` to do the\nallocation.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 10 ++++++++++\n reftable/basics.h | 1 +\n 2 files changed, 11 insertions(+)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex cf072935c88..4adc98cf5de 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -43,6 +43,16 @@ void *reftable_calloc(size_t nelem, size_t elsize)\n \treturn p;\n }\n \n+char *reftable_strdup(const char *str)\n+{\n+\tsize_t len = strlen(str);\n+\tchar *result = reftable_malloc(len + 1);\n+\tif (!result)\n+\t\treturn NULL;\n+\tmemcpy(result, str, len + 1);\n+\treturn result;\n+}\n+\n void reftable_set_alloc(void *(*malloc)(size_t),\n \t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n {\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 4e2e76014aa..f107e148605 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -54,6 +54,7 @@ void *reftable_malloc(size_t sz);\n void *reftable_realloc(void *p, size_t sz);\n void reftable_free(void *p);\n void *reftable_calloc(size_t nelem, size_t elsize);\n+char *reftable_strdup(const char *str);\n \n #define REFTABLE_ALLOC_ARRAY(x, alloc) (x) = reftable_malloc(st_mult(sizeof(*(x)), (alloc)))\n #define REFTABLE_CALLOC_ARRAY(x, alloc) (x) = reftable_calloc((alloc), sizeof(*(x)))\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502873","messageId":"e6ded75f630ea309d5b76126560a0ec3d526bf71.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 04/22] reftable/basics: handle allocation failures in `reftable_calloc()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:28:26Z","receivedAt":"2024-09-16T12:28:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_calloc()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 4adc98cf5de..b404900b5d9 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -39,6 +39,8 @@ void *reftable_calloc(size_t nelem, size_t elsize)\n {\n \tsize_t sz = st_mult(nelem, elsize);\n \tvoid *p = reftable_malloc(sz);\n+\tif (!p)\n+\t\treturn NULL;\n \tmemset(p, 0, sz);\n \treturn p;\n }\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502874","messageId":"1f98abe9812e7fe951a62680e842c592231f2bf3.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 05/22] reftable/basics: handle allocation failures in `parse_names()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:28:29Z","receivedAt":"2024-09-16T12:28:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `parse_names()` by returning `NULL` in\ncase any allocation fails. While at it, refactor the function to return\nthe array directly instead of assigning it to an out-pointer.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 20 ++++++++++++++++----\n reftable/basics.h | 9 ++++++---\n reftable/stack.c | 6 +++++-\n t/unit-tests/t-reftable-basics.c | 11 ++++++-----\n 4 files changed, 33 insertions(+), 13 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex b404900b5d9..b9bbf061e9d 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -130,14 +130,14 @@ size_t names_length(const char **names)\n \treturn p - names;\n }\n \n-void parse_names(char *buf, int size, char ***namesp)\n+char **parse_names(char *buf, int size)\n {\n \tchar **names = NULL;\n \tsize_t names_cap = 0;\n \tsize_t names_len = 0;\n-\n \tchar *p = buf;\n \tchar *end = buf + size;\n+\n \twhile (p < end) {\n \t\tchar *next = strchr(p, '\\n');\n \t\tif (next && next < end) {\n@@ -147,14 +147,26 @@ void parse_names(char *buf, int size, char ***namesp)\n \t\t}\n \t\tif (p < next) {\n \t\t\tREFTABLE_ALLOC_GROW(names, names_len + 1, names_cap);\n-\t\t\tnames[names_len++] = xstrdup(p);\n+\t\t\tif (!names)\n+\t\t\t\tgoto err;\n+\n+\t\t\tnames[names_len] = strdup(p);\n+\t\t\tif (!names[names_len++])\n+\t\t\t\tgoto err;\n \t\t}\n \t\tp = next + 1;\n \t}\n \n \tREFTABLE_REALLOC_ARRAY(names, names_len + 1);\n \tnames[names_len] = NULL;\n-\t*namesp = names;\n+\n+\treturn names;\n+\n+err:\n+\tfor (size_t i = 0; i < names_len; i++)\n+\t\tfree(names[i]);\n+\tfree(names);\n+\treturn NULL;\n }\n \n int names_equal(const char **a, const char **b)\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex f107e148605..69adeab2e4b 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -38,9 +38,12 @@ size_t binsearch(size_t sz, int (*f)(size_t k, void *args), void *args);\n */\n void free_names(char **a);\n \n-/* parse a newline separated list of names. `size` is the length of the buffer,\n- * without terminating '\\0'. Empty names are discarded. */\n-void parse_names(char *buf, int size, char ***namesp);\n+/*\n+ * Parse a newline separated list of names. `size` is the length of the buffer,\n+ * without terminating '\\0'. Empty names are discarded. Returns a `NULL`\n+ * pointer when allocations fail.\n+ */\n+char **parse_names(char *buf, int size);\n \n /* compares two NULL-terminated arrays of strings. */\n int names_equal(const char **a, const char **b);\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex ce0a35216ba..498fae846d7 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -108,7 +108,11 @@ static int fd_read_lines(int fd, char ***namesp)\n \t}\n \tbuf[size] = 0;\n \n-\tparse_names(buf, size, namesp);\n+\t*namesp = parse_names(buf, size);\n+\tif (!*namesp) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \n done:\n \treftable_free(buf);\ndiff --git a/t/unit-tests/t-reftable-basics.c b/t/unit-tests/t-reftable-basics.c\nindex e5556ebf527..1fa77b6faff 100644\n--- a/t/unit-tests/t-reftable-basics.c\n+++ b/t/unit-tests/t-reftable-basics.c\n@@ -72,13 +72,14 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \tif_test (\"parse_names works for basic input\") {\n \t\tchar in1[] = \"line\\n\";\n \t\tchar in2[] = \"a\\nb\\nc\";\n-\t\tchar **out = NULL;\n-\t\tparse_names(in1, strlen(in1), &out);\n+\t\tchar **out = parse_names(in1, strlen(in1));\n+\t\tcheck(out != NULL);\n \t\tcheck_str(out[0], \"line\");\n \t\tcheck(!out[1]);\n \t\tfree_names(out);\n \n-\t\tparse_names(in2, strlen(in2), &out);\n+\t\tout = parse_names(in2, strlen(in2));\n+\t\tcheck(out != NULL);\n \t\tcheck_str(out[0], \"a\");\n \t\tcheck_str(out[1], \"b\");\n \t\tcheck_str(out[2], \"c\");\n@@ -88,8 +89,8 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \n \tif_test (\"parse_names drops empty string\") {\n \t\tchar in[] = \"a\\n\\nb\\n\";\n-\t\tchar **out = NULL;\n-\t\tparse_names(in, strlen(in), &out);\n+\t\tchar **out = parse_names(in, strlen(in));\n+\t\tcheck(out != NULL);\n \t\tcheck_str(out[0], \"a\");\n \t\t/* simply '\\n' should be dropped as empty string */\n \t\tcheck_str(out[1], \"b\");\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502875","messageId":"fa32be01e508c1f90096e25fe14708587b649013.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 06/22] reftable/record: handle allocation failures on copy","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:28:34Z","receivedAt":"2024-09-16T12:28:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when copying records. While at it, convert\nfrom `xstrdup()` to `reftable_strdup()`. Adapt callsites to check for\nerror codes.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/record.c | 84 +++++++++++++++++++++++++++++++++--------------\n reftable/record.h | 6 ++--\n 2 files changed, 63 insertions(+), 27 deletions(-)\n\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 6b5a075b921..60fd33c9c94 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -215,13 +215,14 @@ static void reftable_ref_record_key(const void *r, struct strbuf *dest)\n \tstrbuf_addstr(dest, rec->refname);\n }\n \n-static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size)\n+static int reftable_ref_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n {\n \tstruct reftable_ref_record *ref = rec;\n \tconst struct reftable_ref_record *src = src_rec;\n \tchar *refname = NULL;\n \tsize_t refname_cap = 0;\n+\tint err;\n \n \tassert(hash_size > 0);\n \n@@ -236,6 +237,11 @@ static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n \n \t\tREFTABLE_ALLOC_GROW(ref->refname, refname_len + 1,\n \t\t\t\t ref->refname_cap);\n+\t\tif (!ref->refname) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n+\n \t\tmemcpy(ref->refname, src->refname, refname_len);\n \t\tref->refname[refname_len] = 0;\n \t}\n@@ -254,9 +260,17 @@ static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n \t\t src->value.val2.target_value, hash_size);\n \t\tbreak;\n \tcase REFTABLE_REF_SYMREF:\n-\t\tref->value.symref = xstrdup(src->value.symref);\n+\t\tref->value.symref = reftable_strdup(src->value.symref);\n+\t\tif (!ref->value.symref) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n \t\tbreak;\n \t}\n+\n+\terr = 0;\n+out:\n+\treturn err;\n }\n \n static void reftable_ref_record_release_void(void *rec)\n@@ -457,23 +471,28 @@ static void reftable_obj_record_release(void *rec)\n \tmemset(obj, 0, sizeof(struct reftable_obj_record));\n }\n \n-static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size UNUSED)\n+static int reftable_obj_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size UNUSED)\n {\n \tstruct reftable_obj_record *obj = rec;\n-\tconst struct reftable_obj_record *src =\n-\t\t(const struct reftable_obj_record *)src_rec;\n+\tconst struct reftable_obj_record *src = src_rec;\n \n \treftable_obj_record_release(obj);\n \n \tREFTABLE_ALLOC_ARRAY(obj->hash_prefix, src->hash_prefix_len);\n+\tif (!obj->hash_prefix)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tobj->hash_prefix_len = src->hash_prefix_len;\n \tif (src->hash_prefix_len)\n \t\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n \n \tREFTABLE_ALLOC_ARRAY(obj->offsets, src->offset_len);\n+\tif (!obj->offsets)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tobj->offset_len = src->offset_len;\n \tCOPY_ARRAY(obj->offsets, src->offsets, src->offset_len);\n+\n+\treturn 0;\n }\n \n static uint8_t reftable_obj_record_val_type(const void *rec)\n@@ -646,33 +665,44 @@ static void reftable_log_record_key(const void *r, struct strbuf *dest)\n \tstrbuf_add(dest, i64, sizeof(i64));\n }\n \n-static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size)\n+static int reftable_log_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n {\n \tstruct reftable_log_record *dst = rec;\n \tconst struct reftable_log_record *src =\n \t\t(const struct reftable_log_record *)src_rec;\n+\tint ret;\n \n \treftable_log_record_release(dst);\n \t*dst = *src;\n+\n \tif (dst->refname) {\n-\t\tdst->refname = xstrdup(dst->refname);\n+\t\tdst->refname = reftable_strdup(dst->refname);\n+\t\tif (!dst->refname) {\n+\t\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n \t}\n+\n \tswitch (dst->value_type) {\n \tcase REFTABLE_LOG_DELETION:\n \t\tbreak;\n \tcase REFTABLE_LOG_UPDATE:\n-\t\tif (dst->value.update.email) {\n+\t\tif (dst->value.update.email)\n \t\t\tdst->value.update.email =\n-\t\t\t\txstrdup(dst->value.update.email);\n-\t\t}\n-\t\tif (dst->value.update.name) {\n+\t\t\t\treftable_strdup(dst->value.update.email);\n+\t\tif (dst->value.update.name)\n \t\t\tdst->value.update.name =\n-\t\t\t\txstrdup(dst->value.update.name);\n-\t\t}\n-\t\tif (dst->value.update.message) {\n+\t\t\t\treftable_strdup(dst->value.update.name);\n+\t\tif (dst->value.update.message)\n \t\t\tdst->value.update.message =\n-\t\t\t\txstrdup(dst->value.update.message);\n+\t\t\t\treftable_strdup(dst->value.update.message);\n+\n+\t\tif (!dst->value.update.email ||\n+\t\t !dst->value.update.name ||\n+\t\t !dst->value.update.message) {\n+\t\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n \t\t}\n \n \t\tmemcpy(dst->value.update.new_hash,\n@@ -681,6 +711,10 @@ static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n \t\t src->value.update.old_hash, hash_size);\n \t\tbreak;\n \t}\n+\n+\tret = 0;\n+out:\n+\treturn ret;\n }\n \n static void reftable_log_record_release_void(void *rec)\n@@ -954,8 +988,8 @@ static void reftable_index_record_key(const void *r, struct strbuf *dest)\n \tstrbuf_addbuf(dest, &rec->last_key);\n }\n \n-static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size UNUSED)\n+static int reftable_index_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size UNUSED)\n {\n \tstruct reftable_index_record *dst = rec;\n \tconst struct reftable_index_record *src = src_rec;\n@@ -963,6 +997,8 @@ static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n \tstrbuf_reset(&dst->last_key);\n \tstrbuf_addbuf(&dst->last_key, &src->last_key);\n \tdst->offset = src->offset;\n+\n+\treturn 0;\n }\n \n static void reftable_index_record_release(void *rec)\n@@ -1054,14 +1090,14 @@ int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n \t\t\t\t\t\t dest, hash_size);\n }\n \n-void reftable_record_copy_from(struct reftable_record *rec,\n+int reftable_record_copy_from(struct reftable_record *rec,\n \t\t\t struct reftable_record *src, int hash_size)\n {\n \tassert(src->type == rec->type);\n \n-\treftable_record_vtable(rec)->copy_from(reftable_record_data(rec),\n-\t\t\t\t\t reftable_record_data(src),\n-\t\t\t\t\t hash_size);\n+\treturn reftable_record_vtable(rec)->copy_from(reftable_record_data(rec),\n+\t\t\t\t\t\t reftable_record_data(src),\n+\t\t\t\t\t\t hash_size);\n }\n \n uint8_t reftable_record_val_type(struct reftable_record *rec)\ndiff --git a/reftable/record.h b/reftable/record.h\nindex 5003bacdb0c..0f53ba54434 100644\n--- a/reftable/record.h\n+++ b/reftable/record.h\n@@ -44,7 +44,7 @@ struct reftable_record_vtable {\n \t/* The record type of ('r' for ref). */\n \tuint8_t type;\n \n-\tvoid (*copy_from)(void *dest, const void *src, int hash_size);\n+\tint (*copy_from)(void *dest, const void *src, int hash_size);\n \n \t/* a value of [0..7], indicating record subvariants (eg. ref vs. symref\n \t * vs ref deletion) */\n@@ -137,8 +137,8 @@ void reftable_record_init(struct reftable_record *rec, uint8_t typ);\n int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b);\n int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);\n void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);\n-void reftable_record_copy_from(struct reftable_record *rec,\n-\t\t\t struct reftable_record *src, int hash_size);\n+int reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size);\n uint8_t reftable_record_val_type(struct reftable_record *rec);\n int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n \t\t\t int hash_size);\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502876","messageId":"372e36f880e02558f83713068194231c876c545b.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 07/22] reftable/record: handle allocation failures when decoding records","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:28:37Z","receivedAt":"2024-09-16T12:28:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when decoding records. While at it, fix some\nerror codes to be `REFTABLE_FORMAT_ERROR`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/record.c | 80 +++++++++++++++++++++++++++++++++++++----------\n 1 file changed, 63 insertions(+), 17 deletions(-)\n\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 60fd33c9c94..787e134c9a0 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -359,7 +359,7 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tuint64_t update_index = 0;\n \tconst char *refname = NULL;\n \tsize_t refname_cap = 0;\n-\tint n;\n+\tint n, err;\n \n \tassert(hash_size > 0);\n \n@@ -375,6 +375,10 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tSWAP(r->refname_cap, refname_cap);\n \n \tREFTABLE_ALLOC_GROW(r->refname, key.len + 1, r->refname_cap);\n+\tif (!r->refname) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \tmemcpy(r->refname, key.buf, key.len);\n \tr->refname[key.len] = 0;\n \n@@ -383,7 +387,8 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tswitch (val_type) {\n \tcase REFTABLE_REF_VAL1:\n \t\tif (in.len < hash_size) {\n-\t\t\treturn -1;\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n \t\t}\n \n \t\tmemcpy(r->value.val1, in.buf, hash_size);\n@@ -392,7 +397,8 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \n \tcase REFTABLE_REF_VAL2:\n \t\tif (in.len < 2 * hash_size) {\n-\t\t\treturn -1;\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n \t\t}\n \n \t\tmemcpy(r->value.val2.value, in.buf, hash_size);\n@@ -405,7 +411,8 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tcase REFTABLE_REF_SYMREF: {\n \t\tint n = decode_string(scratch, in);\n \t\tif (n < 0) {\n-\t\t\treturn -1;\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n \t\t}\n \t\tstring_view_consume(&in, n);\n \t\tr->value.symref = strbuf_detach(scratch, NULL);\n@@ -419,6 +426,9 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \t}\n \n \treturn start.len - in.len;\n+\n+done:\n+\treturn err;\n }\n \n static int reftable_ref_record_is_deletion_void(const void *p)\n@@ -552,6 +562,8 @@ static int reftable_obj_record_decode(void *rec, struct strbuf key,\n \treftable_obj_record_release(r);\n \n \tREFTABLE_ALLOC_ARRAY(r->hash_prefix, key.len);\n+\tif (!r->hash_prefix)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tmemcpy(r->hash_prefix, key.buf, key.len);\n \tr->hash_prefix_len = key.len;\n \n@@ -570,6 +582,8 @@ static int reftable_obj_record_decode(void *rec, struct strbuf key,\n \t\treturn start.len - in.len;\n \n \tREFTABLE_ALLOC_ARRAY(r->offsets, count);\n+\tif (!r->offsets)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tr->offset_len = count;\n \n \tn = get_var_int(&r->offsets[0], &in);\n@@ -801,12 +815,17 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \tstruct reftable_log_record *r = rec;\n \tuint64_t max = 0;\n \tuint64_t ts = 0;\n-\tint n;\n+\tint err, n;\n \n \tif (key.len <= 9 || key.buf[key.len - 9] != 0)\n \t\treturn REFTABLE_FORMAT_ERROR;\n \n \tREFTABLE_ALLOC_GROW(r->refname, key.len - 8, r->refname_cap);\n+\tif (!r->refname) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tmemcpy(r->refname, key.buf, key.len - 8);\n \tts = get_be64(key.buf + key.len - 8);\n \n@@ -829,8 +848,10 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \tif (val_type == REFTABLE_LOG_DELETION)\n \t\treturn 0;\n \n-\tif (in.len < 2 * hash_size)\n-\t\treturn REFTABLE_FORMAT_ERROR;\n+\tif (in.len < 2 * hash_size) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n \n \tmemcpy(r->value.update.old_hash, in.buf, hash_size);\n \tmemcpy(r->value.update.new_hash, in.buf + hash_size, hash_size);\n@@ -838,8 +859,10 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \tstring_view_consume(&in, 2 * hash_size);\n \n \tn = decode_string(scratch, in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \n \t/*\n@@ -850,52 +873,75 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \t */\n \tif (!r->value.update.name ||\n \t strcmp(r->value.update.name, scratch->buf)) {\n-\t\tr->value.update.name =\n-\t\t\treftable_realloc(r->value.update.name, scratch->len + 1);\n+\t\tchar *name = reftable_realloc(r->value.update.name, scratch->len + 1);\n+\t\tif (!name) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tr->value.update.name = name;\n \t\tmemcpy(r->value.update.name, scratch->buf, scratch->len);\n \t\tr->value.update.name[scratch->len] = 0;\n \t}\n \n \tn = decode_string(scratch, in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \n \t/* Same as above, but for the reflog email. */\n \tif (!r->value.update.email ||\n \t strcmp(r->value.update.email, scratch->buf)) {\n-\t\tr->value.update.email =\n-\t\t\treftable_realloc(r->value.update.email, scratch->len + 1);\n+\t\tchar *email = reftable_realloc(r->value.update.email, scratch->len + 1);\n+\t\tif (!email) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tr->value.update.email = email;\n \t\tmemcpy(r->value.update.email, scratch->buf, scratch->len);\n \t\tr->value.update.email[scratch->len] = 0;\n \t}\n \n \tts = 0;\n \tn = get_var_int(&ts, &in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \tr->value.update.time = ts;\n-\tif (in.len < 2)\n+\tif (in.len < 2) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \n \tr->value.update.tz_offset = get_be16(in.buf);\n \tstring_view_consume(&in, 2);\n \n \tn = decode_string(scratch, in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \n \tREFTABLE_ALLOC_GROW(r->value.update.message, scratch->len + 1,\n \t\t\t r->value.update.message_cap);\n+\tif (!r->value.update.message) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tmemcpy(r->value.update.message, scratch->buf, scratch->len);\n \tr->value.update.message[scratch->len] = 0;\n \n \treturn start.len - in.len;\n \n done:\n-\treturn REFTABLE_FORMAT_ERROR;\n+\treturn err;\n }\n \n static int null_streq(const char *a, const char *b)\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502877","messageId":"7404d648a9d4c28db0ceb94961accfb272b80b76.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 08/22] reftable/writer: handle allocation failures in `writer_index_hash()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:28:39Z","receivedAt":"2024-09-16T12:28:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation errors in `writer_index_hash()`. Adjust its only\ncaller in `reftable_writer_add_ref()` accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/writer.c | 61 +++++++++++++++++++++++++++++------------------\n 1 file changed, 38 insertions(+), 23 deletions(-)\n\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 9d5e6072bc6..ed61aaf59c5 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -186,18 +186,22 @@ static int obj_index_tree_node_compare(const void *a, const void *b)\n \t\t\t &((const struct obj_index_tree_node *)b)->hash);\n }\n \n-static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n+static int writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n {\n \tuint64_t off = w->next;\n-\n \tstruct obj_index_tree_node want = { .hash = *hash };\n+\tstruct obj_index_tree_node *key;\n+\tstruct tree_node *node;\n \n-\tstruct tree_node *node = tree_search(&want, &w->obj_index_tree,\n-\t\t\t\t\t &obj_index_tree_node_compare, 0);\n-\tstruct obj_index_tree_node *key = NULL;\n+\tnode = tree_search(&want, &w->obj_index_tree,\n+\t\t\t &obj_index_tree_node_compare, 0);\n \tif (!node) {\n \t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n-\t\tkey = reftable_malloc(sizeof(struct obj_index_tree_node));\n+\n+\t\tkey = reftable_malloc(sizeof(*key));\n+\t\tif (!key)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \t\t*key = empty;\n \n \t\tstrbuf_reset(&key->hash);\n@@ -208,12 +212,15 @@ static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n \t\tkey = node->key;\n \t}\n \n-\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off) {\n-\t\treturn;\n-\t}\n+\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off)\n+\t\treturn 0;\n \n \tREFTABLE_ALLOC_GROW(key->offsets, key->offset_len + 1, key->offset_cap);\n+\tif (!key->offsets)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tkey->offsets[key->offset_len++] = off;\n+\n+\treturn 0;\n }\n \n static int writer_add_record(struct reftable_writer *w,\n@@ -284,11 +291,11 @@ int reftable_writer_add_ref(struct reftable_writer *w,\n \t\t\t.ref = *ref\n \t\t},\n \t};\n-\tint err = 0;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint err;\n \n-\tif (!ref->refname)\n-\t\treturn REFTABLE_API_ERROR;\n-\tif (ref->update_index < w->min_update_index ||\n+\tif (!ref->refname ||\n+\t ref->update_index < w->min_update_index ||\n \t ref->update_index > w->max_update_index)\n \t\treturn REFTABLE_API_ERROR;\n \n@@ -296,24 +303,32 @@ int reftable_writer_add_ref(struct reftable_writer *w,\n \n \terr = writer_add_record(w, &rec);\n \tif (err < 0)\n-\t\treturn err;\n+\t\tgoto out;\n \n \tif (!w->opts.skip_index_objects && reftable_ref_record_val1(ref)) {\n-\t\tstruct strbuf h = STRBUF_INIT;\n-\t\tstrbuf_add(&h, (char *)reftable_ref_record_val1(ref),\n+\t\tstrbuf_add(&buf, (char *)reftable_ref_record_val1(ref),\n \t\t\t hash_size(w->opts.hash_id));\n-\t\twriter_index_hash(w, &h);\n-\t\tstrbuf_release(&h);\n+\n+\t\terr = writer_index_hash(w, &buf);\n+\t\tif (err < 0)\n+\t\t\tgoto out;\n \t}\n \n \tif (!w->opts.skip_index_objects && reftable_ref_record_val2(ref)) {\n-\t\tstruct strbuf h = STRBUF_INIT;\n-\t\tstrbuf_add(&h, reftable_ref_record_val2(ref),\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_add(&buf, reftable_ref_record_val2(ref),\n \t\t\t hash_size(w->opts.hash_id));\n-\t\twriter_index_hash(w, &h);\n-\t\tstrbuf_release(&h);\n+\n+\t\terr = writer_index_hash(w, &buf);\n+\t\tif (err < 0)\n+\t\t\tgoto out;\n \t}\n-\treturn 0;\n+\n+\terr = 0;\n+\n+out:\n+\tstrbuf_release(&buf);\n+\treturn err;\n }\n \n int reftable_writer_add_refs(struct reftable_writer *w,\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502878","messageId":"d6afb30221e8faf878126f61a149d002ce5ffc3c.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 09/22] reftable/writer: handle allocation failures in `reftable_new_writer()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:28:42Z","receivedAt":"2024-09-16T12:28:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_new_writer()`. Adapt the\nfunction to return an error code to return such failures. While at it,\nrename it to match our code style as we have to touch up every callsite\nanyway.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/reftable-writer.h | 12 +++++++-----\n reftable/stack.c | 14 ++++++++++----\n reftable/writer.c | 22 ++++++++++++++++------\n t/unit-tests/lib-reftable.c | 8 +++++---\n 4 files changed, 38 insertions(+), 18 deletions(-)\n\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 189b1f4144f..43623dc7c30 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -90,11 +90,13 @@ struct reftable_stats {\n \tint object_id_len;\n };\n \n-/* reftable_new_writer creates a new writer */\n-struct reftable_writer *\n-reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n-\t\t int (*flush_func)(void *),\n-\t\t void *writer_arg, const struct reftable_write_options *opts);\n+struct reftable_writer;\n+\n+/* Create a new writer. */\n+int reftable_writer_new(struct reftable_writer **out,\n+\t\t\tssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t\tint (*flush_func)(void *),\n+\t\t\tvoid *writer_arg, const struct reftable_write_options *opts);\n \n /* Set the range of update indices for the records we will add. When writing a\n table into a stack, the min should be at least\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 498fae846d7..ea21ca6e5f5 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -808,8 +808,11 @@ int reftable_addition_add(struct reftable_addition *add,\n \t}\n \ttab_fd = get_tempfile_fd(tab_file);\n \n-\twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, &tab_fd,\n-\t\t\t\t &add->stack->opts);\n+\terr = reftable_writer_new(&wr, reftable_fd_write, reftable_fd_flush,\n+\t\t\t\t &tab_fd, &add->stack->opts);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = write_table(wr, arg);\n \tif (err < 0)\n \t\tgoto done;\n@@ -898,8 +901,11 @@ static int stack_compact_locked(struct reftable_stack *st,\n \t\tgoto done;\n \t}\n \n-\twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush,\n-\t\t\t\t &tab_fd, &st->opts);\n+\terr = reftable_writer_new(&wr, reftable_fd_write, reftable_fd_flush,\n+\t\t\t\t &tab_fd, &st->opts);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = stack_write_compact(st, wr, first, last, config);\n \tif (err < 0)\n \t\tgoto done;\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex ed61aaf59c5..54ec822e1cd 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -117,13 +117,17 @@ static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n \tw->block_writer->restart_interval = w->opts.restart_interval;\n }\n \n-struct reftable_writer *\n-reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n-\t\t int (*flush_func)(void *),\n-\t\t void *writer_arg, const struct reftable_write_options *_opts)\n+int reftable_writer_new(struct reftable_writer **out,\n+\t\t\tssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t\tint (*flush_func)(void *),\n+\t\t\tvoid *writer_arg, const struct reftable_write_options *_opts)\n {\n-\tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n \tstruct reftable_write_options opts = {0};\n+\tstruct reftable_writer *wp;\n+\n+\twp = reftable_calloc(1, sizeof(*wp));\n+\tif (!wp)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \n \tif (_opts)\n \t\topts = *_opts;\n@@ -134,13 +138,19 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \tstrbuf_init(&wp->block_writer_data.last_key, 0);\n \tstrbuf_init(&wp->last_key, 0);\n \tREFTABLE_CALLOC_ARRAY(wp->block, opts.block_size);\n+\tif (!wp->block) {\n+\t\tfree(wp);\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t}\n \twp->write = writer_func;\n \twp->write_arg = writer_arg;\n \twp->opts = opts;\n \twp->flush = flush_func;\n \twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n \n-\treturn wp;\n+\t*out = wp;\n+\n+\treturn 0;\n }\n \n void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\ndiff --git a/t/unit-tests/lib-reftable.c b/t/unit-tests/lib-reftable.c\nindex ab1fa44a282..54c26c43e77 100644\n--- a/t/unit-tests/lib-reftable.c\n+++ b/t/unit-tests/lib-reftable.c\n@@ -22,9 +22,11 @@ static int strbuf_writer_flush(void *arg UNUSED)\n struct reftable_writer *t_reftable_strbuf_writer(struct strbuf *buf,\n \t\t\t\t\t\t struct reftable_write_options *opts)\n {\n-\treturn reftable_new_writer(&strbuf_writer_write,\n-\t\t\t\t &strbuf_writer_flush,\n-\t\t\t\t buf, opts);\n+\tstruct reftable_writer *writer;\n+\tint ret = reftable_writer_new(&writer, &strbuf_writer_write, &strbuf_writer_flush,\n+\t\t\t\t buf, opts);\n+\tcheck(!ret);\n+\treturn writer;\n }\n \n void t_reftable_write_to_buf(struct strbuf *buf,\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502879","messageId":"349fd785a81e38b212f5c62a00dcbfa63341499f.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 10/22] reftable/merged: handle allocation failures in `merged_table_init_iter()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:28:45Z","receivedAt":"2024-09-16T12:28:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `merged_table_init_iter()`. While at it,\nmerge `merged_iter_init()` into the function. It only has a single\ncaller and merging them makes it easier to handle allocation failures\nconsistently.\n\nThis change also requires us to adapt `reftable_stack_init_*_iterator()`\nto bubble up the new error codes of `merged_table_iter_init()`. Adapt\ncallsites accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 39 +++++++++++++----\n reftable/merged.c | 74 ++++++++++++++++++++------------\n reftable/merged.h | 6 +--\n reftable/reftable-merged.h | 8 ++--\n reftable/reftable-stack.h | 8 ++--\n reftable/stack.c | 34 ++++++++++-----\n t/helper/test-reftable.c | 10 ++++-\n t/unit-tests/t-reftable-merged.c | 12 ++++--\n t/unit-tests/t-reftable-stack.c | 4 +-\n 9 files changed, 131 insertions(+), 64 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 5c241097a4e..9c08c46c85e 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1299,7 +1299,9 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t\t\tstruct reftable_log_record log = {0};\n \t\t\tstruct reftable_iterator it = {0};\n \n-\t\t\treftable_stack_init_log_iterator(arg->stack, &it);\n+\t\t\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto done;\n \n \t\t\t/*\n \t\t\t * When deleting refs we also delete all reflog entries\n@@ -1669,7 +1671,10 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \t * copy over all log entries from the old reflog. Last but not least,\n \t * when renaming we also have to delete all the old reflog entries.\n \t */\n-\treftable_stack_init_log_iterator(arg->stack, &it);\n+\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, arg->oldname);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1890,7 +1895,10 @@ static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftabl\n \tif (ret < 0)\n \t\tgoto done;\n \n-\treftable_stack_init_log_iterator(stack, &iter->iter);\n+\tret = reftable_stack_init_log_iterator(stack, &iter->iter);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&iter->iter, \"\");\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1957,7 +1965,10 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, refname);\n \twhile (!ret) {\n \t\tret = reftable_iterator_next_log(&it, &log);\n@@ -1973,6 +1984,7 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n \t\t\tbreak;\n \t}\n \n+done:\n \treftable_log_record_release(&log);\n \treftable_iterator_destroy(&it);\n \treturn ret;\n@@ -1994,7 +2006,10 @@ static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store,\n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, refname);\n \twhile (!ret) {\n \t\tstruct reftable_log_record log = {0};\n@@ -2044,7 +2059,10 @@ static int reftable_be_reflog_exists(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, refname);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -2150,7 +2168,9 @@ static int write_reflog_delete_table(struct reftable_writer *writer, void *cb_da\n \n \treftable_writer_set_limits(writer, ts, ts);\n \n-\treftable_stack_init_log_iterator(arg->stack, &it);\n+\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\tif (ret < 0)\n+\t\tgoto out;\n \n \t/*\n \t * In order to delete a table we need to delete all reflog entries one\n@@ -2174,6 +2194,7 @@ static int write_reflog_delete_table(struct reftable_writer *writer, void *cb_da\n \t\tret = reftable_writer_add_log(writer, &tombstone);\n \t}\n \n+out:\n \treftable_log_record_release(&log);\n \treftable_iterator_destroy(&it);\n \treturn ret;\n@@ -2312,7 +2333,9 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n \n \tret = reftable_iterator_seek_log(&it, refname);\n \tif (ret < 0)\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 128a810c55d..5a8ea8ae779 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -30,22 +30,6 @@ struct merged_iter {\n \tssize_t advance_index;\n };\n \n-static void merged_iter_init(struct merged_iter *mi,\n-\t\t\t struct reftable_merged_table *mt,\n-\t\t\t uint8_t typ)\n-{\n-\tmemset(mi, 0, sizeof(*mi));\n-\tmi->advance_index = -1;\n-\tmi->suppress_deletions = mt->suppress_deletions;\n-\n-\tREFTABLE_CALLOC_ARRAY(mi->subiters, mt->readers_len);\n-\tfor (size_t i = 0; i < mt->readers_len; i++) {\n-\t\treftable_record_init(&mi->subiters[i].rec, typ);\n-\t\treader_init_iter(mt->readers[i], &mi->subiters[i].iter, typ);\n-\t}\n-\tmi->subiters_len = mt->readers_len;\n-}\n-\n static void merged_iter_close(void *p)\n {\n \tstruct merged_iter *mi = p;\n@@ -244,25 +228,61 @@ reftable_merged_table_min_update_index(struct reftable_merged_table *mt)\n \treturn mt->min;\n }\n \n-void merged_table_init_iter(struct reftable_merged_table *mt,\n-\t\t\t struct reftable_iterator *it,\n-\t\t\t uint8_t typ)\n+int merged_table_init_iter(struct reftable_merged_table *mt,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t uint8_t typ)\n {\n-\tstruct merged_iter *mi = reftable_malloc(sizeof(*mi));\n-\tmerged_iter_init(mi, mt, typ);\n+\tstruct merged_subiter *subiters;\n+\tstruct merged_iter *mi = NULL;\n+\tint ret;\n+\n+\tREFTABLE_CALLOC_ARRAY(subiters, mt->readers_len);\n+\tif (!subiters) {\n+\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n+\tfor (size_t i = 0; i < mt->readers_len; i++) {\n+\t\treftable_record_init(&subiters[i].rec, typ);\n+\t\treader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n+\t}\n+\n+\tREFTABLE_CALLOC_ARRAY(mi, 1);\n+\tif (!mi) {\n+\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\tmi->advance_index = -1;\n+\tmi->suppress_deletions = mt->suppress_deletions;\n+\tmi->subiters = subiters;\n+\tmi->subiters_len = mt->readers_len;\n+\n \titerator_from_merged_iter(it, mi);\n+\tret = 0;\n+\n+out:\n+\tif (ret < 0) {\n+\t\tfor (size_t i = 0; subiters && i < mt->readers_len; i++) {\n+\t\t\treftable_iterator_destroy(&subiters[i].iter);\n+\t\t\treftable_record_release(&subiters[i].rec);\n+\t\t}\n+\t\tfree(subiters);\n+\t\tfree(mi);\n+\t}\n+\n+\treturn ret;\n }\n \n-void reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it)\n+int reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(mt, it, BLOCK_TYPE_REF);\n+\treturn merged_table_init_iter(mt, it, BLOCK_TYPE_REF);\n }\n \n-void reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it)\n+int reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(mt, it, BLOCK_TYPE_LOG);\n+\treturn merged_table_init_iter(mt, it, BLOCK_TYPE_LOG);\n }\n \n uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\ndiff --git a/reftable/merged.h b/reftable/merged.h\nindex de5fd33f010..89bd0c4b35b 100644\n--- a/reftable/merged.h\n+++ b/reftable/merged.h\n@@ -26,8 +26,8 @@ struct reftable_merged_table {\n \n struct reftable_iterator;\n \n-void merged_table_init_iter(struct reftable_merged_table *mt,\n-\t\t\t struct reftable_iterator *it,\n-\t\t\t uint8_t typ);\n+int merged_table_init_iter(struct reftable_merged_table *mt,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t uint8_t typ);\n \n #endif\ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nindex 16d19f8df20..a970d5dd89a 100644\n--- a/reftable/reftable-merged.h\n+++ b/reftable/reftable-merged.h\n@@ -37,12 +37,12 @@ int reftable_merged_table_new(struct reftable_merged_table **dest,\n \t\t\t uint32_t hash_id);\n \n /* Initialize a merged table iterator for reading refs. */\n-void reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it);\n+int reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it);\n \n /* Initialize a merged table iterator for reading logs. */\n-void reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it);\n+int reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it);\n \n /* returns the max update_index covered by this merged table. */\n uint64_t\ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nindex f4f8cabc7fb..e958f911b4e 100644\n--- a/reftable/reftable-stack.h\n+++ b/reftable/reftable-stack.h\n@@ -73,16 +73,16 @@ struct reftable_iterator;\n * be used to iterate through refs. The iterator is valid until the next reload\n * or write.\n */\n-void reftable_stack_init_ref_iterator(struct reftable_stack *st,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_stack_init_ref_iterator(struct reftable_stack *st,\n+\t\t\t\t struct reftable_iterator *it);\n \n /*\n * Initialize an iterator for the merged tables contained in the stack that can\n * be used to iterate through logs. The iterator is valid until the next reload\n * or write.\n */\n-void reftable_stack_init_log_iterator(struct reftable_stack *st,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_stack_init_log_iterator(struct reftable_stack *st,\n+\t\t\t\t struct reftable_iterator *it);\n \n /* returns the merged_table for seeking. This table is valid until the\n * next write or reload, and should not be closed or deleted.\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex ea21ca6e5f5..bb4d2309189 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -136,18 +136,18 @@ int read_lines(const char *filename, char ***namesp)\n \treturn err;\n }\n \n-void reftable_stack_init_ref_iterator(struct reftable_stack *st,\n+int reftable_stack_init_ref_iterator(struct reftable_stack *st,\n \t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(reftable_stack_merged_table(st),\n-\t\t\t it, BLOCK_TYPE_REF);\n+\treturn merged_table_init_iter(reftable_stack_merged_table(st),\n+\t\t\t\t it, BLOCK_TYPE_REF);\n }\n \n-void reftable_stack_init_log_iterator(struct reftable_stack *st,\n-\t\t\t\t struct reftable_iterator *it)\n+int reftable_stack_init_log_iterator(struct reftable_stack *st,\n+\t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(reftable_stack_merged_table(st),\n-\t\t\t it, BLOCK_TYPE_LOG);\n+\treturn merged_table_init_iter(reftable_stack_merged_table(st),\n+\t\t\t\t it, BLOCK_TYPE_LOG);\n }\n \n struct reftable_merged_table *\n@@ -952,7 +952,10 @@ static int stack_write_compact(struct reftable_stack *st,\n \tif (err < 0)\n \t\tgoto done;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tif (err < 0)\n \t\tgoto done;\n@@ -977,7 +980,10 @@ static int stack_write_compact(struct reftable_stack *st,\n \t}\n \treftable_iterator_destroy(&it);\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tif (err < 0)\n \t\tgoto done;\n@@ -1496,7 +1502,10 @@ int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n \tstruct reftable_iterator it = { 0 };\n \tint ret;\n \n-\treftable_merged_table_init_ref_iterator(st->merged, &it);\n+\tret = reftable_merged_table_init_ref_iterator(st->merged, &it);\n+\tif (ret)\n+\t\tgoto out;\n+\n \tret = reftable_iterator_seek_ref(&it, refname);\n \tif (ret)\n \t\tgoto out;\n@@ -1523,7 +1532,10 @@ int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n \tstruct reftable_iterator it = {0};\n \tint err;\n \n-\treftable_stack_init_log_iterator(st, &it);\n+\terr = reftable_stack_init_log_iterator(st, &it);\n+\tif (err)\n+\t\tgoto done;\n+\n \terr = reftable_iterator_seek_log(&it, refname);\n \tif (err)\n \t\tgoto done;\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 29d4e9a755d..5c8849d115b 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -28,7 +28,10 @@ static int dump_table(struct reftable_merged_table *mt)\n \tconst struct git_hash_algo *algop;\n \tint err;\n \n-\treftable_merged_table_init_ref_iterator(mt, &it);\n+\terr = reftable_merged_table_init_ref_iterator(mt, &it);\n+\tif (err < 0)\n+\t\treturn err;\n+\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tif (err < 0)\n \t\treturn err;\n@@ -63,7 +66,10 @@ static int dump_table(struct reftable_merged_table *mt)\n \treftable_iterator_destroy(&it);\n \treftable_ref_record_release(&ref);\n \n-\treftable_merged_table_init_log_iterator(mt, &it);\n+\terr = reftable_merged_table_init_log_iterator(mt, &it);\n+\tif (err < 0)\n+\t\treturn err;\n+\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tif (err < 0)\n \t\treturn err;\ndiff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c\nindex 19e54bdfb8b..3d2848632db 100644\n--- a/t/unit-tests/t-reftable-merged.c\n+++ b/t/unit-tests/t-reftable-merged.c\n@@ -82,7 +82,8 @@ static void t_merged_single_record(void)\n \tstruct reftable_iterator it = { 0 };\n \tint err;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"a\");\n \tcheck(!err);\n \n@@ -161,7 +162,8 @@ static void t_merged_refs(void)\n \tsize_t cap = 0;\n \tsize_t i;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"a\");\n \tcheck(!err);\n \tcheck_int(reftable_merged_table_hash_id(mt), ==, GIT_SHA1_FORMAT_ID);\n@@ -367,7 +369,8 @@ static void t_merged_logs(void)\n \tsize_t cap = 0;\n \tsize_t i;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"a\");\n \tcheck(!err);\n \tcheck_int(reftable_merged_table_hash_id(mt), ==, GIT_SHA1_FORMAT_ID);\n@@ -390,7 +393,8 @@ static void t_merged_logs(void)\n \t\tcheck(reftable_log_record_equal(want[i], &out[i],\n \t\t\t\t\t\t GIT_SHA1_RAWSZ));\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log_at(&it, \"a\", 2);\n \tcheck(!err);\n \treftable_log_record_release(&out[0]);\ndiff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c\nindex 65e513d5ec8..6e41a4cf7e5 100644\n--- a/t/unit-tests/t-reftable-stack.c\n+++ b/t/unit-tests/t-reftable-stack.c\n@@ -599,7 +599,9 @@ static void t_reftable_stack_iterator(void)\n \n \treftable_iterator_destroy(&it);\n \n-\treftable_stack_init_log_iterator(st, &it);\n+\terr = reftable_stack_init_log_iterator(st, &it);\n+\tcheck(!err);\n+\n \treftable_iterator_seek_log(&it, logs[0].refname);\n \tfor (i = 0; ; i++) {\n \t\tstruct reftable_log_record log = { 0 };\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502880","messageId":"c4985e64ce2ea13e1a367d8e058f40e97ec5689a.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 11/22] reftable/reader: handle allocation failures for unindexed reader","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:28:47Z","receivedAt":"2024-09-16T12:28:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when creating unindexed readers.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/reader.c | 23 +++++++++++++++++------\n 1 file changed, 17 insertions(+), 6 deletions(-)\n\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 6494ce2e327..485ee085dac 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -734,19 +734,30 @@ static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n \tREFTABLE_ALLOC_ARRAY(ti, 1);\n \ttable_iter_init(ti, r);\n \terr = table_iter_seek_start(ti, BLOCK_TYPE_REF, 0);\n-\tif (err < 0) {\n-\t\treftable_free(ti);\n-\t\treturn err;\n-\t}\n+\tif (err < 0)\n+\t\tgoto out;\n \n-\tfilter = reftable_malloc(sizeof(struct filtering_ref_iterator));\n+\tfilter = reftable_malloc(sizeof(*filter));\n+\tif (!filter) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n \t*filter = empty;\n \n \tstrbuf_add(&filter->oid, oid, oid_len);\n \titerator_from_table_iter(&filter->it, ti);\n \n \titerator_from_filtering_ref_iterator(it, filter);\n-\treturn 0;\n+\n+\terr = 0;\n+\n+out:\n+\tif (err < 0) {\n+\t\tif (ti)\n+\t\t\ttable_iter_close(ti);\n+\t\treftable_free(ti);\n+\t}\n+\treturn err;\n }\n \n int reftable_reader_refs_for(struct reftable_reader *r,\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502881","messageId":"94c85ffd5ec6a508f8c8c6d2ca30760a8a3614de.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 12/22] reftable/reader: handle allocation failures in `reader_init_iter()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:28:52Z","receivedAt":"2024-09-16T12:28:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reader_init_iter()`. This requires us to\nalso adapt `reftable_reader_init_*_iterator()` to bubble up the new\nerror codes. Adapt callers accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 4 +++-\n reftable/reader.c | 28 +++++++++++++++---------\n reftable/reader.h | 6 ++---\n reftable/reftable-reader.h | 8 +++----\n t/unit-tests/t-reftable-readwrite.c | 34 +++++++++++++++++++----------\n 5 files changed, 50 insertions(+), 30 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 5a8ea8ae779..741f62ea638 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -244,7 +244,9 @@ int merged_table_init_iter(struct reftable_merged_table *mt,\n \n \tfor (size_t i = 0; i < mt->readers_len; i++) {\n \t\treftable_record_init(&subiters[i].rec, typ);\n-\t\treader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n+\t\tret = reader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n+\t\tif (ret < 0)\n+\t\t\tgoto out;\n \t}\n \n \tREFTABLE_CALLOC_ARRAY(mi, 1);\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 485ee085dac..f696e992dfc 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -554,32 +554,37 @@ static void iterator_from_table_iter(struct reftable_iterator *it,\n \tit->ops = &table_iter_vtable;\n }\n \n-void reader_init_iter(struct reftable_reader *r,\n-\t\t struct reftable_iterator *it,\n-\t\t uint8_t typ)\n+int reader_init_iter(struct reftable_reader *r,\n+\t\t struct reftable_iterator *it,\n+\t\t uint8_t typ)\n {\n \tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n \n \tif (offs->is_present) {\n \t\tstruct table_iter *ti;\n \t\tREFTABLE_ALLOC_ARRAY(ti, 1);\n+\t\tif (!ti)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \t\ttable_iter_init(ti, r);\n \t\titerator_from_table_iter(it, ti);\n \t} else {\n \t\titerator_set_empty(it);\n \t}\n+\n+\treturn 0;\n }\n \n-void reftable_reader_init_ref_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it)\n+int reftable_reader_init_ref_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it)\n {\n-\treader_init_iter(r, it, BLOCK_TYPE_REF);\n+\treturn reader_init_iter(r, it, BLOCK_TYPE_REF);\n }\n \n-void reftable_reader_init_log_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it)\n+int reftable_reader_init_log_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it)\n {\n-\treader_init_iter(r, it, BLOCK_TYPE_LOG);\n+\treturn reader_init_iter(r, it, BLOCK_TYPE_LOG);\n }\n \n int reftable_reader_new(struct reftable_reader **out,\n@@ -689,7 +694,10 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n \tstruct indexed_table_ref_iter *itr = NULL;\n \n \t/* Look through the reverse index. */\n-\treader_init_iter(r, &oit, BLOCK_TYPE_OBJ);\n+\terr = reader_init_iter(r, &oit, BLOCK_TYPE_OBJ);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = iterator_seek(&oit, &want);\n \tif (err != 0)\n \t\tgoto done;\ndiff --git a/reftable/reader.h b/reftable/reader.h\nindex 3710ee09b4c..02d10c5d37e 100644\n--- a/reftable/reader.h\n+++ b/reftable/reader.h\n@@ -56,9 +56,9 @@ struct reftable_reader {\n \n const char *reader_name(struct reftable_reader *r);\n \n-void reader_init_iter(struct reftable_reader *r,\n-\t\t struct reftable_iterator *it,\n-\t\t uint8_t typ);\n+int reader_init_iter(struct reftable_reader *r,\n+\t\t struct reftable_iterator *it,\n+\t\t uint8_t typ);\n \n /* initialize a block reader to read from `r` */\n int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nindex a600452b565..6a2d0b693f5 100644\n--- a/reftable/reftable-reader.h\n+++ b/reftable/reftable-reader.h\n@@ -46,12 +46,12 @@ void reftable_reader_incref(struct reftable_reader *reader);\n void reftable_reader_decref(struct reftable_reader *reader);\n \n /* Initialize a reftable iterator for reading refs. */\n-void reftable_reader_init_ref_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_reader_init_ref_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it);\n \n /* Initialize a reftable iterator for reading logs. */\n-void reftable_reader_init_log_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_reader_init_log_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it);\n \n /* returns the hash ID used in this table. */\n uint32_t reftable_reader_hash_id(struct reftable_reader *r);\ndiff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c\nindex e1b235a5f13..acca927a2cf 100644\n--- a/t/unit-tests/t-reftable-readwrite.c\n+++ b/t/unit-tests/t-reftable-readwrite.c\n@@ -206,7 +206,8 @@ static void t_log_write_read(void)\n \terr = reftable_reader_new(&reader, &source, \"file.log\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \n \terr = reftable_iterator_seek_ref(&it, names[N - 1]);\n \tcheck(!err);\n@@ -221,8 +222,8 @@ static void t_log_write_read(void)\n \treftable_iterator_destroy(&it);\n \treftable_ref_record_release(&ref);\n \n-\treftable_reader_init_log_iterator(reader, &it);\n-\n+\terr = reftable_reader_init_log_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tcheck(!err);\n \n@@ -296,7 +297,8 @@ static void t_log_zlib_corruption(void)\n \terr = reftable_reader_new(&reader, &source, \"file.log\");\n \tcheck(!err);\n \n-\treftable_reader_init_log_iterator(reader, &it);\n+\terr = reftable_reader_init_log_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"refname\");\n \tcheck_int(err, ==, REFTABLE_ZLIB_ERROR);\n \n@@ -325,7 +327,8 @@ static void t_table_read_write_sequential(void)\n \terr = reftable_reader_new(&reader, &source, \"file.ref\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tcheck(!err);\n \n@@ -376,7 +379,8 @@ static void t_table_read_api(void)\n \terr = reftable_reader_new(&reader, &source, \"file.ref\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, names[0]);\n \tcheck(!err);\n \n@@ -419,7 +423,8 @@ static void t_table_read_write_seek(int index, int hash_id)\n \t}\n \n \tfor (i = 1; i < N; i++) {\n-\t\treftable_reader_init_ref_iterator(reader, &it);\n+\t\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\t\tcheck(!err);\n \t\terr = reftable_iterator_seek_ref(&it, names[i]);\n \t\tcheck(!err);\n \t\terr = reftable_iterator_next_ref(&it, &ref);\n@@ -435,7 +440,8 @@ static void t_table_read_write_seek(int index, int hash_id)\n \tstrbuf_addstr(&pastLast, names[N - 1]);\n \tstrbuf_addstr(&pastLast, \"/\");\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, pastLast.buf);\n \tif (err == 0) {\n \t\tstruct reftable_ref_record ref = { 0 };\n@@ -534,7 +540,8 @@ static void t_table_refs_for(int indexed)\n \tif (!indexed)\n \t\treader->obj_offsets.is_present = 0;\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tcheck(!err);\n \treftable_iterator_destroy(&it);\n@@ -593,7 +600,8 @@ static void t_write_empty_table(void)\n \terr = reftable_reader_new(&rd, &source, \"filename\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(rd, &it);\n+\terr = reftable_reader_init_ref_iterator(rd, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tcheck(!err);\n \n@@ -802,7 +810,8 @@ static void t_write_multiple_indices(void)\n \t * Seeking the log uses the log index now. In case there is any\n \t * confusion regarding indices we would notice here.\n \t */\n-\treftable_reader_init_log_iterator(reader, &it);\n+\terr = reftable_reader_init_log_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tcheck(!err);\n \n@@ -858,7 +867,8 @@ static void t_write_multi_level_index(void)\n \t/*\n \t * Seeking the last ref should work as expected.\n \t */\n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"refs/heads/199\");\n \tcheck(!err);\n \n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502882","messageId":"6ef25b6c65513e837f9ef8faf1dfa8f4761ff546.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 13/22] reftable/stack: handle allocation failures on reload","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:28:55Z","receivedAt":"2024-09-16T12:28:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_stack_reload_once()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 34 +++++++++++++++++++++++++---------\n 1 file changed, 25 insertions(+), 9 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex bb4d2309189..060b2c1b90d 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -214,13 +214,13 @@ void reftable_stack_destroy(struct reftable_stack *st)\n }\n \n static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n-\t\t\t\t\t\t int cur_len)\n+\t\t\t\t\t\t size_t cur_len)\n {\n \tstruct reftable_reader **cur = reftable_calloc(cur_len, sizeof(*cur));\n-\tint i = 0;\n-\tfor (i = 0; i < cur_len; i++) {\n+\tif (!cur)\n+\t\treturn NULL;\n+\tfor (size_t i = 0; i < cur_len; i++)\n \t\tcur[i] = st->readers[i];\n-\t}\n \treturn cur;\n }\n \n@@ -229,18 +229,30 @@ static int reftable_stack_reload_once(struct reftable_stack *st,\n \t\t\t\t int reuse_open)\n {\n \tsize_t cur_len = !st->merged ? 0 : st->merged->readers_len;\n-\tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n+\tstruct reftable_reader **cur;\n \tstruct reftable_reader **reused = NULL;\n-\tsize_t reused_len = 0, reused_alloc = 0;\n-\tsize_t names_len = names_length(names);\n-\tstruct reftable_reader **new_readers =\n-\t\treftable_calloc(names_len, sizeof(*new_readers));\n+\tstruct reftable_reader **new_readers;\n+\tsize_t reused_len = 0, reused_alloc = 0, names_len;\n \tsize_t new_readers_len = 0;\n \tstruct reftable_merged_table *new_merged = NULL;\n \tstruct strbuf table_path = STRBUF_INIT;\n \tint err = 0;\n \tsize_t i;\n \n+\tcur = stack_copy_readers(st, cur_len);\n+\tif (!cur) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tnames_len = names_length(names);\n+\n+\tnew_readers = reftable_calloc(names_len, sizeof(*new_readers));\n+\tif (!new_readers) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \twhile (*names) {\n \t\tstruct reftable_reader *rd = NULL;\n \t\tconst char *name = *names++;\n@@ -261,6 +273,10 @@ static int reftable_stack_reload_once(struct reftable_stack *st,\n \t\t\t\t * do by bumping their refcount.\n \t\t\t\t */\n \t\t\t\tREFTABLE_ALLOC_GROW(reused, reused_len + 1, reused_alloc);\n+\t\t\t\tif (!reused) {\n+\t\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n \t\t\t\treused[reused_len++] = rd;\n \t\t\t\treftable_reader_incref(rd);\n \t\t\t\tbreak;\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502883","messageId":"f5583cc2300eeebd14dfeb0b5106a6e80418c681.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 14/22] reftable/stack: handle allocation failures in `reftable_new_stack()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:28:58Z","receivedAt":"2024-09-16T12:29:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_new_stack()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 34 ++++++++++++++++++++++++++--------\n 1 file changed, 26 insertions(+), 8 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 060b2c1b90d..1b77c9d0146 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -56,10 +56,16 @@ static int reftable_fd_flush(void *arg)\n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \t\t const struct reftable_write_options *_opts)\n {\n-\tstruct reftable_stack *p = reftable_calloc(1, sizeof(*p));\n \tstruct strbuf list_file_name = STRBUF_INIT;\n-\tstruct reftable_write_options opts = {0};\n-\tint err = 0;\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct reftable_stack *p;\n+\tint err;\n+\n+\tp = reftable_calloc(1, sizeof(*p));\n+\tif (!p) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n \n \tif (_opts)\n \t\topts = *_opts;\n@@ -74,15 +80,23 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \n \tp->list_file = strbuf_detach(&list_file_name, NULL);\n \tp->list_fd = -1;\n-\tp->reftable_dir = xstrdup(dir);\n \tp->opts = opts;\n+\tp->reftable_dir = reftable_strdup(dir);\n+\tif (!p->reftable_dir) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n \n \terr = reftable_stack_reload_maybe_reuse(p, 1);\n-\tif (err < 0) {\n+\tif (err < 0)\n+\t\tgoto out;\n+\n+\t*dest = p;\n+\terr = 0;\n+\n+out:\n+\tif (err < 0)\n \t\treftable_stack_destroy(p);\n-\t} else {\n-\t\t*dest = p;\n-\t}\n \treturn err;\n }\n \n@@ -171,6 +185,10 @@ void reftable_stack_destroy(struct reftable_stack *st)\n {\n \tchar **names = NULL;\n \tint err = 0;\n+\n+\tif (!st)\n+\t\treturn;\n+\n \tif (st->merged) {\n \t\treftable_merged_table_free(st->merged);\n \t\tst->merged = NULL;\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502884","messageId":"3393a59ce4260cf2c9d948f6c37e101954e5f97e.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 15/22] reftable/stack: handle allocation failures in `stack_compact_range()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:29:00Z","receivedAt":"2024-09-16T12:29:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `stack_compact_range()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 19 +++++++++++++++++--\n 1 file changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 1b77c9d0146..2e6dd513d72 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1131,6 +1131,11 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * from the point of view of the newer process.\n \t */\n \tREFTABLE_CALLOC_ARRAY(table_locks, last - first + 1);\n+\tif (!table_locks) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tfor (i = last + 1; i > first; i--) {\n \t\tstack_filename(&table_name, st, reader_name(st->readers[i - 1]));\n \n@@ -1312,8 +1317,18 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\t * thus have to allocate `readers_len + 1` many entries.\n \t\t */\n \t\tREFTABLE_CALLOC_ARRAY(names, st->merged->readers_len + 1);\n-\t\tfor (size_t i = 0; i < st->merged->readers_len; i++)\n-\t\t\tnames[i] = xstrdup(st->readers[i]->name);\n+\t\tif (!names) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tfor (size_t i = 0; i < st->merged->readers_len; i++) {\n+\t\t\tnames[i] = reftable_strdup(st->readers[i]->name);\n+\t\t\tif (!names[i]) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n \t\tfirst_to_replace = first;\n \t\tlast_to_replace = last;\n \t}\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502885","messageId":"2c19f258fe7ed789f1900ce208cb48c1a3b3b571.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 16/22] reftable/stack: handle allocation failures in auto compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:29:03Z","receivedAt":"2024-09-16T12:29:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_stack_auto_compact()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 16 ++++++++++++----\n 1 file changed, 12 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 2e6dd513d72..990784d9d2f 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1518,6 +1518,8 @@ static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n \tuint64_t *sizes;\n \n \tREFTABLE_CALLOC_ARRAY(sizes, st->merged->readers_len);\n+\tif (!sizes)\n+\t\treturn NULL;\n \n \tfor (size_t i = 0; i < st->merged->readers_len; i++)\n \t\tsizes[i] = st->readers[i]->size - overhead;\n@@ -1527,11 +1529,17 @@ static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n \n int reftable_stack_auto_compact(struct reftable_stack *st)\n {\n-\tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n-\tstruct segment seg =\n-\t\tsuggest_compaction_segment(sizes, st->merged->readers_len,\n-\t\t\t\t\t st->opts.auto_compaction_factor);\n+\tstruct segment seg;\n+\tuint64_t *sizes;\n+\n+\tsizes = stack_table_sizes_for_compaction(st);\n+\tif (!sizes)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n+\tseg = suggest_compaction_segment(sizes, st->merged->readers_len,\n+\t\t\t\t\t st->opts.auto_compaction_factor);\n \treftable_free(sizes);\n+\n \tif (segment_size(&seg) > 0)\n \t\treturn stack_compact_range(st, seg.start, seg.end - 1,\n \t\t\t\t\t NULL, STACK_COMPACT_RANGE_BEST_EFFORT);\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502886","messageId":"32fead57de989335b17d16f63c1cd144460495a1.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 17/22] reftable/iter: handle allocation failures when creating indexed table iter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:29:06Z","receivedAt":"2024-09-16T12:29:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `new_indexed_table_ref_iter()`. While at\nit, rename the function to match our coding style.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/iter.c | 22 ++++++++++++++++------\n reftable/iter.h | 2 +-\n reftable/reader.c | 7 ++++++-\n 3 files changed, 23 insertions(+), 8 deletions(-)\n\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex 416a9f6996b..41bdfbb13f9 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -181,14 +181,20 @@ static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\n \t}\n }\n \n-int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+int indexed_table_ref_iter_new(struct indexed_table_ref_iter **dest,\n \t\t\t struct reftable_reader *r, uint8_t *oid,\n \t\t\t int oid_len, uint64_t *offsets, int offset_len)\n {\n \tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n-\tstruct indexed_table_ref_iter *itr = reftable_calloc(1, sizeof(*itr));\n+\tstruct indexed_table_ref_iter *itr;\n \tint err = 0;\n \n+\titr = reftable_calloc(1, sizeof(*itr));\n+\tif (!itr) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n \t*itr = empty;\n \titr->r = r;\n \tstrbuf_add(&itr->oid, oid, oid_len);\n@@ -197,11 +203,15 @@ int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n \titr->offset_len = offset_len;\n \n \terr = indexed_table_ref_iter_next_block(itr);\n-\tif (err < 0) {\n+\tif (err < 0)\n+\t\tgoto out;\n+\n+\t*dest = itr;\n+\terr = 0;\n+\n+out:\n+\tif (err < 0)\n \t\treftable_free(itr);\n-\t} else {\n-\t\t*dest = itr;\n-\t}\n \treturn err;\n }\n \ndiff --git a/reftable/iter.h b/reftable/iter.h\nindex befc4597df1..b3225bc7add 100644\n--- a/reftable/iter.h\n+++ b/reftable/iter.h\n@@ -82,7 +82,7 @@ void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n \t\t\t\t\t struct indexed_table_ref_iter *itr);\n \n /* Takes ownership of `offsets` */\n-int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+int indexed_table_ref_iter_new(struct indexed_table_ref_iter **dest,\n \t\t\t struct reftable_reader *r, uint8_t *oid,\n \t\t\t int oid_len, uint64_t *offsets, int offset_len);\n \ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex f696e992dfc..0179e4e73dd 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -715,7 +715,7 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n \t\tgoto done;\n \t}\n \n-\terr = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),\n+\terr = indexed_table_ref_iter_new(&itr, r, oid, hash_size(r->hash_id),\n \t\t\t\t\t got.u.obj.offsets,\n \t\t\t\t\t got.u.obj.offset_len);\n \tif (err < 0)\n@@ -740,6 +740,11 @@ static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n \tint err;\n \n \tREFTABLE_ALLOC_ARRAY(ti, 1);\n+\tif (!ti) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n \ttable_iter_init(ti, r);\n \terr = table_iter_seek_start(ti, BLOCK_TYPE_REF, 0);\n \tif (err < 0)\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502887","messageId":"4d672ab9b15fcced1eef887fb22fc4aefdd12b90.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 18/22] reftable/blocksource: handle allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:29:15Z","receivedAt":"2024-09-16T12:29:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in the blocksource code.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/blocksource.c | 25 ++++++++++++++++++++-----\n 1 file changed, 20 insertions(+), 5 deletions(-)\n\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex e93cac9bb6f..6b570751be6 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -30,6 +30,8 @@ static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n \tstruct strbuf *b = v;\n \tassert(off + size <= b->len);\n \tREFTABLE_CALLOC_ARRAY(dest->data, size);\n+\tif (!dest->data)\n+\t\treturn -1;\n \tmemcpy(dest->data, b->buf + off, size);\n \tdest->len = size;\n \treturn size;\n@@ -98,27 +100,40 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,\n {\n \tstruct file_block_source *p;\n \tstruct stat st;\n-\tint fd;\n+\tint fd, err;\n \n \tfd = open(name, O_RDONLY);\n \tif (fd < 0) {\n \t\tif (errno == ENOENT)\n \t\t\treturn REFTABLE_NOT_EXIST_ERROR;\n-\t\treturn -1;\n+\t\terr = -1;\n+\t\tgoto out;\n \t}\n \n \tif (fstat(fd, &st) < 0) {\n-\t\tclose(fd);\n-\t\treturn REFTABLE_IO_ERROR;\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto out;\n \t}\n \n \tREFTABLE_CALLOC_ARRAY(p, 1);\n+\tif (!p) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n \tp->size = st.st_size;\n \tp->data = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n-\tclose(fd);\n \n \tassert(!bs->ops);\n \tbs->ops = &file_vtable;\n \tbs->arg = p;\n+\n+\terr = 0;\n+\n+out:\n+\tif (fd >= 0)\n+\t\tclose(fd);\n+\tif (err < 0)\n+\t\tfree(p);\n \treturn 0;\n }\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502888","messageId":"4fa47a559b267c7ea074e4c3832211517e0e98de.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 19/22] reftable/block: handle allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:29:18Z","receivedAt":"2024-09-16T12:29:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `block_writer_init()` and\n`block_reader_init()`. This requires us to bubble up error codes into\n`writer_reinit_block_writer()`. Adapt call sites accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/block.c | 23 +++++++++++++--\n reftable/block.h | 4 +--\n reftable/writer.c | 51 ++++++++++++++++++++++-----------\n t/unit-tests/t-reftable-block.c | 20 +++++++------\n 4 files changed, 69 insertions(+), 29 deletions(-)\n\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 00030eee065..bfa7dc61bf5 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -52,6 +52,8 @@ static int block_writer_register_restart(struct block_writer *w, int n,\n \t\treturn -1;\n \tif (is_restart) {\n \t\tREFTABLE_ALLOC_GROW(w->restarts, w->restart_len + 1, w->restart_cap);\n+\t\tif (!w->restarts)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \t\tw->restarts[w->restart_len++] = w->next;\n \t}\n \n@@ -63,8 +65,8 @@ static int block_writer_register_restart(struct block_writer *w, int n,\n \treturn 0;\n }\n \n-void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n-\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n+int block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n {\n \tbw->buf = buf;\n \tbw->hash_size = hash_size;\n@@ -78,8 +80,12 @@ void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n \tbw->last_key.len = 0;\n \tif (!bw->zstream) {\n \t\tREFTABLE_CALLOC_ARRAY(bw->zstream, 1);\n+\t\tif (!bw->zstream)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \t\tdeflateInit(bw->zstream, 9);\n \t}\n+\n+\treturn 0;\n }\n \n uint8_t block_writer_type(struct block_writer *bw)\n@@ -163,6 +169,10 @@ int block_writer_finish(struct block_writer *w)\n \t\t */\n \t\tcompressed_len = deflateBound(w->zstream, src_len);\n \t\tREFTABLE_ALLOC_GROW(w->compressed, compressed_len, w->compressed_cap);\n+\t\tif (!w->compressed) {\n+\t\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\treturn ret;\n+\t\t}\n \n \t\tw->zstream->next_out = w->compressed;\n \t\tw->zstream->avail_out = compressed_len;\n@@ -219,12 +229,21 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,\n \t\t/* Log blocks specify the *uncompressed* size in their header. */\n \t\tREFTABLE_ALLOC_GROW(br->uncompressed_data, sz,\n \t\t\t\t br->uncompressed_cap);\n+\t\tif (!br->uncompressed_data) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n \n \t\t/* Copy over the block header verbatim. It's not compressed. */\n \t\tmemcpy(br->uncompressed_data, block->data, block_header_skip);\n \n \t\tif (!br->zstream) {\n \t\t\tREFTABLE_CALLOC_ARRAY(br->zstream, 1);\n+\t\t\tif (!br->zstream) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\n \t\t\terr = inflateInit(br->zstream);\n \t\t} else {\n \t\t\terr = inflateReset(br->zstream);\ndiff --git a/reftable/block.h b/reftable/block.h\nindex 1c8f25ee6eb..18d7ea03373 100644\n--- a/reftable/block.h\n+++ b/reftable/block.h\n@@ -45,8 +45,8 @@ struct block_writer {\n /*\n * initializes the blockwriter to write `typ` entries, using `buf` as temporary\n * storage. `buf` is not owned by the block_writer. */\n-void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n-\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n+int block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n \n /* returns the block type (eg. 'r' for ref records. */\n uint8_t block_writer_type(struct block_writer *bw);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 54ec822e1cd..712a84791b2 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -102,19 +102,24 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \treturn header_size(writer_version(w));\n }\n \n-static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n+static int writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n {\n-\tint block_start = 0;\n-\tif (w->next == 0) {\n+\tint block_start = 0, ret;\n+\n+\tif (w->next == 0)\n \t\tblock_start = header_size(writer_version(w));\n-\t}\n \n \tstrbuf_reset(&w->last_key);\n-\tblock_writer_init(&w->block_writer_data, typ, w->block,\n-\t\t\t w->opts.block_size, block_start,\n-\t\t\t hash_size(w->opts.hash_id));\n+\tret = block_writer_init(&w->block_writer_data, typ, w->block,\n+\t\t\t\tw->opts.block_size, block_start,\n+\t\t\t\thash_size(w->opts.hash_id));\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n \tw->block_writer = &w->block_writer_data;\n \tw->block_writer->restart_interval = w->opts.restart_interval;\n+\n+\treturn 0;\n }\n \n int reftable_writer_new(struct reftable_writer **out,\n@@ -247,8 +252,11 @@ static int writer_add_record(struct reftable_writer *w,\n \n \tstrbuf_reset(&w->last_key);\n \tstrbuf_addbuf(&w->last_key, &key);\n-\tif (!w->block_writer)\n-\t\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\tif (!w->block_writer) {\n+\t\terr = writer_reinit_block_writer(w, reftable_record_type(rec));\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t}\n \n \tif (block_writer_type(w->block_writer) != reftable_record_type(rec))\n \t\tBUG(\"record of type %d added to writer of type %d\",\n@@ -271,7 +279,9 @@ static int writer_add_record(struct reftable_writer *w,\n \terr = writer_flush_block(w);\n \tif (err < 0)\n \t\tgoto done;\n-\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\terr = writer_reinit_block_writer(w, reftable_record_type(rec));\n+\tif (err < 0)\n+\t\tgoto done;\n \n \t/*\n \t * Try to add the record to the writer again. If this still fails then\n@@ -461,7 +471,9 @@ static int writer_finish_section(struct reftable_writer *w)\n \n \t\tmax_level++;\n \t\tindex_start = w->next;\n-\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\t\terr = writer_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n \n \t\tidx = w->index;\n \t\tidx_len = w->index_len;\n@@ -555,7 +567,10 @@ static void write_object_record(void *void_arg, void *key)\n \tif (arg->err < 0)\n \t\tgoto done;\n \n-\twriter_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\targ->err = writer_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n \targ->err = block_writer_add(arg->w->block_writer, &rec);\n \tif (arg->err == 0)\n \t\tgoto done;\n@@ -584,16 +599,18 @@ static int writer_dump_object_index(struct reftable_writer *w)\n \tstruct common_prefix_arg common = {\n \t\t.max = 1,\t\t/* obj_id_len should be >= 2. */\n \t};\n-\tif (w->obj_index_tree) {\n+\tint err;\n+\n+\tif (w->obj_index_tree)\n \t\tinfix_walk(w->obj_index_tree, &update_common, &common);\n-\t}\n \tw->stats.object_id_len = common.max + 1;\n \n-\twriter_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\terr = writer_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\tif (err < 0)\n+\t\treturn err;\n \n-\tif (w->obj_index_tree) {\n+\tif (w->obj_index_tree)\n \t\tinfix_walk(w->obj_index_tree, &write_object_record, &closure);\n-\t}\n \n \tif (closure.err < 0)\n \t\treturn closure.err;\ndiff --git a/t/unit-tests/t-reftable-block.c b/t/unit-tests/t-reftable-block.c\nindex f1a49485e23..e52a612e852 100644\n--- a/t/unit-tests/t-reftable-block.c\n+++ b/t/unit-tests/t-reftable-block.c\n@@ -34,8 +34,9 @@ static void t_ref_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \trec.u.ref.refname = (char *) \"\";\n \trec.u.ref.value_type = REFTABLE_REF_DELETION;\n@@ -126,8 +127,9 @@ static void t_log_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n \t\trec.u.log.refname = xstrfmt(\"branch%02\"PRIuMAX , (uintmax_t)i);\n@@ -214,8 +216,9 @@ static void t_obj_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n \t\tuint8_t bytes[] = { i, i + 1, i + 2, i + 3, i + 5 }, *allocated;\n@@ -296,8 +299,9 @@ static void t_index_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n \t\tstrbuf_init(&recs[i].u.idx.last_key, 9);\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502889","messageId":"a569c39fd8e663cca78dcb6025b36ab9366ab995.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 20/22] reftable/pq: handle allocation failures when adding entries","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:29:21Z","receivedAt":"2024-09-16T12:29:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when adding entries to the pqueue. Adapt its\nonly caller accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 5 ++++-\n reftable/pq.c | 7 ++++++-\n reftable/pq.h | 2 +-\n 3 files changed, 11 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 741f62ea638..2c20845d624 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -54,7 +54,10 @@ static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)\n \tif (err)\n \t\treturn err;\n \n-\tmerged_iter_pqueue_add(&mi->pq, &e);\n+\terr = merged_iter_pqueue_add(&mi->pq, &e);\n+\tif (err)\n+\t\treturn err;\n+\n \treturn 0;\n }\n \ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex 2b5b7d1c0e2..03b99122829 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -8,6 +8,7 @@ license that can be found in the LICENSE file or at\n \n #include \"pq.h\"\n \n+#include \"reftable-error.h\"\n #include \"reftable-record.h\"\n #include \"system.h\"\n #include \"basics.h\"\n@@ -44,11 +45,13 @@ struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n \treturn e;\n }\n \n-void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e)\n+int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e)\n {\n \tsize_t i = 0;\n \n \tREFTABLE_ALLOC_GROW(pq->heap, pq->len + 1, pq->cap);\n+\tif (!pq->heap)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tpq->heap[pq->len++] = *e;\n \n \ti = pq->len - 1;\n@@ -59,6 +62,8 @@ void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry\n \t\tSWAP(pq->heap[j], pq->heap[i]);\n \t\ti = j;\n \t}\n+\n+\treturn 0;\n }\n \n void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)\ndiff --git a/reftable/pq.h b/reftable/pq.h\nindex 707bd26767b..83c062eecac 100644\n--- a/reftable/pq.h\n+++ b/reftable/pq.h\n@@ -23,7 +23,7 @@ struct merged_iter_pqueue {\n };\n \n struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n-void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e);\n+int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e);\n void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n int pq_less(struct pq_entry *a, struct pq_entry *b);\n \n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502890","messageId":"99bca688f2a393190bb032238fb36e01088ff24c.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 21/22] reftable/tree: handle allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:29:23Z","receivedAt":"2024-09-16T12:29:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The tree interfaces of the reftable library handle both insertion and\nsearching of tree nodes with a single function, where the behaviour is\naltered between the two via an `insert` bit. This makes it quit awkward\nto handle allocation failures because on inserting we'd have to check\nfor `NULL` pointers and return an error, whereas on searching entries we\ndon't have to handle it as an allocation error.\n\nSplit up concerns of this function into two separate functions, one for\ninserting entries and one for searching entries. This makes it easy for\nus to check for allocation errors as `tree_insert()` should never return\na `NULL` pointer now. Adapt callers accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/tree.c | 42 +++++++++++++++++++++++-----------\n reftable/tree.h | 21 +++++++++++++----\n reftable/writer.c | 7 +++---\n t/unit-tests/t-reftable-tree.c | 10 ++++----\n 4 files changed, 54 insertions(+), 26 deletions(-)\n\ndiff --git a/reftable/tree.c b/reftable/tree.c\nindex 5ffb2e0d690..f4dbe720901 100644\n--- a/reftable/tree.c\n+++ b/reftable/tree.c\n@@ -11,28 +11,44 @@ license that can be found in the LICENSE file or at\n \n #include \"basics.h\"\n \n-struct tree_node *tree_search(void *key, struct tree_node **rootp,\n-\t\t\t int (*compare)(const void *, const void *),\n-\t\t\t int insert)\n+struct tree_node *tree_search(struct tree_node *tree,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *))\n {\n \tint res;\n+\tif (!tree)\n+\t\treturn NULL;\n+\tres = compare(key, tree->key);\n+\tif (res < 0)\n+\t\treturn tree_search(tree->left, key, compare);\n+\telse if (res > 0)\n+\t\treturn tree_search(tree->right, key, compare);\n+\treturn tree;\n+}\n+\n+struct tree_node *tree_insert(struct tree_node **rootp,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *))\n+{\n+\tint res;\n+\n \tif (!*rootp) {\n-\t\tif (!insert) {\n+\t\tstruct tree_node *n;\n+\n+\t\tREFTABLE_CALLOC_ARRAY(n, 1);\n+\t\tif (!n)\n \t\t\treturn NULL;\n-\t\t} else {\n-\t\t\tstruct tree_node *n;\n-\t\t\tREFTABLE_CALLOC_ARRAY(n, 1);\n-\t\t\tn->key = key;\n-\t\t\t*rootp = n;\n-\t\t\treturn *rootp;\n-\t\t}\n+\n+\t\tn->key = key;\n+\t\t*rootp = n;\n+\t\treturn *rootp;\n \t}\n \n \tres = compare(key, (*rootp)->key);\n \tif (res < 0)\n-\t\treturn tree_search(key, &(*rootp)->left, compare, insert);\n+\t\treturn tree_insert(&(*rootp)->left, key, compare);\n \telse if (res > 0)\n-\t\treturn tree_search(key, &(*rootp)->right, compare, insert);\n+\t\treturn tree_insert(&(*rootp)->right, key, compare);\n \treturn *rootp;\n }\n \ndiff --git a/reftable/tree.h b/reftable/tree.h\nindex fbdd002e23a..9604453b6d5 100644\n--- a/reftable/tree.h\n+++ b/reftable/tree.h\n@@ -15,12 +15,23 @@ struct tree_node {\n \tstruct tree_node *left, *right;\n };\n \n-/* looks for `key` in `rootp` using `compare` as comparison function. If insert\n- * is set, insert the key if it's not found. Else, return NULL.\n+/*\n+ * Search the tree for the node matching the given key using `compare` as\n+ * comparison function. Returns the node whose key matches or `NULL` in case\n+ * the key does not exist in the tree.\n+ */\n+struct tree_node *tree_search(struct tree_node *tree,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *));\n+\n+/*\n+ * Insert a node into the tree. Returns the newly inserted node if the key does\n+ * not yet exist. Otherwise it returns the preexisting node. Returns `NULL`\n+ * when allocating the new node fails.\n */\n-struct tree_node *tree_search(void *key, struct tree_node **rootp,\n-\t\t\t int (*compare)(const void *, const void *),\n-\t\t\t int insert);\n+struct tree_node *tree_insert(struct tree_node **rootp,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *));\n \n /* performs an infix walk of the tree. */\n void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 712a84791b2..44bed92f467 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -208,8 +208,7 @@ static int writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n \tstruct obj_index_tree_node *key;\n \tstruct tree_node *node;\n \n-\tnode = tree_search(&want, &w->obj_index_tree,\n-\t\t\t &obj_index_tree_node_compare, 0);\n+\tnode = tree_search(w->obj_index_tree, &want, &obj_index_tree_node_compare);\n \tif (!node) {\n \t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n \n@@ -221,8 +220,8 @@ static int writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n \n \t\tstrbuf_reset(&key->hash);\n \t\tstrbuf_addbuf(&key->hash, hash);\n-\t\ttree_search((void *)key, &w->obj_index_tree,\n-\t\t\t &obj_index_tree_node_compare, 1);\n+\t\ttree_insert(&w->obj_index_tree, key,\n+\t\t\t &obj_index_tree_node_compare);\n \t} else {\n \t\tkey = node->key;\n \t}\ndiff --git a/t/unit-tests/t-reftable-tree.c b/t/unit-tests/t-reftable-tree.c\nindex 700479d34b1..79b175a45a7 100644\n--- a/t/unit-tests/t-reftable-tree.c\n+++ b/t/unit-tests/t-reftable-tree.c\n@@ -37,16 +37,17 @@ static void t_tree_search(void)\n \t * values[1] and values[10] (inclusive) in the tree.\n \t */\n \tdo {\n-\t\tnodes[i] = tree_search(&values[i], &root, &t_compare, 1);\n+\t\tnodes[i] = tree_insert(&root, &values[i], &t_compare);\n+\t\tcheck(nodes[i] != NULL);\n \t\ti = (i * 7) % 11;\n \t} while (i != 1);\n \n \tfor (i = 1; i < ARRAY_SIZE(nodes); i++) {\n \t\tcheck_pointer_eq(&values[i], nodes[i]->key);\n-\t\tcheck_pointer_eq(nodes[i], tree_search(&values[i], &root, &t_compare, 0));\n+\t\tcheck_pointer_eq(nodes[i], tree_search(root, &values[i], &t_compare));\n \t}\n \n-\tcheck(!tree_search(values, &root, t_compare, 0));\n+\tcheck(!tree_search(root, values, t_compare));\n \ttree_free(root);\n }\n \n@@ -62,7 +63,8 @@ static void t_infix_walk(void)\n \tsize_t count = 0;\n \n \tdo {\n-\t\ttree_search(&values[i], &root, t_compare, 1);\n+\t\tstruct tree_node *node = tree_insert(&root, &values[i], t_compare);\n+\t\tcheck(node != NULL);\n \t\ti = (i * 7) % 11;\n \t\tcount++;\n \t} while (i != 1);\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"502891","messageId":"02073dafb96d58b76d37f2dad5efe9807634dc82.1726489647.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH 22/22] reftable: handle trivial allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-16T12:29:26Z","receivedAt":"2024-09-16T12:29:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle trivial allocation failures in the reftable library and its unit\ntests.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 3 +++\n reftable/reader.c | 10 +++++++++-\n reftable/stack.c | 20 ++++++++++++++++++++\n reftable/writer.c | 13 +++++++++++--\n t/unit-tests/t-reftable-block.c | 4 ++++\n t/unit-tests/t-reftable-merged.c | 4 ++++\n t/unit-tests/t-reftable-readwrite.c | 27 ++++++++++++++++-----------\n 7 files changed, 67 insertions(+), 14 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 2c20845d624..52bc66c5273 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -203,6 +203,9 @@ int reftable_merged_table_new(struct reftable_merged_table **dest,\n \t}\n \n \tREFTABLE_CALLOC_ARRAY(m, 1);\n+\tif (!m)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \tm->readers = readers;\n \tm->readers_len = n;\n \tm->min = first_min;\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 0179e4e73dd..98e7aa26373 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -598,6 +598,10 @@ int reftable_reader_new(struct reftable_reader **out,\n \tint err;\n \n \tREFTABLE_CALLOC_ARRAY(r, 1);\n+\tif (!r) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \n \t/*\n \t * We need one extra byte to read the type of first block. We also\n@@ -627,7 +631,11 @@ int reftable_reader_new(struct reftable_reader **out,\n \n \tr->size = file_size - footer_size(r->version);\n \tr->source = *source;\n-\tr->name = xstrdup(name);\n+\tr->name = reftable_strdup(name);\n+\tif (!r->name) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \tr->hash_id = 0;\n \tr->refcount = 1;\n \ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 990784d9d2f..7df28ab3438 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -116,6 +116,11 @@ static int fd_read_lines(int fd, char ***namesp)\n \t}\n \n \tREFTABLE_ALLOC_ARRAY(buf, size + 1);\n+\tif (!buf) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tif (read_in_full(fd, buf, size) != size) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n@@ -140,6 +145,8 @@ int read_lines(const char *filename, char ***namesp)\n \tif (fd < 0) {\n \t\tif (errno == ENOENT) {\n \t\t\tREFTABLE_CALLOC_ARRAY(*namesp, 1);\n+\t\t\tif (!*namesp)\n+\t\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \t\t\treturn 0;\n \t\t}\n \n@@ -420,6 +427,10 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\t\t}\n \n \t\t\tREFTABLE_CALLOC_ARRAY(names, 1);\n+\t\t\tif (!names) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto out;\n+\t\t\t}\n \t\t} else {\n \t\t\terr = fd_read_lines(fd, &names);\n \t\t\tif (err < 0)\n@@ -779,7 +790,11 @@ int reftable_stack_new_addition(struct reftable_addition **dest,\n {\n \tint err = 0;\n \tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n+\n \tREFTABLE_CALLOC_ARRAY(*dest, 1);\n+\tif (!*dest)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \t**dest = empty;\n \terr = reftable_stack_init_addition(*dest, st);\n \tif (err) {\n@@ -886,7 +901,12 @@ int reftable_addition_add(struct reftable_addition *add,\n \n \tREFTABLE_ALLOC_GROW(add->new_tables, add->new_tables_len + 1,\n \t\t\t add->new_tables_cap);\n+\tif (!add->new_tables) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \tadd->new_tables[add->new_tables_len++] = strbuf_detach(&next_name, NULL);\n+\n done:\n \tdelete_tempfile(&tab_file);\n \tstrbuf_release(&temp_tab_file_name);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 44bed92f467..ad4718edecb 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -49,8 +49,14 @@ static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n {\n \tint n = 0;\n \tif (w->pending_padding > 0) {\n-\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding, sizeof(*zeroed));\n-\t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n+\t\tuint8_t *zeroed;\n+\t\tint n;\n+\n+\t\tzeroed = reftable_calloc(w->pending_padding, sizeof(*zeroed));\n+\t\tif (!zeroed)\n+\t\t\treturn -1;\n+\n+\t\tn = w->write(w->write_arg, zeroed, w->pending_padding);\n \t\tif (n < 0)\n \t\t\treturn n;\n \n@@ -767,6 +773,9 @@ static int writer_flush_nonempty_block(struct reftable_writer *w)\n \t * case we will end up with a multi-level index.\n \t */\n \tREFTABLE_ALLOC_GROW(w->index, w->index_len + 1, w->index_cap);\n+\tif (!w->index)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \tindex_record.offset = w->next;\n \tstrbuf_reset(&index_record.last_key);\n \tstrbuf_addbuf(&index_record.last_key, &w->block_writer->last_key);\ndiff --git a/t/unit-tests/t-reftable-block.c b/t/unit-tests/t-reftable-block.c\nindex e52a612e852..d470060e8be 100644\n--- a/t/unit-tests/t-reftable-block.c\n+++ b/t/unit-tests/t-reftable-block.c\n@@ -32,6 +32,7 @@ static void t_ref_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n@@ -125,6 +126,7 @@ static void t_log_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size,\n@@ -214,6 +216,7 @@ static void t_obj_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size,\n@@ -297,6 +300,7 @@ static void t_index_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size,\ndiff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c\nindex 3d2848632db..3c84363e980 100644\n--- a/t/unit-tests/t-reftable-merged.c\n+++ b/t/unit-tests/t-reftable-merged.c\n@@ -29,7 +29,9 @@ merged_table_from_records(struct reftable_ref_record **refs,\n \tint err;\n \n \tREFTABLE_CALLOC_ARRAY(*readers, n);\n+\tcheck(*readers != NULL);\n \tREFTABLE_CALLOC_ARRAY(*source, n);\n+\tcheck(*source != NULL);\n \n \tfor (size_t i = 0; i < n; i++) {\n \t\tt_reftable_write_to_buf(&buf[i], refs[i], sizes[i], NULL, 0, &opts);\n@@ -285,7 +287,9 @@ merged_table_from_log_records(struct reftable_log_record **logs,\n \tint err;\n \n \tREFTABLE_CALLOC_ARRAY(*readers, n);\n+\tcheck(*readers != NULL);\n \tREFTABLE_CALLOC_ARRAY(*source, n);\n+\tcheck(*source != NULL);\n \n \tfor (size_t i = 0; i < n; i++) {\n \t\tt_reftable_write_to_buf(&buf[i], NULL, 0, logs[i], sizes[i], &opts);\ndiff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c\nindex acca927a2cf..bfa069caff7 100644\n--- a/t/unit-tests/t-reftable-readwrite.c\n+++ b/t/unit-tests/t-reftable-readwrite.c\n@@ -52,8 +52,11 @@ static void write_table(char ***names, struct strbuf *buf, int N,\n \tint i;\n \n \tREFTABLE_CALLOC_ARRAY(*names, N + 1);\n+\tcheck(*names != NULL);\n \tREFTABLE_CALLOC_ARRAY(refs, N);\n+\tcheck(refs != NULL);\n \tREFTABLE_CALLOC_ARRAY(logs, N);\n+\tcheck(logs != NULL);\n \n \tfor (i = 0; i < N; i++) {\n \t\trefs[i].refname = (*names)[i] = xstrfmt(\"refs/heads/branch%02d\", i);\n@@ -150,23 +153,25 @@ static void t_log_overflow(void)\n \n static void t_log_write_read(void)\n {\n-\tint N = 2;\n-\tchar **names = reftable_calloc(N + 1, sizeof(*names));\n-\tint err;\n \tstruct reftable_write_options opts = {\n \t\t.block_size = 256,\n \t};\n \tstruct reftable_ref_record ref = { 0 };\n-\tint i = 0;\n \tstruct reftable_log_record log = { 0 };\n-\tint n;\n \tstruct reftable_iterator it = { 0 };\n \tstruct reftable_reader *reader;\n \tstruct reftable_block_source source = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w = t_reftable_strbuf_writer(&buf, &opts);\n \tconst struct reftable_stats *stats = NULL;\n+\tint N = 2, err, i, n;\n+\tchar **names;\n+\n+\tnames = reftable_calloc(N + 1, sizeof(*names));\n+\tcheck(names != NULL);\n+\n \treftable_writer_set_limits(w, 0, N);\n+\n \tfor (i = 0; i < N; i++) {\n \t\tchar name[256];\n \t\tstruct reftable_ref_record ref = { 0 };\n@@ -178,6 +183,7 @@ static void t_log_write_read(void)\n \t\terr = reftable_writer_add_ref(w, &ref);\n \t\tcheck(!err);\n \t}\n+\n \tfor (i = 0; i < N; i++) {\n \t\tstruct reftable_log_record log = { 0 };\n \n@@ -476,8 +482,7 @@ static void t_table_read_write_seek_index(void)\n \n static void t_table_refs_for(int indexed)\n {\n-\tint N = 50;\n-\tchar **want_names = reftable_calloc(N + 1, sizeof(*want_names));\n+\tchar **want_names;\n \tint want_names_len = 0;\n \tuint8_t want_hash[GIT_SHA1_RAWSZ];\n \n@@ -485,15 +490,15 @@ static void t_table_refs_for(int indexed)\n \t\t.block_size = 256,\n \t};\n \tstruct reftable_ref_record ref = { 0 };\n-\tint i = 0;\n-\tint n;\n-\tint err;\n \tstruct reftable_reader *reader;\n \tstruct reftable_block_source source = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w = t_reftable_strbuf_writer(&buf, &opts);\n \tstruct reftable_iterator it = { 0 };\n-\tint j;\n+\tint N = 50, n, j, err, i;\n+\n+\twant_names = reftable_calloc(N + 1, sizeof(*want_names));\n+\tcheck(want_names != NULL);\n \n \tt_reftable_set_hash(want_hash, 4, GIT_SHA1_FORMAT_ID);\n \n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503203","messageId":"xmqqed5cajd5.fsf@gitster.g","threadId":"62121","inReplyTo":"e6ded75f630ea309d5b76126560a0ec3d526bf71.1726489647.git.ps@pks.im","subject":"Re: [PATCH 04/22] reftable/basics: handle allocation failures in `reftable_calloc()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-09-21T19:37:26Z","receivedAt":"2024-09-21T19:37:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> Handle allocation failures in `reftable_calloc()`.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> reftable/basics.c | 2 ++\n> 1 file changed, 2 insertions(+)\n>\n> diff --git a/reftable/basics.c b/reftable/basics.c\n> index 4adc98cf5de..b404900b5d9 100644\n> --- a/reftable/basics.c\n> +++ b/reftable/basics.c\n> @@ -39,6 +39,8 @@ void *reftable_calloc(size_t nelem, size_t elsize)\n> {\n> \tsize_t sz = st_mult(nelem, elsize);\n> \tvoid *p = reftable_malloc(sz);\n> +\tif (!p)\n> +\t\treturn NULL;\n> \tmemset(p, 0, sz);\n> \treturn p;\n> }\n\nSince this series is not about eradicating all avenues in reftable\nlibrary code that can lead to die(), but only about dealing with\nallocation errors from the underlying malloc/realloc routines, I\nthink the code posted is perfectly fine as-is as a part of this\nseries, but since I noticed something, let me comment before I\nforget.\n\nWhen st_mult() detects overflow, you'd still die(), wouldn't you?\n\nWe'd probably want a variant of st_mult() that lets us notice a\ncondition that would yield too large a result, and code the above\nlike so,\n\n\tsize_t sz;\n\tvoid *p;\n\n\tif (st_mult_gently(nelem, elsize, &sz) ||\n !((p = reftable_malloc(sz))))\n\t\treturn NULL;\n\tmemset(p, 0, sz);\n\treturn p;\n\nor use the underlying helper ourselves, and say\n\n\tsize_t sz;\n\tvoid *p;\n\n\tif (unsigned_mult_overflows(nelem, elsize)) ||\n !((sz = nelem * elsize, p = reftable_malloc(sz))))\n\t\treturn NULL;\n\tmemset(p, 0, sz);\n\treturn p;\n\nwhich lets us without an extra helper but after writing it myself, I\nfind it a bit too wordy.\n\nIn a sense, it is on the borderline to handle st_mult() overflow in\nthis function for a topic whose theme is about allocation failures.\n\nFrom the point of view of callers of reftable_calloc(), whether the\narguments they are feeding the function is too large to be\nmultiplied or whether the request is too big for the underlying\nallocator to handle, the end result should be the same: they\nrequested too large an allocation.\n\nSo I wouldn't complain that it is out of scope, if use of st_mult()\nthat computes the allocation size is fixed as part of this series.\nBut as I already said, I am also OK if we leave it to a separate\nseries to tackle other potential callers of die().\n\nThanks.\n"},{"id":"503209","messageId":"xmqq7cb49pby.fsf@gitster.g","threadId":"62121","inReplyTo":"32fead57de989335b17d16f63c1cd144460495a1.1726489647.git.ps@pks.im","subject":"Re: [PATCH 17/22] reftable/iter: handle allocation failures when creating indexed table iter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-09-22T06:26:09Z","receivedAt":"2024-09-22T06:26:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> -int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n> +int indexed_table_ref_iter_new(struct indexed_table_ref_iter **dest,\n> \t\t\t struct reftable_reader *r, uint8_t *oid,\n> \t\t\t int oid_len, uint64_t *offsets, int offset_len)\n> {\n> \tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n> -\tstruct indexed_table_ref_iter *itr = reftable_calloc(1, sizeof(*itr));\n> +\tstruct indexed_table_ref_iter *itr;\n> \tint err = 0;\n> \n> +\titr = reftable_calloc(1, sizeof(*itr));\n> +\tif (!itr) {\n> +\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n> +\t\tgoto out;\n> +\t}\n> +\n> \t*itr = empty;\n> \titr->r = r;\n> \tstrbuf_add(&itr->oid, oid, oid_len);\n> @@ -197,11 +203,15 @@ int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n> \titr->offset_len = offset_len;\n> \n> \terr = indexed_table_ref_iter_next_block(itr);\n> -\tif (err < 0) {\n> +\tif (err < 0)\n> +\t\tgoto out;\n> +\n> +\t*dest = itr;\n> +\terr = 0;\n> +\n> +out:\n> +\tif (err < 0)\n> \t\treftable_free(itr);\n> -\t} else {\n> -\t\t*dest = itr;\n> -\t}\n> \treturn err;\n> }\n\nUnless the service the helper function offers is to upgrade an\nexisting resource (e.g., realloc() taking a pointer and give an\nenlarged piece of memory), it may be a safer calling convention to\npromise that *dest is cleared to NULL when the function fails,\ninstead of promising that *dest is left intact. The caller, when it\nneeds to evantually release the resource acquired here, has to\nremember what the returned value (i.e., err) was, in order to decide\nif it needs to call the release helper on *dest it obtained from us.\n\nThe only caller seems to initialize *dest to NULL itself, so it does\nnot matter in the current code, though.\n"},{"id":"503304","messageId":"ZvJSyT8OsUVLln4A@pks.im","threadId":"62121","inReplyTo":"xmqqed5cajd5.fsf@gitster.g","subject":"Re: [PATCH 04/22] reftable/basics: handle allocation failures in `reftable_calloc()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T05:48:57Z","receivedAt":"2024-09-24T05:49:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Sep 21, 2024 at 12:37:26PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > Handle allocation failures in `reftable_calloc()`.\n> >\n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > reftable/basics.c | 2 ++\n> > 1 file changed, 2 insertions(+)\n> >\n> > diff --git a/reftable/basics.c b/reftable/basics.c\n> > index 4adc98cf5de..b404900b5d9 100644\n> > --- a/reftable/basics.c\n> > +++ b/reftable/basics.c\n> > @@ -39,6 +39,8 @@ void *reftable_calloc(size_t nelem, size_t elsize)\n> > {\n> > \tsize_t sz = st_mult(nelem, elsize);\n> > \tvoid *p = reftable_malloc(sz);\n> > +\tif (!p)\n> > +\t\treturn NULL;\n> > \tmemset(p, 0, sz);\n> > \treturn p;\n> > }\n> \n> Since this series is not about eradicating all avenues in reftable\n> library code that can lead to die(), but only about dealing with\n> allocation errors from the underlying malloc/realloc routines, I\n> think the code posted is perfectly fine as-is as a part of this\n> series, but since I noticed something, let me comment before I\n> forget.\n> \n> When st_mult() detects overflow, you'd still die(), wouldn't you?\n\nTrue.\n\n> We'd probably want a variant of st_mult() that lets us notice a\n> condition that would yield too large a result, and code the above\n> like so,\n> \n> \tsize_t sz;\n> \tvoid *p;\n> \n> \tif (st_mult_gently(nelem, elsize, &sz) ||\n> !((p = reftable_malloc(sz))))\n> \t\treturn NULL;\n> \tmemset(p, 0, sz);\n> \treturn p;\n> \n> or use the underlying helper ourselves, and say\n> \n> \tsize_t sz;\n> \tvoid *p;\n> \n> \tif (unsigned_mult_overflows(nelem, elsize)) ||\n> !((sz = nelem * elsize, p = reftable_malloc(sz))))\n> \t\treturn NULL;\n> \tmemset(p, 0, sz);\n> \treturn p;\n> \n> which lets us without an extra helper but after writing it myself, I\n> find it a bit too wordy.\n\nYeah, we'll have to have something like this eventually.\n\n> In a sense, it is on the borderline to handle st_mult() overflow in\n> this function for a topic whose theme is about allocation failures.\n> \n> From the point of view of callers of reftable_calloc(), whether the\n> arguments they are feeding the function is too large to be\n> multiplied or whether the request is too big for the underlying\n> allocator to handle, the end result should be the same: they\n> requested too large an allocation.\n> \n> So I wouldn't complain that it is out of scope, if use of st_mult()\n> that computes the allocation size is fixed as part of this series.\n> But as I already said, I am also OK if we leave it to a separate\n> series to tackle other potential callers of die().\n\nI'd leave it as-is for now, but I do have it on my agenda to address\nthis, as well. I already have it as part of my third patch series in\nthis context where I completely detangle the reftable library from the\nrest of Git to make it a reusable library for libgit2 and the likes.\n\nIn any case, there's another, bigger elephant in the room: `struct\nstrbuf`. I will have to introduce a reftable-specific buffer type for\nthis such that we can handle allocation failures here, too. While the\nalternative would be to amend `struct strbuf` itself to do that, I don't\nquite think that we should do it in \"core\" Git itself for now. And it is\nanother excuse for me for why the reftable code should have its own type\ninstead of relying on `struct strbuf` in the context of making it a\nstandalone library again.\n\nPatrick\n"},{"id":"503305","messageId":"ZvJS0Nt-1_Bbq5fB@pks.im","threadId":"62121","inReplyTo":"xmqq7cb49pby.fsf@gitster.g","subject":"Re: [PATCH 17/22] reftable/iter: handle allocation failures when creating indexed table iter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T05:49:04Z","receivedAt":"2024-09-24T05:49:10Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Sep 21, 2024 at 11:26:09PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > -int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n> > +int indexed_table_ref_iter_new(struct indexed_table_ref_iter **dest,\n> > \t\t\t struct reftable_reader *r, uint8_t *oid,\n> > \t\t\t int oid_len, uint64_t *offsets, int offset_len)\n> > {\n> > \tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n> > -\tstruct indexed_table_ref_iter *itr = reftable_calloc(1, sizeof(*itr));\n> > +\tstruct indexed_table_ref_iter *itr;\n> > \tint err = 0;\n> > \n> > +\titr = reftable_calloc(1, sizeof(*itr));\n> > +\tif (!itr) {\n> > +\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n> > +\t\tgoto out;\n> > +\t}\n> > +\n> > \t*itr = empty;\n> > \titr->r = r;\n> > \tstrbuf_add(&itr->oid, oid, oid_len);\n> > @@ -197,11 +203,15 @@ int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n> > \titr->offset_len = offset_len;\n> > \n> > \terr = indexed_table_ref_iter_next_block(itr);\n> > -\tif (err < 0) {\n> > +\tif (err < 0)\n> > +\t\tgoto out;\n> > +\n> > +\t*dest = itr;\n> > +\terr = 0;\n> > +\n> > +out:\n> > +\tif (err < 0)\n> > \t\treftable_free(itr);\n> > -\t} else {\n> > -\t\t*dest = itr;\n> > -\t}\n> > \treturn err;\n> > }\n> \n> Unless the service the helper function offers is to upgrade an\n> existing resource (e.g., realloc() taking a pointer and give an\n> enlarged piece of memory), it may be a safer calling convention to\n> promise that *dest is cleared to NULL when the function fails,\n> instead of promising that *dest is left intact. The caller, when it\n> needs to evantually release the resource acquired here, has to\n> remember what the returned value (i.e., err) was, in order to decide\n> if it needs to call the release helper on *dest it obtained from us.\n> \n> The only caller seems to initialize *dest to NULL itself, so it does\n> not matter in the current code, though.\n\nI don't really see it as much of a problem here, mostly because this\nfunction is internal to the reftable library anyway. Also, callers\nessentially have to NULL-initialize the variable anyway once there are\nmultiple error paths and if they want to free it, because otherwise they\ncould end up freeing a uninitialized pointer.\n\nOn the other hand it doesn't hurt much to assign `NULL` on the error\npath either, so I'll just do that.\n\nPatrick\n"},{"id":"503306","messageId":"ZvJWCauoFpjYcpKs@pks.im","threadId":"62121","inReplyTo":"ZvJSyT8OsUVLln4A@pks.im","subject":"Re: [PATCH 04/22] reftable/basics: handle allocation failures in `reftable_calloc()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:02:49Z","receivedAt":"2024-09-24T06:02:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Sep 24, 2024 at 07:48:57AM +0200, Patrick Steinhardt wrote:\n> On Sat, Sep 21, 2024 at 12:37:26PM -0700, Junio C Hamano wrote:\n> > Patrick Steinhardt writes:\n> > In a sense, it is on the borderline to handle st_mult() overflow in\n> > this function for a topic whose theme is about allocation failures.\n> > \n> > From the point of view of callers of reftable_calloc(), whether the\n> > arguments they are feeding the function is too large to be\n> > multiplied or whether the request is too big for the underlying\n> > allocator to handle, the end result should be the same: they\n> > requested too large an allocation.\n> > \n> > So I wouldn't complain that it is out of scope, if use of st_mult()\n> > that computes the allocation size is fixed as part of this series.\n> > But as I already said, I am also OK if we leave it to a separate\n> > series to tackle other potential callers of die().\n> \n> I'd leave it as-is for now, but I do have it on my agenda to address\n> this, as well. I already have it as part of my third patch series in\n> this context where I completely detangle the reftable library from the\n> rest of Git to make it a reusable library for libgit2 and the likes.\n\nWell, you know. I reroll the series anyway, so I'll just make the change\nnow.\n\nPatrick\n"},{"id":"503307","messageId":"cover.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH v2 00/22] reftable: handle allocation errors","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:31:49Z","receivedAt":"2024-09-24T06:31:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the second patch series that converts the reftable library to\nstart handling allocation errors. This is done such that the reftable\nlibrary can truly behave like a library and let its callers handle such\nconditions.\n\nChanges compared to v1:\n\n - Remove our use of `st_mult()` in `reftable_calloc()`.\n\n - Initialize the `*dest` pointer with `NULL` in case creating a new\n indexed table iterator fails.\n\nThanks!\n\nPatrick\n\nPatrick Steinhardt (22):\n reftable/error: introduce out-of-memory error code\n reftable/basics: merge \"publicbasics\" into \"basics\"\n reftable: introduce `reftable_strdup()`\n reftable/basics: handle allocation failures in `reftable_calloc()`\n reftable/basics: handle allocation failures in `parse_names()`\n reftable/record: handle allocation failures on copy\n reftable/record: handle allocation failures when decoding records\n reftable/writer: handle allocation failures in `writer_index_hash()`\n reftable/writer: handle allocation failures in `reftable_new_writer()`\n reftable/merged: handle allocation failures in\n `merged_table_init_iter()`\n reftable/reader: handle allocation failures for unindexed reader\n reftable/reader: handle allocation failures in `reader_init_iter()`\n reftable/stack: handle allocation failures on reload\n reftable/stack: handle allocation failures in `reftable_new_stack()`\n reftable/stack: handle allocation failures in `stack_compact_range()`\n reftable/stack: handle allocation failures in auto compaction\n reftable/iter: handle allocation failures when creating indexed table\n iter\n reftable/blocksource: handle allocation failures\n reftable/block: handle allocation failures\n reftable/pq: handle allocation failures when adding entries\n reftable/tree: handle allocation failures\n reftable: handle trivial allocation failures\n\n Makefile | 1 -\n refs/reftable-backend.c | 39 ++++--\n reftable/basics.c | 92 ++++++++++++++-\n reftable/basics.h | 13 +-\n reftable/block.c | 23 +++-\n reftable/block.h | 4 +-\n reftable/blocksource.c | 25 +++-\n reftable/error.c | 2 +\n reftable/iter.c | 20 +++-\n reftable/iter.h | 2 +-\n reftable/merged.c | 84 ++++++++-----\n reftable/merged.h | 6 +-\n reftable/pq.c | 7 +-\n reftable/pq.h | 2 +-\n reftable/publicbasics.c | 66 -----------\n reftable/reader.c | 68 ++++++++---\n reftable/reader.h | 6 +-\n reftable/record.c | 164 +++++++++++++++++++-------\n reftable/record.h | 6 +-\n reftable/reftable-basics.h | 18 +++\n reftable/reftable-error.h | 3 +\n reftable/reftable-malloc.h | 18 ---\n reftable/reftable-merged.h | 8 +-\n reftable/reftable-reader.h | 8 +-\n reftable/reftable-stack.h | 8 +-\n reftable/reftable-writer.h | 12 +-\n reftable/stack.c | 177 ++++++++++++++++++++++------\n reftable/tree.c | 42 +++++--\n reftable/tree.h | 21 +++-\n reftable/writer.c | 150 +++++++++++++++--------\n t/helper/test-reftable.c | 10 +-\n t/unit-tests/lib-reftable.c | 8 +-\n t/unit-tests/t-reftable-basics.c | 11 +-\n t/unit-tests/t-reftable-block.c | 24 ++--\n t/unit-tests/t-reftable-merged.c | 16 ++-\n t/unit-tests/t-reftable-readwrite.c | 61 ++++++----\n t/unit-tests/t-reftable-stack.c | 4 +-\n t/unit-tests/t-reftable-tree.c | 10 +-\n 38 files changed, 853 insertions(+), 386 deletions(-)\n delete mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-basics.h\n delete mode 100644 reftable/reftable-malloc.h\n\nRange-diff against v1:\n 1: 8c99ecc3255 = 1: 8c99ecc3255 reftable/error: introduce out-of-memory error code\n 2: 4dcdf1d48ec = 2: 4dcdf1d48ec reftable/basics: merge \"publicbasics\" into \"basics\"\n 3: 21fa9b15d96 = 3: 21fa9b15d96 reftable: introduce `reftable_strdup()`\n 4: e6ded75f630 ! 4: f6ad92ffd01 reftable/basics: handle allocation failures in `reftable_calloc()`\n @@ Commit message\n \n Handle allocation failures in `reftable_calloc()`.\n \n + While at it, remove our use of `st_mult()` that would cause us to die on\n + an overflow. From the caller's point of view there is not much of a\n + difference between arguments that are too large to be multiplied and a\n + request that is too big to handle by the allocator: in both cases the\n + allocation cannot be fulfilled. And in neither of these cases do we want\n + the reftable library to die.\n +\n + While we could use `unsigned_mult_overflows()` to handle the overflow\n + gracefully, we instead open-code it to further our goal of converting\n + the reftable codebase to become a standalone library that can be reused\n + by external projects.\n +\n Signed-off-by: Patrick Steinhardt \n \n ## reftable/basics.c ##\n -@@ reftable/basics.c: void *reftable_calloc(size_t nelem, size_t elsize)\n +@@ reftable/basics.c: void reftable_free(void *p)\n + \n + void *reftable_calloc(size_t nelem, size_t elsize)\n {\n - \tsize_t sz = st_mult(nelem, elsize);\n - \tvoid *p = reftable_malloc(sz);\n +-\tsize_t sz = st_mult(nelem, elsize);\n +-\tvoid *p = reftable_malloc(sz);\n +-\tmemset(p, 0, sz);\n ++\tvoid *p;\n ++\n ++\tif (nelem && elsize > SIZE_MAX / nelem)\n ++\t\treturn NULL;\n ++\n ++\tp = reftable_malloc(nelem * elsize);\n +\tif (!p)\n +\t\treturn NULL;\n - \tmemset(p, 0, sz);\n ++\n ++\tmemset(p, 0, nelem * elsize);\n \treturn p;\n }\n + \n 5: 1f98abe9812 = 5: ad028020df7 reftable/basics: handle allocation failures in `parse_names()`\n 6: fa32be01e50 = 6: df713fbe08c reftable/record: handle allocation failures on copy\n 7: 372e36f880e = 7: 870bb003c0a reftable/record: handle allocation failures when decoding records\n 8: 7404d648a9d = 8: 1d47e425009 reftable/writer: handle allocation failures in `writer_index_hash()`\n 9: d6afb30221e = 9: caa71f0a775 reftable/writer: handle allocation failures in `reftable_new_writer()`\n10: 349fd785a81 = 10: a84e9cadae4 reftable/merged: handle allocation failures in `merged_table_init_iter()`\n11: c4985e64ce2 = 11: 20d38330141 reftable/reader: handle allocation failures for unindexed reader\n12: 94c85ffd5ec = 12: e35c3a705d5 reftable/reader: handle allocation failures in `reader_init_iter()`\n13: 6ef25b6c655 = 13: ca3b57f151e reftable/stack: handle allocation failures on reload\n14: f5583cc2300 = 14: 7377421a632 reftable/stack: handle allocation failures in `reftable_new_stack()`\n15: 3393a59ce42 = 15: 244e8667c5d reftable/stack: handle allocation failures in `stack_compact_range()`\n16: 2c19f258fe7 = 16: 99f4868c38f reftable/stack: handle allocation failures in auto compaction\n17: 32fead57de9 ! 17: 271839a6260 reftable/iter: handle allocation failures when creating indexed table iter\n @@ reftable/iter.c: int new_indexed_table_ref_iter(struct indexed_table_ref_iter **\n \titr->offset_len = offset_len;\n \n \terr = indexed_table_ref_iter_next_block(itr);\n --\tif (err < 0) {\n +\tif (err < 0)\n +\t\tgoto out;\n +\n @@ reftable/iter.c: int new_indexed_table_ref_iter(struct indexed_table_ref_iter **\n +\terr = 0;\n +\n +out:\n -+\tif (err < 0)\n + \tif (err < 0) {\n ++\t\t*dest = NULL;\n \t\treftable_free(itr);\n -\t} else {\n -\t\t*dest = itr;\n --\t}\n + \t}\n \treturn err;\n }\n - \n \n ## reftable/iter.h ##\n @@ reftable/iter.h: void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n18: 4d672ab9b15 = 18: c41808e9d75 reftable/blocksource: handle allocation failures\n19: 4fa47a559b2 = 19: 9348b0a4575 reftable/block: handle allocation failures\n20: a569c39fd8e = 20: 445daf9464d reftable/pq: handle allocation failures when adding entries\n21: 99bca688f2a = 21: c31f9e53f52 reftable/tree: handle allocation failures\n22: 02073dafb96 = 22: d0fe9993716 reftable: handle trivial allocation failures\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503308","messageId":"8c99ecc3255b1b10cdec047da01804dc23a7d287.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 01/22] reftable/error: introduce out-of-memory error code","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:31:52Z","receivedAt":"2024-09-24T06:31:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library does not use the same memory allocation functions\nas the rest of the Git codebase. Instead, as the reftable library is\nsupposed to be usable as a standalone library without Git, it provides a\nset of pluggable memory allocators.\n\nCompared to `xmalloc()` and friends these allocators are _not_ expected\nto die when an allocation fails. This design choice is concious, as a\nlibrary should leave it to its caller to handle any kind of error. While\nit is very likely that the caller cannot really do much in the case of\nan out-of-memory situation anyway, we are not the ones to make that\ndecision.\n\nCuriously though, we never handle allocation errors even though memory\nallocation functions are allowed to fail. And as we do not plug in Git's\nmemory allocator via `reftable_set_alloc()` either the consequence is\nthat we'd instead segfault as soon as we run out of memory.\n\nWhile the easy fix would be to wire up `xmalloc()` and friends, it\nwould only fix the usage of the reftable library in Git itself. Other\nusers like libgit2, which is about to revive its efforts to land a\nbackend for reftables, wouldn't be able to benefit from this solution.\n\nInstead, we are about to do it the hard way: adapt all allocation sites\nto perform error checking. Introduce a new error code for out-of-memory\nerrors that we will wire up in subsequent steps.\n\nThis commit also serves as the motivator for all the remaining steps in\nthis series such that we do not have to repeat the same arguments in\nevery single subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/error.c | 2 ++\n reftable/reftable-error.h | 3 +++\n 2 files changed, 5 insertions(+)\n\ndiff --git a/reftable/error.c b/reftable/error.c\nindex a25f28a43eb..660d0296170 100644\n--- a/reftable/error.c\n+++ b/reftable/error.c\n@@ -35,6 +35,8 @@ const char *reftable_error_str(int err)\n \t\treturn \"entry too large\";\n \tcase REFTABLE_OUTDATED_ERROR:\n \t\treturn \"data concurrently modified\";\n+\tcase REFTABLE_OUT_OF_MEMORY_ERROR:\n+\t\treturn \"out of memory\";\n \tcase -1:\n \t\treturn \"general error\";\n \tdefault:\ndiff --git a/reftable/reftable-error.h b/reftable/reftable-error.h\nindex 6368cd9ed9d..f4048265629 100644\n--- a/reftable/reftable-error.h\n+++ b/reftable/reftable-error.h\n@@ -57,6 +57,9 @@ enum reftable_error {\n \n \t/* Trying to write out-of-date data. */\n \tREFTABLE_OUTDATED_ERROR = -12,\n+\n+\t/* An allocation has failed due to an out-of-memory situation. */\n+\tREFTABLE_OUT_OF_MEMORY_ERROR = -13,\n };\n \n /* convert the numeric error code to a string. The string should not be\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503309","messageId":"4dcdf1d48ecf9a05cfa643d5a6165240898811a7.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 02/22] reftable/basics: merge \"publicbasics\" into \"basics\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:31:54Z","receivedAt":"2024-09-24T06:31:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The split between \"basics\" and \"publicbasics\" is somewhat arbitrary and\nnot in line with how we typically structure code in the reftable\nlibrary. While we do indeed split up headers into a public and internal\npart, we don't do that for the compilation unit itself. Furthermore, the\ndeclarations for \"publicbasics.c\" are in \"reftable-malloc.h\", which\nisn't in line with our naming schema, either.\n\nFix these inconsistencies by:\n\n - Merging \"publicbasics.c\" into \"basics.c\".\n\n - Renaming \"reftable-malloc.h\" to \"reftable-basics.h\" as the public\n header.\n\nSigned-off-by: Patrick Steinhardt \n---\n Makefile | 1 -\n reftable/basics.c | 55 +++++++++++++++++++++++++++++++\n reftable/basics.h | 3 ++\n reftable/publicbasics.c | 66 --------------------------------------\n reftable/reftable-basics.h | 18 +++++++++++\n reftable/reftable-malloc.h | 18 -----------\n 6 files changed, 76 insertions(+), 85 deletions(-)\n delete mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-basics.h\n delete mode 100644 reftable/reftable-malloc.h\n\ndiff --git a/Makefile b/Makefile\nindex e3abf0ba831..39b10923d49 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2683,7 +2683,6 @@ REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n-REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/merged.o\n REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 0058619ca67..cf072935c88 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -7,6 +7,49 @@ license that can be found in the LICENSE file or at\n */\n \n #include \"basics.h\"\n+#include \"reftable-basics.h\"\n+\n+static void *(*reftable_malloc_ptr)(size_t sz);\n+static void *(*reftable_realloc_ptr)(void *, size_t);\n+static void (*reftable_free_ptr)(void *);\n+\n+void *reftable_malloc(size_t sz)\n+{\n+\tif (reftable_malloc_ptr)\n+\t\treturn (*reftable_malloc_ptr)(sz);\n+\treturn malloc(sz);\n+}\n+\n+void *reftable_realloc(void *p, size_t sz)\n+{\n+\tif (reftable_realloc_ptr)\n+\t\treturn (*reftable_realloc_ptr)(p, sz);\n+\treturn realloc(p, sz);\n+}\n+\n+void reftable_free(void *p)\n+{\n+\tif (reftable_free_ptr)\n+\t\treftable_free_ptr(p);\n+\telse\n+\t\tfree(p);\n+}\n+\n+void *reftable_calloc(size_t nelem, size_t elsize)\n+{\n+\tsize_t sz = st_mult(nelem, elsize);\n+\tvoid *p = reftable_malloc(sz);\n+\tmemset(p, 0, sz);\n+\treturn p;\n+}\n+\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n+{\n+\treftable_malloc_ptr = malloc;\n+\treftable_realloc_ptr = realloc;\n+\treftable_free_ptr = free;\n+}\n \n void put_be24(uint8_t *out, uint32_t i)\n {\n@@ -121,3 +164,15 @@ int common_prefix_size(struct strbuf *a, struct strbuf *b)\n \n \treturn p;\n }\n+\n+int hash_size(uint32_t id)\n+{\n+\tswitch (id) {\n+\tcase 0:\n+\tcase GIT_SHA1_FORMAT_ID:\n+\t\treturn GIT_SHA1_RAWSZ;\n+\tcase GIT_SHA256_FORMAT_ID:\n+\t\treturn GIT_SHA256_RAWSZ;\n+\t}\n+\tabort();\n+}\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex c8fec68d4e8..4e2e76014aa 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -14,6 +14,7 @@ license that can be found in the LICENSE file or at\n */\n \n #include \"system.h\"\n+#include \"reftable-basics.h\"\n \n /* Bigendian en/decoding of integers */\n \n@@ -71,4 +72,6 @@ void *reftable_calloc(size_t nelem, size_t elsize);\n struct strbuf;\n int common_prefix_size(struct strbuf *a, struct strbuf *b);\n \n+int hash_size(uint32_t id);\n+\n #endif\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\ndeleted file mode 100644\nindex 44b84a125e4..00000000000\n--- a/reftable/publicbasics.c\n+++ /dev/null\n@@ -1,66 +0,0 @@\n-/*\n-Copyright 2020 Google LLC\n-\n-Use of this source code is governed by a BSD-style\n-license that can be found in the LICENSE file or at\n-https://developers.google.com/open-source/licenses/bsd\n-*/\n-\n-#include \"system.h\"\n-#include \"reftable-malloc.h\"\n-\n-#include \"basics.h\"\n-\n-static void *(*reftable_malloc_ptr)(size_t sz);\n-static void *(*reftable_realloc_ptr)(void *, size_t);\n-static void (*reftable_free_ptr)(void *);\n-\n-void *reftable_malloc(size_t sz)\n-{\n-\tif (reftable_malloc_ptr)\n-\t\treturn (*reftable_malloc_ptr)(sz);\n-\treturn malloc(sz);\n-}\n-\n-void *reftable_realloc(void *p, size_t sz)\n-{\n-\tif (reftable_realloc_ptr)\n-\t\treturn (*reftable_realloc_ptr)(p, sz);\n-\treturn realloc(p, sz);\n-}\n-\n-void reftable_free(void *p)\n-{\n-\tif (reftable_free_ptr)\n-\t\treftable_free_ptr(p);\n-\telse\n-\t\tfree(p);\n-}\n-\n-void *reftable_calloc(size_t nelem, size_t elsize)\n-{\n-\tsize_t sz = st_mult(nelem, elsize);\n-\tvoid *p = reftable_malloc(sz);\n-\tmemset(p, 0, sz);\n-\treturn p;\n-}\n-\n-void reftable_set_alloc(void *(*malloc)(size_t),\n-\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n-{\n-\treftable_malloc_ptr = malloc;\n-\treftable_realloc_ptr = realloc;\n-\treftable_free_ptr = free;\n-}\n-\n-int hash_size(uint32_t id)\n-{\n-\tswitch (id) {\n-\tcase 0:\n-\tcase GIT_SHA1_FORMAT_ID:\n-\t\treturn GIT_SHA1_RAWSZ;\n-\tcase GIT_SHA256_FORMAT_ID:\n-\t\treturn GIT_SHA256_RAWSZ;\n-\t}\n-\tabort();\n-}\ndiff --git a/reftable/reftable-basics.h b/reftable/reftable-basics.h\nnew file mode 100644\nindex 00000000000..6e8e636b716\n--- /dev/null\n+++ b/reftable/reftable-basics.h\n@@ -0,0 +1,18 @@\n+/*\n+ * Copyright 2020 Google LLC\n+ *\n+ * Use of this source code is governed by a BSD-style\n+ * license that can be found in the LICENSE file or at\n+ * https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_BASICS_H\n+#define REFTABLE_BASICS_H\n+\n+#include \n+\n+/* Overrides the functions to use for memory management. */\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n+\n+#endif\ndiff --git a/reftable/reftable-malloc.h b/reftable/reftable-malloc.h\ndeleted file mode 100644\nindex 5f2185f1f34..00000000000\n--- a/reftable/reftable-malloc.h\n+++ /dev/null\n@@ -1,18 +0,0 @@\n-/*\n-Copyright 2020 Google LLC\n-\n-Use of this source code is governed by a BSD-style\n-license that can be found in the LICENSE file or at\n-https://developers.google.com/open-source/licenses/bsd\n-*/\n-\n-#ifndef REFTABLE_H\n-#define REFTABLE_H\n-\n-#include \n-\n-/* Overrides the functions to use for memory management. */\n-void reftable_set_alloc(void *(*malloc)(size_t),\n-\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n-\n-#endif\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503310","messageId":"21fa9b15d96809728455943877c1b32781364041.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 03/22] reftable: introduce `reftable_strdup()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:31:57Z","receivedAt":"2024-09-24T06:32:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library provides the ability to swap out allocators. There\nis a gap here though, because we continue to use `xstrdup()` even in the\ncase where all the other allocators have been swapped out.\n\nIntroduce `reftable_strdup()` that uses `reftable_malloc()` to do the\nallocation.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 10 ++++++++++\n reftable/basics.h | 1 +\n 2 files changed, 11 insertions(+)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex cf072935c88..4adc98cf5de 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -43,6 +43,16 @@ void *reftable_calloc(size_t nelem, size_t elsize)\n \treturn p;\n }\n \n+char *reftable_strdup(const char *str)\n+{\n+\tsize_t len = strlen(str);\n+\tchar *result = reftable_malloc(len + 1);\n+\tif (!result)\n+\t\treturn NULL;\n+\tmemcpy(result, str, len + 1);\n+\treturn result;\n+}\n+\n void reftable_set_alloc(void *(*malloc)(size_t),\n \t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n {\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 4e2e76014aa..f107e148605 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -54,6 +54,7 @@ void *reftable_malloc(size_t sz);\n void *reftable_realloc(void *p, size_t sz);\n void reftable_free(void *p);\n void *reftable_calloc(size_t nelem, size_t elsize);\n+char *reftable_strdup(const char *str);\n \n #define REFTABLE_ALLOC_ARRAY(x, alloc) (x) = reftable_malloc(st_mult(sizeof(*(x)), (alloc)))\n #define REFTABLE_CALLOC_ARRAY(x, alloc) (x) = reftable_calloc((alloc), sizeof(*(x)))\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503311","messageId":"f6ad92ffd01c442dacd3ac6aa448891028636636.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 04/22] reftable/basics: handle allocation failures in `reftable_calloc()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:32:00Z","receivedAt":"2024-09-24T06:32:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_calloc()`.\n\nWhile at it, remove our use of `st_mult()` that would cause us to die on\nan overflow. From the caller's point of view there is not much of a\ndifference between arguments that are too large to be multiplied and a\nrequest that is too big to handle by the allocator: in both cases the\nallocation cannot be fulfilled. And in neither of these cases do we want\nthe reftable library to die.\n\nWhile we could use `unsigned_mult_overflows()` to handle the overflow\ngracefully, we instead open-code it to further our goal of converting\nthe reftable codebase to become a standalone library that can be reused\nby external projects.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 13 ++++++++++---\n 1 file changed, 10 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 4adc98cf5de..3350bbffa23 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -37,9 +37,16 @@ void reftable_free(void *p)\n \n void *reftable_calloc(size_t nelem, size_t elsize)\n {\n-\tsize_t sz = st_mult(nelem, elsize);\n-\tvoid *p = reftable_malloc(sz);\n-\tmemset(p, 0, sz);\n+\tvoid *p;\n+\n+\tif (nelem && elsize > SIZE_MAX / nelem)\n+\t\treturn NULL;\n+\n+\tp = reftable_malloc(nelem * elsize);\n+\tif (!p)\n+\t\treturn NULL;\n+\n+\tmemset(p, 0, nelem * elsize);\n \treturn p;\n }\n \n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503312","messageId":"ad028020df717a57879cf3fdf5a3648bfebd2c78.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 05/22] reftable/basics: handle allocation failures in `parse_names()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:32:03Z","receivedAt":"2024-09-24T06:32:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `parse_names()` by returning `NULL` in\ncase any allocation fails. While at it, refactor the function to return\nthe array directly instead of assigning it to an out-pointer.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 20 ++++++++++++++++----\n reftable/basics.h | 9 ++++++---\n reftable/stack.c | 6 +++++-\n t/unit-tests/t-reftable-basics.c | 11 ++++++-----\n 4 files changed, 33 insertions(+), 13 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 3350bbffa23..82bfb807631 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -135,14 +135,14 @@ size_t names_length(const char **names)\n \treturn p - names;\n }\n \n-void parse_names(char *buf, int size, char ***namesp)\n+char **parse_names(char *buf, int size)\n {\n \tchar **names = NULL;\n \tsize_t names_cap = 0;\n \tsize_t names_len = 0;\n-\n \tchar *p = buf;\n \tchar *end = buf + size;\n+\n \twhile (p < end) {\n \t\tchar *next = strchr(p, '\\n');\n \t\tif (next && next < end) {\n@@ -152,14 +152,26 @@ void parse_names(char *buf, int size, char ***namesp)\n \t\t}\n \t\tif (p < next) {\n \t\t\tREFTABLE_ALLOC_GROW(names, names_len + 1, names_cap);\n-\t\t\tnames[names_len++] = xstrdup(p);\n+\t\t\tif (!names)\n+\t\t\t\tgoto err;\n+\n+\t\t\tnames[names_len] = strdup(p);\n+\t\t\tif (!names[names_len++])\n+\t\t\t\tgoto err;\n \t\t}\n \t\tp = next + 1;\n \t}\n \n \tREFTABLE_REALLOC_ARRAY(names, names_len + 1);\n \tnames[names_len] = NULL;\n-\t*namesp = names;\n+\n+\treturn names;\n+\n+err:\n+\tfor (size_t i = 0; i < names_len; i++)\n+\t\tfree(names[i]);\n+\tfree(names);\n+\treturn NULL;\n }\n \n int names_equal(const char **a, const char **b)\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex f107e148605..69adeab2e4b 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -38,9 +38,12 @@ size_t binsearch(size_t sz, int (*f)(size_t k, void *args), void *args);\n */\n void free_names(char **a);\n \n-/* parse a newline separated list of names. `size` is the length of the buffer,\n- * without terminating '\\0'. Empty names are discarded. */\n-void parse_names(char *buf, int size, char ***namesp);\n+/*\n+ * Parse a newline separated list of names. `size` is the length of the buffer,\n+ * without terminating '\\0'. Empty names are discarded. Returns a `NULL`\n+ * pointer when allocations fail.\n+ */\n+char **parse_names(char *buf, int size);\n \n /* compares two NULL-terminated arrays of strings. */\n int names_equal(const char **a, const char **b);\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex ce0a35216ba..498fae846d7 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -108,7 +108,11 @@ static int fd_read_lines(int fd, char ***namesp)\n \t}\n \tbuf[size] = 0;\n \n-\tparse_names(buf, size, namesp);\n+\t*namesp = parse_names(buf, size);\n+\tif (!*namesp) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \n done:\n \treftable_free(buf);\ndiff --git a/t/unit-tests/t-reftable-basics.c b/t/unit-tests/t-reftable-basics.c\nindex e5556ebf527..1fa77b6faff 100644\n--- a/t/unit-tests/t-reftable-basics.c\n+++ b/t/unit-tests/t-reftable-basics.c\n@@ -72,13 +72,14 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \tif_test (\"parse_names works for basic input\") {\n \t\tchar in1[] = \"line\\n\";\n \t\tchar in2[] = \"a\\nb\\nc\";\n-\t\tchar **out = NULL;\n-\t\tparse_names(in1, strlen(in1), &out);\n+\t\tchar **out = parse_names(in1, strlen(in1));\n+\t\tcheck(out != NULL);\n \t\tcheck_str(out[0], \"line\");\n \t\tcheck(!out[1]);\n \t\tfree_names(out);\n \n-\t\tparse_names(in2, strlen(in2), &out);\n+\t\tout = parse_names(in2, strlen(in2));\n+\t\tcheck(out != NULL);\n \t\tcheck_str(out[0], \"a\");\n \t\tcheck_str(out[1], \"b\");\n \t\tcheck_str(out[2], \"c\");\n@@ -88,8 +89,8 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \n \tif_test (\"parse_names drops empty string\") {\n \t\tchar in[] = \"a\\n\\nb\\n\";\n-\t\tchar **out = NULL;\n-\t\tparse_names(in, strlen(in), &out);\n+\t\tchar **out = parse_names(in, strlen(in));\n+\t\tcheck(out != NULL);\n \t\tcheck_str(out[0], \"a\");\n \t\t/* simply '\\n' should be dropped as empty string */\n \t\tcheck_str(out[1], \"b\");\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503313","messageId":"df713fbe08cdc4bfa0060568d875606dab218fec.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 06/22] reftable/record: handle allocation failures on copy","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:32:09Z","receivedAt":"2024-09-24T06:32:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when copying records. While at it, convert\nfrom `xstrdup()` to `reftable_strdup()`. Adapt callsites to check for\nerror codes.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/record.c | 84 +++++++++++++++++++++++++++++++++--------------\n reftable/record.h | 6 ++--\n 2 files changed, 63 insertions(+), 27 deletions(-)\n\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 6b5a075b921..60fd33c9c94 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -215,13 +215,14 @@ static void reftable_ref_record_key(const void *r, struct strbuf *dest)\n \tstrbuf_addstr(dest, rec->refname);\n }\n \n-static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size)\n+static int reftable_ref_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n {\n \tstruct reftable_ref_record *ref = rec;\n \tconst struct reftable_ref_record *src = src_rec;\n \tchar *refname = NULL;\n \tsize_t refname_cap = 0;\n+\tint err;\n \n \tassert(hash_size > 0);\n \n@@ -236,6 +237,11 @@ static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n \n \t\tREFTABLE_ALLOC_GROW(ref->refname, refname_len + 1,\n \t\t\t\t ref->refname_cap);\n+\t\tif (!ref->refname) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n+\n \t\tmemcpy(ref->refname, src->refname, refname_len);\n \t\tref->refname[refname_len] = 0;\n \t}\n@@ -254,9 +260,17 @@ static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n \t\t src->value.val2.target_value, hash_size);\n \t\tbreak;\n \tcase REFTABLE_REF_SYMREF:\n-\t\tref->value.symref = xstrdup(src->value.symref);\n+\t\tref->value.symref = reftable_strdup(src->value.symref);\n+\t\tif (!ref->value.symref) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n \t\tbreak;\n \t}\n+\n+\terr = 0;\n+out:\n+\treturn err;\n }\n \n static void reftable_ref_record_release_void(void *rec)\n@@ -457,23 +471,28 @@ static void reftable_obj_record_release(void *rec)\n \tmemset(obj, 0, sizeof(struct reftable_obj_record));\n }\n \n-static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size UNUSED)\n+static int reftable_obj_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size UNUSED)\n {\n \tstruct reftable_obj_record *obj = rec;\n-\tconst struct reftable_obj_record *src =\n-\t\t(const struct reftable_obj_record *)src_rec;\n+\tconst struct reftable_obj_record *src = src_rec;\n \n \treftable_obj_record_release(obj);\n \n \tREFTABLE_ALLOC_ARRAY(obj->hash_prefix, src->hash_prefix_len);\n+\tif (!obj->hash_prefix)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tobj->hash_prefix_len = src->hash_prefix_len;\n \tif (src->hash_prefix_len)\n \t\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n \n \tREFTABLE_ALLOC_ARRAY(obj->offsets, src->offset_len);\n+\tif (!obj->offsets)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tobj->offset_len = src->offset_len;\n \tCOPY_ARRAY(obj->offsets, src->offsets, src->offset_len);\n+\n+\treturn 0;\n }\n \n static uint8_t reftable_obj_record_val_type(const void *rec)\n@@ -646,33 +665,44 @@ static void reftable_log_record_key(const void *r, struct strbuf *dest)\n \tstrbuf_add(dest, i64, sizeof(i64));\n }\n \n-static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size)\n+static int reftable_log_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n {\n \tstruct reftable_log_record *dst = rec;\n \tconst struct reftable_log_record *src =\n \t\t(const struct reftable_log_record *)src_rec;\n+\tint ret;\n \n \treftable_log_record_release(dst);\n \t*dst = *src;\n+\n \tif (dst->refname) {\n-\t\tdst->refname = xstrdup(dst->refname);\n+\t\tdst->refname = reftable_strdup(dst->refname);\n+\t\tif (!dst->refname) {\n+\t\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n \t}\n+\n \tswitch (dst->value_type) {\n \tcase REFTABLE_LOG_DELETION:\n \t\tbreak;\n \tcase REFTABLE_LOG_UPDATE:\n-\t\tif (dst->value.update.email) {\n+\t\tif (dst->value.update.email)\n \t\t\tdst->value.update.email =\n-\t\t\t\txstrdup(dst->value.update.email);\n-\t\t}\n-\t\tif (dst->value.update.name) {\n+\t\t\t\treftable_strdup(dst->value.update.email);\n+\t\tif (dst->value.update.name)\n \t\t\tdst->value.update.name =\n-\t\t\t\txstrdup(dst->value.update.name);\n-\t\t}\n-\t\tif (dst->value.update.message) {\n+\t\t\t\treftable_strdup(dst->value.update.name);\n+\t\tif (dst->value.update.message)\n \t\t\tdst->value.update.message =\n-\t\t\t\txstrdup(dst->value.update.message);\n+\t\t\t\treftable_strdup(dst->value.update.message);\n+\n+\t\tif (!dst->value.update.email ||\n+\t\t !dst->value.update.name ||\n+\t\t !dst->value.update.message) {\n+\t\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n \t\t}\n \n \t\tmemcpy(dst->value.update.new_hash,\n@@ -681,6 +711,10 @@ static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n \t\t src->value.update.old_hash, hash_size);\n \t\tbreak;\n \t}\n+\n+\tret = 0;\n+out:\n+\treturn ret;\n }\n \n static void reftable_log_record_release_void(void *rec)\n@@ -954,8 +988,8 @@ static void reftable_index_record_key(const void *r, struct strbuf *dest)\n \tstrbuf_addbuf(dest, &rec->last_key);\n }\n \n-static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size UNUSED)\n+static int reftable_index_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size UNUSED)\n {\n \tstruct reftable_index_record *dst = rec;\n \tconst struct reftable_index_record *src = src_rec;\n@@ -963,6 +997,8 @@ static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n \tstrbuf_reset(&dst->last_key);\n \tstrbuf_addbuf(&dst->last_key, &src->last_key);\n \tdst->offset = src->offset;\n+\n+\treturn 0;\n }\n \n static void reftable_index_record_release(void *rec)\n@@ -1054,14 +1090,14 @@ int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n \t\t\t\t\t\t dest, hash_size);\n }\n \n-void reftable_record_copy_from(struct reftable_record *rec,\n+int reftable_record_copy_from(struct reftable_record *rec,\n \t\t\t struct reftable_record *src, int hash_size)\n {\n \tassert(src->type == rec->type);\n \n-\treftable_record_vtable(rec)->copy_from(reftable_record_data(rec),\n-\t\t\t\t\t reftable_record_data(src),\n-\t\t\t\t\t hash_size);\n+\treturn reftable_record_vtable(rec)->copy_from(reftable_record_data(rec),\n+\t\t\t\t\t\t reftable_record_data(src),\n+\t\t\t\t\t\t hash_size);\n }\n \n uint8_t reftable_record_val_type(struct reftable_record *rec)\ndiff --git a/reftable/record.h b/reftable/record.h\nindex 5003bacdb0c..0f53ba54434 100644\n--- a/reftable/record.h\n+++ b/reftable/record.h\n@@ -44,7 +44,7 @@ struct reftable_record_vtable {\n \t/* The record type of ('r' for ref). */\n \tuint8_t type;\n \n-\tvoid (*copy_from)(void *dest, const void *src, int hash_size);\n+\tint (*copy_from)(void *dest, const void *src, int hash_size);\n \n \t/* a value of [0..7], indicating record subvariants (eg. ref vs. symref\n \t * vs ref deletion) */\n@@ -137,8 +137,8 @@ void reftable_record_init(struct reftable_record *rec, uint8_t typ);\n int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b);\n int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);\n void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);\n-void reftable_record_copy_from(struct reftable_record *rec,\n-\t\t\t struct reftable_record *src, int hash_size);\n+int reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size);\n uint8_t reftable_record_val_type(struct reftable_record *rec);\n int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n \t\t\t int hash_size);\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503314","messageId":"870bb003c0abd117dce2ae0fcf8c3010c8d1b334.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 07/22] reftable/record: handle allocation failures when decoding records","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:32:12Z","receivedAt":"2024-09-24T06:32:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when decoding records. While at it, fix some\nerror codes to be `REFTABLE_FORMAT_ERROR`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/record.c | 80 +++++++++++++++++++++++++++++++++++++----------\n 1 file changed, 63 insertions(+), 17 deletions(-)\n\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 60fd33c9c94..787e134c9a0 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -359,7 +359,7 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tuint64_t update_index = 0;\n \tconst char *refname = NULL;\n \tsize_t refname_cap = 0;\n-\tint n;\n+\tint n, err;\n \n \tassert(hash_size > 0);\n \n@@ -375,6 +375,10 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tSWAP(r->refname_cap, refname_cap);\n \n \tREFTABLE_ALLOC_GROW(r->refname, key.len + 1, r->refname_cap);\n+\tif (!r->refname) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \tmemcpy(r->refname, key.buf, key.len);\n \tr->refname[key.len] = 0;\n \n@@ -383,7 +387,8 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tswitch (val_type) {\n \tcase REFTABLE_REF_VAL1:\n \t\tif (in.len < hash_size) {\n-\t\t\treturn -1;\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n \t\t}\n \n \t\tmemcpy(r->value.val1, in.buf, hash_size);\n@@ -392,7 +397,8 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \n \tcase REFTABLE_REF_VAL2:\n \t\tif (in.len < 2 * hash_size) {\n-\t\t\treturn -1;\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n \t\t}\n \n \t\tmemcpy(r->value.val2.value, in.buf, hash_size);\n@@ -405,7 +411,8 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tcase REFTABLE_REF_SYMREF: {\n \t\tint n = decode_string(scratch, in);\n \t\tif (n < 0) {\n-\t\t\treturn -1;\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n \t\t}\n \t\tstring_view_consume(&in, n);\n \t\tr->value.symref = strbuf_detach(scratch, NULL);\n@@ -419,6 +426,9 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \t}\n \n \treturn start.len - in.len;\n+\n+done:\n+\treturn err;\n }\n \n static int reftable_ref_record_is_deletion_void(const void *p)\n@@ -552,6 +562,8 @@ static int reftable_obj_record_decode(void *rec, struct strbuf key,\n \treftable_obj_record_release(r);\n \n \tREFTABLE_ALLOC_ARRAY(r->hash_prefix, key.len);\n+\tif (!r->hash_prefix)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tmemcpy(r->hash_prefix, key.buf, key.len);\n \tr->hash_prefix_len = key.len;\n \n@@ -570,6 +582,8 @@ static int reftable_obj_record_decode(void *rec, struct strbuf key,\n \t\treturn start.len - in.len;\n \n \tREFTABLE_ALLOC_ARRAY(r->offsets, count);\n+\tif (!r->offsets)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tr->offset_len = count;\n \n \tn = get_var_int(&r->offsets[0], &in);\n@@ -801,12 +815,17 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \tstruct reftable_log_record *r = rec;\n \tuint64_t max = 0;\n \tuint64_t ts = 0;\n-\tint n;\n+\tint err, n;\n \n \tif (key.len <= 9 || key.buf[key.len - 9] != 0)\n \t\treturn REFTABLE_FORMAT_ERROR;\n \n \tREFTABLE_ALLOC_GROW(r->refname, key.len - 8, r->refname_cap);\n+\tif (!r->refname) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tmemcpy(r->refname, key.buf, key.len - 8);\n \tts = get_be64(key.buf + key.len - 8);\n \n@@ -829,8 +848,10 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \tif (val_type == REFTABLE_LOG_DELETION)\n \t\treturn 0;\n \n-\tif (in.len < 2 * hash_size)\n-\t\treturn REFTABLE_FORMAT_ERROR;\n+\tif (in.len < 2 * hash_size) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n \n \tmemcpy(r->value.update.old_hash, in.buf, hash_size);\n \tmemcpy(r->value.update.new_hash, in.buf + hash_size, hash_size);\n@@ -838,8 +859,10 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \tstring_view_consume(&in, 2 * hash_size);\n \n \tn = decode_string(scratch, in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \n \t/*\n@@ -850,52 +873,75 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \t */\n \tif (!r->value.update.name ||\n \t strcmp(r->value.update.name, scratch->buf)) {\n-\t\tr->value.update.name =\n-\t\t\treftable_realloc(r->value.update.name, scratch->len + 1);\n+\t\tchar *name = reftable_realloc(r->value.update.name, scratch->len + 1);\n+\t\tif (!name) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tr->value.update.name = name;\n \t\tmemcpy(r->value.update.name, scratch->buf, scratch->len);\n \t\tr->value.update.name[scratch->len] = 0;\n \t}\n \n \tn = decode_string(scratch, in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \n \t/* Same as above, but for the reflog email. */\n \tif (!r->value.update.email ||\n \t strcmp(r->value.update.email, scratch->buf)) {\n-\t\tr->value.update.email =\n-\t\t\treftable_realloc(r->value.update.email, scratch->len + 1);\n+\t\tchar *email = reftable_realloc(r->value.update.email, scratch->len + 1);\n+\t\tif (!email) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tr->value.update.email = email;\n \t\tmemcpy(r->value.update.email, scratch->buf, scratch->len);\n \t\tr->value.update.email[scratch->len] = 0;\n \t}\n \n \tts = 0;\n \tn = get_var_int(&ts, &in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \tr->value.update.time = ts;\n-\tif (in.len < 2)\n+\tif (in.len < 2) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \n \tr->value.update.tz_offset = get_be16(in.buf);\n \tstring_view_consume(&in, 2);\n \n \tn = decode_string(scratch, in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \n \tREFTABLE_ALLOC_GROW(r->value.update.message, scratch->len + 1,\n \t\t\t r->value.update.message_cap);\n+\tif (!r->value.update.message) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tmemcpy(r->value.update.message, scratch->buf, scratch->len);\n \tr->value.update.message[scratch->len] = 0;\n \n \treturn start.len - in.len;\n \n done:\n-\treturn REFTABLE_FORMAT_ERROR;\n+\treturn err;\n }\n \n static int null_streq(const char *a, const char *b)\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503315","messageId":"1d47e4250096b4a2e5bd07c357b096d09e74ca07.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 08/22] reftable/writer: handle allocation failures in `writer_index_hash()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:32:15Z","receivedAt":"2024-09-24T06:32:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation errors in `writer_index_hash()`. Adjust its only\ncaller in `reftable_writer_add_ref()` accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/writer.c | 61 +++++++++++++++++++++++++++++------------------\n 1 file changed, 38 insertions(+), 23 deletions(-)\n\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 9d5e6072bc6..ed61aaf59c5 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -186,18 +186,22 @@ static int obj_index_tree_node_compare(const void *a, const void *b)\n \t\t\t &((const struct obj_index_tree_node *)b)->hash);\n }\n \n-static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n+static int writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n {\n \tuint64_t off = w->next;\n-\n \tstruct obj_index_tree_node want = { .hash = *hash };\n+\tstruct obj_index_tree_node *key;\n+\tstruct tree_node *node;\n \n-\tstruct tree_node *node = tree_search(&want, &w->obj_index_tree,\n-\t\t\t\t\t &obj_index_tree_node_compare, 0);\n-\tstruct obj_index_tree_node *key = NULL;\n+\tnode = tree_search(&want, &w->obj_index_tree,\n+\t\t\t &obj_index_tree_node_compare, 0);\n \tif (!node) {\n \t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n-\t\tkey = reftable_malloc(sizeof(struct obj_index_tree_node));\n+\n+\t\tkey = reftable_malloc(sizeof(*key));\n+\t\tif (!key)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \t\t*key = empty;\n \n \t\tstrbuf_reset(&key->hash);\n@@ -208,12 +212,15 @@ static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n \t\tkey = node->key;\n \t}\n \n-\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off) {\n-\t\treturn;\n-\t}\n+\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off)\n+\t\treturn 0;\n \n \tREFTABLE_ALLOC_GROW(key->offsets, key->offset_len + 1, key->offset_cap);\n+\tif (!key->offsets)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tkey->offsets[key->offset_len++] = off;\n+\n+\treturn 0;\n }\n \n static int writer_add_record(struct reftable_writer *w,\n@@ -284,11 +291,11 @@ int reftable_writer_add_ref(struct reftable_writer *w,\n \t\t\t.ref = *ref\n \t\t},\n \t};\n-\tint err = 0;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint err;\n \n-\tif (!ref->refname)\n-\t\treturn REFTABLE_API_ERROR;\n-\tif (ref->update_index < w->min_update_index ||\n+\tif (!ref->refname ||\n+\t ref->update_index < w->min_update_index ||\n \t ref->update_index > w->max_update_index)\n \t\treturn REFTABLE_API_ERROR;\n \n@@ -296,24 +303,32 @@ int reftable_writer_add_ref(struct reftable_writer *w,\n \n \terr = writer_add_record(w, &rec);\n \tif (err < 0)\n-\t\treturn err;\n+\t\tgoto out;\n \n \tif (!w->opts.skip_index_objects && reftable_ref_record_val1(ref)) {\n-\t\tstruct strbuf h = STRBUF_INIT;\n-\t\tstrbuf_add(&h, (char *)reftable_ref_record_val1(ref),\n+\t\tstrbuf_add(&buf, (char *)reftable_ref_record_val1(ref),\n \t\t\t hash_size(w->opts.hash_id));\n-\t\twriter_index_hash(w, &h);\n-\t\tstrbuf_release(&h);\n+\n+\t\terr = writer_index_hash(w, &buf);\n+\t\tif (err < 0)\n+\t\t\tgoto out;\n \t}\n \n \tif (!w->opts.skip_index_objects && reftable_ref_record_val2(ref)) {\n-\t\tstruct strbuf h = STRBUF_INIT;\n-\t\tstrbuf_add(&h, reftable_ref_record_val2(ref),\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_add(&buf, reftable_ref_record_val2(ref),\n \t\t\t hash_size(w->opts.hash_id));\n-\t\twriter_index_hash(w, &h);\n-\t\tstrbuf_release(&h);\n+\n+\t\terr = writer_index_hash(w, &buf);\n+\t\tif (err < 0)\n+\t\t\tgoto out;\n \t}\n-\treturn 0;\n+\n+\terr = 0;\n+\n+out:\n+\tstrbuf_release(&buf);\n+\treturn err;\n }\n \n int reftable_writer_add_refs(struct reftable_writer *w,\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503316","messageId":"caa71f0a775698872ae855381851121f1796f290.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 09/22] reftable/writer: handle allocation failures in `reftable_new_writer()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:32:18Z","receivedAt":"2024-09-24T06:32:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_new_writer()`. Adapt the\nfunction to return an error code to return such failures. While at it,\nrename it to match our code style as we have to touch up every callsite\nanyway.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/reftable-writer.h | 12 +++++++-----\n reftable/stack.c | 14 ++++++++++----\n reftable/writer.c | 22 ++++++++++++++++------\n t/unit-tests/lib-reftable.c | 8 +++++---\n 4 files changed, 38 insertions(+), 18 deletions(-)\n\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 189b1f4144f..43623dc7c30 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -90,11 +90,13 @@ struct reftable_stats {\n \tint object_id_len;\n };\n \n-/* reftable_new_writer creates a new writer */\n-struct reftable_writer *\n-reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n-\t\t int (*flush_func)(void *),\n-\t\t void *writer_arg, const struct reftable_write_options *opts);\n+struct reftable_writer;\n+\n+/* Create a new writer. */\n+int reftable_writer_new(struct reftable_writer **out,\n+\t\t\tssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t\tint (*flush_func)(void *),\n+\t\t\tvoid *writer_arg, const struct reftable_write_options *opts);\n \n /* Set the range of update indices for the records we will add. When writing a\n table into a stack, the min should be at least\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 498fae846d7..ea21ca6e5f5 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -808,8 +808,11 @@ int reftable_addition_add(struct reftable_addition *add,\n \t}\n \ttab_fd = get_tempfile_fd(tab_file);\n \n-\twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, &tab_fd,\n-\t\t\t\t &add->stack->opts);\n+\terr = reftable_writer_new(&wr, reftable_fd_write, reftable_fd_flush,\n+\t\t\t\t &tab_fd, &add->stack->opts);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = write_table(wr, arg);\n \tif (err < 0)\n \t\tgoto done;\n@@ -898,8 +901,11 @@ static int stack_compact_locked(struct reftable_stack *st,\n \t\tgoto done;\n \t}\n \n-\twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush,\n-\t\t\t\t &tab_fd, &st->opts);\n+\terr = reftable_writer_new(&wr, reftable_fd_write, reftable_fd_flush,\n+\t\t\t\t &tab_fd, &st->opts);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = stack_write_compact(st, wr, first, last, config);\n \tif (err < 0)\n \t\tgoto done;\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex ed61aaf59c5..54ec822e1cd 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -117,13 +117,17 @@ static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n \tw->block_writer->restart_interval = w->opts.restart_interval;\n }\n \n-struct reftable_writer *\n-reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n-\t\t int (*flush_func)(void *),\n-\t\t void *writer_arg, const struct reftable_write_options *_opts)\n+int reftable_writer_new(struct reftable_writer **out,\n+\t\t\tssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t\tint (*flush_func)(void *),\n+\t\t\tvoid *writer_arg, const struct reftable_write_options *_opts)\n {\n-\tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n \tstruct reftable_write_options opts = {0};\n+\tstruct reftable_writer *wp;\n+\n+\twp = reftable_calloc(1, sizeof(*wp));\n+\tif (!wp)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \n \tif (_opts)\n \t\topts = *_opts;\n@@ -134,13 +138,19 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \tstrbuf_init(&wp->block_writer_data.last_key, 0);\n \tstrbuf_init(&wp->last_key, 0);\n \tREFTABLE_CALLOC_ARRAY(wp->block, opts.block_size);\n+\tif (!wp->block) {\n+\t\tfree(wp);\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t}\n \twp->write = writer_func;\n \twp->write_arg = writer_arg;\n \twp->opts = opts;\n \twp->flush = flush_func;\n \twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n \n-\treturn wp;\n+\t*out = wp;\n+\n+\treturn 0;\n }\n \n void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\ndiff --git a/t/unit-tests/lib-reftable.c b/t/unit-tests/lib-reftable.c\nindex ab1fa44a282..54c26c43e77 100644\n--- a/t/unit-tests/lib-reftable.c\n+++ b/t/unit-tests/lib-reftable.c\n@@ -22,9 +22,11 @@ static int strbuf_writer_flush(void *arg UNUSED)\n struct reftable_writer *t_reftable_strbuf_writer(struct strbuf *buf,\n \t\t\t\t\t\t struct reftable_write_options *opts)\n {\n-\treturn reftable_new_writer(&strbuf_writer_write,\n-\t\t\t\t &strbuf_writer_flush,\n-\t\t\t\t buf, opts);\n+\tstruct reftable_writer *writer;\n+\tint ret = reftable_writer_new(&writer, &strbuf_writer_write, &strbuf_writer_flush,\n+\t\t\t\t buf, opts);\n+\tcheck(!ret);\n+\treturn writer;\n }\n \n void t_reftable_write_to_buf(struct strbuf *buf,\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503317","messageId":"a84e9cadae486557caa92a4962ad880f0cf58420.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 10/22] reftable/merged: handle allocation failures in `merged_table_init_iter()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:32:21Z","receivedAt":"2024-09-24T06:32:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `merged_table_init_iter()`. While at it,\nmerge `merged_iter_init()` into the function. It only has a single\ncaller and merging them makes it easier to handle allocation failures\nconsistently.\n\nThis change also requires us to adapt `reftable_stack_init_*_iterator()`\nto bubble up the new error codes of `merged_table_iter_init()`. Adapt\ncallsites accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 39 +++++++++++++----\n reftable/merged.c | 74 ++++++++++++++++++++------------\n reftable/merged.h | 6 +--\n reftable/reftable-merged.h | 8 ++--\n reftable/reftable-stack.h | 8 ++--\n reftable/stack.c | 34 ++++++++++-----\n t/helper/test-reftable.c | 10 ++++-\n t/unit-tests/t-reftable-merged.c | 12 ++++--\n t/unit-tests/t-reftable-stack.c | 4 +-\n 9 files changed, 131 insertions(+), 64 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 5c241097a4e..9c08c46c85e 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1299,7 +1299,9 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t\t\tstruct reftable_log_record log = {0};\n \t\t\tstruct reftable_iterator it = {0};\n \n-\t\t\treftable_stack_init_log_iterator(arg->stack, &it);\n+\t\t\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto done;\n \n \t\t\t/*\n \t\t\t * When deleting refs we also delete all reflog entries\n@@ -1669,7 +1671,10 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \t * copy over all log entries from the old reflog. Last but not least,\n \t * when renaming we also have to delete all the old reflog entries.\n \t */\n-\treftable_stack_init_log_iterator(arg->stack, &it);\n+\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, arg->oldname);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1890,7 +1895,10 @@ static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftabl\n \tif (ret < 0)\n \t\tgoto done;\n \n-\treftable_stack_init_log_iterator(stack, &iter->iter);\n+\tret = reftable_stack_init_log_iterator(stack, &iter->iter);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&iter->iter, \"\");\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1957,7 +1965,10 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, refname);\n \twhile (!ret) {\n \t\tret = reftable_iterator_next_log(&it, &log);\n@@ -1973,6 +1984,7 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n \t\t\tbreak;\n \t}\n \n+done:\n \treftable_log_record_release(&log);\n \treftable_iterator_destroy(&it);\n \treturn ret;\n@@ -1994,7 +2006,10 @@ static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store,\n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, refname);\n \twhile (!ret) {\n \t\tstruct reftable_log_record log = {0};\n@@ -2044,7 +2059,10 @@ static int reftable_be_reflog_exists(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, refname);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -2150,7 +2168,9 @@ static int write_reflog_delete_table(struct reftable_writer *writer, void *cb_da\n \n \treftable_writer_set_limits(writer, ts, ts);\n \n-\treftable_stack_init_log_iterator(arg->stack, &it);\n+\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\tif (ret < 0)\n+\t\tgoto out;\n \n \t/*\n \t * In order to delete a table we need to delete all reflog entries one\n@@ -2174,6 +2194,7 @@ static int write_reflog_delete_table(struct reftable_writer *writer, void *cb_da\n \t\tret = reftable_writer_add_log(writer, &tombstone);\n \t}\n \n+out:\n \treftable_log_record_release(&log);\n \treftable_iterator_destroy(&it);\n \treturn ret;\n@@ -2312,7 +2333,9 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n \n \tret = reftable_iterator_seek_log(&it, refname);\n \tif (ret < 0)\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 128a810c55d..5a8ea8ae779 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -30,22 +30,6 @@ struct merged_iter {\n \tssize_t advance_index;\n };\n \n-static void merged_iter_init(struct merged_iter *mi,\n-\t\t\t struct reftable_merged_table *mt,\n-\t\t\t uint8_t typ)\n-{\n-\tmemset(mi, 0, sizeof(*mi));\n-\tmi->advance_index = -1;\n-\tmi->suppress_deletions = mt->suppress_deletions;\n-\n-\tREFTABLE_CALLOC_ARRAY(mi->subiters, mt->readers_len);\n-\tfor (size_t i = 0; i < mt->readers_len; i++) {\n-\t\treftable_record_init(&mi->subiters[i].rec, typ);\n-\t\treader_init_iter(mt->readers[i], &mi->subiters[i].iter, typ);\n-\t}\n-\tmi->subiters_len = mt->readers_len;\n-}\n-\n static void merged_iter_close(void *p)\n {\n \tstruct merged_iter *mi = p;\n@@ -244,25 +228,61 @@ reftable_merged_table_min_update_index(struct reftable_merged_table *mt)\n \treturn mt->min;\n }\n \n-void merged_table_init_iter(struct reftable_merged_table *mt,\n-\t\t\t struct reftable_iterator *it,\n-\t\t\t uint8_t typ)\n+int merged_table_init_iter(struct reftable_merged_table *mt,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t uint8_t typ)\n {\n-\tstruct merged_iter *mi = reftable_malloc(sizeof(*mi));\n-\tmerged_iter_init(mi, mt, typ);\n+\tstruct merged_subiter *subiters;\n+\tstruct merged_iter *mi = NULL;\n+\tint ret;\n+\n+\tREFTABLE_CALLOC_ARRAY(subiters, mt->readers_len);\n+\tif (!subiters) {\n+\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n+\tfor (size_t i = 0; i < mt->readers_len; i++) {\n+\t\treftable_record_init(&subiters[i].rec, typ);\n+\t\treader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n+\t}\n+\n+\tREFTABLE_CALLOC_ARRAY(mi, 1);\n+\tif (!mi) {\n+\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\tmi->advance_index = -1;\n+\tmi->suppress_deletions = mt->suppress_deletions;\n+\tmi->subiters = subiters;\n+\tmi->subiters_len = mt->readers_len;\n+\n \titerator_from_merged_iter(it, mi);\n+\tret = 0;\n+\n+out:\n+\tif (ret < 0) {\n+\t\tfor (size_t i = 0; subiters && i < mt->readers_len; i++) {\n+\t\t\treftable_iterator_destroy(&subiters[i].iter);\n+\t\t\treftable_record_release(&subiters[i].rec);\n+\t\t}\n+\t\tfree(subiters);\n+\t\tfree(mi);\n+\t}\n+\n+\treturn ret;\n }\n \n-void reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it)\n+int reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(mt, it, BLOCK_TYPE_REF);\n+\treturn merged_table_init_iter(mt, it, BLOCK_TYPE_REF);\n }\n \n-void reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it)\n+int reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(mt, it, BLOCK_TYPE_LOG);\n+\treturn merged_table_init_iter(mt, it, BLOCK_TYPE_LOG);\n }\n \n uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\ndiff --git a/reftable/merged.h b/reftable/merged.h\nindex de5fd33f010..89bd0c4b35b 100644\n--- a/reftable/merged.h\n+++ b/reftable/merged.h\n@@ -26,8 +26,8 @@ struct reftable_merged_table {\n \n struct reftable_iterator;\n \n-void merged_table_init_iter(struct reftable_merged_table *mt,\n-\t\t\t struct reftable_iterator *it,\n-\t\t\t uint8_t typ);\n+int merged_table_init_iter(struct reftable_merged_table *mt,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t uint8_t typ);\n \n #endif\ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nindex 16d19f8df20..a970d5dd89a 100644\n--- a/reftable/reftable-merged.h\n+++ b/reftable/reftable-merged.h\n@@ -37,12 +37,12 @@ int reftable_merged_table_new(struct reftable_merged_table **dest,\n \t\t\t uint32_t hash_id);\n \n /* Initialize a merged table iterator for reading refs. */\n-void reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it);\n+int reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it);\n \n /* Initialize a merged table iterator for reading logs. */\n-void reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it);\n+int reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it);\n \n /* returns the max update_index covered by this merged table. */\n uint64_t\ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nindex f4f8cabc7fb..e958f911b4e 100644\n--- a/reftable/reftable-stack.h\n+++ b/reftable/reftable-stack.h\n@@ -73,16 +73,16 @@ struct reftable_iterator;\n * be used to iterate through refs. The iterator is valid until the next reload\n * or write.\n */\n-void reftable_stack_init_ref_iterator(struct reftable_stack *st,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_stack_init_ref_iterator(struct reftable_stack *st,\n+\t\t\t\t struct reftable_iterator *it);\n \n /*\n * Initialize an iterator for the merged tables contained in the stack that can\n * be used to iterate through logs. The iterator is valid until the next reload\n * or write.\n */\n-void reftable_stack_init_log_iterator(struct reftable_stack *st,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_stack_init_log_iterator(struct reftable_stack *st,\n+\t\t\t\t struct reftable_iterator *it);\n \n /* returns the merged_table for seeking. This table is valid until the\n * next write or reload, and should not be closed or deleted.\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex ea21ca6e5f5..bb4d2309189 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -136,18 +136,18 @@ int read_lines(const char *filename, char ***namesp)\n \treturn err;\n }\n \n-void reftable_stack_init_ref_iterator(struct reftable_stack *st,\n+int reftable_stack_init_ref_iterator(struct reftable_stack *st,\n \t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(reftable_stack_merged_table(st),\n-\t\t\t it, BLOCK_TYPE_REF);\n+\treturn merged_table_init_iter(reftable_stack_merged_table(st),\n+\t\t\t\t it, BLOCK_TYPE_REF);\n }\n \n-void reftable_stack_init_log_iterator(struct reftable_stack *st,\n-\t\t\t\t struct reftable_iterator *it)\n+int reftable_stack_init_log_iterator(struct reftable_stack *st,\n+\t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(reftable_stack_merged_table(st),\n-\t\t\t it, BLOCK_TYPE_LOG);\n+\treturn merged_table_init_iter(reftable_stack_merged_table(st),\n+\t\t\t\t it, BLOCK_TYPE_LOG);\n }\n \n struct reftable_merged_table *\n@@ -952,7 +952,10 @@ static int stack_write_compact(struct reftable_stack *st,\n \tif (err < 0)\n \t\tgoto done;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tif (err < 0)\n \t\tgoto done;\n@@ -977,7 +980,10 @@ static int stack_write_compact(struct reftable_stack *st,\n \t}\n \treftable_iterator_destroy(&it);\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tif (err < 0)\n \t\tgoto done;\n@@ -1496,7 +1502,10 @@ int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n \tstruct reftable_iterator it = { 0 };\n \tint ret;\n \n-\treftable_merged_table_init_ref_iterator(st->merged, &it);\n+\tret = reftable_merged_table_init_ref_iterator(st->merged, &it);\n+\tif (ret)\n+\t\tgoto out;\n+\n \tret = reftable_iterator_seek_ref(&it, refname);\n \tif (ret)\n \t\tgoto out;\n@@ -1523,7 +1532,10 @@ int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n \tstruct reftable_iterator it = {0};\n \tint err;\n \n-\treftable_stack_init_log_iterator(st, &it);\n+\terr = reftable_stack_init_log_iterator(st, &it);\n+\tif (err)\n+\t\tgoto done;\n+\n \terr = reftable_iterator_seek_log(&it, refname);\n \tif (err)\n \t\tgoto done;\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 29d4e9a755d..5c8849d115b 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -28,7 +28,10 @@ static int dump_table(struct reftable_merged_table *mt)\n \tconst struct git_hash_algo *algop;\n \tint err;\n \n-\treftable_merged_table_init_ref_iterator(mt, &it);\n+\terr = reftable_merged_table_init_ref_iterator(mt, &it);\n+\tif (err < 0)\n+\t\treturn err;\n+\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tif (err < 0)\n \t\treturn err;\n@@ -63,7 +66,10 @@ static int dump_table(struct reftable_merged_table *mt)\n \treftable_iterator_destroy(&it);\n \treftable_ref_record_release(&ref);\n \n-\treftable_merged_table_init_log_iterator(mt, &it);\n+\terr = reftable_merged_table_init_log_iterator(mt, &it);\n+\tif (err < 0)\n+\t\treturn err;\n+\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tif (err < 0)\n \t\treturn err;\ndiff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c\nindex 19e54bdfb8b..3d2848632db 100644\n--- a/t/unit-tests/t-reftable-merged.c\n+++ b/t/unit-tests/t-reftable-merged.c\n@@ -82,7 +82,8 @@ static void t_merged_single_record(void)\n \tstruct reftable_iterator it = { 0 };\n \tint err;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"a\");\n \tcheck(!err);\n \n@@ -161,7 +162,8 @@ static void t_merged_refs(void)\n \tsize_t cap = 0;\n \tsize_t i;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"a\");\n \tcheck(!err);\n \tcheck_int(reftable_merged_table_hash_id(mt), ==, GIT_SHA1_FORMAT_ID);\n@@ -367,7 +369,8 @@ static void t_merged_logs(void)\n \tsize_t cap = 0;\n \tsize_t i;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"a\");\n \tcheck(!err);\n \tcheck_int(reftable_merged_table_hash_id(mt), ==, GIT_SHA1_FORMAT_ID);\n@@ -390,7 +393,8 @@ static void t_merged_logs(void)\n \t\tcheck(reftable_log_record_equal(want[i], &out[i],\n \t\t\t\t\t\t GIT_SHA1_RAWSZ));\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log_at(&it, \"a\", 2);\n \tcheck(!err);\n \treftable_log_record_release(&out[0]);\ndiff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c\nindex 65e513d5ec8..6e41a4cf7e5 100644\n--- a/t/unit-tests/t-reftable-stack.c\n+++ b/t/unit-tests/t-reftable-stack.c\n@@ -599,7 +599,9 @@ static void t_reftable_stack_iterator(void)\n \n \treftable_iterator_destroy(&it);\n \n-\treftable_stack_init_log_iterator(st, &it);\n+\terr = reftable_stack_init_log_iterator(st, &it);\n+\tcheck(!err);\n+\n \treftable_iterator_seek_log(&it, logs[0].refname);\n \tfor (i = 0; ; i++) {\n \t\tstruct reftable_log_record log = { 0 };\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503318","messageId":"20d3833014190af7cd42f3d6be79e944d4fd0de7.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 11/22] reftable/reader: handle allocation failures for unindexed reader","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:32:27Z","receivedAt":"2024-09-24T06:32:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when creating unindexed readers.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/reader.c | 23 +++++++++++++++++------\n 1 file changed, 17 insertions(+), 6 deletions(-)\n\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 6494ce2e327..485ee085dac 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -734,19 +734,30 @@ static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n \tREFTABLE_ALLOC_ARRAY(ti, 1);\n \ttable_iter_init(ti, r);\n \terr = table_iter_seek_start(ti, BLOCK_TYPE_REF, 0);\n-\tif (err < 0) {\n-\t\treftable_free(ti);\n-\t\treturn err;\n-\t}\n+\tif (err < 0)\n+\t\tgoto out;\n \n-\tfilter = reftable_malloc(sizeof(struct filtering_ref_iterator));\n+\tfilter = reftable_malloc(sizeof(*filter));\n+\tif (!filter) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n \t*filter = empty;\n \n \tstrbuf_add(&filter->oid, oid, oid_len);\n \titerator_from_table_iter(&filter->it, ti);\n \n \titerator_from_filtering_ref_iterator(it, filter);\n-\treturn 0;\n+\n+\terr = 0;\n+\n+out:\n+\tif (err < 0) {\n+\t\tif (ti)\n+\t\t\ttable_iter_close(ti);\n+\t\treftable_free(ti);\n+\t}\n+\treturn err;\n }\n \n int reftable_reader_refs_for(struct reftable_reader *r,\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503319","messageId":"e35c3a705d5d41c1f011ee129bf9747e5d5df31e.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 12/22] reftable/reader: handle allocation failures in `reader_init_iter()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:32:30Z","receivedAt":"2024-09-24T06:32:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reader_init_iter()`. This requires us to\nalso adapt `reftable_reader_init_*_iterator()` to bubble up the new\nerror codes. Adapt callers accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 4 +++-\n reftable/reader.c | 28 +++++++++++++++---------\n reftable/reader.h | 6 ++---\n reftable/reftable-reader.h | 8 +++----\n t/unit-tests/t-reftable-readwrite.c | 34 +++++++++++++++++++----------\n 5 files changed, 50 insertions(+), 30 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 5a8ea8ae779..741f62ea638 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -244,7 +244,9 @@ int merged_table_init_iter(struct reftable_merged_table *mt,\n \n \tfor (size_t i = 0; i < mt->readers_len; i++) {\n \t\treftable_record_init(&subiters[i].rec, typ);\n-\t\treader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n+\t\tret = reader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n+\t\tif (ret < 0)\n+\t\t\tgoto out;\n \t}\n \n \tREFTABLE_CALLOC_ARRAY(mi, 1);\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 485ee085dac..f696e992dfc 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -554,32 +554,37 @@ static void iterator_from_table_iter(struct reftable_iterator *it,\n \tit->ops = &table_iter_vtable;\n }\n \n-void reader_init_iter(struct reftable_reader *r,\n-\t\t struct reftable_iterator *it,\n-\t\t uint8_t typ)\n+int reader_init_iter(struct reftable_reader *r,\n+\t\t struct reftable_iterator *it,\n+\t\t uint8_t typ)\n {\n \tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n \n \tif (offs->is_present) {\n \t\tstruct table_iter *ti;\n \t\tREFTABLE_ALLOC_ARRAY(ti, 1);\n+\t\tif (!ti)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \t\ttable_iter_init(ti, r);\n \t\titerator_from_table_iter(it, ti);\n \t} else {\n \t\titerator_set_empty(it);\n \t}\n+\n+\treturn 0;\n }\n \n-void reftable_reader_init_ref_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it)\n+int reftable_reader_init_ref_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it)\n {\n-\treader_init_iter(r, it, BLOCK_TYPE_REF);\n+\treturn reader_init_iter(r, it, BLOCK_TYPE_REF);\n }\n \n-void reftable_reader_init_log_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it)\n+int reftable_reader_init_log_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it)\n {\n-\treader_init_iter(r, it, BLOCK_TYPE_LOG);\n+\treturn reader_init_iter(r, it, BLOCK_TYPE_LOG);\n }\n \n int reftable_reader_new(struct reftable_reader **out,\n@@ -689,7 +694,10 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n \tstruct indexed_table_ref_iter *itr = NULL;\n \n \t/* Look through the reverse index. */\n-\treader_init_iter(r, &oit, BLOCK_TYPE_OBJ);\n+\terr = reader_init_iter(r, &oit, BLOCK_TYPE_OBJ);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = iterator_seek(&oit, &want);\n \tif (err != 0)\n \t\tgoto done;\ndiff --git a/reftable/reader.h b/reftable/reader.h\nindex 3710ee09b4c..02d10c5d37e 100644\n--- a/reftable/reader.h\n+++ b/reftable/reader.h\n@@ -56,9 +56,9 @@ struct reftable_reader {\n \n const char *reader_name(struct reftable_reader *r);\n \n-void reader_init_iter(struct reftable_reader *r,\n-\t\t struct reftable_iterator *it,\n-\t\t uint8_t typ);\n+int reader_init_iter(struct reftable_reader *r,\n+\t\t struct reftable_iterator *it,\n+\t\t uint8_t typ);\n \n /* initialize a block reader to read from `r` */\n int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nindex a600452b565..6a2d0b693f5 100644\n--- a/reftable/reftable-reader.h\n+++ b/reftable/reftable-reader.h\n@@ -46,12 +46,12 @@ void reftable_reader_incref(struct reftable_reader *reader);\n void reftable_reader_decref(struct reftable_reader *reader);\n \n /* Initialize a reftable iterator for reading refs. */\n-void reftable_reader_init_ref_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_reader_init_ref_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it);\n \n /* Initialize a reftable iterator for reading logs. */\n-void reftable_reader_init_log_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_reader_init_log_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it);\n \n /* returns the hash ID used in this table. */\n uint32_t reftable_reader_hash_id(struct reftable_reader *r);\ndiff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c\nindex e1b235a5f13..acca927a2cf 100644\n--- a/t/unit-tests/t-reftable-readwrite.c\n+++ b/t/unit-tests/t-reftable-readwrite.c\n@@ -206,7 +206,8 @@ static void t_log_write_read(void)\n \terr = reftable_reader_new(&reader, &source, \"file.log\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \n \terr = reftable_iterator_seek_ref(&it, names[N - 1]);\n \tcheck(!err);\n@@ -221,8 +222,8 @@ static void t_log_write_read(void)\n \treftable_iterator_destroy(&it);\n \treftable_ref_record_release(&ref);\n \n-\treftable_reader_init_log_iterator(reader, &it);\n-\n+\terr = reftable_reader_init_log_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tcheck(!err);\n \n@@ -296,7 +297,8 @@ static void t_log_zlib_corruption(void)\n \terr = reftable_reader_new(&reader, &source, \"file.log\");\n \tcheck(!err);\n \n-\treftable_reader_init_log_iterator(reader, &it);\n+\terr = reftable_reader_init_log_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"refname\");\n \tcheck_int(err, ==, REFTABLE_ZLIB_ERROR);\n \n@@ -325,7 +327,8 @@ static void t_table_read_write_sequential(void)\n \terr = reftable_reader_new(&reader, &source, \"file.ref\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tcheck(!err);\n \n@@ -376,7 +379,8 @@ static void t_table_read_api(void)\n \terr = reftable_reader_new(&reader, &source, \"file.ref\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, names[0]);\n \tcheck(!err);\n \n@@ -419,7 +423,8 @@ static void t_table_read_write_seek(int index, int hash_id)\n \t}\n \n \tfor (i = 1; i < N; i++) {\n-\t\treftable_reader_init_ref_iterator(reader, &it);\n+\t\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\t\tcheck(!err);\n \t\terr = reftable_iterator_seek_ref(&it, names[i]);\n \t\tcheck(!err);\n \t\terr = reftable_iterator_next_ref(&it, &ref);\n@@ -435,7 +440,8 @@ static void t_table_read_write_seek(int index, int hash_id)\n \tstrbuf_addstr(&pastLast, names[N - 1]);\n \tstrbuf_addstr(&pastLast, \"/\");\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, pastLast.buf);\n \tif (err == 0) {\n \t\tstruct reftable_ref_record ref = { 0 };\n@@ -534,7 +540,8 @@ static void t_table_refs_for(int indexed)\n \tif (!indexed)\n \t\treader->obj_offsets.is_present = 0;\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tcheck(!err);\n \treftable_iterator_destroy(&it);\n@@ -593,7 +600,8 @@ static void t_write_empty_table(void)\n \terr = reftable_reader_new(&rd, &source, \"filename\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(rd, &it);\n+\terr = reftable_reader_init_ref_iterator(rd, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tcheck(!err);\n \n@@ -802,7 +810,8 @@ static void t_write_multiple_indices(void)\n \t * Seeking the log uses the log index now. In case there is any\n \t * confusion regarding indices we would notice here.\n \t */\n-\treftable_reader_init_log_iterator(reader, &it);\n+\terr = reftable_reader_init_log_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tcheck(!err);\n \n@@ -858,7 +867,8 @@ static void t_write_multi_level_index(void)\n \t/*\n \t * Seeking the last ref should work as expected.\n \t */\n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"refs/heads/199\");\n \tcheck(!err);\n \n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503320","messageId":"ca3b57f151ed8a2b318112fe1ea00e5d0cc00f8e.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 13/22] reftable/stack: handle allocation failures on reload","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:32:33Z","receivedAt":"2024-09-24T06:32:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_stack_reload_once()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 34 +++++++++++++++++++++++++---------\n 1 file changed, 25 insertions(+), 9 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex bb4d2309189..060b2c1b90d 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -214,13 +214,13 @@ void reftable_stack_destroy(struct reftable_stack *st)\n }\n \n static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n-\t\t\t\t\t\t int cur_len)\n+\t\t\t\t\t\t size_t cur_len)\n {\n \tstruct reftable_reader **cur = reftable_calloc(cur_len, sizeof(*cur));\n-\tint i = 0;\n-\tfor (i = 0; i < cur_len; i++) {\n+\tif (!cur)\n+\t\treturn NULL;\n+\tfor (size_t i = 0; i < cur_len; i++)\n \t\tcur[i] = st->readers[i];\n-\t}\n \treturn cur;\n }\n \n@@ -229,18 +229,30 @@ static int reftable_stack_reload_once(struct reftable_stack *st,\n \t\t\t\t int reuse_open)\n {\n \tsize_t cur_len = !st->merged ? 0 : st->merged->readers_len;\n-\tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n+\tstruct reftable_reader **cur;\n \tstruct reftable_reader **reused = NULL;\n-\tsize_t reused_len = 0, reused_alloc = 0;\n-\tsize_t names_len = names_length(names);\n-\tstruct reftable_reader **new_readers =\n-\t\treftable_calloc(names_len, sizeof(*new_readers));\n+\tstruct reftable_reader **new_readers;\n+\tsize_t reused_len = 0, reused_alloc = 0, names_len;\n \tsize_t new_readers_len = 0;\n \tstruct reftable_merged_table *new_merged = NULL;\n \tstruct strbuf table_path = STRBUF_INIT;\n \tint err = 0;\n \tsize_t i;\n \n+\tcur = stack_copy_readers(st, cur_len);\n+\tif (!cur) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tnames_len = names_length(names);\n+\n+\tnew_readers = reftable_calloc(names_len, sizeof(*new_readers));\n+\tif (!new_readers) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \twhile (*names) {\n \t\tstruct reftable_reader *rd = NULL;\n \t\tconst char *name = *names++;\n@@ -261,6 +273,10 @@ static int reftable_stack_reload_once(struct reftable_stack *st,\n \t\t\t\t * do by bumping their refcount.\n \t\t\t\t */\n \t\t\t\tREFTABLE_ALLOC_GROW(reused, reused_len + 1, reused_alloc);\n+\t\t\t\tif (!reused) {\n+\t\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n \t\t\t\treused[reused_len++] = rd;\n \t\t\t\treftable_reader_incref(rd);\n \t\t\t\tbreak;\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503321","messageId":"7377421a632bf7a3517036316eccfb11fae91611.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 14/22] reftable/stack: handle allocation failures in `reftable_new_stack()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:32:35Z","receivedAt":"2024-09-24T06:32:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_new_stack()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 34 ++++++++++++++++++++++++++--------\n 1 file changed, 26 insertions(+), 8 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 060b2c1b90d..1b77c9d0146 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -56,10 +56,16 @@ static int reftable_fd_flush(void *arg)\n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \t\t const struct reftable_write_options *_opts)\n {\n-\tstruct reftable_stack *p = reftable_calloc(1, sizeof(*p));\n \tstruct strbuf list_file_name = STRBUF_INIT;\n-\tstruct reftable_write_options opts = {0};\n-\tint err = 0;\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct reftable_stack *p;\n+\tint err;\n+\n+\tp = reftable_calloc(1, sizeof(*p));\n+\tif (!p) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n \n \tif (_opts)\n \t\topts = *_opts;\n@@ -74,15 +80,23 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \n \tp->list_file = strbuf_detach(&list_file_name, NULL);\n \tp->list_fd = -1;\n-\tp->reftable_dir = xstrdup(dir);\n \tp->opts = opts;\n+\tp->reftable_dir = reftable_strdup(dir);\n+\tif (!p->reftable_dir) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n \n \terr = reftable_stack_reload_maybe_reuse(p, 1);\n-\tif (err < 0) {\n+\tif (err < 0)\n+\t\tgoto out;\n+\n+\t*dest = p;\n+\terr = 0;\n+\n+out:\n+\tif (err < 0)\n \t\treftable_stack_destroy(p);\n-\t} else {\n-\t\t*dest = p;\n-\t}\n \treturn err;\n }\n \n@@ -171,6 +185,10 @@ void reftable_stack_destroy(struct reftable_stack *st)\n {\n \tchar **names = NULL;\n \tint err = 0;\n+\n+\tif (!st)\n+\t\treturn;\n+\n \tif (st->merged) {\n \t\treftable_merged_table_free(st->merged);\n \t\tst->merged = NULL;\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503322","messageId":"244e8667c5db13b4034af747326ce2822a0f9f72.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 15/22] reftable/stack: handle allocation failures in `stack_compact_range()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:32:39Z","receivedAt":"2024-09-24T06:32:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `stack_compact_range()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 19 +++++++++++++++++--\n 1 file changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 1b77c9d0146..2e6dd513d72 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1131,6 +1131,11 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * from the point of view of the newer process.\n \t */\n \tREFTABLE_CALLOC_ARRAY(table_locks, last - first + 1);\n+\tif (!table_locks) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tfor (i = last + 1; i > first; i--) {\n \t\tstack_filename(&table_name, st, reader_name(st->readers[i - 1]));\n \n@@ -1312,8 +1317,18 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\t * thus have to allocate `readers_len + 1` many entries.\n \t\t */\n \t\tREFTABLE_CALLOC_ARRAY(names, st->merged->readers_len + 1);\n-\t\tfor (size_t i = 0; i < st->merged->readers_len; i++)\n-\t\t\tnames[i] = xstrdup(st->readers[i]->name);\n+\t\tif (!names) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tfor (size_t i = 0; i < st->merged->readers_len; i++) {\n+\t\t\tnames[i] = reftable_strdup(st->readers[i]->name);\n+\t\t\tif (!names[i]) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n \t\tfirst_to_replace = first;\n \t\tlast_to_replace = last;\n \t}\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503323","messageId":"99f4868c38f48ccff91f8a9366dad7444d493786.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 16/22] reftable/stack: handle allocation failures in auto compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:32:44Z","receivedAt":"2024-09-24T06:32:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_stack_auto_compact()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 16 ++++++++++++----\n 1 file changed, 12 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 2e6dd513d72..990784d9d2f 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1518,6 +1518,8 @@ static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n \tuint64_t *sizes;\n \n \tREFTABLE_CALLOC_ARRAY(sizes, st->merged->readers_len);\n+\tif (!sizes)\n+\t\treturn NULL;\n \n \tfor (size_t i = 0; i < st->merged->readers_len; i++)\n \t\tsizes[i] = st->readers[i]->size - overhead;\n@@ -1527,11 +1529,17 @@ static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n \n int reftable_stack_auto_compact(struct reftable_stack *st)\n {\n-\tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n-\tstruct segment seg =\n-\t\tsuggest_compaction_segment(sizes, st->merged->readers_len,\n-\t\t\t\t\t st->opts.auto_compaction_factor);\n+\tstruct segment seg;\n+\tuint64_t *sizes;\n+\n+\tsizes = stack_table_sizes_for_compaction(st);\n+\tif (!sizes)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n+\tseg = suggest_compaction_segment(sizes, st->merged->readers_len,\n+\t\t\t\t\t st->opts.auto_compaction_factor);\n \treftable_free(sizes);\n+\n \tif (segment_size(&seg) > 0)\n \t\treturn stack_compact_range(st, seg.start, seg.end - 1,\n \t\t\t\t\t NULL, STACK_COMPACT_RANGE_BEST_EFFORT);\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503324","messageId":"271839a62609b00c2f6d572ca8d5bf135d54c5dd.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 17/22] reftable/iter: handle allocation failures when creating indexed table iter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:32:47Z","receivedAt":"2024-09-24T06:32:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `new_indexed_table_ref_iter()`. While at\nit, rename the function to match our coding style.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/iter.c | 20 ++++++++++++++++----\n reftable/iter.h | 2 +-\n reftable/reader.c | 7 ++++++-\n 3 files changed, 23 insertions(+), 6 deletions(-)\n\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex 416a9f6996b..d319538f804 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -181,14 +181,20 @@ static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\n \t}\n }\n \n-int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+int indexed_table_ref_iter_new(struct indexed_table_ref_iter **dest,\n \t\t\t struct reftable_reader *r, uint8_t *oid,\n \t\t\t int oid_len, uint64_t *offsets, int offset_len)\n {\n \tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n-\tstruct indexed_table_ref_iter *itr = reftable_calloc(1, sizeof(*itr));\n+\tstruct indexed_table_ref_iter *itr;\n \tint err = 0;\n \n+\titr = reftable_calloc(1, sizeof(*itr));\n+\tif (!itr) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n \t*itr = empty;\n \titr->r = r;\n \tstrbuf_add(&itr->oid, oid, oid_len);\n@@ -197,10 +203,16 @@ int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n \titr->offset_len = offset_len;\n \n \terr = indexed_table_ref_iter_next_block(itr);\n+\tif (err < 0)\n+\t\tgoto out;\n+\n+\t*dest = itr;\n+\terr = 0;\n+\n+out:\n \tif (err < 0) {\n+\t\t*dest = NULL;\n \t\treftable_free(itr);\n-\t} else {\n-\t\t*dest = itr;\n \t}\n \treturn err;\n }\ndiff --git a/reftable/iter.h b/reftable/iter.h\nindex befc4597df1..b3225bc7add 100644\n--- a/reftable/iter.h\n+++ b/reftable/iter.h\n@@ -82,7 +82,7 @@ void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n \t\t\t\t\t struct indexed_table_ref_iter *itr);\n \n /* Takes ownership of `offsets` */\n-int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+int indexed_table_ref_iter_new(struct indexed_table_ref_iter **dest,\n \t\t\t struct reftable_reader *r, uint8_t *oid,\n \t\t\t int oid_len, uint64_t *offsets, int offset_len);\n \ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex f696e992dfc..0179e4e73dd 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -715,7 +715,7 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n \t\tgoto done;\n \t}\n \n-\terr = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),\n+\terr = indexed_table_ref_iter_new(&itr, r, oid, hash_size(r->hash_id),\n \t\t\t\t\t got.u.obj.offsets,\n \t\t\t\t\t got.u.obj.offset_len);\n \tif (err < 0)\n@@ -740,6 +740,11 @@ static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n \tint err;\n \n \tREFTABLE_ALLOC_ARRAY(ti, 1);\n+\tif (!ti) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n \ttable_iter_init(ti, r);\n \terr = table_iter_seek_start(ti, BLOCK_TYPE_REF, 0);\n \tif (err < 0)\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503325","messageId":"c41808e9d75fe4a791ffb1980fa1da6e424c9a14.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 18/22] reftable/blocksource: handle allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:32:50Z","receivedAt":"2024-09-24T06:32:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in the blocksource code.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/blocksource.c | 25 ++++++++++++++++++++-----\n 1 file changed, 20 insertions(+), 5 deletions(-)\n\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex e93cac9bb6f..6b570751be6 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -30,6 +30,8 @@ static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n \tstruct strbuf *b = v;\n \tassert(off + size <= b->len);\n \tREFTABLE_CALLOC_ARRAY(dest->data, size);\n+\tif (!dest->data)\n+\t\treturn -1;\n \tmemcpy(dest->data, b->buf + off, size);\n \tdest->len = size;\n \treturn size;\n@@ -98,27 +100,40 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,\n {\n \tstruct file_block_source *p;\n \tstruct stat st;\n-\tint fd;\n+\tint fd, err;\n \n \tfd = open(name, O_RDONLY);\n \tif (fd < 0) {\n \t\tif (errno == ENOENT)\n \t\t\treturn REFTABLE_NOT_EXIST_ERROR;\n-\t\treturn -1;\n+\t\terr = -1;\n+\t\tgoto out;\n \t}\n \n \tif (fstat(fd, &st) < 0) {\n-\t\tclose(fd);\n-\t\treturn REFTABLE_IO_ERROR;\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto out;\n \t}\n \n \tREFTABLE_CALLOC_ARRAY(p, 1);\n+\tif (!p) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n \tp->size = st.st_size;\n \tp->data = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n-\tclose(fd);\n \n \tassert(!bs->ops);\n \tbs->ops = &file_vtable;\n \tbs->arg = p;\n+\n+\terr = 0;\n+\n+out:\n+\tif (fd >= 0)\n+\t\tclose(fd);\n+\tif (err < 0)\n+\t\tfree(p);\n \treturn 0;\n }\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503326","messageId":"9348b0a45759f45fec178e2a2aa607ec43dbec60.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 19/22] reftable/block: handle allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:32:53Z","receivedAt":"2024-09-24T06:32:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `block_writer_init()` and\n`block_reader_init()`. This requires us to bubble up error codes into\n`writer_reinit_block_writer()`. Adapt call sites accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/block.c | 23 +++++++++++++--\n reftable/block.h | 4 +--\n reftable/writer.c | 51 ++++++++++++++++++++++-----------\n t/unit-tests/t-reftable-block.c | 20 +++++++------\n 4 files changed, 69 insertions(+), 29 deletions(-)\n\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 00030eee065..bfa7dc61bf5 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -52,6 +52,8 @@ static int block_writer_register_restart(struct block_writer *w, int n,\n \t\treturn -1;\n \tif (is_restart) {\n \t\tREFTABLE_ALLOC_GROW(w->restarts, w->restart_len + 1, w->restart_cap);\n+\t\tif (!w->restarts)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \t\tw->restarts[w->restart_len++] = w->next;\n \t}\n \n@@ -63,8 +65,8 @@ static int block_writer_register_restart(struct block_writer *w, int n,\n \treturn 0;\n }\n \n-void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n-\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n+int block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n {\n \tbw->buf = buf;\n \tbw->hash_size = hash_size;\n@@ -78,8 +80,12 @@ void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n \tbw->last_key.len = 0;\n \tif (!bw->zstream) {\n \t\tREFTABLE_CALLOC_ARRAY(bw->zstream, 1);\n+\t\tif (!bw->zstream)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \t\tdeflateInit(bw->zstream, 9);\n \t}\n+\n+\treturn 0;\n }\n \n uint8_t block_writer_type(struct block_writer *bw)\n@@ -163,6 +169,10 @@ int block_writer_finish(struct block_writer *w)\n \t\t */\n \t\tcompressed_len = deflateBound(w->zstream, src_len);\n \t\tREFTABLE_ALLOC_GROW(w->compressed, compressed_len, w->compressed_cap);\n+\t\tif (!w->compressed) {\n+\t\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\treturn ret;\n+\t\t}\n \n \t\tw->zstream->next_out = w->compressed;\n \t\tw->zstream->avail_out = compressed_len;\n@@ -219,12 +229,21 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,\n \t\t/* Log blocks specify the *uncompressed* size in their header. */\n \t\tREFTABLE_ALLOC_GROW(br->uncompressed_data, sz,\n \t\t\t\t br->uncompressed_cap);\n+\t\tif (!br->uncompressed_data) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n \n \t\t/* Copy over the block header verbatim. It's not compressed. */\n \t\tmemcpy(br->uncompressed_data, block->data, block_header_skip);\n \n \t\tif (!br->zstream) {\n \t\t\tREFTABLE_CALLOC_ARRAY(br->zstream, 1);\n+\t\t\tif (!br->zstream) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\n \t\t\terr = inflateInit(br->zstream);\n \t\t} else {\n \t\t\terr = inflateReset(br->zstream);\ndiff --git a/reftable/block.h b/reftable/block.h\nindex 1c8f25ee6eb..18d7ea03373 100644\n--- a/reftable/block.h\n+++ b/reftable/block.h\n@@ -45,8 +45,8 @@ struct block_writer {\n /*\n * initializes the blockwriter to write `typ` entries, using `buf` as temporary\n * storage. `buf` is not owned by the block_writer. */\n-void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n-\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n+int block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n \n /* returns the block type (eg. 'r' for ref records. */\n uint8_t block_writer_type(struct block_writer *bw);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 54ec822e1cd..712a84791b2 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -102,19 +102,24 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \treturn header_size(writer_version(w));\n }\n \n-static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n+static int writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n {\n-\tint block_start = 0;\n-\tif (w->next == 0) {\n+\tint block_start = 0, ret;\n+\n+\tif (w->next == 0)\n \t\tblock_start = header_size(writer_version(w));\n-\t}\n \n \tstrbuf_reset(&w->last_key);\n-\tblock_writer_init(&w->block_writer_data, typ, w->block,\n-\t\t\t w->opts.block_size, block_start,\n-\t\t\t hash_size(w->opts.hash_id));\n+\tret = block_writer_init(&w->block_writer_data, typ, w->block,\n+\t\t\t\tw->opts.block_size, block_start,\n+\t\t\t\thash_size(w->opts.hash_id));\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n \tw->block_writer = &w->block_writer_data;\n \tw->block_writer->restart_interval = w->opts.restart_interval;\n+\n+\treturn 0;\n }\n \n int reftable_writer_new(struct reftable_writer **out,\n@@ -247,8 +252,11 @@ static int writer_add_record(struct reftable_writer *w,\n \n \tstrbuf_reset(&w->last_key);\n \tstrbuf_addbuf(&w->last_key, &key);\n-\tif (!w->block_writer)\n-\t\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\tif (!w->block_writer) {\n+\t\terr = writer_reinit_block_writer(w, reftable_record_type(rec));\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t}\n \n \tif (block_writer_type(w->block_writer) != reftable_record_type(rec))\n \t\tBUG(\"record of type %d added to writer of type %d\",\n@@ -271,7 +279,9 @@ static int writer_add_record(struct reftable_writer *w,\n \terr = writer_flush_block(w);\n \tif (err < 0)\n \t\tgoto done;\n-\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\terr = writer_reinit_block_writer(w, reftable_record_type(rec));\n+\tif (err < 0)\n+\t\tgoto done;\n \n \t/*\n \t * Try to add the record to the writer again. If this still fails then\n@@ -461,7 +471,9 @@ static int writer_finish_section(struct reftable_writer *w)\n \n \t\tmax_level++;\n \t\tindex_start = w->next;\n-\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\t\terr = writer_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n \n \t\tidx = w->index;\n \t\tidx_len = w->index_len;\n@@ -555,7 +567,10 @@ static void write_object_record(void *void_arg, void *key)\n \tif (arg->err < 0)\n \t\tgoto done;\n \n-\twriter_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\targ->err = writer_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n \targ->err = block_writer_add(arg->w->block_writer, &rec);\n \tif (arg->err == 0)\n \t\tgoto done;\n@@ -584,16 +599,18 @@ static int writer_dump_object_index(struct reftable_writer *w)\n \tstruct common_prefix_arg common = {\n \t\t.max = 1,\t\t/* obj_id_len should be >= 2. */\n \t};\n-\tif (w->obj_index_tree) {\n+\tint err;\n+\n+\tif (w->obj_index_tree)\n \t\tinfix_walk(w->obj_index_tree, &update_common, &common);\n-\t}\n \tw->stats.object_id_len = common.max + 1;\n \n-\twriter_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\terr = writer_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\tif (err < 0)\n+\t\treturn err;\n \n-\tif (w->obj_index_tree) {\n+\tif (w->obj_index_tree)\n \t\tinfix_walk(w->obj_index_tree, &write_object_record, &closure);\n-\t}\n \n \tif (closure.err < 0)\n \t\treturn closure.err;\ndiff --git a/t/unit-tests/t-reftable-block.c b/t/unit-tests/t-reftable-block.c\nindex f1a49485e23..e52a612e852 100644\n--- a/t/unit-tests/t-reftable-block.c\n+++ b/t/unit-tests/t-reftable-block.c\n@@ -34,8 +34,9 @@ static void t_ref_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \trec.u.ref.refname = (char *) \"\";\n \trec.u.ref.value_type = REFTABLE_REF_DELETION;\n@@ -126,8 +127,9 @@ static void t_log_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n \t\trec.u.log.refname = xstrfmt(\"branch%02\"PRIuMAX , (uintmax_t)i);\n@@ -214,8 +216,9 @@ static void t_obj_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n \t\tuint8_t bytes[] = { i, i + 1, i + 2, i + 3, i + 5 }, *allocated;\n@@ -296,8 +299,9 @@ static void t_index_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n \t\tstrbuf_init(&recs[i].u.idx.last_key, 9);\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503327","messageId":"445daf9464dbc6c7d6c4de31944c6a6b733a1a8b.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 20/22] reftable/pq: handle allocation failures when adding entries","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:32:56Z","receivedAt":"2024-09-24T06:33:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when adding entries to the pqueue. Adapt its\nonly caller accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 5 ++++-\n reftable/pq.c | 7 ++++++-\n reftable/pq.h | 2 +-\n 3 files changed, 11 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 741f62ea638..2c20845d624 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -54,7 +54,10 @@ static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)\n \tif (err)\n \t\treturn err;\n \n-\tmerged_iter_pqueue_add(&mi->pq, &e);\n+\terr = merged_iter_pqueue_add(&mi->pq, &e);\n+\tif (err)\n+\t\treturn err;\n+\n \treturn 0;\n }\n \ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex 2b5b7d1c0e2..03b99122829 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -8,6 +8,7 @@ license that can be found in the LICENSE file or at\n \n #include \"pq.h\"\n \n+#include \"reftable-error.h\"\n #include \"reftable-record.h\"\n #include \"system.h\"\n #include \"basics.h\"\n@@ -44,11 +45,13 @@ struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n \treturn e;\n }\n \n-void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e)\n+int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e)\n {\n \tsize_t i = 0;\n \n \tREFTABLE_ALLOC_GROW(pq->heap, pq->len + 1, pq->cap);\n+\tif (!pq->heap)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tpq->heap[pq->len++] = *e;\n \n \ti = pq->len - 1;\n@@ -59,6 +62,8 @@ void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry\n \t\tSWAP(pq->heap[j], pq->heap[i]);\n \t\ti = j;\n \t}\n+\n+\treturn 0;\n }\n \n void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)\ndiff --git a/reftable/pq.h b/reftable/pq.h\nindex 707bd26767b..83c062eecac 100644\n--- a/reftable/pq.h\n+++ b/reftable/pq.h\n@@ -23,7 +23,7 @@ struct merged_iter_pqueue {\n };\n \n struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n-void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e);\n+int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e);\n void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n int pq_less(struct pq_entry *a, struct pq_entry *b);\n \n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503328","messageId":"c31f9e53f529055bbc1eb5022befb497c1682d19.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 21/22] reftable/tree: handle allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:33:01Z","receivedAt":"2024-09-24T06:33:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The tree interfaces of the reftable library handle both insertion and\nsearching of tree nodes with a single function, where the behaviour is\naltered between the two via an `insert` bit. This makes it quit awkward\nto handle allocation failures because on inserting we'd have to check\nfor `NULL` pointers and return an error, whereas on searching entries we\ndon't have to handle it as an allocation error.\n\nSplit up concerns of this function into two separate functions, one for\ninserting entries and one for searching entries. This makes it easy for\nus to check for allocation errors as `tree_insert()` should never return\na `NULL` pointer now. Adapt callers accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/tree.c | 42 +++++++++++++++++++++++-----------\n reftable/tree.h | 21 +++++++++++++----\n reftable/writer.c | 7 +++---\n t/unit-tests/t-reftable-tree.c | 10 ++++----\n 4 files changed, 54 insertions(+), 26 deletions(-)\n\ndiff --git a/reftable/tree.c b/reftable/tree.c\nindex 5ffb2e0d690..f4dbe720901 100644\n--- a/reftable/tree.c\n+++ b/reftable/tree.c\n@@ -11,28 +11,44 @@ license that can be found in the LICENSE file or at\n \n #include \"basics.h\"\n \n-struct tree_node *tree_search(void *key, struct tree_node **rootp,\n-\t\t\t int (*compare)(const void *, const void *),\n-\t\t\t int insert)\n+struct tree_node *tree_search(struct tree_node *tree,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *))\n {\n \tint res;\n+\tif (!tree)\n+\t\treturn NULL;\n+\tres = compare(key, tree->key);\n+\tif (res < 0)\n+\t\treturn tree_search(tree->left, key, compare);\n+\telse if (res > 0)\n+\t\treturn tree_search(tree->right, key, compare);\n+\treturn tree;\n+}\n+\n+struct tree_node *tree_insert(struct tree_node **rootp,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *))\n+{\n+\tint res;\n+\n \tif (!*rootp) {\n-\t\tif (!insert) {\n+\t\tstruct tree_node *n;\n+\n+\t\tREFTABLE_CALLOC_ARRAY(n, 1);\n+\t\tif (!n)\n \t\t\treturn NULL;\n-\t\t} else {\n-\t\t\tstruct tree_node *n;\n-\t\t\tREFTABLE_CALLOC_ARRAY(n, 1);\n-\t\t\tn->key = key;\n-\t\t\t*rootp = n;\n-\t\t\treturn *rootp;\n-\t\t}\n+\n+\t\tn->key = key;\n+\t\t*rootp = n;\n+\t\treturn *rootp;\n \t}\n \n \tres = compare(key, (*rootp)->key);\n \tif (res < 0)\n-\t\treturn tree_search(key, &(*rootp)->left, compare, insert);\n+\t\treturn tree_insert(&(*rootp)->left, key, compare);\n \telse if (res > 0)\n-\t\treturn tree_search(key, &(*rootp)->right, compare, insert);\n+\t\treturn tree_insert(&(*rootp)->right, key, compare);\n \treturn *rootp;\n }\n \ndiff --git a/reftable/tree.h b/reftable/tree.h\nindex fbdd002e23a..9604453b6d5 100644\n--- a/reftable/tree.h\n+++ b/reftable/tree.h\n@@ -15,12 +15,23 @@ struct tree_node {\n \tstruct tree_node *left, *right;\n };\n \n-/* looks for `key` in `rootp` using `compare` as comparison function. If insert\n- * is set, insert the key if it's not found. Else, return NULL.\n+/*\n+ * Search the tree for the node matching the given key using `compare` as\n+ * comparison function. Returns the node whose key matches or `NULL` in case\n+ * the key does not exist in the tree.\n+ */\n+struct tree_node *tree_search(struct tree_node *tree,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *));\n+\n+/*\n+ * Insert a node into the tree. Returns the newly inserted node if the key does\n+ * not yet exist. Otherwise it returns the preexisting node. Returns `NULL`\n+ * when allocating the new node fails.\n */\n-struct tree_node *tree_search(void *key, struct tree_node **rootp,\n-\t\t\t int (*compare)(const void *, const void *),\n-\t\t\t int insert);\n+struct tree_node *tree_insert(struct tree_node **rootp,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *));\n \n /* performs an infix walk of the tree. */\n void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 712a84791b2..44bed92f467 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -208,8 +208,7 @@ static int writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n \tstruct obj_index_tree_node *key;\n \tstruct tree_node *node;\n \n-\tnode = tree_search(&want, &w->obj_index_tree,\n-\t\t\t &obj_index_tree_node_compare, 0);\n+\tnode = tree_search(w->obj_index_tree, &want, &obj_index_tree_node_compare);\n \tif (!node) {\n \t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n \n@@ -221,8 +220,8 @@ static int writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n \n \t\tstrbuf_reset(&key->hash);\n \t\tstrbuf_addbuf(&key->hash, hash);\n-\t\ttree_search((void *)key, &w->obj_index_tree,\n-\t\t\t &obj_index_tree_node_compare, 1);\n+\t\ttree_insert(&w->obj_index_tree, key,\n+\t\t\t &obj_index_tree_node_compare);\n \t} else {\n \t\tkey = node->key;\n \t}\ndiff --git a/t/unit-tests/t-reftable-tree.c b/t/unit-tests/t-reftable-tree.c\nindex 700479d34b1..79b175a45a7 100644\n--- a/t/unit-tests/t-reftable-tree.c\n+++ b/t/unit-tests/t-reftable-tree.c\n@@ -37,16 +37,17 @@ static void t_tree_search(void)\n \t * values[1] and values[10] (inclusive) in the tree.\n \t */\n \tdo {\n-\t\tnodes[i] = tree_search(&values[i], &root, &t_compare, 1);\n+\t\tnodes[i] = tree_insert(&root, &values[i], &t_compare);\n+\t\tcheck(nodes[i] != NULL);\n \t\ti = (i * 7) % 11;\n \t} while (i != 1);\n \n \tfor (i = 1; i < ARRAY_SIZE(nodes); i++) {\n \t\tcheck_pointer_eq(&values[i], nodes[i]->key);\n-\t\tcheck_pointer_eq(nodes[i], tree_search(&values[i], &root, &t_compare, 0));\n+\t\tcheck_pointer_eq(nodes[i], tree_search(root, &values[i], &t_compare));\n \t}\n \n-\tcheck(!tree_search(values, &root, t_compare, 0));\n+\tcheck(!tree_search(root, values, t_compare));\n \ttree_free(root);\n }\n \n@@ -62,7 +63,8 @@ static void t_infix_walk(void)\n \tsize_t count = 0;\n \n \tdo {\n-\t\ttree_search(&values[i], &root, t_compare, 1);\n+\t\tstruct tree_node *node = tree_insert(&root, &values[i], t_compare);\n+\t\tcheck(node != NULL);\n \t\ti = (i * 7) % 11;\n \t\tcount++;\n \t} while (i != 1);\n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503329","messageId":"d0fe9993716cb213f695584bf9363382894eb36a.1727158127.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727158127.git.ps@pks.im","subject":"[PATCH v2 22/22] reftable: handle trivial allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-24T06:33:05Z","receivedAt":"2024-09-24T06:33:10Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle trivial allocation failures in the reftable library and its unit\ntests.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 3 +++\n reftable/reader.c | 10 +++++++++-\n reftable/stack.c | 20 ++++++++++++++++++++\n reftable/writer.c | 13 +++++++++++--\n t/unit-tests/t-reftable-block.c | 4 ++++\n t/unit-tests/t-reftable-merged.c | 4 ++++\n t/unit-tests/t-reftable-readwrite.c | 27 ++++++++++++++++-----------\n 7 files changed, 67 insertions(+), 14 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 2c20845d624..52bc66c5273 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -203,6 +203,9 @@ int reftable_merged_table_new(struct reftable_merged_table **dest,\n \t}\n \n \tREFTABLE_CALLOC_ARRAY(m, 1);\n+\tif (!m)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \tm->readers = readers;\n \tm->readers_len = n;\n \tm->min = first_min;\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 0179e4e73dd..98e7aa26373 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -598,6 +598,10 @@ int reftable_reader_new(struct reftable_reader **out,\n \tint err;\n \n \tREFTABLE_CALLOC_ARRAY(r, 1);\n+\tif (!r) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \n \t/*\n \t * We need one extra byte to read the type of first block. We also\n@@ -627,7 +631,11 @@ int reftable_reader_new(struct reftable_reader **out,\n \n \tr->size = file_size - footer_size(r->version);\n \tr->source = *source;\n-\tr->name = xstrdup(name);\n+\tr->name = reftable_strdup(name);\n+\tif (!r->name) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \tr->hash_id = 0;\n \tr->refcount = 1;\n \ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 990784d9d2f..7df28ab3438 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -116,6 +116,11 @@ static int fd_read_lines(int fd, char ***namesp)\n \t}\n \n \tREFTABLE_ALLOC_ARRAY(buf, size + 1);\n+\tif (!buf) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tif (read_in_full(fd, buf, size) != size) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n@@ -140,6 +145,8 @@ int read_lines(const char *filename, char ***namesp)\n \tif (fd < 0) {\n \t\tif (errno == ENOENT) {\n \t\t\tREFTABLE_CALLOC_ARRAY(*namesp, 1);\n+\t\t\tif (!*namesp)\n+\t\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \t\t\treturn 0;\n \t\t}\n \n@@ -420,6 +427,10 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\t\t}\n \n \t\t\tREFTABLE_CALLOC_ARRAY(names, 1);\n+\t\t\tif (!names) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto out;\n+\t\t\t}\n \t\t} else {\n \t\t\terr = fd_read_lines(fd, &names);\n \t\t\tif (err < 0)\n@@ -779,7 +790,11 @@ int reftable_stack_new_addition(struct reftable_addition **dest,\n {\n \tint err = 0;\n \tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n+\n \tREFTABLE_CALLOC_ARRAY(*dest, 1);\n+\tif (!*dest)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \t**dest = empty;\n \terr = reftable_stack_init_addition(*dest, st);\n \tif (err) {\n@@ -886,7 +901,12 @@ int reftable_addition_add(struct reftable_addition *add,\n \n \tREFTABLE_ALLOC_GROW(add->new_tables, add->new_tables_len + 1,\n \t\t\t add->new_tables_cap);\n+\tif (!add->new_tables) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \tadd->new_tables[add->new_tables_len++] = strbuf_detach(&next_name, NULL);\n+\n done:\n \tdelete_tempfile(&tab_file);\n \tstrbuf_release(&temp_tab_file_name);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 44bed92f467..ad4718edecb 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -49,8 +49,14 @@ static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n {\n \tint n = 0;\n \tif (w->pending_padding > 0) {\n-\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding, sizeof(*zeroed));\n-\t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n+\t\tuint8_t *zeroed;\n+\t\tint n;\n+\n+\t\tzeroed = reftable_calloc(w->pending_padding, sizeof(*zeroed));\n+\t\tif (!zeroed)\n+\t\t\treturn -1;\n+\n+\t\tn = w->write(w->write_arg, zeroed, w->pending_padding);\n \t\tif (n < 0)\n \t\t\treturn n;\n \n@@ -767,6 +773,9 @@ static int writer_flush_nonempty_block(struct reftable_writer *w)\n \t * case we will end up with a multi-level index.\n \t */\n \tREFTABLE_ALLOC_GROW(w->index, w->index_len + 1, w->index_cap);\n+\tif (!w->index)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \tindex_record.offset = w->next;\n \tstrbuf_reset(&index_record.last_key);\n \tstrbuf_addbuf(&index_record.last_key, &w->block_writer->last_key);\ndiff --git a/t/unit-tests/t-reftable-block.c b/t/unit-tests/t-reftable-block.c\nindex e52a612e852..d470060e8be 100644\n--- a/t/unit-tests/t-reftable-block.c\n+++ b/t/unit-tests/t-reftable-block.c\n@@ -32,6 +32,7 @@ static void t_ref_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n@@ -125,6 +126,7 @@ static void t_log_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size,\n@@ -214,6 +216,7 @@ static void t_obj_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size,\n@@ -297,6 +300,7 @@ static void t_index_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size,\ndiff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c\nindex 3d2848632db..3c84363e980 100644\n--- a/t/unit-tests/t-reftable-merged.c\n+++ b/t/unit-tests/t-reftable-merged.c\n@@ -29,7 +29,9 @@ merged_table_from_records(struct reftable_ref_record **refs,\n \tint err;\n \n \tREFTABLE_CALLOC_ARRAY(*readers, n);\n+\tcheck(*readers != NULL);\n \tREFTABLE_CALLOC_ARRAY(*source, n);\n+\tcheck(*source != NULL);\n \n \tfor (size_t i = 0; i < n; i++) {\n \t\tt_reftable_write_to_buf(&buf[i], refs[i], sizes[i], NULL, 0, &opts);\n@@ -285,7 +287,9 @@ merged_table_from_log_records(struct reftable_log_record **logs,\n \tint err;\n \n \tREFTABLE_CALLOC_ARRAY(*readers, n);\n+\tcheck(*readers != NULL);\n \tREFTABLE_CALLOC_ARRAY(*source, n);\n+\tcheck(*source != NULL);\n \n \tfor (size_t i = 0; i < n; i++) {\n \t\tt_reftable_write_to_buf(&buf[i], NULL, 0, logs[i], sizes[i], &opts);\ndiff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c\nindex acca927a2cf..bfa069caff7 100644\n--- a/t/unit-tests/t-reftable-readwrite.c\n+++ b/t/unit-tests/t-reftable-readwrite.c\n@@ -52,8 +52,11 @@ static void write_table(char ***names, struct strbuf *buf, int N,\n \tint i;\n \n \tREFTABLE_CALLOC_ARRAY(*names, N + 1);\n+\tcheck(*names != NULL);\n \tREFTABLE_CALLOC_ARRAY(refs, N);\n+\tcheck(refs != NULL);\n \tREFTABLE_CALLOC_ARRAY(logs, N);\n+\tcheck(logs != NULL);\n \n \tfor (i = 0; i < N; i++) {\n \t\trefs[i].refname = (*names)[i] = xstrfmt(\"refs/heads/branch%02d\", i);\n@@ -150,23 +153,25 @@ static void t_log_overflow(void)\n \n static void t_log_write_read(void)\n {\n-\tint N = 2;\n-\tchar **names = reftable_calloc(N + 1, sizeof(*names));\n-\tint err;\n \tstruct reftable_write_options opts = {\n \t\t.block_size = 256,\n \t};\n \tstruct reftable_ref_record ref = { 0 };\n-\tint i = 0;\n \tstruct reftable_log_record log = { 0 };\n-\tint n;\n \tstruct reftable_iterator it = { 0 };\n \tstruct reftable_reader *reader;\n \tstruct reftable_block_source source = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w = t_reftable_strbuf_writer(&buf, &opts);\n \tconst struct reftable_stats *stats = NULL;\n+\tint N = 2, err, i, n;\n+\tchar **names;\n+\n+\tnames = reftable_calloc(N + 1, sizeof(*names));\n+\tcheck(names != NULL);\n+\n \treftable_writer_set_limits(w, 0, N);\n+\n \tfor (i = 0; i < N; i++) {\n \t\tchar name[256];\n \t\tstruct reftable_ref_record ref = { 0 };\n@@ -178,6 +183,7 @@ static void t_log_write_read(void)\n \t\terr = reftable_writer_add_ref(w, &ref);\n \t\tcheck(!err);\n \t}\n+\n \tfor (i = 0; i < N; i++) {\n \t\tstruct reftable_log_record log = { 0 };\n \n@@ -476,8 +482,7 @@ static void t_table_read_write_seek_index(void)\n \n static void t_table_refs_for(int indexed)\n {\n-\tint N = 50;\n-\tchar **want_names = reftable_calloc(N + 1, sizeof(*want_names));\n+\tchar **want_names;\n \tint want_names_len = 0;\n \tuint8_t want_hash[GIT_SHA1_RAWSZ];\n \n@@ -485,15 +490,15 @@ static void t_table_refs_for(int indexed)\n \t\t.block_size = 256,\n \t};\n \tstruct reftable_ref_record ref = { 0 };\n-\tint i = 0;\n-\tint n;\n-\tint err;\n \tstruct reftable_reader *reader;\n \tstruct reftable_block_source source = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w = t_reftable_strbuf_writer(&buf, &opts);\n \tstruct reftable_iterator it = { 0 };\n-\tint j;\n+\tint N = 50, n, j, err, i;\n+\n+\twant_names = reftable_calloc(N + 1, sizeof(*want_names));\n+\tcheck(want_names != NULL);\n \n \tt_reftable_set_hash(want_hash, 4, GIT_SHA1_FORMAT_ID);\n \n-- \n2.46.0.551.gc5ee8f2d1c.dirty\n\n"},{"id":"503369","messageId":"xmqqy13huhuf.fsf@gitster.g","threadId":"62121","inReplyTo":"ZvJSyT8OsUVLln4A@pks.im","subject":"Re: [PATCH 04/22] reftable/basics: handle allocation failures in `reftable_calloc()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-09-24T16:39:04Z","receivedAt":"2024-09-24T16:39:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> In any case, there's another, bigger elephant in the room: `struct\n> strbuf`. I will have to introduce a reftable-specific buffer type for\n> this such that we can handle allocation failures here, too.\n\nOh, that one.\n\nIf you are letting your callers to register allocation callbacks,\nyou'd want to make sure any code you use, in cluding strbuf, would\ntake pieces of memory from and return them to the registered\nallocation routines.\n\n> While the\n> alternative would be to amend `struct strbuf` itself to do that, I don't\n> quite think that we should do it in \"core\" Git itself for now.\n\nPunting is of course an option, but what do the \"libification\" folks\nwant to see happen (Emily Cc'ed, not necessarily because she\npersonally did work in libification, but she knows who is responding\nto the list this week)? Wouldn't we eventually want these common\nthings not to die but to gracefully fail?\n"},{"id":"503370","messageId":"xmqqikulugwj.fsf@gitster.g","threadId":"62121","inReplyTo":"f6ad92ffd01c442dacd3ac6aa448891028636636.1727158127.git.ps@pks.im","subject":"Re: [PATCH v2 04/22] reftable/basics: handle allocation failures in `reftable_calloc()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-09-24T16:59:24Z","receivedAt":"2024-09-24T16:59:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> void *reftable_calloc(size_t nelem, size_t elsize)\n> {\n> -\tsize_t sz = st_mult(nelem, elsize);\n> -\tvoid *p = reftable_malloc(sz);\n> -\tmemset(p, 0, sz);\n> +\tvoid *p;\n> +\n> +\tif (nelem && elsize > SIZE_MAX / nelem)\n> +\t\treturn NULL;\n\nNow it is open coded, it strikes me that the check is a bit overly\nconservative.\n\nIf we are trying to allocate slightly than half of SIZE_MAX by\nasking elsize==1 and nelem==(SIZE_MAX / 2 + 10), we'd say that\n(elsize * nelem) would not fit size_t and fail the allocation.\n\nFor the purpose of this caller, it is not a practical issue, as it\nis likely that you'd not be able to obtain slightly more than half\nyour address space out of a single allocation anyway.\n\nBut it illustrates why open coding is not necessarily an excellent\nidea in the longer term, doesn't it? When unsigned_mult_overflows()\nis updated to avoid such a false positive, how would we remember\nthat we need to update this copy we?\n\n> +\tp = reftable_malloc(nelem * elsize);\n> +\tif (!p)\n> +\t\treturn NULL;\n> +\n> +\tmemset(p, 0, nelem * elsize);\n> \treturn p;\n> }\n"},{"id":"503444","messageId":"1abdf814-e75e-499b-b5ed-094b52cfca35@web.de","threadId":"62121","inReplyTo":"ad028020df717a57879cf3fdf5a3648bfebd2c78.1727158127.git.ps@pks.im","subject":"Re: [PATCH v2 05/22] reftable/basics: handle allocation failures in `parse_names()`","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2024-09-24T22:19:29Z","receivedAt":"2024-09-24T22:19:43Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 24.09.24 um 08:32 schrieb Patrick Steinhardt:\n> diff --git a/reftable/basics.c b/reftable/basics.c\n> index 3350bbffa23..82bfb807631 100644\n> --- a/reftable/basics.c\n> +++ b/reftable/basics.c\n> @@ -152,14 +152,26 @@ void parse_names(char *buf, int size, char ***namesp)\n> \t\t}\n> \t\tif (p < next) {\n> \t\t\tREFTABLE_ALLOC_GROW(names, names_len + 1, names_cap);\n> -\t\t\tnames[names_len++] = xstrdup(p);\n> +\t\t\tif (!names)\n> +\t\t\t\tgoto err;\n> +\n> +\t\t\tnames[names_len] = strdup(p);\n> +\t\t\tif (!names[names_len++])\n> +\t\t\t\tgoto err;\n> \t\t}\n> \t\tp = next + 1;\n> \t}\n>\n> \tREFTABLE_REALLOC_ARRAY(names, names_len + 1);\n> \tnames[names_len] = NULL;\n> -\t*namesp = names;\n> +\n> +\treturn names;\n> +\n> +err:\n> +\tfor (size_t i = 0; i < names_len; i++)\n> +\t\tfree(names[i]);\n> +\tfree(names);\n\nThese are allocated with reftable_realloc() under the hood, so they'd\nbetter be passed to reftable_free() instead, no?\n\nRené\n\n"},{"id":"503540","messageId":"ZvVO73dp4dbtxKi8@pks.im","threadId":"62121","inReplyTo":"1abdf814-e75e-499b-b5ed-094b52cfca35@web.de","subject":"Re: [PATCH v2 05/22] reftable/basics: handle allocation failures in `parse_names()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-26T12:09:29Z","receivedAt":"2024-09-26T12:09:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Sep 25, 2024 at 12:19:29AM +0200, René Scharfe wrote:\n> Am 24.09.24 um 08:32 schrieb Patrick Steinhardt:\n> > diff --git a/reftable/basics.c b/reftable/basics.c\n> > index 3350bbffa23..82bfb807631 100644\n> > --- a/reftable/basics.c\n> > +++ b/reftable/basics.c\n> > @@ -152,14 +152,26 @@ void parse_names(char *buf, int size, char ***namesp)\n> > \t\t}\n> > \t\tif (p < next) {\n> > \t\t\tREFTABLE_ALLOC_GROW(names, names_len + 1, names_cap);\n> > -\t\t\tnames[names_len++] = xstrdup(p);\n> > +\t\t\tif (!names)\n> > +\t\t\t\tgoto err;\n> > +\n> > +\t\t\tnames[names_len] = strdup(p);\n> > +\t\t\tif (!names[names_len++])\n> > +\t\t\t\tgoto err;\n> > \t\t}\n> > \t\tp = next + 1;\n> > \t}\n> >\n> > \tREFTABLE_REALLOC_ARRAY(names, names_len + 1);\n> > \tnames[names_len] = NULL;\n> > -\t*namesp = names;\n> > +\n> > +\treturn names;\n> > +\n> > +err:\n> > +\tfor (size_t i = 0; i < names_len; i++)\n> > +\t\tfree(names[i]);\n> > +\tfree(names);\n> \n> These are allocated with reftable_realloc() under the hood, so they'd\n> better be passed to reftable_free() instead, no?\n\nOh yeah, good catch! I'll wait a bit longer for feedback before sending\nout a revised version.\n\nPatrick\n"},{"id":"503541","messageId":"ZvVPiIzzLTTb75b8@pks.im","threadId":"62121","inReplyTo":"xmqqikulugwj.fsf@gitster.g","subject":"Re: [PATCH v2 04/22] reftable/basics: handle allocation failures in `reftable_calloc()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-26T12:11:52Z","receivedAt":"2024-09-26T12:12:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Sep 24, 2024 at 09:59:24AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > void *reftable_calloc(size_t nelem, size_t elsize)\n> > {\n> > -\tsize_t sz = st_mult(nelem, elsize);\n> > -\tvoid *p = reftable_malloc(sz);\n> > -\tmemset(p, 0, sz);\n> > +\tvoid *p;\n> > +\n> > +\tif (nelem && elsize > SIZE_MAX / nelem)\n> > +\t\treturn NULL;\n> \n> Now it is open coded, it strikes me that the check is a bit overly\n> conservative.\n> \n> If we are trying to allocate slightly than half of SIZE_MAX by\n> asking elsize==1 and nelem==(SIZE_MAX / 2 + 10), we'd say that\n> (elsize * nelem) would not fit size_t and fail the allocation.\n> \n> For the purpose of this caller, it is not a practical issue, as it\n> is likely that you'd not be able to obtain slightly more than half\n> your address space out of a single allocation anyway.\n> \n> But it illustrates why open coding is not necessarily an excellent\n> idea in the longer term, doesn't it? When unsigned_mult_overflows()\n> is updated to avoid such a false positive, how would we remember\n> that we need to update this copy we?\n\nI agree in general, but with the reftable library I'm stuck between a\nrock and a hard place. My goal is to make it fully reusable by other\nprojects without first having to do surgery on their side. While having\nsomething like `st_mult()` is simple enough to port over, the biggest\nproblem I have is that over time we sneak in more and more code from the\nGit codebase. The result is death by a thousand cuts.\n\nNow if we had a single header that exposes a small set of functions\nwithout _anything_ else it could work alright. But I rather doubt that\nwe really want to have a standalone header for `st_mult()` that we can\ninclude. But without such a standalone header it is simply too easy to\nstart building on top of Git features by accident.\n\nSo I'm instead aiming to go a different way and fully cut the reftable\ncode loose from Git. So even if we e.g. eventually want `struct strbuf`\nreturn errors on failure, it would only address one part of my problem.\n\nA couple months ago I said that I'll try to write something like shims\nthat wrap our types in reftable types such that other projects can\nprovide implementations for such shims themselves. I tried to make that\nwork, but the result was an unholy mess that really didn't make any\nsense whatsoever. Most of the features that I need from the Git codebase\ncan be provided in a couple of lines of code (`struct strbuf` is roughly\n50 lines for example), plus maybe a callback function that can be\nprovided to wire things up on the user side (`register_tempfiles()` for\nexample). So once I saw that those wrappers are harder to use and result\nin roughly the same lines of code I decided to scrap that approach and\ninstead try to convert it fully.\n\nSo yeah, overall we shouldn't open-code things like this. But I don't\nreally see another way to do this for the reftable library.\n\nPatrick\n"},{"id":"503573","messageId":"xmqqzfnul7fg.fsf@gitster.g","threadId":"62121","inReplyTo":"ZvVPiIzzLTTb75b8@pks.im","subject":"Re: [PATCH v2 04/22] reftable/basics: handle allocation failures in `reftable_calloc()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-09-26T16:13:23Z","receivedAt":"2024-09-26T16:13:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> But it illustrates why open coding is not necessarily an excellent\n>> idea in the longer term, doesn't it? When unsigned_mult_overflows()\n>> is updated to avoid such a false positive, how would we remember\n>> that we need to update this copy we?\n>\n> I agree in general, but with the reftable library I'm stuck between a\n> rock and a hard place. My goal is to make it fully reusable by other\n> projects without first having to do surgery on their side. While having\n> something like `st_mult()` is simple enough to port over, the biggest\n> problem I have is that over time we sneak in more and more code from the\n> Git codebase. The result is death by a thousand cuts.\n\n> Now if we had a single header that exposes a small set of functions\n> without _anything_ else it could work alright. But I rather doubt that\n> we really want to have a standalone header for `st_mult()` that we can\n> include. But without such a standalone header it is simply too easy to\n> start building on top of Git features by accident.\n>\n> So I'm instead aiming to go a different way and fully cut the reftable\n> code loose from Git. So even if we e.g. eventually want `struct strbuf`\n> return errors on failure, it would only address one part of my problem.\n\nThe dependency to \"strbuf\" (just as an example) was added initially\nfairly early. Soon after 27f7ed2a (reftable: add LICENSE,\n2021-10-07) added the reftable/ hierarchy, e303bf22 (reftable:\n(de)serialization for the polymorphic record type., 2021-10-07). I\nsomehow had an impression that reftable \"library\" started without\nany Git dependency and then use of our helper functions seemed\nthrough from the shim layer, but it was pretty much part of the\nlibrary from day one, it seems.\n\n> A couple months ago I said that I'll try to write something like shims\n> that wrap our types in reftable types such that other projects can\n> provide implementations for such shims themselves. I tried to make that\n> work, but the result was an unholy mess that really didn't make any\n> sense whatsoever. Most of the features that I need from the Git codebase\n> can be provided in a couple of lines of code (`struct strbuf` is roughly\n> 50 lines for example), plus maybe a callback function that can be\n> provided to wire things up on the user side (`register_tempfiles()` for\n> example). So once I saw that those wrappers are harder to use and result\n> in roughly the same lines of code I decided to scrap that approach and\n> instead try to convert it fully.\n>\n> So yeah, overall we shouldn't open-code things like this. But I don't\n> really see another way to do this for the reftable library.\n\nBut isn't all of the above what Libification ought to be about? I\nwas hoping that the reftable polishing would not have to be done by\nyou alone, and you would recruit those who are into libification of\nother parts of Git codebase to help cleaning up these fringe (from\nthe point of view of reftable) interfaces.\n\nWhat do the libification folks feel about this (folks involved in\nlibgit-sys CC'ed; of course all others who are interested in the\nlibification topic are welcome to comment)?\n\nThanks.\n"},{"id":"503603","messageId":"ZvZCdcpifMpmKajx@pks.im","threadId":"62121","inReplyTo":"xmqqzfnul7fg.fsf@gitster.g","subject":"Re: [PATCH v2 04/22] reftable/basics: handle allocation failures in `reftable_calloc()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-27T05:28:31Z","receivedAt":"2024-09-27T05:28:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Sep 26, 2024 at 09:13:23AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> >> But it illustrates why open coding is not necessarily an excellent\n> >> idea in the longer term, doesn't it? When unsigned_mult_overflows()\n> >> is updated to avoid such a false positive, how would we remember\n> >> that we need to update this copy we?\n> >\n> > I agree in general, but with the reftable library I'm stuck between a\n> > rock and a hard place. My goal is to make it fully reusable by other\n> > projects without first having to do surgery on their side. While having\n> > something like `st_mult()` is simple enough to port over, the biggest\n> > problem I have is that over time we sneak in more and more code from the\n> > Git codebase. The result is death by a thousand cuts.\n> \n> > Now if we had a single header that exposes a small set of functions\n> > without _anything_ else it could work alright. But I rather doubt that\n> > we really want to have a standalone header for `st_mult()` that we can\n> > include. But without such a standalone header it is simply too easy to\n> > start building on top of Git features by accident.\n> >\n> > So I'm instead aiming to go a different way and fully cut the reftable\n> > code loose from Git. So even if we e.g. eventually want `struct strbuf`\n> > return errors on failure, it would only address one part of my problem.\n> \n> The dependency to \"strbuf\" (just as an example) was added initially\n> fairly early. Soon after 27f7ed2a (reftable: add LICENSE,\n> 2021-10-07) added the reftable/ hierarchy, e303bf22 (reftable:\n> (de)serialization for the polymorphic record type., 2021-10-07). I\n> somehow had an impression that reftable \"library\" started without\n> any Git dependency and then use of our helper functions seemed\n> through from the shim layer, but it was pretty much part of the\n> library from day one, it seems.\n\nI think the history goes a bit different. Initially, the reftable\nlibrary was developed completely outside of the Git tree in [1]. It did\nnot have any external dependencies and didn't use any of the Git code.\n\nDuring the upstreaming process the discussion rather quickly swayed into\nthe direction of making Git the canonical upstream instead of using a\nseparate repository that hosts the code. See e.g. [2], which was still\ntracking a VERSION file to keep track of the upstream version that the\ncode was pulled from. In [3] the discussion started about upstream\nrequiring a CLA, which was an (understandable) non-starter. I cannot\nfind the statement, but eventually we decided that canonical upstream\nshould be Git, and the VERSION file was never committed.\n\nBut with that decision, now that the reftable library was part of Git,\nthe reviews also started to note that it really should use functions\nprovided by Git. The initial version of the patch series didn't yet have\nany of that. From my point of view that was a big mistake, as the result\nis that Git is _not_ reusable by other projects in its current state.\n\n[1]: https://github.com/google/reftable/tree/master/c\n[2]: <2106ff286b1135f9428529d9fc392edc127e960c.1580134944.git.gitgitgadget@gmail.com>\n[3]: <20200207001612.GF6573@camp.crustytoothpaste.net>\n\n> > A couple months ago I said that I'll try to write something like shims\n> > that wrap our types in reftable types such that other projects can\n> > provide implementations for such shims themselves. I tried to make that\n> > work, but the result was an unholy mess that really didn't make any\n> > sense whatsoever. Most of the features that I need from the Git codebase\n> > can be provided in a couple of lines of code (`struct strbuf` is roughly\n> > 50 lines for example), plus maybe a callback function that can be\n> > provided to wire things up on the user side (`register_tempfiles()` for\n> > example). So once I saw that those wrappers are harder to use and result\n> > in roughly the same lines of code I decided to scrap that approach and\n> > instead try to convert it fully.\n> >\n> > So yeah, overall we shouldn't open-code things like this. But I don't\n> > really see another way to do this for the reftable library.\n> \n> But isn't all of the above what Libification ought to be about? I\n> was hoping that the reftable polishing would not have to be done by\n> you alone, and you would recruit those who are into libification of\n> other parts of Git codebase to help cleaning up these fringe (from\n> the point of view of reftable) interfaces.\n> \n> What do the libification folks feel about this (folks involved in\n> libgit-sys CC'ed; of course all others who are interested in the\n> libification topic are welcome to comment)?\n\nI think it's orthogonal to that. The libification effort doesn't really\ncare about exposing low-level details like `st_mult()` as a library, or\neven the `strbuf` interface. It cares about making Git functionality\navailable, which is on a much higher conceptual level.\n\nFurthermore, to the best of my knowledge the intent wasn't ever to\nprovide the ability to take a couple of C files and have them be easy to\nbuild as part of another project. The goal is to have proper interfaces\nto our code, but whether the code itself has internal interdependencies\nis not really much of a concern (globals are a different story).\n\nMy goal is different: I want to `cp -r` the \"reftable/\" directory into\nlibgit2 or any other project that wants to implement a reftable backend\nwithout having to care about all the different dependencies that it may\npull in or having to carefully massage all of the includes.\n\nNow if it was a single standalone file that one would have to copy in\naddition to that I'd be fine. But \"strbuf.c' isn't that at all: it ropes\nin \"gettext.h\", \"hex-ll.h\", \"string-list.h\", \"utf-8.h\", \"date.h\". All of\nwhich is not needed at all, but the person who does the porting now has\nto carefully figure out what can and cannot be removed from both\n\"strbuf.h\" and \"strbuf.c\".\n\nSo... it gets out of hand really fast. We would have to significantly\nchange the way we develop the Git codebase if we wanted to make this a\ngood experience. But enforcing very strict coding guidelines onto all of\nour codebase doesn't feel reasonable to me, which is why I am instead\ntrying to reinstate the state where the reftable library was decoupled\nfrom the rest of the Git codebase.\n\nThe difference here is roughly 100 to 200 lines of code, which I don't\nthink is much of a maintenance burden by itself. In fact, we'll even\nstart to share the maintenance burden with libgit2, because they would\nbe able to use the exact same copy as we do and thus contribute back\nbugfixes and improvements for discoveries that their additional test\ncoverage brings us.\n\nI hope that this all clarifies my point of view :) I've also Cc'd\nHan-Wen in case I've made any mistakes regarding the original intent.\n\nPatrick\n"},{"id":"503614","messageId":"CAOw_e7YZNUa_Fu58C73kyc85f+_1Z1ew3p9=AekQGiqmXeQK5w@mail.gmail.com","threadId":"62121","inReplyTo":"ZvZCdcpifMpmKajx@pks.im","subject":"Re: [PATCH v2 04/22] reftable/basics: handle allocation failures in `reftable_calloc()`","fromName":"Han-Wen Nienhuys","fromEmail":"hanwenn@gmail.com","sentAt":"2024-09-27T12:21:27Z","receivedAt":"2024-09-27T12:21:41Z","isPatch":true,"sender":{"key":"hanwenn@gmail.com","avatar":"https://gravatar.com/avatar/058832acb8d613baeb6ce9d21b009d9772424a309b9a521330423daef909a27a?d=mp&s=160"},"body":"On Fri, Sep 27, 2024 at 11:31 AM Patrick Steinhardt wrote:\n> > The dependency to \"strbuf\" (just as an example) was added initially\n> > fairly early. Soon after 27f7ed2a (reftable: add LICENSE,\n> > 2021-10-07) added the reftable/ hierarchy, e303bf22 (reftable:\n> > (de)serialization for the polymorphic record type., 2021-10-07). I\n\nFor historical understanding, these commits are meaningless because\nthey were a bulk import. You should look at\nhttps://github.com/hanwen/reftable instead.\n\n> > somehow had an impression that reftable \"library\" started without\n> > any Git dependency and then use of our helper functions seemed\n> > through from the shim layer, but it was pretty much part of the\n> > library from day one, it seems.\n>\n> I think the history goes a bit different. Initially, the reftable\n> library was developed completely outside of the Git tree in [1]. It did\n> not have any external dependencies and didn't use any of the Git code.\n\nCorrect. strbuf was originally called 'slice', because it was a direct\ntranslation of the []byte type in Go. I renamed it to avoid offending\nthe sensibilities of Git reviewers. See here\nhttps://github.com/hanwen/reftable/commit/06d9728b4fedb9ed996ac1ae48343e3879114e1b\n. It was a change that made me a little sad, because the strbuf type\nonly works with explicit initialization (STRBUF_INIT),\nwhich means that every struct that transitively includes a strbuf must\nnow have a XXX_INIT macro. It just required 12 strbuf functions, but I\nsuppose more have crept in over time.\n\n> The difference here is roughly 100 to 200 lines of code, which I don't\n> think is much of a maintenance burden by itself. In fact, we'll even\n\nI think Patrick's plan is sound. The amount of code to make reftable\nwork both standalone and in Git is small. It just needs some\ndiscipline on both the Git and libgit2 side to not add additional\ndependencies.\n\n-- \nHan-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen\n"},{"id":"503616","messageId":"xmqqplopgm08.fsf@gitster.g","threadId":"62121","inReplyTo":"CAOw_e7YZNUa_Fu58C73kyc85f+_1Z1ew3p9=AekQGiqmXeQK5w@mail.gmail.com","subject":"Re: [PATCH v2 04/22] reftable/basics: handle allocation failures in `reftable_calloc()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-09-27T15:21:59Z","receivedAt":"2024-09-27T15:22:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Han-Wen Nienhuys writes:\n\n> I think Patrick's plan is sound. The amount of code to make reftable\n> work both standalone and in Git is small. It just needs some\n> discipline on both the Git and libgit2 side to not add additional\n> dependencies.\n\nThanks for a sanity check. Very much appreciated.\n"},{"id":"503673","messageId":"cover.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH v3 00/22] refatble: handle allocation errors","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:08:08Z","receivedAt":"2024-09-30T08:08:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the third version of this patch series that converts the\nreftable library to start handling allocation errors. This is done such\nthat the reftable library can truly behave like a library and let its\ncallers handle such conditions.\n\nThere is only a single change compared to v1, namely that we use\n`reftable_free()` instead of `free()`.\n\nThanks!\n\nPatrick\n\nPatrick Steinhardt (22):\n reftable/error: introduce out-of-memory error code\n reftable/basics: merge \"publicbasics\" into \"basics\"\n reftable: introduce `reftable_strdup()`\n reftable/basics: handle allocation failures in `reftable_calloc()`\n reftable/basics: handle allocation failures in `parse_names()`\n reftable/record: handle allocation failures on copy\n reftable/record: handle allocation failures when decoding records\n reftable/writer: handle allocation failures in `writer_index_hash()`\n reftable/writer: handle allocation failures in `reftable_new_writer()`\n reftable/merged: handle allocation failures in\n `merged_table_init_iter()`\n reftable/reader: handle allocation failures for unindexed reader\n reftable/reader: handle allocation failures in `reader_init_iter()`\n reftable/stack: handle allocation failures on reload\n reftable/stack: handle allocation failures in `reftable_new_stack()`\n reftable/stack: handle allocation failures in `stack_compact_range()`\n reftable/stack: handle allocation failures in auto compaction\n reftable/iter: handle allocation failures when creating indexed table\n iter\n reftable/blocksource: handle allocation failures\n reftable/block: handle allocation failures\n reftable/pq: handle allocation failures when adding entries\n reftable/tree: handle allocation failures\n reftable: handle trivial allocation failures\n\n Makefile | 1 -\n refs/reftable-backend.c | 39 ++++--\n reftable/basics.c | 92 ++++++++++++++-\n reftable/basics.h | 13 +-\n reftable/block.c | 23 +++-\n reftable/block.h | 4 +-\n reftable/blocksource.c | 25 +++-\n reftable/error.c | 2 +\n reftable/iter.c | 20 +++-\n reftable/iter.h | 2 +-\n reftable/merged.c | 84 ++++++++-----\n reftable/merged.h | 6 +-\n reftable/pq.c | 7 +-\n reftable/pq.h | 2 +-\n reftable/publicbasics.c | 66 -----------\n reftable/reader.c | 68 ++++++++---\n reftable/reader.h | 6 +-\n reftable/record.c | 164 +++++++++++++++++++-------\n reftable/record.h | 6 +-\n reftable/reftable-basics.h | 18 +++\n reftable/reftable-error.h | 3 +\n reftable/reftable-malloc.h | 18 ---\n reftable/reftable-merged.h | 8 +-\n reftable/reftable-reader.h | 8 +-\n reftable/reftable-stack.h | 8 +-\n reftable/reftable-writer.h | 12 +-\n reftable/stack.c | 177 ++++++++++++++++++++++------\n reftable/tree.c | 42 +++++--\n reftable/tree.h | 21 +++-\n reftable/writer.c | 150 +++++++++++++++--------\n t/helper/test-reftable.c | 10 +-\n t/unit-tests/lib-reftable.c | 8 +-\n t/unit-tests/t-reftable-basics.c | 11 +-\n t/unit-tests/t-reftable-block.c | 24 ++--\n t/unit-tests/t-reftable-merged.c | 16 ++-\n t/unit-tests/t-reftable-readwrite.c | 61 ++++++----\n t/unit-tests/t-reftable-stack.c | 4 +-\n t/unit-tests/t-reftable-tree.c | 10 +-\n 38 files changed, 853 insertions(+), 386 deletions(-)\n delete mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-basics.h\n delete mode 100644 reftable/reftable-malloc.h\n\nRange-diff against v2:\n 1: 8c99ecc325 = 1: 8c99ecc325 reftable/error: introduce out-of-memory error code\n 2: 4dcdf1d48e = 2: 4dcdf1d48e reftable/basics: merge \"publicbasics\" into \"basics\"\n 3: 21fa9b15d9 = 3: 21fa9b15d9 reftable: introduce `reftable_strdup()`\n 4: f6ad92ffd0 = 4: f6ad92ffd0 reftable/basics: handle allocation failures in `reftable_calloc()`\n 5: ad028020df ! 5: 922783708d reftable/basics: handle allocation failures in `parse_names()`\n @@ reftable/basics.c: void parse_names(char *buf, int size, char ***namesp)\n +\n +err:\n +\tfor (size_t i = 0; i < names_len; i++)\n -+\t\tfree(names[i]);\n -+\tfree(names);\n ++\t\treftable_free(names[i]);\n ++\treftable_free(names);\n +\treturn NULL;\n }\n \n 6: df713fbe08 = 6: d4676e8a6a reftable/record: handle allocation failures on copy\n 7: 870bb003c0 = 7: 874e6a1c20 reftable/record: handle allocation failures when decoding records\n 8: 1d47e42500 = 8: 42bc57fd00 reftable/writer: handle allocation failures in `writer_index_hash()`\n 9: caa71f0a77 = 9: 9edd1d84cd reftable/writer: handle allocation failures in `reftable_new_writer()`\n10: a84e9cadae = 10: d4004a7f43 reftable/merged: handle allocation failures in `merged_table_init_iter()`\n11: 20d3833014 = 11: b2bd142021 reftable/reader: handle allocation failures for unindexed reader\n12: e35c3a705d = 12: 9b5fd52862 reftable/reader: handle allocation failures in `reader_init_iter()`\n13: ca3b57f151 = 13: 0f3e3d1585 reftable/stack: handle allocation failures on reload\n14: 7377421a63 = 14: c88645a251 reftable/stack: handle allocation failures in `reftable_new_stack()`\n15: 244e8667c5 = 15: 92ad4fc934 reftable/stack: handle allocation failures in `stack_compact_range()`\n16: 99f4868c38 = 16: 19acebd919 reftable/stack: handle allocation failures in auto compaction\n17: 271839a626 = 17: 620658bffc reftable/iter: handle allocation failures when creating indexed table iter\n18: c41808e9d7 = 18: 48047dd4c9 reftable/blocksource: handle allocation failures\n19: 9348b0a457 = 19: 08685605ba reftable/block: handle allocation failures\n20: 445daf9464 = 20: a66937b7af reftable/pq: handle allocation failures when adding entries\n21: c31f9e53f5 = 21: c291114553 reftable/tree: handle allocation failures\n22: d0fe999371 = 22: 20f50c446a reftable: handle trivial allocation failures\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503674","messageId":"8c99ecc3255b1b10cdec047da01804dc23a7d287.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 01/22] reftable/error: introduce out-of-memory error code","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:08:11Z","receivedAt":"2024-09-30T08:08:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library does not use the same memory allocation functions\nas the rest of the Git codebase. Instead, as the reftable library is\nsupposed to be usable as a standalone library without Git, it provides a\nset of pluggable memory allocators.\n\nCompared to `xmalloc()` and friends these allocators are _not_ expected\nto die when an allocation fails. This design choice is concious, as a\nlibrary should leave it to its caller to handle any kind of error. While\nit is very likely that the caller cannot really do much in the case of\nan out-of-memory situation anyway, we are not the ones to make that\ndecision.\n\nCuriously though, we never handle allocation errors even though memory\nallocation functions are allowed to fail. And as we do not plug in Git's\nmemory allocator via `reftable_set_alloc()` either the consequence is\nthat we'd instead segfault as soon as we run out of memory.\n\nWhile the easy fix would be to wire up `xmalloc()` and friends, it\nwould only fix the usage of the reftable library in Git itself. Other\nusers like libgit2, which is about to revive its efforts to land a\nbackend for reftables, wouldn't be able to benefit from this solution.\n\nInstead, we are about to do it the hard way: adapt all allocation sites\nto perform error checking. Introduce a new error code for out-of-memory\nerrors that we will wire up in subsequent steps.\n\nThis commit also serves as the motivator for all the remaining steps in\nthis series such that we do not have to repeat the same arguments in\nevery single subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/error.c | 2 ++\n reftable/reftable-error.h | 3 +++\n 2 files changed, 5 insertions(+)\n\ndiff --git a/reftable/error.c b/reftable/error.c\nindex a25f28a43e..660d029617 100644\n--- a/reftable/error.c\n+++ b/reftable/error.c\n@@ -35,6 +35,8 @@ const char *reftable_error_str(int err)\n \t\treturn \"entry too large\";\n \tcase REFTABLE_OUTDATED_ERROR:\n \t\treturn \"data concurrently modified\";\n+\tcase REFTABLE_OUT_OF_MEMORY_ERROR:\n+\t\treturn \"out of memory\";\n \tcase -1:\n \t\treturn \"general error\";\n \tdefault:\ndiff --git a/reftable/reftable-error.h b/reftable/reftable-error.h\nindex 6368cd9ed9..f404826562 100644\n--- a/reftable/reftable-error.h\n+++ b/reftable/reftable-error.h\n@@ -57,6 +57,9 @@ enum reftable_error {\n \n \t/* Trying to write out-of-date data. */\n \tREFTABLE_OUTDATED_ERROR = -12,\n+\n+\t/* An allocation has failed due to an out-of-memory situation. */\n+\tREFTABLE_OUT_OF_MEMORY_ERROR = -13,\n };\n \n /* convert the numeric error code to a string. The string should not be\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503675","messageId":"4dcdf1d48ecf9a05cfa643d5a6165240898811a7.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 02/22] reftable/basics: merge \"publicbasics\" into \"basics\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:08:13Z","receivedAt":"2024-09-30T08:08:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The split between \"basics\" and \"publicbasics\" is somewhat arbitrary and\nnot in line with how we typically structure code in the reftable\nlibrary. While we do indeed split up headers into a public and internal\npart, we don't do that for the compilation unit itself. Furthermore, the\ndeclarations for \"publicbasics.c\" are in \"reftable-malloc.h\", which\nisn't in line with our naming schema, either.\n\nFix these inconsistencies by:\n\n - Merging \"publicbasics.c\" into \"basics.c\".\n\n - Renaming \"reftable-malloc.h\" to \"reftable-basics.h\" as the public\n header.\n\nSigned-off-by: Patrick Steinhardt \n---\n Makefile | 1 -\n reftable/basics.c | 55 +++++++++++++++++++++++++++++++\n reftable/basics.h | 3 ++\n reftable/publicbasics.c | 66 --------------------------------------\n reftable/reftable-basics.h | 18 +++++++++++\n reftable/reftable-malloc.h | 18 -----------\n 6 files changed, 76 insertions(+), 85 deletions(-)\n delete mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-basics.h\n delete mode 100644 reftable/reftable-malloc.h\n\ndiff --git a/Makefile b/Makefile\nindex e3abf0ba83..39b10923d4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2683,7 +2683,6 @@ REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n-REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/merged.o\n REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 0058619ca6..cf072935c8 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -7,6 +7,49 @@ license that can be found in the LICENSE file or at\n */\n \n #include \"basics.h\"\n+#include \"reftable-basics.h\"\n+\n+static void *(*reftable_malloc_ptr)(size_t sz);\n+static void *(*reftable_realloc_ptr)(void *, size_t);\n+static void (*reftable_free_ptr)(void *);\n+\n+void *reftable_malloc(size_t sz)\n+{\n+\tif (reftable_malloc_ptr)\n+\t\treturn (*reftable_malloc_ptr)(sz);\n+\treturn malloc(sz);\n+}\n+\n+void *reftable_realloc(void *p, size_t sz)\n+{\n+\tif (reftable_realloc_ptr)\n+\t\treturn (*reftable_realloc_ptr)(p, sz);\n+\treturn realloc(p, sz);\n+}\n+\n+void reftable_free(void *p)\n+{\n+\tif (reftable_free_ptr)\n+\t\treftable_free_ptr(p);\n+\telse\n+\t\tfree(p);\n+}\n+\n+void *reftable_calloc(size_t nelem, size_t elsize)\n+{\n+\tsize_t sz = st_mult(nelem, elsize);\n+\tvoid *p = reftable_malloc(sz);\n+\tmemset(p, 0, sz);\n+\treturn p;\n+}\n+\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n+{\n+\treftable_malloc_ptr = malloc;\n+\treftable_realloc_ptr = realloc;\n+\treftable_free_ptr = free;\n+}\n \n void put_be24(uint8_t *out, uint32_t i)\n {\n@@ -121,3 +164,15 @@ int common_prefix_size(struct strbuf *a, struct strbuf *b)\n \n \treturn p;\n }\n+\n+int hash_size(uint32_t id)\n+{\n+\tswitch (id) {\n+\tcase 0:\n+\tcase GIT_SHA1_FORMAT_ID:\n+\t\treturn GIT_SHA1_RAWSZ;\n+\tcase GIT_SHA256_FORMAT_ID:\n+\t\treturn GIT_SHA256_RAWSZ;\n+\t}\n+\tabort();\n+}\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex c8fec68d4e..4e2e76014a 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -14,6 +14,7 @@ license that can be found in the LICENSE file or at\n */\n \n #include \"system.h\"\n+#include \"reftable-basics.h\"\n \n /* Bigendian en/decoding of integers */\n \n@@ -71,4 +72,6 @@ void *reftable_calloc(size_t nelem, size_t elsize);\n struct strbuf;\n int common_prefix_size(struct strbuf *a, struct strbuf *b);\n \n+int hash_size(uint32_t id);\n+\n #endif\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\ndeleted file mode 100644\nindex 44b84a125e..0000000000\n--- a/reftable/publicbasics.c\n+++ /dev/null\n@@ -1,66 +0,0 @@\n-/*\n-Copyright 2020 Google LLC\n-\n-Use of this source code is governed by a BSD-style\n-license that can be found in the LICENSE file or at\n-https://developers.google.com/open-source/licenses/bsd\n-*/\n-\n-#include \"system.h\"\n-#include \"reftable-malloc.h\"\n-\n-#include \"basics.h\"\n-\n-static void *(*reftable_malloc_ptr)(size_t sz);\n-static void *(*reftable_realloc_ptr)(void *, size_t);\n-static void (*reftable_free_ptr)(void *);\n-\n-void *reftable_malloc(size_t sz)\n-{\n-\tif (reftable_malloc_ptr)\n-\t\treturn (*reftable_malloc_ptr)(sz);\n-\treturn malloc(sz);\n-}\n-\n-void *reftable_realloc(void *p, size_t sz)\n-{\n-\tif (reftable_realloc_ptr)\n-\t\treturn (*reftable_realloc_ptr)(p, sz);\n-\treturn realloc(p, sz);\n-}\n-\n-void reftable_free(void *p)\n-{\n-\tif (reftable_free_ptr)\n-\t\treftable_free_ptr(p);\n-\telse\n-\t\tfree(p);\n-}\n-\n-void *reftable_calloc(size_t nelem, size_t elsize)\n-{\n-\tsize_t sz = st_mult(nelem, elsize);\n-\tvoid *p = reftable_malloc(sz);\n-\tmemset(p, 0, sz);\n-\treturn p;\n-}\n-\n-void reftable_set_alloc(void *(*malloc)(size_t),\n-\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n-{\n-\treftable_malloc_ptr = malloc;\n-\treftable_realloc_ptr = realloc;\n-\treftable_free_ptr = free;\n-}\n-\n-int hash_size(uint32_t id)\n-{\n-\tswitch (id) {\n-\tcase 0:\n-\tcase GIT_SHA1_FORMAT_ID:\n-\t\treturn GIT_SHA1_RAWSZ;\n-\tcase GIT_SHA256_FORMAT_ID:\n-\t\treturn GIT_SHA256_RAWSZ;\n-\t}\n-\tabort();\n-}\ndiff --git a/reftable/reftable-basics.h b/reftable/reftable-basics.h\nnew file mode 100644\nindex 0000000000..6e8e636b71\n--- /dev/null\n+++ b/reftable/reftable-basics.h\n@@ -0,0 +1,18 @@\n+/*\n+ * Copyright 2020 Google LLC\n+ *\n+ * Use of this source code is governed by a BSD-style\n+ * license that can be found in the LICENSE file or at\n+ * https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_BASICS_H\n+#define REFTABLE_BASICS_H\n+\n+#include \n+\n+/* Overrides the functions to use for memory management. */\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n+\n+#endif\ndiff --git a/reftable/reftable-malloc.h b/reftable/reftable-malloc.h\ndeleted file mode 100644\nindex 5f2185f1f3..0000000000\n--- a/reftable/reftable-malloc.h\n+++ /dev/null\n@@ -1,18 +0,0 @@\n-/*\n-Copyright 2020 Google LLC\n-\n-Use of this source code is governed by a BSD-style\n-license that can be found in the LICENSE file or at\n-https://developers.google.com/open-source/licenses/bsd\n-*/\n-\n-#ifndef REFTABLE_H\n-#define REFTABLE_H\n-\n-#include \n-\n-/* Overrides the functions to use for memory management. */\n-void reftable_set_alloc(void *(*malloc)(size_t),\n-\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n-\n-#endif\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503676","messageId":"21fa9b15d96809728455943877c1b32781364041.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 03/22] reftable: introduce `reftable_strdup()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:08:16Z","receivedAt":"2024-09-30T08:08:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library provides the ability to swap out allocators. There\nis a gap here though, because we continue to use `xstrdup()` even in the\ncase where all the other allocators have been swapped out.\n\nIntroduce `reftable_strdup()` that uses `reftable_malloc()` to do the\nallocation.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 10 ++++++++++\n reftable/basics.h | 1 +\n 2 files changed, 11 insertions(+)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex cf072935c8..4adc98cf5d 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -43,6 +43,16 @@ void *reftable_calloc(size_t nelem, size_t elsize)\n \treturn p;\n }\n \n+char *reftable_strdup(const char *str)\n+{\n+\tsize_t len = strlen(str);\n+\tchar *result = reftable_malloc(len + 1);\n+\tif (!result)\n+\t\treturn NULL;\n+\tmemcpy(result, str, len + 1);\n+\treturn result;\n+}\n+\n void reftable_set_alloc(void *(*malloc)(size_t),\n \t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n {\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 4e2e76014a..f107e14860 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -54,6 +54,7 @@ void *reftable_malloc(size_t sz);\n void *reftable_realloc(void *p, size_t sz);\n void reftable_free(void *p);\n void *reftable_calloc(size_t nelem, size_t elsize);\n+char *reftable_strdup(const char *str);\n \n #define REFTABLE_ALLOC_ARRAY(x, alloc) (x) = reftable_malloc(st_mult(sizeof(*(x)), (alloc)))\n #define REFTABLE_CALLOC_ARRAY(x, alloc) (x) = reftable_calloc((alloc), sizeof(*(x)))\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503677","messageId":"f6ad92ffd01c442dacd3ac6aa448891028636636.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 04/22] reftable/basics: handle allocation failures in `reftable_calloc()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:08:19Z","receivedAt":"2024-09-30T08:08:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_calloc()`.\n\nWhile at it, remove our use of `st_mult()` that would cause us to die on\nan overflow. From the caller's point of view there is not much of a\ndifference between arguments that are too large to be multiplied and a\nrequest that is too big to handle by the allocator: in both cases the\nallocation cannot be fulfilled. And in neither of these cases do we want\nthe reftable library to die.\n\nWhile we could use `unsigned_mult_overflows()` to handle the overflow\ngracefully, we instead open-code it to further our goal of converting\nthe reftable codebase to become a standalone library that can be reused\nby external projects.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 13 ++++++++++---\n 1 file changed, 10 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 4adc98cf5d..3350bbffa2 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -37,9 +37,16 @@ void reftable_free(void *p)\n \n void *reftable_calloc(size_t nelem, size_t elsize)\n {\n-\tsize_t sz = st_mult(nelem, elsize);\n-\tvoid *p = reftable_malloc(sz);\n-\tmemset(p, 0, sz);\n+\tvoid *p;\n+\n+\tif (nelem && elsize > SIZE_MAX / nelem)\n+\t\treturn NULL;\n+\n+\tp = reftable_malloc(nelem * elsize);\n+\tif (!p)\n+\t\treturn NULL;\n+\n+\tmemset(p, 0, nelem * elsize);\n \treturn p;\n }\n \n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503678","messageId":"922783708def23c7e0c73db846076cf5e242154b.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 05/22] reftable/basics: handle allocation failures in `parse_names()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:08:24Z","receivedAt":"2024-09-30T08:08:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `parse_names()` by returning `NULL` in\ncase any allocation fails. While at it, refactor the function to return\nthe array directly instead of assigning it to an out-pointer.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 20 ++++++++++++++++----\n reftable/basics.h | 9 ++++++---\n reftable/stack.c | 6 +++++-\n t/unit-tests/t-reftable-basics.c | 11 ++++++-----\n 4 files changed, 33 insertions(+), 13 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 3350bbffa2..5ae6e6acdd 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -135,14 +135,14 @@ size_t names_length(const char **names)\n \treturn p - names;\n }\n \n-void parse_names(char *buf, int size, char ***namesp)\n+char **parse_names(char *buf, int size)\n {\n \tchar **names = NULL;\n \tsize_t names_cap = 0;\n \tsize_t names_len = 0;\n-\n \tchar *p = buf;\n \tchar *end = buf + size;\n+\n \twhile (p < end) {\n \t\tchar *next = strchr(p, '\\n');\n \t\tif (next && next < end) {\n@@ -152,14 +152,26 @@ void parse_names(char *buf, int size, char ***namesp)\n \t\t}\n \t\tif (p < next) {\n \t\t\tREFTABLE_ALLOC_GROW(names, names_len + 1, names_cap);\n-\t\t\tnames[names_len++] = xstrdup(p);\n+\t\t\tif (!names)\n+\t\t\t\tgoto err;\n+\n+\t\t\tnames[names_len] = strdup(p);\n+\t\t\tif (!names[names_len++])\n+\t\t\t\tgoto err;\n \t\t}\n \t\tp = next + 1;\n \t}\n \n \tREFTABLE_REALLOC_ARRAY(names, names_len + 1);\n \tnames[names_len] = NULL;\n-\t*namesp = names;\n+\n+\treturn names;\n+\n+err:\n+\tfor (size_t i = 0; i < names_len; i++)\n+\t\treftable_free(names[i]);\n+\treftable_free(names);\n+\treturn NULL;\n }\n \n int names_equal(const char **a, const char **b)\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex f107e14860..69adeab2e4 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -38,9 +38,12 @@ size_t binsearch(size_t sz, int (*f)(size_t k, void *args), void *args);\n */\n void free_names(char **a);\n \n-/* parse a newline separated list of names. `size` is the length of the buffer,\n- * without terminating '\\0'. Empty names are discarded. */\n-void parse_names(char *buf, int size, char ***namesp);\n+/*\n+ * Parse a newline separated list of names. `size` is the length of the buffer,\n+ * without terminating '\\0'. Empty names are discarded. Returns a `NULL`\n+ * pointer when allocations fail.\n+ */\n+char **parse_names(char *buf, int size);\n \n /* compares two NULL-terminated arrays of strings. */\n int names_equal(const char **a, const char **b);\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex ce0a35216b..498fae846d 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -108,7 +108,11 @@ static int fd_read_lines(int fd, char ***namesp)\n \t}\n \tbuf[size] = 0;\n \n-\tparse_names(buf, size, namesp);\n+\t*namesp = parse_names(buf, size);\n+\tif (!*namesp) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \n done:\n \treftable_free(buf);\ndiff --git a/t/unit-tests/t-reftable-basics.c b/t/unit-tests/t-reftable-basics.c\nindex e5556ebf52..1fa77b6faf 100644\n--- a/t/unit-tests/t-reftable-basics.c\n+++ b/t/unit-tests/t-reftable-basics.c\n@@ -72,13 +72,14 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \tif_test (\"parse_names works for basic input\") {\n \t\tchar in1[] = \"line\\n\";\n \t\tchar in2[] = \"a\\nb\\nc\";\n-\t\tchar **out = NULL;\n-\t\tparse_names(in1, strlen(in1), &out);\n+\t\tchar **out = parse_names(in1, strlen(in1));\n+\t\tcheck(out != NULL);\n \t\tcheck_str(out[0], \"line\");\n \t\tcheck(!out[1]);\n \t\tfree_names(out);\n \n-\t\tparse_names(in2, strlen(in2), &out);\n+\t\tout = parse_names(in2, strlen(in2));\n+\t\tcheck(out != NULL);\n \t\tcheck_str(out[0], \"a\");\n \t\tcheck_str(out[1], \"b\");\n \t\tcheck_str(out[2], \"c\");\n@@ -88,8 +89,8 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \n \tif_test (\"parse_names drops empty string\") {\n \t\tchar in[] = \"a\\n\\nb\\n\";\n-\t\tchar **out = NULL;\n-\t\tparse_names(in, strlen(in), &out);\n+\t\tchar **out = parse_names(in, strlen(in));\n+\t\tcheck(out != NULL);\n \t\tcheck_str(out[0], \"a\");\n \t\t/* simply '\\n' should be dropped as empty string */\n \t\tcheck_str(out[1], \"b\");\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503679","messageId":"d4676e8a6a952ff43761b305fd77915e26b692f9.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 06/22] reftable/record: handle allocation failures on copy","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:08:26Z","receivedAt":"2024-09-30T08:08:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when copying records. While at it, convert\nfrom `xstrdup()` to `reftable_strdup()`. Adapt callsites to check for\nerror codes.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/record.c | 84 +++++++++++++++++++++++++++++++++--------------\n reftable/record.h | 6 ++--\n 2 files changed, 63 insertions(+), 27 deletions(-)\n\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 6b5a075b92..60fd33c9c9 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -215,13 +215,14 @@ static void reftable_ref_record_key(const void *r, struct strbuf *dest)\n \tstrbuf_addstr(dest, rec->refname);\n }\n \n-static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size)\n+static int reftable_ref_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n {\n \tstruct reftable_ref_record *ref = rec;\n \tconst struct reftable_ref_record *src = src_rec;\n \tchar *refname = NULL;\n \tsize_t refname_cap = 0;\n+\tint err;\n \n \tassert(hash_size > 0);\n \n@@ -236,6 +237,11 @@ static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n \n \t\tREFTABLE_ALLOC_GROW(ref->refname, refname_len + 1,\n \t\t\t\t ref->refname_cap);\n+\t\tif (!ref->refname) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n+\n \t\tmemcpy(ref->refname, src->refname, refname_len);\n \t\tref->refname[refname_len] = 0;\n \t}\n@@ -254,9 +260,17 @@ static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n \t\t src->value.val2.target_value, hash_size);\n \t\tbreak;\n \tcase REFTABLE_REF_SYMREF:\n-\t\tref->value.symref = xstrdup(src->value.symref);\n+\t\tref->value.symref = reftable_strdup(src->value.symref);\n+\t\tif (!ref->value.symref) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n \t\tbreak;\n \t}\n+\n+\terr = 0;\n+out:\n+\treturn err;\n }\n \n static void reftable_ref_record_release_void(void *rec)\n@@ -457,23 +471,28 @@ static void reftable_obj_record_release(void *rec)\n \tmemset(obj, 0, sizeof(struct reftable_obj_record));\n }\n \n-static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size UNUSED)\n+static int reftable_obj_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size UNUSED)\n {\n \tstruct reftable_obj_record *obj = rec;\n-\tconst struct reftable_obj_record *src =\n-\t\t(const struct reftable_obj_record *)src_rec;\n+\tconst struct reftable_obj_record *src = src_rec;\n \n \treftable_obj_record_release(obj);\n \n \tREFTABLE_ALLOC_ARRAY(obj->hash_prefix, src->hash_prefix_len);\n+\tif (!obj->hash_prefix)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tobj->hash_prefix_len = src->hash_prefix_len;\n \tif (src->hash_prefix_len)\n \t\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n \n \tREFTABLE_ALLOC_ARRAY(obj->offsets, src->offset_len);\n+\tif (!obj->offsets)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tobj->offset_len = src->offset_len;\n \tCOPY_ARRAY(obj->offsets, src->offsets, src->offset_len);\n+\n+\treturn 0;\n }\n \n static uint8_t reftable_obj_record_val_type(const void *rec)\n@@ -646,33 +665,44 @@ static void reftable_log_record_key(const void *r, struct strbuf *dest)\n \tstrbuf_add(dest, i64, sizeof(i64));\n }\n \n-static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size)\n+static int reftable_log_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n {\n \tstruct reftable_log_record *dst = rec;\n \tconst struct reftable_log_record *src =\n \t\t(const struct reftable_log_record *)src_rec;\n+\tint ret;\n \n \treftable_log_record_release(dst);\n \t*dst = *src;\n+\n \tif (dst->refname) {\n-\t\tdst->refname = xstrdup(dst->refname);\n+\t\tdst->refname = reftable_strdup(dst->refname);\n+\t\tif (!dst->refname) {\n+\t\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n \t}\n+\n \tswitch (dst->value_type) {\n \tcase REFTABLE_LOG_DELETION:\n \t\tbreak;\n \tcase REFTABLE_LOG_UPDATE:\n-\t\tif (dst->value.update.email) {\n+\t\tif (dst->value.update.email)\n \t\t\tdst->value.update.email =\n-\t\t\t\txstrdup(dst->value.update.email);\n-\t\t}\n-\t\tif (dst->value.update.name) {\n+\t\t\t\treftable_strdup(dst->value.update.email);\n+\t\tif (dst->value.update.name)\n \t\t\tdst->value.update.name =\n-\t\t\t\txstrdup(dst->value.update.name);\n-\t\t}\n-\t\tif (dst->value.update.message) {\n+\t\t\t\treftable_strdup(dst->value.update.name);\n+\t\tif (dst->value.update.message)\n \t\t\tdst->value.update.message =\n-\t\t\t\txstrdup(dst->value.update.message);\n+\t\t\t\treftable_strdup(dst->value.update.message);\n+\n+\t\tif (!dst->value.update.email ||\n+\t\t !dst->value.update.name ||\n+\t\t !dst->value.update.message) {\n+\t\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n \t\t}\n \n \t\tmemcpy(dst->value.update.new_hash,\n@@ -681,6 +711,10 @@ static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n \t\t src->value.update.old_hash, hash_size);\n \t\tbreak;\n \t}\n+\n+\tret = 0;\n+out:\n+\treturn ret;\n }\n \n static void reftable_log_record_release_void(void *rec)\n@@ -954,8 +988,8 @@ static void reftable_index_record_key(const void *r, struct strbuf *dest)\n \tstrbuf_addbuf(dest, &rec->last_key);\n }\n \n-static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size UNUSED)\n+static int reftable_index_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size UNUSED)\n {\n \tstruct reftable_index_record *dst = rec;\n \tconst struct reftable_index_record *src = src_rec;\n@@ -963,6 +997,8 @@ static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n \tstrbuf_reset(&dst->last_key);\n \tstrbuf_addbuf(&dst->last_key, &src->last_key);\n \tdst->offset = src->offset;\n+\n+\treturn 0;\n }\n \n static void reftable_index_record_release(void *rec)\n@@ -1054,14 +1090,14 @@ int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n \t\t\t\t\t\t dest, hash_size);\n }\n \n-void reftable_record_copy_from(struct reftable_record *rec,\n+int reftable_record_copy_from(struct reftable_record *rec,\n \t\t\t struct reftable_record *src, int hash_size)\n {\n \tassert(src->type == rec->type);\n \n-\treftable_record_vtable(rec)->copy_from(reftable_record_data(rec),\n-\t\t\t\t\t reftable_record_data(src),\n-\t\t\t\t\t hash_size);\n+\treturn reftable_record_vtable(rec)->copy_from(reftable_record_data(rec),\n+\t\t\t\t\t\t reftable_record_data(src),\n+\t\t\t\t\t\t hash_size);\n }\n \n uint8_t reftable_record_val_type(struct reftable_record *rec)\ndiff --git a/reftable/record.h b/reftable/record.h\nindex 5003bacdb0..0f53ba5443 100644\n--- a/reftable/record.h\n+++ b/reftable/record.h\n@@ -44,7 +44,7 @@ struct reftable_record_vtable {\n \t/* The record type of ('r' for ref). */\n \tuint8_t type;\n \n-\tvoid (*copy_from)(void *dest, const void *src, int hash_size);\n+\tint (*copy_from)(void *dest, const void *src, int hash_size);\n \n \t/* a value of [0..7], indicating record subvariants (eg. ref vs. symref\n \t * vs ref deletion) */\n@@ -137,8 +137,8 @@ void reftable_record_init(struct reftable_record *rec, uint8_t typ);\n int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b);\n int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);\n void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);\n-void reftable_record_copy_from(struct reftable_record *rec,\n-\t\t\t struct reftable_record *src, int hash_size);\n+int reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size);\n uint8_t reftable_record_val_type(struct reftable_record *rec);\n int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n \t\t\t int hash_size);\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503680","messageId":"874e6a1c207b9684e533ce8372d2537c07abf43a.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 07/22] reftable/record: handle allocation failures when decoding records","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:08:29Z","receivedAt":"2024-09-30T08:08:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when decoding records. While at it, fix some\nerror codes to be `REFTABLE_FORMAT_ERROR`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/record.c | 80 +++++++++++++++++++++++++++++++++++++----------\n 1 file changed, 63 insertions(+), 17 deletions(-)\n\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 60fd33c9c9..787e134c9a 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -359,7 +359,7 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tuint64_t update_index = 0;\n \tconst char *refname = NULL;\n \tsize_t refname_cap = 0;\n-\tint n;\n+\tint n, err;\n \n \tassert(hash_size > 0);\n \n@@ -375,6 +375,10 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tSWAP(r->refname_cap, refname_cap);\n \n \tREFTABLE_ALLOC_GROW(r->refname, key.len + 1, r->refname_cap);\n+\tif (!r->refname) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \tmemcpy(r->refname, key.buf, key.len);\n \tr->refname[key.len] = 0;\n \n@@ -383,7 +387,8 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tswitch (val_type) {\n \tcase REFTABLE_REF_VAL1:\n \t\tif (in.len < hash_size) {\n-\t\t\treturn -1;\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n \t\t}\n \n \t\tmemcpy(r->value.val1, in.buf, hash_size);\n@@ -392,7 +397,8 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \n \tcase REFTABLE_REF_VAL2:\n \t\tif (in.len < 2 * hash_size) {\n-\t\t\treturn -1;\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n \t\t}\n \n \t\tmemcpy(r->value.val2.value, in.buf, hash_size);\n@@ -405,7 +411,8 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tcase REFTABLE_REF_SYMREF: {\n \t\tint n = decode_string(scratch, in);\n \t\tif (n < 0) {\n-\t\t\treturn -1;\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n \t\t}\n \t\tstring_view_consume(&in, n);\n \t\tr->value.symref = strbuf_detach(scratch, NULL);\n@@ -419,6 +426,9 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \t}\n \n \treturn start.len - in.len;\n+\n+done:\n+\treturn err;\n }\n \n static int reftable_ref_record_is_deletion_void(const void *p)\n@@ -552,6 +562,8 @@ static int reftable_obj_record_decode(void *rec, struct strbuf key,\n \treftable_obj_record_release(r);\n \n \tREFTABLE_ALLOC_ARRAY(r->hash_prefix, key.len);\n+\tif (!r->hash_prefix)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tmemcpy(r->hash_prefix, key.buf, key.len);\n \tr->hash_prefix_len = key.len;\n \n@@ -570,6 +582,8 @@ static int reftable_obj_record_decode(void *rec, struct strbuf key,\n \t\treturn start.len - in.len;\n \n \tREFTABLE_ALLOC_ARRAY(r->offsets, count);\n+\tif (!r->offsets)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tr->offset_len = count;\n \n \tn = get_var_int(&r->offsets[0], &in);\n@@ -801,12 +815,17 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \tstruct reftable_log_record *r = rec;\n \tuint64_t max = 0;\n \tuint64_t ts = 0;\n-\tint n;\n+\tint err, n;\n \n \tif (key.len <= 9 || key.buf[key.len - 9] != 0)\n \t\treturn REFTABLE_FORMAT_ERROR;\n \n \tREFTABLE_ALLOC_GROW(r->refname, key.len - 8, r->refname_cap);\n+\tif (!r->refname) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tmemcpy(r->refname, key.buf, key.len - 8);\n \tts = get_be64(key.buf + key.len - 8);\n \n@@ -829,8 +848,10 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \tif (val_type == REFTABLE_LOG_DELETION)\n \t\treturn 0;\n \n-\tif (in.len < 2 * hash_size)\n-\t\treturn REFTABLE_FORMAT_ERROR;\n+\tif (in.len < 2 * hash_size) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n \n \tmemcpy(r->value.update.old_hash, in.buf, hash_size);\n \tmemcpy(r->value.update.new_hash, in.buf + hash_size, hash_size);\n@@ -838,8 +859,10 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \tstring_view_consume(&in, 2 * hash_size);\n \n \tn = decode_string(scratch, in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \n \t/*\n@@ -850,52 +873,75 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \t */\n \tif (!r->value.update.name ||\n \t strcmp(r->value.update.name, scratch->buf)) {\n-\t\tr->value.update.name =\n-\t\t\treftable_realloc(r->value.update.name, scratch->len + 1);\n+\t\tchar *name = reftable_realloc(r->value.update.name, scratch->len + 1);\n+\t\tif (!name) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tr->value.update.name = name;\n \t\tmemcpy(r->value.update.name, scratch->buf, scratch->len);\n \t\tr->value.update.name[scratch->len] = 0;\n \t}\n \n \tn = decode_string(scratch, in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \n \t/* Same as above, but for the reflog email. */\n \tif (!r->value.update.email ||\n \t strcmp(r->value.update.email, scratch->buf)) {\n-\t\tr->value.update.email =\n-\t\t\treftable_realloc(r->value.update.email, scratch->len + 1);\n+\t\tchar *email = reftable_realloc(r->value.update.email, scratch->len + 1);\n+\t\tif (!email) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tr->value.update.email = email;\n \t\tmemcpy(r->value.update.email, scratch->buf, scratch->len);\n \t\tr->value.update.email[scratch->len] = 0;\n \t}\n \n \tts = 0;\n \tn = get_var_int(&ts, &in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \tr->value.update.time = ts;\n-\tif (in.len < 2)\n+\tif (in.len < 2) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \n \tr->value.update.tz_offset = get_be16(in.buf);\n \tstring_view_consume(&in, 2);\n \n \tn = decode_string(scratch, in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \n \tREFTABLE_ALLOC_GROW(r->value.update.message, scratch->len + 1,\n \t\t\t r->value.update.message_cap);\n+\tif (!r->value.update.message) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tmemcpy(r->value.update.message, scratch->buf, scratch->len);\n \tr->value.update.message[scratch->len] = 0;\n \n \treturn start.len - in.len;\n \n done:\n-\treturn REFTABLE_FORMAT_ERROR;\n+\treturn err;\n }\n \n static int null_streq(const char *a, const char *b)\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503681","messageId":"42bc57fd001b04e799a43e2749715f2696ffe0db.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 08/22] reftable/writer: handle allocation failures in `writer_index_hash()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:08:32Z","receivedAt":"2024-09-30T08:08:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation errors in `writer_index_hash()`. Adjust its only\ncaller in `reftable_writer_add_ref()` accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/writer.c | 61 +++++++++++++++++++++++++++++------------------\n 1 file changed, 38 insertions(+), 23 deletions(-)\n\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 9d5e6072bc..ed61aaf59c 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -186,18 +186,22 @@ static int obj_index_tree_node_compare(const void *a, const void *b)\n \t\t\t &((const struct obj_index_tree_node *)b)->hash);\n }\n \n-static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n+static int writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n {\n \tuint64_t off = w->next;\n-\n \tstruct obj_index_tree_node want = { .hash = *hash };\n+\tstruct obj_index_tree_node *key;\n+\tstruct tree_node *node;\n \n-\tstruct tree_node *node = tree_search(&want, &w->obj_index_tree,\n-\t\t\t\t\t &obj_index_tree_node_compare, 0);\n-\tstruct obj_index_tree_node *key = NULL;\n+\tnode = tree_search(&want, &w->obj_index_tree,\n+\t\t\t &obj_index_tree_node_compare, 0);\n \tif (!node) {\n \t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n-\t\tkey = reftable_malloc(sizeof(struct obj_index_tree_node));\n+\n+\t\tkey = reftable_malloc(sizeof(*key));\n+\t\tif (!key)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \t\t*key = empty;\n \n \t\tstrbuf_reset(&key->hash);\n@@ -208,12 +212,15 @@ static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n \t\tkey = node->key;\n \t}\n \n-\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off) {\n-\t\treturn;\n-\t}\n+\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off)\n+\t\treturn 0;\n \n \tREFTABLE_ALLOC_GROW(key->offsets, key->offset_len + 1, key->offset_cap);\n+\tif (!key->offsets)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tkey->offsets[key->offset_len++] = off;\n+\n+\treturn 0;\n }\n \n static int writer_add_record(struct reftable_writer *w,\n@@ -284,11 +291,11 @@ int reftable_writer_add_ref(struct reftable_writer *w,\n \t\t\t.ref = *ref\n \t\t},\n \t};\n-\tint err = 0;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint err;\n \n-\tif (!ref->refname)\n-\t\treturn REFTABLE_API_ERROR;\n-\tif (ref->update_index < w->min_update_index ||\n+\tif (!ref->refname ||\n+\t ref->update_index < w->min_update_index ||\n \t ref->update_index > w->max_update_index)\n \t\treturn REFTABLE_API_ERROR;\n \n@@ -296,24 +303,32 @@ int reftable_writer_add_ref(struct reftable_writer *w,\n \n \terr = writer_add_record(w, &rec);\n \tif (err < 0)\n-\t\treturn err;\n+\t\tgoto out;\n \n \tif (!w->opts.skip_index_objects && reftable_ref_record_val1(ref)) {\n-\t\tstruct strbuf h = STRBUF_INIT;\n-\t\tstrbuf_add(&h, (char *)reftable_ref_record_val1(ref),\n+\t\tstrbuf_add(&buf, (char *)reftable_ref_record_val1(ref),\n \t\t\t hash_size(w->opts.hash_id));\n-\t\twriter_index_hash(w, &h);\n-\t\tstrbuf_release(&h);\n+\n+\t\terr = writer_index_hash(w, &buf);\n+\t\tif (err < 0)\n+\t\t\tgoto out;\n \t}\n \n \tif (!w->opts.skip_index_objects && reftable_ref_record_val2(ref)) {\n-\t\tstruct strbuf h = STRBUF_INIT;\n-\t\tstrbuf_add(&h, reftable_ref_record_val2(ref),\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_add(&buf, reftable_ref_record_val2(ref),\n \t\t\t hash_size(w->opts.hash_id));\n-\t\twriter_index_hash(w, &h);\n-\t\tstrbuf_release(&h);\n+\n+\t\terr = writer_index_hash(w, &buf);\n+\t\tif (err < 0)\n+\t\t\tgoto out;\n \t}\n-\treturn 0;\n+\n+\terr = 0;\n+\n+out:\n+\tstrbuf_release(&buf);\n+\treturn err;\n }\n \n int reftable_writer_add_refs(struct reftable_writer *w,\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503682","messageId":"9edd1d84cdbd53d966ff5cfe9b75281dd5966b07.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 09/22] reftable/writer: handle allocation failures in `reftable_new_writer()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:08:34Z","receivedAt":"2024-09-30T08:08:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_new_writer()`. Adapt the\nfunction to return an error code to return such failures. While at it,\nrename it to match our code style as we have to touch up every callsite\nanyway.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/reftable-writer.h | 12 +++++++-----\n reftable/stack.c | 14 ++++++++++----\n reftable/writer.c | 22 ++++++++++++++++------\n t/unit-tests/lib-reftable.c | 8 +++++---\n 4 files changed, 38 insertions(+), 18 deletions(-)\n\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 189b1f4144..43623dc7c3 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -90,11 +90,13 @@ struct reftable_stats {\n \tint object_id_len;\n };\n \n-/* reftable_new_writer creates a new writer */\n-struct reftable_writer *\n-reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n-\t\t int (*flush_func)(void *),\n-\t\t void *writer_arg, const struct reftable_write_options *opts);\n+struct reftable_writer;\n+\n+/* Create a new writer. */\n+int reftable_writer_new(struct reftable_writer **out,\n+\t\t\tssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t\tint (*flush_func)(void *),\n+\t\t\tvoid *writer_arg, const struct reftable_write_options *opts);\n \n /* Set the range of update indices for the records we will add. When writing a\n table into a stack, the min should be at least\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 498fae846d..ea21ca6e5f 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -808,8 +808,11 @@ int reftable_addition_add(struct reftable_addition *add,\n \t}\n \ttab_fd = get_tempfile_fd(tab_file);\n \n-\twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, &tab_fd,\n-\t\t\t\t &add->stack->opts);\n+\terr = reftable_writer_new(&wr, reftable_fd_write, reftable_fd_flush,\n+\t\t\t\t &tab_fd, &add->stack->opts);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = write_table(wr, arg);\n \tif (err < 0)\n \t\tgoto done;\n@@ -898,8 +901,11 @@ static int stack_compact_locked(struct reftable_stack *st,\n \t\tgoto done;\n \t}\n \n-\twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush,\n-\t\t\t\t &tab_fd, &st->opts);\n+\terr = reftable_writer_new(&wr, reftable_fd_write, reftable_fd_flush,\n+\t\t\t\t &tab_fd, &st->opts);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = stack_write_compact(st, wr, first, last, config);\n \tif (err < 0)\n \t\tgoto done;\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex ed61aaf59c..54ec822e1c 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -117,13 +117,17 @@ static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n \tw->block_writer->restart_interval = w->opts.restart_interval;\n }\n \n-struct reftable_writer *\n-reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n-\t\t int (*flush_func)(void *),\n-\t\t void *writer_arg, const struct reftable_write_options *_opts)\n+int reftable_writer_new(struct reftable_writer **out,\n+\t\t\tssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t\tint (*flush_func)(void *),\n+\t\t\tvoid *writer_arg, const struct reftable_write_options *_opts)\n {\n-\tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n \tstruct reftable_write_options opts = {0};\n+\tstruct reftable_writer *wp;\n+\n+\twp = reftable_calloc(1, sizeof(*wp));\n+\tif (!wp)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \n \tif (_opts)\n \t\topts = *_opts;\n@@ -134,13 +138,19 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \tstrbuf_init(&wp->block_writer_data.last_key, 0);\n \tstrbuf_init(&wp->last_key, 0);\n \tREFTABLE_CALLOC_ARRAY(wp->block, opts.block_size);\n+\tif (!wp->block) {\n+\t\tfree(wp);\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t}\n \twp->write = writer_func;\n \twp->write_arg = writer_arg;\n \twp->opts = opts;\n \twp->flush = flush_func;\n \twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n \n-\treturn wp;\n+\t*out = wp;\n+\n+\treturn 0;\n }\n \n void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\ndiff --git a/t/unit-tests/lib-reftable.c b/t/unit-tests/lib-reftable.c\nindex ab1fa44a28..54c26c43e7 100644\n--- a/t/unit-tests/lib-reftable.c\n+++ b/t/unit-tests/lib-reftable.c\n@@ -22,9 +22,11 @@ static int strbuf_writer_flush(void *arg UNUSED)\n struct reftable_writer *t_reftable_strbuf_writer(struct strbuf *buf,\n \t\t\t\t\t\t struct reftable_write_options *opts)\n {\n-\treturn reftable_new_writer(&strbuf_writer_write,\n-\t\t\t\t &strbuf_writer_flush,\n-\t\t\t\t buf, opts);\n+\tstruct reftable_writer *writer;\n+\tint ret = reftable_writer_new(&writer, &strbuf_writer_write, &strbuf_writer_flush,\n+\t\t\t\t buf, opts);\n+\tcheck(!ret);\n+\treturn writer;\n }\n \n void t_reftable_write_to_buf(struct strbuf *buf,\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503683","messageId":"d4004a7f4370e1fa79fb3fca5faf463016e1580c.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 10/22] reftable/merged: handle allocation failures in `merged_table_init_iter()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:08:37Z","receivedAt":"2024-09-30T08:08:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `merged_table_init_iter()`. While at it,\nmerge `merged_iter_init()` into the function. It only has a single\ncaller and merging them makes it easier to handle allocation failures\nconsistently.\n\nThis change also requires us to adapt `reftable_stack_init_*_iterator()`\nto bubble up the new error codes of `merged_table_iter_init()`. Adapt\ncallsites accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 39 +++++++++++++----\n reftable/merged.c | 74 ++++++++++++++++++++------------\n reftable/merged.h | 6 +--\n reftable/reftable-merged.h | 8 ++--\n reftable/reftable-stack.h | 8 ++--\n reftable/stack.c | 34 ++++++++++-----\n t/helper/test-reftable.c | 10 ++++-\n t/unit-tests/t-reftable-merged.c | 12 ++++--\n t/unit-tests/t-reftable-stack.c | 4 +-\n 9 files changed, 131 insertions(+), 64 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 5c241097a4..9c08c46c85 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1299,7 +1299,9 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t\t\tstruct reftable_log_record log = {0};\n \t\t\tstruct reftable_iterator it = {0};\n \n-\t\t\treftable_stack_init_log_iterator(arg->stack, &it);\n+\t\t\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto done;\n \n \t\t\t/*\n \t\t\t * When deleting refs we also delete all reflog entries\n@@ -1669,7 +1671,10 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \t * copy over all log entries from the old reflog. Last but not least,\n \t * when renaming we also have to delete all the old reflog entries.\n \t */\n-\treftable_stack_init_log_iterator(arg->stack, &it);\n+\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, arg->oldname);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1890,7 +1895,10 @@ static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftabl\n \tif (ret < 0)\n \t\tgoto done;\n \n-\treftable_stack_init_log_iterator(stack, &iter->iter);\n+\tret = reftable_stack_init_log_iterator(stack, &iter->iter);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&iter->iter, \"\");\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1957,7 +1965,10 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, refname);\n \twhile (!ret) {\n \t\tret = reftable_iterator_next_log(&it, &log);\n@@ -1973,6 +1984,7 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n \t\t\tbreak;\n \t}\n \n+done:\n \treftable_log_record_release(&log);\n \treftable_iterator_destroy(&it);\n \treturn ret;\n@@ -1994,7 +2006,10 @@ static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store,\n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, refname);\n \twhile (!ret) {\n \t\tstruct reftable_log_record log = {0};\n@@ -2044,7 +2059,10 @@ static int reftable_be_reflog_exists(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, refname);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -2150,7 +2168,9 @@ static int write_reflog_delete_table(struct reftable_writer *writer, void *cb_da\n \n \treftable_writer_set_limits(writer, ts, ts);\n \n-\treftable_stack_init_log_iterator(arg->stack, &it);\n+\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\tif (ret < 0)\n+\t\tgoto out;\n \n \t/*\n \t * In order to delete a table we need to delete all reflog entries one\n@@ -2174,6 +2194,7 @@ static int write_reflog_delete_table(struct reftable_writer *writer, void *cb_da\n \t\tret = reftable_writer_add_log(writer, &tombstone);\n \t}\n \n+out:\n \treftable_log_record_release(&log);\n \treftable_iterator_destroy(&it);\n \treturn ret;\n@@ -2312,7 +2333,9 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n \n \tret = reftable_iterator_seek_log(&it, refname);\n \tif (ret < 0)\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 128a810c55..5a8ea8ae77 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -30,22 +30,6 @@ struct merged_iter {\n \tssize_t advance_index;\n };\n \n-static void merged_iter_init(struct merged_iter *mi,\n-\t\t\t struct reftable_merged_table *mt,\n-\t\t\t uint8_t typ)\n-{\n-\tmemset(mi, 0, sizeof(*mi));\n-\tmi->advance_index = -1;\n-\tmi->suppress_deletions = mt->suppress_deletions;\n-\n-\tREFTABLE_CALLOC_ARRAY(mi->subiters, mt->readers_len);\n-\tfor (size_t i = 0; i < mt->readers_len; i++) {\n-\t\treftable_record_init(&mi->subiters[i].rec, typ);\n-\t\treader_init_iter(mt->readers[i], &mi->subiters[i].iter, typ);\n-\t}\n-\tmi->subiters_len = mt->readers_len;\n-}\n-\n static void merged_iter_close(void *p)\n {\n \tstruct merged_iter *mi = p;\n@@ -244,25 +228,61 @@ reftable_merged_table_min_update_index(struct reftable_merged_table *mt)\n \treturn mt->min;\n }\n \n-void merged_table_init_iter(struct reftable_merged_table *mt,\n-\t\t\t struct reftable_iterator *it,\n-\t\t\t uint8_t typ)\n+int merged_table_init_iter(struct reftable_merged_table *mt,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t uint8_t typ)\n {\n-\tstruct merged_iter *mi = reftable_malloc(sizeof(*mi));\n-\tmerged_iter_init(mi, mt, typ);\n+\tstruct merged_subiter *subiters;\n+\tstruct merged_iter *mi = NULL;\n+\tint ret;\n+\n+\tREFTABLE_CALLOC_ARRAY(subiters, mt->readers_len);\n+\tif (!subiters) {\n+\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n+\tfor (size_t i = 0; i < mt->readers_len; i++) {\n+\t\treftable_record_init(&subiters[i].rec, typ);\n+\t\treader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n+\t}\n+\n+\tREFTABLE_CALLOC_ARRAY(mi, 1);\n+\tif (!mi) {\n+\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\tmi->advance_index = -1;\n+\tmi->suppress_deletions = mt->suppress_deletions;\n+\tmi->subiters = subiters;\n+\tmi->subiters_len = mt->readers_len;\n+\n \titerator_from_merged_iter(it, mi);\n+\tret = 0;\n+\n+out:\n+\tif (ret < 0) {\n+\t\tfor (size_t i = 0; subiters && i < mt->readers_len; i++) {\n+\t\t\treftable_iterator_destroy(&subiters[i].iter);\n+\t\t\treftable_record_release(&subiters[i].rec);\n+\t\t}\n+\t\tfree(subiters);\n+\t\tfree(mi);\n+\t}\n+\n+\treturn ret;\n }\n \n-void reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it)\n+int reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(mt, it, BLOCK_TYPE_REF);\n+\treturn merged_table_init_iter(mt, it, BLOCK_TYPE_REF);\n }\n \n-void reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it)\n+int reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(mt, it, BLOCK_TYPE_LOG);\n+\treturn merged_table_init_iter(mt, it, BLOCK_TYPE_LOG);\n }\n \n uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\ndiff --git a/reftable/merged.h b/reftable/merged.h\nindex de5fd33f01..89bd0c4b35 100644\n--- a/reftable/merged.h\n+++ b/reftable/merged.h\n@@ -26,8 +26,8 @@ struct reftable_merged_table {\n \n struct reftable_iterator;\n \n-void merged_table_init_iter(struct reftable_merged_table *mt,\n-\t\t\t struct reftable_iterator *it,\n-\t\t\t uint8_t typ);\n+int merged_table_init_iter(struct reftable_merged_table *mt,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t uint8_t typ);\n \n #endif\ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nindex 16d19f8df2..a970d5dd89 100644\n--- a/reftable/reftable-merged.h\n+++ b/reftable/reftable-merged.h\n@@ -37,12 +37,12 @@ int reftable_merged_table_new(struct reftable_merged_table **dest,\n \t\t\t uint32_t hash_id);\n \n /* Initialize a merged table iterator for reading refs. */\n-void reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it);\n+int reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it);\n \n /* Initialize a merged table iterator for reading logs. */\n-void reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it);\n+int reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it);\n \n /* returns the max update_index covered by this merged table. */\n uint64_t\ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nindex f4f8cabc7f..e958f911b4 100644\n--- a/reftable/reftable-stack.h\n+++ b/reftable/reftable-stack.h\n@@ -73,16 +73,16 @@ struct reftable_iterator;\n * be used to iterate through refs. The iterator is valid until the next reload\n * or write.\n */\n-void reftable_stack_init_ref_iterator(struct reftable_stack *st,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_stack_init_ref_iterator(struct reftable_stack *st,\n+\t\t\t\t struct reftable_iterator *it);\n \n /*\n * Initialize an iterator for the merged tables contained in the stack that can\n * be used to iterate through logs. The iterator is valid until the next reload\n * or write.\n */\n-void reftable_stack_init_log_iterator(struct reftable_stack *st,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_stack_init_log_iterator(struct reftable_stack *st,\n+\t\t\t\t struct reftable_iterator *it);\n \n /* returns the merged_table for seeking. This table is valid until the\n * next write or reload, and should not be closed or deleted.\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex ea21ca6e5f..bb4d230918 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -136,18 +136,18 @@ int read_lines(const char *filename, char ***namesp)\n \treturn err;\n }\n \n-void reftable_stack_init_ref_iterator(struct reftable_stack *st,\n+int reftable_stack_init_ref_iterator(struct reftable_stack *st,\n \t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(reftable_stack_merged_table(st),\n-\t\t\t it, BLOCK_TYPE_REF);\n+\treturn merged_table_init_iter(reftable_stack_merged_table(st),\n+\t\t\t\t it, BLOCK_TYPE_REF);\n }\n \n-void reftable_stack_init_log_iterator(struct reftable_stack *st,\n-\t\t\t\t struct reftable_iterator *it)\n+int reftable_stack_init_log_iterator(struct reftable_stack *st,\n+\t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(reftable_stack_merged_table(st),\n-\t\t\t it, BLOCK_TYPE_LOG);\n+\treturn merged_table_init_iter(reftable_stack_merged_table(st),\n+\t\t\t\t it, BLOCK_TYPE_LOG);\n }\n \n struct reftable_merged_table *\n@@ -952,7 +952,10 @@ static int stack_write_compact(struct reftable_stack *st,\n \tif (err < 0)\n \t\tgoto done;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tif (err < 0)\n \t\tgoto done;\n@@ -977,7 +980,10 @@ static int stack_write_compact(struct reftable_stack *st,\n \t}\n \treftable_iterator_destroy(&it);\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tif (err < 0)\n \t\tgoto done;\n@@ -1496,7 +1502,10 @@ int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n \tstruct reftable_iterator it = { 0 };\n \tint ret;\n \n-\treftable_merged_table_init_ref_iterator(st->merged, &it);\n+\tret = reftable_merged_table_init_ref_iterator(st->merged, &it);\n+\tif (ret)\n+\t\tgoto out;\n+\n \tret = reftable_iterator_seek_ref(&it, refname);\n \tif (ret)\n \t\tgoto out;\n@@ -1523,7 +1532,10 @@ int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n \tstruct reftable_iterator it = {0};\n \tint err;\n \n-\treftable_stack_init_log_iterator(st, &it);\n+\terr = reftable_stack_init_log_iterator(st, &it);\n+\tif (err)\n+\t\tgoto done;\n+\n \terr = reftable_iterator_seek_log(&it, refname);\n \tif (err)\n \t\tgoto done;\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 29d4e9a755..5c8849d115 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -28,7 +28,10 @@ static int dump_table(struct reftable_merged_table *mt)\n \tconst struct git_hash_algo *algop;\n \tint err;\n \n-\treftable_merged_table_init_ref_iterator(mt, &it);\n+\terr = reftable_merged_table_init_ref_iterator(mt, &it);\n+\tif (err < 0)\n+\t\treturn err;\n+\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tif (err < 0)\n \t\treturn err;\n@@ -63,7 +66,10 @@ static int dump_table(struct reftable_merged_table *mt)\n \treftable_iterator_destroy(&it);\n \treftable_ref_record_release(&ref);\n \n-\treftable_merged_table_init_log_iterator(mt, &it);\n+\terr = reftable_merged_table_init_log_iterator(mt, &it);\n+\tif (err < 0)\n+\t\treturn err;\n+\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tif (err < 0)\n \t\treturn err;\ndiff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c\nindex 19e54bdfb8..3d2848632d 100644\n--- a/t/unit-tests/t-reftable-merged.c\n+++ b/t/unit-tests/t-reftable-merged.c\n@@ -82,7 +82,8 @@ static void t_merged_single_record(void)\n \tstruct reftable_iterator it = { 0 };\n \tint err;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"a\");\n \tcheck(!err);\n \n@@ -161,7 +162,8 @@ static void t_merged_refs(void)\n \tsize_t cap = 0;\n \tsize_t i;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"a\");\n \tcheck(!err);\n \tcheck_int(reftable_merged_table_hash_id(mt), ==, GIT_SHA1_FORMAT_ID);\n@@ -367,7 +369,8 @@ static void t_merged_logs(void)\n \tsize_t cap = 0;\n \tsize_t i;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"a\");\n \tcheck(!err);\n \tcheck_int(reftable_merged_table_hash_id(mt), ==, GIT_SHA1_FORMAT_ID);\n@@ -390,7 +393,8 @@ static void t_merged_logs(void)\n \t\tcheck(reftable_log_record_equal(want[i], &out[i],\n \t\t\t\t\t\t GIT_SHA1_RAWSZ));\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log_at(&it, \"a\", 2);\n \tcheck(!err);\n \treftable_log_record_release(&out[0]);\ndiff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c\nindex 65e513d5ec..6e41a4cf7e 100644\n--- a/t/unit-tests/t-reftable-stack.c\n+++ b/t/unit-tests/t-reftable-stack.c\n@@ -599,7 +599,9 @@ static void t_reftable_stack_iterator(void)\n \n \treftable_iterator_destroy(&it);\n \n-\treftable_stack_init_log_iterator(st, &it);\n+\terr = reftable_stack_init_log_iterator(st, &it);\n+\tcheck(!err);\n+\n \treftable_iterator_seek_log(&it, logs[0].refname);\n \tfor (i = 0; ; i++) {\n \t\tstruct reftable_log_record log = { 0 };\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503684","messageId":"b2bd142021a7c390250811d5dd980b8b1757a96b.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 11/22] reftable/reader: handle allocation failures for unindexed reader","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:08:42Z","receivedAt":"2024-09-30T08:08:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when creating unindexed readers.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/reader.c | 23 +++++++++++++++++------\n 1 file changed, 17 insertions(+), 6 deletions(-)\n\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 6494ce2e32..485ee085da 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -734,19 +734,30 @@ static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n \tREFTABLE_ALLOC_ARRAY(ti, 1);\n \ttable_iter_init(ti, r);\n \terr = table_iter_seek_start(ti, BLOCK_TYPE_REF, 0);\n-\tif (err < 0) {\n-\t\treftable_free(ti);\n-\t\treturn err;\n-\t}\n+\tif (err < 0)\n+\t\tgoto out;\n \n-\tfilter = reftable_malloc(sizeof(struct filtering_ref_iterator));\n+\tfilter = reftable_malloc(sizeof(*filter));\n+\tif (!filter) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n \t*filter = empty;\n \n \tstrbuf_add(&filter->oid, oid, oid_len);\n \titerator_from_table_iter(&filter->it, ti);\n \n \titerator_from_filtering_ref_iterator(it, filter);\n-\treturn 0;\n+\n+\terr = 0;\n+\n+out:\n+\tif (err < 0) {\n+\t\tif (ti)\n+\t\t\ttable_iter_close(ti);\n+\t\treftable_free(ti);\n+\t}\n+\treturn err;\n }\n \n int reftable_reader_refs_for(struct reftable_reader *r,\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503685","messageId":"9b5fd52862bf9ee4367481e285990883e8547ab4.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 12/22] reftable/reader: handle allocation failures in `reader_init_iter()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:08:45Z","receivedAt":"2024-09-30T08:08:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reader_init_iter()`. This requires us to\nalso adapt `reftable_reader_init_*_iterator()` to bubble up the new\nerror codes. Adapt callers accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 4 +++-\n reftable/reader.c | 28 +++++++++++++++---------\n reftable/reader.h | 6 ++---\n reftable/reftable-reader.h | 8 +++----\n t/unit-tests/t-reftable-readwrite.c | 34 +++++++++++++++++++----------\n 5 files changed, 50 insertions(+), 30 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 5a8ea8ae77..741f62ea63 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -244,7 +244,9 @@ int merged_table_init_iter(struct reftable_merged_table *mt,\n \n \tfor (size_t i = 0; i < mt->readers_len; i++) {\n \t\treftable_record_init(&subiters[i].rec, typ);\n-\t\treader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n+\t\tret = reader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n+\t\tif (ret < 0)\n+\t\t\tgoto out;\n \t}\n \n \tREFTABLE_CALLOC_ARRAY(mi, 1);\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 485ee085da..f696e992df 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -554,32 +554,37 @@ static void iterator_from_table_iter(struct reftable_iterator *it,\n \tit->ops = &table_iter_vtable;\n }\n \n-void reader_init_iter(struct reftable_reader *r,\n-\t\t struct reftable_iterator *it,\n-\t\t uint8_t typ)\n+int reader_init_iter(struct reftable_reader *r,\n+\t\t struct reftable_iterator *it,\n+\t\t uint8_t typ)\n {\n \tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n \n \tif (offs->is_present) {\n \t\tstruct table_iter *ti;\n \t\tREFTABLE_ALLOC_ARRAY(ti, 1);\n+\t\tif (!ti)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \t\ttable_iter_init(ti, r);\n \t\titerator_from_table_iter(it, ti);\n \t} else {\n \t\titerator_set_empty(it);\n \t}\n+\n+\treturn 0;\n }\n \n-void reftable_reader_init_ref_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it)\n+int reftable_reader_init_ref_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it)\n {\n-\treader_init_iter(r, it, BLOCK_TYPE_REF);\n+\treturn reader_init_iter(r, it, BLOCK_TYPE_REF);\n }\n \n-void reftable_reader_init_log_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it)\n+int reftable_reader_init_log_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it)\n {\n-\treader_init_iter(r, it, BLOCK_TYPE_LOG);\n+\treturn reader_init_iter(r, it, BLOCK_TYPE_LOG);\n }\n \n int reftable_reader_new(struct reftable_reader **out,\n@@ -689,7 +694,10 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n \tstruct indexed_table_ref_iter *itr = NULL;\n \n \t/* Look through the reverse index. */\n-\treader_init_iter(r, &oit, BLOCK_TYPE_OBJ);\n+\terr = reader_init_iter(r, &oit, BLOCK_TYPE_OBJ);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = iterator_seek(&oit, &want);\n \tif (err != 0)\n \t\tgoto done;\ndiff --git a/reftable/reader.h b/reftable/reader.h\nindex 3710ee09b4..02d10c5d37 100644\n--- a/reftable/reader.h\n+++ b/reftable/reader.h\n@@ -56,9 +56,9 @@ struct reftable_reader {\n \n const char *reader_name(struct reftable_reader *r);\n \n-void reader_init_iter(struct reftable_reader *r,\n-\t\t struct reftable_iterator *it,\n-\t\t uint8_t typ);\n+int reader_init_iter(struct reftable_reader *r,\n+\t\t struct reftable_iterator *it,\n+\t\t uint8_t typ);\n \n /* initialize a block reader to read from `r` */\n int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nindex a600452b56..6a2d0b693f 100644\n--- a/reftable/reftable-reader.h\n+++ b/reftable/reftable-reader.h\n@@ -46,12 +46,12 @@ void reftable_reader_incref(struct reftable_reader *reader);\n void reftable_reader_decref(struct reftable_reader *reader);\n \n /* Initialize a reftable iterator for reading refs. */\n-void reftable_reader_init_ref_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_reader_init_ref_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it);\n \n /* Initialize a reftable iterator for reading logs. */\n-void reftable_reader_init_log_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_reader_init_log_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it);\n \n /* returns the hash ID used in this table. */\n uint32_t reftable_reader_hash_id(struct reftable_reader *r);\ndiff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c\nindex e1b235a5f1..acca927a2c 100644\n--- a/t/unit-tests/t-reftable-readwrite.c\n+++ b/t/unit-tests/t-reftable-readwrite.c\n@@ -206,7 +206,8 @@ static void t_log_write_read(void)\n \terr = reftable_reader_new(&reader, &source, \"file.log\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \n \terr = reftable_iterator_seek_ref(&it, names[N - 1]);\n \tcheck(!err);\n@@ -221,8 +222,8 @@ static void t_log_write_read(void)\n \treftable_iterator_destroy(&it);\n \treftable_ref_record_release(&ref);\n \n-\treftable_reader_init_log_iterator(reader, &it);\n-\n+\terr = reftable_reader_init_log_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tcheck(!err);\n \n@@ -296,7 +297,8 @@ static void t_log_zlib_corruption(void)\n \terr = reftable_reader_new(&reader, &source, \"file.log\");\n \tcheck(!err);\n \n-\treftable_reader_init_log_iterator(reader, &it);\n+\terr = reftable_reader_init_log_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"refname\");\n \tcheck_int(err, ==, REFTABLE_ZLIB_ERROR);\n \n@@ -325,7 +327,8 @@ static void t_table_read_write_sequential(void)\n \terr = reftable_reader_new(&reader, &source, \"file.ref\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tcheck(!err);\n \n@@ -376,7 +379,8 @@ static void t_table_read_api(void)\n \terr = reftable_reader_new(&reader, &source, \"file.ref\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, names[0]);\n \tcheck(!err);\n \n@@ -419,7 +423,8 @@ static void t_table_read_write_seek(int index, int hash_id)\n \t}\n \n \tfor (i = 1; i < N; i++) {\n-\t\treftable_reader_init_ref_iterator(reader, &it);\n+\t\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\t\tcheck(!err);\n \t\terr = reftable_iterator_seek_ref(&it, names[i]);\n \t\tcheck(!err);\n \t\terr = reftable_iterator_next_ref(&it, &ref);\n@@ -435,7 +440,8 @@ static void t_table_read_write_seek(int index, int hash_id)\n \tstrbuf_addstr(&pastLast, names[N - 1]);\n \tstrbuf_addstr(&pastLast, \"/\");\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, pastLast.buf);\n \tif (err == 0) {\n \t\tstruct reftable_ref_record ref = { 0 };\n@@ -534,7 +540,8 @@ static void t_table_refs_for(int indexed)\n \tif (!indexed)\n \t\treader->obj_offsets.is_present = 0;\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tcheck(!err);\n \treftable_iterator_destroy(&it);\n@@ -593,7 +600,8 @@ static void t_write_empty_table(void)\n \terr = reftable_reader_new(&rd, &source, \"filename\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(rd, &it);\n+\terr = reftable_reader_init_ref_iterator(rd, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tcheck(!err);\n \n@@ -802,7 +810,8 @@ static void t_write_multiple_indices(void)\n \t * Seeking the log uses the log index now. In case there is any\n \t * confusion regarding indices we would notice here.\n \t */\n-\treftable_reader_init_log_iterator(reader, &it);\n+\terr = reftable_reader_init_log_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tcheck(!err);\n \n@@ -858,7 +867,8 @@ static void t_write_multi_level_index(void)\n \t/*\n \t * Seeking the last ref should work as expected.\n \t */\n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"refs/heads/199\");\n \tcheck(!err);\n \n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503686","messageId":"0f3e3d1585777813f52caa82ae873c37620d96ad.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 13/22] reftable/stack: handle allocation failures on reload","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:08:47Z","receivedAt":"2024-09-30T08:08:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_stack_reload_once()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 34 +++++++++++++++++++++++++---------\n 1 file changed, 25 insertions(+), 9 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex bb4d230918..060b2c1b90 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -214,13 +214,13 @@ void reftable_stack_destroy(struct reftable_stack *st)\n }\n \n static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n-\t\t\t\t\t\t int cur_len)\n+\t\t\t\t\t\t size_t cur_len)\n {\n \tstruct reftable_reader **cur = reftable_calloc(cur_len, sizeof(*cur));\n-\tint i = 0;\n-\tfor (i = 0; i < cur_len; i++) {\n+\tif (!cur)\n+\t\treturn NULL;\n+\tfor (size_t i = 0; i < cur_len; i++)\n \t\tcur[i] = st->readers[i];\n-\t}\n \treturn cur;\n }\n \n@@ -229,18 +229,30 @@ static int reftable_stack_reload_once(struct reftable_stack *st,\n \t\t\t\t int reuse_open)\n {\n \tsize_t cur_len = !st->merged ? 0 : st->merged->readers_len;\n-\tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n+\tstruct reftable_reader **cur;\n \tstruct reftable_reader **reused = NULL;\n-\tsize_t reused_len = 0, reused_alloc = 0;\n-\tsize_t names_len = names_length(names);\n-\tstruct reftable_reader **new_readers =\n-\t\treftable_calloc(names_len, sizeof(*new_readers));\n+\tstruct reftable_reader **new_readers;\n+\tsize_t reused_len = 0, reused_alloc = 0, names_len;\n \tsize_t new_readers_len = 0;\n \tstruct reftable_merged_table *new_merged = NULL;\n \tstruct strbuf table_path = STRBUF_INIT;\n \tint err = 0;\n \tsize_t i;\n \n+\tcur = stack_copy_readers(st, cur_len);\n+\tif (!cur) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tnames_len = names_length(names);\n+\n+\tnew_readers = reftable_calloc(names_len, sizeof(*new_readers));\n+\tif (!new_readers) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \twhile (*names) {\n \t\tstruct reftable_reader *rd = NULL;\n \t\tconst char *name = *names++;\n@@ -261,6 +273,10 @@ static int reftable_stack_reload_once(struct reftable_stack *st,\n \t\t\t\t * do by bumping their refcount.\n \t\t\t\t */\n \t\t\t\tREFTABLE_ALLOC_GROW(reused, reused_len + 1, reused_alloc);\n+\t\t\t\tif (!reused) {\n+\t\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n \t\t\t\treused[reused_len++] = rd;\n \t\t\t\treftable_reader_incref(rd);\n \t\t\t\tbreak;\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503687","messageId":"c88645a251494de3dffea054887ddbe84468f948.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 14/22] reftable/stack: handle allocation failures in `reftable_new_stack()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:08:50Z","receivedAt":"2024-09-30T08:08:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_new_stack()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 34 ++++++++++++++++++++++++++--------\n 1 file changed, 26 insertions(+), 8 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 060b2c1b90..1b77c9d014 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -56,10 +56,16 @@ static int reftable_fd_flush(void *arg)\n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \t\t const struct reftable_write_options *_opts)\n {\n-\tstruct reftable_stack *p = reftable_calloc(1, sizeof(*p));\n \tstruct strbuf list_file_name = STRBUF_INIT;\n-\tstruct reftable_write_options opts = {0};\n-\tint err = 0;\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct reftable_stack *p;\n+\tint err;\n+\n+\tp = reftable_calloc(1, sizeof(*p));\n+\tif (!p) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n \n \tif (_opts)\n \t\topts = *_opts;\n@@ -74,15 +80,23 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \n \tp->list_file = strbuf_detach(&list_file_name, NULL);\n \tp->list_fd = -1;\n-\tp->reftable_dir = xstrdup(dir);\n \tp->opts = opts;\n+\tp->reftable_dir = reftable_strdup(dir);\n+\tif (!p->reftable_dir) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n \n \terr = reftable_stack_reload_maybe_reuse(p, 1);\n-\tif (err < 0) {\n+\tif (err < 0)\n+\t\tgoto out;\n+\n+\t*dest = p;\n+\terr = 0;\n+\n+out:\n+\tif (err < 0)\n \t\treftable_stack_destroy(p);\n-\t} else {\n-\t\t*dest = p;\n-\t}\n \treturn err;\n }\n \n@@ -171,6 +185,10 @@ void reftable_stack_destroy(struct reftable_stack *st)\n {\n \tchar **names = NULL;\n \tint err = 0;\n+\n+\tif (!st)\n+\t\treturn;\n+\n \tif (st->merged) {\n \t\treftable_merged_table_free(st->merged);\n \t\tst->merged = NULL;\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503688","messageId":"92ad4fc9346aaff0f6d15a6f5556d150fd5c8f1d.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 15/22] reftable/stack: handle allocation failures in `stack_compact_range()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:08:53Z","receivedAt":"2024-09-30T08:08:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `stack_compact_range()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 19 +++++++++++++++++--\n 1 file changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 1b77c9d014..2e6dd513d7 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1131,6 +1131,11 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * from the point of view of the newer process.\n \t */\n \tREFTABLE_CALLOC_ARRAY(table_locks, last - first + 1);\n+\tif (!table_locks) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tfor (i = last + 1; i > first; i--) {\n \t\tstack_filename(&table_name, st, reader_name(st->readers[i - 1]));\n \n@@ -1312,8 +1317,18 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\t * thus have to allocate `readers_len + 1` many entries.\n \t\t */\n \t\tREFTABLE_CALLOC_ARRAY(names, st->merged->readers_len + 1);\n-\t\tfor (size_t i = 0; i < st->merged->readers_len; i++)\n-\t\t\tnames[i] = xstrdup(st->readers[i]->name);\n+\t\tif (!names) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tfor (size_t i = 0; i < st->merged->readers_len; i++) {\n+\t\t\tnames[i] = reftable_strdup(st->readers[i]->name);\n+\t\t\tif (!names[i]) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n \t\tfirst_to_replace = first;\n \t\tlast_to_replace = last;\n \t}\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503689","messageId":"19acebd9195e7ba587ae17154d5ad7d0d218ec10.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 16/22] reftable/stack: handle allocation failures in auto compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:08:55Z","receivedAt":"2024-09-30T08:09:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_stack_auto_compact()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 16 ++++++++++++----\n 1 file changed, 12 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 2e6dd513d7..990784d9d2 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1518,6 +1518,8 @@ static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n \tuint64_t *sizes;\n \n \tREFTABLE_CALLOC_ARRAY(sizes, st->merged->readers_len);\n+\tif (!sizes)\n+\t\treturn NULL;\n \n \tfor (size_t i = 0; i < st->merged->readers_len; i++)\n \t\tsizes[i] = st->readers[i]->size - overhead;\n@@ -1527,11 +1529,17 @@ static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n \n int reftable_stack_auto_compact(struct reftable_stack *st)\n {\n-\tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n-\tstruct segment seg =\n-\t\tsuggest_compaction_segment(sizes, st->merged->readers_len,\n-\t\t\t\t\t st->opts.auto_compaction_factor);\n+\tstruct segment seg;\n+\tuint64_t *sizes;\n+\n+\tsizes = stack_table_sizes_for_compaction(st);\n+\tif (!sizes)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n+\tseg = suggest_compaction_segment(sizes, st->merged->readers_len,\n+\t\t\t\t\t st->opts.auto_compaction_factor);\n \treftable_free(sizes);\n+\n \tif (segment_size(&seg) > 0)\n \t\treturn stack_compact_range(st, seg.start, seg.end - 1,\n \t\t\t\t\t NULL, STACK_COMPACT_RANGE_BEST_EFFORT);\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503690","messageId":"620658bffcc0570410fb2299c568f7c285a91336.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 17/22] reftable/iter: handle allocation failures when creating indexed table iter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:09:00Z","receivedAt":"2024-09-30T08:09:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `new_indexed_table_ref_iter()`. While at\nit, rename the function to match our coding style.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/iter.c | 20 ++++++++++++++++----\n reftable/iter.h | 2 +-\n reftable/reader.c | 7 ++++++-\n 3 files changed, 23 insertions(+), 6 deletions(-)\n\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex 416a9f6996..d319538f80 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -181,14 +181,20 @@ static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\n \t}\n }\n \n-int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+int indexed_table_ref_iter_new(struct indexed_table_ref_iter **dest,\n \t\t\t struct reftable_reader *r, uint8_t *oid,\n \t\t\t int oid_len, uint64_t *offsets, int offset_len)\n {\n \tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n-\tstruct indexed_table_ref_iter *itr = reftable_calloc(1, sizeof(*itr));\n+\tstruct indexed_table_ref_iter *itr;\n \tint err = 0;\n \n+\titr = reftable_calloc(1, sizeof(*itr));\n+\tif (!itr) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n \t*itr = empty;\n \titr->r = r;\n \tstrbuf_add(&itr->oid, oid, oid_len);\n@@ -197,10 +203,16 @@ int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n \titr->offset_len = offset_len;\n \n \terr = indexed_table_ref_iter_next_block(itr);\n+\tif (err < 0)\n+\t\tgoto out;\n+\n+\t*dest = itr;\n+\terr = 0;\n+\n+out:\n \tif (err < 0) {\n+\t\t*dest = NULL;\n \t\treftable_free(itr);\n-\t} else {\n-\t\t*dest = itr;\n \t}\n \treturn err;\n }\ndiff --git a/reftable/iter.h b/reftable/iter.h\nindex befc4597df..b3225bc7ad 100644\n--- a/reftable/iter.h\n+++ b/reftable/iter.h\n@@ -82,7 +82,7 @@ void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n \t\t\t\t\t struct indexed_table_ref_iter *itr);\n \n /* Takes ownership of `offsets` */\n-int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+int indexed_table_ref_iter_new(struct indexed_table_ref_iter **dest,\n \t\t\t struct reftable_reader *r, uint8_t *oid,\n \t\t\t int oid_len, uint64_t *offsets, int offset_len);\n \ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex f696e992df..0179e4e73d 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -715,7 +715,7 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n \t\tgoto done;\n \t}\n \n-\terr = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),\n+\terr = indexed_table_ref_iter_new(&itr, r, oid, hash_size(r->hash_id),\n \t\t\t\t\t got.u.obj.offsets,\n \t\t\t\t\t got.u.obj.offset_len);\n \tif (err < 0)\n@@ -740,6 +740,11 @@ static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n \tint err;\n \n \tREFTABLE_ALLOC_ARRAY(ti, 1);\n+\tif (!ti) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n \ttable_iter_init(ti, r);\n \terr = table_iter_seek_start(ti, BLOCK_TYPE_REF, 0);\n \tif (err < 0)\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503691","messageId":"48047dd4c9f20c320a2a596d3596687afc838740.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 18/22] reftable/blocksource: handle allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:09:03Z","receivedAt":"2024-09-30T08:09:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in the blocksource code.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/blocksource.c | 25 ++++++++++++++++++++-----\n 1 file changed, 20 insertions(+), 5 deletions(-)\n\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex e93cac9bb6..6b570751be 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -30,6 +30,8 @@ static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n \tstruct strbuf *b = v;\n \tassert(off + size <= b->len);\n \tREFTABLE_CALLOC_ARRAY(dest->data, size);\n+\tif (!dest->data)\n+\t\treturn -1;\n \tmemcpy(dest->data, b->buf + off, size);\n \tdest->len = size;\n \treturn size;\n@@ -98,27 +100,40 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,\n {\n \tstruct file_block_source *p;\n \tstruct stat st;\n-\tint fd;\n+\tint fd, err;\n \n \tfd = open(name, O_RDONLY);\n \tif (fd < 0) {\n \t\tif (errno == ENOENT)\n \t\t\treturn REFTABLE_NOT_EXIST_ERROR;\n-\t\treturn -1;\n+\t\terr = -1;\n+\t\tgoto out;\n \t}\n \n \tif (fstat(fd, &st) < 0) {\n-\t\tclose(fd);\n-\t\treturn REFTABLE_IO_ERROR;\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto out;\n \t}\n \n \tREFTABLE_CALLOC_ARRAY(p, 1);\n+\tif (!p) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n \tp->size = st.st_size;\n \tp->data = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n-\tclose(fd);\n \n \tassert(!bs->ops);\n \tbs->ops = &file_vtable;\n \tbs->arg = p;\n+\n+\terr = 0;\n+\n+out:\n+\tif (fd >= 0)\n+\t\tclose(fd);\n+\tif (err < 0)\n+\t\tfree(p);\n \treturn 0;\n }\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503692","messageId":"08685605ba6a672811c7dc4c11f831b09f1a62a8.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 19/22] reftable/block: handle allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:09:05Z","receivedAt":"2024-09-30T08:09:10Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `block_writer_init()` and\n`block_reader_init()`. This requires us to bubble up error codes into\n`writer_reinit_block_writer()`. Adapt call sites accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/block.c | 23 +++++++++++++--\n reftable/block.h | 4 +--\n reftable/writer.c | 51 ++++++++++++++++++++++-----------\n t/unit-tests/t-reftable-block.c | 20 +++++++------\n 4 files changed, 69 insertions(+), 29 deletions(-)\n\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 00030eee06..bfa7dc61bf 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -52,6 +52,8 @@ static int block_writer_register_restart(struct block_writer *w, int n,\n \t\treturn -1;\n \tif (is_restart) {\n \t\tREFTABLE_ALLOC_GROW(w->restarts, w->restart_len + 1, w->restart_cap);\n+\t\tif (!w->restarts)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \t\tw->restarts[w->restart_len++] = w->next;\n \t}\n \n@@ -63,8 +65,8 @@ static int block_writer_register_restart(struct block_writer *w, int n,\n \treturn 0;\n }\n \n-void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n-\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n+int block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n {\n \tbw->buf = buf;\n \tbw->hash_size = hash_size;\n@@ -78,8 +80,12 @@ void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n \tbw->last_key.len = 0;\n \tif (!bw->zstream) {\n \t\tREFTABLE_CALLOC_ARRAY(bw->zstream, 1);\n+\t\tif (!bw->zstream)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \t\tdeflateInit(bw->zstream, 9);\n \t}\n+\n+\treturn 0;\n }\n \n uint8_t block_writer_type(struct block_writer *bw)\n@@ -163,6 +169,10 @@ int block_writer_finish(struct block_writer *w)\n \t\t */\n \t\tcompressed_len = deflateBound(w->zstream, src_len);\n \t\tREFTABLE_ALLOC_GROW(w->compressed, compressed_len, w->compressed_cap);\n+\t\tif (!w->compressed) {\n+\t\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\treturn ret;\n+\t\t}\n \n \t\tw->zstream->next_out = w->compressed;\n \t\tw->zstream->avail_out = compressed_len;\n@@ -219,12 +229,21 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,\n \t\t/* Log blocks specify the *uncompressed* size in their header. */\n \t\tREFTABLE_ALLOC_GROW(br->uncompressed_data, sz,\n \t\t\t\t br->uncompressed_cap);\n+\t\tif (!br->uncompressed_data) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n \n \t\t/* Copy over the block header verbatim. It's not compressed. */\n \t\tmemcpy(br->uncompressed_data, block->data, block_header_skip);\n \n \t\tif (!br->zstream) {\n \t\t\tREFTABLE_CALLOC_ARRAY(br->zstream, 1);\n+\t\t\tif (!br->zstream) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\n \t\t\terr = inflateInit(br->zstream);\n \t\t} else {\n \t\t\terr = inflateReset(br->zstream);\ndiff --git a/reftable/block.h b/reftable/block.h\nindex 1c8f25ee6e..18d7ea0337 100644\n--- a/reftable/block.h\n+++ b/reftable/block.h\n@@ -45,8 +45,8 @@ struct block_writer {\n /*\n * initializes the blockwriter to write `typ` entries, using `buf` as temporary\n * storage. `buf` is not owned by the block_writer. */\n-void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n-\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n+int block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n \n /* returns the block type (eg. 'r' for ref records. */\n uint8_t block_writer_type(struct block_writer *bw);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 54ec822e1c..712a84791b 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -102,19 +102,24 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \treturn header_size(writer_version(w));\n }\n \n-static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n+static int writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n {\n-\tint block_start = 0;\n-\tif (w->next == 0) {\n+\tint block_start = 0, ret;\n+\n+\tif (w->next == 0)\n \t\tblock_start = header_size(writer_version(w));\n-\t}\n \n \tstrbuf_reset(&w->last_key);\n-\tblock_writer_init(&w->block_writer_data, typ, w->block,\n-\t\t\t w->opts.block_size, block_start,\n-\t\t\t hash_size(w->opts.hash_id));\n+\tret = block_writer_init(&w->block_writer_data, typ, w->block,\n+\t\t\t\tw->opts.block_size, block_start,\n+\t\t\t\thash_size(w->opts.hash_id));\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n \tw->block_writer = &w->block_writer_data;\n \tw->block_writer->restart_interval = w->opts.restart_interval;\n+\n+\treturn 0;\n }\n \n int reftable_writer_new(struct reftable_writer **out,\n@@ -247,8 +252,11 @@ static int writer_add_record(struct reftable_writer *w,\n \n \tstrbuf_reset(&w->last_key);\n \tstrbuf_addbuf(&w->last_key, &key);\n-\tif (!w->block_writer)\n-\t\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\tif (!w->block_writer) {\n+\t\terr = writer_reinit_block_writer(w, reftable_record_type(rec));\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t}\n \n \tif (block_writer_type(w->block_writer) != reftable_record_type(rec))\n \t\tBUG(\"record of type %d added to writer of type %d\",\n@@ -271,7 +279,9 @@ static int writer_add_record(struct reftable_writer *w,\n \terr = writer_flush_block(w);\n \tif (err < 0)\n \t\tgoto done;\n-\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\terr = writer_reinit_block_writer(w, reftable_record_type(rec));\n+\tif (err < 0)\n+\t\tgoto done;\n \n \t/*\n \t * Try to add the record to the writer again. If this still fails then\n@@ -461,7 +471,9 @@ static int writer_finish_section(struct reftable_writer *w)\n \n \t\tmax_level++;\n \t\tindex_start = w->next;\n-\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\t\terr = writer_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n \n \t\tidx = w->index;\n \t\tidx_len = w->index_len;\n@@ -555,7 +567,10 @@ static void write_object_record(void *void_arg, void *key)\n \tif (arg->err < 0)\n \t\tgoto done;\n \n-\twriter_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\targ->err = writer_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n \targ->err = block_writer_add(arg->w->block_writer, &rec);\n \tif (arg->err == 0)\n \t\tgoto done;\n@@ -584,16 +599,18 @@ static int writer_dump_object_index(struct reftable_writer *w)\n \tstruct common_prefix_arg common = {\n \t\t.max = 1,\t\t/* obj_id_len should be >= 2. */\n \t};\n-\tif (w->obj_index_tree) {\n+\tint err;\n+\n+\tif (w->obj_index_tree)\n \t\tinfix_walk(w->obj_index_tree, &update_common, &common);\n-\t}\n \tw->stats.object_id_len = common.max + 1;\n \n-\twriter_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\terr = writer_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\tif (err < 0)\n+\t\treturn err;\n \n-\tif (w->obj_index_tree) {\n+\tif (w->obj_index_tree)\n \t\tinfix_walk(w->obj_index_tree, &write_object_record, &closure);\n-\t}\n \n \tif (closure.err < 0)\n \t\treturn closure.err;\ndiff --git a/t/unit-tests/t-reftable-block.c b/t/unit-tests/t-reftable-block.c\nindex f1a49485e2..e52a612e85 100644\n--- a/t/unit-tests/t-reftable-block.c\n+++ b/t/unit-tests/t-reftable-block.c\n@@ -34,8 +34,9 @@ static void t_ref_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \trec.u.ref.refname = (char *) \"\";\n \trec.u.ref.value_type = REFTABLE_REF_DELETION;\n@@ -126,8 +127,9 @@ static void t_log_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n \t\trec.u.log.refname = xstrfmt(\"branch%02\"PRIuMAX , (uintmax_t)i);\n@@ -214,8 +216,9 @@ static void t_obj_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n \t\tuint8_t bytes[] = { i, i + 1, i + 2, i + 3, i + 5 }, *allocated;\n@@ -296,8 +299,9 @@ static void t_index_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n \t\tstrbuf_init(&recs[i].u.idx.last_key, 9);\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503693","messageId":"a66937b7afbdbf7eb33c099056a6a0117844cbd9.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 20/22] reftable/pq: handle allocation failures when adding entries","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:09:08Z","receivedAt":"2024-09-30T08:09:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when adding entries to the pqueue. Adapt its\nonly caller accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 5 ++++-\n reftable/pq.c | 7 ++++++-\n reftable/pq.h | 2 +-\n 3 files changed, 11 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 741f62ea63..2c20845d62 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -54,7 +54,10 @@ static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)\n \tif (err)\n \t\treturn err;\n \n-\tmerged_iter_pqueue_add(&mi->pq, &e);\n+\terr = merged_iter_pqueue_add(&mi->pq, &e);\n+\tif (err)\n+\t\treturn err;\n+\n \treturn 0;\n }\n \ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex 2b5b7d1c0e..03b9912282 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -8,6 +8,7 @@ license that can be found in the LICENSE file or at\n \n #include \"pq.h\"\n \n+#include \"reftable-error.h\"\n #include \"reftable-record.h\"\n #include \"system.h\"\n #include \"basics.h\"\n@@ -44,11 +45,13 @@ struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n \treturn e;\n }\n \n-void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e)\n+int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e)\n {\n \tsize_t i = 0;\n \n \tREFTABLE_ALLOC_GROW(pq->heap, pq->len + 1, pq->cap);\n+\tif (!pq->heap)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tpq->heap[pq->len++] = *e;\n \n \ti = pq->len - 1;\n@@ -59,6 +62,8 @@ void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry\n \t\tSWAP(pq->heap[j], pq->heap[i]);\n \t\ti = j;\n \t}\n+\n+\treturn 0;\n }\n \n void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)\ndiff --git a/reftable/pq.h b/reftable/pq.h\nindex 707bd26767..83c062eeca 100644\n--- a/reftable/pq.h\n+++ b/reftable/pq.h\n@@ -23,7 +23,7 @@ struct merged_iter_pqueue {\n };\n \n struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n-void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e);\n+int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e);\n void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n int pq_less(struct pq_entry *a, struct pq_entry *b);\n \n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503694","messageId":"c291114553016985f5ba635457cbf9c87c05ad2d.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 21/22] reftable/tree: handle allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:09:11Z","receivedAt":"2024-09-30T08:09:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The tree interfaces of the reftable library handle both insertion and\nsearching of tree nodes with a single function, where the behaviour is\naltered between the two via an `insert` bit. This makes it quit awkward\nto handle allocation failures because on inserting we'd have to check\nfor `NULL` pointers and return an error, whereas on searching entries we\ndon't have to handle it as an allocation error.\n\nSplit up concerns of this function into two separate functions, one for\ninserting entries and one for searching entries. This makes it easy for\nus to check for allocation errors as `tree_insert()` should never return\na `NULL` pointer now. Adapt callers accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/tree.c | 42 +++++++++++++++++++++++-----------\n reftable/tree.h | 21 +++++++++++++----\n reftable/writer.c | 7 +++---\n t/unit-tests/t-reftable-tree.c | 10 ++++----\n 4 files changed, 54 insertions(+), 26 deletions(-)\n\ndiff --git a/reftable/tree.c b/reftable/tree.c\nindex 5ffb2e0d69..f4dbe72090 100644\n--- a/reftable/tree.c\n+++ b/reftable/tree.c\n@@ -11,28 +11,44 @@ license that can be found in the LICENSE file or at\n \n #include \"basics.h\"\n \n-struct tree_node *tree_search(void *key, struct tree_node **rootp,\n-\t\t\t int (*compare)(const void *, const void *),\n-\t\t\t int insert)\n+struct tree_node *tree_search(struct tree_node *tree,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *))\n {\n \tint res;\n+\tif (!tree)\n+\t\treturn NULL;\n+\tres = compare(key, tree->key);\n+\tif (res < 0)\n+\t\treturn tree_search(tree->left, key, compare);\n+\telse if (res > 0)\n+\t\treturn tree_search(tree->right, key, compare);\n+\treturn tree;\n+}\n+\n+struct tree_node *tree_insert(struct tree_node **rootp,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *))\n+{\n+\tint res;\n+\n \tif (!*rootp) {\n-\t\tif (!insert) {\n+\t\tstruct tree_node *n;\n+\n+\t\tREFTABLE_CALLOC_ARRAY(n, 1);\n+\t\tif (!n)\n \t\t\treturn NULL;\n-\t\t} else {\n-\t\t\tstruct tree_node *n;\n-\t\t\tREFTABLE_CALLOC_ARRAY(n, 1);\n-\t\t\tn->key = key;\n-\t\t\t*rootp = n;\n-\t\t\treturn *rootp;\n-\t\t}\n+\n+\t\tn->key = key;\n+\t\t*rootp = n;\n+\t\treturn *rootp;\n \t}\n \n \tres = compare(key, (*rootp)->key);\n \tif (res < 0)\n-\t\treturn tree_search(key, &(*rootp)->left, compare, insert);\n+\t\treturn tree_insert(&(*rootp)->left, key, compare);\n \telse if (res > 0)\n-\t\treturn tree_search(key, &(*rootp)->right, compare, insert);\n+\t\treturn tree_insert(&(*rootp)->right, key, compare);\n \treturn *rootp;\n }\n \ndiff --git a/reftable/tree.h b/reftable/tree.h\nindex fbdd002e23..9604453b6d 100644\n--- a/reftable/tree.h\n+++ b/reftable/tree.h\n@@ -15,12 +15,23 @@ struct tree_node {\n \tstruct tree_node *left, *right;\n };\n \n-/* looks for `key` in `rootp` using `compare` as comparison function. If insert\n- * is set, insert the key if it's not found. Else, return NULL.\n+/*\n+ * Search the tree for the node matching the given key using `compare` as\n+ * comparison function. Returns the node whose key matches or `NULL` in case\n+ * the key does not exist in the tree.\n+ */\n+struct tree_node *tree_search(struct tree_node *tree,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *));\n+\n+/*\n+ * Insert a node into the tree. Returns the newly inserted node if the key does\n+ * not yet exist. Otherwise it returns the preexisting node. Returns `NULL`\n+ * when allocating the new node fails.\n */\n-struct tree_node *tree_search(void *key, struct tree_node **rootp,\n-\t\t\t int (*compare)(const void *, const void *),\n-\t\t\t int insert);\n+struct tree_node *tree_insert(struct tree_node **rootp,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *));\n \n /* performs an infix walk of the tree. */\n void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 712a84791b..44bed92f46 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -208,8 +208,7 @@ static int writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n \tstruct obj_index_tree_node *key;\n \tstruct tree_node *node;\n \n-\tnode = tree_search(&want, &w->obj_index_tree,\n-\t\t\t &obj_index_tree_node_compare, 0);\n+\tnode = tree_search(w->obj_index_tree, &want, &obj_index_tree_node_compare);\n \tif (!node) {\n \t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n \n@@ -221,8 +220,8 @@ static int writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n \n \t\tstrbuf_reset(&key->hash);\n \t\tstrbuf_addbuf(&key->hash, hash);\n-\t\ttree_search((void *)key, &w->obj_index_tree,\n-\t\t\t &obj_index_tree_node_compare, 1);\n+\t\ttree_insert(&w->obj_index_tree, key,\n+\t\t\t &obj_index_tree_node_compare);\n \t} else {\n \t\tkey = node->key;\n \t}\ndiff --git a/t/unit-tests/t-reftable-tree.c b/t/unit-tests/t-reftable-tree.c\nindex 700479d34b..79b175a45a 100644\n--- a/t/unit-tests/t-reftable-tree.c\n+++ b/t/unit-tests/t-reftable-tree.c\n@@ -37,16 +37,17 @@ static void t_tree_search(void)\n \t * values[1] and values[10] (inclusive) in the tree.\n \t */\n \tdo {\n-\t\tnodes[i] = tree_search(&values[i], &root, &t_compare, 1);\n+\t\tnodes[i] = tree_insert(&root, &values[i], &t_compare);\n+\t\tcheck(nodes[i] != NULL);\n \t\ti = (i * 7) % 11;\n \t} while (i != 1);\n \n \tfor (i = 1; i < ARRAY_SIZE(nodes); i++) {\n \t\tcheck_pointer_eq(&values[i], nodes[i]->key);\n-\t\tcheck_pointer_eq(nodes[i], tree_search(&values[i], &root, &t_compare, 0));\n+\t\tcheck_pointer_eq(nodes[i], tree_search(root, &values[i], &t_compare));\n \t}\n \n-\tcheck(!tree_search(values, &root, t_compare, 0));\n+\tcheck(!tree_search(root, values, t_compare));\n \ttree_free(root);\n }\n \n@@ -62,7 +63,8 @@ static void t_infix_walk(void)\n \tsize_t count = 0;\n \n \tdo {\n-\t\ttree_search(&values[i], &root, t_compare, 1);\n+\t\tstruct tree_node *node = tree_insert(&root, &values[i], t_compare);\n+\t\tcheck(node != NULL);\n \t\ti = (i * 7) % 11;\n \t\tcount++;\n \t} while (i != 1);\n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503695","messageId":"20f50c446abeb563ecc8c6f8a8be0349abee4dcb.1727680272.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"[PATCH v3 22/22] reftable: handle trivial allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T08:09:13Z","receivedAt":"2024-09-30T08:09:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle trivial allocation failures in the reftable library and its unit\ntests.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 3 +++\n reftable/reader.c | 10 +++++++++-\n reftable/stack.c | 20 ++++++++++++++++++++\n reftable/writer.c | 13 +++++++++++--\n t/unit-tests/t-reftable-block.c | 4 ++++\n t/unit-tests/t-reftable-merged.c | 4 ++++\n t/unit-tests/t-reftable-readwrite.c | 27 ++++++++++++++++-----------\n 7 files changed, 67 insertions(+), 14 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 2c20845d62..52bc66c527 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -203,6 +203,9 @@ int reftable_merged_table_new(struct reftable_merged_table **dest,\n \t}\n \n \tREFTABLE_CALLOC_ARRAY(m, 1);\n+\tif (!m)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \tm->readers = readers;\n \tm->readers_len = n;\n \tm->min = first_min;\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 0179e4e73d..98e7aa2637 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -598,6 +598,10 @@ int reftable_reader_new(struct reftable_reader **out,\n \tint err;\n \n \tREFTABLE_CALLOC_ARRAY(r, 1);\n+\tif (!r) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \n \t/*\n \t * We need one extra byte to read the type of first block. We also\n@@ -627,7 +631,11 @@ int reftable_reader_new(struct reftable_reader **out,\n \n \tr->size = file_size - footer_size(r->version);\n \tr->source = *source;\n-\tr->name = xstrdup(name);\n+\tr->name = reftable_strdup(name);\n+\tif (!r->name) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \tr->hash_id = 0;\n \tr->refcount = 1;\n \ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 990784d9d2..7df28ab343 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -116,6 +116,11 @@ static int fd_read_lines(int fd, char ***namesp)\n \t}\n \n \tREFTABLE_ALLOC_ARRAY(buf, size + 1);\n+\tif (!buf) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tif (read_in_full(fd, buf, size) != size) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n@@ -140,6 +145,8 @@ int read_lines(const char *filename, char ***namesp)\n \tif (fd < 0) {\n \t\tif (errno == ENOENT) {\n \t\t\tREFTABLE_CALLOC_ARRAY(*namesp, 1);\n+\t\t\tif (!*namesp)\n+\t\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \t\t\treturn 0;\n \t\t}\n \n@@ -420,6 +427,10 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\t\t}\n \n \t\t\tREFTABLE_CALLOC_ARRAY(names, 1);\n+\t\t\tif (!names) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto out;\n+\t\t\t}\n \t\t} else {\n \t\t\terr = fd_read_lines(fd, &names);\n \t\t\tif (err < 0)\n@@ -779,7 +790,11 @@ int reftable_stack_new_addition(struct reftable_addition **dest,\n {\n \tint err = 0;\n \tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n+\n \tREFTABLE_CALLOC_ARRAY(*dest, 1);\n+\tif (!*dest)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \t**dest = empty;\n \terr = reftable_stack_init_addition(*dest, st);\n \tif (err) {\n@@ -886,7 +901,12 @@ int reftable_addition_add(struct reftable_addition *add,\n \n \tREFTABLE_ALLOC_GROW(add->new_tables, add->new_tables_len + 1,\n \t\t\t add->new_tables_cap);\n+\tif (!add->new_tables) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \tadd->new_tables[add->new_tables_len++] = strbuf_detach(&next_name, NULL);\n+\n done:\n \tdelete_tempfile(&tab_file);\n \tstrbuf_release(&temp_tab_file_name);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 44bed92f46..ad4718edec 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -49,8 +49,14 @@ static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n {\n \tint n = 0;\n \tif (w->pending_padding > 0) {\n-\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding, sizeof(*zeroed));\n-\t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n+\t\tuint8_t *zeroed;\n+\t\tint n;\n+\n+\t\tzeroed = reftable_calloc(w->pending_padding, sizeof(*zeroed));\n+\t\tif (!zeroed)\n+\t\t\treturn -1;\n+\n+\t\tn = w->write(w->write_arg, zeroed, w->pending_padding);\n \t\tif (n < 0)\n \t\t\treturn n;\n \n@@ -767,6 +773,9 @@ static int writer_flush_nonempty_block(struct reftable_writer *w)\n \t * case we will end up with a multi-level index.\n \t */\n \tREFTABLE_ALLOC_GROW(w->index, w->index_len + 1, w->index_cap);\n+\tif (!w->index)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \tindex_record.offset = w->next;\n \tstrbuf_reset(&index_record.last_key);\n \tstrbuf_addbuf(&index_record.last_key, &w->block_writer->last_key);\ndiff --git a/t/unit-tests/t-reftable-block.c b/t/unit-tests/t-reftable-block.c\nindex e52a612e85..d470060e8b 100644\n--- a/t/unit-tests/t-reftable-block.c\n+++ b/t/unit-tests/t-reftable-block.c\n@@ -32,6 +32,7 @@ static void t_ref_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n@@ -125,6 +126,7 @@ static void t_log_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size,\n@@ -214,6 +216,7 @@ static void t_obj_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size,\n@@ -297,6 +300,7 @@ static void t_index_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size,\ndiff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c\nindex 3d2848632d..3c84363e98 100644\n--- a/t/unit-tests/t-reftable-merged.c\n+++ b/t/unit-tests/t-reftable-merged.c\n@@ -29,7 +29,9 @@ merged_table_from_records(struct reftable_ref_record **refs,\n \tint err;\n \n \tREFTABLE_CALLOC_ARRAY(*readers, n);\n+\tcheck(*readers != NULL);\n \tREFTABLE_CALLOC_ARRAY(*source, n);\n+\tcheck(*source != NULL);\n \n \tfor (size_t i = 0; i < n; i++) {\n \t\tt_reftable_write_to_buf(&buf[i], refs[i], sizes[i], NULL, 0, &opts);\n@@ -285,7 +287,9 @@ merged_table_from_log_records(struct reftable_log_record **logs,\n \tint err;\n \n \tREFTABLE_CALLOC_ARRAY(*readers, n);\n+\tcheck(*readers != NULL);\n \tREFTABLE_CALLOC_ARRAY(*source, n);\n+\tcheck(*source != NULL);\n \n \tfor (size_t i = 0; i < n; i++) {\n \t\tt_reftable_write_to_buf(&buf[i], NULL, 0, logs[i], sizes[i], &opts);\ndiff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c\nindex acca927a2c..bfa069caff 100644\n--- a/t/unit-tests/t-reftable-readwrite.c\n+++ b/t/unit-tests/t-reftable-readwrite.c\n@@ -52,8 +52,11 @@ static void write_table(char ***names, struct strbuf *buf, int N,\n \tint i;\n \n \tREFTABLE_CALLOC_ARRAY(*names, N + 1);\n+\tcheck(*names != NULL);\n \tREFTABLE_CALLOC_ARRAY(refs, N);\n+\tcheck(refs != NULL);\n \tREFTABLE_CALLOC_ARRAY(logs, N);\n+\tcheck(logs != NULL);\n \n \tfor (i = 0; i < N; i++) {\n \t\trefs[i].refname = (*names)[i] = xstrfmt(\"refs/heads/branch%02d\", i);\n@@ -150,23 +153,25 @@ static void t_log_overflow(void)\n \n static void t_log_write_read(void)\n {\n-\tint N = 2;\n-\tchar **names = reftable_calloc(N + 1, sizeof(*names));\n-\tint err;\n \tstruct reftable_write_options opts = {\n \t\t.block_size = 256,\n \t};\n \tstruct reftable_ref_record ref = { 0 };\n-\tint i = 0;\n \tstruct reftable_log_record log = { 0 };\n-\tint n;\n \tstruct reftable_iterator it = { 0 };\n \tstruct reftable_reader *reader;\n \tstruct reftable_block_source source = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w = t_reftable_strbuf_writer(&buf, &opts);\n \tconst struct reftable_stats *stats = NULL;\n+\tint N = 2, err, i, n;\n+\tchar **names;\n+\n+\tnames = reftable_calloc(N + 1, sizeof(*names));\n+\tcheck(names != NULL);\n+\n \treftable_writer_set_limits(w, 0, N);\n+\n \tfor (i = 0; i < N; i++) {\n \t\tchar name[256];\n \t\tstruct reftable_ref_record ref = { 0 };\n@@ -178,6 +183,7 @@ static void t_log_write_read(void)\n \t\terr = reftable_writer_add_ref(w, &ref);\n \t\tcheck(!err);\n \t}\n+\n \tfor (i = 0; i < N; i++) {\n \t\tstruct reftable_log_record log = { 0 };\n \n@@ -476,8 +482,7 @@ static void t_table_read_write_seek_index(void)\n \n static void t_table_refs_for(int indexed)\n {\n-\tint N = 50;\n-\tchar **want_names = reftable_calloc(N + 1, sizeof(*want_names));\n+\tchar **want_names;\n \tint want_names_len = 0;\n \tuint8_t want_hash[GIT_SHA1_RAWSZ];\n \n@@ -485,15 +490,15 @@ static void t_table_refs_for(int indexed)\n \t\t.block_size = 256,\n \t};\n \tstruct reftable_ref_record ref = { 0 };\n-\tint i = 0;\n-\tint n;\n-\tint err;\n \tstruct reftable_reader *reader;\n \tstruct reftable_block_source source = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w = t_reftable_strbuf_writer(&buf, &opts);\n \tstruct reftable_iterator it = { 0 };\n-\tint j;\n+\tint N = 50, n, j, err, i;\n+\n+\twant_names = reftable_calloc(N + 1, sizeof(*want_names));\n+\tcheck(want_names != NULL);\n \n \tt_reftable_set_hash(want_hash, 4, GIT_SHA1_FORMAT_ID);\n \n-- \n2.46.2.852.g229c0bf0e5.dirty\n\n"},{"id":"503750","messageId":"5d37ff9a-9fdd-49c5-b18e-7b49f9d9350b@web.de","threadId":"62121","inReplyTo":"922783708def23c7e0c73db846076cf5e242154b.1727680272.git.ps@pks.im","subject":"Re: [PATCH v3 05/22] reftable/basics: handle allocation failures in `parse_names()`","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2024-09-30T17:40:47Z","receivedAt":"2024-09-30T17:40:54Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 30.09.24 um 10:08 schrieb Patrick Steinhardt:\n> diff --git a/reftable/basics.c b/reftable/basics.c\n> index 3350bbffa2..5ae6e6acdd 100644\n> --- a/reftable/basics.c\n> +++ b/reftable/basics.c\n> @@ -152,14 +152,26 @@ void parse_names(char *buf, int size, char ***namesp)\n> \t\t}\n> \t\tif (p < next) {\n> \t\t\tREFTABLE_ALLOC_GROW(names, names_len + 1, names_cap);\n> -\t\t\tnames[names_len++] = xstrdup(p);\n> +\t\t\tif (!names)\n> +\t\t\t\tgoto err;\n> +\n> +\t\t\tnames[names_len] = strdup(p);\n> +\t\t\tif (!names[names_len++])\n> +\t\t\t\tgoto err;\n> \t\t}\n> \t\tp = next + 1;\n> \t}\n>\n> \tREFTABLE_REALLOC_ARRAY(names, names_len + 1);\n> \tnames[names_len] = NULL;\n> -\t*namesp = names;\n> +\n> +\treturn names;\n> +\n> +err:\n> +\tfor (size_t i = 0; i < names_len; i++)\n> +\t\treftable_free(names[i]);\n\nIf you use reftable_free() here, you'd better use reftable_strdup()\ninstead of strdup(3) above.\n\nRené\n\n"},{"id":"503752","messageId":"64253c71-915d-4862-8fa2-555ca2fc63cc@web.de","threadId":"62121","inReplyTo":"9edd1d84cdbd53d966ff5cfe9b75281dd5966b07.1727680272.git.ps@pks.im","subject":"Re: [PATCH v3 09/22] reftable/writer: handle allocation failures in `reftable_new_writer()`","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2024-09-30T17:40:48Z","receivedAt":"2024-09-30T17:41:02Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 30.09.24 um 10:08 schrieb Patrick Steinhardt:\n> diff --git a/reftable/writer.c b/reftable/writer.c\n> index ed61aaf59c..54ec822e1c 100644\n> --- a/reftable/writer.c\n> +++ b/reftable/writer.c\n> @@ -117,13 +117,17 @@ static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n> \tw->block_writer->restart_interval = w->opts.restart_interval;\n> }\n>\n> -struct reftable_writer *\n> -reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n> -\t\t int (*flush_func)(void *),\n> -\t\t void *writer_arg, const struct reftable_write_options *_opts)\n> +int reftable_writer_new(struct reftable_writer **out,\n> +\t\t\tssize_t (*writer_func)(void *, const void *, size_t),\n> +\t\t\tint (*flush_func)(void *),\n> +\t\t\tvoid *writer_arg, const struct reftable_write_options *_opts)\n> {\n> -\tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n> \tstruct reftable_write_options opts = {0};\n> +\tstruct reftable_writer *wp;\n> +\n> +\twp = reftable_calloc(1, sizeof(*wp));\n> +\tif (!wp)\n> +\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n>\n> \tif (_opts)\n> \t\topts = *_opts;\n> @@ -134,13 +138,19 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n> \tstrbuf_init(&wp->block_writer_data.last_key, 0);\n> \tstrbuf_init(&wp->last_key, 0);\n> \tREFTABLE_CALLOC_ARRAY(wp->block, opts.block_size);\n> +\tif (!wp->block) {\n> +\t\tfree(wp);\n\nBetter use reftable_free() to free it, since you use reftable_calloc()\nto allocate it above.\n\nPerhaps ban free(3), strdup(3) etc. at the end of reftable/basics.h,\nbanned.h style?\n\nRené\n\n"},{"id":"503755","messageId":"xmqqy1396m4e.fsf@gitster.g","threadId":"62121","inReplyTo":"cover.1727680272.git.ps@pks.im","subject":"Re: [PATCH v3 00/22] refatble: handle allocation errors","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-09-30T18:18:41Z","receivedAt":"2024-09-30T18:18:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> this is the third version of this patch series that converts the\n> reftable library to start handling allocation errors. This is done such\n> that the reftable library can truly behave like a library and let its\n> callers handle such conditions.\n>\n> There is only a single change compared to v1, namely that we use\n> `reftable_free()` instead of `free()`.\n\nIndeed that is the only change in this iteration.\n\nLooking good.\n\nThanks.\n"},{"id":"503757","messageId":"Zvrsd5JQr8RAhSaN@pks.im","threadId":"62121","inReplyTo":"64253c71-915d-4862-8fa2-555ca2fc63cc@web.de","subject":"Re: [PATCH v3 09/22] reftable/writer: handle allocation failures in `reftable_new_writer()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T18:22:47Z","receivedAt":"2024-09-30T18:22:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Sep 30, 2024 at 07:40:48PM +0200, René Scharfe wrote:\n> Am 30.09.24 um 10:08 schrieb Patrick Steinhardt:\n> > diff --git a/reftable/writer.c b/reftable/writer.c\n> > index ed61aaf59c..54ec822e1c 100644\n> > --- a/reftable/writer.c\n> > +++ b/reftable/writer.c\n> > @@ -117,13 +117,17 @@ static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n> > \tw->block_writer->restart_interval = w->opts.restart_interval;\n> > }\n> >\n> > -struct reftable_writer *\n> > -reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n> > -\t\t int (*flush_func)(void *),\n> > -\t\t void *writer_arg, const struct reftable_write_options *_opts)\n> > +int reftable_writer_new(struct reftable_writer **out,\n> > +\t\t\tssize_t (*writer_func)(void *, const void *, size_t),\n> > +\t\t\tint (*flush_func)(void *),\n> > +\t\t\tvoid *writer_arg, const struct reftable_write_options *_opts)\n> > {\n> > -\tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n> > \tstruct reftable_write_options opts = {0};\n> > +\tstruct reftable_writer *wp;\n> > +\n> > +\twp = reftable_calloc(1, sizeof(*wp));\n> > +\tif (!wp)\n> > +\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n> >\n> > \tif (_opts)\n> > \t\topts = *_opts;\n> > @@ -134,13 +138,19 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n> > \tstrbuf_init(&wp->block_writer_data.last_key, 0);\n> > \tstrbuf_init(&wp->last_key, 0);\n> > \tREFTABLE_CALLOC_ARRAY(wp->block, opts.block_size);\n> > +\tif (!wp->block) {\n> > +\t\tfree(wp);\n> \n> Better use reftable_free() to free it, since you use reftable_calloc()\n> to allocate it above.\n> \n> Perhaps ban free(3), strdup(3) etc. at the end of reftable/basics.h,\n> banned.h style?\n\nUgh. I was relying too much on your review having mentioned all cases,\nbut I really should've double-checked the other patches, too. Mind you,\nI really don't mean to blame you here, I blame myself.\n\nIn any case, banning these functions via \"reftable/basics.h\" certainly\nseems like a good idea. It's just too easy to screw up by accident. Will\nfix tomorrow.\n\nThanks!\n\nPatrick\n"},{"id":"503760","messageId":"xmqqed516jow.fsf@gitster.g","threadId":"62121","inReplyTo":"Zvrsd5JQr8RAhSaN@pks.im","subject":"Re: [PATCH v3 09/22] reftable/writer: handle allocation failures in `reftable_new_writer()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-09-30T19:11:11Z","receivedAt":"2024-09-30T19:11:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> Perhaps ban free(3), strdup(3) etc. at the end of reftable/basics.h,\n>> banned.h style?\n>\n> Ugh. I was relying too much on your review having mentioned all cases,\n> but I really should've double-checked the other patches, too. Mind you,\n> I really don't mean to blame you here, I blame myself.\n>\n> In any case, banning these functions via \"reftable/basics.h\" certainly\n> seems like a good idea. It's just too easy to screw up by accident. Will\n> fix tomorrow.\n\nSounds like a good plan to do the banned.h-style annotations.\n\nThanks.\n"},{"id":"503803","messageId":"cover.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH v4 00/25] reftable: handle allocation errors","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:41:42Z","receivedAt":"2024-10-01T09:41:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the fourth version of this patch series that converts the\nreftable library to start handling allocation errors. This is done such\nthat the reftable library can truly behave like a library and let its\ncallers handle such conditions.\n\nChanges compared to v3:\n\n - Fix some additional sites where we use strdup(3P)/free(3P) by\n accident.\n\n - Convert preexisting callsites of free(3P) to use reftable_free().\n\n - Introduce a REFTABLE_FREE_AND_NULL() macro and use it.\n\n - Ban standard allocators in reftable code \"banned.h\"-style.\n\nThanks!\n\nPatrick\n\nPatrick Steinhardt (25):\n reftable/error: introduce out-of-memory error code\n reftable/basics: merge \"publicbasics\" into \"basics\"\n reftable: introduce `reftable_strdup()`\n reftable/basics: handle allocation failures in `reftable_calloc()`\n reftable/basics: handle allocation failures in `parse_names()`\n reftable/record: handle allocation failures on copy\n reftable/record: handle allocation failures when decoding records\n reftable/writer: handle allocation failures in `writer_index_hash()`\n reftable/writer: handle allocation failures in `reftable_new_writer()`\n reftable/merged: handle allocation failures in\n `merged_table_init_iter()`\n reftable/reader: handle allocation failures for unindexed reader\n reftable/reader: handle allocation failures in `reader_init_iter()`\n reftable/stack: handle allocation failures on reload\n reftable/stack: handle allocation failures in `reftable_new_stack()`\n reftable/stack: handle allocation failures in `stack_compact_range()`\n reftable/stack: handle allocation failures in auto compaction\n reftable/iter: handle allocation failures when creating indexed table\n iter\n reftable/blocksource: handle allocation failures\n reftable/block: handle allocation failures\n reftable/pq: handle allocation failures when adding entries\n reftable/tree: handle allocation failures\n reftable: handle trivial allocation failures\n reftable: fix calls to free(3P)\n reftable: introduce `REFTABLE_FREE_AND_NULL()`\n reftable/basics: ban standard allocator functions\n\n Makefile | 1 -\n refs/reftable-backend.c | 39 ++++--\n reftable/basics.c | 93 +++++++++++++-\n reftable/basics.h | 23 +++-\n reftable/block.c | 29 ++++-\n reftable/block.h | 4 +-\n reftable/blocksource.c | 25 +++-\n reftable/error.c | 2 +\n reftable/iter.c | 22 +++-\n reftable/iter.h | 2 +-\n reftable/merged.c | 84 ++++++++-----\n reftable/merged.h | 6 +-\n reftable/pq.c | 9 +-\n reftable/pq.h | 2 +-\n reftable/publicbasics.c | 66 ----------\n reftable/reader.c | 70 ++++++++---\n reftable/reader.h | 6 +-\n reftable/record.c | 174 +++++++++++++++++++-------\n reftable/record.h | 6 +-\n reftable/reftable-basics.h | 18 +++\n reftable/reftable-error.h | 3 +\n reftable/reftable-malloc.h | 18 ---\n reftable/reftable-merged.h | 8 +-\n reftable/reftable-reader.h | 8 +-\n reftable/reftable-stack.h | 8 +-\n reftable/reftable-writer.h | 12 +-\n reftable/stack.c | 187 +++++++++++++++++++++-------\n reftable/tree.c | 42 +++++--\n reftable/tree.h | 21 +++-\n reftable/writer.c | 154 +++++++++++++++--------\n t/helper/test-reftable.c | 10 +-\n t/unit-tests/lib-reftable.c | 8 +-\n t/unit-tests/t-reftable-basics.c | 11 +-\n t/unit-tests/t-reftable-block.c | 24 ++--\n t/unit-tests/t-reftable-merged.c | 16 ++-\n t/unit-tests/t-reftable-readwrite.c | 65 ++++++----\n t/unit-tests/t-reftable-stack.c | 6 +-\n t/unit-tests/t-reftable-tree.c | 10 +-\n 38 files changed, 885 insertions(+), 407 deletions(-)\n delete mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-basics.h\n delete mode 100644 reftable/reftable-malloc.h\n\nRange-diff against v3:\n 1: 8c99ecc325 = 1: 94eaef3ae5 reftable/error: introduce out-of-memory error code\n 2: 4dcdf1d48e = 2: fe55051cb7 reftable/basics: merge \"publicbasics\" into \"basics\"\n 3: 21fa9b15d9 = 3: b1a10d41d3 reftable: introduce `reftable_strdup()`\n 4: f6ad92ffd0 = 4: fd9141e985 reftable/basics: handle allocation failures in `reftable_calloc()`\n 5: 922783708d ! 5: bdfddbebce reftable/basics: handle allocation failures in `parse_names()`\n @@ reftable/basics.c: void parse_names(char *buf, int size, char ***namesp)\n +\t\t\tif (!names)\n +\t\t\t\tgoto err;\n +\n -+\t\t\tnames[names_len] = strdup(p);\n ++\t\t\tnames[names_len] = reftable_strdup(p);\n +\t\t\tif (!names[names_len++])\n +\t\t\t\tgoto err;\n \t\t}\n 6: d4676e8a6a = 6: 756a32c285 reftable/record: handle allocation failures on copy\n 7: 874e6a1c20 = 7: ca64971e7b reftable/record: handle allocation failures when decoding records\n 8: 42bc57fd00 = 8: fc2f113cba reftable/writer: handle allocation failures in `writer_index_hash()`\n 9: 9edd1d84cd ! 9: 0ed99e0bdf reftable/writer: handle allocation failures in `reftable_new_writer()`\n @@ reftable/writer.c: reftable_new_writer(ssize_t (*writer_func)(void *, const void\n \tstrbuf_init(&wp->last_key, 0);\n \tREFTABLE_CALLOC_ARRAY(wp->block, opts.block_size);\n +\tif (!wp->block) {\n -+\t\tfree(wp);\n ++\t\treftable_free(wp);\n +\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n +\t}\n \twp->write = writer_func;\n10: d4004a7f43 ! 10: 8dfbfd9286 reftable/merged: handle allocation failures in `merged_table_init_iter()`\n @@ reftable/merged.c: reftable_merged_table_min_update_index(struct reftable_merged\n +\t\t\treftable_iterator_destroy(&subiters[i].iter);\n +\t\t\treftable_record_release(&subiters[i].rec);\n +\t\t}\n -+\t\tfree(subiters);\n -+\t\tfree(mi);\n ++\t\treftable_free(subiters);\n ++\t\treftable_free(mi);\n +\t}\n +\n +\treturn ret;\n11: b2bd142021 = 11: 7b592a6c6b reftable/reader: handle allocation failures for unindexed reader\n12: 9b5fd52862 = 12: ceed838265 reftable/reader: handle allocation failures in `reader_init_iter()`\n13: 0f3e3d1585 = 13: 1e997a5766 reftable/stack: handle allocation failures on reload\n14: c88645a251 = 14: 40d4d81378 reftable/stack: handle allocation failures in `reftable_new_stack()`\n15: 92ad4fc934 = 15: 6aaae4baa6 reftable/stack: handle allocation failures in `stack_compact_range()`\n16: 19acebd919 = 16: 103a59ef0a reftable/stack: handle allocation failures in auto compaction\n17: 620658bffc = 17: 6799d299fe reftable/iter: handle allocation failures when creating indexed table iter\n18: 48047dd4c9 ! 18: c7e54d71d7 reftable/blocksource: handle allocation failures\n @@ reftable/blocksource.c: int reftable_block_source_from_file(struct reftable_bloc\n +\tif (fd >= 0)\n +\t\tclose(fd);\n +\tif (err < 0)\n -+\t\tfree(p);\n ++\t\treftable_free(p);\n \treturn 0;\n }\n19: 08685605ba = 19: 92d39b9021 reftable/block: handle allocation failures\n20: a66937b7af = 20: 3416004e0d reftable/pq: handle allocation failures when adding entries\n21: c291114553 = 21: a363930488 reftable/tree: handle allocation failures\n22: 20f50c446a = 22: 28661500ff reftable: handle trivial allocation failures\n -: ---------- > 23: 228cc81263 reftable: fix calls to free(3P)\n -: ---------- > 24: 1c66f6ef8d reftable: introduce `REFTABLE_FREE_AND_NULL()`\n -: ---------- > 25: 764961e6f0 reftable/basics: ban standard allocator functions\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503804","messageId":"94eaef3ae5ee008b5140272e4ce572c041f9c5a2.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 01/25] reftable/error: introduce out-of-memory error code","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:41:45Z","receivedAt":"2024-10-01T09:41:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library does not use the same memory allocation functions\nas the rest of the Git codebase. Instead, as the reftable library is\nsupposed to be usable as a standalone library without Git, it provides a\nset of pluggable memory allocators.\n\nCompared to `xmalloc()` and friends these allocators are _not_ expected\nto die when an allocation fails. This design choice is concious, as a\nlibrary should leave it to its caller to handle any kind of error. While\nit is very likely that the caller cannot really do much in the case of\nan out-of-memory situation anyway, we are not the ones to make that\ndecision.\n\nCuriously though, we never handle allocation errors even though memory\nallocation functions are allowed to fail. And as we do not plug in Git's\nmemory allocator via `reftable_set_alloc()` either the consequence is\nthat we'd instead segfault as soon as we run out of memory.\n\nWhile the easy fix would be to wire up `xmalloc()` and friends, it\nwould only fix the usage of the reftable library in Git itself. Other\nusers like libgit2, which is about to revive its efforts to land a\nbackend for reftables, wouldn't be able to benefit from this solution.\n\nInstead, we are about to do it the hard way: adapt all allocation sites\nto perform error checking. Introduce a new error code for out-of-memory\nerrors that we will wire up in subsequent steps.\n\nThis commit also serves as the motivator for all the remaining steps in\nthis series such that we do not have to repeat the same arguments in\nevery single subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/error.c | 2 ++\n reftable/reftable-error.h | 3 +++\n 2 files changed, 5 insertions(+)\n\ndiff --git a/reftable/error.c b/reftable/error.c\nindex a25f28a43e..660d029617 100644\n--- a/reftable/error.c\n+++ b/reftable/error.c\n@@ -35,6 +35,8 @@ const char *reftable_error_str(int err)\n \t\treturn \"entry too large\";\n \tcase REFTABLE_OUTDATED_ERROR:\n \t\treturn \"data concurrently modified\";\n+\tcase REFTABLE_OUT_OF_MEMORY_ERROR:\n+\t\treturn \"out of memory\";\n \tcase -1:\n \t\treturn \"general error\";\n \tdefault:\ndiff --git a/reftable/reftable-error.h b/reftable/reftable-error.h\nindex 6368cd9ed9..f404826562 100644\n--- a/reftable/reftable-error.h\n+++ b/reftable/reftable-error.h\n@@ -57,6 +57,9 @@ enum reftable_error {\n \n \t/* Trying to write out-of-date data. */\n \tREFTABLE_OUTDATED_ERROR = -12,\n+\n+\t/* An allocation has failed due to an out-of-memory situation. */\n+\tREFTABLE_OUT_OF_MEMORY_ERROR = -13,\n };\n \n /* convert the numeric error code to a string. The string should not be\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503805","messageId":"fe55051cb79015960729d7a06f21f43138585177.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 02/25] reftable/basics: merge \"publicbasics\" into \"basics\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:41:47Z","receivedAt":"2024-10-01T09:41:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The split between \"basics\" and \"publicbasics\" is somewhat arbitrary and\nnot in line with how we typically structure code in the reftable\nlibrary. While we do indeed split up headers into a public and internal\npart, we don't do that for the compilation unit itself. Furthermore, the\ndeclarations for \"publicbasics.c\" are in \"reftable-malloc.h\", which\nisn't in line with our naming schema, either.\n\nFix these inconsistencies by:\n\n - Merging \"publicbasics.c\" into \"basics.c\".\n\n - Renaming \"reftable-malloc.h\" to \"reftable-basics.h\" as the public\n header.\n\nSigned-off-by: Patrick Steinhardt \n---\n Makefile | 1 -\n reftable/basics.c | 55 +++++++++++++++++++++++++++++++\n reftable/basics.h | 3 ++\n reftable/publicbasics.c | 66 --------------------------------------\n reftable/reftable-basics.h | 18 +++++++++++\n reftable/reftable-malloc.h | 18 -----------\n 6 files changed, 76 insertions(+), 85 deletions(-)\n delete mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-basics.h\n delete mode 100644 reftable/reftable-malloc.h\n\ndiff --git a/Makefile b/Makefile\nindex e3abf0ba83..39b10923d4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2683,7 +2683,6 @@ REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n-REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/merged.o\n REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 0058619ca6..cf072935c8 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -7,6 +7,49 @@ license that can be found in the LICENSE file or at\n */\n \n #include \"basics.h\"\n+#include \"reftable-basics.h\"\n+\n+static void *(*reftable_malloc_ptr)(size_t sz);\n+static void *(*reftable_realloc_ptr)(void *, size_t);\n+static void (*reftable_free_ptr)(void *);\n+\n+void *reftable_malloc(size_t sz)\n+{\n+\tif (reftable_malloc_ptr)\n+\t\treturn (*reftable_malloc_ptr)(sz);\n+\treturn malloc(sz);\n+}\n+\n+void *reftable_realloc(void *p, size_t sz)\n+{\n+\tif (reftable_realloc_ptr)\n+\t\treturn (*reftable_realloc_ptr)(p, sz);\n+\treturn realloc(p, sz);\n+}\n+\n+void reftable_free(void *p)\n+{\n+\tif (reftable_free_ptr)\n+\t\treftable_free_ptr(p);\n+\telse\n+\t\tfree(p);\n+}\n+\n+void *reftable_calloc(size_t nelem, size_t elsize)\n+{\n+\tsize_t sz = st_mult(nelem, elsize);\n+\tvoid *p = reftable_malloc(sz);\n+\tmemset(p, 0, sz);\n+\treturn p;\n+}\n+\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n+{\n+\treftable_malloc_ptr = malloc;\n+\treftable_realloc_ptr = realloc;\n+\treftable_free_ptr = free;\n+}\n \n void put_be24(uint8_t *out, uint32_t i)\n {\n@@ -121,3 +164,15 @@ int common_prefix_size(struct strbuf *a, struct strbuf *b)\n \n \treturn p;\n }\n+\n+int hash_size(uint32_t id)\n+{\n+\tswitch (id) {\n+\tcase 0:\n+\tcase GIT_SHA1_FORMAT_ID:\n+\t\treturn GIT_SHA1_RAWSZ;\n+\tcase GIT_SHA256_FORMAT_ID:\n+\t\treturn GIT_SHA256_RAWSZ;\n+\t}\n+\tabort();\n+}\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex c8fec68d4e..4e2e76014a 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -14,6 +14,7 @@ license that can be found in the LICENSE file or at\n */\n \n #include \"system.h\"\n+#include \"reftable-basics.h\"\n \n /* Bigendian en/decoding of integers */\n \n@@ -71,4 +72,6 @@ void *reftable_calloc(size_t nelem, size_t elsize);\n struct strbuf;\n int common_prefix_size(struct strbuf *a, struct strbuf *b);\n \n+int hash_size(uint32_t id);\n+\n #endif\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\ndeleted file mode 100644\nindex 44b84a125e..0000000000\n--- a/reftable/publicbasics.c\n+++ /dev/null\n@@ -1,66 +0,0 @@\n-/*\n-Copyright 2020 Google LLC\n-\n-Use of this source code is governed by a BSD-style\n-license that can be found in the LICENSE file or at\n-https://developers.google.com/open-source/licenses/bsd\n-*/\n-\n-#include \"system.h\"\n-#include \"reftable-malloc.h\"\n-\n-#include \"basics.h\"\n-\n-static void *(*reftable_malloc_ptr)(size_t sz);\n-static void *(*reftable_realloc_ptr)(void *, size_t);\n-static void (*reftable_free_ptr)(void *);\n-\n-void *reftable_malloc(size_t sz)\n-{\n-\tif (reftable_malloc_ptr)\n-\t\treturn (*reftable_malloc_ptr)(sz);\n-\treturn malloc(sz);\n-}\n-\n-void *reftable_realloc(void *p, size_t sz)\n-{\n-\tif (reftable_realloc_ptr)\n-\t\treturn (*reftable_realloc_ptr)(p, sz);\n-\treturn realloc(p, sz);\n-}\n-\n-void reftable_free(void *p)\n-{\n-\tif (reftable_free_ptr)\n-\t\treftable_free_ptr(p);\n-\telse\n-\t\tfree(p);\n-}\n-\n-void *reftable_calloc(size_t nelem, size_t elsize)\n-{\n-\tsize_t sz = st_mult(nelem, elsize);\n-\tvoid *p = reftable_malloc(sz);\n-\tmemset(p, 0, sz);\n-\treturn p;\n-}\n-\n-void reftable_set_alloc(void *(*malloc)(size_t),\n-\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n-{\n-\treftable_malloc_ptr = malloc;\n-\treftable_realloc_ptr = realloc;\n-\treftable_free_ptr = free;\n-}\n-\n-int hash_size(uint32_t id)\n-{\n-\tswitch (id) {\n-\tcase 0:\n-\tcase GIT_SHA1_FORMAT_ID:\n-\t\treturn GIT_SHA1_RAWSZ;\n-\tcase GIT_SHA256_FORMAT_ID:\n-\t\treturn GIT_SHA256_RAWSZ;\n-\t}\n-\tabort();\n-}\ndiff --git a/reftable/reftable-basics.h b/reftable/reftable-basics.h\nnew file mode 100644\nindex 0000000000..6e8e636b71\n--- /dev/null\n+++ b/reftable/reftable-basics.h\n@@ -0,0 +1,18 @@\n+/*\n+ * Copyright 2020 Google LLC\n+ *\n+ * Use of this source code is governed by a BSD-style\n+ * license that can be found in the LICENSE file or at\n+ * https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_BASICS_H\n+#define REFTABLE_BASICS_H\n+\n+#include \n+\n+/* Overrides the functions to use for memory management. */\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n+\n+#endif\ndiff --git a/reftable/reftable-malloc.h b/reftable/reftable-malloc.h\ndeleted file mode 100644\nindex 5f2185f1f3..0000000000\n--- a/reftable/reftable-malloc.h\n+++ /dev/null\n@@ -1,18 +0,0 @@\n-/*\n-Copyright 2020 Google LLC\n-\n-Use of this source code is governed by a BSD-style\n-license that can be found in the LICENSE file or at\n-https://developers.google.com/open-source/licenses/bsd\n-*/\n-\n-#ifndef REFTABLE_H\n-#define REFTABLE_H\n-\n-#include \n-\n-/* Overrides the functions to use for memory management. */\n-void reftable_set_alloc(void *(*malloc)(size_t),\n-\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n-\n-#endif\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503806","messageId":"b1a10d41d3798c98c53b251df9077fcc5eaef560.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 03/25] reftable: introduce `reftable_strdup()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:41:50Z","receivedAt":"2024-10-01T09:41:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library provides the ability to swap out allocators. There\nis a gap here though, because we continue to use `xstrdup()` even in the\ncase where all the other allocators have been swapped out.\n\nIntroduce `reftable_strdup()` that uses `reftable_malloc()` to do the\nallocation.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 10 ++++++++++\n reftable/basics.h | 1 +\n 2 files changed, 11 insertions(+)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex cf072935c8..4adc98cf5d 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -43,6 +43,16 @@ void *reftable_calloc(size_t nelem, size_t elsize)\n \treturn p;\n }\n \n+char *reftable_strdup(const char *str)\n+{\n+\tsize_t len = strlen(str);\n+\tchar *result = reftable_malloc(len + 1);\n+\tif (!result)\n+\t\treturn NULL;\n+\tmemcpy(result, str, len + 1);\n+\treturn result;\n+}\n+\n void reftable_set_alloc(void *(*malloc)(size_t),\n \t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n {\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 4e2e76014a..f107e14860 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -54,6 +54,7 @@ void *reftable_malloc(size_t sz);\n void *reftable_realloc(void *p, size_t sz);\n void reftable_free(void *p);\n void *reftable_calloc(size_t nelem, size_t elsize);\n+char *reftable_strdup(const char *str);\n \n #define REFTABLE_ALLOC_ARRAY(x, alloc) (x) = reftable_malloc(st_mult(sizeof(*(x)), (alloc)))\n #define REFTABLE_CALLOC_ARRAY(x, alloc) (x) = reftable_calloc((alloc), sizeof(*(x)))\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503807","messageId":"fd9141e985bd83d8374ec339dbdce7a2350aa31d.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 04/25] reftable/basics: handle allocation failures in `reftable_calloc()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:41:53Z","receivedAt":"2024-10-01T09:42:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_calloc()`.\n\nWhile at it, remove our use of `st_mult()` that would cause us to die on\nan overflow. From the caller's point of view there is not much of a\ndifference between arguments that are too large to be multiplied and a\nrequest that is too big to handle by the allocator: in both cases the\nallocation cannot be fulfilled. And in neither of these cases do we want\nthe reftable library to die.\n\nWhile we could use `unsigned_mult_overflows()` to handle the overflow\ngracefully, we instead open-code it to further our goal of converting\nthe reftable codebase to become a standalone library that can be reused\nby external projects.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 13 ++++++++++---\n 1 file changed, 10 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 4adc98cf5d..3350bbffa2 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -37,9 +37,16 @@ void reftable_free(void *p)\n \n void *reftable_calloc(size_t nelem, size_t elsize)\n {\n-\tsize_t sz = st_mult(nelem, elsize);\n-\tvoid *p = reftable_malloc(sz);\n-\tmemset(p, 0, sz);\n+\tvoid *p;\n+\n+\tif (nelem && elsize > SIZE_MAX / nelem)\n+\t\treturn NULL;\n+\n+\tp = reftable_malloc(nelem * elsize);\n+\tif (!p)\n+\t\treturn NULL;\n+\n+\tmemset(p, 0, nelem * elsize);\n \treturn p;\n }\n \n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503808","messageId":"bdfddbebce9f77959fd9544cd5ba3496d5b9dccf.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 05/25] reftable/basics: handle allocation failures in `parse_names()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:41:59Z","receivedAt":"2024-10-01T09:42:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `parse_names()` by returning `NULL` in\ncase any allocation fails. While at it, refactor the function to return\nthe array directly instead of assigning it to an out-pointer.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 20 ++++++++++++++++----\n reftable/basics.h | 9 ++++++---\n reftable/stack.c | 6 +++++-\n t/unit-tests/t-reftable-basics.c | 11 ++++++-----\n 4 files changed, 33 insertions(+), 13 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 3350bbffa2..ea53cf102a 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -135,14 +135,14 @@ size_t names_length(const char **names)\n \treturn p - names;\n }\n \n-void parse_names(char *buf, int size, char ***namesp)\n+char **parse_names(char *buf, int size)\n {\n \tchar **names = NULL;\n \tsize_t names_cap = 0;\n \tsize_t names_len = 0;\n-\n \tchar *p = buf;\n \tchar *end = buf + size;\n+\n \twhile (p < end) {\n \t\tchar *next = strchr(p, '\\n');\n \t\tif (next && next < end) {\n@@ -152,14 +152,26 @@ void parse_names(char *buf, int size, char ***namesp)\n \t\t}\n \t\tif (p < next) {\n \t\t\tREFTABLE_ALLOC_GROW(names, names_len + 1, names_cap);\n-\t\t\tnames[names_len++] = xstrdup(p);\n+\t\t\tif (!names)\n+\t\t\t\tgoto err;\n+\n+\t\t\tnames[names_len] = reftable_strdup(p);\n+\t\t\tif (!names[names_len++])\n+\t\t\t\tgoto err;\n \t\t}\n \t\tp = next + 1;\n \t}\n \n \tREFTABLE_REALLOC_ARRAY(names, names_len + 1);\n \tnames[names_len] = NULL;\n-\t*namesp = names;\n+\n+\treturn names;\n+\n+err:\n+\tfor (size_t i = 0; i < names_len; i++)\n+\t\treftable_free(names[i]);\n+\treftable_free(names);\n+\treturn NULL;\n }\n \n int names_equal(const char **a, const char **b)\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex f107e14860..69adeab2e4 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -38,9 +38,12 @@ size_t binsearch(size_t sz, int (*f)(size_t k, void *args), void *args);\n */\n void free_names(char **a);\n \n-/* parse a newline separated list of names. `size` is the length of the buffer,\n- * without terminating '\\0'. Empty names are discarded. */\n-void parse_names(char *buf, int size, char ***namesp);\n+/*\n+ * Parse a newline separated list of names. `size` is the length of the buffer,\n+ * without terminating '\\0'. Empty names are discarded. Returns a `NULL`\n+ * pointer when allocations fail.\n+ */\n+char **parse_names(char *buf, int size);\n \n /* compares two NULL-terminated arrays of strings. */\n int names_equal(const char **a, const char **b);\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex ce0a35216b..498fae846d 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -108,7 +108,11 @@ static int fd_read_lines(int fd, char ***namesp)\n \t}\n \tbuf[size] = 0;\n \n-\tparse_names(buf, size, namesp);\n+\t*namesp = parse_names(buf, size);\n+\tif (!*namesp) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \n done:\n \treftable_free(buf);\ndiff --git a/t/unit-tests/t-reftable-basics.c b/t/unit-tests/t-reftable-basics.c\nindex e5556ebf52..1fa77b6faf 100644\n--- a/t/unit-tests/t-reftable-basics.c\n+++ b/t/unit-tests/t-reftable-basics.c\n@@ -72,13 +72,14 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \tif_test (\"parse_names works for basic input\") {\n \t\tchar in1[] = \"line\\n\";\n \t\tchar in2[] = \"a\\nb\\nc\";\n-\t\tchar **out = NULL;\n-\t\tparse_names(in1, strlen(in1), &out);\n+\t\tchar **out = parse_names(in1, strlen(in1));\n+\t\tcheck(out != NULL);\n \t\tcheck_str(out[0], \"line\");\n \t\tcheck(!out[1]);\n \t\tfree_names(out);\n \n-\t\tparse_names(in2, strlen(in2), &out);\n+\t\tout = parse_names(in2, strlen(in2));\n+\t\tcheck(out != NULL);\n \t\tcheck_str(out[0], \"a\");\n \t\tcheck_str(out[1], \"b\");\n \t\tcheck_str(out[2], \"c\");\n@@ -88,8 +89,8 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \n \tif_test (\"parse_names drops empty string\") {\n \t\tchar in[] = \"a\\n\\nb\\n\";\n-\t\tchar **out = NULL;\n-\t\tparse_names(in, strlen(in), &out);\n+\t\tchar **out = parse_names(in, strlen(in));\n+\t\tcheck(out != NULL);\n \t\tcheck_str(out[0], \"a\");\n \t\t/* simply '\\n' should be dropped as empty string */\n \t\tcheck_str(out[1], \"b\");\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503809","messageId":"756a32c28558bfaa360304712110443d643fc303.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 06/25] reftable/record: handle allocation failures on copy","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:42:01Z","receivedAt":"2024-10-01T09:42:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when copying records. While at it, convert\nfrom `xstrdup()` to `reftable_strdup()`. Adapt callsites to check for\nerror codes.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/record.c | 84 +++++++++++++++++++++++++++++++++--------------\n reftable/record.h | 6 ++--\n 2 files changed, 63 insertions(+), 27 deletions(-)\n\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 6b5a075b92..60fd33c9c9 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -215,13 +215,14 @@ static void reftable_ref_record_key(const void *r, struct strbuf *dest)\n \tstrbuf_addstr(dest, rec->refname);\n }\n \n-static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size)\n+static int reftable_ref_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n {\n \tstruct reftable_ref_record *ref = rec;\n \tconst struct reftable_ref_record *src = src_rec;\n \tchar *refname = NULL;\n \tsize_t refname_cap = 0;\n+\tint err;\n \n \tassert(hash_size > 0);\n \n@@ -236,6 +237,11 @@ static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n \n \t\tREFTABLE_ALLOC_GROW(ref->refname, refname_len + 1,\n \t\t\t\t ref->refname_cap);\n+\t\tif (!ref->refname) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n+\n \t\tmemcpy(ref->refname, src->refname, refname_len);\n \t\tref->refname[refname_len] = 0;\n \t}\n@@ -254,9 +260,17 @@ static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n \t\t src->value.val2.target_value, hash_size);\n \t\tbreak;\n \tcase REFTABLE_REF_SYMREF:\n-\t\tref->value.symref = xstrdup(src->value.symref);\n+\t\tref->value.symref = reftable_strdup(src->value.symref);\n+\t\tif (!ref->value.symref) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n \t\tbreak;\n \t}\n+\n+\terr = 0;\n+out:\n+\treturn err;\n }\n \n static void reftable_ref_record_release_void(void *rec)\n@@ -457,23 +471,28 @@ static void reftable_obj_record_release(void *rec)\n \tmemset(obj, 0, sizeof(struct reftable_obj_record));\n }\n \n-static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size UNUSED)\n+static int reftable_obj_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size UNUSED)\n {\n \tstruct reftable_obj_record *obj = rec;\n-\tconst struct reftable_obj_record *src =\n-\t\t(const struct reftable_obj_record *)src_rec;\n+\tconst struct reftable_obj_record *src = src_rec;\n \n \treftable_obj_record_release(obj);\n \n \tREFTABLE_ALLOC_ARRAY(obj->hash_prefix, src->hash_prefix_len);\n+\tif (!obj->hash_prefix)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tobj->hash_prefix_len = src->hash_prefix_len;\n \tif (src->hash_prefix_len)\n \t\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n \n \tREFTABLE_ALLOC_ARRAY(obj->offsets, src->offset_len);\n+\tif (!obj->offsets)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tobj->offset_len = src->offset_len;\n \tCOPY_ARRAY(obj->offsets, src->offsets, src->offset_len);\n+\n+\treturn 0;\n }\n \n static uint8_t reftable_obj_record_val_type(const void *rec)\n@@ -646,33 +665,44 @@ static void reftable_log_record_key(const void *r, struct strbuf *dest)\n \tstrbuf_add(dest, i64, sizeof(i64));\n }\n \n-static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size)\n+static int reftable_log_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n {\n \tstruct reftable_log_record *dst = rec;\n \tconst struct reftable_log_record *src =\n \t\t(const struct reftable_log_record *)src_rec;\n+\tint ret;\n \n \treftable_log_record_release(dst);\n \t*dst = *src;\n+\n \tif (dst->refname) {\n-\t\tdst->refname = xstrdup(dst->refname);\n+\t\tdst->refname = reftable_strdup(dst->refname);\n+\t\tif (!dst->refname) {\n+\t\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n \t}\n+\n \tswitch (dst->value_type) {\n \tcase REFTABLE_LOG_DELETION:\n \t\tbreak;\n \tcase REFTABLE_LOG_UPDATE:\n-\t\tif (dst->value.update.email) {\n+\t\tif (dst->value.update.email)\n \t\t\tdst->value.update.email =\n-\t\t\t\txstrdup(dst->value.update.email);\n-\t\t}\n-\t\tif (dst->value.update.name) {\n+\t\t\t\treftable_strdup(dst->value.update.email);\n+\t\tif (dst->value.update.name)\n \t\t\tdst->value.update.name =\n-\t\t\t\txstrdup(dst->value.update.name);\n-\t\t}\n-\t\tif (dst->value.update.message) {\n+\t\t\t\treftable_strdup(dst->value.update.name);\n+\t\tif (dst->value.update.message)\n \t\t\tdst->value.update.message =\n-\t\t\t\txstrdup(dst->value.update.message);\n+\t\t\t\treftable_strdup(dst->value.update.message);\n+\n+\t\tif (!dst->value.update.email ||\n+\t\t !dst->value.update.name ||\n+\t\t !dst->value.update.message) {\n+\t\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n \t\t}\n \n \t\tmemcpy(dst->value.update.new_hash,\n@@ -681,6 +711,10 @@ static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n \t\t src->value.update.old_hash, hash_size);\n \t\tbreak;\n \t}\n+\n+\tret = 0;\n+out:\n+\treturn ret;\n }\n \n static void reftable_log_record_release_void(void *rec)\n@@ -954,8 +988,8 @@ static void reftable_index_record_key(const void *r, struct strbuf *dest)\n \tstrbuf_addbuf(dest, &rec->last_key);\n }\n \n-static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size UNUSED)\n+static int reftable_index_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size UNUSED)\n {\n \tstruct reftable_index_record *dst = rec;\n \tconst struct reftable_index_record *src = src_rec;\n@@ -963,6 +997,8 @@ static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n \tstrbuf_reset(&dst->last_key);\n \tstrbuf_addbuf(&dst->last_key, &src->last_key);\n \tdst->offset = src->offset;\n+\n+\treturn 0;\n }\n \n static void reftable_index_record_release(void *rec)\n@@ -1054,14 +1090,14 @@ int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n \t\t\t\t\t\t dest, hash_size);\n }\n \n-void reftable_record_copy_from(struct reftable_record *rec,\n+int reftable_record_copy_from(struct reftable_record *rec,\n \t\t\t struct reftable_record *src, int hash_size)\n {\n \tassert(src->type == rec->type);\n \n-\treftable_record_vtable(rec)->copy_from(reftable_record_data(rec),\n-\t\t\t\t\t reftable_record_data(src),\n-\t\t\t\t\t hash_size);\n+\treturn reftable_record_vtable(rec)->copy_from(reftable_record_data(rec),\n+\t\t\t\t\t\t reftable_record_data(src),\n+\t\t\t\t\t\t hash_size);\n }\n \n uint8_t reftable_record_val_type(struct reftable_record *rec)\ndiff --git a/reftable/record.h b/reftable/record.h\nindex 5003bacdb0..0f53ba5443 100644\n--- a/reftable/record.h\n+++ b/reftable/record.h\n@@ -44,7 +44,7 @@ struct reftable_record_vtable {\n \t/* The record type of ('r' for ref). */\n \tuint8_t type;\n \n-\tvoid (*copy_from)(void *dest, const void *src, int hash_size);\n+\tint (*copy_from)(void *dest, const void *src, int hash_size);\n \n \t/* a value of [0..7], indicating record subvariants (eg. ref vs. symref\n \t * vs ref deletion) */\n@@ -137,8 +137,8 @@ void reftable_record_init(struct reftable_record *rec, uint8_t typ);\n int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b);\n int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);\n void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);\n-void reftable_record_copy_from(struct reftable_record *rec,\n-\t\t\t struct reftable_record *src, int hash_size);\n+int reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size);\n uint8_t reftable_record_val_type(struct reftable_record *rec);\n int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n \t\t\t int hash_size);\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503810","messageId":"ca64971e7b892524e553d20e6726c95dd1865787.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 07/25] reftable/record: handle allocation failures when decoding records","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:42:04Z","receivedAt":"2024-10-01T09:42:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when decoding records. While at it, fix some\nerror codes to be `REFTABLE_FORMAT_ERROR`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/record.c | 80 +++++++++++++++++++++++++++++++++++++----------\n 1 file changed, 63 insertions(+), 17 deletions(-)\n\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 60fd33c9c9..787e134c9a 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -359,7 +359,7 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tuint64_t update_index = 0;\n \tconst char *refname = NULL;\n \tsize_t refname_cap = 0;\n-\tint n;\n+\tint n, err;\n \n \tassert(hash_size > 0);\n \n@@ -375,6 +375,10 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tSWAP(r->refname_cap, refname_cap);\n \n \tREFTABLE_ALLOC_GROW(r->refname, key.len + 1, r->refname_cap);\n+\tif (!r->refname) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \tmemcpy(r->refname, key.buf, key.len);\n \tr->refname[key.len] = 0;\n \n@@ -383,7 +387,8 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tswitch (val_type) {\n \tcase REFTABLE_REF_VAL1:\n \t\tif (in.len < hash_size) {\n-\t\t\treturn -1;\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n \t\t}\n \n \t\tmemcpy(r->value.val1, in.buf, hash_size);\n@@ -392,7 +397,8 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \n \tcase REFTABLE_REF_VAL2:\n \t\tif (in.len < 2 * hash_size) {\n-\t\t\treturn -1;\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n \t\t}\n \n \t\tmemcpy(r->value.val2.value, in.buf, hash_size);\n@@ -405,7 +411,8 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tcase REFTABLE_REF_SYMREF: {\n \t\tint n = decode_string(scratch, in);\n \t\tif (n < 0) {\n-\t\t\treturn -1;\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n \t\t}\n \t\tstring_view_consume(&in, n);\n \t\tr->value.symref = strbuf_detach(scratch, NULL);\n@@ -419,6 +426,9 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \t}\n \n \treturn start.len - in.len;\n+\n+done:\n+\treturn err;\n }\n \n static int reftable_ref_record_is_deletion_void(const void *p)\n@@ -552,6 +562,8 @@ static int reftable_obj_record_decode(void *rec, struct strbuf key,\n \treftable_obj_record_release(r);\n \n \tREFTABLE_ALLOC_ARRAY(r->hash_prefix, key.len);\n+\tif (!r->hash_prefix)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tmemcpy(r->hash_prefix, key.buf, key.len);\n \tr->hash_prefix_len = key.len;\n \n@@ -570,6 +582,8 @@ static int reftable_obj_record_decode(void *rec, struct strbuf key,\n \t\treturn start.len - in.len;\n \n \tREFTABLE_ALLOC_ARRAY(r->offsets, count);\n+\tif (!r->offsets)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tr->offset_len = count;\n \n \tn = get_var_int(&r->offsets[0], &in);\n@@ -801,12 +815,17 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \tstruct reftable_log_record *r = rec;\n \tuint64_t max = 0;\n \tuint64_t ts = 0;\n-\tint n;\n+\tint err, n;\n \n \tif (key.len <= 9 || key.buf[key.len - 9] != 0)\n \t\treturn REFTABLE_FORMAT_ERROR;\n \n \tREFTABLE_ALLOC_GROW(r->refname, key.len - 8, r->refname_cap);\n+\tif (!r->refname) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tmemcpy(r->refname, key.buf, key.len - 8);\n \tts = get_be64(key.buf + key.len - 8);\n \n@@ -829,8 +848,10 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \tif (val_type == REFTABLE_LOG_DELETION)\n \t\treturn 0;\n \n-\tif (in.len < 2 * hash_size)\n-\t\treturn REFTABLE_FORMAT_ERROR;\n+\tif (in.len < 2 * hash_size) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n \n \tmemcpy(r->value.update.old_hash, in.buf, hash_size);\n \tmemcpy(r->value.update.new_hash, in.buf + hash_size, hash_size);\n@@ -838,8 +859,10 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \tstring_view_consume(&in, 2 * hash_size);\n \n \tn = decode_string(scratch, in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \n \t/*\n@@ -850,52 +873,75 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \t */\n \tif (!r->value.update.name ||\n \t strcmp(r->value.update.name, scratch->buf)) {\n-\t\tr->value.update.name =\n-\t\t\treftable_realloc(r->value.update.name, scratch->len + 1);\n+\t\tchar *name = reftable_realloc(r->value.update.name, scratch->len + 1);\n+\t\tif (!name) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tr->value.update.name = name;\n \t\tmemcpy(r->value.update.name, scratch->buf, scratch->len);\n \t\tr->value.update.name[scratch->len] = 0;\n \t}\n \n \tn = decode_string(scratch, in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \n \t/* Same as above, but for the reflog email. */\n \tif (!r->value.update.email ||\n \t strcmp(r->value.update.email, scratch->buf)) {\n-\t\tr->value.update.email =\n-\t\t\treftable_realloc(r->value.update.email, scratch->len + 1);\n+\t\tchar *email = reftable_realloc(r->value.update.email, scratch->len + 1);\n+\t\tif (!email) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tr->value.update.email = email;\n \t\tmemcpy(r->value.update.email, scratch->buf, scratch->len);\n \t\tr->value.update.email[scratch->len] = 0;\n \t}\n \n \tts = 0;\n \tn = get_var_int(&ts, &in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \tr->value.update.time = ts;\n-\tif (in.len < 2)\n+\tif (in.len < 2) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \n \tr->value.update.tz_offset = get_be16(in.buf);\n \tstring_view_consume(&in, 2);\n \n \tn = decode_string(scratch, in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \n \tREFTABLE_ALLOC_GROW(r->value.update.message, scratch->len + 1,\n \t\t\t r->value.update.message_cap);\n+\tif (!r->value.update.message) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tmemcpy(r->value.update.message, scratch->buf, scratch->len);\n \tr->value.update.message[scratch->len] = 0;\n \n \treturn start.len - in.len;\n \n done:\n-\treturn REFTABLE_FORMAT_ERROR;\n+\treturn err;\n }\n \n static int null_streq(const char *a, const char *b)\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503811","messageId":"fc2f113cba3542fa8d37fc20d7825b70022f55aa.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 08/25] reftable/writer: handle allocation failures in `writer_index_hash()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:42:07Z","receivedAt":"2024-10-01T09:42:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation errors in `writer_index_hash()`. Adjust its only\ncaller in `reftable_writer_add_ref()` accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/writer.c | 61 +++++++++++++++++++++++++++++------------------\n 1 file changed, 38 insertions(+), 23 deletions(-)\n\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 9d5e6072bc..ed61aaf59c 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -186,18 +186,22 @@ static int obj_index_tree_node_compare(const void *a, const void *b)\n \t\t\t &((const struct obj_index_tree_node *)b)->hash);\n }\n \n-static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n+static int writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n {\n \tuint64_t off = w->next;\n-\n \tstruct obj_index_tree_node want = { .hash = *hash };\n+\tstruct obj_index_tree_node *key;\n+\tstruct tree_node *node;\n \n-\tstruct tree_node *node = tree_search(&want, &w->obj_index_tree,\n-\t\t\t\t\t &obj_index_tree_node_compare, 0);\n-\tstruct obj_index_tree_node *key = NULL;\n+\tnode = tree_search(&want, &w->obj_index_tree,\n+\t\t\t &obj_index_tree_node_compare, 0);\n \tif (!node) {\n \t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n-\t\tkey = reftable_malloc(sizeof(struct obj_index_tree_node));\n+\n+\t\tkey = reftable_malloc(sizeof(*key));\n+\t\tif (!key)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \t\t*key = empty;\n \n \t\tstrbuf_reset(&key->hash);\n@@ -208,12 +212,15 @@ static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n \t\tkey = node->key;\n \t}\n \n-\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off) {\n-\t\treturn;\n-\t}\n+\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off)\n+\t\treturn 0;\n \n \tREFTABLE_ALLOC_GROW(key->offsets, key->offset_len + 1, key->offset_cap);\n+\tif (!key->offsets)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tkey->offsets[key->offset_len++] = off;\n+\n+\treturn 0;\n }\n \n static int writer_add_record(struct reftable_writer *w,\n@@ -284,11 +291,11 @@ int reftable_writer_add_ref(struct reftable_writer *w,\n \t\t\t.ref = *ref\n \t\t},\n \t};\n-\tint err = 0;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint err;\n \n-\tif (!ref->refname)\n-\t\treturn REFTABLE_API_ERROR;\n-\tif (ref->update_index < w->min_update_index ||\n+\tif (!ref->refname ||\n+\t ref->update_index < w->min_update_index ||\n \t ref->update_index > w->max_update_index)\n \t\treturn REFTABLE_API_ERROR;\n \n@@ -296,24 +303,32 @@ int reftable_writer_add_ref(struct reftable_writer *w,\n \n \terr = writer_add_record(w, &rec);\n \tif (err < 0)\n-\t\treturn err;\n+\t\tgoto out;\n \n \tif (!w->opts.skip_index_objects && reftable_ref_record_val1(ref)) {\n-\t\tstruct strbuf h = STRBUF_INIT;\n-\t\tstrbuf_add(&h, (char *)reftable_ref_record_val1(ref),\n+\t\tstrbuf_add(&buf, (char *)reftable_ref_record_val1(ref),\n \t\t\t hash_size(w->opts.hash_id));\n-\t\twriter_index_hash(w, &h);\n-\t\tstrbuf_release(&h);\n+\n+\t\terr = writer_index_hash(w, &buf);\n+\t\tif (err < 0)\n+\t\t\tgoto out;\n \t}\n \n \tif (!w->opts.skip_index_objects && reftable_ref_record_val2(ref)) {\n-\t\tstruct strbuf h = STRBUF_INIT;\n-\t\tstrbuf_add(&h, reftable_ref_record_val2(ref),\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_add(&buf, reftable_ref_record_val2(ref),\n \t\t\t hash_size(w->opts.hash_id));\n-\t\twriter_index_hash(w, &h);\n-\t\tstrbuf_release(&h);\n+\n+\t\terr = writer_index_hash(w, &buf);\n+\t\tif (err < 0)\n+\t\t\tgoto out;\n \t}\n-\treturn 0;\n+\n+\terr = 0;\n+\n+out:\n+\tstrbuf_release(&buf);\n+\treturn err;\n }\n \n int reftable_writer_add_refs(struct reftable_writer *w,\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503812","messageId":"0ed99e0bdf19fa1e396312b79acf47e9ba8e6ae5.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 09/25] reftable/writer: handle allocation failures in `reftable_new_writer()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:42:09Z","receivedAt":"2024-10-01T09:42:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_new_writer()`. Adapt the\nfunction to return an error code to return such failures. While at it,\nrename it to match our code style as we have to touch up every callsite\nanyway.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/reftable-writer.h | 12 +++++++-----\n reftable/stack.c | 14 ++++++++++----\n reftable/writer.c | 22 ++++++++++++++++------\n t/unit-tests/lib-reftable.c | 8 +++++---\n 4 files changed, 38 insertions(+), 18 deletions(-)\n\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 189b1f4144..43623dc7c3 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -90,11 +90,13 @@ struct reftable_stats {\n \tint object_id_len;\n };\n \n-/* reftable_new_writer creates a new writer */\n-struct reftable_writer *\n-reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n-\t\t int (*flush_func)(void *),\n-\t\t void *writer_arg, const struct reftable_write_options *opts);\n+struct reftable_writer;\n+\n+/* Create a new writer. */\n+int reftable_writer_new(struct reftable_writer **out,\n+\t\t\tssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t\tint (*flush_func)(void *),\n+\t\t\tvoid *writer_arg, const struct reftable_write_options *opts);\n \n /* Set the range of update indices for the records we will add. When writing a\n table into a stack, the min should be at least\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 498fae846d..ea21ca6e5f 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -808,8 +808,11 @@ int reftable_addition_add(struct reftable_addition *add,\n \t}\n \ttab_fd = get_tempfile_fd(tab_file);\n \n-\twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, &tab_fd,\n-\t\t\t\t &add->stack->opts);\n+\terr = reftable_writer_new(&wr, reftable_fd_write, reftable_fd_flush,\n+\t\t\t\t &tab_fd, &add->stack->opts);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = write_table(wr, arg);\n \tif (err < 0)\n \t\tgoto done;\n@@ -898,8 +901,11 @@ static int stack_compact_locked(struct reftable_stack *st,\n \t\tgoto done;\n \t}\n \n-\twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush,\n-\t\t\t\t &tab_fd, &st->opts);\n+\terr = reftable_writer_new(&wr, reftable_fd_write, reftable_fd_flush,\n+\t\t\t\t &tab_fd, &st->opts);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = stack_write_compact(st, wr, first, last, config);\n \tif (err < 0)\n \t\tgoto done;\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex ed61aaf59c..8ab2e916d3 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -117,13 +117,17 @@ static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n \tw->block_writer->restart_interval = w->opts.restart_interval;\n }\n \n-struct reftable_writer *\n-reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n-\t\t int (*flush_func)(void *),\n-\t\t void *writer_arg, const struct reftable_write_options *_opts)\n+int reftable_writer_new(struct reftable_writer **out,\n+\t\t\tssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t\tint (*flush_func)(void *),\n+\t\t\tvoid *writer_arg, const struct reftable_write_options *_opts)\n {\n-\tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n \tstruct reftable_write_options opts = {0};\n+\tstruct reftable_writer *wp;\n+\n+\twp = reftable_calloc(1, sizeof(*wp));\n+\tif (!wp)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \n \tif (_opts)\n \t\topts = *_opts;\n@@ -134,13 +138,19 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \tstrbuf_init(&wp->block_writer_data.last_key, 0);\n \tstrbuf_init(&wp->last_key, 0);\n \tREFTABLE_CALLOC_ARRAY(wp->block, opts.block_size);\n+\tif (!wp->block) {\n+\t\treftable_free(wp);\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t}\n \twp->write = writer_func;\n \twp->write_arg = writer_arg;\n \twp->opts = opts;\n \twp->flush = flush_func;\n \twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n \n-\treturn wp;\n+\t*out = wp;\n+\n+\treturn 0;\n }\n \n void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\ndiff --git a/t/unit-tests/lib-reftable.c b/t/unit-tests/lib-reftable.c\nindex ab1fa44a28..54c26c43e7 100644\n--- a/t/unit-tests/lib-reftable.c\n+++ b/t/unit-tests/lib-reftable.c\n@@ -22,9 +22,11 @@ static int strbuf_writer_flush(void *arg UNUSED)\n struct reftable_writer *t_reftable_strbuf_writer(struct strbuf *buf,\n \t\t\t\t\t\t struct reftable_write_options *opts)\n {\n-\treturn reftable_new_writer(&strbuf_writer_write,\n-\t\t\t\t &strbuf_writer_flush,\n-\t\t\t\t buf, opts);\n+\tstruct reftable_writer *writer;\n+\tint ret = reftable_writer_new(&writer, &strbuf_writer_write, &strbuf_writer_flush,\n+\t\t\t\t buf, opts);\n+\tcheck(!ret);\n+\treturn writer;\n }\n \n void t_reftable_write_to_buf(struct strbuf *buf,\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503813","messageId":"8dfbfd9286c730fc63d40c6d095876fb1317cdcf.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 10/25] reftable/merged: handle allocation failures in `merged_table_init_iter()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:42:12Z","receivedAt":"2024-10-01T09:42:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `merged_table_init_iter()`. While at it,\nmerge `merged_iter_init()` into the function. It only has a single\ncaller and merging them makes it easier to handle allocation failures\nconsistently.\n\nThis change also requires us to adapt `reftable_stack_init_*_iterator()`\nto bubble up the new error codes of `merged_table_iter_init()`. Adapt\ncallsites accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 39 +++++++++++++----\n reftable/merged.c | 74 ++++++++++++++++++++------------\n reftable/merged.h | 6 +--\n reftable/reftable-merged.h | 8 ++--\n reftable/reftable-stack.h | 8 ++--\n reftable/stack.c | 34 ++++++++++-----\n t/helper/test-reftable.c | 10 ++++-\n t/unit-tests/t-reftable-merged.c | 12 ++++--\n t/unit-tests/t-reftable-stack.c | 4 +-\n 9 files changed, 131 insertions(+), 64 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 5c241097a4..9c08c46c85 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1299,7 +1299,9 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t\t\tstruct reftable_log_record log = {0};\n \t\t\tstruct reftable_iterator it = {0};\n \n-\t\t\treftable_stack_init_log_iterator(arg->stack, &it);\n+\t\t\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto done;\n \n \t\t\t/*\n \t\t\t * When deleting refs we also delete all reflog entries\n@@ -1669,7 +1671,10 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \t * copy over all log entries from the old reflog. Last but not least,\n \t * when renaming we also have to delete all the old reflog entries.\n \t */\n-\treftable_stack_init_log_iterator(arg->stack, &it);\n+\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, arg->oldname);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1890,7 +1895,10 @@ static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftabl\n \tif (ret < 0)\n \t\tgoto done;\n \n-\treftable_stack_init_log_iterator(stack, &iter->iter);\n+\tret = reftable_stack_init_log_iterator(stack, &iter->iter);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&iter->iter, \"\");\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1957,7 +1965,10 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, refname);\n \twhile (!ret) {\n \t\tret = reftable_iterator_next_log(&it, &log);\n@@ -1973,6 +1984,7 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n \t\t\tbreak;\n \t}\n \n+done:\n \treftable_log_record_release(&log);\n \treftable_iterator_destroy(&it);\n \treturn ret;\n@@ -1994,7 +2006,10 @@ static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store,\n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, refname);\n \twhile (!ret) {\n \t\tstruct reftable_log_record log = {0};\n@@ -2044,7 +2059,10 @@ static int reftable_be_reflog_exists(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, refname);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -2150,7 +2168,9 @@ static int write_reflog_delete_table(struct reftable_writer *writer, void *cb_da\n \n \treftable_writer_set_limits(writer, ts, ts);\n \n-\treftable_stack_init_log_iterator(arg->stack, &it);\n+\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\tif (ret < 0)\n+\t\tgoto out;\n \n \t/*\n \t * In order to delete a table we need to delete all reflog entries one\n@@ -2174,6 +2194,7 @@ static int write_reflog_delete_table(struct reftable_writer *writer, void *cb_da\n \t\tret = reftable_writer_add_log(writer, &tombstone);\n \t}\n \n+out:\n \treftable_log_record_release(&log);\n \treftable_iterator_destroy(&it);\n \treturn ret;\n@@ -2312,7 +2333,9 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n \n \tret = reftable_iterator_seek_log(&it, refname);\n \tif (ret < 0)\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 128a810c55..de4f81abaf 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -30,22 +30,6 @@ struct merged_iter {\n \tssize_t advance_index;\n };\n \n-static void merged_iter_init(struct merged_iter *mi,\n-\t\t\t struct reftable_merged_table *mt,\n-\t\t\t uint8_t typ)\n-{\n-\tmemset(mi, 0, sizeof(*mi));\n-\tmi->advance_index = -1;\n-\tmi->suppress_deletions = mt->suppress_deletions;\n-\n-\tREFTABLE_CALLOC_ARRAY(mi->subiters, mt->readers_len);\n-\tfor (size_t i = 0; i < mt->readers_len; i++) {\n-\t\treftable_record_init(&mi->subiters[i].rec, typ);\n-\t\treader_init_iter(mt->readers[i], &mi->subiters[i].iter, typ);\n-\t}\n-\tmi->subiters_len = mt->readers_len;\n-}\n-\n static void merged_iter_close(void *p)\n {\n \tstruct merged_iter *mi = p;\n@@ -244,25 +228,61 @@ reftable_merged_table_min_update_index(struct reftable_merged_table *mt)\n \treturn mt->min;\n }\n \n-void merged_table_init_iter(struct reftable_merged_table *mt,\n-\t\t\t struct reftable_iterator *it,\n-\t\t\t uint8_t typ)\n+int merged_table_init_iter(struct reftable_merged_table *mt,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t uint8_t typ)\n {\n-\tstruct merged_iter *mi = reftable_malloc(sizeof(*mi));\n-\tmerged_iter_init(mi, mt, typ);\n+\tstruct merged_subiter *subiters;\n+\tstruct merged_iter *mi = NULL;\n+\tint ret;\n+\n+\tREFTABLE_CALLOC_ARRAY(subiters, mt->readers_len);\n+\tif (!subiters) {\n+\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n+\tfor (size_t i = 0; i < mt->readers_len; i++) {\n+\t\treftable_record_init(&subiters[i].rec, typ);\n+\t\treader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n+\t}\n+\n+\tREFTABLE_CALLOC_ARRAY(mi, 1);\n+\tif (!mi) {\n+\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\tmi->advance_index = -1;\n+\tmi->suppress_deletions = mt->suppress_deletions;\n+\tmi->subiters = subiters;\n+\tmi->subiters_len = mt->readers_len;\n+\n \titerator_from_merged_iter(it, mi);\n+\tret = 0;\n+\n+out:\n+\tif (ret < 0) {\n+\t\tfor (size_t i = 0; subiters && i < mt->readers_len; i++) {\n+\t\t\treftable_iterator_destroy(&subiters[i].iter);\n+\t\t\treftable_record_release(&subiters[i].rec);\n+\t\t}\n+\t\treftable_free(subiters);\n+\t\treftable_free(mi);\n+\t}\n+\n+\treturn ret;\n }\n \n-void reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it)\n+int reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(mt, it, BLOCK_TYPE_REF);\n+\treturn merged_table_init_iter(mt, it, BLOCK_TYPE_REF);\n }\n \n-void reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it)\n+int reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(mt, it, BLOCK_TYPE_LOG);\n+\treturn merged_table_init_iter(mt, it, BLOCK_TYPE_LOG);\n }\n \n uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\ndiff --git a/reftable/merged.h b/reftable/merged.h\nindex de5fd33f01..89bd0c4b35 100644\n--- a/reftable/merged.h\n+++ b/reftable/merged.h\n@@ -26,8 +26,8 @@ struct reftable_merged_table {\n \n struct reftable_iterator;\n \n-void merged_table_init_iter(struct reftable_merged_table *mt,\n-\t\t\t struct reftable_iterator *it,\n-\t\t\t uint8_t typ);\n+int merged_table_init_iter(struct reftable_merged_table *mt,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t uint8_t typ);\n \n #endif\ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nindex 16d19f8df2..a970d5dd89 100644\n--- a/reftable/reftable-merged.h\n+++ b/reftable/reftable-merged.h\n@@ -37,12 +37,12 @@ int reftable_merged_table_new(struct reftable_merged_table **dest,\n \t\t\t uint32_t hash_id);\n \n /* Initialize a merged table iterator for reading refs. */\n-void reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it);\n+int reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it);\n \n /* Initialize a merged table iterator for reading logs. */\n-void reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it);\n+int reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it);\n \n /* returns the max update_index covered by this merged table. */\n uint64_t\ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nindex f4f8cabc7f..e958f911b4 100644\n--- a/reftable/reftable-stack.h\n+++ b/reftable/reftable-stack.h\n@@ -73,16 +73,16 @@ struct reftable_iterator;\n * be used to iterate through refs. The iterator is valid until the next reload\n * or write.\n */\n-void reftable_stack_init_ref_iterator(struct reftable_stack *st,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_stack_init_ref_iterator(struct reftable_stack *st,\n+\t\t\t\t struct reftable_iterator *it);\n \n /*\n * Initialize an iterator for the merged tables contained in the stack that can\n * be used to iterate through logs. The iterator is valid until the next reload\n * or write.\n */\n-void reftable_stack_init_log_iterator(struct reftable_stack *st,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_stack_init_log_iterator(struct reftable_stack *st,\n+\t\t\t\t struct reftable_iterator *it);\n \n /* returns the merged_table for seeking. This table is valid until the\n * next write or reload, and should not be closed or deleted.\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex ea21ca6e5f..bb4d230918 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -136,18 +136,18 @@ int read_lines(const char *filename, char ***namesp)\n \treturn err;\n }\n \n-void reftable_stack_init_ref_iterator(struct reftable_stack *st,\n+int reftable_stack_init_ref_iterator(struct reftable_stack *st,\n \t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(reftable_stack_merged_table(st),\n-\t\t\t it, BLOCK_TYPE_REF);\n+\treturn merged_table_init_iter(reftable_stack_merged_table(st),\n+\t\t\t\t it, BLOCK_TYPE_REF);\n }\n \n-void reftable_stack_init_log_iterator(struct reftable_stack *st,\n-\t\t\t\t struct reftable_iterator *it)\n+int reftable_stack_init_log_iterator(struct reftable_stack *st,\n+\t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(reftable_stack_merged_table(st),\n-\t\t\t it, BLOCK_TYPE_LOG);\n+\treturn merged_table_init_iter(reftable_stack_merged_table(st),\n+\t\t\t\t it, BLOCK_TYPE_LOG);\n }\n \n struct reftable_merged_table *\n@@ -952,7 +952,10 @@ static int stack_write_compact(struct reftable_stack *st,\n \tif (err < 0)\n \t\tgoto done;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tif (err < 0)\n \t\tgoto done;\n@@ -977,7 +980,10 @@ static int stack_write_compact(struct reftable_stack *st,\n \t}\n \treftable_iterator_destroy(&it);\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tif (err < 0)\n \t\tgoto done;\n@@ -1496,7 +1502,10 @@ int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n \tstruct reftable_iterator it = { 0 };\n \tint ret;\n \n-\treftable_merged_table_init_ref_iterator(st->merged, &it);\n+\tret = reftable_merged_table_init_ref_iterator(st->merged, &it);\n+\tif (ret)\n+\t\tgoto out;\n+\n \tret = reftable_iterator_seek_ref(&it, refname);\n \tif (ret)\n \t\tgoto out;\n@@ -1523,7 +1532,10 @@ int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n \tstruct reftable_iterator it = {0};\n \tint err;\n \n-\treftable_stack_init_log_iterator(st, &it);\n+\terr = reftable_stack_init_log_iterator(st, &it);\n+\tif (err)\n+\t\tgoto done;\n+\n \terr = reftable_iterator_seek_log(&it, refname);\n \tif (err)\n \t\tgoto done;\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 29d4e9a755..5c8849d115 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -28,7 +28,10 @@ static int dump_table(struct reftable_merged_table *mt)\n \tconst struct git_hash_algo *algop;\n \tint err;\n \n-\treftable_merged_table_init_ref_iterator(mt, &it);\n+\terr = reftable_merged_table_init_ref_iterator(mt, &it);\n+\tif (err < 0)\n+\t\treturn err;\n+\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tif (err < 0)\n \t\treturn err;\n@@ -63,7 +66,10 @@ static int dump_table(struct reftable_merged_table *mt)\n \treftable_iterator_destroy(&it);\n \treftable_ref_record_release(&ref);\n \n-\treftable_merged_table_init_log_iterator(mt, &it);\n+\terr = reftable_merged_table_init_log_iterator(mt, &it);\n+\tif (err < 0)\n+\t\treturn err;\n+\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tif (err < 0)\n \t\treturn err;\ndiff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c\nindex 19e54bdfb8..3d2848632d 100644\n--- a/t/unit-tests/t-reftable-merged.c\n+++ b/t/unit-tests/t-reftable-merged.c\n@@ -82,7 +82,8 @@ static void t_merged_single_record(void)\n \tstruct reftable_iterator it = { 0 };\n \tint err;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"a\");\n \tcheck(!err);\n \n@@ -161,7 +162,8 @@ static void t_merged_refs(void)\n \tsize_t cap = 0;\n \tsize_t i;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"a\");\n \tcheck(!err);\n \tcheck_int(reftable_merged_table_hash_id(mt), ==, GIT_SHA1_FORMAT_ID);\n@@ -367,7 +369,8 @@ static void t_merged_logs(void)\n \tsize_t cap = 0;\n \tsize_t i;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"a\");\n \tcheck(!err);\n \tcheck_int(reftable_merged_table_hash_id(mt), ==, GIT_SHA1_FORMAT_ID);\n@@ -390,7 +393,8 @@ static void t_merged_logs(void)\n \t\tcheck(reftable_log_record_equal(want[i], &out[i],\n \t\t\t\t\t\t GIT_SHA1_RAWSZ));\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log_at(&it, \"a\", 2);\n \tcheck(!err);\n \treftable_log_record_release(&out[0]);\ndiff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c\nindex 65e513d5ec..6e41a4cf7e 100644\n--- a/t/unit-tests/t-reftable-stack.c\n+++ b/t/unit-tests/t-reftable-stack.c\n@@ -599,7 +599,9 @@ static void t_reftable_stack_iterator(void)\n \n \treftable_iterator_destroy(&it);\n \n-\treftable_stack_init_log_iterator(st, &it);\n+\terr = reftable_stack_init_log_iterator(st, &it);\n+\tcheck(!err);\n+\n \treftable_iterator_seek_log(&it, logs[0].refname);\n \tfor (i = 0; ; i++) {\n \t\tstruct reftable_log_record log = { 0 };\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503814","messageId":"7b592a6c6b8b08a7e8089364abfc6d2d6f6e83e4.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 11/25] reftable/reader: handle allocation failures for unindexed reader","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:42:18Z","receivedAt":"2024-10-01T09:42:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when creating unindexed readers.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/reader.c | 23 +++++++++++++++++------\n 1 file changed, 17 insertions(+), 6 deletions(-)\n\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 6494ce2e32..485ee085da 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -734,19 +734,30 @@ static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n \tREFTABLE_ALLOC_ARRAY(ti, 1);\n \ttable_iter_init(ti, r);\n \terr = table_iter_seek_start(ti, BLOCK_TYPE_REF, 0);\n-\tif (err < 0) {\n-\t\treftable_free(ti);\n-\t\treturn err;\n-\t}\n+\tif (err < 0)\n+\t\tgoto out;\n \n-\tfilter = reftable_malloc(sizeof(struct filtering_ref_iterator));\n+\tfilter = reftable_malloc(sizeof(*filter));\n+\tif (!filter) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n \t*filter = empty;\n \n \tstrbuf_add(&filter->oid, oid, oid_len);\n \titerator_from_table_iter(&filter->it, ti);\n \n \titerator_from_filtering_ref_iterator(it, filter);\n-\treturn 0;\n+\n+\terr = 0;\n+\n+out:\n+\tif (err < 0) {\n+\t\tif (ti)\n+\t\t\ttable_iter_close(ti);\n+\t\treftable_free(ti);\n+\t}\n+\treturn err;\n }\n \n int reftable_reader_refs_for(struct reftable_reader *r,\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503815","messageId":"ceed83826537cca26cc04d237efb298343dcde14.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 12/25] reftable/reader: handle allocation failures in `reader_init_iter()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:42:20Z","receivedAt":"2024-10-01T09:42:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reader_init_iter()`. This requires us to\nalso adapt `reftable_reader_init_*_iterator()` to bubble up the new\nerror codes. Adapt callers accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 4 +++-\n reftable/reader.c | 28 +++++++++++++++---------\n reftable/reader.h | 6 ++---\n reftable/reftable-reader.h | 8 +++----\n t/unit-tests/t-reftable-readwrite.c | 34 +++++++++++++++++++----------\n 5 files changed, 50 insertions(+), 30 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex de4f81abaf..69790c345c 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -244,7 +244,9 @@ int merged_table_init_iter(struct reftable_merged_table *mt,\n \n \tfor (size_t i = 0; i < mt->readers_len; i++) {\n \t\treftable_record_init(&subiters[i].rec, typ);\n-\t\treader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n+\t\tret = reader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n+\t\tif (ret < 0)\n+\t\t\tgoto out;\n \t}\n \n \tREFTABLE_CALLOC_ARRAY(mi, 1);\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 485ee085da..f696e992df 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -554,32 +554,37 @@ static void iterator_from_table_iter(struct reftable_iterator *it,\n \tit->ops = &table_iter_vtable;\n }\n \n-void reader_init_iter(struct reftable_reader *r,\n-\t\t struct reftable_iterator *it,\n-\t\t uint8_t typ)\n+int reader_init_iter(struct reftable_reader *r,\n+\t\t struct reftable_iterator *it,\n+\t\t uint8_t typ)\n {\n \tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n \n \tif (offs->is_present) {\n \t\tstruct table_iter *ti;\n \t\tREFTABLE_ALLOC_ARRAY(ti, 1);\n+\t\tif (!ti)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \t\ttable_iter_init(ti, r);\n \t\titerator_from_table_iter(it, ti);\n \t} else {\n \t\titerator_set_empty(it);\n \t}\n+\n+\treturn 0;\n }\n \n-void reftable_reader_init_ref_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it)\n+int reftable_reader_init_ref_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it)\n {\n-\treader_init_iter(r, it, BLOCK_TYPE_REF);\n+\treturn reader_init_iter(r, it, BLOCK_TYPE_REF);\n }\n \n-void reftable_reader_init_log_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it)\n+int reftable_reader_init_log_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it)\n {\n-\treader_init_iter(r, it, BLOCK_TYPE_LOG);\n+\treturn reader_init_iter(r, it, BLOCK_TYPE_LOG);\n }\n \n int reftable_reader_new(struct reftable_reader **out,\n@@ -689,7 +694,10 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n \tstruct indexed_table_ref_iter *itr = NULL;\n \n \t/* Look through the reverse index. */\n-\treader_init_iter(r, &oit, BLOCK_TYPE_OBJ);\n+\terr = reader_init_iter(r, &oit, BLOCK_TYPE_OBJ);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = iterator_seek(&oit, &want);\n \tif (err != 0)\n \t\tgoto done;\ndiff --git a/reftable/reader.h b/reftable/reader.h\nindex 3710ee09b4..02d10c5d37 100644\n--- a/reftable/reader.h\n+++ b/reftable/reader.h\n@@ -56,9 +56,9 @@ struct reftable_reader {\n \n const char *reader_name(struct reftable_reader *r);\n \n-void reader_init_iter(struct reftable_reader *r,\n-\t\t struct reftable_iterator *it,\n-\t\t uint8_t typ);\n+int reader_init_iter(struct reftable_reader *r,\n+\t\t struct reftable_iterator *it,\n+\t\t uint8_t typ);\n \n /* initialize a block reader to read from `r` */\n int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nindex a600452b56..6a2d0b693f 100644\n--- a/reftable/reftable-reader.h\n+++ b/reftable/reftable-reader.h\n@@ -46,12 +46,12 @@ void reftable_reader_incref(struct reftable_reader *reader);\n void reftable_reader_decref(struct reftable_reader *reader);\n \n /* Initialize a reftable iterator for reading refs. */\n-void reftable_reader_init_ref_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_reader_init_ref_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it);\n \n /* Initialize a reftable iterator for reading logs. */\n-void reftable_reader_init_log_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_reader_init_log_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it);\n \n /* returns the hash ID used in this table. */\n uint32_t reftable_reader_hash_id(struct reftable_reader *r);\ndiff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c\nindex e1b235a5f1..acca927a2c 100644\n--- a/t/unit-tests/t-reftable-readwrite.c\n+++ b/t/unit-tests/t-reftable-readwrite.c\n@@ -206,7 +206,8 @@ static void t_log_write_read(void)\n \terr = reftable_reader_new(&reader, &source, \"file.log\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \n \terr = reftable_iterator_seek_ref(&it, names[N - 1]);\n \tcheck(!err);\n@@ -221,8 +222,8 @@ static void t_log_write_read(void)\n \treftable_iterator_destroy(&it);\n \treftable_ref_record_release(&ref);\n \n-\treftable_reader_init_log_iterator(reader, &it);\n-\n+\terr = reftable_reader_init_log_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tcheck(!err);\n \n@@ -296,7 +297,8 @@ static void t_log_zlib_corruption(void)\n \terr = reftable_reader_new(&reader, &source, \"file.log\");\n \tcheck(!err);\n \n-\treftable_reader_init_log_iterator(reader, &it);\n+\terr = reftable_reader_init_log_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"refname\");\n \tcheck_int(err, ==, REFTABLE_ZLIB_ERROR);\n \n@@ -325,7 +327,8 @@ static void t_table_read_write_sequential(void)\n \terr = reftable_reader_new(&reader, &source, \"file.ref\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tcheck(!err);\n \n@@ -376,7 +379,8 @@ static void t_table_read_api(void)\n \terr = reftable_reader_new(&reader, &source, \"file.ref\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, names[0]);\n \tcheck(!err);\n \n@@ -419,7 +423,8 @@ static void t_table_read_write_seek(int index, int hash_id)\n \t}\n \n \tfor (i = 1; i < N; i++) {\n-\t\treftable_reader_init_ref_iterator(reader, &it);\n+\t\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\t\tcheck(!err);\n \t\terr = reftable_iterator_seek_ref(&it, names[i]);\n \t\tcheck(!err);\n \t\terr = reftable_iterator_next_ref(&it, &ref);\n@@ -435,7 +440,8 @@ static void t_table_read_write_seek(int index, int hash_id)\n \tstrbuf_addstr(&pastLast, names[N - 1]);\n \tstrbuf_addstr(&pastLast, \"/\");\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, pastLast.buf);\n \tif (err == 0) {\n \t\tstruct reftable_ref_record ref = { 0 };\n@@ -534,7 +540,8 @@ static void t_table_refs_for(int indexed)\n \tif (!indexed)\n \t\treader->obj_offsets.is_present = 0;\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tcheck(!err);\n \treftable_iterator_destroy(&it);\n@@ -593,7 +600,8 @@ static void t_write_empty_table(void)\n \terr = reftable_reader_new(&rd, &source, \"filename\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(rd, &it);\n+\terr = reftable_reader_init_ref_iterator(rd, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tcheck(!err);\n \n@@ -802,7 +810,8 @@ static void t_write_multiple_indices(void)\n \t * Seeking the log uses the log index now. In case there is any\n \t * confusion regarding indices we would notice here.\n \t */\n-\treftable_reader_init_log_iterator(reader, &it);\n+\terr = reftable_reader_init_log_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tcheck(!err);\n \n@@ -858,7 +867,8 @@ static void t_write_multi_level_index(void)\n \t/*\n \t * Seeking the last ref should work as expected.\n \t */\n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"refs/heads/199\");\n \tcheck(!err);\n \n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503816","messageId":"1e997a576693a9ae1841b14ef7f2a02841694bfb.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 13/25] reftable/stack: handle allocation failures on reload","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:42:23Z","receivedAt":"2024-10-01T09:42:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_stack_reload_once()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 34 +++++++++++++++++++++++++---------\n 1 file changed, 25 insertions(+), 9 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex bb4d230918..060b2c1b90 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -214,13 +214,13 @@ void reftable_stack_destroy(struct reftable_stack *st)\n }\n \n static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n-\t\t\t\t\t\t int cur_len)\n+\t\t\t\t\t\t size_t cur_len)\n {\n \tstruct reftable_reader **cur = reftable_calloc(cur_len, sizeof(*cur));\n-\tint i = 0;\n-\tfor (i = 0; i < cur_len; i++) {\n+\tif (!cur)\n+\t\treturn NULL;\n+\tfor (size_t i = 0; i < cur_len; i++)\n \t\tcur[i] = st->readers[i];\n-\t}\n \treturn cur;\n }\n \n@@ -229,18 +229,30 @@ static int reftable_stack_reload_once(struct reftable_stack *st,\n \t\t\t\t int reuse_open)\n {\n \tsize_t cur_len = !st->merged ? 0 : st->merged->readers_len;\n-\tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n+\tstruct reftable_reader **cur;\n \tstruct reftable_reader **reused = NULL;\n-\tsize_t reused_len = 0, reused_alloc = 0;\n-\tsize_t names_len = names_length(names);\n-\tstruct reftable_reader **new_readers =\n-\t\treftable_calloc(names_len, sizeof(*new_readers));\n+\tstruct reftable_reader **new_readers;\n+\tsize_t reused_len = 0, reused_alloc = 0, names_len;\n \tsize_t new_readers_len = 0;\n \tstruct reftable_merged_table *new_merged = NULL;\n \tstruct strbuf table_path = STRBUF_INIT;\n \tint err = 0;\n \tsize_t i;\n \n+\tcur = stack_copy_readers(st, cur_len);\n+\tif (!cur) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tnames_len = names_length(names);\n+\n+\tnew_readers = reftable_calloc(names_len, sizeof(*new_readers));\n+\tif (!new_readers) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \twhile (*names) {\n \t\tstruct reftable_reader *rd = NULL;\n \t\tconst char *name = *names++;\n@@ -261,6 +273,10 @@ static int reftable_stack_reload_once(struct reftable_stack *st,\n \t\t\t\t * do by bumping their refcount.\n \t\t\t\t */\n \t\t\t\tREFTABLE_ALLOC_GROW(reused, reused_len + 1, reused_alloc);\n+\t\t\t\tif (!reused) {\n+\t\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n \t\t\t\treused[reused_len++] = rd;\n \t\t\t\treftable_reader_incref(rd);\n \t\t\t\tbreak;\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503817","messageId":"40d4d81378dbeffa0a916aca7d747ee9a045e045.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 14/25] reftable/stack: handle allocation failures in `reftable_new_stack()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:42:26Z","receivedAt":"2024-10-01T09:42:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_new_stack()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 34 ++++++++++++++++++++++++++--------\n 1 file changed, 26 insertions(+), 8 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 060b2c1b90..1b77c9d014 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -56,10 +56,16 @@ static int reftable_fd_flush(void *arg)\n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \t\t const struct reftable_write_options *_opts)\n {\n-\tstruct reftable_stack *p = reftable_calloc(1, sizeof(*p));\n \tstruct strbuf list_file_name = STRBUF_INIT;\n-\tstruct reftable_write_options opts = {0};\n-\tint err = 0;\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct reftable_stack *p;\n+\tint err;\n+\n+\tp = reftable_calloc(1, sizeof(*p));\n+\tif (!p) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n \n \tif (_opts)\n \t\topts = *_opts;\n@@ -74,15 +80,23 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \n \tp->list_file = strbuf_detach(&list_file_name, NULL);\n \tp->list_fd = -1;\n-\tp->reftable_dir = xstrdup(dir);\n \tp->opts = opts;\n+\tp->reftable_dir = reftable_strdup(dir);\n+\tif (!p->reftable_dir) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n \n \terr = reftable_stack_reload_maybe_reuse(p, 1);\n-\tif (err < 0) {\n+\tif (err < 0)\n+\t\tgoto out;\n+\n+\t*dest = p;\n+\terr = 0;\n+\n+out:\n+\tif (err < 0)\n \t\treftable_stack_destroy(p);\n-\t} else {\n-\t\t*dest = p;\n-\t}\n \treturn err;\n }\n \n@@ -171,6 +185,10 @@ void reftable_stack_destroy(struct reftable_stack *st)\n {\n \tchar **names = NULL;\n \tint err = 0;\n+\n+\tif (!st)\n+\t\treturn;\n+\n \tif (st->merged) {\n \t\treftable_merged_table_free(st->merged);\n \t\tst->merged = NULL;\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503818","messageId":"6aaae4baa6ec9bccf8f8e8a517dcebda995b65b2.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 15/25] reftable/stack: handle allocation failures in `stack_compact_range()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:42:29Z","receivedAt":"2024-10-01T09:42:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `stack_compact_range()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 19 +++++++++++++++++--\n 1 file changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 1b77c9d014..2e6dd513d7 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1131,6 +1131,11 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * from the point of view of the newer process.\n \t */\n \tREFTABLE_CALLOC_ARRAY(table_locks, last - first + 1);\n+\tif (!table_locks) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tfor (i = last + 1; i > first; i--) {\n \t\tstack_filename(&table_name, st, reader_name(st->readers[i - 1]));\n \n@@ -1312,8 +1317,18 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\t * thus have to allocate `readers_len + 1` many entries.\n \t\t */\n \t\tREFTABLE_CALLOC_ARRAY(names, st->merged->readers_len + 1);\n-\t\tfor (size_t i = 0; i < st->merged->readers_len; i++)\n-\t\t\tnames[i] = xstrdup(st->readers[i]->name);\n+\t\tif (!names) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tfor (size_t i = 0; i < st->merged->readers_len; i++) {\n+\t\t\tnames[i] = reftable_strdup(st->readers[i]->name);\n+\t\t\tif (!names[i]) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n \t\tfirst_to_replace = first;\n \t\tlast_to_replace = last;\n \t}\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503819","messageId":"103a59ef0a5318b2ecef7ba65b7bb0c049b6dc6c.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 16/25] reftable/stack: handle allocation failures in auto compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:42:31Z","receivedAt":"2024-10-01T09:42:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_stack_auto_compact()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 16 ++++++++++++----\n 1 file changed, 12 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 2e6dd513d7..990784d9d2 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1518,6 +1518,8 @@ static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n \tuint64_t *sizes;\n \n \tREFTABLE_CALLOC_ARRAY(sizes, st->merged->readers_len);\n+\tif (!sizes)\n+\t\treturn NULL;\n \n \tfor (size_t i = 0; i < st->merged->readers_len; i++)\n \t\tsizes[i] = st->readers[i]->size - overhead;\n@@ -1527,11 +1529,17 @@ static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n \n int reftable_stack_auto_compact(struct reftable_stack *st)\n {\n-\tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n-\tstruct segment seg =\n-\t\tsuggest_compaction_segment(sizes, st->merged->readers_len,\n-\t\t\t\t\t st->opts.auto_compaction_factor);\n+\tstruct segment seg;\n+\tuint64_t *sizes;\n+\n+\tsizes = stack_table_sizes_for_compaction(st);\n+\tif (!sizes)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n+\tseg = suggest_compaction_segment(sizes, st->merged->readers_len,\n+\t\t\t\t\t st->opts.auto_compaction_factor);\n \treftable_free(sizes);\n+\n \tif (segment_size(&seg) > 0)\n \t\treturn stack_compact_range(st, seg.start, seg.end - 1,\n \t\t\t\t\t NULL, STACK_COMPACT_RANGE_BEST_EFFORT);\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503820","messageId":"6799d299fe6dffcc6323f09dda6cd6cd51996056.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 17/25] reftable/iter: handle allocation failures when creating indexed table iter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:42:36Z","receivedAt":"2024-10-01T09:42:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `new_indexed_table_ref_iter()`. While at\nit, rename the function to match our coding style.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/iter.c | 20 ++++++++++++++++----\n reftable/iter.h | 2 +-\n reftable/reader.c | 7 ++++++-\n 3 files changed, 23 insertions(+), 6 deletions(-)\n\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex 416a9f6996..d319538f80 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -181,14 +181,20 @@ static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\n \t}\n }\n \n-int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+int indexed_table_ref_iter_new(struct indexed_table_ref_iter **dest,\n \t\t\t struct reftable_reader *r, uint8_t *oid,\n \t\t\t int oid_len, uint64_t *offsets, int offset_len)\n {\n \tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n-\tstruct indexed_table_ref_iter *itr = reftable_calloc(1, sizeof(*itr));\n+\tstruct indexed_table_ref_iter *itr;\n \tint err = 0;\n \n+\titr = reftable_calloc(1, sizeof(*itr));\n+\tif (!itr) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n \t*itr = empty;\n \titr->r = r;\n \tstrbuf_add(&itr->oid, oid, oid_len);\n@@ -197,10 +203,16 @@ int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n \titr->offset_len = offset_len;\n \n \terr = indexed_table_ref_iter_next_block(itr);\n+\tif (err < 0)\n+\t\tgoto out;\n+\n+\t*dest = itr;\n+\terr = 0;\n+\n+out:\n \tif (err < 0) {\n+\t\t*dest = NULL;\n \t\treftable_free(itr);\n-\t} else {\n-\t\t*dest = itr;\n \t}\n \treturn err;\n }\ndiff --git a/reftable/iter.h b/reftable/iter.h\nindex befc4597df..b3225bc7ad 100644\n--- a/reftable/iter.h\n+++ b/reftable/iter.h\n@@ -82,7 +82,7 @@ void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n \t\t\t\t\t struct indexed_table_ref_iter *itr);\n \n /* Takes ownership of `offsets` */\n-int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+int indexed_table_ref_iter_new(struct indexed_table_ref_iter **dest,\n \t\t\t struct reftable_reader *r, uint8_t *oid,\n \t\t\t int oid_len, uint64_t *offsets, int offset_len);\n \ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex f696e992df..0179e4e73d 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -715,7 +715,7 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n \t\tgoto done;\n \t}\n \n-\terr = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),\n+\terr = indexed_table_ref_iter_new(&itr, r, oid, hash_size(r->hash_id),\n \t\t\t\t\t got.u.obj.offsets,\n \t\t\t\t\t got.u.obj.offset_len);\n \tif (err < 0)\n@@ -740,6 +740,11 @@ static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n \tint err;\n \n \tREFTABLE_ALLOC_ARRAY(ti, 1);\n+\tif (!ti) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n \ttable_iter_init(ti, r);\n \terr = table_iter_seek_start(ti, BLOCK_TYPE_REF, 0);\n \tif (err < 0)\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503821","messageId":"c7e54d71d758e92ceafefbb8cd272920616c2453.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 18/25] reftable/blocksource: handle allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:42:39Z","receivedAt":"2024-10-01T09:42:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in the blocksource code.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/blocksource.c | 25 ++++++++++++++++++++-----\n 1 file changed, 20 insertions(+), 5 deletions(-)\n\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex e93cac9bb6..a2a6a196d5 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -30,6 +30,8 @@ static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n \tstruct strbuf *b = v;\n \tassert(off + size <= b->len);\n \tREFTABLE_CALLOC_ARRAY(dest->data, size);\n+\tif (!dest->data)\n+\t\treturn -1;\n \tmemcpy(dest->data, b->buf + off, size);\n \tdest->len = size;\n \treturn size;\n@@ -98,27 +100,40 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,\n {\n \tstruct file_block_source *p;\n \tstruct stat st;\n-\tint fd;\n+\tint fd, err;\n \n \tfd = open(name, O_RDONLY);\n \tif (fd < 0) {\n \t\tif (errno == ENOENT)\n \t\t\treturn REFTABLE_NOT_EXIST_ERROR;\n-\t\treturn -1;\n+\t\terr = -1;\n+\t\tgoto out;\n \t}\n \n \tif (fstat(fd, &st) < 0) {\n-\t\tclose(fd);\n-\t\treturn REFTABLE_IO_ERROR;\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto out;\n \t}\n \n \tREFTABLE_CALLOC_ARRAY(p, 1);\n+\tif (!p) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n \tp->size = st.st_size;\n \tp->data = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n-\tclose(fd);\n \n \tassert(!bs->ops);\n \tbs->ops = &file_vtable;\n \tbs->arg = p;\n+\n+\terr = 0;\n+\n+out:\n+\tif (fd >= 0)\n+\t\tclose(fd);\n+\tif (err < 0)\n+\t\treftable_free(p);\n \treturn 0;\n }\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503822","messageId":"92d39b9021afc2dfcfa3d236c7c86286a7a4e101.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 19/25] reftable/block: handle allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:42:42Z","receivedAt":"2024-10-01T09:42:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `block_writer_init()` and\n`block_reader_init()`. This requires us to bubble up error codes into\n`writer_reinit_block_writer()`. Adapt call sites accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/block.c | 23 +++++++++++++--\n reftable/block.h | 4 +--\n reftable/writer.c | 51 ++++++++++++++++++++++-----------\n t/unit-tests/t-reftable-block.c | 20 +++++++------\n 4 files changed, 69 insertions(+), 29 deletions(-)\n\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 00030eee06..bfa7dc61bf 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -52,6 +52,8 @@ static int block_writer_register_restart(struct block_writer *w, int n,\n \t\treturn -1;\n \tif (is_restart) {\n \t\tREFTABLE_ALLOC_GROW(w->restarts, w->restart_len + 1, w->restart_cap);\n+\t\tif (!w->restarts)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \t\tw->restarts[w->restart_len++] = w->next;\n \t}\n \n@@ -63,8 +65,8 @@ static int block_writer_register_restart(struct block_writer *w, int n,\n \treturn 0;\n }\n \n-void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n-\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n+int block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n {\n \tbw->buf = buf;\n \tbw->hash_size = hash_size;\n@@ -78,8 +80,12 @@ void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n \tbw->last_key.len = 0;\n \tif (!bw->zstream) {\n \t\tREFTABLE_CALLOC_ARRAY(bw->zstream, 1);\n+\t\tif (!bw->zstream)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \t\tdeflateInit(bw->zstream, 9);\n \t}\n+\n+\treturn 0;\n }\n \n uint8_t block_writer_type(struct block_writer *bw)\n@@ -163,6 +169,10 @@ int block_writer_finish(struct block_writer *w)\n \t\t */\n \t\tcompressed_len = deflateBound(w->zstream, src_len);\n \t\tREFTABLE_ALLOC_GROW(w->compressed, compressed_len, w->compressed_cap);\n+\t\tif (!w->compressed) {\n+\t\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\treturn ret;\n+\t\t}\n \n \t\tw->zstream->next_out = w->compressed;\n \t\tw->zstream->avail_out = compressed_len;\n@@ -219,12 +229,21 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,\n \t\t/* Log blocks specify the *uncompressed* size in their header. */\n \t\tREFTABLE_ALLOC_GROW(br->uncompressed_data, sz,\n \t\t\t\t br->uncompressed_cap);\n+\t\tif (!br->uncompressed_data) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n \n \t\t/* Copy over the block header verbatim. It's not compressed. */\n \t\tmemcpy(br->uncompressed_data, block->data, block_header_skip);\n \n \t\tif (!br->zstream) {\n \t\t\tREFTABLE_CALLOC_ARRAY(br->zstream, 1);\n+\t\t\tif (!br->zstream) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\n \t\t\terr = inflateInit(br->zstream);\n \t\t} else {\n \t\t\terr = inflateReset(br->zstream);\ndiff --git a/reftable/block.h b/reftable/block.h\nindex 1c8f25ee6e..18d7ea0337 100644\n--- a/reftable/block.h\n+++ b/reftable/block.h\n@@ -45,8 +45,8 @@ struct block_writer {\n /*\n * initializes the blockwriter to write `typ` entries, using `buf` as temporary\n * storage. `buf` is not owned by the block_writer. */\n-void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n-\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n+int block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n \n /* returns the block type (eg. 'r' for ref records. */\n uint8_t block_writer_type(struct block_writer *bw);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 8ab2e916d3..791e246337 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -102,19 +102,24 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \treturn header_size(writer_version(w));\n }\n \n-static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n+static int writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n {\n-\tint block_start = 0;\n-\tif (w->next == 0) {\n+\tint block_start = 0, ret;\n+\n+\tif (w->next == 0)\n \t\tblock_start = header_size(writer_version(w));\n-\t}\n \n \tstrbuf_reset(&w->last_key);\n-\tblock_writer_init(&w->block_writer_data, typ, w->block,\n-\t\t\t w->opts.block_size, block_start,\n-\t\t\t hash_size(w->opts.hash_id));\n+\tret = block_writer_init(&w->block_writer_data, typ, w->block,\n+\t\t\t\tw->opts.block_size, block_start,\n+\t\t\t\thash_size(w->opts.hash_id));\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n \tw->block_writer = &w->block_writer_data;\n \tw->block_writer->restart_interval = w->opts.restart_interval;\n+\n+\treturn 0;\n }\n \n int reftable_writer_new(struct reftable_writer **out,\n@@ -247,8 +252,11 @@ static int writer_add_record(struct reftable_writer *w,\n \n \tstrbuf_reset(&w->last_key);\n \tstrbuf_addbuf(&w->last_key, &key);\n-\tif (!w->block_writer)\n-\t\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\tif (!w->block_writer) {\n+\t\terr = writer_reinit_block_writer(w, reftable_record_type(rec));\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t}\n \n \tif (block_writer_type(w->block_writer) != reftable_record_type(rec))\n \t\tBUG(\"record of type %d added to writer of type %d\",\n@@ -271,7 +279,9 @@ static int writer_add_record(struct reftable_writer *w,\n \terr = writer_flush_block(w);\n \tif (err < 0)\n \t\tgoto done;\n-\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\terr = writer_reinit_block_writer(w, reftable_record_type(rec));\n+\tif (err < 0)\n+\t\tgoto done;\n \n \t/*\n \t * Try to add the record to the writer again. If this still fails then\n@@ -461,7 +471,9 @@ static int writer_finish_section(struct reftable_writer *w)\n \n \t\tmax_level++;\n \t\tindex_start = w->next;\n-\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\t\terr = writer_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n \n \t\tidx = w->index;\n \t\tidx_len = w->index_len;\n@@ -555,7 +567,10 @@ static void write_object_record(void *void_arg, void *key)\n \tif (arg->err < 0)\n \t\tgoto done;\n \n-\twriter_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\targ->err = writer_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n \targ->err = block_writer_add(arg->w->block_writer, &rec);\n \tif (arg->err == 0)\n \t\tgoto done;\n@@ -584,16 +599,18 @@ static int writer_dump_object_index(struct reftable_writer *w)\n \tstruct common_prefix_arg common = {\n \t\t.max = 1,\t\t/* obj_id_len should be >= 2. */\n \t};\n-\tif (w->obj_index_tree) {\n+\tint err;\n+\n+\tif (w->obj_index_tree)\n \t\tinfix_walk(w->obj_index_tree, &update_common, &common);\n-\t}\n \tw->stats.object_id_len = common.max + 1;\n \n-\twriter_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\terr = writer_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\tif (err < 0)\n+\t\treturn err;\n \n-\tif (w->obj_index_tree) {\n+\tif (w->obj_index_tree)\n \t\tinfix_walk(w->obj_index_tree, &write_object_record, &closure);\n-\t}\n \n \tif (closure.err < 0)\n \t\treturn closure.err;\ndiff --git a/t/unit-tests/t-reftable-block.c b/t/unit-tests/t-reftable-block.c\nindex f1a49485e2..e52a612e85 100644\n--- a/t/unit-tests/t-reftable-block.c\n+++ b/t/unit-tests/t-reftable-block.c\n@@ -34,8 +34,9 @@ static void t_ref_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \trec.u.ref.refname = (char *) \"\";\n \trec.u.ref.value_type = REFTABLE_REF_DELETION;\n@@ -126,8 +127,9 @@ static void t_log_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n \t\trec.u.log.refname = xstrfmt(\"branch%02\"PRIuMAX , (uintmax_t)i);\n@@ -214,8 +216,9 @@ static void t_obj_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n \t\tuint8_t bytes[] = { i, i + 1, i + 2, i + 3, i + 5 }, *allocated;\n@@ -296,8 +299,9 @@ static void t_index_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n \t\tstrbuf_init(&recs[i].u.idx.last_key, 9);\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503823","messageId":"3416004e0d73ec46be616ca2809bb40d4d10c44a.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 20/25] reftable/pq: handle allocation failures when adding entries","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:42:44Z","receivedAt":"2024-10-01T09:42:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when adding entries to the pqueue. Adapt its\nonly caller accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 5 ++++-\n reftable/pq.c | 7 ++++++-\n reftable/pq.h | 2 +-\n 3 files changed, 11 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 69790c345c..8e202a8efd 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -54,7 +54,10 @@ static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)\n \tif (err)\n \t\treturn err;\n \n-\tmerged_iter_pqueue_add(&mi->pq, &e);\n+\terr = merged_iter_pqueue_add(&mi->pq, &e);\n+\tif (err)\n+\t\treturn err;\n+\n \treturn 0;\n }\n \ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex 2b5b7d1c0e..03b9912282 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -8,6 +8,7 @@ license that can be found in the LICENSE file or at\n \n #include \"pq.h\"\n \n+#include \"reftable-error.h\"\n #include \"reftable-record.h\"\n #include \"system.h\"\n #include \"basics.h\"\n@@ -44,11 +45,13 @@ struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n \treturn e;\n }\n \n-void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e)\n+int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e)\n {\n \tsize_t i = 0;\n \n \tREFTABLE_ALLOC_GROW(pq->heap, pq->len + 1, pq->cap);\n+\tif (!pq->heap)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tpq->heap[pq->len++] = *e;\n \n \ti = pq->len - 1;\n@@ -59,6 +62,8 @@ void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry\n \t\tSWAP(pq->heap[j], pq->heap[i]);\n \t\ti = j;\n \t}\n+\n+\treturn 0;\n }\n \n void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)\ndiff --git a/reftable/pq.h b/reftable/pq.h\nindex 707bd26767..83c062eeca 100644\n--- a/reftable/pq.h\n+++ b/reftable/pq.h\n@@ -23,7 +23,7 @@ struct merged_iter_pqueue {\n };\n \n struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n-void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e);\n+int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e);\n void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n int pq_less(struct pq_entry *a, struct pq_entry *b);\n \n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503824","messageId":"a3639304884db6547f43461afe7aa9800ff24dc7.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 21/25] reftable/tree: handle allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:42:47Z","receivedAt":"2024-10-01T09:42:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The tree interfaces of the reftable library handle both insertion and\nsearching of tree nodes with a single function, where the behaviour is\naltered between the two via an `insert` bit. This makes it quit awkward\nto handle allocation failures because on inserting we'd have to check\nfor `NULL` pointers and return an error, whereas on searching entries we\ndon't have to handle it as an allocation error.\n\nSplit up concerns of this function into two separate functions, one for\ninserting entries and one for searching entries. This makes it easy for\nus to check for allocation errors as `tree_insert()` should never return\na `NULL` pointer now. Adapt callers accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/tree.c | 42 +++++++++++++++++++++++-----------\n reftable/tree.h | 21 +++++++++++++----\n reftable/writer.c | 7 +++---\n t/unit-tests/t-reftable-tree.c | 10 ++++----\n 4 files changed, 54 insertions(+), 26 deletions(-)\n\ndiff --git a/reftable/tree.c b/reftable/tree.c\nindex 5ffb2e0d69..f4dbe72090 100644\n--- a/reftable/tree.c\n+++ b/reftable/tree.c\n@@ -11,28 +11,44 @@ license that can be found in the LICENSE file or at\n \n #include \"basics.h\"\n \n-struct tree_node *tree_search(void *key, struct tree_node **rootp,\n-\t\t\t int (*compare)(const void *, const void *),\n-\t\t\t int insert)\n+struct tree_node *tree_search(struct tree_node *tree,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *))\n {\n \tint res;\n+\tif (!tree)\n+\t\treturn NULL;\n+\tres = compare(key, tree->key);\n+\tif (res < 0)\n+\t\treturn tree_search(tree->left, key, compare);\n+\telse if (res > 0)\n+\t\treturn tree_search(tree->right, key, compare);\n+\treturn tree;\n+}\n+\n+struct tree_node *tree_insert(struct tree_node **rootp,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *))\n+{\n+\tint res;\n+\n \tif (!*rootp) {\n-\t\tif (!insert) {\n+\t\tstruct tree_node *n;\n+\n+\t\tREFTABLE_CALLOC_ARRAY(n, 1);\n+\t\tif (!n)\n \t\t\treturn NULL;\n-\t\t} else {\n-\t\t\tstruct tree_node *n;\n-\t\t\tREFTABLE_CALLOC_ARRAY(n, 1);\n-\t\t\tn->key = key;\n-\t\t\t*rootp = n;\n-\t\t\treturn *rootp;\n-\t\t}\n+\n+\t\tn->key = key;\n+\t\t*rootp = n;\n+\t\treturn *rootp;\n \t}\n \n \tres = compare(key, (*rootp)->key);\n \tif (res < 0)\n-\t\treturn tree_search(key, &(*rootp)->left, compare, insert);\n+\t\treturn tree_insert(&(*rootp)->left, key, compare);\n \telse if (res > 0)\n-\t\treturn tree_search(key, &(*rootp)->right, compare, insert);\n+\t\treturn tree_insert(&(*rootp)->right, key, compare);\n \treturn *rootp;\n }\n \ndiff --git a/reftable/tree.h b/reftable/tree.h\nindex fbdd002e23..9604453b6d 100644\n--- a/reftable/tree.h\n+++ b/reftable/tree.h\n@@ -15,12 +15,23 @@ struct tree_node {\n \tstruct tree_node *left, *right;\n };\n \n-/* looks for `key` in `rootp` using `compare` as comparison function. If insert\n- * is set, insert the key if it's not found. Else, return NULL.\n+/*\n+ * Search the tree for the node matching the given key using `compare` as\n+ * comparison function. Returns the node whose key matches or `NULL` in case\n+ * the key does not exist in the tree.\n+ */\n+struct tree_node *tree_search(struct tree_node *tree,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *));\n+\n+/*\n+ * Insert a node into the tree. Returns the newly inserted node if the key does\n+ * not yet exist. Otherwise it returns the preexisting node. Returns `NULL`\n+ * when allocating the new node fails.\n */\n-struct tree_node *tree_search(void *key, struct tree_node **rootp,\n-\t\t\t int (*compare)(const void *, const void *),\n-\t\t\t int insert);\n+struct tree_node *tree_insert(struct tree_node **rootp,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *));\n \n /* performs an infix walk of the tree. */\n void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 791e246337..e180c10840 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -208,8 +208,7 @@ static int writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n \tstruct obj_index_tree_node *key;\n \tstruct tree_node *node;\n \n-\tnode = tree_search(&want, &w->obj_index_tree,\n-\t\t\t &obj_index_tree_node_compare, 0);\n+\tnode = tree_search(w->obj_index_tree, &want, &obj_index_tree_node_compare);\n \tif (!node) {\n \t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n \n@@ -221,8 +220,8 @@ static int writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n \n \t\tstrbuf_reset(&key->hash);\n \t\tstrbuf_addbuf(&key->hash, hash);\n-\t\ttree_search((void *)key, &w->obj_index_tree,\n-\t\t\t &obj_index_tree_node_compare, 1);\n+\t\ttree_insert(&w->obj_index_tree, key,\n+\t\t\t &obj_index_tree_node_compare);\n \t} else {\n \t\tkey = node->key;\n \t}\ndiff --git a/t/unit-tests/t-reftable-tree.c b/t/unit-tests/t-reftable-tree.c\nindex 700479d34b..79b175a45a 100644\n--- a/t/unit-tests/t-reftable-tree.c\n+++ b/t/unit-tests/t-reftable-tree.c\n@@ -37,16 +37,17 @@ static void t_tree_search(void)\n \t * values[1] and values[10] (inclusive) in the tree.\n \t */\n \tdo {\n-\t\tnodes[i] = tree_search(&values[i], &root, &t_compare, 1);\n+\t\tnodes[i] = tree_insert(&root, &values[i], &t_compare);\n+\t\tcheck(nodes[i] != NULL);\n \t\ti = (i * 7) % 11;\n \t} while (i != 1);\n \n \tfor (i = 1; i < ARRAY_SIZE(nodes); i++) {\n \t\tcheck_pointer_eq(&values[i], nodes[i]->key);\n-\t\tcheck_pointer_eq(nodes[i], tree_search(&values[i], &root, &t_compare, 0));\n+\t\tcheck_pointer_eq(nodes[i], tree_search(root, &values[i], &t_compare));\n \t}\n \n-\tcheck(!tree_search(values, &root, t_compare, 0));\n+\tcheck(!tree_search(root, values, t_compare));\n \ttree_free(root);\n }\n \n@@ -62,7 +63,8 @@ static void t_infix_walk(void)\n \tsize_t count = 0;\n \n \tdo {\n-\t\ttree_search(&values[i], &root, t_compare, 1);\n+\t\tstruct tree_node *node = tree_insert(&root, &values[i], t_compare);\n+\t\tcheck(node != NULL);\n \t\ti = (i * 7) % 11;\n \t\tcount++;\n \t} while (i != 1);\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503825","messageId":"28661500ff1710e866966b93c1d7c7d820f4c511.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 22/25] reftable: handle trivial allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:42:50Z","receivedAt":"2024-10-01T09:42:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle trivial allocation failures in the reftable library and its unit\ntests.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 3 +++\n reftable/reader.c | 10 +++++++++-\n reftable/stack.c | 20 ++++++++++++++++++++\n reftable/writer.c | 13 +++++++++++--\n t/unit-tests/t-reftable-block.c | 4 ++++\n t/unit-tests/t-reftable-merged.c | 4 ++++\n t/unit-tests/t-reftable-readwrite.c | 27 ++++++++++++++++-----------\n 7 files changed, 67 insertions(+), 14 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 8e202a8efd..514d6facf4 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -203,6 +203,9 @@ int reftable_merged_table_new(struct reftable_merged_table **dest,\n \t}\n \n \tREFTABLE_CALLOC_ARRAY(m, 1);\n+\tif (!m)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \tm->readers = readers;\n \tm->readers_len = n;\n \tm->min = first_min;\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 0179e4e73d..98e7aa2637 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -598,6 +598,10 @@ int reftable_reader_new(struct reftable_reader **out,\n \tint err;\n \n \tREFTABLE_CALLOC_ARRAY(r, 1);\n+\tif (!r) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \n \t/*\n \t * We need one extra byte to read the type of first block. We also\n@@ -627,7 +631,11 @@ int reftable_reader_new(struct reftable_reader **out,\n \n \tr->size = file_size - footer_size(r->version);\n \tr->source = *source;\n-\tr->name = xstrdup(name);\n+\tr->name = reftable_strdup(name);\n+\tif (!r->name) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \tr->hash_id = 0;\n \tr->refcount = 1;\n \ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 990784d9d2..7df28ab343 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -116,6 +116,11 @@ static int fd_read_lines(int fd, char ***namesp)\n \t}\n \n \tREFTABLE_ALLOC_ARRAY(buf, size + 1);\n+\tif (!buf) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tif (read_in_full(fd, buf, size) != size) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n@@ -140,6 +145,8 @@ int read_lines(const char *filename, char ***namesp)\n \tif (fd < 0) {\n \t\tif (errno == ENOENT) {\n \t\t\tREFTABLE_CALLOC_ARRAY(*namesp, 1);\n+\t\t\tif (!*namesp)\n+\t\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \t\t\treturn 0;\n \t\t}\n \n@@ -420,6 +427,10 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\t\t}\n \n \t\t\tREFTABLE_CALLOC_ARRAY(names, 1);\n+\t\t\tif (!names) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto out;\n+\t\t\t}\n \t\t} else {\n \t\t\terr = fd_read_lines(fd, &names);\n \t\t\tif (err < 0)\n@@ -779,7 +790,11 @@ int reftable_stack_new_addition(struct reftable_addition **dest,\n {\n \tint err = 0;\n \tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n+\n \tREFTABLE_CALLOC_ARRAY(*dest, 1);\n+\tif (!*dest)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \t**dest = empty;\n \terr = reftable_stack_init_addition(*dest, st);\n \tif (err) {\n@@ -886,7 +901,12 @@ int reftable_addition_add(struct reftable_addition *add,\n \n \tREFTABLE_ALLOC_GROW(add->new_tables, add->new_tables_len + 1,\n \t\t\t add->new_tables_cap);\n+\tif (!add->new_tables) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \tadd->new_tables[add->new_tables_len++] = strbuf_detach(&next_name, NULL);\n+\n done:\n \tdelete_tempfile(&tab_file);\n \tstrbuf_release(&temp_tab_file_name);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex e180c10840..550172e65c 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -49,8 +49,14 @@ static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n {\n \tint n = 0;\n \tif (w->pending_padding > 0) {\n-\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding, sizeof(*zeroed));\n-\t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n+\t\tuint8_t *zeroed;\n+\t\tint n;\n+\n+\t\tzeroed = reftable_calloc(w->pending_padding, sizeof(*zeroed));\n+\t\tif (!zeroed)\n+\t\t\treturn -1;\n+\n+\t\tn = w->write(w->write_arg, zeroed, w->pending_padding);\n \t\tif (n < 0)\n \t\t\treturn n;\n \n@@ -767,6 +773,9 @@ static int writer_flush_nonempty_block(struct reftable_writer *w)\n \t * case we will end up with a multi-level index.\n \t */\n \tREFTABLE_ALLOC_GROW(w->index, w->index_len + 1, w->index_cap);\n+\tif (!w->index)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \tindex_record.offset = w->next;\n \tstrbuf_reset(&index_record.last_key);\n \tstrbuf_addbuf(&index_record.last_key, &w->block_writer->last_key);\ndiff --git a/t/unit-tests/t-reftable-block.c b/t/unit-tests/t-reftable-block.c\nindex e52a612e85..d470060e8b 100644\n--- a/t/unit-tests/t-reftable-block.c\n+++ b/t/unit-tests/t-reftable-block.c\n@@ -32,6 +32,7 @@ static void t_ref_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n@@ -125,6 +126,7 @@ static void t_log_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size,\n@@ -214,6 +216,7 @@ static void t_obj_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size,\n@@ -297,6 +300,7 @@ static void t_index_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size,\ndiff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c\nindex 3d2848632d..3c84363e98 100644\n--- a/t/unit-tests/t-reftable-merged.c\n+++ b/t/unit-tests/t-reftable-merged.c\n@@ -29,7 +29,9 @@ merged_table_from_records(struct reftable_ref_record **refs,\n \tint err;\n \n \tREFTABLE_CALLOC_ARRAY(*readers, n);\n+\tcheck(*readers != NULL);\n \tREFTABLE_CALLOC_ARRAY(*source, n);\n+\tcheck(*source != NULL);\n \n \tfor (size_t i = 0; i < n; i++) {\n \t\tt_reftable_write_to_buf(&buf[i], refs[i], sizes[i], NULL, 0, &opts);\n@@ -285,7 +287,9 @@ merged_table_from_log_records(struct reftable_log_record **logs,\n \tint err;\n \n \tREFTABLE_CALLOC_ARRAY(*readers, n);\n+\tcheck(*readers != NULL);\n \tREFTABLE_CALLOC_ARRAY(*source, n);\n+\tcheck(*source != NULL);\n \n \tfor (size_t i = 0; i < n; i++) {\n \t\tt_reftable_write_to_buf(&buf[i], NULL, 0, logs[i], sizes[i], &opts);\ndiff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c\nindex acca927a2c..bfa069caff 100644\n--- a/t/unit-tests/t-reftable-readwrite.c\n+++ b/t/unit-tests/t-reftable-readwrite.c\n@@ -52,8 +52,11 @@ static void write_table(char ***names, struct strbuf *buf, int N,\n \tint i;\n \n \tREFTABLE_CALLOC_ARRAY(*names, N + 1);\n+\tcheck(*names != NULL);\n \tREFTABLE_CALLOC_ARRAY(refs, N);\n+\tcheck(refs != NULL);\n \tREFTABLE_CALLOC_ARRAY(logs, N);\n+\tcheck(logs != NULL);\n \n \tfor (i = 0; i < N; i++) {\n \t\trefs[i].refname = (*names)[i] = xstrfmt(\"refs/heads/branch%02d\", i);\n@@ -150,23 +153,25 @@ static void t_log_overflow(void)\n \n static void t_log_write_read(void)\n {\n-\tint N = 2;\n-\tchar **names = reftable_calloc(N + 1, sizeof(*names));\n-\tint err;\n \tstruct reftable_write_options opts = {\n \t\t.block_size = 256,\n \t};\n \tstruct reftable_ref_record ref = { 0 };\n-\tint i = 0;\n \tstruct reftable_log_record log = { 0 };\n-\tint n;\n \tstruct reftable_iterator it = { 0 };\n \tstruct reftable_reader *reader;\n \tstruct reftable_block_source source = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w = t_reftable_strbuf_writer(&buf, &opts);\n \tconst struct reftable_stats *stats = NULL;\n+\tint N = 2, err, i, n;\n+\tchar **names;\n+\n+\tnames = reftable_calloc(N + 1, sizeof(*names));\n+\tcheck(names != NULL);\n+\n \treftable_writer_set_limits(w, 0, N);\n+\n \tfor (i = 0; i < N; i++) {\n \t\tchar name[256];\n \t\tstruct reftable_ref_record ref = { 0 };\n@@ -178,6 +183,7 @@ static void t_log_write_read(void)\n \t\terr = reftable_writer_add_ref(w, &ref);\n \t\tcheck(!err);\n \t}\n+\n \tfor (i = 0; i < N; i++) {\n \t\tstruct reftable_log_record log = { 0 };\n \n@@ -476,8 +482,7 @@ static void t_table_read_write_seek_index(void)\n \n static void t_table_refs_for(int indexed)\n {\n-\tint N = 50;\n-\tchar **want_names = reftable_calloc(N + 1, sizeof(*want_names));\n+\tchar **want_names;\n \tint want_names_len = 0;\n \tuint8_t want_hash[GIT_SHA1_RAWSZ];\n \n@@ -485,15 +490,15 @@ static void t_table_refs_for(int indexed)\n \t\t.block_size = 256,\n \t};\n \tstruct reftable_ref_record ref = { 0 };\n-\tint i = 0;\n-\tint n;\n-\tint err;\n \tstruct reftable_reader *reader;\n \tstruct reftable_block_source source = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w = t_reftable_strbuf_writer(&buf, &opts);\n \tstruct reftable_iterator it = { 0 };\n-\tint j;\n+\tint N = 50, n, j, err, i;\n+\n+\twant_names = reftable_calloc(N + 1, sizeof(*want_names));\n+\tcheck(want_names != NULL);\n \n \tt_reftable_set_hash(want_hash, 4, GIT_SHA1_FORMAT_ID);\n \n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503826","messageId":"228cc812632d3c310d9b4f02cacc06fe60f6c23c.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 23/25] reftable: fix calls to free(3P)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:42:55Z","receivedAt":"2024-10-01T09:42:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"There are a small set of calls to free(3P) in the reftable library. As\nthe reftable allocators are pluggable we should rather call the reftable\nspecific function, which is `reftable_free()`.\n\nConvert the code accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 2 +-\n t/unit-tests/t-reftable-readwrite.c | 4 ++--\n 2 files changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 7df28ab343..b2babe8e3d 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1421,7 +1421,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\tstruct lock_file *table_lock = &table_locks[i];\n \t\tchar *table_path = get_locked_file_path(table_lock);\n \t\tunlink(table_path);\n-\t\tfree(table_path);\n+\t\treftable_free(table_path);\n \t}\n \n done:\ndiff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c\nindex bfa069caff..27ce84445e 100644\n--- a/t/unit-tests/t-reftable-readwrite.c\n+++ b/t/unit-tests/t-reftable-readwrite.c\n@@ -76,8 +76,8 @@ static void write_table(char ***names, struct strbuf *buf, int N,\n \n \tt_reftable_write_to_buf(buf, refs, N, logs, N, &opts);\n \n-\tfree(refs);\n-\tfree(logs);\n+\treftable_free(refs);\n+\treftable_free(logs);\n }\n \n static void t_log_buffer_size(void)\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503827","messageId":"1c66f6ef8d9827653fd9df960330431877e1dbc9.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 24/25] reftable: introduce `REFTABLE_FREE_AND_NULL()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:43:00Z","receivedAt":"2024-10-01T09:43:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have several calls to `FREE_AND_NULL()` in the reftable library,\nwhich of course uses free(3P). As the reftable allocators are pluggable\nwe should rather call the reftable specific function, which is\n`reftable_free()`.\n\nIntroduce a new macro `REFTABLE_FREE_AND_NULL()` and adapt the callsites\naccordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 1 +\n reftable/block.c | 6 +++---\n reftable/iter.c | 2 +-\n reftable/pq.c | 2 +-\n reftable/reader.c | 2 +-\n reftable/record.c | 10 +++++-----\n reftable/stack.c | 8 ++++----\n reftable/writer.c | 4 ++--\n t/unit-tests/t-reftable-stack.c | 2 +-\n 9 files changed, 19 insertions(+), 18 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 69adeab2e4..7f0f20e50c 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -71,6 +71,7 @@ char *reftable_strdup(const char *str);\n \t\t\tREFTABLE_REALLOC_ARRAY(x, alloc); \\\n \t\t} \\\n \t} while (0)\n+#define REFTABLE_FREE_AND_NULL(p) do { reftable_free(p); (p) = NULL; } while (0)\n \n /* Find the longest shared prefix size of `a` and `b` */\n struct strbuf;\ndiff --git a/reftable/block.c b/reftable/block.c\nindex bfa7dc61bf..8d41a2f99e 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -551,9 +551,9 @@ int block_iter_seek_key(struct block_iter *it, const struct block_reader *br,\n void block_writer_release(struct block_writer *bw)\n {\n \tdeflateEnd(bw->zstream);\n-\tFREE_AND_NULL(bw->zstream);\n-\tFREE_AND_NULL(bw->restarts);\n-\tFREE_AND_NULL(bw->compressed);\n+\tREFTABLE_FREE_AND_NULL(bw->zstream);\n+\tREFTABLE_FREE_AND_NULL(bw->restarts);\n+\tREFTABLE_FREE_AND_NULL(bw->compressed);\n \tstrbuf_release(&bw->last_key);\n \t/* the block is not owned. */\n }\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex d319538f80..d926db653b 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -237,7 +237,7 @@ void reftable_iterator_destroy(struct reftable_iterator *it)\n \t\treturn;\n \tit->ops->close(it->iter_arg);\n \tit->ops = NULL;\n-\tFREE_AND_NULL(it->iter_arg);\n+\tREFTABLE_FREE_AND_NULL(it->iter_arg);\n }\n \n int reftable_iterator_seek_ref(struct reftable_iterator *it,\ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex 03b9912282..6ee1164dd3 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -68,6 +68,6 @@ int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry\n \n void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)\n {\n-\tFREE_AND_NULL(pq->heap);\n+\tREFTABLE_FREE_AND_NULL(pq->heap);\n \tmemset(pq, 0, sizeof(*pq));\n }\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 98e7aa2637..8d37253922 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -678,7 +678,7 @@ void reftable_reader_decref(struct reftable_reader *r)\n \tif (--r->refcount)\n \t\treturn;\n \tblock_source_close(&r->source);\n-\tFREE_AND_NULL(r->name);\n+\tREFTABLE_FREE_AND_NULL(r->name);\n \treftable_free(r);\n }\n \ndiff --git a/reftable/record.c b/reftable/record.c\nindex 787e134c9a..30d563e16d 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -476,8 +476,8 @@ static void reftable_obj_record_key(const void *r, struct strbuf *dest)\n static void reftable_obj_record_release(void *rec)\n {\n \tstruct reftable_obj_record *obj = rec;\n-\tFREE_AND_NULL(obj->hash_prefix);\n-\tFREE_AND_NULL(obj->offsets);\n+\tREFTABLE_FREE_AND_NULL(obj->hash_prefix);\n+\tREFTABLE_FREE_AND_NULL(obj->offsets);\n \tmemset(obj, 0, sizeof(struct reftable_obj_record));\n }\n \n@@ -834,10 +834,10 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \tif (val_type != r->value_type) {\n \t\tswitch (r->value_type) {\n \t\tcase REFTABLE_LOG_UPDATE:\n-\t\t\tFREE_AND_NULL(r->value.update.message);\n+\t\t\tREFTABLE_FREE_AND_NULL(r->value.update.message);\n \t\t\tr->value.update.message_cap = 0;\n-\t\t\tFREE_AND_NULL(r->value.update.email);\n-\t\t\tFREE_AND_NULL(r->value.update.name);\n+\t\t\tREFTABLE_FREE_AND_NULL(r->value.update.email);\n+\t\t\tREFTABLE_FREE_AND_NULL(r->value.update.name);\n \t\t\tbreak;\n \t\tcase REFTABLE_LOG_DELETION:\n \t\t\tbreak;\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex b2babe8e3d..63976e5cea 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -203,7 +203,7 @@ void reftable_stack_destroy(struct reftable_stack *st)\n \n \terr = read_lines(st->list_file, &names);\n \tif (err < 0) {\n-\t\tFREE_AND_NULL(names);\n+\t\tREFTABLE_FREE_AND_NULL(names);\n \t}\n \n \tif (st->readers) {\n@@ -224,7 +224,7 @@ void reftable_stack_destroy(struct reftable_stack *st)\n \t\t}\n \t\tstrbuf_release(&filename);\n \t\tst->readers_len = 0;\n-\t\tFREE_AND_NULL(st->readers);\n+\t\tREFTABLE_FREE_AND_NULL(st->readers);\n \t}\n \n \tif (st->list_fd >= 0) {\n@@ -232,8 +232,8 @@ void reftable_stack_destroy(struct reftable_stack *st)\n \t\tst->list_fd = -1;\n \t}\n \n-\tFREE_AND_NULL(st->list_file);\n-\tFREE_AND_NULL(st->reftable_dir);\n+\tREFTABLE_FREE_AND_NULL(st->list_file);\n+\tREFTABLE_FREE_AND_NULL(st->reftable_dir);\n \treftable_free(st);\n \tfree_names(names);\n }\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 550172e65c..b032a47dec 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -593,7 +593,7 @@ static void object_record_free(void *void_arg UNUSED, void *key)\n {\n \tstruct obj_index_tree_node *entry = key;\n \n-\tFREE_AND_NULL(entry->offsets);\n+\tREFTABLE_FREE_AND_NULL(entry->offsets);\n \tstrbuf_release(&entry->hash);\n \treftable_free(entry);\n }\n@@ -709,7 +709,7 @@ static void writer_clear_index(struct reftable_writer *w)\n {\n \tfor (size_t i = 0; w->index && i < w->index_len; i++)\n \t\tstrbuf_release(&w->index[i].last_key);\n-\tFREE_AND_NULL(w->index);\n+\tREFTABLE_FREE_AND_NULL(w->index);\n \tw->index_len = 0;\n \tw->index_cap = 0;\n }\ndiff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c\nindex 6e41a4cf7e..cc2db2cdef 100644\n--- a/t/unit-tests/t-reftable-stack.c\n+++ b/t/unit-tests/t-reftable-stack.c\n@@ -1149,7 +1149,7 @@ static void unclean_stack_close(struct reftable_stack *st)\n \tfor (size_t i = 0; i < st->readers_len; i++)\n \t\treftable_reader_decref(st->readers[i]);\n \tst->readers_len = 0;\n-\tFREE_AND_NULL(st->readers);\n+\tREFTABLE_FREE_AND_NULL(st->readers);\n }\n \n static void t_reftable_stack_compaction_concurrent_clean(void)\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503828","messageId":"764961e6f02b8e5788bce75830559b70c2e6f231.1727774935.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"[PATCH v4 25/25] reftable/basics: ban standard allocator functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T09:43:03Z","receivedAt":"2024-10-01T09:43:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library uses pluggable allocators, which means that we\nshouldn't ever use the standard allocator functions. But it is an easy\nmistake to make to accidentally use e.g. free(3P) instead of the\nreftable-specific `reftable_free()` function, and we do not have any\nmechanism to detect this misuse right now.\n\nIntroduce a couple of macros that ban the standard allocators, similar\nto how we do it in \"banned.h\".\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 1 +\n reftable/basics.h | 9 +++++++++\n 2 files changed, 10 insertions(+)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex ea53cf102a..c8396dc525 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -6,6 +6,7 @@ license that can be found in the LICENSE file or at\n https://developers.google.com/open-source/licenses/bsd\n */\n \n+#define REFTABLE_ALLOW_BANNED_ALLOCATORS\n #include \"basics.h\"\n #include \"reftable-basics.h\"\n \ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 7f0f20e50c..428e8d1e57 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -73,6 +73,15 @@ char *reftable_strdup(const char *str);\n \t} while (0)\n #define REFTABLE_FREE_AND_NULL(p) do { reftable_free(p); (p) = NULL; } while (0)\n \n+#ifndef REFTABLE_ALLOW_BANNED_ALLOCATORS\n+# define REFTABLE_BANNED(func) use_reftable_##func##_instead\n+# define malloc(sz) REFTABLE_BANNED(malloc)\n+# define realloc(ptr, sz) REFTABLE_BANNED(realloc)\n+# define free(ptr) REFTABLE_BANNED(free)\n+# define calloc(nelem, elsize) REFTABLE_BANNED(calloc)\n+# define strdup(str) REFTABLE_BANNED(strdup)\n+#endif\n+\n /* Find the longest shared prefix size of `a` and `b` */\n struct strbuf;\n int common_prefix_size(struct strbuf *a, struct strbuf *b);\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503852","messageId":"xmqqzfnnyakq.fsf@gitster.g","threadId":"62121","inReplyTo":"cover.1727774935.git.ps@pks.im","subject":"Re: [PATCH v4 00/25] reftable: handle allocation errors","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-10-01T17:52:53Z","receivedAt":"2024-10-01T17:52:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> this is the fourth version of this patch series that converts the\n> reftable library to start handling allocation errors. This is done such\n> that the reftable library can truly behave like a library and let its\n> callers handle such conditions.\n>\n> Changes compared to v3:\n>\n> - Fix some additional sites where we use strdup(3P)/free(3P) by\n> accident.\n>\n> - Convert preexisting callsites of free(3P) to use reftable_free().\n>\n> - Introduce a REFTABLE_FREE_AND_NULL() macro and use it.\n>\n> - Ban standard allocators in reftable code \"banned.h\"-style.\n\nWith this patch, there is only one hit from\n\n $ git grep '[^_]free(' reftable/\n\nand no hits from\n\n $ git grep '[^_]FREE_AND_NULL(' reftable/\n\nwhich is just as expected. Nicely done.\n\nShouldn't we add FREE_AND_NULL() to the banned list as well in the\nlast step?\n\nThanks.\n\n\n"},{"id":"503855","messageId":"7740463b-84b4-4a69-961c-278ce2901f0d@web.de","threadId":"62121","inReplyTo":"xmqqzfnnyakq.fsf@gitster.g","subject":"Re: [PATCH v4 00/25] reftable: handle allocation errors","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2024-10-01T18:30:23Z","receivedAt":"2024-10-01T18:30:31Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 01.10.24 um 19:52 schrieb Junio C Hamano:\n> Shouldn't we add FREE_AND_NULL() to the banned list as well in the\n> last step?\n\nAnd perhaps the wrapper.h functions like xmalloc()? At least as long as\ngit-compat-util.h is included by reftable/system.h. Can be done later,\nof course, no need to reroll just for that.\n\nRené\n\n"},{"id":"503865","messageId":"xmqqjzervd69.fsf@gitster.g","threadId":"62121","inReplyTo":"7740463b-84b4-4a69-961c-278ce2901f0d@web.de","subject":"Re: [PATCH v4 00/25] reftable: handle allocation errors","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-10-01T19:25:02Z","receivedAt":"2024-10-01T19:25:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe writes:\n\n> Am 01.10.24 um 19:52 schrieb Junio C Hamano:\n>> Shouldn't we add FREE_AND_NULL() to the banned list as well in the\n>> last step?\n>\n> And perhaps the wrapper.h functions like xmalloc()? At least as long as\n> git-compat-util.h is included by reftable/system.h. Can be done later,\n> of course, no need to reroll just for that.\n\nYeah, and I agree FREE_AND_NULL() falls into the same bucket as xmalloc()\nand friends, so no need to reroll just for that, either.\n\nThanks.\n"},{"id":"503873","messageId":"xmqqh69vsail.fsf@gitster.g","threadId":"62121","inReplyTo":"764961e6f02b8e5788bce75830559b70c2e6f231.1727774935.git.ps@pks.im","subject":"Re: [PATCH v4 25/25] reftable/basics: ban standard allocator functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-10-01T22:50:42Z","receivedAt":"2024-10-01T22:50:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> +#ifndef REFTABLE_ALLOW_BANNED_ALLOCATORS\n> +# define REFTABLE_BANNED(func) use_reftable_##func##_instead\n\nWe'd need to mimic banned.h a bit better, by adding\n\n#undef malloc\n#undef realloc\n#undef free\n#undef calloc\n#undef strdup\n\nbefore (potentially re-)defining them.\n\n> +# define malloc(sz) REFTABLE_BANNED(malloc)\n> +# define realloc(ptr, sz) REFTABLE_BANNED(realloc)\n> +# define free(ptr) REFTABLE_BANNED(free)\n> +# define calloc(nelem, elsize) REFTABLE_BANNED(calloc)\n> +# define strdup(str) REFTABLE_BANNED(strdup)\n> +#endif\n"},{"id":"503887","messageId":"ZvzMIWNfzmZL7eHS@pks.im","threadId":"62121","inReplyTo":"xmqqjzervd69.fsf@gitster.g","subject":"Re: [PATCH v4 00/25] reftable: handle allocation errors","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T04:29:27Z","receivedAt":"2024-10-02T04:29:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 01, 2024 at 12:25:02PM -0700, Junio C Hamano wrote:\n> René Scharfe writes:\n> \n> > Am 01.10.24 um 19:52 schrieb Junio C Hamano:\n> >> Shouldn't we add FREE_AND_NULL() to the banned list as well in the\n> >> last step?\n> >\n> > And perhaps the wrapper.h functions like xmalloc()? At least as long as\n> > git-compat-util.h is included by reftable/system.h. Can be done later,\n> > of course, no need to reroll just for that.\n> \n> Yeah, and I agree FREE_AND_NULL() falls into the same bucket as xmalloc()\n> and friends, so no need to reroll just for that, either.\n\n`FREE_AND_NULL()` would already be detected because it expands to code\nthat contains `free()`. Same for `REALLOC_ARRAY()` and the likes.\n\nI'm overall a bit more hesitant to also start banning all the Git\nspecific wrappers, mostly because there are so many of them. My goal is\nto rather make them inaccessible in the \"reftable/\" code in the first\nplace so that we don't have to play whack-a-mole. My plan here is to\nsplit out the POSIX-providing bits from \"git-compat-util.h\" into a new\nheader \"compat/posix.h\" that we can include in \"reftable/system.h\".\n\nBut it'll take a while to get there :)\n\nThanks for your inputs!\n\nPatrick\n"},{"id":"503888","messageId":"ZvzMbxdfRZU9pQ76@pks.im","threadId":"62121","inReplyTo":"xmqqh69vsail.fsf@gitster.g","subject":"Re: [PATCH v4 25/25] reftable/basics: ban standard allocator functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T04:30:48Z","receivedAt":"2024-10-02T04:30:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 01, 2024 at 03:50:42PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > +#ifndef REFTABLE_ALLOW_BANNED_ALLOCATORS\n> > +# define REFTABLE_BANNED(func) use_reftable_##func##_instead\n> \n> We'd need to mimic banned.h a bit better, by adding\n> \n> #undef malloc\n> #undef realloc\n> #undef free\n> #undef calloc\n> #undef strdup\n> \n> before (potentially re-)defining them.\n> \n> > +# define malloc(sz) REFTABLE_BANNED(malloc)\n> > +# define realloc(ptr, sz) REFTABLE_BANNED(realloc)\n> > +# define free(ptr) REFTABLE_BANNED(free)\n> > +# define calloc(nelem, elsize) REFTABLE_BANNED(calloc)\n> > +# define strdup(str) REFTABLE_BANNED(strdup)\n> > +#endif\n\nOh, indeed, that's something I missed. Will do for a hopefully last\nreroll.\n\nPatrick\n"},{"id":"503901","messageId":"cover.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1726489647.git.ps@pks.im","subject":"[PATCH v5 00/25] reftable: handle allocation errors","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:55:22Z","receivedAt":"2024-10-02T10:55:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the fifth version of this patch series that converts the\nreftable library to start handling allocation errors. This is done such\nthat the reftable library can truly behave like a library and let its\ncallers handle such conditions.\n\nChanges compared to v4:\n\n - `#undef` standard allocators before banning them.\n\n - Explain why we don't ban `FREE_AND_NULL()` and related as well as\n `xmalloc()` and related.\n\nThanks!\n\nPatrick\n\nPatrick Steinhardt (25):\n reftable/error: introduce out-of-memory error code\n reftable/basics: merge \"publicbasics\" into \"basics\"\n reftable: introduce `reftable_strdup()`\n reftable/basics: handle allocation failures in `reftable_calloc()`\n reftable/basics: handle allocation failures in `parse_names()`\n reftable/record: handle allocation failures on copy\n reftable/record: handle allocation failures when decoding records\n reftable/writer: handle allocation failures in `writer_index_hash()`\n reftable/writer: handle allocation failures in `reftable_new_writer()`\n reftable/merged: handle allocation failures in\n `merged_table_init_iter()`\n reftable/reader: handle allocation failures for unindexed reader\n reftable/reader: handle allocation failures in `reader_init_iter()`\n reftable/stack: handle allocation failures on reload\n reftable/stack: handle allocation failures in `reftable_new_stack()`\n reftable/stack: handle allocation failures in `stack_compact_range()`\n reftable/stack: handle allocation failures in auto compaction\n reftable/iter: handle allocation failures when creating indexed table\n iter\n reftable/blocksource: handle allocation failures\n reftable/block: handle allocation failures\n reftable/pq: handle allocation failures when adding entries\n reftable/tree: handle allocation failures\n reftable: handle trivial allocation failures\n reftable: fix calls to free(3P)\n reftable: introduce `REFTABLE_FREE_AND_NULL()`\n reftable/basics: ban standard allocator functions\n\n Makefile | 1 -\n refs/reftable-backend.c | 39 ++++--\n reftable/basics.c | 93 +++++++++++++-\n reftable/basics.h | 28 ++++-\n reftable/block.c | 29 ++++-\n reftable/block.h | 4 +-\n reftable/blocksource.c | 25 +++-\n reftable/error.c | 2 +\n reftable/iter.c | 22 +++-\n reftable/iter.h | 2 +-\n reftable/merged.c | 84 ++++++++-----\n reftable/merged.h | 6 +-\n reftable/pq.c | 9 +-\n reftable/pq.h | 2 +-\n reftable/publicbasics.c | 66 ----------\n reftable/reader.c | 70 ++++++++---\n reftable/reader.h | 6 +-\n reftable/record.c | 174 +++++++++++++++++++-------\n reftable/record.h | 6 +-\n reftable/reftable-basics.h | 18 +++\n reftable/reftable-error.h | 3 +\n reftable/reftable-malloc.h | 18 ---\n reftable/reftable-merged.h | 8 +-\n reftable/reftable-reader.h | 8 +-\n reftable/reftable-stack.h | 8 +-\n reftable/reftable-writer.h | 12 +-\n reftable/stack.c | 187 +++++++++++++++++++++-------\n reftable/tree.c | 42 +++++--\n reftable/tree.h | 21 +++-\n reftable/writer.c | 154 +++++++++++++++--------\n t/helper/test-reftable.c | 10 +-\n t/unit-tests/lib-reftable.c | 8 +-\n t/unit-tests/t-reftable-basics.c | 11 +-\n t/unit-tests/t-reftable-block.c | 24 ++--\n t/unit-tests/t-reftable-merged.c | 16 ++-\n t/unit-tests/t-reftable-readwrite.c | 65 ++++++----\n t/unit-tests/t-reftable-stack.c | 6 +-\n t/unit-tests/t-reftable-tree.c | 10 +-\n 38 files changed, 890 insertions(+), 407 deletions(-)\n delete mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-basics.h\n delete mode 100644 reftable/reftable-malloc.h\n\nRange-diff against v4:\n 1: 94eaef3ae5 = 1: 94eaef3ae5 reftable/error: introduce out-of-memory error code\n 2: fe55051cb7 = 2: fe55051cb7 reftable/basics: merge \"publicbasics\" into \"basics\"\n 3: b1a10d41d3 = 3: b1a10d41d3 reftable: introduce `reftable_strdup()`\n 4: fd9141e985 = 4: fd9141e985 reftable/basics: handle allocation failures in `reftable_calloc()`\n 5: bdfddbebce = 5: bdfddbebce reftable/basics: handle allocation failures in `parse_names()`\n 6: 756a32c285 = 6: 756a32c285 reftable/record: handle allocation failures on copy\n 7: ca64971e7b = 7: ca64971e7b reftable/record: handle allocation failures when decoding records\n 8: fc2f113cba = 8: fc2f113cba reftable/writer: handle allocation failures in `writer_index_hash()`\n 9: 0ed99e0bdf = 9: 0ed99e0bdf reftable/writer: handle allocation failures in `reftable_new_writer()`\n10: 8dfbfd9286 = 10: 8dfbfd9286 reftable/merged: handle allocation failures in `merged_table_init_iter()`\n11: 7b592a6c6b = 11: 7b592a6c6b reftable/reader: handle allocation failures for unindexed reader\n12: ceed838265 = 12: ceed838265 reftable/reader: handle allocation failures in `reader_init_iter()`\n13: 1e997a5766 = 13: 1e997a5766 reftable/stack: handle allocation failures on reload\n14: 40d4d81378 = 14: 40d4d81378 reftable/stack: handle allocation failures in `reftable_new_stack()`\n15: 6aaae4baa6 = 15: 6aaae4baa6 reftable/stack: handle allocation failures in `stack_compact_range()`\n16: 103a59ef0a = 16: 103a59ef0a reftable/stack: handle allocation failures in auto compaction\n17: 6799d299fe = 17: 6799d299fe reftable/iter: handle allocation failures when creating indexed table iter\n18: c7e54d71d7 = 18: c7e54d71d7 reftable/blocksource: handle allocation failures\n19: 92d39b9021 = 19: 92d39b9021 reftable/block: handle allocation failures\n20: 3416004e0d = 20: 3416004e0d reftable/pq: handle allocation failures when adding entries\n21: a363930488 = 21: a363930488 reftable/tree: handle allocation failures\n22: 28661500ff = 22: 28661500ff reftable: handle trivial allocation failures\n23: 228cc81263 = 23: 228cc81263 reftable: fix calls to free(3P)\n24: 1c66f6ef8d = 24: 1c66f6ef8d reftable: introduce `REFTABLE_FREE_AND_NULL()`\n25: 764961e6f0 ! 25: bc80f1e726 reftable/basics: ban standard allocator functions\n @@ Commit message\n Introduce a couple of macros that ban the standard allocators, similar\n to how we do it in \"banned.h\".\n \n + Note that we do not ban the following two classes of functions:\n +\n + - Macros like `FREE_AND_NULL()` or `REALLOC_ARRAY()`. As those expand\n + to code that contains already-banned functions we'd get a compiler\n + error even without banning those macros explicitly.\n +\n + - Git-specific allocators like `xmalloc()` and friends. The primary\n + reason is that there are simply too many of them, so we're rather\n + aiming for best effort here. Furthermore, the eventual goal is to\n + make them unavailable in the reftable library place by not pulling\n + them in via \"git-compat-utils.h\" anymore.\n +\n Signed-off-by: Patrick Steinhardt \n \n ## reftable/basics.c ##\n @@ reftable/basics.h: char *reftable_strdup(const char *str);\n \n +#ifndef REFTABLE_ALLOW_BANNED_ALLOCATORS\n +# define REFTABLE_BANNED(func) use_reftable_##func##_instead\n ++# undef malloc\n +# define malloc(sz) REFTABLE_BANNED(malloc)\n ++# undef realloc\n +# define realloc(ptr, sz) REFTABLE_BANNED(realloc)\n ++# undef free\n +# define free(ptr) REFTABLE_BANNED(free)\n ++# undef calloc\n +# define calloc(nelem, elsize) REFTABLE_BANNED(calloc)\n ++# undef strdup\n +# define strdup(str) REFTABLE_BANNED(strdup)\n +#endif\n +\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503902","messageId":"94eaef3ae5ee008b5140272e4ce572c041f9c5a2.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 01/25] reftable/error: introduce out-of-memory error code","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:55:24Z","receivedAt":"2024-10-02T10:55:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library does not use the same memory allocation functions\nas the rest of the Git codebase. Instead, as the reftable library is\nsupposed to be usable as a standalone library without Git, it provides a\nset of pluggable memory allocators.\n\nCompared to `xmalloc()` and friends these allocators are _not_ expected\nto die when an allocation fails. This design choice is concious, as a\nlibrary should leave it to its caller to handle any kind of error. While\nit is very likely that the caller cannot really do much in the case of\nan out-of-memory situation anyway, we are not the ones to make that\ndecision.\n\nCuriously though, we never handle allocation errors even though memory\nallocation functions are allowed to fail. And as we do not plug in Git's\nmemory allocator via `reftable_set_alloc()` either the consequence is\nthat we'd instead segfault as soon as we run out of memory.\n\nWhile the easy fix would be to wire up `xmalloc()` and friends, it\nwould only fix the usage of the reftable library in Git itself. Other\nusers like libgit2, which is about to revive its efforts to land a\nbackend for reftables, wouldn't be able to benefit from this solution.\n\nInstead, we are about to do it the hard way: adapt all allocation sites\nto perform error checking. Introduce a new error code for out-of-memory\nerrors that we will wire up in subsequent steps.\n\nThis commit also serves as the motivator for all the remaining steps in\nthis series such that we do not have to repeat the same arguments in\nevery single subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/error.c | 2 ++\n reftable/reftable-error.h | 3 +++\n 2 files changed, 5 insertions(+)\n\ndiff --git a/reftable/error.c b/reftable/error.c\nindex a25f28a43e..660d029617 100644\n--- a/reftable/error.c\n+++ b/reftable/error.c\n@@ -35,6 +35,8 @@ const char *reftable_error_str(int err)\n \t\treturn \"entry too large\";\n \tcase REFTABLE_OUTDATED_ERROR:\n \t\treturn \"data concurrently modified\";\n+\tcase REFTABLE_OUT_OF_MEMORY_ERROR:\n+\t\treturn \"out of memory\";\n \tcase -1:\n \t\treturn \"general error\";\n \tdefault:\ndiff --git a/reftable/reftable-error.h b/reftable/reftable-error.h\nindex 6368cd9ed9..f404826562 100644\n--- a/reftable/reftable-error.h\n+++ b/reftable/reftable-error.h\n@@ -57,6 +57,9 @@ enum reftable_error {\n \n \t/* Trying to write out-of-date data. */\n \tREFTABLE_OUTDATED_ERROR = -12,\n+\n+\t/* An allocation has failed due to an out-of-memory situation. */\n+\tREFTABLE_OUT_OF_MEMORY_ERROR = -13,\n };\n \n /* convert the numeric error code to a string. The string should not be\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503903","messageId":"fe55051cb79015960729d7a06f21f43138585177.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 02/25] reftable/basics: merge \"publicbasics\" into \"basics\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:55:27Z","receivedAt":"2024-10-02T10:55:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The split between \"basics\" and \"publicbasics\" is somewhat arbitrary and\nnot in line with how we typically structure code in the reftable\nlibrary. While we do indeed split up headers into a public and internal\npart, we don't do that for the compilation unit itself. Furthermore, the\ndeclarations for \"publicbasics.c\" are in \"reftable-malloc.h\", which\nisn't in line with our naming schema, either.\n\nFix these inconsistencies by:\n\n - Merging \"publicbasics.c\" into \"basics.c\".\n\n - Renaming \"reftable-malloc.h\" to \"reftable-basics.h\" as the public\n header.\n\nSigned-off-by: Patrick Steinhardt \n---\n Makefile | 1 -\n reftable/basics.c | 55 +++++++++++++++++++++++++++++++\n reftable/basics.h | 3 ++\n reftable/publicbasics.c | 66 --------------------------------------\n reftable/reftable-basics.h | 18 +++++++++++\n reftable/reftable-malloc.h | 18 -----------\n 6 files changed, 76 insertions(+), 85 deletions(-)\n delete mode 100644 reftable/publicbasics.c\n create mode 100644 reftable/reftable-basics.h\n delete mode 100644 reftable/reftable-malloc.h\n\ndiff --git a/Makefile b/Makefile\nindex e3abf0ba83..39b10923d4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2683,7 +2683,6 @@ REFTABLE_OBJS += reftable/error.o\n REFTABLE_OBJS += reftable/block.o\n REFTABLE_OBJS += reftable/blocksource.o\n REFTABLE_OBJS += reftable/iter.o\n-REFTABLE_OBJS += reftable/publicbasics.o\n REFTABLE_OBJS += reftable/merged.o\n REFTABLE_OBJS += reftable/pq.o\n REFTABLE_OBJS += reftable/reader.o\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 0058619ca6..cf072935c8 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -7,6 +7,49 @@ license that can be found in the LICENSE file or at\n */\n \n #include \"basics.h\"\n+#include \"reftable-basics.h\"\n+\n+static void *(*reftable_malloc_ptr)(size_t sz);\n+static void *(*reftable_realloc_ptr)(void *, size_t);\n+static void (*reftable_free_ptr)(void *);\n+\n+void *reftable_malloc(size_t sz)\n+{\n+\tif (reftable_malloc_ptr)\n+\t\treturn (*reftable_malloc_ptr)(sz);\n+\treturn malloc(sz);\n+}\n+\n+void *reftable_realloc(void *p, size_t sz)\n+{\n+\tif (reftable_realloc_ptr)\n+\t\treturn (*reftable_realloc_ptr)(p, sz);\n+\treturn realloc(p, sz);\n+}\n+\n+void reftable_free(void *p)\n+{\n+\tif (reftable_free_ptr)\n+\t\treftable_free_ptr(p);\n+\telse\n+\t\tfree(p);\n+}\n+\n+void *reftable_calloc(size_t nelem, size_t elsize)\n+{\n+\tsize_t sz = st_mult(nelem, elsize);\n+\tvoid *p = reftable_malloc(sz);\n+\tmemset(p, 0, sz);\n+\treturn p;\n+}\n+\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n+{\n+\treftable_malloc_ptr = malloc;\n+\treftable_realloc_ptr = realloc;\n+\treftable_free_ptr = free;\n+}\n \n void put_be24(uint8_t *out, uint32_t i)\n {\n@@ -121,3 +164,15 @@ int common_prefix_size(struct strbuf *a, struct strbuf *b)\n \n \treturn p;\n }\n+\n+int hash_size(uint32_t id)\n+{\n+\tswitch (id) {\n+\tcase 0:\n+\tcase GIT_SHA1_FORMAT_ID:\n+\t\treturn GIT_SHA1_RAWSZ;\n+\tcase GIT_SHA256_FORMAT_ID:\n+\t\treturn GIT_SHA256_RAWSZ;\n+\t}\n+\tabort();\n+}\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex c8fec68d4e..4e2e76014a 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -14,6 +14,7 @@ license that can be found in the LICENSE file or at\n */\n \n #include \"system.h\"\n+#include \"reftable-basics.h\"\n \n /* Bigendian en/decoding of integers */\n \n@@ -71,4 +72,6 @@ void *reftable_calloc(size_t nelem, size_t elsize);\n struct strbuf;\n int common_prefix_size(struct strbuf *a, struct strbuf *b);\n \n+int hash_size(uint32_t id);\n+\n #endif\ndiff --git a/reftable/publicbasics.c b/reftable/publicbasics.c\ndeleted file mode 100644\nindex 44b84a125e..0000000000\n--- a/reftable/publicbasics.c\n+++ /dev/null\n@@ -1,66 +0,0 @@\n-/*\n-Copyright 2020 Google LLC\n-\n-Use of this source code is governed by a BSD-style\n-license that can be found in the LICENSE file or at\n-https://developers.google.com/open-source/licenses/bsd\n-*/\n-\n-#include \"system.h\"\n-#include \"reftable-malloc.h\"\n-\n-#include \"basics.h\"\n-\n-static void *(*reftable_malloc_ptr)(size_t sz);\n-static void *(*reftable_realloc_ptr)(void *, size_t);\n-static void (*reftable_free_ptr)(void *);\n-\n-void *reftable_malloc(size_t sz)\n-{\n-\tif (reftable_malloc_ptr)\n-\t\treturn (*reftable_malloc_ptr)(sz);\n-\treturn malloc(sz);\n-}\n-\n-void *reftable_realloc(void *p, size_t sz)\n-{\n-\tif (reftable_realloc_ptr)\n-\t\treturn (*reftable_realloc_ptr)(p, sz);\n-\treturn realloc(p, sz);\n-}\n-\n-void reftable_free(void *p)\n-{\n-\tif (reftable_free_ptr)\n-\t\treftable_free_ptr(p);\n-\telse\n-\t\tfree(p);\n-}\n-\n-void *reftable_calloc(size_t nelem, size_t elsize)\n-{\n-\tsize_t sz = st_mult(nelem, elsize);\n-\tvoid *p = reftable_malloc(sz);\n-\tmemset(p, 0, sz);\n-\treturn p;\n-}\n-\n-void reftable_set_alloc(void *(*malloc)(size_t),\n-\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n-{\n-\treftable_malloc_ptr = malloc;\n-\treftable_realloc_ptr = realloc;\n-\treftable_free_ptr = free;\n-}\n-\n-int hash_size(uint32_t id)\n-{\n-\tswitch (id) {\n-\tcase 0:\n-\tcase GIT_SHA1_FORMAT_ID:\n-\t\treturn GIT_SHA1_RAWSZ;\n-\tcase GIT_SHA256_FORMAT_ID:\n-\t\treturn GIT_SHA256_RAWSZ;\n-\t}\n-\tabort();\n-}\ndiff --git a/reftable/reftable-basics.h b/reftable/reftable-basics.h\nnew file mode 100644\nindex 0000000000..6e8e636b71\n--- /dev/null\n+++ b/reftable/reftable-basics.h\n@@ -0,0 +1,18 @@\n+/*\n+ * Copyright 2020 Google LLC\n+ *\n+ * Use of this source code is governed by a BSD-style\n+ * license that can be found in the LICENSE file or at\n+ * https://developers.google.com/open-source/licenses/bsd\n+*/\n+\n+#ifndef REFTABLE_BASICS_H\n+#define REFTABLE_BASICS_H\n+\n+#include \n+\n+/* Overrides the functions to use for memory management. */\n+void reftable_set_alloc(void *(*malloc)(size_t),\n+\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n+\n+#endif\ndiff --git a/reftable/reftable-malloc.h b/reftable/reftable-malloc.h\ndeleted file mode 100644\nindex 5f2185f1f3..0000000000\n--- a/reftable/reftable-malloc.h\n+++ /dev/null\n@@ -1,18 +0,0 @@\n-/*\n-Copyright 2020 Google LLC\n-\n-Use of this source code is governed by a BSD-style\n-license that can be found in the LICENSE file or at\n-https://developers.google.com/open-source/licenses/bsd\n-*/\n-\n-#ifndef REFTABLE_H\n-#define REFTABLE_H\n-\n-#include \n-\n-/* Overrides the functions to use for memory management. */\n-void reftable_set_alloc(void *(*malloc)(size_t),\n-\t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *));\n-\n-#endif\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503904","messageId":"b1a10d41d3798c98c53b251df9077fcc5eaef560.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 03/25] reftable: introduce `reftable_strdup()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:55:30Z","receivedAt":"2024-10-02T10:55:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library provides the ability to swap out allocators. There\nis a gap here though, because we continue to use `xstrdup()` even in the\ncase where all the other allocators have been swapped out.\n\nIntroduce `reftable_strdup()` that uses `reftable_malloc()` to do the\nallocation.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 10 ++++++++++\n reftable/basics.h | 1 +\n 2 files changed, 11 insertions(+)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex cf072935c8..4adc98cf5d 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -43,6 +43,16 @@ void *reftable_calloc(size_t nelem, size_t elsize)\n \treturn p;\n }\n \n+char *reftable_strdup(const char *str)\n+{\n+\tsize_t len = strlen(str);\n+\tchar *result = reftable_malloc(len + 1);\n+\tif (!result)\n+\t\treturn NULL;\n+\tmemcpy(result, str, len + 1);\n+\treturn result;\n+}\n+\n void reftable_set_alloc(void *(*malloc)(size_t),\n \t\t\tvoid *(*realloc)(void *, size_t), void (*free)(void *))\n {\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 4e2e76014a..f107e14860 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -54,6 +54,7 @@ void *reftable_malloc(size_t sz);\n void *reftable_realloc(void *p, size_t sz);\n void reftable_free(void *p);\n void *reftable_calloc(size_t nelem, size_t elsize);\n+char *reftable_strdup(const char *str);\n \n #define REFTABLE_ALLOC_ARRAY(x, alloc) (x) = reftable_malloc(st_mult(sizeof(*(x)), (alloc)))\n #define REFTABLE_CALLOC_ARRAY(x, alloc) (x) = reftable_calloc((alloc), sizeof(*(x)))\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503905","messageId":"fd9141e985bd83d8374ec339dbdce7a2350aa31d.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 04/25] reftable/basics: handle allocation failures in `reftable_calloc()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:55:35Z","receivedAt":"2024-10-02T10:55:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_calloc()`.\n\nWhile at it, remove our use of `st_mult()` that would cause us to die on\nan overflow. From the caller's point of view there is not much of a\ndifference between arguments that are too large to be multiplied and a\nrequest that is too big to handle by the allocator: in both cases the\nallocation cannot be fulfilled. And in neither of these cases do we want\nthe reftable library to die.\n\nWhile we could use `unsigned_mult_overflows()` to handle the overflow\ngracefully, we instead open-code it to further our goal of converting\nthe reftable codebase to become a standalone library that can be reused\nby external projects.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 13 ++++++++++---\n 1 file changed, 10 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 4adc98cf5d..3350bbffa2 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -37,9 +37,16 @@ void reftable_free(void *p)\n \n void *reftable_calloc(size_t nelem, size_t elsize)\n {\n-\tsize_t sz = st_mult(nelem, elsize);\n-\tvoid *p = reftable_malloc(sz);\n-\tmemset(p, 0, sz);\n+\tvoid *p;\n+\n+\tif (nelem && elsize > SIZE_MAX / nelem)\n+\t\treturn NULL;\n+\n+\tp = reftable_malloc(nelem * elsize);\n+\tif (!p)\n+\t\treturn NULL;\n+\n+\tmemset(p, 0, nelem * elsize);\n \treturn p;\n }\n \n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503906","messageId":"bdfddbebce9f77959fd9544cd5ba3496d5b9dccf.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 05/25] reftable/basics: handle allocation failures in `parse_names()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:55:38Z","receivedAt":"2024-10-02T10:55:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `parse_names()` by returning `NULL` in\ncase any allocation fails. While at it, refactor the function to return\nthe array directly instead of assigning it to an out-pointer.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 20 ++++++++++++++++----\n reftable/basics.h | 9 ++++++---\n reftable/stack.c | 6 +++++-\n t/unit-tests/t-reftable-basics.c | 11 ++++++-----\n 4 files changed, 33 insertions(+), 13 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 3350bbffa2..ea53cf102a 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -135,14 +135,14 @@ size_t names_length(const char **names)\n \treturn p - names;\n }\n \n-void parse_names(char *buf, int size, char ***namesp)\n+char **parse_names(char *buf, int size)\n {\n \tchar **names = NULL;\n \tsize_t names_cap = 0;\n \tsize_t names_len = 0;\n-\n \tchar *p = buf;\n \tchar *end = buf + size;\n+\n \twhile (p < end) {\n \t\tchar *next = strchr(p, '\\n');\n \t\tif (next && next < end) {\n@@ -152,14 +152,26 @@ void parse_names(char *buf, int size, char ***namesp)\n \t\t}\n \t\tif (p < next) {\n \t\t\tREFTABLE_ALLOC_GROW(names, names_len + 1, names_cap);\n-\t\t\tnames[names_len++] = xstrdup(p);\n+\t\t\tif (!names)\n+\t\t\t\tgoto err;\n+\n+\t\t\tnames[names_len] = reftable_strdup(p);\n+\t\t\tif (!names[names_len++])\n+\t\t\t\tgoto err;\n \t\t}\n \t\tp = next + 1;\n \t}\n \n \tREFTABLE_REALLOC_ARRAY(names, names_len + 1);\n \tnames[names_len] = NULL;\n-\t*namesp = names;\n+\n+\treturn names;\n+\n+err:\n+\tfor (size_t i = 0; i < names_len; i++)\n+\t\treftable_free(names[i]);\n+\treftable_free(names);\n+\treturn NULL;\n }\n \n int names_equal(const char **a, const char **b)\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex f107e14860..69adeab2e4 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -38,9 +38,12 @@ size_t binsearch(size_t sz, int (*f)(size_t k, void *args), void *args);\n */\n void free_names(char **a);\n \n-/* parse a newline separated list of names. `size` is the length of the buffer,\n- * without terminating '\\0'. Empty names are discarded. */\n-void parse_names(char *buf, int size, char ***namesp);\n+/*\n+ * Parse a newline separated list of names. `size` is the length of the buffer,\n+ * without terminating '\\0'. Empty names are discarded. Returns a `NULL`\n+ * pointer when allocations fail.\n+ */\n+char **parse_names(char *buf, int size);\n \n /* compares two NULL-terminated arrays of strings. */\n int names_equal(const char **a, const char **b);\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex ce0a35216b..498fae846d 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -108,7 +108,11 @@ static int fd_read_lines(int fd, char ***namesp)\n \t}\n \tbuf[size] = 0;\n \n-\tparse_names(buf, size, namesp);\n+\t*namesp = parse_names(buf, size);\n+\tif (!*namesp) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \n done:\n \treftable_free(buf);\ndiff --git a/t/unit-tests/t-reftable-basics.c b/t/unit-tests/t-reftable-basics.c\nindex e5556ebf52..1fa77b6faf 100644\n--- a/t/unit-tests/t-reftable-basics.c\n+++ b/t/unit-tests/t-reftable-basics.c\n@@ -72,13 +72,14 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \tif_test (\"parse_names works for basic input\") {\n \t\tchar in1[] = \"line\\n\";\n \t\tchar in2[] = \"a\\nb\\nc\";\n-\t\tchar **out = NULL;\n-\t\tparse_names(in1, strlen(in1), &out);\n+\t\tchar **out = parse_names(in1, strlen(in1));\n+\t\tcheck(out != NULL);\n \t\tcheck_str(out[0], \"line\");\n \t\tcheck(!out[1]);\n \t\tfree_names(out);\n \n-\t\tparse_names(in2, strlen(in2), &out);\n+\t\tout = parse_names(in2, strlen(in2));\n+\t\tcheck(out != NULL);\n \t\tcheck_str(out[0], \"a\");\n \t\tcheck_str(out[1], \"b\");\n \t\tcheck_str(out[2], \"c\");\n@@ -88,8 +89,8 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \n \tif_test (\"parse_names drops empty string\") {\n \t\tchar in[] = \"a\\n\\nb\\n\";\n-\t\tchar **out = NULL;\n-\t\tparse_names(in, strlen(in), &out);\n+\t\tchar **out = parse_names(in, strlen(in));\n+\t\tcheck(out != NULL);\n \t\tcheck_str(out[0], \"a\");\n \t\t/* simply '\\n' should be dropped as empty string */\n \t\tcheck_str(out[1], \"b\");\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503907","messageId":"756a32c28558bfaa360304712110443d643fc303.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 06/25] reftable/record: handle allocation failures on copy","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:55:40Z","receivedAt":"2024-10-02T10:55:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when copying records. While at it, convert\nfrom `xstrdup()` to `reftable_strdup()`. Adapt callsites to check for\nerror codes.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/record.c | 84 +++++++++++++++++++++++++++++++++--------------\n reftable/record.h | 6 ++--\n 2 files changed, 63 insertions(+), 27 deletions(-)\n\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 6b5a075b92..60fd33c9c9 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -215,13 +215,14 @@ static void reftable_ref_record_key(const void *r, struct strbuf *dest)\n \tstrbuf_addstr(dest, rec->refname);\n }\n \n-static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size)\n+static int reftable_ref_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n {\n \tstruct reftable_ref_record *ref = rec;\n \tconst struct reftable_ref_record *src = src_rec;\n \tchar *refname = NULL;\n \tsize_t refname_cap = 0;\n+\tint err;\n \n \tassert(hash_size > 0);\n \n@@ -236,6 +237,11 @@ static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n \n \t\tREFTABLE_ALLOC_GROW(ref->refname, refname_len + 1,\n \t\t\t\t ref->refname_cap);\n+\t\tif (!ref->refname) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n+\n \t\tmemcpy(ref->refname, src->refname, refname_len);\n \t\tref->refname[refname_len] = 0;\n \t}\n@@ -254,9 +260,17 @@ static void reftable_ref_record_copy_from(void *rec, const void *src_rec,\n \t\t src->value.val2.target_value, hash_size);\n \t\tbreak;\n \tcase REFTABLE_REF_SYMREF:\n-\t\tref->value.symref = xstrdup(src->value.symref);\n+\t\tref->value.symref = reftable_strdup(src->value.symref);\n+\t\tif (!ref->value.symref) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n \t\tbreak;\n \t}\n+\n+\terr = 0;\n+out:\n+\treturn err;\n }\n \n static void reftable_ref_record_release_void(void *rec)\n@@ -457,23 +471,28 @@ static void reftable_obj_record_release(void *rec)\n \tmemset(obj, 0, sizeof(struct reftable_obj_record));\n }\n \n-static void reftable_obj_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size UNUSED)\n+static int reftable_obj_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size UNUSED)\n {\n \tstruct reftable_obj_record *obj = rec;\n-\tconst struct reftable_obj_record *src =\n-\t\t(const struct reftable_obj_record *)src_rec;\n+\tconst struct reftable_obj_record *src = src_rec;\n \n \treftable_obj_record_release(obj);\n \n \tREFTABLE_ALLOC_ARRAY(obj->hash_prefix, src->hash_prefix_len);\n+\tif (!obj->hash_prefix)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tobj->hash_prefix_len = src->hash_prefix_len;\n \tif (src->hash_prefix_len)\n \t\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n \n \tREFTABLE_ALLOC_ARRAY(obj->offsets, src->offset_len);\n+\tif (!obj->offsets)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tobj->offset_len = src->offset_len;\n \tCOPY_ARRAY(obj->offsets, src->offsets, src->offset_len);\n+\n+\treturn 0;\n }\n \n static uint8_t reftable_obj_record_val_type(const void *rec)\n@@ -646,33 +665,44 @@ static void reftable_log_record_key(const void *r, struct strbuf *dest)\n \tstrbuf_add(dest, i64, sizeof(i64));\n }\n \n-static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size)\n+static int reftable_log_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size)\n {\n \tstruct reftable_log_record *dst = rec;\n \tconst struct reftable_log_record *src =\n \t\t(const struct reftable_log_record *)src_rec;\n+\tint ret;\n \n \treftable_log_record_release(dst);\n \t*dst = *src;\n+\n \tif (dst->refname) {\n-\t\tdst->refname = xstrdup(dst->refname);\n+\t\tdst->refname = reftable_strdup(dst->refname);\n+\t\tif (!dst->refname) {\n+\t\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n+\t\t}\n \t}\n+\n \tswitch (dst->value_type) {\n \tcase REFTABLE_LOG_DELETION:\n \t\tbreak;\n \tcase REFTABLE_LOG_UPDATE:\n-\t\tif (dst->value.update.email) {\n+\t\tif (dst->value.update.email)\n \t\t\tdst->value.update.email =\n-\t\t\t\txstrdup(dst->value.update.email);\n-\t\t}\n-\t\tif (dst->value.update.name) {\n+\t\t\t\treftable_strdup(dst->value.update.email);\n+\t\tif (dst->value.update.name)\n \t\t\tdst->value.update.name =\n-\t\t\t\txstrdup(dst->value.update.name);\n-\t\t}\n-\t\tif (dst->value.update.message) {\n+\t\t\t\treftable_strdup(dst->value.update.name);\n+\t\tif (dst->value.update.message)\n \t\t\tdst->value.update.message =\n-\t\t\t\txstrdup(dst->value.update.message);\n+\t\t\t\treftable_strdup(dst->value.update.message);\n+\n+\t\tif (!dst->value.update.email ||\n+\t\t !dst->value.update.name ||\n+\t\t !dst->value.update.message) {\n+\t\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto out;\n \t\t}\n \n \t\tmemcpy(dst->value.update.new_hash,\n@@ -681,6 +711,10 @@ static void reftable_log_record_copy_from(void *rec, const void *src_rec,\n \t\t src->value.update.old_hash, hash_size);\n \t\tbreak;\n \t}\n+\n+\tret = 0;\n+out:\n+\treturn ret;\n }\n \n static void reftable_log_record_release_void(void *rec)\n@@ -954,8 +988,8 @@ static void reftable_index_record_key(const void *r, struct strbuf *dest)\n \tstrbuf_addbuf(dest, &rec->last_key);\n }\n \n-static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t int hash_size UNUSED)\n+static int reftable_index_record_copy_from(void *rec, const void *src_rec,\n+\t\t\t\t\t int hash_size UNUSED)\n {\n \tstruct reftable_index_record *dst = rec;\n \tconst struct reftable_index_record *src = src_rec;\n@@ -963,6 +997,8 @@ static void reftable_index_record_copy_from(void *rec, const void *src_rec,\n \tstrbuf_reset(&dst->last_key);\n \tstrbuf_addbuf(&dst->last_key, &src->last_key);\n \tdst->offset = src->offset;\n+\n+\treturn 0;\n }\n \n static void reftable_index_record_release(void *rec)\n@@ -1054,14 +1090,14 @@ int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n \t\t\t\t\t\t dest, hash_size);\n }\n \n-void reftable_record_copy_from(struct reftable_record *rec,\n+int reftable_record_copy_from(struct reftable_record *rec,\n \t\t\t struct reftable_record *src, int hash_size)\n {\n \tassert(src->type == rec->type);\n \n-\treftable_record_vtable(rec)->copy_from(reftable_record_data(rec),\n-\t\t\t\t\t reftable_record_data(src),\n-\t\t\t\t\t hash_size);\n+\treturn reftable_record_vtable(rec)->copy_from(reftable_record_data(rec),\n+\t\t\t\t\t\t reftable_record_data(src),\n+\t\t\t\t\t\t hash_size);\n }\n \n uint8_t reftable_record_val_type(struct reftable_record *rec)\ndiff --git a/reftable/record.h b/reftable/record.h\nindex 5003bacdb0..0f53ba5443 100644\n--- a/reftable/record.h\n+++ b/reftable/record.h\n@@ -44,7 +44,7 @@ struct reftable_record_vtable {\n \t/* The record type of ('r' for ref). */\n \tuint8_t type;\n \n-\tvoid (*copy_from)(void *dest, const void *src, int hash_size);\n+\tint (*copy_from)(void *dest, const void *src, int hash_size);\n \n \t/* a value of [0..7], indicating record subvariants (eg. ref vs. symref\n \t * vs ref deletion) */\n@@ -137,8 +137,8 @@ void reftable_record_init(struct reftable_record *rec, uint8_t typ);\n int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b);\n int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);\n void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);\n-void reftable_record_copy_from(struct reftable_record *rec,\n-\t\t\t struct reftable_record *src, int hash_size);\n+int reftable_record_copy_from(struct reftable_record *rec,\n+\t\t\t struct reftable_record *src, int hash_size);\n uint8_t reftable_record_val_type(struct reftable_record *rec);\n int reftable_record_encode(struct reftable_record *rec, struct string_view dest,\n \t\t\t int hash_size);\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503908","messageId":"ca64971e7b892524e553d20e6726c95dd1865787.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 07/25] reftable/record: handle allocation failures when decoding records","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:55:43Z","receivedAt":"2024-10-02T10:55:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when decoding records. While at it, fix some\nerror codes to be `REFTABLE_FORMAT_ERROR`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/record.c | 80 +++++++++++++++++++++++++++++++++++++----------\n 1 file changed, 63 insertions(+), 17 deletions(-)\n\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 60fd33c9c9..787e134c9a 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -359,7 +359,7 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tuint64_t update_index = 0;\n \tconst char *refname = NULL;\n \tsize_t refname_cap = 0;\n-\tint n;\n+\tint n, err;\n \n \tassert(hash_size > 0);\n \n@@ -375,6 +375,10 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tSWAP(r->refname_cap, refname_cap);\n \n \tREFTABLE_ALLOC_GROW(r->refname, key.len + 1, r->refname_cap);\n+\tif (!r->refname) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \tmemcpy(r->refname, key.buf, key.len);\n \tr->refname[key.len] = 0;\n \n@@ -383,7 +387,8 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tswitch (val_type) {\n \tcase REFTABLE_REF_VAL1:\n \t\tif (in.len < hash_size) {\n-\t\t\treturn -1;\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n \t\t}\n \n \t\tmemcpy(r->value.val1, in.buf, hash_size);\n@@ -392,7 +397,8 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \n \tcase REFTABLE_REF_VAL2:\n \t\tif (in.len < 2 * hash_size) {\n-\t\t\treturn -1;\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n \t\t}\n \n \t\tmemcpy(r->value.val2.value, in.buf, hash_size);\n@@ -405,7 +411,8 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \tcase REFTABLE_REF_SYMREF: {\n \t\tint n = decode_string(scratch, in);\n \t\tif (n < 0) {\n-\t\t\treturn -1;\n+\t\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\t\tgoto done;\n \t\t}\n \t\tstring_view_consume(&in, n);\n \t\tr->value.symref = strbuf_detach(scratch, NULL);\n@@ -419,6 +426,9 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,\n \t}\n \n \treturn start.len - in.len;\n+\n+done:\n+\treturn err;\n }\n \n static int reftable_ref_record_is_deletion_void(const void *p)\n@@ -552,6 +562,8 @@ static int reftable_obj_record_decode(void *rec, struct strbuf key,\n \treftable_obj_record_release(r);\n \n \tREFTABLE_ALLOC_ARRAY(r->hash_prefix, key.len);\n+\tif (!r->hash_prefix)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tmemcpy(r->hash_prefix, key.buf, key.len);\n \tr->hash_prefix_len = key.len;\n \n@@ -570,6 +582,8 @@ static int reftable_obj_record_decode(void *rec, struct strbuf key,\n \t\treturn start.len - in.len;\n \n \tREFTABLE_ALLOC_ARRAY(r->offsets, count);\n+\tif (!r->offsets)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tr->offset_len = count;\n \n \tn = get_var_int(&r->offsets[0], &in);\n@@ -801,12 +815,17 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \tstruct reftable_log_record *r = rec;\n \tuint64_t max = 0;\n \tuint64_t ts = 0;\n-\tint n;\n+\tint err, n;\n \n \tif (key.len <= 9 || key.buf[key.len - 9] != 0)\n \t\treturn REFTABLE_FORMAT_ERROR;\n \n \tREFTABLE_ALLOC_GROW(r->refname, key.len - 8, r->refname_cap);\n+\tif (!r->refname) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tmemcpy(r->refname, key.buf, key.len - 8);\n \tts = get_be64(key.buf + key.len - 8);\n \n@@ -829,8 +848,10 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \tif (val_type == REFTABLE_LOG_DELETION)\n \t\treturn 0;\n \n-\tif (in.len < 2 * hash_size)\n-\t\treturn REFTABLE_FORMAT_ERROR;\n+\tif (in.len < 2 * hash_size) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n+\t\tgoto done;\n+\t}\n \n \tmemcpy(r->value.update.old_hash, in.buf, hash_size);\n \tmemcpy(r->value.update.new_hash, in.buf + hash_size, hash_size);\n@@ -838,8 +859,10 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \tstring_view_consume(&in, 2 * hash_size);\n \n \tn = decode_string(scratch, in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \n \t/*\n@@ -850,52 +873,75 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \t */\n \tif (!r->value.update.name ||\n \t strcmp(r->value.update.name, scratch->buf)) {\n-\t\tr->value.update.name =\n-\t\t\treftable_realloc(r->value.update.name, scratch->len + 1);\n+\t\tchar *name = reftable_realloc(r->value.update.name, scratch->len + 1);\n+\t\tif (!name) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tr->value.update.name = name;\n \t\tmemcpy(r->value.update.name, scratch->buf, scratch->len);\n \t\tr->value.update.name[scratch->len] = 0;\n \t}\n \n \tn = decode_string(scratch, in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \n \t/* Same as above, but for the reflog email. */\n \tif (!r->value.update.email ||\n \t strcmp(r->value.update.email, scratch->buf)) {\n-\t\tr->value.update.email =\n-\t\t\treftable_realloc(r->value.update.email, scratch->len + 1);\n+\t\tchar *email = reftable_realloc(r->value.update.email, scratch->len + 1);\n+\t\tif (!email) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tr->value.update.email = email;\n \t\tmemcpy(r->value.update.email, scratch->buf, scratch->len);\n \t\tr->value.update.email[scratch->len] = 0;\n \t}\n \n \tts = 0;\n \tn = get_var_int(&ts, &in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \tr->value.update.time = ts;\n-\tif (in.len < 2)\n+\tif (in.len < 2) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \n \tr->value.update.tz_offset = get_be16(in.buf);\n \tstring_view_consume(&in, 2);\n \n \tn = decode_string(scratch, in);\n-\tif (n < 0)\n+\tif (n < 0) {\n+\t\terr = REFTABLE_FORMAT_ERROR;\n \t\tgoto done;\n+\t}\n \tstring_view_consume(&in, n);\n \n \tREFTABLE_ALLOC_GROW(r->value.update.message, scratch->len + 1,\n \t\t\t r->value.update.message_cap);\n+\tif (!r->value.update.message) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tmemcpy(r->value.update.message, scratch->buf, scratch->len);\n \tr->value.update.message[scratch->len] = 0;\n \n \treturn start.len - in.len;\n \n done:\n-\treturn REFTABLE_FORMAT_ERROR;\n+\treturn err;\n }\n \n static int null_streq(const char *a, const char *b)\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503909","messageId":"fc2f113cba3542fa8d37fc20d7825b70022f55aa.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 08/25] reftable/writer: handle allocation failures in `writer_index_hash()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:55:45Z","receivedAt":"2024-10-02T10:55:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation errors in `writer_index_hash()`. Adjust its only\ncaller in `reftable_writer_add_ref()` accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/writer.c | 61 +++++++++++++++++++++++++++++------------------\n 1 file changed, 38 insertions(+), 23 deletions(-)\n\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 9d5e6072bc..ed61aaf59c 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -186,18 +186,22 @@ static int obj_index_tree_node_compare(const void *a, const void *b)\n \t\t\t &((const struct obj_index_tree_node *)b)->hash);\n }\n \n-static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n+static int writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n {\n \tuint64_t off = w->next;\n-\n \tstruct obj_index_tree_node want = { .hash = *hash };\n+\tstruct obj_index_tree_node *key;\n+\tstruct tree_node *node;\n \n-\tstruct tree_node *node = tree_search(&want, &w->obj_index_tree,\n-\t\t\t\t\t &obj_index_tree_node_compare, 0);\n-\tstruct obj_index_tree_node *key = NULL;\n+\tnode = tree_search(&want, &w->obj_index_tree,\n+\t\t\t &obj_index_tree_node_compare, 0);\n \tif (!node) {\n \t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n-\t\tkey = reftable_malloc(sizeof(struct obj_index_tree_node));\n+\n+\t\tkey = reftable_malloc(sizeof(*key));\n+\t\tif (!key)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \t\t*key = empty;\n \n \t\tstrbuf_reset(&key->hash);\n@@ -208,12 +212,15 @@ static void writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n \t\tkey = node->key;\n \t}\n \n-\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off) {\n-\t\treturn;\n-\t}\n+\tif (key->offset_len > 0 && key->offsets[key->offset_len - 1] == off)\n+\t\treturn 0;\n \n \tREFTABLE_ALLOC_GROW(key->offsets, key->offset_len + 1, key->offset_cap);\n+\tif (!key->offsets)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tkey->offsets[key->offset_len++] = off;\n+\n+\treturn 0;\n }\n \n static int writer_add_record(struct reftable_writer *w,\n@@ -284,11 +291,11 @@ int reftable_writer_add_ref(struct reftable_writer *w,\n \t\t\t.ref = *ref\n \t\t},\n \t};\n-\tint err = 0;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint err;\n \n-\tif (!ref->refname)\n-\t\treturn REFTABLE_API_ERROR;\n-\tif (ref->update_index < w->min_update_index ||\n+\tif (!ref->refname ||\n+\t ref->update_index < w->min_update_index ||\n \t ref->update_index > w->max_update_index)\n \t\treturn REFTABLE_API_ERROR;\n \n@@ -296,24 +303,32 @@ int reftable_writer_add_ref(struct reftable_writer *w,\n \n \terr = writer_add_record(w, &rec);\n \tif (err < 0)\n-\t\treturn err;\n+\t\tgoto out;\n \n \tif (!w->opts.skip_index_objects && reftable_ref_record_val1(ref)) {\n-\t\tstruct strbuf h = STRBUF_INIT;\n-\t\tstrbuf_add(&h, (char *)reftable_ref_record_val1(ref),\n+\t\tstrbuf_add(&buf, (char *)reftable_ref_record_val1(ref),\n \t\t\t hash_size(w->opts.hash_id));\n-\t\twriter_index_hash(w, &h);\n-\t\tstrbuf_release(&h);\n+\n+\t\terr = writer_index_hash(w, &buf);\n+\t\tif (err < 0)\n+\t\t\tgoto out;\n \t}\n \n \tif (!w->opts.skip_index_objects && reftable_ref_record_val2(ref)) {\n-\t\tstruct strbuf h = STRBUF_INIT;\n-\t\tstrbuf_add(&h, reftable_ref_record_val2(ref),\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_add(&buf, reftable_ref_record_val2(ref),\n \t\t\t hash_size(w->opts.hash_id));\n-\t\twriter_index_hash(w, &h);\n-\t\tstrbuf_release(&h);\n+\n+\t\terr = writer_index_hash(w, &buf);\n+\t\tif (err < 0)\n+\t\t\tgoto out;\n \t}\n-\treturn 0;\n+\n+\terr = 0;\n+\n+out:\n+\tstrbuf_release(&buf);\n+\treturn err;\n }\n \n int reftable_writer_add_refs(struct reftable_writer *w,\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503910","messageId":"0ed99e0bdf19fa1e396312b79acf47e9ba8e6ae5.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 09/25] reftable/writer: handle allocation failures in `reftable_new_writer()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:55:48Z","receivedAt":"2024-10-02T10:55:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_new_writer()`. Adapt the\nfunction to return an error code to return such failures. While at it,\nrename it to match our code style as we have to touch up every callsite\nanyway.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/reftable-writer.h | 12 +++++++-----\n reftable/stack.c | 14 ++++++++++----\n reftable/writer.c | 22 ++++++++++++++++------\n t/unit-tests/lib-reftable.c | 8 +++++---\n 4 files changed, 38 insertions(+), 18 deletions(-)\n\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex 189b1f4144..43623dc7c3 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -90,11 +90,13 @@ struct reftable_stats {\n \tint object_id_len;\n };\n \n-/* reftable_new_writer creates a new writer */\n-struct reftable_writer *\n-reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n-\t\t int (*flush_func)(void *),\n-\t\t void *writer_arg, const struct reftable_write_options *opts);\n+struct reftable_writer;\n+\n+/* Create a new writer. */\n+int reftable_writer_new(struct reftable_writer **out,\n+\t\t\tssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t\tint (*flush_func)(void *),\n+\t\t\tvoid *writer_arg, const struct reftable_write_options *opts);\n \n /* Set the range of update indices for the records we will add. When writing a\n table into a stack, the min should be at least\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 498fae846d..ea21ca6e5f 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -808,8 +808,11 @@ int reftable_addition_add(struct reftable_addition *add,\n \t}\n \ttab_fd = get_tempfile_fd(tab_file);\n \n-\twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, &tab_fd,\n-\t\t\t\t &add->stack->opts);\n+\terr = reftable_writer_new(&wr, reftable_fd_write, reftable_fd_flush,\n+\t\t\t\t &tab_fd, &add->stack->opts);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = write_table(wr, arg);\n \tif (err < 0)\n \t\tgoto done;\n@@ -898,8 +901,11 @@ static int stack_compact_locked(struct reftable_stack *st,\n \t\tgoto done;\n \t}\n \n-\twr = reftable_new_writer(reftable_fd_write, reftable_fd_flush,\n-\t\t\t\t &tab_fd, &st->opts);\n+\terr = reftable_writer_new(&wr, reftable_fd_write, reftable_fd_flush,\n+\t\t\t\t &tab_fd, &st->opts);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = stack_write_compact(st, wr, first, last, config);\n \tif (err < 0)\n \t\tgoto done;\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex ed61aaf59c..8ab2e916d3 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -117,13 +117,17 @@ static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n \tw->block_writer->restart_interval = w->opts.restart_interval;\n }\n \n-struct reftable_writer *\n-reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n-\t\t int (*flush_func)(void *),\n-\t\t void *writer_arg, const struct reftable_write_options *_opts)\n+int reftable_writer_new(struct reftable_writer **out,\n+\t\t\tssize_t (*writer_func)(void *, const void *, size_t),\n+\t\t\tint (*flush_func)(void *),\n+\t\t\tvoid *writer_arg, const struct reftable_write_options *_opts)\n {\n-\tstruct reftable_writer *wp = reftable_calloc(1, sizeof(*wp));\n \tstruct reftable_write_options opts = {0};\n+\tstruct reftable_writer *wp;\n+\n+\twp = reftable_calloc(1, sizeof(*wp));\n+\tif (!wp)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \n \tif (_opts)\n \t\topts = *_opts;\n@@ -134,13 +138,19 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t),\n \tstrbuf_init(&wp->block_writer_data.last_key, 0);\n \tstrbuf_init(&wp->last_key, 0);\n \tREFTABLE_CALLOC_ARRAY(wp->block, opts.block_size);\n+\tif (!wp->block) {\n+\t\treftable_free(wp);\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t}\n \twp->write = writer_func;\n \twp->write_arg = writer_arg;\n \twp->opts = opts;\n \twp->flush = flush_func;\n \twriter_reinit_block_writer(wp, BLOCK_TYPE_REF);\n \n-\treturn wp;\n+\t*out = wp;\n+\n+\treturn 0;\n }\n \n void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,\ndiff --git a/t/unit-tests/lib-reftable.c b/t/unit-tests/lib-reftable.c\nindex ab1fa44a28..54c26c43e7 100644\n--- a/t/unit-tests/lib-reftable.c\n+++ b/t/unit-tests/lib-reftable.c\n@@ -22,9 +22,11 @@ static int strbuf_writer_flush(void *arg UNUSED)\n struct reftable_writer *t_reftable_strbuf_writer(struct strbuf *buf,\n \t\t\t\t\t\t struct reftable_write_options *opts)\n {\n-\treturn reftable_new_writer(&strbuf_writer_write,\n-\t\t\t\t &strbuf_writer_flush,\n-\t\t\t\t buf, opts);\n+\tstruct reftable_writer *writer;\n+\tint ret = reftable_writer_new(&writer, &strbuf_writer_write, &strbuf_writer_flush,\n+\t\t\t\t buf, opts);\n+\tcheck(!ret);\n+\treturn writer;\n }\n \n void t_reftable_write_to_buf(struct strbuf *buf,\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503911","messageId":"8dfbfd9286c730fc63d40c6d095876fb1317cdcf.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 10/25] reftable/merged: handle allocation failures in `merged_table_init_iter()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:55:54Z","receivedAt":"2024-10-02T10:55:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `merged_table_init_iter()`. While at it,\nmerge `merged_iter_init()` into the function. It only has a single\ncaller and merging them makes it easier to handle allocation failures\nconsistently.\n\nThis change also requires us to adapt `reftable_stack_init_*_iterator()`\nto bubble up the new error codes of `merged_table_iter_init()`. Adapt\ncallsites accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 39 +++++++++++++----\n reftable/merged.c | 74 ++++++++++++++++++++------------\n reftable/merged.h | 6 +--\n reftable/reftable-merged.h | 8 ++--\n reftable/reftable-stack.h | 8 ++--\n reftable/stack.c | 34 ++++++++++-----\n t/helper/test-reftable.c | 10 ++++-\n t/unit-tests/t-reftable-merged.c | 12 ++++--\n t/unit-tests/t-reftable-stack.c | 4 +-\n 9 files changed, 131 insertions(+), 64 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 5c241097a4..9c08c46c85 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1299,7 +1299,9 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t\t\tstruct reftable_log_record log = {0};\n \t\t\tstruct reftable_iterator it = {0};\n \n-\t\t\treftable_stack_init_log_iterator(arg->stack, &it);\n+\t\t\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\t\t\tif (ret < 0)\n+\t\t\t\tgoto done;\n \n \t\t\t/*\n \t\t\t * When deleting refs we also delete all reflog entries\n@@ -1669,7 +1671,10 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \t * copy over all log entries from the old reflog. Last but not least,\n \t * when renaming we also have to delete all the old reflog entries.\n \t */\n-\treftable_stack_init_log_iterator(arg->stack, &it);\n+\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, arg->oldname);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1890,7 +1895,10 @@ static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftabl\n \tif (ret < 0)\n \t\tgoto done;\n \n-\treftable_stack_init_log_iterator(stack, &iter->iter);\n+\tret = reftable_stack_init_log_iterator(stack, &iter->iter);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&iter->iter, \"\");\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1957,7 +1965,10 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, refname);\n \twhile (!ret) {\n \t\tret = reftable_iterator_next_log(&it, &log);\n@@ -1973,6 +1984,7 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n \t\t\tbreak;\n \t}\n \n+done:\n \treftable_log_record_release(&log);\n \treftable_iterator_destroy(&it);\n \treturn ret;\n@@ -1994,7 +2006,10 @@ static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store,\n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, refname);\n \twhile (!ret) {\n \t\tstruct reftable_log_record log = {0};\n@@ -2044,7 +2059,10 @@ static int reftable_be_reflog_exists(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \tret = reftable_iterator_seek_log(&it, refname);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -2150,7 +2168,9 @@ static int write_reflog_delete_table(struct reftable_writer *writer, void *cb_da\n \n \treftable_writer_set_limits(writer, ts, ts);\n \n-\treftable_stack_init_log_iterator(arg->stack, &it);\n+\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\tif (ret < 0)\n+\t\tgoto out;\n \n \t/*\n \t * In order to delete a table we need to delete all reflog entries one\n@@ -2174,6 +2194,7 @@ static int write_reflog_delete_table(struct reftable_writer *writer, void *cb_da\n \t\tret = reftable_writer_add_log(writer, &tombstone);\n \t}\n \n+out:\n \treftable_log_record_release(&log);\n \treftable_iterator_destroy(&it);\n \treturn ret;\n@@ -2312,7 +2333,9 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\treftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tif (ret < 0)\n+\t\tgoto done;\n \n \tret = reftable_iterator_seek_log(&it, refname);\n \tif (ret < 0)\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 128a810c55..de4f81abaf 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -30,22 +30,6 @@ struct merged_iter {\n \tssize_t advance_index;\n };\n \n-static void merged_iter_init(struct merged_iter *mi,\n-\t\t\t struct reftable_merged_table *mt,\n-\t\t\t uint8_t typ)\n-{\n-\tmemset(mi, 0, sizeof(*mi));\n-\tmi->advance_index = -1;\n-\tmi->suppress_deletions = mt->suppress_deletions;\n-\n-\tREFTABLE_CALLOC_ARRAY(mi->subiters, mt->readers_len);\n-\tfor (size_t i = 0; i < mt->readers_len; i++) {\n-\t\treftable_record_init(&mi->subiters[i].rec, typ);\n-\t\treader_init_iter(mt->readers[i], &mi->subiters[i].iter, typ);\n-\t}\n-\tmi->subiters_len = mt->readers_len;\n-}\n-\n static void merged_iter_close(void *p)\n {\n \tstruct merged_iter *mi = p;\n@@ -244,25 +228,61 @@ reftable_merged_table_min_update_index(struct reftable_merged_table *mt)\n \treturn mt->min;\n }\n \n-void merged_table_init_iter(struct reftable_merged_table *mt,\n-\t\t\t struct reftable_iterator *it,\n-\t\t\t uint8_t typ)\n+int merged_table_init_iter(struct reftable_merged_table *mt,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t uint8_t typ)\n {\n-\tstruct merged_iter *mi = reftable_malloc(sizeof(*mi));\n-\tmerged_iter_init(mi, mt, typ);\n+\tstruct merged_subiter *subiters;\n+\tstruct merged_iter *mi = NULL;\n+\tint ret;\n+\n+\tREFTABLE_CALLOC_ARRAY(subiters, mt->readers_len);\n+\tif (!subiters) {\n+\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n+\tfor (size_t i = 0; i < mt->readers_len; i++) {\n+\t\treftable_record_init(&subiters[i].rec, typ);\n+\t\treader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n+\t}\n+\n+\tREFTABLE_CALLOC_ARRAY(mi, 1);\n+\tif (!mi) {\n+\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\tmi->advance_index = -1;\n+\tmi->suppress_deletions = mt->suppress_deletions;\n+\tmi->subiters = subiters;\n+\tmi->subiters_len = mt->readers_len;\n+\n \titerator_from_merged_iter(it, mi);\n+\tret = 0;\n+\n+out:\n+\tif (ret < 0) {\n+\t\tfor (size_t i = 0; subiters && i < mt->readers_len; i++) {\n+\t\t\treftable_iterator_destroy(&subiters[i].iter);\n+\t\t\treftable_record_release(&subiters[i].rec);\n+\t\t}\n+\t\treftable_free(subiters);\n+\t\treftable_free(mi);\n+\t}\n+\n+\treturn ret;\n }\n \n-void reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it)\n+int reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(mt, it, BLOCK_TYPE_REF);\n+\treturn merged_table_init_iter(mt, it, BLOCK_TYPE_REF);\n }\n \n-void reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it)\n+int reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(mt, it, BLOCK_TYPE_LOG);\n+\treturn merged_table_init_iter(mt, it, BLOCK_TYPE_LOG);\n }\n \n uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt)\ndiff --git a/reftable/merged.h b/reftable/merged.h\nindex de5fd33f01..89bd0c4b35 100644\n--- a/reftable/merged.h\n+++ b/reftable/merged.h\n@@ -26,8 +26,8 @@ struct reftable_merged_table {\n \n struct reftable_iterator;\n \n-void merged_table_init_iter(struct reftable_merged_table *mt,\n-\t\t\t struct reftable_iterator *it,\n-\t\t\t uint8_t typ);\n+int merged_table_init_iter(struct reftable_merged_table *mt,\n+\t\t\t struct reftable_iterator *it,\n+\t\t\t uint8_t typ);\n \n #endif\ndiff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h\nindex 16d19f8df2..a970d5dd89 100644\n--- a/reftable/reftable-merged.h\n+++ b/reftable/reftable-merged.h\n@@ -37,12 +37,12 @@ int reftable_merged_table_new(struct reftable_merged_table **dest,\n \t\t\t uint32_t hash_id);\n \n /* Initialize a merged table iterator for reading refs. */\n-void reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it);\n+int reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it);\n \n /* Initialize a merged table iterator for reading logs. */\n-void reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n-\t\t\t\t\t struct reftable_iterator *it);\n+int reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt,\n+\t\t\t\t\t struct reftable_iterator *it);\n \n /* returns the max update_index covered by this merged table. */\n uint64_t\ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nindex f4f8cabc7f..e958f911b4 100644\n--- a/reftable/reftable-stack.h\n+++ b/reftable/reftable-stack.h\n@@ -73,16 +73,16 @@ struct reftable_iterator;\n * be used to iterate through refs. The iterator is valid until the next reload\n * or write.\n */\n-void reftable_stack_init_ref_iterator(struct reftable_stack *st,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_stack_init_ref_iterator(struct reftable_stack *st,\n+\t\t\t\t struct reftable_iterator *it);\n \n /*\n * Initialize an iterator for the merged tables contained in the stack that can\n * be used to iterate through logs. The iterator is valid until the next reload\n * or write.\n */\n-void reftable_stack_init_log_iterator(struct reftable_stack *st,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_stack_init_log_iterator(struct reftable_stack *st,\n+\t\t\t\t struct reftable_iterator *it);\n \n /* returns the merged_table for seeking. This table is valid until the\n * next write or reload, and should not be closed or deleted.\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex ea21ca6e5f..bb4d230918 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -136,18 +136,18 @@ int read_lines(const char *filename, char ***namesp)\n \treturn err;\n }\n \n-void reftable_stack_init_ref_iterator(struct reftable_stack *st,\n+int reftable_stack_init_ref_iterator(struct reftable_stack *st,\n \t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(reftable_stack_merged_table(st),\n-\t\t\t it, BLOCK_TYPE_REF);\n+\treturn merged_table_init_iter(reftable_stack_merged_table(st),\n+\t\t\t\t it, BLOCK_TYPE_REF);\n }\n \n-void reftable_stack_init_log_iterator(struct reftable_stack *st,\n-\t\t\t\t struct reftable_iterator *it)\n+int reftable_stack_init_log_iterator(struct reftable_stack *st,\n+\t\t\t\t struct reftable_iterator *it)\n {\n-\tmerged_table_init_iter(reftable_stack_merged_table(st),\n-\t\t\t it, BLOCK_TYPE_LOG);\n+\treturn merged_table_init_iter(reftable_stack_merged_table(st),\n+\t\t\t\t it, BLOCK_TYPE_LOG);\n }\n \n struct reftable_merged_table *\n@@ -952,7 +952,10 @@ static int stack_write_compact(struct reftable_stack *st,\n \tif (err < 0)\n \t\tgoto done;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tif (err < 0)\n \t\tgoto done;\n@@ -977,7 +980,10 @@ static int stack_write_compact(struct reftable_stack *st,\n \t}\n \treftable_iterator_destroy(&it);\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tif (err < 0)\n \t\tgoto done;\n@@ -1496,7 +1502,10 @@ int reftable_stack_read_ref(struct reftable_stack *st, const char *refname,\n \tstruct reftable_iterator it = { 0 };\n \tint ret;\n \n-\treftable_merged_table_init_ref_iterator(st->merged, &it);\n+\tret = reftable_merged_table_init_ref_iterator(st->merged, &it);\n+\tif (ret)\n+\t\tgoto out;\n+\n \tret = reftable_iterator_seek_ref(&it, refname);\n \tif (ret)\n \t\tgoto out;\n@@ -1523,7 +1532,10 @@ int reftable_stack_read_log(struct reftable_stack *st, const char *refname,\n \tstruct reftable_iterator it = {0};\n \tint err;\n \n-\treftable_stack_init_log_iterator(st, &it);\n+\terr = reftable_stack_init_log_iterator(st, &it);\n+\tif (err)\n+\t\tgoto done;\n+\n \terr = reftable_iterator_seek_log(&it, refname);\n \tif (err)\n \t\tgoto done;\ndiff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c\nindex 29d4e9a755..5c8849d115 100644\n--- a/t/helper/test-reftable.c\n+++ b/t/helper/test-reftable.c\n@@ -28,7 +28,10 @@ static int dump_table(struct reftable_merged_table *mt)\n \tconst struct git_hash_algo *algop;\n \tint err;\n \n-\treftable_merged_table_init_ref_iterator(mt, &it);\n+\terr = reftable_merged_table_init_ref_iterator(mt, &it);\n+\tif (err < 0)\n+\t\treturn err;\n+\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tif (err < 0)\n \t\treturn err;\n@@ -63,7 +66,10 @@ static int dump_table(struct reftable_merged_table *mt)\n \treftable_iterator_destroy(&it);\n \treftable_ref_record_release(&ref);\n \n-\treftable_merged_table_init_log_iterator(mt, &it);\n+\terr = reftable_merged_table_init_log_iterator(mt, &it);\n+\tif (err < 0)\n+\t\treturn err;\n+\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tif (err < 0)\n \t\treturn err;\ndiff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c\nindex 19e54bdfb8..3d2848632d 100644\n--- a/t/unit-tests/t-reftable-merged.c\n+++ b/t/unit-tests/t-reftable-merged.c\n@@ -82,7 +82,8 @@ static void t_merged_single_record(void)\n \tstruct reftable_iterator it = { 0 };\n \tint err;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"a\");\n \tcheck(!err);\n \n@@ -161,7 +162,8 @@ static void t_merged_refs(void)\n \tsize_t cap = 0;\n \tsize_t i;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"a\");\n \tcheck(!err);\n \tcheck_int(reftable_merged_table_hash_id(mt), ==, GIT_SHA1_FORMAT_ID);\n@@ -367,7 +369,8 @@ static void t_merged_logs(void)\n \tsize_t cap = 0;\n \tsize_t i;\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"a\");\n \tcheck(!err);\n \tcheck_int(reftable_merged_table_hash_id(mt), ==, GIT_SHA1_FORMAT_ID);\n@@ -390,7 +393,8 @@ static void t_merged_logs(void)\n \t\tcheck(reftable_log_record_equal(want[i], &out[i],\n \t\t\t\t\t\t GIT_SHA1_RAWSZ));\n \n-\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\terr = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log_at(&it, \"a\", 2);\n \tcheck(!err);\n \treftable_log_record_release(&out[0]);\ndiff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c\nindex 65e513d5ec..6e41a4cf7e 100644\n--- a/t/unit-tests/t-reftable-stack.c\n+++ b/t/unit-tests/t-reftable-stack.c\n@@ -599,7 +599,9 @@ static void t_reftable_stack_iterator(void)\n \n \treftable_iterator_destroy(&it);\n \n-\treftable_stack_init_log_iterator(st, &it);\n+\terr = reftable_stack_init_log_iterator(st, &it);\n+\tcheck(!err);\n+\n \treftable_iterator_seek_log(&it, logs[0].refname);\n \tfor (i = 0; ; i++) {\n \t\tstruct reftable_log_record log = { 0 };\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503912","messageId":"7b592a6c6b8b08a7e8089364abfc6d2d6f6e83e4.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 11/25] reftable/reader: handle allocation failures for unindexed reader","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:55:56Z","receivedAt":"2024-10-02T10:56:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when creating unindexed readers.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/reader.c | 23 +++++++++++++++++------\n 1 file changed, 17 insertions(+), 6 deletions(-)\n\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 6494ce2e32..485ee085da 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -734,19 +734,30 @@ static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n \tREFTABLE_ALLOC_ARRAY(ti, 1);\n \ttable_iter_init(ti, r);\n \terr = table_iter_seek_start(ti, BLOCK_TYPE_REF, 0);\n-\tif (err < 0) {\n-\t\treftable_free(ti);\n-\t\treturn err;\n-\t}\n+\tif (err < 0)\n+\t\tgoto out;\n \n-\tfilter = reftable_malloc(sizeof(struct filtering_ref_iterator));\n+\tfilter = reftable_malloc(sizeof(*filter));\n+\tif (!filter) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n \t*filter = empty;\n \n \tstrbuf_add(&filter->oid, oid, oid_len);\n \titerator_from_table_iter(&filter->it, ti);\n \n \titerator_from_filtering_ref_iterator(it, filter);\n-\treturn 0;\n+\n+\terr = 0;\n+\n+out:\n+\tif (err < 0) {\n+\t\tif (ti)\n+\t\t\ttable_iter_close(ti);\n+\t\treftable_free(ti);\n+\t}\n+\treturn err;\n }\n \n int reftable_reader_refs_for(struct reftable_reader *r,\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503913","messageId":"ceed83826537cca26cc04d237efb298343dcde14.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 12/25] reftable/reader: handle allocation failures in `reader_init_iter()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:55:59Z","receivedAt":"2024-10-02T10:56:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reader_init_iter()`. This requires us to\nalso adapt `reftable_reader_init_*_iterator()` to bubble up the new\nerror codes. Adapt callers accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 4 +++-\n reftable/reader.c | 28 +++++++++++++++---------\n reftable/reader.h | 6 ++---\n reftable/reftable-reader.h | 8 +++----\n t/unit-tests/t-reftable-readwrite.c | 34 +++++++++++++++++++----------\n 5 files changed, 50 insertions(+), 30 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex de4f81abaf..69790c345c 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -244,7 +244,9 @@ int merged_table_init_iter(struct reftable_merged_table *mt,\n \n \tfor (size_t i = 0; i < mt->readers_len; i++) {\n \t\treftable_record_init(&subiters[i].rec, typ);\n-\t\treader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n+\t\tret = reader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n+\t\tif (ret < 0)\n+\t\t\tgoto out;\n \t}\n \n \tREFTABLE_CALLOC_ARRAY(mi, 1);\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 485ee085da..f696e992df 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -554,32 +554,37 @@ static void iterator_from_table_iter(struct reftable_iterator *it,\n \tit->ops = &table_iter_vtable;\n }\n \n-void reader_init_iter(struct reftable_reader *r,\n-\t\t struct reftable_iterator *it,\n-\t\t uint8_t typ)\n+int reader_init_iter(struct reftable_reader *r,\n+\t\t struct reftable_iterator *it,\n+\t\t uint8_t typ)\n {\n \tstruct reftable_reader_offsets *offs = reader_offsets_for(r, typ);\n \n \tif (offs->is_present) {\n \t\tstruct table_iter *ti;\n \t\tREFTABLE_ALLOC_ARRAY(ti, 1);\n+\t\tif (!ti)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \t\ttable_iter_init(ti, r);\n \t\titerator_from_table_iter(it, ti);\n \t} else {\n \t\titerator_set_empty(it);\n \t}\n+\n+\treturn 0;\n }\n \n-void reftable_reader_init_ref_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it)\n+int reftable_reader_init_ref_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it)\n {\n-\treader_init_iter(r, it, BLOCK_TYPE_REF);\n+\treturn reader_init_iter(r, it, BLOCK_TYPE_REF);\n }\n \n-void reftable_reader_init_log_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it)\n+int reftable_reader_init_log_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it)\n {\n-\treader_init_iter(r, it, BLOCK_TYPE_LOG);\n+\treturn reader_init_iter(r, it, BLOCK_TYPE_LOG);\n }\n \n int reftable_reader_new(struct reftable_reader **out,\n@@ -689,7 +694,10 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n \tstruct indexed_table_ref_iter *itr = NULL;\n \n \t/* Look through the reverse index. */\n-\treader_init_iter(r, &oit, BLOCK_TYPE_OBJ);\n+\terr = reader_init_iter(r, &oit, BLOCK_TYPE_OBJ);\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = iterator_seek(&oit, &want);\n \tif (err != 0)\n \t\tgoto done;\ndiff --git a/reftable/reader.h b/reftable/reader.h\nindex 3710ee09b4..02d10c5d37 100644\n--- a/reftable/reader.h\n+++ b/reftable/reader.h\n@@ -56,9 +56,9 @@ struct reftable_reader {\n \n const char *reader_name(struct reftable_reader *r);\n \n-void reader_init_iter(struct reftable_reader *r,\n-\t\t struct reftable_iterator *it,\n-\t\t uint8_t typ);\n+int reader_init_iter(struct reftable_reader *r,\n+\t\t struct reftable_iterator *it,\n+\t\t uint8_t typ);\n \n /* initialize a block reader to read from `r` */\n int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,\ndiff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h\nindex a600452b56..6a2d0b693f 100644\n--- a/reftable/reftable-reader.h\n+++ b/reftable/reftable-reader.h\n@@ -46,12 +46,12 @@ void reftable_reader_incref(struct reftable_reader *reader);\n void reftable_reader_decref(struct reftable_reader *reader);\n \n /* Initialize a reftable iterator for reading refs. */\n-void reftable_reader_init_ref_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_reader_init_ref_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it);\n \n /* Initialize a reftable iterator for reading logs. */\n-void reftable_reader_init_log_iterator(struct reftable_reader *r,\n-\t\t\t\t struct reftable_iterator *it);\n+int reftable_reader_init_log_iterator(struct reftable_reader *r,\n+\t\t\t\t struct reftable_iterator *it);\n \n /* returns the hash ID used in this table. */\n uint32_t reftable_reader_hash_id(struct reftable_reader *r);\ndiff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c\nindex e1b235a5f1..acca927a2c 100644\n--- a/t/unit-tests/t-reftable-readwrite.c\n+++ b/t/unit-tests/t-reftable-readwrite.c\n@@ -206,7 +206,8 @@ static void t_log_write_read(void)\n \terr = reftable_reader_new(&reader, &source, \"file.log\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \n \terr = reftable_iterator_seek_ref(&it, names[N - 1]);\n \tcheck(!err);\n@@ -221,8 +222,8 @@ static void t_log_write_read(void)\n \treftable_iterator_destroy(&it);\n \treftable_ref_record_release(&ref);\n \n-\treftable_reader_init_log_iterator(reader, &it);\n-\n+\terr = reftable_reader_init_log_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tcheck(!err);\n \n@@ -296,7 +297,8 @@ static void t_log_zlib_corruption(void)\n \terr = reftable_reader_new(&reader, &source, \"file.log\");\n \tcheck(!err);\n \n-\treftable_reader_init_log_iterator(reader, &it);\n+\terr = reftable_reader_init_log_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"refname\");\n \tcheck_int(err, ==, REFTABLE_ZLIB_ERROR);\n \n@@ -325,7 +327,8 @@ static void t_table_read_write_sequential(void)\n \terr = reftable_reader_new(&reader, &source, \"file.ref\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tcheck(!err);\n \n@@ -376,7 +379,8 @@ static void t_table_read_api(void)\n \terr = reftable_reader_new(&reader, &source, \"file.ref\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, names[0]);\n \tcheck(!err);\n \n@@ -419,7 +423,8 @@ static void t_table_read_write_seek(int index, int hash_id)\n \t}\n \n \tfor (i = 1; i < N; i++) {\n-\t\treftable_reader_init_ref_iterator(reader, &it);\n+\t\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\t\tcheck(!err);\n \t\terr = reftable_iterator_seek_ref(&it, names[i]);\n \t\tcheck(!err);\n \t\terr = reftable_iterator_next_ref(&it, &ref);\n@@ -435,7 +440,8 @@ static void t_table_read_write_seek(int index, int hash_id)\n \tstrbuf_addstr(&pastLast, names[N - 1]);\n \tstrbuf_addstr(&pastLast, \"/\");\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, pastLast.buf);\n \tif (err == 0) {\n \t\tstruct reftable_ref_record ref = { 0 };\n@@ -534,7 +540,8 @@ static void t_table_refs_for(int indexed)\n \tif (!indexed)\n \t\treader->obj_offsets.is_present = 0;\n \n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tcheck(!err);\n \treftable_iterator_destroy(&it);\n@@ -593,7 +600,8 @@ static void t_write_empty_table(void)\n \terr = reftable_reader_new(&rd, &source, \"filename\");\n \tcheck(!err);\n \n-\treftable_reader_init_ref_iterator(rd, &it);\n+\terr = reftable_reader_init_ref_iterator(rd, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"\");\n \tcheck(!err);\n \n@@ -802,7 +810,8 @@ static void t_write_multiple_indices(void)\n \t * Seeking the log uses the log index now. In case there is any\n \t * confusion regarding indices we would notice here.\n \t */\n-\treftable_reader_init_log_iterator(reader, &it);\n+\terr = reftable_reader_init_log_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_log(&it, \"\");\n \tcheck(!err);\n \n@@ -858,7 +867,8 @@ static void t_write_multi_level_index(void)\n \t/*\n \t * Seeking the last ref should work as expected.\n \t */\n-\treftable_reader_init_ref_iterator(reader, &it);\n+\terr = reftable_reader_init_ref_iterator(reader, &it);\n+\tcheck(!err);\n \terr = reftable_iterator_seek_ref(&it, \"refs/heads/199\");\n \tcheck(!err);\n \n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503914","messageId":"1e997a576693a9ae1841b14ef7f2a02841694bfb.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 13/25] reftable/stack: handle allocation failures on reload","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:56:01Z","receivedAt":"2024-10-02T10:56:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_stack_reload_once()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 34 +++++++++++++++++++++++++---------\n 1 file changed, 25 insertions(+), 9 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex bb4d230918..060b2c1b90 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -214,13 +214,13 @@ void reftable_stack_destroy(struct reftable_stack *st)\n }\n \n static struct reftable_reader **stack_copy_readers(struct reftable_stack *st,\n-\t\t\t\t\t\t int cur_len)\n+\t\t\t\t\t\t size_t cur_len)\n {\n \tstruct reftable_reader **cur = reftable_calloc(cur_len, sizeof(*cur));\n-\tint i = 0;\n-\tfor (i = 0; i < cur_len; i++) {\n+\tif (!cur)\n+\t\treturn NULL;\n+\tfor (size_t i = 0; i < cur_len; i++)\n \t\tcur[i] = st->readers[i];\n-\t}\n \treturn cur;\n }\n \n@@ -229,18 +229,30 @@ static int reftable_stack_reload_once(struct reftable_stack *st,\n \t\t\t\t int reuse_open)\n {\n \tsize_t cur_len = !st->merged ? 0 : st->merged->readers_len;\n-\tstruct reftable_reader **cur = stack_copy_readers(st, cur_len);\n+\tstruct reftable_reader **cur;\n \tstruct reftable_reader **reused = NULL;\n-\tsize_t reused_len = 0, reused_alloc = 0;\n-\tsize_t names_len = names_length(names);\n-\tstruct reftable_reader **new_readers =\n-\t\treftable_calloc(names_len, sizeof(*new_readers));\n+\tstruct reftable_reader **new_readers;\n+\tsize_t reused_len = 0, reused_alloc = 0, names_len;\n \tsize_t new_readers_len = 0;\n \tstruct reftable_merged_table *new_merged = NULL;\n \tstruct strbuf table_path = STRBUF_INIT;\n \tint err = 0;\n \tsize_t i;\n \n+\tcur = stack_copy_readers(st, cur_len);\n+\tif (!cur) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tnames_len = names_length(names);\n+\n+\tnew_readers = reftable_calloc(names_len, sizeof(*new_readers));\n+\tif (!new_readers) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \twhile (*names) {\n \t\tstruct reftable_reader *rd = NULL;\n \t\tconst char *name = *names++;\n@@ -261,6 +273,10 @@ static int reftable_stack_reload_once(struct reftable_stack *st,\n \t\t\t\t * do by bumping their refcount.\n \t\t\t\t */\n \t\t\t\tREFTABLE_ALLOC_GROW(reused, reused_len + 1, reused_alloc);\n+\t\t\t\tif (!reused) {\n+\t\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n \t\t\t\treused[reused_len++] = rd;\n \t\t\t\treftable_reader_incref(rd);\n \t\t\t\tbreak;\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503915","messageId":"40d4d81378dbeffa0a916aca7d747ee9a045e045.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 14/25] reftable/stack: handle allocation failures in `reftable_new_stack()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:56:04Z","receivedAt":"2024-10-02T10:56:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_new_stack()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 34 ++++++++++++++++++++++++++--------\n 1 file changed, 26 insertions(+), 8 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 060b2c1b90..1b77c9d014 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -56,10 +56,16 @@ static int reftable_fd_flush(void *arg)\n int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \t\t const struct reftable_write_options *_opts)\n {\n-\tstruct reftable_stack *p = reftable_calloc(1, sizeof(*p));\n \tstruct strbuf list_file_name = STRBUF_INIT;\n-\tstruct reftable_write_options opts = {0};\n-\tint err = 0;\n+\tstruct reftable_write_options opts = { 0 };\n+\tstruct reftable_stack *p;\n+\tint err;\n+\n+\tp = reftable_calloc(1, sizeof(*p));\n+\tif (!p) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n \n \tif (_opts)\n \t\topts = *_opts;\n@@ -74,15 +80,23 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \n \tp->list_file = strbuf_detach(&list_file_name, NULL);\n \tp->list_fd = -1;\n-\tp->reftable_dir = xstrdup(dir);\n \tp->opts = opts;\n+\tp->reftable_dir = reftable_strdup(dir);\n+\tif (!p->reftable_dir) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n \n \terr = reftable_stack_reload_maybe_reuse(p, 1);\n-\tif (err < 0) {\n+\tif (err < 0)\n+\t\tgoto out;\n+\n+\t*dest = p;\n+\terr = 0;\n+\n+out:\n+\tif (err < 0)\n \t\treftable_stack_destroy(p);\n-\t} else {\n-\t\t*dest = p;\n-\t}\n \treturn err;\n }\n \n@@ -171,6 +185,10 @@ void reftable_stack_destroy(struct reftable_stack *st)\n {\n \tchar **names = NULL;\n \tint err = 0;\n+\n+\tif (!st)\n+\t\treturn;\n+\n \tif (st->merged) {\n \t\treftable_merged_table_free(st->merged);\n \t\tst->merged = NULL;\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503916","messageId":"6aaae4baa6ec9bccf8f8e8a517dcebda995b65b2.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 15/25] reftable/stack: handle allocation failures in `stack_compact_range()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:56:07Z","receivedAt":"2024-10-02T10:56:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `stack_compact_range()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 19 +++++++++++++++++--\n 1 file changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 1b77c9d014..2e6dd513d7 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1131,6 +1131,11 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * from the point of view of the newer process.\n \t */\n \tREFTABLE_CALLOC_ARRAY(table_locks, last - first + 1);\n+\tif (!table_locks) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tfor (i = last + 1; i > first; i--) {\n \t\tstack_filename(&table_name, st, reader_name(st->readers[i - 1]));\n \n@@ -1312,8 +1317,18 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\t * thus have to allocate `readers_len + 1` many entries.\n \t\t */\n \t\tREFTABLE_CALLOC_ARRAY(names, st->merged->readers_len + 1);\n-\t\tfor (size_t i = 0; i < st->merged->readers_len; i++)\n-\t\t\tnames[i] = xstrdup(st->readers[i]->name);\n+\t\tif (!names) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tfor (size_t i = 0; i < st->merged->readers_len; i++) {\n+\t\t\tnames[i] = reftable_strdup(st->readers[i]->name);\n+\t\t\tif (!names[i]) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n \t\tfirst_to_replace = first;\n \t\tlast_to_replace = last;\n \t}\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503917","messageId":"103a59ef0a5318b2ecef7ba65b7bb0c049b6dc6c.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 16/25] reftable/stack: handle allocation failures in auto compaction","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:56:11Z","receivedAt":"2024-10-02T10:56:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `reftable_stack_auto_compact()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 16 ++++++++++++----\n 1 file changed, 12 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 2e6dd513d7..990784d9d2 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1518,6 +1518,8 @@ static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n \tuint64_t *sizes;\n \n \tREFTABLE_CALLOC_ARRAY(sizes, st->merged->readers_len);\n+\tif (!sizes)\n+\t\treturn NULL;\n \n \tfor (size_t i = 0; i < st->merged->readers_len; i++)\n \t\tsizes[i] = st->readers[i]->size - overhead;\n@@ -1527,11 +1529,17 @@ static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st)\n \n int reftable_stack_auto_compact(struct reftable_stack *st)\n {\n-\tuint64_t *sizes = stack_table_sizes_for_compaction(st);\n-\tstruct segment seg =\n-\t\tsuggest_compaction_segment(sizes, st->merged->readers_len,\n-\t\t\t\t\t st->opts.auto_compaction_factor);\n+\tstruct segment seg;\n+\tuint64_t *sizes;\n+\n+\tsizes = stack_table_sizes_for_compaction(st);\n+\tif (!sizes)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n+\tseg = suggest_compaction_segment(sizes, st->merged->readers_len,\n+\t\t\t\t\t st->opts.auto_compaction_factor);\n \treftable_free(sizes);\n+\n \tif (segment_size(&seg) > 0)\n \t\treturn stack_compact_range(st, seg.start, seg.end - 1,\n \t\t\t\t\t NULL, STACK_COMPACT_RANGE_BEST_EFFORT);\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503918","messageId":"6799d299fe6dffcc6323f09dda6cd6cd51996056.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 17/25] reftable/iter: handle allocation failures when creating indexed table iter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:56:14Z","receivedAt":"2024-10-02T10:56:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `new_indexed_table_ref_iter()`. While at\nit, rename the function to match our coding style.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/iter.c | 20 ++++++++++++++++----\n reftable/iter.h | 2 +-\n reftable/reader.c | 7 ++++++-\n 3 files changed, 23 insertions(+), 6 deletions(-)\n\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex 416a9f6996..d319538f80 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -181,14 +181,20 @@ static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\n \t}\n }\n \n-int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+int indexed_table_ref_iter_new(struct indexed_table_ref_iter **dest,\n \t\t\t struct reftable_reader *r, uint8_t *oid,\n \t\t\t int oid_len, uint64_t *offsets, int offset_len)\n {\n \tstruct indexed_table_ref_iter empty = INDEXED_TABLE_REF_ITER_INIT;\n-\tstruct indexed_table_ref_iter *itr = reftable_calloc(1, sizeof(*itr));\n+\tstruct indexed_table_ref_iter *itr;\n \tint err = 0;\n \n+\titr = reftable_calloc(1, sizeof(*itr));\n+\tif (!itr) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n \t*itr = empty;\n \titr->r = r;\n \tstrbuf_add(&itr->oid, oid, oid_len);\n@@ -197,10 +203,16 @@ int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n \titr->offset_len = offset_len;\n \n \terr = indexed_table_ref_iter_next_block(itr);\n+\tif (err < 0)\n+\t\tgoto out;\n+\n+\t*dest = itr;\n+\terr = 0;\n+\n+out:\n \tif (err < 0) {\n+\t\t*dest = NULL;\n \t\treftable_free(itr);\n-\t} else {\n-\t\t*dest = itr;\n \t}\n \treturn err;\n }\ndiff --git a/reftable/iter.h b/reftable/iter.h\nindex befc4597df..b3225bc7ad 100644\n--- a/reftable/iter.h\n+++ b/reftable/iter.h\n@@ -82,7 +82,7 @@ void iterator_from_indexed_table_ref_iter(struct reftable_iterator *it,\n \t\t\t\t\t struct indexed_table_ref_iter *itr);\n \n /* Takes ownership of `offsets` */\n-int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,\n+int indexed_table_ref_iter_new(struct indexed_table_ref_iter **dest,\n \t\t\t struct reftable_reader *r, uint8_t *oid,\n \t\t\t int oid_len, uint64_t *offsets, int offset_len);\n \ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex f696e992df..0179e4e73d 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -715,7 +715,7 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,\n \t\tgoto done;\n \t}\n \n-\terr = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),\n+\terr = indexed_table_ref_iter_new(&itr, r, oid, hash_size(r->hash_id),\n \t\t\t\t\t got.u.obj.offsets,\n \t\t\t\t\t got.u.obj.offset_len);\n \tif (err < 0)\n@@ -740,6 +740,11 @@ static int reftable_reader_refs_for_unindexed(struct reftable_reader *r,\n \tint err;\n \n \tREFTABLE_ALLOC_ARRAY(ti, 1);\n+\tif (!ti) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n \ttable_iter_init(ti, r);\n \terr = table_iter_seek_start(ti, BLOCK_TYPE_REF, 0);\n \tif (err < 0)\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503919","messageId":"c7e54d71d758e92ceafefbb8cd272920616c2453.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 18/25] reftable/blocksource: handle allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:56:17Z","receivedAt":"2024-10-02T10:56:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in the blocksource code.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/blocksource.c | 25 ++++++++++++++++++++-----\n 1 file changed, 20 insertions(+), 5 deletions(-)\n\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex e93cac9bb6..a2a6a196d5 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -30,6 +30,8 @@ static int strbuf_read_block(void *v, struct reftable_block *dest, uint64_t off,\n \tstruct strbuf *b = v;\n \tassert(off + size <= b->len);\n \tREFTABLE_CALLOC_ARRAY(dest->data, size);\n+\tif (!dest->data)\n+\t\treturn -1;\n \tmemcpy(dest->data, b->buf + off, size);\n \tdest->len = size;\n \treturn size;\n@@ -98,27 +100,40 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,\n {\n \tstruct file_block_source *p;\n \tstruct stat st;\n-\tint fd;\n+\tint fd, err;\n \n \tfd = open(name, O_RDONLY);\n \tif (fd < 0) {\n \t\tif (errno == ENOENT)\n \t\t\treturn REFTABLE_NOT_EXIST_ERROR;\n-\t\treturn -1;\n+\t\terr = -1;\n+\t\tgoto out;\n \t}\n \n \tif (fstat(fd, &st) < 0) {\n-\t\tclose(fd);\n-\t\treturn REFTABLE_IO_ERROR;\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tgoto out;\n \t}\n \n \tREFTABLE_CALLOC_ARRAY(p, 1);\n+\tif (!p) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto out;\n+\t}\n+\n \tp->size = st.st_size;\n \tp->data = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n-\tclose(fd);\n \n \tassert(!bs->ops);\n \tbs->ops = &file_vtable;\n \tbs->arg = p;\n+\n+\terr = 0;\n+\n+out:\n+\tif (fd >= 0)\n+\t\tclose(fd);\n+\tif (err < 0)\n+\t\treftable_free(p);\n \treturn 0;\n }\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503920","messageId":"92d39b9021afc2dfcfa3d236c7c86286a7a4e101.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 19/25] reftable/block: handle allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:56:19Z","receivedAt":"2024-10-02T10:56:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures in `block_writer_init()` and\n`block_reader_init()`. This requires us to bubble up error codes into\n`writer_reinit_block_writer()`. Adapt call sites accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/block.c | 23 +++++++++++++--\n reftable/block.h | 4 +--\n reftable/writer.c | 51 ++++++++++++++++++++++-----------\n t/unit-tests/t-reftable-block.c | 20 +++++++------\n 4 files changed, 69 insertions(+), 29 deletions(-)\n\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 00030eee06..bfa7dc61bf 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -52,6 +52,8 @@ static int block_writer_register_restart(struct block_writer *w, int n,\n \t\treturn -1;\n \tif (is_restart) {\n \t\tREFTABLE_ALLOC_GROW(w->restarts, w->restart_len + 1, w->restart_cap);\n+\t\tif (!w->restarts)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \t\tw->restarts[w->restart_len++] = w->next;\n \t}\n \n@@ -63,8 +65,8 @@ static int block_writer_register_restart(struct block_writer *w, int n,\n \treturn 0;\n }\n \n-void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n-\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n+int block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size)\n {\n \tbw->buf = buf;\n \tbw->hash_size = hash_size;\n@@ -78,8 +80,12 @@ void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n \tbw->last_key.len = 0;\n \tif (!bw->zstream) {\n \t\tREFTABLE_CALLOC_ARRAY(bw->zstream, 1);\n+\t\tif (!bw->zstream)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \t\tdeflateInit(bw->zstream, 9);\n \t}\n+\n+\treturn 0;\n }\n \n uint8_t block_writer_type(struct block_writer *bw)\n@@ -163,6 +169,10 @@ int block_writer_finish(struct block_writer *w)\n \t\t */\n \t\tcompressed_len = deflateBound(w->zstream, src_len);\n \t\tREFTABLE_ALLOC_GROW(w->compressed, compressed_len, w->compressed_cap);\n+\t\tif (!w->compressed) {\n+\t\t\tret = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\treturn ret;\n+\t\t}\n \n \t\tw->zstream->next_out = w->compressed;\n \t\tw->zstream->avail_out = compressed_len;\n@@ -219,12 +229,21 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,\n \t\t/* Log blocks specify the *uncompressed* size in their header. */\n \t\tREFTABLE_ALLOC_GROW(br->uncompressed_data, sz,\n \t\t\t\t br->uncompressed_cap);\n+\t\tif (!br->uncompressed_data) {\n+\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n \n \t\t/* Copy over the block header verbatim. It's not compressed. */\n \t\tmemcpy(br->uncompressed_data, block->data, block_header_skip);\n \n \t\tif (!br->zstream) {\n \t\t\tREFTABLE_CALLOC_ARRAY(br->zstream, 1);\n+\t\t\tif (!br->zstream) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\n \t\t\terr = inflateInit(br->zstream);\n \t\t} else {\n \t\t\terr = inflateReset(br->zstream);\ndiff --git a/reftable/block.h b/reftable/block.h\nindex 1c8f25ee6e..18d7ea0337 100644\n--- a/reftable/block.h\n+++ b/reftable/block.h\n@@ -45,8 +45,8 @@ struct block_writer {\n /*\n * initializes the blockwriter to write `typ` entries, using `buf` as temporary\n * storage. `buf` is not owned by the block_writer. */\n-void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n-\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n+int block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,\n+\t\t uint32_t block_size, uint32_t header_off, int hash_size);\n \n /* returns the block type (eg. 'r' for ref records. */\n uint8_t block_writer_type(struct block_writer *bw);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 8ab2e916d3..791e246337 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -102,19 +102,24 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \treturn header_size(writer_version(w));\n }\n \n-static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n+static int writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)\n {\n-\tint block_start = 0;\n-\tif (w->next == 0) {\n+\tint block_start = 0, ret;\n+\n+\tif (w->next == 0)\n \t\tblock_start = header_size(writer_version(w));\n-\t}\n \n \tstrbuf_reset(&w->last_key);\n-\tblock_writer_init(&w->block_writer_data, typ, w->block,\n-\t\t\t w->opts.block_size, block_start,\n-\t\t\t hash_size(w->opts.hash_id));\n+\tret = block_writer_init(&w->block_writer_data, typ, w->block,\n+\t\t\t\tw->opts.block_size, block_start,\n+\t\t\t\thash_size(w->opts.hash_id));\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n \tw->block_writer = &w->block_writer_data;\n \tw->block_writer->restart_interval = w->opts.restart_interval;\n+\n+\treturn 0;\n }\n \n int reftable_writer_new(struct reftable_writer **out,\n@@ -247,8 +252,11 @@ static int writer_add_record(struct reftable_writer *w,\n \n \tstrbuf_reset(&w->last_key);\n \tstrbuf_addbuf(&w->last_key, &key);\n-\tif (!w->block_writer)\n-\t\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\tif (!w->block_writer) {\n+\t\terr = writer_reinit_block_writer(w, reftable_record_type(rec));\n+\t\tif (err < 0)\n+\t\t\tgoto done;\n+\t}\n \n \tif (block_writer_type(w->block_writer) != reftable_record_type(rec))\n \t\tBUG(\"record of type %d added to writer of type %d\",\n@@ -271,7 +279,9 @@ static int writer_add_record(struct reftable_writer *w,\n \terr = writer_flush_block(w);\n \tif (err < 0)\n \t\tgoto done;\n-\twriter_reinit_block_writer(w, reftable_record_type(rec));\n+\terr = writer_reinit_block_writer(w, reftable_record_type(rec));\n+\tif (err < 0)\n+\t\tgoto done;\n \n \t/*\n \t * Try to add the record to the writer again. If this still fails then\n@@ -461,7 +471,9 @@ static int writer_finish_section(struct reftable_writer *w)\n \n \t\tmax_level++;\n \t\tindex_start = w->next;\n-\t\twriter_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\t\terr = writer_reinit_block_writer(w, BLOCK_TYPE_INDEX);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n \n \t\tidx = w->index;\n \t\tidx_len = w->index_len;\n@@ -555,7 +567,10 @@ static void write_object_record(void *void_arg, void *key)\n \tif (arg->err < 0)\n \t\tgoto done;\n \n-\twriter_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\targ->err = writer_reinit_block_writer(arg->w, BLOCK_TYPE_OBJ);\n+\tif (arg->err < 0)\n+\t\tgoto done;\n+\n \targ->err = block_writer_add(arg->w->block_writer, &rec);\n \tif (arg->err == 0)\n \t\tgoto done;\n@@ -584,16 +599,18 @@ static int writer_dump_object_index(struct reftable_writer *w)\n \tstruct common_prefix_arg common = {\n \t\t.max = 1,\t\t/* obj_id_len should be >= 2. */\n \t};\n-\tif (w->obj_index_tree) {\n+\tint err;\n+\n+\tif (w->obj_index_tree)\n \t\tinfix_walk(w->obj_index_tree, &update_common, &common);\n-\t}\n \tw->stats.object_id_len = common.max + 1;\n \n-\twriter_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\terr = writer_reinit_block_writer(w, BLOCK_TYPE_OBJ);\n+\tif (err < 0)\n+\t\treturn err;\n \n-\tif (w->obj_index_tree) {\n+\tif (w->obj_index_tree)\n \t\tinfix_walk(w->obj_index_tree, &write_object_record, &closure);\n-\t}\n \n \tif (closure.err < 0)\n \t\treturn closure.err;\ndiff --git a/t/unit-tests/t-reftable-block.c b/t/unit-tests/t-reftable-block.c\nindex f1a49485e2..e52a612e85 100644\n--- a/t/unit-tests/t-reftable-block.c\n+++ b/t/unit-tests/t-reftable-block.c\n@@ -34,8 +34,9 @@ static void t_ref_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \trec.u.ref.refname = (char *) \"\";\n \trec.u.ref.value_type = REFTABLE_REF_DELETION;\n@@ -126,8 +127,9 @@ static void t_log_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n \t\trec.u.log.refname = xstrfmt(\"branch%02\"PRIuMAX , (uintmax_t)i);\n@@ -214,8 +216,9 @@ static void t_obj_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n \t\tuint8_t bytes[] = { i, i + 1, i + 2, i + 3, i + 5 }, *allocated;\n@@ -296,8 +299,9 @@ static void t_index_block_read_write(void)\n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n-\tblock_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size,\n-\t\t\t header_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tret = block_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size,\n+\t\t\t\theader_off, hash_size(GIT_SHA1_FORMAT_ID));\n+\tcheck(!ret);\n \n \tfor (i = 0; i < N; i++) {\n \t\tstrbuf_init(&recs[i].u.idx.last_key, 9);\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503921","messageId":"3416004e0d73ec46be616ca2809bb40d4d10c44a.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 20/25] reftable/pq: handle allocation failures when adding entries","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:56:22Z","receivedAt":"2024-10-02T10:56:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle allocation failures when adding entries to the pqueue. Adapt its\nonly caller accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 5 ++++-\n reftable/pq.c | 7 ++++++-\n reftable/pq.h | 2 +-\n 3 files changed, 11 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 69790c345c..8e202a8efd 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -54,7 +54,10 @@ static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)\n \tif (err)\n \t\treturn err;\n \n-\tmerged_iter_pqueue_add(&mi->pq, &e);\n+\terr = merged_iter_pqueue_add(&mi->pq, &e);\n+\tif (err)\n+\t\treturn err;\n+\n \treturn 0;\n }\n \ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex 2b5b7d1c0e..03b9912282 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -8,6 +8,7 @@ license that can be found in the LICENSE file or at\n \n #include \"pq.h\"\n \n+#include \"reftable-error.h\"\n #include \"reftable-record.h\"\n #include \"system.h\"\n #include \"basics.h\"\n@@ -44,11 +45,13 @@ struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n \treturn e;\n }\n \n-void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e)\n+int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e)\n {\n \tsize_t i = 0;\n \n \tREFTABLE_ALLOC_GROW(pq->heap, pq->len + 1, pq->cap);\n+\tif (!pq->heap)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tpq->heap[pq->len++] = *e;\n \n \ti = pq->len - 1;\n@@ -59,6 +62,8 @@ void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry\n \t\tSWAP(pq->heap[j], pq->heap[i]);\n \t\ti = j;\n \t}\n+\n+\treturn 0;\n }\n \n void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)\ndiff --git a/reftable/pq.h b/reftable/pq.h\nindex 707bd26767..83c062eeca 100644\n--- a/reftable/pq.h\n+++ b/reftable/pq.h\n@@ -23,7 +23,7 @@ struct merged_iter_pqueue {\n };\n \n struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n-void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e);\n+int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e);\n void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n int pq_less(struct pq_entry *a, struct pq_entry *b);\n \n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503922","messageId":"a3639304884db6547f43461afe7aa9800ff24dc7.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 21/25] reftable/tree: handle allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:56:25Z","receivedAt":"2024-10-02T10:56:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The tree interfaces of the reftable library handle both insertion and\nsearching of tree nodes with a single function, where the behaviour is\naltered between the two via an `insert` bit. This makes it quit awkward\nto handle allocation failures because on inserting we'd have to check\nfor `NULL` pointers and return an error, whereas on searching entries we\ndon't have to handle it as an allocation error.\n\nSplit up concerns of this function into two separate functions, one for\ninserting entries and one for searching entries. This makes it easy for\nus to check for allocation errors as `tree_insert()` should never return\na `NULL` pointer now. Adapt callers accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/tree.c | 42 +++++++++++++++++++++++-----------\n reftable/tree.h | 21 +++++++++++++----\n reftable/writer.c | 7 +++---\n t/unit-tests/t-reftable-tree.c | 10 ++++----\n 4 files changed, 54 insertions(+), 26 deletions(-)\n\ndiff --git a/reftable/tree.c b/reftable/tree.c\nindex 5ffb2e0d69..f4dbe72090 100644\n--- a/reftable/tree.c\n+++ b/reftable/tree.c\n@@ -11,28 +11,44 @@ license that can be found in the LICENSE file or at\n \n #include \"basics.h\"\n \n-struct tree_node *tree_search(void *key, struct tree_node **rootp,\n-\t\t\t int (*compare)(const void *, const void *),\n-\t\t\t int insert)\n+struct tree_node *tree_search(struct tree_node *tree,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *))\n {\n \tint res;\n+\tif (!tree)\n+\t\treturn NULL;\n+\tres = compare(key, tree->key);\n+\tif (res < 0)\n+\t\treturn tree_search(tree->left, key, compare);\n+\telse if (res > 0)\n+\t\treturn tree_search(tree->right, key, compare);\n+\treturn tree;\n+}\n+\n+struct tree_node *tree_insert(struct tree_node **rootp,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *))\n+{\n+\tint res;\n+\n \tif (!*rootp) {\n-\t\tif (!insert) {\n+\t\tstruct tree_node *n;\n+\n+\t\tREFTABLE_CALLOC_ARRAY(n, 1);\n+\t\tif (!n)\n \t\t\treturn NULL;\n-\t\t} else {\n-\t\t\tstruct tree_node *n;\n-\t\t\tREFTABLE_CALLOC_ARRAY(n, 1);\n-\t\t\tn->key = key;\n-\t\t\t*rootp = n;\n-\t\t\treturn *rootp;\n-\t\t}\n+\n+\t\tn->key = key;\n+\t\t*rootp = n;\n+\t\treturn *rootp;\n \t}\n \n \tres = compare(key, (*rootp)->key);\n \tif (res < 0)\n-\t\treturn tree_search(key, &(*rootp)->left, compare, insert);\n+\t\treturn tree_insert(&(*rootp)->left, key, compare);\n \telse if (res > 0)\n-\t\treturn tree_search(key, &(*rootp)->right, compare, insert);\n+\t\treturn tree_insert(&(*rootp)->right, key, compare);\n \treturn *rootp;\n }\n \ndiff --git a/reftable/tree.h b/reftable/tree.h\nindex fbdd002e23..9604453b6d 100644\n--- a/reftable/tree.h\n+++ b/reftable/tree.h\n@@ -15,12 +15,23 @@ struct tree_node {\n \tstruct tree_node *left, *right;\n };\n \n-/* looks for `key` in `rootp` using `compare` as comparison function. If insert\n- * is set, insert the key if it's not found. Else, return NULL.\n+/*\n+ * Search the tree for the node matching the given key using `compare` as\n+ * comparison function. Returns the node whose key matches or `NULL` in case\n+ * the key does not exist in the tree.\n+ */\n+struct tree_node *tree_search(struct tree_node *tree,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *));\n+\n+/*\n+ * Insert a node into the tree. Returns the newly inserted node if the key does\n+ * not yet exist. Otherwise it returns the preexisting node. Returns `NULL`\n+ * when allocating the new node fails.\n */\n-struct tree_node *tree_search(void *key, struct tree_node **rootp,\n-\t\t\t int (*compare)(const void *, const void *),\n-\t\t\t int insert);\n+struct tree_node *tree_insert(struct tree_node **rootp,\n+\t\t\t void *key,\n+\t\t\t int (*compare)(const void *, const void *));\n \n /* performs an infix walk of the tree. */\n void infix_walk(struct tree_node *t, void (*action)(void *arg, void *key),\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 791e246337..e180c10840 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -208,8 +208,7 @@ static int writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n \tstruct obj_index_tree_node *key;\n \tstruct tree_node *node;\n \n-\tnode = tree_search(&want, &w->obj_index_tree,\n-\t\t\t &obj_index_tree_node_compare, 0);\n+\tnode = tree_search(w->obj_index_tree, &want, &obj_index_tree_node_compare);\n \tif (!node) {\n \t\tstruct obj_index_tree_node empty = OBJ_INDEX_TREE_NODE_INIT;\n \n@@ -221,8 +220,8 @@ static int writer_index_hash(struct reftable_writer *w, struct strbuf *hash)\n \n \t\tstrbuf_reset(&key->hash);\n \t\tstrbuf_addbuf(&key->hash, hash);\n-\t\ttree_search((void *)key, &w->obj_index_tree,\n-\t\t\t &obj_index_tree_node_compare, 1);\n+\t\ttree_insert(&w->obj_index_tree, key,\n+\t\t\t &obj_index_tree_node_compare);\n \t} else {\n \t\tkey = node->key;\n \t}\ndiff --git a/t/unit-tests/t-reftable-tree.c b/t/unit-tests/t-reftable-tree.c\nindex 700479d34b..79b175a45a 100644\n--- a/t/unit-tests/t-reftable-tree.c\n+++ b/t/unit-tests/t-reftable-tree.c\n@@ -37,16 +37,17 @@ static void t_tree_search(void)\n \t * values[1] and values[10] (inclusive) in the tree.\n \t */\n \tdo {\n-\t\tnodes[i] = tree_search(&values[i], &root, &t_compare, 1);\n+\t\tnodes[i] = tree_insert(&root, &values[i], &t_compare);\n+\t\tcheck(nodes[i] != NULL);\n \t\ti = (i * 7) % 11;\n \t} while (i != 1);\n \n \tfor (i = 1; i < ARRAY_SIZE(nodes); i++) {\n \t\tcheck_pointer_eq(&values[i], nodes[i]->key);\n-\t\tcheck_pointer_eq(nodes[i], tree_search(&values[i], &root, &t_compare, 0));\n+\t\tcheck_pointer_eq(nodes[i], tree_search(root, &values[i], &t_compare));\n \t}\n \n-\tcheck(!tree_search(values, &root, t_compare, 0));\n+\tcheck(!tree_search(root, values, t_compare));\n \ttree_free(root);\n }\n \n@@ -62,7 +63,8 @@ static void t_infix_walk(void)\n \tsize_t count = 0;\n \n \tdo {\n-\t\ttree_search(&values[i], &root, t_compare, 1);\n+\t\tstruct tree_node *node = tree_insert(&root, &values[i], t_compare);\n+\t\tcheck(node != NULL);\n \t\ti = (i * 7) % 11;\n \t\tcount++;\n \t} while (i != 1);\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503923","messageId":"28661500ff1710e866966b93c1d7c7d820f4c511.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 22/25] reftable: handle trivial allocation failures","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:56:31Z","receivedAt":"2024-10-02T10:56:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Handle trivial allocation failures in the reftable library and its unit\ntests.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 3 +++\n reftable/reader.c | 10 +++++++++-\n reftable/stack.c | 20 ++++++++++++++++++++\n reftable/writer.c | 13 +++++++++++--\n t/unit-tests/t-reftable-block.c | 4 ++++\n t/unit-tests/t-reftable-merged.c | 4 ++++\n t/unit-tests/t-reftable-readwrite.c | 27 ++++++++++++++++-----------\n 7 files changed, 67 insertions(+), 14 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 8e202a8efd..514d6facf4 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -203,6 +203,9 @@ int reftable_merged_table_new(struct reftable_merged_table **dest,\n \t}\n \n \tREFTABLE_CALLOC_ARRAY(m, 1);\n+\tif (!m)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \tm->readers = readers;\n \tm->readers_len = n;\n \tm->min = first_min;\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 0179e4e73d..98e7aa2637 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -598,6 +598,10 @@ int reftable_reader_new(struct reftable_reader **out,\n \tint err;\n \n \tREFTABLE_CALLOC_ARRAY(r, 1);\n+\tif (!r) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \n \t/*\n \t * We need one extra byte to read the type of first block. We also\n@@ -627,7 +631,11 @@ int reftable_reader_new(struct reftable_reader **out,\n \n \tr->size = file_size - footer_size(r->version);\n \tr->source = *source;\n-\tr->name = xstrdup(name);\n+\tr->name = reftable_strdup(name);\n+\tif (!r->name) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \tr->hash_id = 0;\n \tr->refcount = 1;\n \ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 990784d9d2..7df28ab343 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -116,6 +116,11 @@ static int fd_read_lines(int fd, char ***namesp)\n \t}\n \n \tREFTABLE_ALLOC_ARRAY(buf, size + 1);\n+\tif (!buf) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n+\n \tif (read_in_full(fd, buf, size) != size) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n@@ -140,6 +145,8 @@ int read_lines(const char *filename, char ***namesp)\n \tif (fd < 0) {\n \t\tif (errno == ENOENT) {\n \t\t\tREFTABLE_CALLOC_ARRAY(*namesp, 1);\n+\t\t\tif (!*namesp)\n+\t\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \t\t\treturn 0;\n \t\t}\n \n@@ -420,6 +427,10 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\t\t}\n \n \t\t\tREFTABLE_CALLOC_ARRAY(names, 1);\n+\t\t\tif (!names) {\n+\t\t\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\t\t\tgoto out;\n+\t\t\t}\n \t\t} else {\n \t\t\terr = fd_read_lines(fd, &names);\n \t\t\tif (err < 0)\n@@ -779,7 +790,11 @@ int reftable_stack_new_addition(struct reftable_addition **dest,\n {\n \tint err = 0;\n \tstruct reftable_addition empty = REFTABLE_ADDITION_INIT;\n+\n \tREFTABLE_CALLOC_ARRAY(*dest, 1);\n+\tif (!*dest)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \t**dest = empty;\n \terr = reftable_stack_init_addition(*dest, st);\n \tif (err) {\n@@ -886,7 +901,12 @@ int reftable_addition_add(struct reftable_addition *add,\n \n \tREFTABLE_ALLOC_GROW(add->new_tables, add->new_tables_len + 1,\n \t\t\t add->new_tables_cap);\n+\tif (!add->new_tables) {\n+\t\terr = REFTABLE_OUT_OF_MEMORY_ERROR;\n+\t\tgoto done;\n+\t}\n \tadd->new_tables[add->new_tables_len++] = strbuf_detach(&next_name, NULL);\n+\n done:\n \tdelete_tempfile(&tab_file);\n \tstrbuf_release(&temp_tab_file_name);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex e180c10840..550172e65c 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -49,8 +49,14 @@ static int padded_write(struct reftable_writer *w, uint8_t *data, size_t len,\n {\n \tint n = 0;\n \tif (w->pending_padding > 0) {\n-\t\tuint8_t *zeroed = reftable_calloc(w->pending_padding, sizeof(*zeroed));\n-\t\tint n = w->write(w->write_arg, zeroed, w->pending_padding);\n+\t\tuint8_t *zeroed;\n+\t\tint n;\n+\n+\t\tzeroed = reftable_calloc(w->pending_padding, sizeof(*zeroed));\n+\t\tif (!zeroed)\n+\t\t\treturn -1;\n+\n+\t\tn = w->write(w->write_arg, zeroed, w->pending_padding);\n \t\tif (n < 0)\n \t\t\treturn n;\n \n@@ -767,6 +773,9 @@ static int writer_flush_nonempty_block(struct reftable_writer *w)\n \t * case we will end up with a multi-level index.\n \t */\n \tREFTABLE_ALLOC_GROW(w->index, w->index_len + 1, w->index_cap);\n+\tif (!w->index)\n+\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n \tindex_record.offset = w->next;\n \tstrbuf_reset(&index_record.last_key);\n \tstrbuf_addbuf(&index_record.last_key, &w->block_writer->last_key);\ndiff --git a/t/unit-tests/t-reftable-block.c b/t/unit-tests/t-reftable-block.c\nindex e52a612e85..d470060e8b 100644\n--- a/t/unit-tests/t-reftable-block.c\n+++ b/t/unit-tests/t-reftable-block.c\n@@ -32,6 +32,7 @@ static void t_ref_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,\n@@ -125,6 +126,7 @@ static void t_log_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source ,&buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size,\n@@ -214,6 +216,7 @@ static void t_obj_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size,\n@@ -297,6 +300,7 @@ static void t_index_block_read_write(void)\n \tstruct strbuf want = STRBUF_INIT, buf = STRBUF_INIT;\n \n \tREFTABLE_CALLOC_ARRAY(block.data, block_size);\n+\tcheck(block.data != NULL);\n \tblock.len = block_size;\n \tblock_source_from_strbuf(&block.source, &buf);\n \tret = block_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size,\ndiff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c\nindex 3d2848632d..3c84363e98 100644\n--- a/t/unit-tests/t-reftable-merged.c\n+++ b/t/unit-tests/t-reftable-merged.c\n@@ -29,7 +29,9 @@ merged_table_from_records(struct reftable_ref_record **refs,\n \tint err;\n \n \tREFTABLE_CALLOC_ARRAY(*readers, n);\n+\tcheck(*readers != NULL);\n \tREFTABLE_CALLOC_ARRAY(*source, n);\n+\tcheck(*source != NULL);\n \n \tfor (size_t i = 0; i < n; i++) {\n \t\tt_reftable_write_to_buf(&buf[i], refs[i], sizes[i], NULL, 0, &opts);\n@@ -285,7 +287,9 @@ merged_table_from_log_records(struct reftable_log_record **logs,\n \tint err;\n \n \tREFTABLE_CALLOC_ARRAY(*readers, n);\n+\tcheck(*readers != NULL);\n \tREFTABLE_CALLOC_ARRAY(*source, n);\n+\tcheck(*source != NULL);\n \n \tfor (size_t i = 0; i < n; i++) {\n \t\tt_reftable_write_to_buf(&buf[i], NULL, 0, logs[i], sizes[i], &opts);\ndiff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c\nindex acca927a2c..bfa069caff 100644\n--- a/t/unit-tests/t-reftable-readwrite.c\n+++ b/t/unit-tests/t-reftable-readwrite.c\n@@ -52,8 +52,11 @@ static void write_table(char ***names, struct strbuf *buf, int N,\n \tint i;\n \n \tREFTABLE_CALLOC_ARRAY(*names, N + 1);\n+\tcheck(*names != NULL);\n \tREFTABLE_CALLOC_ARRAY(refs, N);\n+\tcheck(refs != NULL);\n \tREFTABLE_CALLOC_ARRAY(logs, N);\n+\tcheck(logs != NULL);\n \n \tfor (i = 0; i < N; i++) {\n \t\trefs[i].refname = (*names)[i] = xstrfmt(\"refs/heads/branch%02d\", i);\n@@ -150,23 +153,25 @@ static void t_log_overflow(void)\n \n static void t_log_write_read(void)\n {\n-\tint N = 2;\n-\tchar **names = reftable_calloc(N + 1, sizeof(*names));\n-\tint err;\n \tstruct reftable_write_options opts = {\n \t\t.block_size = 256,\n \t};\n \tstruct reftable_ref_record ref = { 0 };\n-\tint i = 0;\n \tstruct reftable_log_record log = { 0 };\n-\tint n;\n \tstruct reftable_iterator it = { 0 };\n \tstruct reftable_reader *reader;\n \tstruct reftable_block_source source = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w = t_reftable_strbuf_writer(&buf, &opts);\n \tconst struct reftable_stats *stats = NULL;\n+\tint N = 2, err, i, n;\n+\tchar **names;\n+\n+\tnames = reftable_calloc(N + 1, sizeof(*names));\n+\tcheck(names != NULL);\n+\n \treftable_writer_set_limits(w, 0, N);\n+\n \tfor (i = 0; i < N; i++) {\n \t\tchar name[256];\n \t\tstruct reftable_ref_record ref = { 0 };\n@@ -178,6 +183,7 @@ static void t_log_write_read(void)\n \t\terr = reftable_writer_add_ref(w, &ref);\n \t\tcheck(!err);\n \t}\n+\n \tfor (i = 0; i < N; i++) {\n \t\tstruct reftable_log_record log = { 0 };\n \n@@ -476,8 +482,7 @@ static void t_table_read_write_seek_index(void)\n \n static void t_table_refs_for(int indexed)\n {\n-\tint N = 50;\n-\tchar **want_names = reftable_calloc(N + 1, sizeof(*want_names));\n+\tchar **want_names;\n \tint want_names_len = 0;\n \tuint8_t want_hash[GIT_SHA1_RAWSZ];\n \n@@ -485,15 +490,15 @@ static void t_table_refs_for(int indexed)\n \t\t.block_size = 256,\n \t};\n \tstruct reftable_ref_record ref = { 0 };\n-\tint i = 0;\n-\tint n;\n-\tint err;\n \tstruct reftable_reader *reader;\n \tstruct reftable_block_source source = { 0 };\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct reftable_writer *w = t_reftable_strbuf_writer(&buf, &opts);\n \tstruct reftable_iterator it = { 0 };\n-\tint j;\n+\tint N = 50, n, j, err, i;\n+\n+\twant_names = reftable_calloc(N + 1, sizeof(*want_names));\n+\tcheck(want_names != NULL);\n \n \tt_reftable_set_hash(want_hash, 4, GIT_SHA1_FORMAT_ID);\n \n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503924","messageId":"228cc812632d3c310d9b4f02cacc06fe60f6c23c.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 23/25] reftable: fix calls to free(3P)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:56:33Z","receivedAt":"2024-10-02T10:56:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"There are a small set of calls to free(3P) in the reftable library. As\nthe reftable allocators are pluggable we should rather call the reftable\nspecific function, which is `reftable_free()`.\n\nConvert the code accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 2 +-\n t/unit-tests/t-reftable-readwrite.c | 4 ++--\n 2 files changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 7df28ab343..b2babe8e3d 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1421,7 +1421,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\tstruct lock_file *table_lock = &table_locks[i];\n \t\tchar *table_path = get_locked_file_path(table_lock);\n \t\tunlink(table_path);\n-\t\tfree(table_path);\n+\t\treftable_free(table_path);\n \t}\n \n done:\ndiff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c\nindex bfa069caff..27ce84445e 100644\n--- a/t/unit-tests/t-reftable-readwrite.c\n+++ b/t/unit-tests/t-reftable-readwrite.c\n@@ -76,8 +76,8 @@ static void write_table(char ***names, struct strbuf *buf, int N,\n \n \tt_reftable_write_to_buf(buf, refs, N, logs, N, &opts);\n \n-\tfree(refs);\n-\tfree(logs);\n+\treftable_free(refs);\n+\treftable_free(logs);\n }\n \n static void t_log_buffer_size(void)\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503925","messageId":"1c66f6ef8d9827653fd9df960330431877e1dbc9.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 24/25] reftable: introduce `REFTABLE_FREE_AND_NULL()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:56:36Z","receivedAt":"2024-10-02T10:56:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have several calls to `FREE_AND_NULL()` in the reftable library,\nwhich of course uses free(3P). As the reftable allocators are pluggable\nwe should rather call the reftable specific function, which is\n`reftable_free()`.\n\nIntroduce a new macro `REFTABLE_FREE_AND_NULL()` and adapt the callsites\naccordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 1 +\n reftable/block.c | 6 +++---\n reftable/iter.c | 2 +-\n reftable/pq.c | 2 +-\n reftable/reader.c | 2 +-\n reftable/record.c | 10 +++++-----\n reftable/stack.c | 8 ++++----\n reftable/writer.c | 4 ++--\n t/unit-tests/t-reftable-stack.c | 2 +-\n 9 files changed, 19 insertions(+), 18 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 69adeab2e4..7f0f20e50c 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -71,6 +71,7 @@ char *reftable_strdup(const char *str);\n \t\t\tREFTABLE_REALLOC_ARRAY(x, alloc); \\\n \t\t} \\\n \t} while (0)\n+#define REFTABLE_FREE_AND_NULL(p) do { reftable_free(p); (p) = NULL; } while (0)\n \n /* Find the longest shared prefix size of `a` and `b` */\n struct strbuf;\ndiff --git a/reftable/block.c b/reftable/block.c\nindex bfa7dc61bf..8d41a2f99e 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -551,9 +551,9 @@ int block_iter_seek_key(struct block_iter *it, const struct block_reader *br,\n void block_writer_release(struct block_writer *bw)\n {\n \tdeflateEnd(bw->zstream);\n-\tFREE_AND_NULL(bw->zstream);\n-\tFREE_AND_NULL(bw->restarts);\n-\tFREE_AND_NULL(bw->compressed);\n+\tREFTABLE_FREE_AND_NULL(bw->zstream);\n+\tREFTABLE_FREE_AND_NULL(bw->restarts);\n+\tREFTABLE_FREE_AND_NULL(bw->compressed);\n \tstrbuf_release(&bw->last_key);\n \t/* the block is not owned. */\n }\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex d319538f80..d926db653b 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -237,7 +237,7 @@ void reftable_iterator_destroy(struct reftable_iterator *it)\n \t\treturn;\n \tit->ops->close(it->iter_arg);\n \tit->ops = NULL;\n-\tFREE_AND_NULL(it->iter_arg);\n+\tREFTABLE_FREE_AND_NULL(it->iter_arg);\n }\n \n int reftable_iterator_seek_ref(struct reftable_iterator *it,\ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex 03b9912282..6ee1164dd3 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -68,6 +68,6 @@ int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry\n \n void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)\n {\n-\tFREE_AND_NULL(pq->heap);\n+\tREFTABLE_FREE_AND_NULL(pq->heap);\n \tmemset(pq, 0, sizeof(*pq));\n }\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 98e7aa2637..8d37253922 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -678,7 +678,7 @@ void reftable_reader_decref(struct reftable_reader *r)\n \tif (--r->refcount)\n \t\treturn;\n \tblock_source_close(&r->source);\n-\tFREE_AND_NULL(r->name);\n+\tREFTABLE_FREE_AND_NULL(r->name);\n \treftable_free(r);\n }\n \ndiff --git a/reftable/record.c b/reftable/record.c\nindex 787e134c9a..30d563e16d 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -476,8 +476,8 @@ static void reftable_obj_record_key(const void *r, struct strbuf *dest)\n static void reftable_obj_record_release(void *rec)\n {\n \tstruct reftable_obj_record *obj = rec;\n-\tFREE_AND_NULL(obj->hash_prefix);\n-\tFREE_AND_NULL(obj->offsets);\n+\tREFTABLE_FREE_AND_NULL(obj->hash_prefix);\n+\tREFTABLE_FREE_AND_NULL(obj->offsets);\n \tmemset(obj, 0, sizeof(struct reftable_obj_record));\n }\n \n@@ -834,10 +834,10 @@ static int reftable_log_record_decode(void *rec, struct strbuf key,\n \tif (val_type != r->value_type) {\n \t\tswitch (r->value_type) {\n \t\tcase REFTABLE_LOG_UPDATE:\n-\t\t\tFREE_AND_NULL(r->value.update.message);\n+\t\t\tREFTABLE_FREE_AND_NULL(r->value.update.message);\n \t\t\tr->value.update.message_cap = 0;\n-\t\t\tFREE_AND_NULL(r->value.update.email);\n-\t\t\tFREE_AND_NULL(r->value.update.name);\n+\t\t\tREFTABLE_FREE_AND_NULL(r->value.update.email);\n+\t\t\tREFTABLE_FREE_AND_NULL(r->value.update.name);\n \t\t\tbreak;\n \t\tcase REFTABLE_LOG_DELETION:\n \t\t\tbreak;\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex b2babe8e3d..63976e5cea 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -203,7 +203,7 @@ void reftable_stack_destroy(struct reftable_stack *st)\n \n \terr = read_lines(st->list_file, &names);\n \tif (err < 0) {\n-\t\tFREE_AND_NULL(names);\n+\t\tREFTABLE_FREE_AND_NULL(names);\n \t}\n \n \tif (st->readers) {\n@@ -224,7 +224,7 @@ void reftable_stack_destroy(struct reftable_stack *st)\n \t\t}\n \t\tstrbuf_release(&filename);\n \t\tst->readers_len = 0;\n-\t\tFREE_AND_NULL(st->readers);\n+\t\tREFTABLE_FREE_AND_NULL(st->readers);\n \t}\n \n \tif (st->list_fd >= 0) {\n@@ -232,8 +232,8 @@ void reftable_stack_destroy(struct reftable_stack *st)\n \t\tst->list_fd = -1;\n \t}\n \n-\tFREE_AND_NULL(st->list_file);\n-\tFREE_AND_NULL(st->reftable_dir);\n+\tREFTABLE_FREE_AND_NULL(st->list_file);\n+\tREFTABLE_FREE_AND_NULL(st->reftable_dir);\n \treftable_free(st);\n \tfree_names(names);\n }\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 550172e65c..b032a47dec 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -593,7 +593,7 @@ static void object_record_free(void *void_arg UNUSED, void *key)\n {\n \tstruct obj_index_tree_node *entry = key;\n \n-\tFREE_AND_NULL(entry->offsets);\n+\tREFTABLE_FREE_AND_NULL(entry->offsets);\n \tstrbuf_release(&entry->hash);\n \treftable_free(entry);\n }\n@@ -709,7 +709,7 @@ static void writer_clear_index(struct reftable_writer *w)\n {\n \tfor (size_t i = 0; w->index && i < w->index_len; i++)\n \t\tstrbuf_release(&w->index[i].last_key);\n-\tFREE_AND_NULL(w->index);\n+\tREFTABLE_FREE_AND_NULL(w->index);\n \tw->index_len = 0;\n \tw->index_cap = 0;\n }\ndiff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c\nindex 6e41a4cf7e..cc2db2cdef 100644\n--- a/t/unit-tests/t-reftable-stack.c\n+++ b/t/unit-tests/t-reftable-stack.c\n@@ -1149,7 +1149,7 @@ static void unclean_stack_close(struct reftable_stack *st)\n \tfor (size_t i = 0; i < st->readers_len; i++)\n \t\treftable_reader_decref(st->readers[i]);\n \tst->readers_len = 0;\n-\tFREE_AND_NULL(st->readers);\n+\tREFTABLE_FREE_AND_NULL(st->readers);\n }\n \n static void t_reftable_stack_compaction_concurrent_clean(void)\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503926","messageId":"bc80f1e7265784ed68436cf1e81b2aa4bdbec309.1727866394.git.ps@pks.im","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"[PATCH v5 25/25] reftable/basics: ban standard allocator functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-02T10:56:39Z","receivedAt":"2024-10-02T10:56:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library uses pluggable allocators, which means that we\nshouldn't ever use the standard allocator functions. But it is an easy\nmistake to make to accidentally use e.g. free(3P) instead of the\nreftable-specific `reftable_free()` function, and we do not have any\nmechanism to detect this misuse right now.\n\nIntroduce a couple of macros that ban the standard allocators, similar\nto how we do it in \"banned.h\".\n\nNote that we do not ban the following two classes of functions:\n\n - Macros like `FREE_AND_NULL()` or `REALLOC_ARRAY()`. As those expand\n to code that contains already-banned functions we'd get a compiler\n error even without banning those macros explicitly.\n\n - Git-specific allocators like `xmalloc()` and friends. The primary\n reason is that there are simply too many of them, so we're rather\n aiming for best effort here. Furthermore, the eventual goal is to\n make them unavailable in the reftable library place by not pulling\n them in via \"git-compat-utils.h\" anymore.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 1 +\n reftable/basics.h | 14 ++++++++++++++\n 2 files changed, 15 insertions(+)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex ea53cf102a..c8396dc525 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -6,6 +6,7 @@ license that can be found in the LICENSE file or at\n https://developers.google.com/open-source/licenses/bsd\n */\n \n+#define REFTABLE_ALLOW_BANNED_ALLOCATORS\n #include \"basics.h\"\n #include \"reftable-basics.h\"\n \ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 7f0f20e50c..4c9ef0fe6c 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -73,6 +73,20 @@ char *reftable_strdup(const char *str);\n \t} while (0)\n #define REFTABLE_FREE_AND_NULL(p) do { reftable_free(p); (p) = NULL; } while (0)\n \n+#ifndef REFTABLE_ALLOW_BANNED_ALLOCATORS\n+# define REFTABLE_BANNED(func) use_reftable_##func##_instead\n+# undef malloc\n+# define malloc(sz) REFTABLE_BANNED(malloc)\n+# undef realloc\n+# define realloc(ptr, sz) REFTABLE_BANNED(realloc)\n+# undef free\n+# define free(ptr) REFTABLE_BANNED(free)\n+# undef calloc\n+# define calloc(nelem, elsize) REFTABLE_BANNED(calloc)\n+# undef strdup\n+# define strdup(str) REFTABLE_BANNED(strdup)\n+#endif\n+\n /* Find the longest shared prefix size of `a` and `b` */\n struct strbuf;\n int common_prefix_size(struct strbuf *a, struct strbuf *b);\n-- \n2.47.0.rc0.dirty\n\n"},{"id":"503968","messageId":"xmqqv7yamldp.fsf@gitster.g","threadId":"62121","inReplyTo":"ZvzMIWNfzmZL7eHS@pks.im","subject":"Re: [PATCH v4 00/25] reftable: handle allocation errors","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-10-02T18:04:50Z","receivedAt":"2024-10-02T18:04:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> `FREE_AND_NULL()` would already be detected because it expands to code\n> that contains `free()`. Same for `REALLOC_ARRAY()` and the likes.\n\nAh, that is very true.\n"},{"id":"503972","messageId":"xmqqv7yal2qu.fsf@gitster.g","threadId":"62121","inReplyTo":"cover.1727866394.git.ps@pks.im","subject":"Re: [PATCH v5 00/25] reftable: handle allocation errors","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-10-02T19:32:41Z","receivedAt":"2024-10-02T19:32:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> Changes compared to v4:\n>\n> - `#undef` standard allocators before banning them.\n>\n> - Explain why we don't ban `FREE_AND_NULL()` and related as well as\n> `xmalloc()` and related.\n\nLooking good.\n\nLet me mark the topic for 'next' so that it can graduate in the\nfirst batch after the upcoming release.\n\nThanks.\n"},{"id":"503982","messageId":"CAPig+cT7nni11bGmNyYCQZ0P0QZ_HG-izPtsymq==sUOkOn-Lw@mail.gmail.com","threadId":"62121","inReplyTo":"bdfddbebce9f77959fd9544cd5ba3496d5b9dccf.1727866394.git.ps@pks.im","subject":"Re: [PATCH v5 05/25] reftable/basics: handle allocation failures in `parse_names()`","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2024-10-02T22:07:14Z","receivedAt":"2024-10-02T22:07:26Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Wed, Oct 2, 2024 at 6:56 AM Patrick Steinhardt wrote:\n> Handle allocation failures in `parse_names()` by returning `NULL` in\n> case any allocation fails. While at it, refactor the function to return\n> the array directly instead of assigning it to an out-pointer.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> diff --git a/reftable/basics.c b/reftable/basics.c\n> @@ -152,14 +152,26 @@ void parse_names(char *buf, int size, char ***namesp)\n> REFTABLE_ALLOC_GROW(names, names_len + 1, names_cap);\n> - names[names_len++] = xstrdup(p);\n> + if (!names)\n> + goto err;\n\nAm I reading this correctly? Presumably, `names_len` can be non-zero\nhere, right? And we now check for names==NULL to detect an allocation\nfailure...\n\n> + names[names_len] = reftable_strdup(p);\n> + if (!names[names_len++])\n> + goto err;\n> }\n> p = next + 1;\n> }\n>\n> REFTABLE_REALLOC_ARRAY(names, names_len + 1);\n> names[names_len] = NULL;\n> - *namesp = names;\n> +\n> + return names;\n> +\n> +err:\n> + for (size_t i = 0; i < names_len; i++)\n> + reftable_free(names[i]);\n\n... and then we potentially index into names[] because `names_len` is\nnon-zero, thus crash because `names` is NULL.\n\n> + reftable_free(names);\n> + return NULL;\n> }\n"},{"id":"504072","messageId":"Zv92BRolDugwLuOn@pks.im","threadId":"62121","inReplyTo":"CAPig+cT7nni11bGmNyYCQZ0P0QZ_HG-izPtsymq==sUOkOn-Lw@mail.gmail.com","subject":"Re: [PATCH v5 05/25] reftable/basics: handle allocation failures in `parse_names()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-04T04:58:53Z","receivedAt":"2024-10-04T04:59:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Oct 02, 2024 at 06:07:14PM -0400, Eric Sunshine wrote:\n> On Wed, Oct 2, 2024 at 6:56 AM Patrick Steinhardt wrote:\n> > Handle allocation failures in `parse_names()` by returning `NULL` in\n> > case any allocation fails. While at it, refactor the function to return\n> > the array directly instead of assigning it to an out-pointer.\n> >\n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > diff --git a/reftable/basics.c b/reftable/basics.c\n> > @@ -152,14 +152,26 @@ void parse_names(char *buf, int size, char ***namesp)\n> > REFTABLE_ALLOC_GROW(names, names_len + 1, names_cap);\n> > - names[names_len++] = xstrdup(p);\n> > + if (!names)\n> > + goto err;\n> \n> Am I reading this correctly? Presumably, `names_len` can be non-zero\n> here, right? And we now check for names==NULL to detect an allocation\n> failure...\n> \n> > + names[names_len] = reftable_strdup(p);\n> > + if (!names[names_len++])\n> > + goto err;\n> > }\n> > p = next + 1;\n> > }\n> >\n> > REFTABLE_REALLOC_ARRAY(names, names_len + 1);\n> > names[names_len] = NULL;\n> > - *namesp = names;\n> > +\n> > + return names;\n> > +\n> > +err:\n> > + for (size_t i = 0; i < names_len; i++)\n> > + reftable_free(names[i]);\n> \n> ... and then we potentially index into names[] because `names_len` is\n> non-zero, thus crash because `names` is NULL.\n> \n> > + reftable_free(names);\n> > + return NULL;\n> > }\n\nGood catch! I think we should queue something like the below on top of\nwhat we already have in `next` now.\n\nPatrick\n\n-- >8 --\n\nSubject: [PATCH] reftable/basics: fix segfault when growing `names` array fails\n\nWhen growing the `names` array fails we would end up with a `NULL`\npointer. This causes two problems:\n\n - We would run into a segfault because we try to free names that we\n have assigned to the array already.\n\n - We lose track of the old array and cannot free its contents.\n\nFix this issue by using a temporary variable. Like this we do not\nclobber the old array that we tried to reallocate, which will remain\nvalid when a call to realloc(3P) fails.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 8 +++++---\n 1 file changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex c8396dc525..df49cc8ef2 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -152,9 +152,11 @@ char **parse_names(char *buf, int size)\n \t\t\tnext = end;\n \t\t}\n \t\tif (p < next) {\n-\t\t\tREFTABLE_ALLOC_GROW(names, names_len + 1, names_cap);\n-\t\t\tif (!names)\n-\t\t\t\tgoto err;\n+\t\t\tchar **names_grown = names;\n+\t\t\tREFTABLE_ALLOC_GROW(names_grown, names_len + 1, names_cap);\n+\t\t\tif (!names_grown)\n+\t\t\t\tgoto err;\n+\t\t\tnames = names_grown;\n \n \t\t\tnames[names_len] = reftable_strdup(p);\n \t\t\tif (!names[names_len++])\n"},{"id":"504073","messageId":"CAPig+cQA=1nXyinWk736BjvMBYYYKXd6JsRDOf8P2=u7TYvF6A@mail.gmail.com","threadId":"62121","inReplyTo":"Zv92BRolDugwLuOn@pks.im","subject":"Re: [PATCH v5 05/25] reftable/basics: handle allocation failures in `parse_names()`","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2024-10-04T05:43:25Z","receivedAt":"2024-10-04T05:43:37Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, Oct 4, 2024 at 12:59 AM Patrick Steinhardt wrote:\n> On Wed, Oct 02, 2024 at 06:07:14PM -0400, Eric Sunshine wrote:\n> > Am I reading this correctly? Presumably, `names_len` can be non-zero\n> > here, right? And we now check for names==NULL to detect an allocation\n> > failure...\n> > ... and then we potentially index into names[] because `names_len` is\n> > non-zero, thus crash because `names` is NULL.\n>\n> Good catch! I think we should queue something like the below on top of\n> what we already have in `next` now.\n>\n> -- >8 --\n>\n> Subject: [PATCH] reftable/basics: fix segfault when growing `names` array fails\n>\n> When growing the `names` array fails we would end up with a `NULL`\n> pointer. This causes two problems:\n>\n> - We would run into a segfault because we try to free names that we\n> have assigned to the array already.\n>\n> - We lose track of the old array and cannot free its contents.\n>\n> Fix this issue by using a temporary variable. Like this we do not\n> clobber the old array that we tried to reallocate, which will remain\n> valid when a call to realloc(3P) fails.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> reftable/basics.c | 8 +++++---\n> 1 file changed, 5 insertions(+), 3 deletions(-)\n>\n> diff --git a/reftable/basics.c b/reftable/basics.c\n> @@ -152,9 +152,11 @@ char **parse_names(char *buf, int size)\n> - REFTABLE_ALLOC_GROW(names, names_len + 1, names_cap);\n> - if (!names)\n> - goto err;\n> + char **names_grown = names;\n> + REFTABLE_ALLOC_GROW(names_grown, names_len + 1, names_cap);\n> + if (!names_grown)\n> + goto err;\n> + names = names_grown;\n\nSeems sensible; it covers both problems described by the commit message.\n"}]}