{"thread":{"id":"62862","subject":"[PATCH 00/19] reftable: stop using \"git-compat-util.h\"","startedAt":"2025-01-27T13:04:18Z","lastAt":"2025-02-17T12:47:19Z","messageCount":123,"participants":["Patrick Steinhardt","Justin Tobler","Junio C Hamano","Johannes Sixt","Toon Claes","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":19},"messages":[{"id":"511242","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","threadId":"62862","inReplyTo":null,"subject":"[PATCH 00/19] reftable: stop using \"git-compat-util.h\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:08Z","receivedAt":"2025-01-27T13:04:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series is the final step to fully decouple the reftable\nlibrary from the rest of the Git codebase. The goal of this is to make\nthe library reusable by other projects like libgit2 by simply copying\nover the source files, making Git the canonical upstream for reftable\nfunctionality.\n\nThis patch series stops using all kinds of helpers exposed by our\n\"git-compat-util.h\" header and open-codes them instead. In order to keep\nus from using these helpers by accident the final step is to pull out\nPOSIX-related bits and pieces into a new \"compat/posix.h\" header, which\nthe reftable library then uses instead of \"git-compat-util.h\".\n\nThe series is built on top of master at 5f8f7081f7 (The third batch,\n2025-01-23) with ps/reftable-sign-compare at 33319b0976 (reftable:\naddress trivial -Wsign-compare warnings, 2025-01-20) merged into it.\nThere is a trivial merge conflict with ps/zlib-ng that can be solved\nlike this:\n\n diff --cc reftable/system.h\n index e4a8944a70,d02eacea8f..0000000000\n --- a/reftable/system.h\n +++ b/reftable/system.h\n @@@ -11,15 -11,9 +11,15 @@@ https://developers.google.com/open-sour\n \n /* This header glues the reftable library to the rest of Git */\n \n -#include \"git-compat-util.h\"\n +#include \"compat/posix.h\"\n - #include \n + #include \"compat/zlib-compat.h\"\n \n +/*\n + * Return a random 32 bit integer. This function is expected to return\n + * pre-seeded data.\n + */\n +uint32_t reftable_rand(void);\n +\n /*\n * An implementation-specific temporary file. By making this specific to the\n * implementation it becomes possible to tie temporary files into any kind of\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (19):\n reftable/stack: stop using `read_in_full()`\n reftable/stack: stop using `write_in_full()`\n reftable/blocksource: stop using `xmmap()`\n reftable/record: stop using `COPY_ARRAY()`\n reftable/record: stop using `BUG()` in `reftable_record_init()`\n reftable/record: don't `BUG()` in `reftable_record_cmp()`\n reftable: stop using `BUG()` in trivial cases\n reftable/basics: stop using `st_mult()` in array allocators\n reftable/basics: provide wrappers for big endian conversion\n reftable/reader: stop using `ARRAY_SIZE()` macro\n reftable/system: introduce `reftable_rand()`\n reftable/stack: stop using `sleep_millisec()`\n reftable/basics: stop using `SWAP()` macro\n reftable/basics: stop using `UNUSED` annotation\n compat/mingw: split out POSIX-related bits\n compat/msvc: split out POSIX-related bits\n git-compat-util.h: split out POSIX-emulating bits\n reftable: decouple from Git codebase by pulling in \"compat/posix.h\"\n Makefile: skip reftable library for Coccinelle\n\n Makefile | 2 +-\n compat/{mingw.c => mingw/compat-util.c} | 0\n compat/mingw/compat-util.h | 220 +++++++++++++\n compat/{mingw.h => mingw/posix.h} | 216 +------------\n compat/{msvc.c => msvc/compat-util.c} | 2 +-\n compat/msvc/compat-util.h | 7 +\n compat/{msvc.h => msvc/posix.h} | 8 +-\n compat/posix.h | 541 ++++++++++++++++++++++++++++++++\n config.mak.uname | 6 +-\n contrib/buildsystems/CMakeLists.txt | 2 +-\n git-compat-util.h | 535 +------------------------------\n meson.build | 8 +-\n reftable/basics.c | 19 --\n reftable/basics.h | 123 +++++++-\n reftable/block.c | 16 +-\n reftable/blocksource.c | 21 +-\n reftable/iter.c | 20 +-\n reftable/merged.c | 27 +-\n reftable/pq.c | 40 ++-\n reftable/pq.h | 2 +-\n reftable/reader.c | 33 +-\n reftable/record.c | 96 +++---\n reftable/record.h | 6 +-\n reftable/stack.c | 52 ++-\n reftable/system.c | 7 +\n reftable/system.h | 9 +-\n reftable/writer.c | 29 +-\n t/unit-tests/t-reftable-basics.c | 28 +-\n t/unit-tests/t-reftable-pq.c | 22 +-\n t/unit-tests/t-reftable-record.c | 42 ++-\n 30 files changed, 1214 insertions(+), 925 deletions(-)\n\n\n---\nbase-commit: 5f8f7081f7761acdf83d0a4c6819fe3d724f01d7\nchange-id: 20241119-pks-reftable-drop-git-compat-util-470f2bfde562\n\n"},{"id":"511243","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-2-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 02/19] reftable/stack: stop using `write_in_full()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:10Z","receivedAt":"2025-01-27T13:04:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Similar to the preceding commit, drop our use of `write_in_full()` and\nimplement a new wrapper `reftable_write_full()` that handles this logic\nfor us. This is done to reduce our dependency on the Git library.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 28 ++++++++++++++++++++++++----\n 1 file changed, 24 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 9490366795..5f155b344b 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -48,6 +48,25 @@ static int stack_fsync(const struct reftable_write_options *opts, int fd)\n \treturn fsync(fd);\n }\n \n+static ssize_t reftable_write_data(int fd, const void *data, size_t size)\n+{\n+\tsize_t total_written = 0;\n+\tconst char *p = data;\n+\n+\twhile (total_written < size) {\n+\t\tssize_t bytes_written = write(fd, p, size - total_written);\n+\t\tif (bytes_written < 0 && (errno == EAGAIN || errno == EINTR))\n+\t\t\tcontinue;\n+\t\tif (bytes_written < 0)\n+\t\t\treturn REFTABLE_IO_ERROR;\n+\n+\t\ttotal_written += bytes_written;\n+\t\tp += bytes_written;\n+\t}\n+\n+\treturn total_written;\n+}\n+\n struct fd_writer {\n \tconst struct reftable_write_options *opts;\n \tint fd;\n@@ -56,7 +75,7 @@ struct fd_writer {\n static ssize_t fd_writer_write(void *arg, const void *data, size_t sz)\n {\n \tstruct fd_writer *writer = arg;\n-\treturn write_in_full(writer->fd, data, sz);\n+\treturn reftable_write_data(writer->fd, data, sz);\n }\n \n static int fd_writer_flush(void *arg)\n@@ -784,7 +803,8 @@ int reftable_addition_commit(struct reftable_addition *add)\n \t\t\tgoto done;\n \t}\n \n-\terr = write_in_full(add->tables_list_lock.fd, table_list.buf, table_list.len);\n+\terr = reftable_write_data(add->tables_list_lock.fd,\n+\t\t\t\t table_list.buf, table_list.len);\n \treftable_buf_release(&table_list);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n@@ -1468,8 +1488,8 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\t\tgoto done;\n \t}\n \n-\terr = write_in_full(tables_list_lock.fd,\n-\t\t\t tables_list_buf.buf, tables_list_buf.len);\n+\terr = reftable_write_data(tables_list_lock.fd,\n+\t\t\t\t tables_list_buf.buf, tables_list_buf.len);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tunlink(new_table_path.buf);\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511245","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-1-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 01/19] reftable/stack: stop using `read_in_full()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:09Z","receivedAt":"2025-01-27T13:04:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"There is a single callsite of `read_in_full()` in the reftable library.\nOpen-code the function to reduce our dependency on the Git library.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 18 ++++++++++++++----\n 1 file changed, 14 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex f7c1845e15..9490366795 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -115,13 +115,16 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \n static int fd_read_lines(int fd, char ***namesp)\n {\n-\toff_t size = lseek(fd, 0, SEEK_END);\n \tchar *buf = NULL;\n \tint err = 0;\n+\toff_t size;\n+\n+\tsize = lseek(fd, 0, SEEK_END);\n \tif (size < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n+\n \terr = lseek(fd, 0, SEEK_SET);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n@@ -134,9 +137,16 @@ static int fd_read_lines(int fd, char ***namesp)\n \t\tgoto done;\n \t}\n \n-\tif (read_in_full(fd, buf, size) != size) {\n-\t\terr = REFTABLE_IO_ERROR;\n-\t\tgoto done;\n+\tfor (size_t total_read = 0; total_read < (size_t) size; ) {\n+\t\tssize_t bytes_read = read(fd, buf + total_read, size - total_read);\n+\t\tif (bytes_read < 0 && (errno == EAGAIN || errno == EINTR))\n+\t\t\tcontinue;\n+\t\tif (bytes_read < 0 || !bytes_read) {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\ttotal_read += bytes_read;\n \t}\n \tbuf[size] = 0;\n \n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511244","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-3-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 03/19] reftable/blocksource: stop using `xmmap()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:11Z","receivedAt":"2025-01-27T13:04:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We use `xmmap()` to map reftables into memory. This function has two\nproblems:\n\n - It causes us to die in case the mmap fails.\n\n - It ties us to the Git codebase.\n\nRefactor the code to use mmap(3p) instead with manual error checking.\nNote that this function may not be the system-provided mmap(3p), but may\npoint to our `git_mmap()` wrapper that emulates the syscall on systems\nthat do not have mmap(3p) available.\n\nFix `reftable_block_source_from_file()` to properly bubble up the error\ncode in case the map(3p) call fails.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/blocksource.c | 11 ++++++++---\n 1 file changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex bba4a45b98..02972c46f4 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -98,7 +98,7 @@ static struct reftable_block_source_vtable file_vtable = {\n int reftable_block_source_from_file(struct reftable_block_source *bs,\n \t\t\t\t const char *name)\n {\n-\tstruct file_block_source *p;\n+\tstruct file_block_source *p = NULL;\n \tstruct stat st;\n \tint fd, err;\n \n@@ -122,7 +122,12 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,\n \t}\n \n \tp->size = st.st_size;\n-\tp->data = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tp->data = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tif (p->data == MAP_FAILED) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tp->data = NULL;\n+\t\tgoto out;\n+\t}\n \n \tassert(!bs->ops);\n \tbs->ops = &file_vtable;\n@@ -135,5 +140,5 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,\n \t\tclose(fd);\n \tif (err < 0)\n \t\treftable_free(p);\n-\treturn 0;\n+\treturn err;\n }\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511246","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-4-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 04/19] reftable/record: stop using `COPY_ARRAY()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:12Z","receivedAt":"2025-01-27T13:04:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Drop our use of `COPY_ARRAY()`, replacing it with an open-coded variant\nthereof. This is done to reduce our dependency on the Git library.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/record.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 8919df8a4d..d1664c47ca 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -508,7 +508,8 @@ static int reftable_obj_record_copy_from(void *rec, const void *src_rec,\n \tif (!obj->offsets)\n \t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tobj->offset_len = src->offset_len;\n-\tCOPY_ARRAY(obj->offsets, src->offsets, src->offset_len);\n+\tif (src->offset_len)\n+\t\tmemcpy(obj->offsets, src->offsets, sizeof(*src->offsets) * src->offset_len);\n \n \treturn 0;\n }\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511247","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-5-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 05/19] reftable/record: stop using `BUG()` in `reftable_record_init()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:13Z","receivedAt":"2025-01-27T13:04:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're aborting the program via `BUG()` in case `reftable_record_init()`\nwas invoked with an unknown record type. This is bad because we may now\ndie in library code, and because it makes us depend on the Git codebase.\n\nRefactor the code such that `reftable_record_init()` can return an error\ncode to the caller. Adapt any callers accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/block.c | 4 +++-\n reftable/merged.c | 5 ++++-\n reftable/reader.c | 5 ++++-\n reftable/record.c | 8 ++++----\n reftable/record.h | 4 ++--\n t/unit-tests/t-reftable-pq.c | 6 +++---\n t/unit-tests/t-reftable-record.c | 2 +-\n 7 files changed, 21 insertions(+), 13 deletions(-)\n\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 8ac865ce78..255d566854 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -509,7 +509,9 @@ int block_iter_seek_key(struct block_iter *it, const struct block_reader *br,\n \tit->block_len = br->block_len;\n \tit->hash_size = br->hash_size;\n \n-\treftable_record_init(&rec, block_reader_type(br));\n+\terr = reftable_record_init(&rec, block_reader_type(br));\n+\tif (err < 0)\n+\t\tgoto done;\n \n \t/*\n \t * We're looking for the last entry less than the wanted key so that\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex e72b39e178..4156eec07f 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -253,7 +253,10 @@ int merged_table_init_iter(struct reftable_merged_table *mt,\n \t}\n \n \tfor (size_t i = 0; i < mt->readers_len; i++) {\n-\t\treftable_record_init(&subiters[i].rec, typ);\n+\t\tret = reftable_record_init(&subiters[i].rec, typ);\n+\t\tif (ret < 0)\n+\t\t\tgoto out;\n+\n \t\tret = reader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n \t\tif (ret < 0)\n \t\t\tgoto out;\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 3f2e4b2800..de6e6dd932 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -360,7 +360,10 @@ static int table_iter_seek_linear(struct table_iter *ti,\n \tstruct reftable_record rec;\n \tint err;\n \n-\treftable_record_init(&rec, reftable_record_type(want));\n+\terr = reftable_record_init(&rec, reftable_record_type(want));\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = reftable_record_key(want, &want_key);\n \tif (err < 0)\n \t\tgoto done;\ndiff --git a/reftable/record.c b/reftable/record.c\nindex d1664c47ca..31985bb977 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -1301,7 +1301,7 @@ reftable_record_vtable(struct reftable_record *rec)\n \tabort();\n }\n \n-void reftable_record_init(struct reftable_record *rec, uint8_t typ)\n+int reftable_record_init(struct reftable_record *rec, uint8_t typ)\n {\n \tmemset(rec, 0, sizeof(*rec));\n \trec->type = typ;\n@@ -1310,11 +1310,11 @@ void reftable_record_init(struct reftable_record *rec, uint8_t typ)\n \tcase BLOCK_TYPE_REF:\n \tcase BLOCK_TYPE_LOG:\n \tcase BLOCK_TYPE_OBJ:\n-\t\treturn;\n+\t\treturn 0;\n \tcase BLOCK_TYPE_INDEX:\n \t\treftable_buf_init(&rec->u.idx.last_key);\n-\t\treturn;\n+\t\treturn 0;\n \tdefault:\n-\t\tBUG(\"unhandled record type\");\n+\t\treturn REFTABLE_API_ERROR;\n \t}\n }\ndiff --git a/reftable/record.h b/reftable/record.h\nindex c7755a4d75..e1846c294b 100644\n--- a/reftable/record.h\n+++ b/reftable/record.h\n@@ -130,8 +130,8 @@ struct reftable_record {\n \t} u;\n };\n \n-/* Initialize the reftable record for the given type */\n-void reftable_record_init(struct reftable_record *rec, uint8_t typ);\n+/* Initialize the reftable record for the given type. */\n+int reftable_record_init(struct reftable_record *rec, uint8_t typ);\n \n /* see struct record_vtable */\n int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b);\ndiff --git a/t/unit-tests/t-reftable-pq.c b/t/unit-tests/t-reftable-pq.c\nindex f3f8a0cdf3..d8a4c283a1 100644\n--- a/t/unit-tests/t-reftable-pq.c\n+++ b/t/unit-tests/t-reftable-pq.c\n@@ -32,7 +32,7 @@ static void t_pq_record(void)\n \tchar *last = NULL;\n \n \tfor (i = 0; i < N; i++) {\n-\t\treftable_record_init(&recs[i], BLOCK_TYPE_REF);\n+\t\tcheck(!reftable_record_init(&recs[i], BLOCK_TYPE_REF));\n \t\trecs[i].u.ref.refname = xstrfmt(\"%02\"PRIuMAX, (uintmax_t)i);\n \t}\n \n@@ -72,7 +72,7 @@ static void t_pq_index(void)\n \tsize_t N = ARRAY_SIZE(recs), i;\n \n \tfor (i = 0; i < N; i++) {\n-\t\treftable_record_init(&recs[i], BLOCK_TYPE_REF);\n+\t\tcheck(!reftable_record_init(&recs[i], BLOCK_TYPE_REF));\n \t\trecs[i].u.ref.refname = (char *) \"refs/heads/master\";\n \t}\n \n@@ -111,7 +111,7 @@ static void t_merged_iter_pqueue_top(void)\n \tsize_t N = ARRAY_SIZE(recs), i;\n \n \tfor (i = 0; i < N; i++) {\n-\t\treftable_record_init(&recs[i], BLOCK_TYPE_REF);\n+\t\tcheck(!reftable_record_init(&recs[i], BLOCK_TYPE_REF));\n \t\trecs[i].u.ref.refname = (char *) \"refs/heads/master\";\n \t}\n \ndiff --git a/t/unit-tests/t-reftable-record.c b/t/unit-tests/t-reftable-record.c\nindex d49d2a2729..6540bd20e3 100644\n--- a/t/unit-tests/t-reftable-record.c\n+++ b/t/unit-tests/t-reftable-record.c\n@@ -17,7 +17,7 @@ static void t_copy(struct reftable_record *rec)\n \tuint8_t typ;\n \n \ttyp = reftable_record_type(rec);\n-\treftable_record_init(©, typ);\n+\tcheck(!reftable_record_init(©, typ));\n \treftable_record_copy_from(©, rec, REFTABLE_HASH_SIZE_SHA1);\n \t/* do it twice to catch memory leaks */\n \treftable_record_copy_from(©, rec, REFTABLE_HASH_SIZE_SHA1);\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511248","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-6-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 06/19] reftable/record: don't `BUG()` in `reftable_record_cmp()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:14Z","receivedAt":"2025-01-27T13:04:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The refatble library aborts with a bug in case `reftable_record_cmp()`\nis invoked with two records of differing types. This would cause the\nprogram to die without the caller being able to handle the error, which\nis not something we want in the context of library code. And it ties us\nto the Git codebase.\n\nRefactor the code such that `reftable_record_cmp()` returns an error\ncode separate from the actual comparison result. This requires us to\nalso adapt some callers up the callchain in a similar fashion.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 20 +++++++++++++++-----\n reftable/pq.c | 36 +++++++++++++++++++++++++++++-------\n reftable/pq.h | 2 +-\n reftable/record.c | 10 ++++++----\n reftable/record.h | 2 +-\n t/unit-tests/t-reftable-pq.c | 16 ++++++++++++----\n t/unit-tests/t-reftable-record.c | 40 ++++++++++++++++++++++++++++------------\n 7 files changed, 92 insertions(+), 34 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 4156eec07f..563864068c 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -66,8 +66,11 @@ static int merged_iter_seek(struct merged_iter *mi, struct reftable_record *want\n \tint err;\n \n \tmi->advance_index = -1;\n-\twhile (!merged_iter_pqueue_is_empty(mi->pq))\n-\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq)) {\n+\t\terr = merged_iter_pqueue_remove(&mi->pq, NULL);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n \n \tfor (size_t i = 0; i < mi->subiters_len; i++) {\n \t\terr = iterator_seek(&mi->subiters[i].iter, want);\n@@ -120,7 +123,9 @@ static int merged_iter_next_entry(struct merged_iter *mi,\n \tif (empty)\n \t\treturn 1;\n \n-\tentry = merged_iter_pqueue_remove(&mi->pq);\n+\terr = merged_iter_pqueue_remove(&mi->pq, &entry);\n+\tif (err < 0)\n+\t\treturn err;\n \n \t/*\n \t One can also use reftable as datacenter-local storage, where the ref\n@@ -134,11 +139,16 @@ static int merged_iter_next_entry(struct merged_iter *mi,\n \t\tstruct pq_entry top = merged_iter_pqueue_top(mi->pq);\n \t\tint cmp;\n \n-\t\tcmp = reftable_record_cmp(top.rec, entry.rec);\n+\t\terr = reftable_record_cmp(top.rec, entry.rec, &cmp);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n \t\tif (cmp > 0)\n \t\t\tbreak;\n \n-\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\t\terr = merged_iter_pqueue_remove(&mi->pq, NULL);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\n \t\terr = merged_iter_advance_subiter(mi, top.index);\n \t\tif (err < 0)\n \t\t\treturn err;\ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex 5591e875e1..ef8035cfd9 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -15,13 +15,18 @@ license that can be found in the LICENSE file or at\n \n int pq_less(struct pq_entry *a, struct pq_entry *b)\n {\n-\tint cmp = reftable_record_cmp(a->rec, b->rec);\n+\tint cmp, err;\n+\n+\terr = reftable_record_cmp(a->rec, b->rec, &cmp);\n+\tif (err < 0)\n+\t\treturn err;\n+\n \tif (cmp == 0)\n \t\treturn a->index > b->index;\n \treturn cmp < 0;\n }\n \n-struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n+int merged_iter_pqueue_remove(struct merged_iter_pqueue *pq, struct pq_entry *out)\n {\n \tsize_t i = 0;\n \tstruct pq_entry e = pq->heap[0];\n@@ -32,17 +37,34 @@ struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n \t\tsize_t min = i;\n \t\tsize_t j = 2 * i + 1;\n \t\tsize_t k = 2 * i + 2;\n-\t\tif (j < pq->len && pq_less(&pq->heap[j], &pq->heap[i]))\n-\t\t\tmin = j;\n-\t\tif (k < pq->len && pq_less(&pq->heap[k], &pq->heap[min]))\n-\t\t\tmin = k;\n+\t\tint cmp;\n+\n+\t\tif (j < pq->len) {\n+\t\t\tcmp = pq_less(&pq->heap[j], &pq->heap[i]);\n+\t\t\tif (cmp < 0)\n+\t\t\t\treturn -1;\n+\t\t\telse if (cmp)\n+\t\t\t\tmin = j;\n+\t\t}\n+\n+\t\tif (k < pq->len) {\n+\t\t\tcmp = pq_less(&pq->heap[k], &pq->heap[min]);\n+\t\t\tif (cmp < 0)\n+\t\t\t\treturn -1;\n+\t\t\telse if (cmp)\n+\t\t\t\tmin = k;\n+\t\t}\n+\n \t\tif (min == i)\n \t\t\tbreak;\n \t\tSWAP(pq->heap[i], pq->heap[min]);\n \t\ti = min;\n \t}\n \n-\treturn e;\n+\tif (out)\n+\t\t*out = e;\n+\n+\treturn 0;\n }\n \n int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e)\ndiff --git a/reftable/pq.h b/reftable/pq.h\nindex 83c062eeca..ff39016445 100644\n--- a/reftable/pq.h\n+++ b/reftable/pq.h\n@@ -22,7 +22,7 @@ struct merged_iter_pqueue {\n \tsize_t cap;\n };\n \n-struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n+int merged_iter_pqueue_remove(struct merged_iter_pqueue *pq, struct pq_entry *out);\n int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e);\n void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n int pq_less(struct pq_entry *a, struct pq_entry *b);\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 31985bb977..da550494a2 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -1190,12 +1190,14 @@ int reftable_record_is_deletion(struct reftable_record *rec)\n \t\treftable_record_data(rec));\n }\n \n-int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b)\n+int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b,\n+\t\t\tint *cmp)\n {\n \tif (a->type != b->type)\n-\t\tBUG(\"cannot compare reftable records of different type\");\n-\treturn reftable_record_vtable(a)->cmp(\n-\t\treftable_record_data(a), reftable_record_data(b));\n+\t\treturn -1;\n+\t*cmp = reftable_record_vtable(a)->cmp(reftable_record_data(a),\n+\t\t\t\t\t reftable_record_data(b));\n+\treturn 0;\n }\n \n int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, uint32_t hash_size)\ndiff --git a/reftable/record.h b/reftable/record.h\nindex e1846c294b..867810a932 100644\n--- a/reftable/record.h\n+++ b/reftable/record.h\n@@ -134,7 +134,7 @@ struct reftable_record {\n int reftable_record_init(struct reftable_record *rec, uint8_t typ);\n \n /* see struct record_vtable */\n-int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b);\n+int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b, int *cmp);\n int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, uint32_t hash_size);\n int reftable_record_key(struct reftable_record *rec, struct reftable_buf *dest);\n int reftable_record_copy_from(struct reftable_record *rec,\ndiff --git a/t/unit-tests/t-reftable-pq.c b/t/unit-tests/t-reftable-pq.c\nindex d8a4c283a1..c128fe8616 100644\n--- a/t/unit-tests/t-reftable-pq.c\n+++ b/t/unit-tests/t-reftable-pq.c\n@@ -21,7 +21,9 @@ static void merged_iter_pqueue_check(const struct merged_iter_pqueue *pq)\n \n static int pq_entry_equal(struct pq_entry *a, struct pq_entry *b)\n {\n-\treturn !reftable_record_cmp(a->rec, b->rec) && (a->index == b->index);\n+\tint cmp;\n+\tcheck(!reftable_record_cmp(a->rec, b->rec, &cmp));\n+\treturn !cmp && (a->index == b->index);\n }\n \n static void t_pq_record(void)\n@@ -49,7 +51,9 @@ static void t_pq_record(void)\n \n \twhile (!merged_iter_pqueue_is_empty(pq)) {\n \t\tstruct pq_entry top = merged_iter_pqueue_top(pq);\n-\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct pq_entry e;\n+\n+\t\tcheck(!merged_iter_pqueue_remove(&pq, &e));\n \t\tmerged_iter_pqueue_check(&pq);\n \n \t\tcheck(pq_entry_equal(&top, &e));\n@@ -90,7 +94,9 @@ static void t_pq_index(void)\n \n \tfor (i = N - 1; i > 0; i--) {\n \t\tstruct pq_entry top = merged_iter_pqueue_top(pq);\n-\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct pq_entry e;\n+\n+\t\tcheck(!merged_iter_pqueue_remove(&pq, &e));\n \t\tmerged_iter_pqueue_check(&pq);\n \n \t\tcheck(pq_entry_equal(&top, &e));\n@@ -129,7 +135,9 @@ static void t_merged_iter_pqueue_top(void)\n \n \tfor (i = N - 1; i > 0; i--) {\n \t\tstruct pq_entry top = merged_iter_pqueue_top(pq);\n-\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct pq_entry e;\n+\n+\t\tcheck(!merged_iter_pqueue_remove(&pq, &e));\n \n \t\tmerged_iter_pqueue_check(&pq);\n \t\tcheck(pq_entry_equal(&top, &e));\ndiff --git a/t/unit-tests/t-reftable-record.c b/t/unit-tests/t-reftable-record.c\nindex 6540bd20e3..5954966373 100644\n--- a/t/unit-tests/t-reftable-record.c\n+++ b/t/unit-tests/t-reftable-record.c\n@@ -100,16 +100,20 @@ static void t_reftable_ref_record_comparison(void)\n \t\t\t.u.ref.value.symref = (char *) \"refs/heads/master\",\n \t\t},\n \t};\n+\tint cmp;\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \n \tin[1].u.ref.value_type = in[0].u.ref.value_type;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n }\n \n static void t_reftable_ref_record_compare_name(void)\n@@ -209,17 +213,20 @@ static void t_reftable_log_record_comparison(void)\n \t\t\t.u.log.update_index = 22,\n \t\t},\n \t};\n+\tint cmp;\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \t/* comparison should be reversed for equal keys, because\n \t * comparison is now performed on the basis of update indices */\n-\tcheck_int(reftable_record_cmp(&in[0], &in[1]), <, 0);\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck_int(cmp, <, 0);\n \n \tin[1].u.log.update_index = in[0].u.log.update_index;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n }\n \n static void t_reftable_log_record_compare_key(void)\n@@ -396,16 +403,20 @@ static void t_reftable_obj_record_comparison(void)\n \t\t\t.u.obj.hash_prefix_len = 5,\n \t\t},\n \t};\n+\tint cmp;\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \n \tin[1].u.obj.offset_len = in[0].u.obj.offset_len;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n }\n \n static void t_reftable_obj_record_roundtrip(void)\n@@ -486,19 +497,24 @@ static void t_reftable_index_record_comparison(void)\n \t\t\t.u.idx.last_key = REFTABLE_BUF_INIT,\n \t\t},\n \t};\n+\tint cmp;\n+\n \tcheck(!reftable_buf_addstr(&in[0].u.idx.last_key, \"refs/heads/master\"));\n \tcheck(!reftable_buf_addstr(&in[1].u.idx.last_key, \"refs/heads/master\"));\n \tcheck(!reftable_buf_addstr(&in[2].u.idx.last_key, \"refs/heads/branch\"));\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \n \tin[1].u.idx.offset = in[0].u.idx.offset;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tfor (size_t i = 0; i < ARRAY_SIZE(in); i++)\n \t\treftable_record_release(&in[i]);\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511249","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-7-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 07/19] reftable: stop using `BUG()` in trivial cases","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:15Z","receivedAt":"2025-01-27T13:04:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Stop using `BUG()` in the remaining trivial cases that we still have in\nthe reftable library. Instead of aborting the program, we'll now bubble\nup a `REFTABLE_API_ERROR` to indicate misuse of the calling conventions.\n\nNote that in both `reftable_reader_{inc,dec}ref()` we simply stop\ncalling `BUG()` altogether. The only situation where the counter should\nbe zero is when the structure has already been free'd anyway, so we\nwould run into undefined behaviour regardless of whether we try to abort\nthe program or not.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/iter.c | 3 +--\n reftable/reader.c | 4 ----\n reftable/writer.c | 5 ++---\n 3 files changed, 3 insertions(+), 9 deletions(-)\n\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex 86e801ca9f..b2ffb09c16 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -146,8 +146,7 @@ static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n static int indexed_table_ref_iter_seek(void *p UNUSED,\n \t\t\t\t struct reftable_record *want UNUSED)\n {\n-\tBUG(\"seeking indexed table is not supported\");\n-\treturn -1;\n+\treturn REFTABLE_API_ERROR;\n }\n \n static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex de6e6dd932..36a5633ede 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -677,8 +677,6 @@ int reftable_reader_new(struct reftable_reader **out,\n \n void reftable_reader_incref(struct reftable_reader *r)\n {\n-\tif (!r->refcount)\n-\t\tBUG(\"cannot increment ref counter of dead reader\");\n \tr->refcount++;\n }\n \n@@ -686,8 +684,6 @@ void reftable_reader_decref(struct reftable_reader *r)\n {\n \tif (!r)\n \t\treturn;\n-\tif (!r->refcount)\n-\t\tBUG(\"cannot decrement ref counter of dead reader\");\n \tif (--r->refcount)\n \t\treturn;\n \tblock_source_close(&r->source);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 91d6629486..155863ee5f 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -158,7 +158,7 @@ int reftable_writer_new(struct reftable_writer **out,\n \t\topts = *_opts;\n \toptions_set_defaults(&opts);\n \tif (opts.block_size >= (1 << 24))\n-\t\tBUG(\"configured block size exceeds 16MB\");\n+\t\treturn REFTABLE_API_ERROR;\n \n \treftable_buf_init(&wp->block_writer_data.last_key);\n \treftable_buf_init(&wp->last_key);\n@@ -289,8 +289,7 @@ static int writer_add_record(struct reftable_writer *w,\n \t}\n \n \tif (block_writer_type(w->block_writer) != reftable_record_type(rec))\n-\t\tBUG(\"record of type %d added to writer of type %d\",\n-\t\t reftable_record_type(rec), block_writer_type(w->block_writer));\n+\t\treturn REFTABLE_API_ERROR;\n \n \t/*\n \t * Try to add the record to the writer. If this succeeds then we're\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511250","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-8-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 08/19] reftable/basics: stop using `st_mult()` in array allocators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:16Z","receivedAt":"2025-01-27T13:04:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're using `st_mult()` as part of our macro helpers that allocate\narrays. This is bad due two two reasons:\n\n - `st_mult()` causes us to die in case the multiplication overflows.\n\n - `st_mult()` ties us to the Git codebase.\n\nRefactor the code to instead detect overflows manually and return an\nerror in such cases.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 36 ++++++++++++++++++++++++++++++++----\n 1 file changed, 32 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex a2a010a0e1..646f8d67f2 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -117,18 +117,46 @@ void reftable_free(void *p);\n void *reftable_calloc(size_t nelem, size_t elsize);\n char *reftable_strdup(const char *str);\n \n-#define REFTABLE_ALLOC_ARRAY(x, alloc) (x) = reftable_malloc(st_mult(sizeof(*(x)), (alloc)))\n+static inline int reftable_alloc_size(size_t nelem, size_t elsize, size_t *out)\n+{\n+\tif (nelem && elsize > SIZE_MAX / nelem)\n+\t\treturn -1;\n+\t*out = nelem * elsize;\n+\treturn 0;\n+}\n+\n+#define REFTABLE_ALLOC_ARRAY(x, alloc) do { \\\n+\t\tsize_t alloc_size; \\\n+\t\tif (reftable_alloc_size(sizeof(*(x)), (alloc), &alloc_size) < 0) { \\\n+\t\t\terrno = ENOMEM; \\\n+\t\t\t(x) = NULL; \\\n+\t\t} else { \\\n+\t\t\t(x) = reftable_malloc(alloc_size); \\\n+\t\t} \\\n+\t} while (0)\n #define REFTABLE_CALLOC_ARRAY(x, alloc) (x) = reftable_calloc((alloc), sizeof(*(x)))\n-#define REFTABLE_REALLOC_ARRAY(x, alloc) (x) = reftable_realloc((x), st_mult(sizeof(*(x)), (alloc)))\n+#define REFTABLE_REALLOC_ARRAY(x, alloc) do { \\\n+\t\tsize_t alloc_size; \\\n+\t\tif (reftable_alloc_size(sizeof(*(x)), (alloc), &alloc_size) < 0) { \\\n+\t\t\terrno = ENOMEM; \\\n+\t\t\t(x) = NULL; \\\n+\t\t} else { \\\n+\t\t\t(x) = reftable_realloc((x), alloc_size); \\\n+\t\t} \\\n+\t} while (0)\n \n static inline void *reftable_alloc_grow(void *p, size_t nelem, size_t elsize,\n \t\t\t\t\tsize_t *allocp)\n {\n \tvoid *new_p;\n-\tsize_t alloc = *allocp * 2 + 1;\n+\tsize_t alloc = *allocp * 2 + 1, alloc_bytes;\n \tif (alloc < nelem)\n \t\talloc = nelem;\n-\tnew_p = reftable_realloc(p, st_mult(elsize, alloc));\n+\tif (reftable_alloc_size(elsize, alloc, &alloc_bytes) < 0) {\n+\t\terrno = ENOMEM;\n+\t\treturn p;\n+\t}\n+\tnew_p = reftable_realloc(p, alloc_bytes);\n \tif (!new_p)\n \t\treturn p;\n \t*allocp = alloc;\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511251","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-9-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 09/19] reftable/basics: provide wrappers for big endian conversion","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:17Z","receivedAt":"2025-01-27T13:04:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're using a mixture of big endian conversion functions provided by\nboth the reftable library, but also by the Git codebase. Refactor the\ncode so that we exclusively use reftable-provided wrappers in order to\nuntangle us from the Git codebase.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 19 ----------\n reftable/basics.h | 76 ++++++++++++++++++++++++++++++++++++++--\n reftable/block.c | 12 +++----\n reftable/reader.c | 22 ++++++------\n reftable/record.c | 8 ++---\n reftable/writer.c | 20 +++++------\n t/unit-tests/t-reftable-basics.c | 28 ++++++++++++---\n 7 files changed, 127 insertions(+), 58 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 3b5ea27bbd..8c4a4433e4 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -147,25 +147,6 @@ char *reftable_buf_detach(struct reftable_buf *buf)\n \treturn result;\n }\n \n-void put_be24(uint8_t *out, uint32_t i)\n-{\n-\tout[0] = (uint8_t)((i >> 16) & 0xff);\n-\tout[1] = (uint8_t)((i >> 8) & 0xff);\n-\tout[2] = (uint8_t)(i & 0xff);\n-}\n-\n-uint32_t get_be24(uint8_t *in)\n-{\n-\treturn (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 |\n-\t (uint32_t)(in[2]);\n-}\n-\n-void put_be16(uint8_t *out, uint16_t i)\n-{\n-\tout[0] = (uint8_t)((i >> 8) & 0xff);\n-\tout[1] = (uint8_t)(i & 0xff);\n-}\n-\n size_t binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n {\n \tsize_t lo = 0;\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 646f8d67f2..c1ddbaec3f 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -76,9 +76,79 @@ char *reftable_buf_detach(struct reftable_buf *buf);\n \n /* Bigendian en/decoding of integers */\n \n-void put_be24(uint8_t *out, uint32_t i);\n-uint32_t get_be24(uint8_t *in);\n-void put_be16(uint8_t *out, uint16_t i);\n+static inline void reftable_put_be16(void *out, uint16_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 8) & 0xff);\n+\tp[1] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline void reftable_put_be24(void *out, uint32_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 16) & 0xff);\n+\tp[1] = (uint8_t)((i >> 8) & 0xff);\n+\tp[2] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline void reftable_put_be32(void *out, uint32_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 24) & 0xff);\n+\tp[1] = (uint8_t)((i >> 16) & 0xff);\n+\tp[2] = (uint8_t)((i >> 8) & 0xff);\n+\tp[3] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline void reftable_put_be64(void *out, uint64_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 56) & 0xff);\n+\tp[1] = (uint8_t)((i >> 48) & 0xff);\n+\tp[2] = (uint8_t)((i >> 40) & 0xff);\n+\tp[3] = (uint8_t)((i >> 32) & 0xff);\n+\tp[4] = (uint8_t)((i >> 24) & 0xff);\n+\tp[5] = (uint8_t)((i >> 16) & 0xff);\n+\tp[6] = (uint8_t)((i >> 8) & 0xff);\n+\tp[7] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline uint16_t reftable_get_be16(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint16_t)(p[0]) << 8 |\n+\t (uint16_t)(p[1]) << 0;\n+}\n+\n+static inline uint32_t reftable_get_be24(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint32_t)(p[0]) << 16 |\n+\t (uint32_t)(p[1]) << 8 |\n+\t (uint32_t)(p[2]) << 0;\n+}\n+\n+static inline uint32_t reftable_get_be32(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint32_t)(p[0]) << 24 |\n+\t (uint32_t)(p[1]) << 16 |\n+\t (uint32_t)(p[2]) << 8|\n+\t (uint32_t)(p[3]) << 0;\n+}\n+\n+static inline uint64_t reftable_get_be64(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint64_t)(p[0]) << 56 |\n+\t (uint64_t)(p[1]) << 48 |\n+\t (uint64_t)(p[2]) << 40 |\n+\t (uint64_t)(p[3]) << 32 |\n+\t (uint64_t)(p[4]) << 24 |\n+\t (uint64_t)(p[5]) << 16 |\n+\t (uint64_t)(p[6]) << 8 |\n+\t (uint64_t)(p[7]) << 0;\n+}\n \n /*\n * find smallest index i in [0, sz) at which `f(i) > 0`, assuming that f is\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 255d566854..373908807e 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -148,13 +148,13 @@ int block_writer_add(struct block_writer *w, struct reftable_record *rec)\n int block_writer_finish(struct block_writer *w)\n {\n \tfor (uint32_t i = 0; i < w->restart_len; i++) {\n-\t\tput_be24(w->block + w->next, w->restarts[i]);\n+\t\treftable_put_be24(w->block + w->next, w->restarts[i]);\n \t\tw->next += 3;\n \t}\n \n-\tput_be16(w->block + w->next, w->restart_len);\n+\treftable_put_be16(w->block + w->next, w->restart_len);\n \tw->next += 2;\n-\tput_be24(w->block + 1 + w->header_off, w->next);\n+\treftable_put_be24(w->block + 1 + w->header_off, w->next);\n \n \t/*\n \t * Log records are stored zlib-compressed. Note that the compression\n@@ -216,7 +216,7 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,\n {\n \tuint32_t full_block_size = table_block_size;\n \tuint8_t typ = block->data[header_off];\n-\tuint32_t sz = get_be24(block->data + header_off + 1);\n+\tuint32_t sz = reftable_get_be24(block->data + header_off + 1);\n \tint err = 0;\n \tuint16_t restart_count = 0;\n \tuint32_t restart_start = 0;\n@@ -300,7 +300,7 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,\n \t\tfull_block_size = sz;\n \t}\n \n-\trestart_count = get_be16(block->data + sz - 2);\n+\trestart_count = reftable_get_be16(block->data + sz - 2);\n \trestart_start = sz - 2 - 3 * restart_count;\n \trestart_bytes = block->data + restart_start;\n \n@@ -355,7 +355,7 @@ int block_reader_first_key(const struct block_reader *br, struct reftable_buf *k\n \n static uint32_t block_reader_restart_offset(const struct block_reader *br, size_t idx)\n {\n-\treturn get_be24(br->restart_bytes + 3 * idx);\n+\treturn reftable_get_be24(br->restart_bytes + 3 * idx);\n }\n \n void block_iter_seek_start(struct block_iter *it, const struct block_reader *br)\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 36a5633ede..bf07a0a586 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -101,18 +101,18 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n \t}\n \n \tf++;\n-\tr->block_size = get_be24(f);\n+\tr->block_size = reftable_get_be24(f);\n \n \tf += 3;\n-\tr->min_update_index = get_be64(f);\n+\tr->min_update_index = reftable_get_be64(f);\n \tf += 8;\n-\tr->max_update_index = get_be64(f);\n+\tr->max_update_index = reftable_get_be64(f);\n \tf += 8;\n \n \tif (r->version == 1) {\n \t\tr->hash_id = REFTABLE_HASH_SHA1;\n \t} else {\n-\t\tswitch (get_be32(f)) {\n+\t\tswitch (reftable_get_be32(f)) {\n \t\tcase REFTABLE_FORMAT_ID_SHA1:\n \t\t\tr->hash_id = REFTABLE_HASH_SHA1;\n \t\t\tbreak;\n@@ -127,24 +127,24 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n \t\tf += 4;\n \t}\n \n-\tr->ref_offsets.index_offset = get_be64(f);\n+\tr->ref_offsets.index_offset = reftable_get_be64(f);\n \tf += 8;\n \n-\tr->obj_offsets.offset = get_be64(f);\n+\tr->obj_offsets.offset = reftable_get_be64(f);\n \tf += 8;\n \n \tr->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1);\n \tr->obj_offsets.offset >>= 5;\n \n-\tr->obj_offsets.index_offset = get_be64(f);\n+\tr->obj_offsets.index_offset = reftable_get_be64(f);\n \tf += 8;\n-\tr->log_offsets.offset = get_be64(f);\n+\tr->log_offsets.offset = reftable_get_be64(f);\n \tf += 8;\n-\tr->log_offsets.index_offset = get_be64(f);\n+\tr->log_offsets.index_offset = reftable_get_be64(f);\n \tf += 8;\n \n \tcomputed_crc = crc32(0, footer, f - footer);\n-\tfile_crc = get_be32(f);\n+\tfile_crc = reftable_get_be32(f);\n \tf += 4;\n \tif (computed_crc != file_crc) {\n \t\terr = REFTABLE_FORMAT_ERROR;\n@@ -214,7 +214,7 @@ static int32_t extract_block_size(uint8_t *data, uint8_t *typ, uint64_t off,\n \n \t*typ = data[0];\n \tif (reftable_is_block_type(*typ)) {\n-\t\tresult = get_be24(data + 1);\n+\t\tresult = reftable_get_be24(data + 1);\n \t}\n \treturn result;\n }\ndiff --git a/reftable/record.c b/reftable/record.c\nindex da550494a2..286659c098 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -684,7 +684,7 @@ static int reftable_log_record_key(const void *r, struct reftable_buf *dest)\n \t\treturn err;\n \n \tts = (~ts) - rec->update_index;\n-\tput_be64(&i64[0], ts);\n+\treftable_put_be64(&i64[0], ts);\n \n \terr = reftable_buf_add(dest, i64, sizeof(i64));\n \tif (err < 0)\n@@ -809,7 +809,7 @@ static int reftable_log_record_encode(const void *rec, struct string_view s,\n \tif (s.len < 2)\n \t\treturn -1;\n \n-\tput_be16(s.buf, r->value.update.tz_offset);\n+\treftable_put_be16(s.buf, r->value.update.tz_offset);\n \tstring_view_consume(&s, 2);\n \n \tn = encode_string(\n@@ -841,7 +841,7 @@ static int reftable_log_record_decode(void *rec, struct reftable_buf key,\n \t}\n \n \tmemcpy(r->refname, key.buf, key.len - 8);\n-\tts = get_be64(key.buf + key.len - 8);\n+\tts = reftable_get_be64((unsigned char *)key.buf + key.len - 8);\n \n \tr->update_index = (~max) - ts;\n \n@@ -932,7 +932,7 @@ static int reftable_log_record_decode(void *rec, struct reftable_buf key,\n \t\tgoto done;\n \t}\n \n-\tr->value.update.tz_offset = get_be16(in.buf);\n+\tr->value.update.tz_offset = reftable_get_be16(in.buf);\n \tstring_view_consume(&in, 2);\n \n \tn = decode_string(scratch, in);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 155863ee5f..5961698311 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -99,9 +99,9 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \n \tdest[4] = writer_version(w);\n \n-\tput_be24(dest + 5, w->opts.block_size);\n-\tput_be64(dest + 8, w->min_update_index);\n-\tput_be64(dest + 16, w->max_update_index);\n+\treftable_put_be24(dest + 5, w->opts.block_size);\n+\treftable_put_be64(dest + 8, w->min_update_index);\n+\treftable_put_be64(dest + 16, w->max_update_index);\n \tif (writer_version(w) == 2) {\n \t\tuint32_t hash_id;\n \n@@ -116,7 +116,7 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \t\t\treturn -1;\n \t\t}\n \n-\t\tput_be32(dest + 24, hash_id);\n+\t\treftable_put_be32(dest + 24, hash_id);\n \t}\n \n \treturn header_size(writer_version(w));\n@@ -717,19 +717,19 @@ int reftable_writer_close(struct reftable_writer *w)\n \t}\n \n \tp += writer_write_header(w, footer);\n-\tput_be64(p, w->stats.ref_stats.index_offset);\n+\treftable_put_be64(p, w->stats.ref_stats.index_offset);\n \tp += 8;\n-\tput_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n+\treftable_put_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n \tp += 8;\n-\tput_be64(p, w->stats.obj_stats.index_offset);\n+\treftable_put_be64(p, w->stats.obj_stats.index_offset);\n \tp += 8;\n \n-\tput_be64(p, w->stats.log_stats.offset);\n+\treftable_put_be64(p, w->stats.log_stats.offset);\n \tp += 8;\n-\tput_be64(p, w->stats.log_stats.index_offset);\n+\treftable_put_be64(p, w->stats.log_stats.index_offset);\n \tp += 8;\n \n-\tput_be32(p, crc32(0, footer, p - footer));\n+\treftable_put_be32(p, crc32(0, footer, p - footer));\n \tp += 4;\n \n \terr = w->flush(w->write_arg);\ndiff --git a/t/unit-tests/t-reftable-basics.c b/t/unit-tests/t-reftable-basics.c\nindex 9ba7eb05ad..c9e751e49e 100644\n--- a/t/unit-tests/t-reftable-basics.c\n+++ b/t/unit-tests/t-reftable-basics.c\n@@ -128,12 +128,30 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \t\treftable_buf_release(&b);\n \t}\n \n-\tif_test (\"put_be24 and get_be24 work\") {\n+\tif_test (\"reftable_put_be64 and reftable_get_be64 work\") {\n+\t\tuint64_t in = 0x1122334455667788;\n+\t\tuint8_t dest[8];\n+\t\tuint64_t out;\n+\t\treftable_put_be64(dest, in);\n+\t\tout = reftable_get_be64(dest);\n+\t\tcheck_int(in, ==, out);\n+\t}\n+\n+\tif_test (\"reftable_put_be32 and reftable_get_be32 work\") {\n+\t\tuint32_t in = 0x11223344;\n+\t\tuint8_t dest[4];\n+\t\tuint32_t out;\n+\t\treftable_put_be32(dest, in);\n+\t\tout = reftable_get_be32(dest);\n+\t\tcheck_int(in, ==, out);\n+\t}\n+\n+\tif_test (\"reftable_put_be24 and reftable_get_be24 work\") {\n \t\tuint32_t in = 0x112233;\n \t\tuint8_t dest[3];\n \t\tuint32_t out;\n-\t\tput_be24(dest, in);\n-\t\tout = get_be24(dest);\n+\t\treftable_put_be24(dest, in);\n+\t\tout = reftable_get_be24(dest);\n \t\tcheck_int(in, ==, out);\n \t}\n \n@@ -141,8 +159,8 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \t\tuint32_t in = 0xfef1;\n \t\tuint8_t dest[3];\n \t\tuint32_t out;\n-\t\tput_be16(dest, in);\n-\t\tout = get_be16(dest);\n+\t\treftable_put_be16(dest, in);\n+\t\tout = reftable_get_be16(dest);\n \t\tcheck_int(in, ==, out);\n \t}\n \n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511252","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-10-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 10/19] reftable/reader: stop using `ARRAY_SIZE()` macro","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:18Z","receivedAt":"2025-01-27T13:04:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have a single user of the `ARRAY_SIZE()` macro in the reftable\nreader. Drop its use to reduce our dependence on the Git codebase.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/reader.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex bf07a0a586..c3a3674665 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -849,7 +849,7 @@ int reftable_reader_print_blocks(const char *tablename)\n \tprintf(\"header:\\n\");\n \tprintf(\" block_size: %d\\n\", r->block_size);\n \n-\tfor (i = 0; i < ARRAY_SIZE(sections); i++) {\n+\tfor (i = 0; i < sizeof(sections) / sizeof(*sections); i++) {\n \t\terr = table_iter_seek_start(&ti, sections[i].type, 0);\n \t\tif (err < 0)\n \t\t\tgoto done;\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511253","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-11-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 11/19] reftable/system: introduce `reftable_rand()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:19Z","receivedAt":"2025-01-27T13:04:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new system-level `reftable_rand()` function that generates a\nsingle unsigned integer for us. The implementation of this function is\nto be provided by the calling codebase, which allows us to more easily\nhook into pre-seeded random number generators.\n\nAdapt the two callsites where we generated random data.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 4 ++--\n reftable/system.c | 5 +++++\n reftable/system.h | 6 ++++++\n 3 files changed, 13 insertions(+), 2 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 5f155b344b..82bdf5169f 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -523,7 +523,7 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\tclose(fd);\n \t\tfd = -1;\n \n-\t\tdelay = delay + (delay * git_rand(CSPRNG_BYTES_INSECURE)) / UINT32_MAX + 1;\n+\t\tdelay = delay + (delay * reftable_rand()) / UINT32_MAX + 1;\n \t\tsleep_millisec(delay);\n \t}\n \n@@ -688,7 +688,7 @@ int reftable_stack_add(struct reftable_stack *st,\n static int format_name(struct reftable_buf *dest, uint64_t min, uint64_t max)\n {\n \tchar buf[100];\n-\tuint32_t rnd = git_rand(CSPRNG_BYTES_INSECURE);\n+\tuint32_t rnd = reftable_rand();\n \tsnprintf(buf, sizeof(buf), \"0x%012\" PRIx64 \"-0x%012\" PRIx64 \"-%08x\",\n \t\t min, max, rnd);\n \treftable_buf_reset(dest);\ndiff --git a/reftable/system.c b/reftable/system.c\nindex adf8e4d30b..e25ccc0da3 100644\n--- a/reftable/system.c\n+++ b/reftable/system.c\n@@ -4,6 +4,11 @@\n #include \"../lockfile.h\"\n #include \"../tempfile.h\"\n \n+uint32_t reftable_rand(void)\n+{\n+\treturn git_rand(CSPRNG_BYTES_INSECURE);\n+}\n+\n int tmpfile_from_pattern(struct reftable_tmpfile *out, const char *pattern)\n {\n \tstruct tempfile *tempfile;\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 7d5f803eeb..3bd4a4e322 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -13,6 +13,12 @@ license that can be found in the LICENSE file or at\n \n #include \"git-compat-util.h\"\n \n+/*\n+ * Return a random 32 bit integer. This function is expected to return\n+ * pre-seeded data.\n+ */\n+uint32_t reftable_rand(void);\n+\n /*\n * An implementation-specific temporary file. By making this specific to the\n * implementation it becomes possible to tie temporary files into any kind of\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511254","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-12-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 12/19] reftable/stack: stop using `sleep_millisec()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:20Z","receivedAt":"2025-01-27T13:04:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor our use of `sleep_millisec()` by open-coding it with poll(3p),\nwhich is the current implementation of this function. Ideally, we'd use\na more direct way to sleep, but there is no equivalent to sleep(3p) that\nwould accept milliseconds as input.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 82bdf5169f..8e9836aef3 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -524,7 +524,7 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\tfd = -1;\n \n \t\tdelay = delay + (delay * reftable_rand()) / UINT32_MAX + 1;\n-\t\tsleep_millisec(delay);\n+\t\tpoll(NULL, 0, delay);\n \t}\n \n out:\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511255","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-13-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 13/19] reftable/basics: stop using `SWAP()` macro","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:21Z","receivedAt":"2025-01-27T13:04:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Stop using `SWAP()` macro in favor of an open-coded variant of it. Note\nthat this also requires us to open-code the build assert that `SWAP()`\nitself uses to verify that the size of both variables matches.\n\nThis is done to reduce our dependency on the Git codebase.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 9 +++++++++\n reftable/merged.c | 2 +-\n reftable/pq.c | 4 ++--\n reftable/record.c | 16 ++++++++--------\n 4 files changed, 20 insertions(+), 11 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex c1ddbaec3f..59000798f0 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -266,6 +266,15 @@ static inline void *reftable_alloc_grow(void *p, size_t nelem, size_t elsize,\n # define strdup(str) REFTABLE_BANNED(strdup)\n #endif\n \n+#define REFTABLE_SWAP(a, b) do {\t\t\t\t\t\t\t\t\\\n+\tvoid *_swap_a_ptr = &(a);\t\t\t\t\t\t\t\t\\\n+\tvoid *_swap_b_ptr = &(b);\t\t\t\t\t\t\t\t\\\n+\tunsigned char _swap_buffer[sizeof(a) - 2 * sizeof(a) * (sizeof(a) != sizeof(b))];\t\\\n+\tmemcpy(_swap_buffer, _swap_a_ptr, sizeof(a));\t\t\t\t\t\t\\\n+\tmemcpy(_swap_a_ptr, _swap_b_ptr, sizeof(a));\t\t\t\t\t\t\\\n+\tmemcpy(_swap_b_ptr, _swap_buffer, sizeof(a));\t\t\t\t\t\t\\\n+} while (0)\n+\n /* Find the longest shared prefix size of `a` and `b` */\n size_t common_prefix_size(struct reftable_buf *a, struct reftable_buf *b);\n \ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 563864068c..4ff1553772 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -155,7 +155,7 @@ static int merged_iter_next_entry(struct merged_iter *mi,\n \t}\n \n \tmi->advance_index = entry.index;\n-\tSWAP(*rec, *entry.rec);\n+\tREFTABLE_SWAP(*rec, *entry.rec);\n \treturn 0;\n }\n \ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex ef8035cfd9..82394a972d 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -57,7 +57,7 @@ int merged_iter_pqueue_remove(struct merged_iter_pqueue *pq, struct pq_entry *ou\n \n \t\tif (min == i)\n \t\t\tbreak;\n-\t\tSWAP(pq->heap[i], pq->heap[min]);\n+\t\tREFTABLE_SWAP(pq->heap[i], pq->heap[min]);\n \t\ti = min;\n \t}\n \n@@ -81,7 +81,7 @@ int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry\n \t\tsize_t j = (i - 1) / 2;\n \t\tif (pq_less(&pq->heap[j], &pq->heap[i]))\n \t\t\tbreak;\n-\t\tSWAP(pq->heap[j], pq->heap[i]);\n+\t\tREFTABLE_SWAP(pq->heap[j], pq->heap[i]);\n \t\ti = j;\n \t}\n \ndiff --git a/reftable/record.c b/reftable/record.c\nindex 286659c098..d86cd65bb4 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -237,11 +237,11 @@ static int reftable_ref_record_copy_from(void *rec, const void *src_rec,\n \tsize_t refname_cap = 0;\n \tint err;\n \n-\tSWAP(refname, ref->refname);\n-\tSWAP(refname_cap, ref->refname_cap);\n+\tREFTABLE_SWAP(refname, ref->refname);\n+\tREFTABLE_SWAP(refname_cap, ref->refname_cap);\n \treftable_ref_record_release(ref);\n-\tSWAP(ref->refname, refname);\n-\tSWAP(ref->refname_cap, refname_cap);\n+\tREFTABLE_SWAP(ref->refname, refname);\n+\tREFTABLE_SWAP(ref->refname_cap, refname_cap);\n \n \tif (src->refname) {\n \t\tsize_t refname_len = strlen(src->refname);\n@@ -376,11 +376,11 @@ static int reftable_ref_record_decode(void *rec, struct reftable_buf key,\n \t\treturn n;\n \tstring_view_consume(&in, n);\n \n-\tSWAP(refname, r->refname);\n-\tSWAP(refname_cap, r->refname_cap);\n+\tREFTABLE_SWAP(refname, r->refname);\n+\tREFTABLE_SWAP(refname_cap, r->refname_cap);\n \treftable_ref_record_release(r);\n-\tSWAP(r->refname, refname);\n-\tSWAP(r->refname_cap, refname_cap);\n+\tREFTABLE_SWAP(r->refname, refname);\n+\tREFTABLE_SWAP(r->refname_cap, refname_cap);\n \n \tREFTABLE_ALLOC_GROW_OR_NULL(r->refname, key.len + 1, r->refname_cap);\n \tif (!r->refname) {\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511256","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-14-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 14/19] reftable/basics: stop using `UNUSED` annotation","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:22Z","receivedAt":"2025-01-27T13:04:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Stop using the `UNUSED` annotation and replace it with a new\n`REFTABLE_UNUSED` macro. The latter is a weaker guarantee compared to\n`UNUSED` as it only suppresses unused parameters without generating a\nwarning in case a parameter marked as unused is in fact used. But it's\ngood enough, and by relaxing the behaviour a bit we avoid having to wire\nup compiler-specific logic.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 2 ++\n reftable/blocksource.c | 10 +++++++---\n reftable/iter.c | 17 ++++++++++++-----\n reftable/record.c | 51 ++++++++++++++++++++++++++++++++++++--------------\n reftable/writer.c | 4 +++-\n 5 files changed, 61 insertions(+), 23 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 59000798f0..4d0645a4e9 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -16,6 +16,8 @@ license that can be found in the LICENSE file or at\n #include \"system.h\"\n #include \"reftable-basics.h\"\n \n+#define REFTABLE_UNUSED(x) (void)(x)\n+\n struct reftable_buf {\n \tsize_t alloc;\n \tsize_t len;\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex 02972c46f4..bfd64b0e48 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -13,15 +13,17 @@ license that can be found in the LICENSE file or at\n #include \"reftable-blocksource.h\"\n #include \"reftable-error.h\"\n \n-static void reftable_buf_return_block(void *b UNUSED, struct reftable_block *dest)\n+static void reftable_buf_return_block(void *b, struct reftable_block *dest)\n {\n+\tREFTABLE_UNUSED(b);\n \tif (dest->len)\n \t\tmemset(dest->data, 0xff, dest->len);\n \treftable_free(dest->data);\n }\n \n-static void reftable_buf_close(void *b UNUSED)\n+static void reftable_buf_close(void *b)\n {\n+\tREFTABLE_UNUSED(b);\n }\n \n static ssize_t reftable_buf_read_block(void *v, struct reftable_block *dest,\n@@ -67,8 +69,10 @@ static uint64_t file_size(void *b)\n \treturn ((struct file_block_source *)b)->size;\n }\n \n-static void file_return_block(void *b UNUSED, struct reftable_block *dest UNUSED)\n+static void file_return_block(void *b, struct reftable_block *dest)\n {\n+\tREFTABLE_UNUSED(b);\n+\tREFTABLE_UNUSED(dest);\n }\n \n static void file_close(void *v)\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex b2ffb09c16..452add2705 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -25,18 +25,23 @@ int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n \treturn it->ops->next(it->iter_arg, rec);\n }\n \n-static int empty_iterator_seek(void *arg UNUSED, struct reftable_record *want UNUSED)\n+static int empty_iterator_seek(void *arg, struct reftable_record *want)\n {\n+\tREFTABLE_UNUSED(arg);\n+\tREFTABLE_UNUSED(want);\n \treturn 0;\n }\n \n-static int empty_iterator_next(void *arg UNUSED, struct reftable_record *rec UNUSED)\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n {\n+\tREFTABLE_UNUSED(arg);\n+\tREFTABLE_UNUSED(rec);\n \treturn 1;\n }\n \n-static void empty_iterator_close(void *arg UNUSED)\n+static void empty_iterator_close(void *arg)\n {\n+\tREFTABLE_UNUSED(arg);\n }\n \n static struct reftable_iterator_vtable empty_vtable = {\n@@ -143,9 +148,11 @@ static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n \treturn 0;\n }\n \n-static int indexed_table_ref_iter_seek(void *p UNUSED,\n-\t\t\t\t struct reftable_record *want UNUSED)\n+static int indexed_table_ref_iter_seek(void *p,\n+\t\t\t\t struct reftable_record *want)\n {\n+\tREFTABLE_UNUSED(p);\n+\tREFTABLE_UNUSED(want);\n \treturn REFTABLE_API_ERROR;\n }\n \ndiff --git a/reftable/record.c b/reftable/record.c\nindex d86cd65bb4..46093ef987 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -490,11 +490,13 @@ static void reftable_obj_record_release(void *rec)\n }\n \n static int reftable_obj_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t\t uint32_t hash_size)\n {\n \tstruct reftable_obj_record *obj = rec;\n \tconst struct reftable_obj_record *src = src_rec;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\n \treftable_obj_record_release(obj);\n \n \tREFTABLE_ALLOC_ARRAY(obj->hash_prefix, src->hash_prefix_len);\n@@ -523,13 +525,16 @@ static uint8_t reftable_obj_record_val_type(const void *rec)\n }\n \n static int reftable_obj_record_encode(const void *rec, struct string_view s,\n-\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t uint32_t hash_size)\n {\n \tconst struct reftable_obj_record *r = rec;\n \tstruct string_view start = s;\n \tint i = 0;\n \tint n = 0;\n \tuint64_t last = 0;\n+\n+\tREFTABLE_UNUSED(hash_size);\n+\n \tif (r->offset_len == 0 || r->offset_len >= 8) {\n \t\tn = put_var_int(&s, r->offset_len);\n \t\tif (n < 0) {\n@@ -558,8 +563,8 @@ static int reftable_obj_record_encode(const void *rec, struct string_view s,\n \n static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n \t\t\t\t uint8_t val_type, struct string_view in,\n-\t\t\t\t uint32_t hash_size UNUSED,\n-\t\t\t\t struct reftable_buf *scratch UNUSED)\n+\t\t\t\t uint32_t hash_size,\n+\t\t\t\t struct reftable_buf *scratch)\n {\n \tstruct string_view start = in;\n \tstruct reftable_obj_record *r = rec;\n@@ -567,6 +572,9 @@ static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n \tint n = 0;\n \tuint64_t last;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\tREFTABLE_UNUSED(scratch);\n+\n \treftable_obj_record_release(r);\n \n \tREFTABLE_ALLOC_ARRAY(r->hash_prefix, key.len);\n@@ -613,17 +621,20 @@ static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n \treturn start.len - in.len;\n }\n \n-static int not_a_deletion(const void *p UNUSED)\n+static int not_a_deletion(const void *p)\n {\n+\tREFTABLE_UNUSED(p);\n \treturn 0;\n }\n \n static int reftable_obj_record_equal_void(const void *a, const void *b,\n-\t\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t\t uint32_t hash_size)\n {\n \tstruct reftable_obj_record *ra = (struct reftable_obj_record *) a;\n \tstruct reftable_obj_record *rb = (struct reftable_obj_record *) b;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\n \tif (ra->hash_prefix_len != rb->hash_prefix_len\n \t || ra->offset_len != rb->offset_len)\n \t\treturn 0;\n@@ -1049,12 +1060,14 @@ static int reftable_index_record_key(const void *r, struct reftable_buf *dest)\n }\n \n static int reftable_index_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t\t uint32_t hash_size)\n {\n \tstruct reftable_index_record *dst = rec;\n \tconst struct reftable_index_record *src = src_rec;\n \tint err;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\n \treftable_buf_reset(&dst->last_key);\n \terr = reftable_buf_add(&dst->last_key, src->last_key.buf, src->last_key.len);\n \tif (err < 0)\n@@ -1070,19 +1083,23 @@ static void reftable_index_record_release(void *rec)\n \treftable_buf_release(&idx->last_key);\n }\n \n-static uint8_t reftable_index_record_val_type(const void *rec UNUSED)\n+static uint8_t reftable_index_record_val_type(const void *rec)\n {\n+\tREFTABLE_UNUSED(rec);\n \treturn 0;\n }\n \n static int reftable_index_record_encode(const void *rec, struct string_view out,\n-\t\t\t\t\tuint32_t hash_size UNUSED)\n+\t\t\t\t\tuint32_t hash_size)\n {\n \tconst struct reftable_index_record *r =\n \t\t(const struct reftable_index_record *)rec;\n \tstruct string_view start = out;\n+\tint n;\n \n-\tint n = put_var_int(&out, r->offset);\n+\tREFTABLE_UNUSED(hash_size);\n+\n+\tn = put_var_int(&out, r->offset);\n \tif (n < 0)\n \t\treturn n;\n \n@@ -1092,15 +1109,19 @@ static int reftable_index_record_encode(const void *rec, struct string_view out,\n }\n \n static int reftable_index_record_decode(void *rec, struct reftable_buf key,\n-\t\t\t\t\tuint8_t val_type UNUSED,\n+\t\t\t\t\tuint8_t val_type,\n \t\t\t\t\tstruct string_view in,\n-\t\t\t\t\tuint32_t hash_size UNUSED,\n-\t\t\t\t\tstruct reftable_buf *scratch UNUSED)\n+\t\t\t\t\tuint32_t hash_size,\n+\t\t\t\t\tstruct reftable_buf *scratch)\n {\n \tstruct string_view start = in;\n \tstruct reftable_index_record *r = rec;\n \tint err, n = 0;\n \n+\tREFTABLE_UNUSED(val_type);\n+\tREFTABLE_UNUSED(hash_size);\n+\tREFTABLE_UNUSED(scratch);\n+\n \treftable_buf_reset(&r->last_key);\n \terr = reftable_buf_add(&r->last_key, key.buf, key.len);\n \tif (err < 0)\n@@ -1115,11 +1136,13 @@ static int reftable_index_record_decode(void *rec, struct reftable_buf key,\n }\n \n static int reftable_index_record_equal(const void *a, const void *b,\n-\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t uint32_t hash_size)\n {\n \tstruct reftable_index_record *ia = (struct reftable_index_record *) a;\n \tstruct reftable_index_record *ib = (struct reftable_index_record *) b;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\n \treturn ia->offset == ib->offset && !reftable_buf_cmp(&ia->last_key, &ib->last_key);\n }\n \ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 5961698311..0040a1b1c4 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -636,10 +636,12 @@ static void write_object_record(void *void_arg, void *key)\n done:;\n }\n \n-static void object_record_free(void *void_arg UNUSED, void *key)\n+static void object_record_free(void *void_arg, void *key)\n {\n \tstruct obj_index_tree_node *entry = key;\n \n+\tREFTABLE_UNUSED(void_arg);\n+\n \tREFTABLE_FREE_AND_NULL(entry->offsets);\n \treftable_buf_release(&entry->hash);\n \treftable_free(entry);\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511257","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-15-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 15/19] compat/mingw: split out POSIX-related bits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:23Z","receivedAt":"2025-01-27T13:04:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Split out the POSIX-related bits from \"compat/mingw.h\". This is in\npreparation for splitting up \"git-compat-utils.h\" into a header that\nprovides POSIX-compatibility and a header that provides common wrappers\nused by the Git project.\n\nSigned-off-by: Patrick Steinhardt \n---\n compat/{mingw.c => mingw/compat-util.c} | 0\n compat/mingw/compat-util.h | 220 ++++++++++++++++++++++++++++++++\n compat/{mingw.h => mingw/posix.h} | 216 +------------------------------\n compat/msvc.c | 2 +-\n compat/msvc.h | 2 +-\n config.mak.uname | 4 +-\n contrib/buildsystems/CMakeLists.txt | 2 +-\n git-compat-util.h | 4 +-\n meson.build | 6 +-\n 9 files changed, 234 insertions(+), 222 deletions(-)\n\ndiff --git a/compat/mingw.c b/compat/mingw/compat-util.c\nsimilarity index 100%\nrename from compat/mingw.c\nrename to compat/mingw/compat-util.c\ndiff --git a/compat/mingw/compat-util.h b/compat/mingw/compat-util.h\nnew file mode 100644\nindex 0000000000..b433762526\n--- /dev/null\n+++ b/compat/mingw/compat-util.h\n@@ -0,0 +1,220 @@\n+#ifndef COMPAT_MINGW_COMPAT_UTIL_H\n+#define COMPAT_MINGW_COMPAT_UTIL_H\n+\n+#include \"compat/mingw/posix.h\"\n+\n+struct config_context;\n+int mingw_core_config(const char *var, const char *value,\n+\t\t const struct config_context *ctx, void *cb);\n+#define platform_core_config mingw_core_config\n+\n+#ifndef NO_OPENSSL\n+#include \n+static inline int mingw_SSL_set_fd(SSL *ssl, int fd)\n+{\n+\treturn SSL_set_fd(ssl, _get_osfhandle(fd));\n+}\n+#define SSL_set_fd mingw_SSL_set_fd\n+\n+static inline int mingw_SSL_set_rfd(SSL *ssl, int fd)\n+{\n+\treturn SSL_set_rfd(ssl, _get_osfhandle(fd));\n+}\n+#define SSL_set_rfd mingw_SSL_set_rfd\n+\n+static inline int mingw_SSL_set_wfd(SSL *ssl, int fd)\n+{\n+\treturn SSL_set_wfd(ssl, _get_osfhandle(fd));\n+}\n+#define SSL_set_wfd mingw_SSL_set_wfd\n+#endif\n+\n+/*\n+ * git specific compatibility\n+ */\n+\n+static inline void convert_slashes(char *path)\n+{\n+\tfor (; *path; path++)\n+\t\tif (*path == '\\\\')\n+\t\t\t*path = '/';\n+}\n+#define PATH_SEP ';'\n+char *mingw_query_user_email(void);\n+#define query_user_email mingw_query_user_email\n+\n+/**\n+ * Verifies that the specified path is owned by the user running the\n+ * current process.\n+ */\n+int is_path_owned_by_current_sid(const char *path, struct strbuf *report);\n+#define is_path_owned_by_current_user is_path_owned_by_current_sid\n+\n+/**\n+ * Verifies that the given path is a valid one on Windows.\n+ *\n+ * In particular, path segments are disallowed which\n+ *\n+ * - end in a period or a space (except the special directories `.` and `..`).\n+ *\n+ * - contain any of the reserved characters, e.g. `:`, `;`, `*`, etc\n+ *\n+ * - correspond to reserved names (such as `AUX`, `PRN`, etc)\n+ *\n+ * The `allow_literal_nul` parameter controls whether the path `NUL` should\n+ * be considered valid (this makes sense e.g. before opening files, as it is\n+ * perfectly legitimate to open `NUL` on Windows, just as it is to open\n+ * `/dev/null` on Unix/Linux).\n+ *\n+ * Returns 1 upon success, otherwise 0.\n+ */\n+int is_valid_win32_path(const char *path, int allow_literal_nul);\n+#define is_valid_path(path) is_valid_win32_path(path, 0)\n+\n+/**\n+ * Converts UTF-8 encoded string to UTF-16LE.\n+ *\n+ * To support repositories with legacy-encoded file names, invalid UTF-8 bytes\n+ * 0xa0 - 0xff are converted to corresponding printable Unicode chars \\u00a0 -\n+ * \\u00ff, and invalid UTF-8 bytes 0x80 - 0x9f (which would make non-printable\n+ * Unicode) are converted to hex-code.\n+ *\n+ * Lead-bytes not followed by an appropriate number of trail-bytes, over-long\n+ * encodings and 4-byte encodings > \\u10ffff are detected as invalid UTF-8.\n+ *\n+ * Maximum space requirement for the target buffer is two wide chars per UTF-8\n+ * char (((strlen(utf) * 2) + 1) [* sizeof(wchar_t)]).\n+ *\n+ * The maximum space is needed only if the entire input string consists of\n+ * invalid UTF-8 bytes in range 0x80-0x9f, as per the following table:\n+ *\n+ * | | UTF-8 | UTF-16 |\n+ * Code point | UTF-8 sequence | bytes | words | ratio\n+ * --------------+-------------------+-------+--------+-------\n+ * 000000-00007f | 0-7f | 1 | 1 | 1\n+ * 000080-0007ff | c2-df + 80-bf | 2 | 1 | 0.5\n+ * 000800-00ffff | e0-ef + 2 * 80-bf | 3 | 1 | 0.33\n+ * 010000-10ffff | f0-f4 + 3 * 80-bf | 4 | 2 (a) | 0.5\n+ * invalid | 80-9f | 1 | 2 (b) | 2\n+ * invalid | a0-ff | 1 | 1 | 1\n+ *\n+ * (a) encoded as UTF-16 surrogate pair\n+ * (b) encoded as two hex digits\n+ *\n+ * Note that, while the UTF-8 encoding scheme can be extended to 5-byte, 6-byte\n+ * or even indefinite-byte sequences, the largest valid code point \\u10ffff\n+ * encodes as only 4 UTF-8 bytes.\n+ *\n+ * Parameters:\n+ * wcs: wide char target buffer\n+ * utf: string to convert\n+ * wcslen: size of target buffer (in wchar_t's)\n+ * utflen: size of string to convert, or -1 if 0-terminated\n+ *\n+ * Returns:\n+ * length of converted string (_wcslen(wcs)), or -1 on failure\n+ *\n+ * Errors:\n+ * EINVAL: one of the input parameters is invalid (e.g. NULL)\n+ * ERANGE: the output buffer is too small\n+ */\n+int xutftowcsn(wchar_t *wcs, const char *utf, size_t wcslen, int utflen);\n+\n+/**\n+ * Simplified variant of xutftowcsn, assumes input string is \\0-terminated.\n+ */\n+static inline int xutftowcs(wchar_t *wcs, const char *utf, size_t wcslen)\n+{\n+\treturn xutftowcsn(wcs, utf, wcslen, -1);\n+}\n+\n+/**\n+ * Simplified file system specific variant of xutftowcsn, assumes output\n+ * buffer size is MAX_PATH wide chars and input string is \\0-terminated,\n+ * fails with ENAMETOOLONG if input string is too long.\n+ */\n+static inline int xutftowcs_path(wchar_t *wcs, const char *utf)\n+{\n+\tint result = xutftowcsn(wcs, utf, MAX_PATH, -1);\n+\tif (result < 0 && errno == ERANGE)\n+\t\terrno = ENAMETOOLONG;\n+\treturn result;\n+}\n+\n+/**\n+ * Converts UTF-16LE encoded string to UTF-8.\n+ *\n+ * Maximum space requirement for the target buffer is three UTF-8 chars per\n+ * wide char ((_wcslen(wcs) * 3) + 1).\n+ *\n+ * The maximum space is needed only if the entire input string consists of\n+ * UTF-16 words in range 0x0800-0xd7ff or 0xe000-0xffff (i.e. \\u0800-\\uffff\n+ * modulo surrogate pairs), as per the following table:\n+ *\n+ * | | UTF-16 | UTF-8 |\n+ * Code point | UTF-16 sequence | words | bytes | ratio\n+ * --------------+-----------------------+--------+-------+-------\n+ * 000000-00007f | 0000-007f | 1 | 1 | 1\n+ * 000080-0007ff | 0080-07ff | 1 | 2 | 2\n+ * 000800-00ffff | 0800-d7ff / e000-ffff | 1 | 3 | 3\n+ * 010000-10ffff | d800-dbff + dc00-dfff | 2 | 4 | 2\n+ *\n+ * Note that invalid code points > 10ffff cannot be represented in UTF-16.\n+ *\n+ * Parameters:\n+ * utf: target buffer\n+ * wcs: wide string to convert\n+ * utflen: size of target buffer\n+ *\n+ * Returns:\n+ * length of converted string, or -1 on failure\n+ *\n+ * Errors:\n+ * EINVAL: one of the input parameters is invalid (e.g. NULL)\n+ * ERANGE: the output buffer is too small\n+ */\n+int xwcstoutf(char *utf, const wchar_t *wcs, size_t utflen);\n+\n+/*\n+ * A critical section used in the implementation of the spawn\n+ * functions (mingw_spawnv[p]e()) and waitpid(). Initialised in\n+ * the replacement main() macro below.\n+ */\n+extern CRITICAL_SECTION pinfo_cs;\n+\n+/*\n+ * Git, like most portable C applications, implements a main() function. On\n+ * Windows, this main() function would receive parameters encoded in the\n+ * current locale, but Git for Windows would prefer UTF-8 encoded parameters.\n+ *\n+ * To make that happen, we still declare main() here, and then declare and\n+ * implement wmain() (which is the Unicode variant of main()) and compile with\n+ * -municode. This wmain() function reencodes the parameters from UTF-16 to\n+ * UTF-8 format, sets up a couple of other things as required on Windows, and\n+ * then hands off to the main() function.\n+ */\n+int wmain(int argc, const wchar_t **w_argv);\n+int main(int argc, const char **argv);\n+\n+/*\n+ * For debugging: if a problem occurs, say, in a Git process that is spawned\n+ * from another Git process which in turn is spawned from yet another Git\n+ * process, it can be quite daunting to figure out what is going on.\n+ *\n+ * Call this function to open a new MinTTY (this assumes you are in Git for\n+ * Windows' SDK) with a GDB that attaches to the current process right away.\n+ */\n+void open_in_gdb(void);\n+\n+/*\n+ * Used by Pthread API implementation for Windows\n+ */\n+int err_win_to_posix(DWORD winerr);\n+\n+#ifndef NO_UNIX_SOCKETS\n+int mingw_have_unix_sockets(void);\n+#undef have_unix_sockets\n+#define have_unix_sockets mingw_have_unix_sockets\n+#endif\n+\n+#endif /* COMPAT_MINGW_COMPAT_UTIL_H */\ndiff --git a/compat/mingw.h b/compat/mingw/posix.h\nsimilarity index 59%\nrename from compat/mingw.h\nrename to compat/mingw/posix.h\nindex ebfb8ba423..8dddfa818d 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw/posix.h\n@@ -1,3 +1,6 @@\n+#ifndef COMPAT_MINGW_POSIX_H\n+#define COMPAT_MINGW_POSIX_H\n+\n #ifdef __MINGW64_VERSION_MAJOR\n #include \n #include \n@@ -11,11 +14,6 @@ typedef _sigset_t sigset_t;\n #undef _POSIX_THREAD_SAFE_FUNCTIONS\n #endif\n \n-struct config_context;\n-int mingw_core_config(const char *var, const char *value,\n-\t\t const struct config_context *ctx, void *cb);\n-#define platform_core_config mingw_core_config\n-\n /*\n * things that are not available in header files\n */\n@@ -180,27 +178,6 @@ int mingw_kill(pid_t pid, int sig);\n #define locate_in_PATH mingw_locate_in_PATH\n char *mingw_locate_in_PATH(const char *cmd);\n \n-#ifndef NO_OPENSSL\n-#include \n-static inline int mingw_SSL_set_fd(SSL *ssl, int fd)\n-{\n-\treturn SSL_set_fd(ssl, _get_osfhandle(fd));\n-}\n-#define SSL_set_fd mingw_SSL_set_fd\n-\n-static inline int mingw_SSL_set_rfd(SSL *ssl, int fd)\n-{\n-\treturn SSL_set_rfd(ssl, _get_osfhandle(fd));\n-}\n-#define SSL_set_rfd mingw_SSL_set_rfd\n-\n-static inline int mingw_SSL_set_wfd(SSL *ssl, int fd)\n-{\n-\treturn SSL_set_wfd(ssl, _get_osfhandle(fd));\n-}\n-#define SSL_set_wfd mingw_SSL_set_wfd\n-#endif\n-\n /*\n * implementations of missing functions\n */\n@@ -444,19 +421,6 @@ int winansi_dup2(int oldfd, int newfd);\n void winansi_init(void);\n HANDLE winansi_get_osfhandle(int fd);\n \n-/*\n- * git specific compatibility\n- */\n-\n-static inline void convert_slashes(char *path)\n-{\n-\tfor (; *path; path++)\n-\t\tif (*path == '\\\\')\n-\t\t\t*path = '/';\n-}\n-#define PATH_SEP ';'\n-char *mingw_query_user_email(void);\n-#define query_user_email mingw_query_user_email\n #if !defined(__MINGW64_VERSION_MAJOR) && (!defined(_MSC_VER) || _MSC_VER < 1800)\n #define PRIuMAX \"I64u\"\n #define PRId64 \"I64d\"\n@@ -464,176 +428,4 @@ char *mingw_query_user_email(void);\n #include \n #endif\n \n-/**\n- * Verifies that the specified path is owned by the user running the\n- * current process.\n- */\n-int is_path_owned_by_current_sid(const char *path, struct strbuf *report);\n-#define is_path_owned_by_current_user is_path_owned_by_current_sid\n-\n-/**\n- * Verifies that the given path is a valid one on Windows.\n- *\n- * In particular, path segments are disallowed which\n- *\n- * - end in a period or a space (except the special directories `.` and `..`).\n- *\n- * - contain any of the reserved characters, e.g. `:`, `;`, `*`, etc\n- *\n- * - correspond to reserved names (such as `AUX`, `PRN`, etc)\n- *\n- * The `allow_literal_nul` parameter controls whether the path `NUL` should\n- * be considered valid (this makes sense e.g. before opening files, as it is\n- * perfectly legitimate to open `NUL` on Windows, just as it is to open\n- * `/dev/null` on Unix/Linux).\n- *\n- * Returns 1 upon success, otherwise 0.\n- */\n-int is_valid_win32_path(const char *path, int allow_literal_nul);\n-#define is_valid_path(path) is_valid_win32_path(path, 0)\n-\n-/**\n- * Converts UTF-8 encoded string to UTF-16LE.\n- *\n- * To support repositories with legacy-encoded file names, invalid UTF-8 bytes\n- * 0xa0 - 0xff are converted to corresponding printable Unicode chars \\u00a0 -\n- * \\u00ff, and invalid UTF-8 bytes 0x80 - 0x9f (which would make non-printable\n- * Unicode) are converted to hex-code.\n- *\n- * Lead-bytes not followed by an appropriate number of trail-bytes, over-long\n- * encodings and 4-byte encodings > \\u10ffff are detected as invalid UTF-8.\n- *\n- * Maximum space requirement for the target buffer is two wide chars per UTF-8\n- * char (((strlen(utf) * 2) + 1) [* sizeof(wchar_t)]).\n- *\n- * The maximum space is needed only if the entire input string consists of\n- * invalid UTF-8 bytes in range 0x80-0x9f, as per the following table:\n- *\n- * | | UTF-8 | UTF-16 |\n- * Code point | UTF-8 sequence | bytes | words | ratio\n- * --------------+-------------------+-------+--------+-------\n- * 000000-00007f | 0-7f | 1 | 1 | 1\n- * 000080-0007ff | c2-df + 80-bf | 2 | 1 | 0.5\n- * 000800-00ffff | e0-ef + 2 * 80-bf | 3 | 1 | 0.33\n- * 010000-10ffff | f0-f4 + 3 * 80-bf | 4 | 2 (a) | 0.5\n- * invalid | 80-9f | 1 | 2 (b) | 2\n- * invalid | a0-ff | 1 | 1 | 1\n- *\n- * (a) encoded as UTF-16 surrogate pair\n- * (b) encoded as two hex digits\n- *\n- * Note that, while the UTF-8 encoding scheme can be extended to 5-byte, 6-byte\n- * or even indefinite-byte sequences, the largest valid code point \\u10ffff\n- * encodes as only 4 UTF-8 bytes.\n- *\n- * Parameters:\n- * wcs: wide char target buffer\n- * utf: string to convert\n- * wcslen: size of target buffer (in wchar_t's)\n- * utflen: size of string to convert, or -1 if 0-terminated\n- *\n- * Returns:\n- * length of converted string (_wcslen(wcs)), or -1 on failure\n- *\n- * Errors:\n- * EINVAL: one of the input parameters is invalid (e.g. NULL)\n- * ERANGE: the output buffer is too small\n- */\n-int xutftowcsn(wchar_t *wcs, const char *utf, size_t wcslen, int utflen);\n-\n-/**\n- * Simplified variant of xutftowcsn, assumes input string is \\0-terminated.\n- */\n-static inline int xutftowcs(wchar_t *wcs, const char *utf, size_t wcslen)\n-{\n-\treturn xutftowcsn(wcs, utf, wcslen, -1);\n-}\n-\n-/**\n- * Simplified file system specific variant of xutftowcsn, assumes output\n- * buffer size is MAX_PATH wide chars and input string is \\0-terminated,\n- * fails with ENAMETOOLONG if input string is too long.\n- */\n-static inline int xutftowcs_path(wchar_t *wcs, const char *utf)\n-{\n-\tint result = xutftowcsn(wcs, utf, MAX_PATH, -1);\n-\tif (result < 0 && errno == ERANGE)\n-\t\terrno = ENAMETOOLONG;\n-\treturn result;\n-}\n-\n-/**\n- * Converts UTF-16LE encoded string to UTF-8.\n- *\n- * Maximum space requirement for the target buffer is three UTF-8 chars per\n- * wide char ((_wcslen(wcs) * 3) + 1).\n- *\n- * The maximum space is needed only if the entire input string consists of\n- * UTF-16 words in range 0x0800-0xd7ff or 0xe000-0xffff (i.e. \\u0800-\\uffff\n- * modulo surrogate pairs), as per the following table:\n- *\n- * | | UTF-16 | UTF-8 |\n- * Code point | UTF-16 sequence | words | bytes | ratio\n- * --------------+-----------------------+--------+-------+-------\n- * 000000-00007f | 0000-007f | 1 | 1 | 1\n- * 000080-0007ff | 0080-07ff | 1 | 2 | 2\n- * 000800-00ffff | 0800-d7ff / e000-ffff | 1 | 3 | 3\n- * 010000-10ffff | d800-dbff + dc00-dfff | 2 | 4 | 2\n- *\n- * Note that invalid code points > 10ffff cannot be represented in UTF-16.\n- *\n- * Parameters:\n- * utf: target buffer\n- * wcs: wide string to convert\n- * utflen: size of target buffer\n- *\n- * Returns:\n- * length of converted string, or -1 on failure\n- *\n- * Errors:\n- * EINVAL: one of the input parameters is invalid (e.g. NULL)\n- * ERANGE: the output buffer is too small\n- */\n-int xwcstoutf(char *utf, const wchar_t *wcs, size_t utflen);\n-\n-/*\n- * A critical section used in the implementation of the spawn\n- * functions (mingw_spawnv[p]e()) and waitpid(). Initialised in\n- * the replacement main() macro below.\n- */\n-extern CRITICAL_SECTION pinfo_cs;\n-\n-/*\n- * Git, like most portable C applications, implements a main() function. On\n- * Windows, this main() function would receive parameters encoded in the\n- * current locale, but Git for Windows would prefer UTF-8 encoded parameters.\n- *\n- * To make that happen, we still declare main() here, and then declare and\n- * implement wmain() (which is the Unicode variant of main()) and compile with\n- * -municode. This wmain() function reencodes the parameters from UTF-16 to\n- * UTF-8 format, sets up a couple of other things as required on Windows, and\n- * then hands off to the main() function.\n- */\n-int wmain(int argc, const wchar_t **w_argv);\n-int main(int argc, const char **argv);\n-\n-/*\n- * For debugging: if a problem occurs, say, in a Git process that is spawned\n- * from another Git process which in turn is spawned from yet another Git\n- * process, it can be quite daunting to figure out what is going on.\n- *\n- * Call this function to open a new MinTTY (this assumes you are in Git for\n- * Windows' SDK) with a GDB that attaches to the current process right away.\n- */\n-void open_in_gdb(void);\n-\n-/*\n- * Used by Pthread API implementation for Windows\n- */\n-int err_win_to_posix(DWORD winerr);\n-\n-#ifndef NO_UNIX_SOCKETS\n-int mingw_have_unix_sockets(void);\n-#undef have_unix_sockets\n-#define have_unix_sockets mingw_have_unix_sockets\n-#endif\n+#endif /* COMPAT_MINGW_POSIX_H */\ndiff --git a/compat/msvc.c b/compat/msvc.c\nindex 71843d7eef..7f228a6399 100644\n--- a/compat/msvc.c\n+++ b/compat/msvc.c\n@@ -3,4 +3,4 @@\n #include \n #include \"../strbuf.h\"\n \n-#include \"mingw.c\"\n+#include \"mingw/compat-util.c\"\ndiff --git a/compat/msvc.h b/compat/msvc.h\nindex 1d7a8c6145..df39978f4f 100644\n--- a/compat/msvc.h\n+++ b/compat/msvc.h\n@@ -28,6 +28,6 @@ typedef int sigset_t;\n /* open for reading, writing, or both (not in fcntl.h) */\n #define O_ACCMODE (_O_RDONLY | _O_WRONLY | _O_RDWR)\n \n-#include \"compat/mingw.h\"\n+#include \"compat/mingw/compat-util.h\"\n \n #endif\ndiff --git a/config.mak.uname b/config.mak.uname\nindex b12d4e168a..cc4152e87e 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -541,7 +541,7 @@ endif\n \n \tEXTRA_PROGRAMS += headless-git$X\n \n-compat/msvc.o: compat/msvc.c compat/mingw.c GIT-CFLAGS\n+compat/msvc.o: compat/msvc.c compat/mingw/compat-util.c GIT-CFLAGS\n endif\n ifeq ($(uname_S),Interix)\n \tNO_INITGROUPS = YesPlease\n@@ -694,7 +694,7 @@ ifeq ($(uname_S),MINGW)\n \tBASIC_LDFLAGS += -municode\n \tCOMPAT_CFLAGS += -DNOGDI -Icompat -Icompat/win32\n \tCOMPAT_CFLAGS += -DSTRIP_EXTENSION=\\\".exe\\\"\n-\tCOMPAT_OBJS += compat/mingw.o compat/winansi.o \\\n+\tCOMPAT_OBJS += compat/mingw/compat-util.o compat/winansi.o \\\n \t\tcompat/win32/trace2_win32_process_info.o \\\n \t\tcompat/win32/flush.o \\\n \t\tcompat/win32/path-utils.o \\\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 10dc54fdcb..65ce8705f0 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -258,7 +258,7 @@ if(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\t\t\tUSE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP\n \t\t\t\tHAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET HAVE_RTLGENRANDOM)\n \tlist(APPEND compat_SOURCES\n-\t\tcompat/mingw.c\n+\t\tcompat/mingw/compat-util.c\n \t\tcompat/winansi.c\n \t\tcompat/win32/flush.c\n \t\tcompat/win32/path-utils.c\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex e283c46c6f..b96fb98e1e 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -304,10 +304,10 @@ static inline int is_xplatform_dir_sep(int c)\n #if defined(__MINGW32__)\n /* pull in Windows compatibility stuff */\n #include \"compat/win32/path-utils.h\"\n-#include \"compat/mingw.h\"\n+#include \"compat/mingw/compat-util.h\"\n #elif defined(_MSC_VER)\n #include \"compat/win32/path-utils.h\"\n-#include \"compat/msvc.h\"\n+#include \"compat/msvc/compat-util.h\"\n #else\n #include \n #include \ndiff --git a/meson.build b/meson.build\nindex 2297f5e9dd..32eb0a86d5 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -1054,7 +1054,7 @@ if host_machine.system() == 'cygwin'\n ]\n elif host_machine.system() == 'windows'\n libgit_sources += [\n- 'compat/mingw.c',\n+ 'compat/mingw/compat-util.c',\n 'compat/winansi.c',\n 'compat/win32/flush.c',\n 'compat/win32/path-utils.c',\n@@ -1157,7 +1157,7 @@ else\n error('Native regex support requested but not found')\n endif\n \n-# setitimer and friends are provided by compat/mingw.c.\n+# setitimer and friends are provided by compat/mingw/compat-util.c.\n if host_machine.system() != 'windows'\n if not compiler.compiles('''\n #include \n@@ -1237,7 +1237,7 @@ if not compiler.has_function('qsort')\n endif\n libgit_sources += 'compat/qsort_s.c'\n \n-# unsetenv is provided by compat/mingw.c.\n+# unsetenv is provided by compat/mingw/compat-util.c.\n if host_machine.system() != 'windows' and not compiler.has_function('unsetenv')\n libgit_c_args += '-DNO_UNSETENV'\n libgit_sources += 'compat/unsetenv.c'\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511258","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-16-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 16/19] compat/msvc: split out POSIX-related bits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:24Z","receivedAt":"2025-01-27T13:04:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Split out the POSIX-related bits from \"compat/msvc.h\". This is in\npreparation for splitting up \"git-compat-utils.h\" into a header that\nprovides POSIX-compatibility and a header that provides common wrappers\nused by the Git project.\n\nSigned-off-by: Patrick Steinhardt \n---\n compat/{msvc.c => msvc/compat-util.c} | 0\n compat/msvc/compat-util.h | 7 +++++++\n compat/{msvc.h => msvc/posix.h} | 8 ++++----\n config.mak.uname | 4 ++--\n meson.build | 4 +++-\n 5 files changed, 16 insertions(+), 7 deletions(-)\n\ndiff --git a/compat/msvc.c b/compat/msvc/compat-util.c\nsimilarity index 100%\nrename from compat/msvc.c\nrename to compat/msvc/compat-util.c\ndiff --git a/compat/msvc/compat-util.h b/compat/msvc/compat-util.h\nnew file mode 100644\nindex 0000000000..e409e9910c\n--- /dev/null\n+++ b/compat/msvc/compat-util.h\n@@ -0,0 +1,7 @@\n+#ifndef COMPAT_MSVC_COMPAT_UTIL_H\n+#define COMPAT_MSVC_COMPAT_UTIL_H\n+\n+#include \"compat/msvc/posix.h\"\n+#include \"compat/mingw/compat-util.h\"\n+\n+#endif /* COMPAT_MSVC_COMPAT_UTIL_H */\ndiff --git a/compat/msvc.h b/compat/msvc/posix.h\nsimilarity index 86%\nrename from compat/msvc.h\nrename to compat/msvc/posix.h\nindex df39978f4f..47062293cb 100644\n--- a/compat/msvc.h\n+++ b/compat/msvc/posix.h\n@@ -1,5 +1,5 @@\n-#ifndef __MSVC__HEAD\n-#define __MSVC__HEAD\n+#ifndef COMPAT_MSVC_POSIX_H\n+#define COMPAT_MSVC_POSIX_H\n \n #include \n #include \n@@ -28,6 +28,6 @@ typedef int sigset_t;\n /* open for reading, writing, or both (not in fcntl.h) */\n #define O_ACCMODE (_O_RDONLY | _O_WRONLY | _O_RDWR)\n \n-#include \"compat/mingw/compat-util.h\"\n+#include \"compat/mingw/posix.h\"\n \n-#endif\n+#endif /* COMPAT_MSVC_POSIX_H */\ndiff --git a/config.mak.uname b/config.mak.uname\nindex cc4152e87e..dbd05adb5d 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -495,7 +495,7 @@ endif\n \tAR = compat/vcbuild/scripts/lib.pl\n \tCFLAGS =\n \tBASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE\n-\tCOMPAT_OBJS = compat/msvc.o compat/winansi.o \\\n+\tCOMPAT_OBJS = compat/msvc/compat-util.o compat/winansi.o \\\n \t\tcompat/win32/flush.o \\\n \t\tcompat/win32/path-utils.o \\\n \t\tcompat/win32/pthread.o compat/win32/syslog.o \\\n@@ -541,7 +541,7 @@ endif\n \n \tEXTRA_PROGRAMS += headless-git$X\n \n-compat/msvc.o: compat/msvc.c compat/mingw/compat-util.c GIT-CFLAGS\n+compat/msvc/compat-util.o: compat/msvc/compat-util.c compat/mingw/compat-util.c GIT-CFLAGS\n endif\n ifeq ($(uname_S),Interix)\n \tNO_INITGROUPS = YesPlease\ndiff --git a/meson.build b/meson.build\nindex 32eb0a86d5..e8abf798f5 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -1054,7 +1054,6 @@ if host_machine.system() == 'cygwin'\n ]\n elif host_machine.system() == 'windows'\n libgit_sources += [\n- 'compat/mingw/compat-util.c',\n 'compat/winansi.c',\n 'compat/win32/flush.c',\n 'compat/win32/path-utils.c',\n@@ -1081,6 +1080,9 @@ elif host_machine.system() == 'windows'\n libgit_include_directories += 'compat/win32'\n if compiler.get_id() == 'msvc'\n libgit_include_directories += 'compat/vcbuild/include'\n+ libgit_sources += 'compat/msvc/compat-util.c'\n+ else\n+ libgit_sources += 'compat/mingw/compat-util.c'\n endif\n endif\n \n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511261","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-17-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 17/19] git-compat-util.h: split out POSIX-emulating bits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:25Z","receivedAt":"2025-01-27T13:04:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"git-compat-util.h\" header is a treasure trove of various bits and\npieces used throughout the project. It basically mixes two different\nthings into one:\n\n - Providing a POSIX-like interface even on platforms that aren't\n POSIX-compliant.\n\n - Providing low-level functionality that is specific to Git.\n\nThis intermixing is a bit of a problem for the reftable library as we\ndon't want to recreate the POSIX-like interface there. But neither do we\nwant to pull in the Git-specific functionality, as it is otherwise quite\neasy to start depedning on the Git codebase again.\n\nSplit out a new header \"compat/posix.h\" that only contains the bits and\npieces relevant for the emulation of POSIX, which we will start using in\nthe next commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n compat/posix.h | 541 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n git-compat-util.h | 531 +----------------------------------------------------\n 2 files changed, 543 insertions(+), 529 deletions(-)\n\ndiff --git a/compat/posix.h b/compat/posix.h\nnew file mode 100644\nindex 0000000000..2b2da97965\n--- /dev/null\n+++ b/compat/posix.h\n@@ -0,0 +1,541 @@\n+#ifndef COMPAT_POSIX_H\n+#define COMPAT_POSIX_H\n+\n+#define _FILE_OFFSET_BITS 64\n+\n+/*\n+ * Derived from Linux \"Features Test Macro\" header\n+ * Convenience macros to test the versions of gcc (or\n+ * a compatible compiler).\n+ * Use them like this:\n+ * #if GIT_GNUC_PREREQ (2,8)\n+ * ... code requiring gcc 2.8 or later ...\n+ * #endif\n+ *\n+ * This macro of course is not part of POSIX, but we need it for the UNUSED\n+ * macro which is used by some of our POSIX compatibility wrappers.\n+*/\n+#if defined(__GNUC__) && defined(__GNUC_MINOR__)\n+# define GIT_GNUC_PREREQ(maj, min) \\\n+\t((__GNUC__ << 16) + __GNUC_MINOR__ >= ((maj) << 16) + (min))\n+#else\n+ #define GIT_GNUC_PREREQ(maj, min) 0\n+#endif\n+\n+/*\n+ * UNUSED marks a function parameter that is always unused. It also\n+ * can be used to annotate a function, a variable, or a type that is\n+ * always unused.\n+ *\n+ * A callback interface may dictate that a function accepts a\n+ * parameter at that position, but the implementation of the function\n+ * may not need to use the parameter. In such a case, mark the parameter\n+ * with UNUSED.\n+ *\n+ * When a parameter may be used or unused, depending on conditional\n+ * compilation, consider using MAYBE_UNUSED instead.\n+ */\n+#if GIT_GNUC_PREREQ(4, 5)\n+#define UNUSED __attribute__((unused)) \\\n+\t__attribute__((deprecated (\"parameter declared as UNUSED\")))\n+#elif defined(__GNUC__)\n+#define UNUSED __attribute__((unused)) \\\n+\t__attribute__((deprecated))\n+#else\n+#define UNUSED\n+#endif\n+\n+#ifdef __MINGW64__\n+#define _POSIX_C_SOURCE 1\n+#elif defined(__sun__)\n+ /*\n+ * On Solaris, when _XOPEN_EXTENDED is set, its header file\n+ * forces the programs to be XPG4v2, defeating any _XOPEN_SOURCE\n+ * setting to say we are XPG5 or XPG6. Also on Solaris,\n+ * XPG6 programs must be compiled with a c99 compiler, while\n+ * non XPG6 programs must be compiled with a pre-c99 compiler.\n+ */\n+# if __STDC_VERSION__ - 0 >= 199901L\n+# define _XOPEN_SOURCE 600\n+# else\n+# define _XOPEN_SOURCE 500\n+# endif\n+#elif !defined(__APPLE__) && !defined(__FreeBSD__) && !defined(__USLC__) && \\\n+ !defined(_M_UNIX) && !defined(__sgi) && !defined(__DragonFly__) && \\\n+ !defined(__TANDEM) && !defined(__QNX__) && !defined(__MirBSD__) && \\\n+ !defined(__CYGWIN__)\n+#define _XOPEN_SOURCE 600 /* glibc2 and AIX 5.3L need 500, OpenBSD needs 600 for S_ISLNK() */\n+#define _XOPEN_SOURCE_EXTENDED 1 /* AIX 5.3L needs this */\n+#endif\n+#define _ALL_SOURCE 1\n+#define _GNU_SOURCE 1\n+#define _BSD_SOURCE 1\n+#define _DEFAULT_SOURCE 1\n+#define _NETBSD_SOURCE 1\n+#define _SGI_SOURCE 1\n+\n+#if defined(WIN32) && !defined(__CYGWIN__) /* Both MinGW and MSVC */\n+# if !defined(_WIN32_WINNT)\n+# define _WIN32_WINNT 0x0600\n+# endif\n+#define WIN32_LEAN_AND_MEAN /* stops windows.h including winsock.h */\n+#include \n+#ifndef NO_UNIX_SOCKETS\n+#include \n+#endif\n+#include \n+#define GIT_WINDOWS_NATIVE\n+#endif\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#ifdef HAVE_STRINGS_H\n+#include /* for strcasecmp() */\n+#endif\n+#include \n+#include \n+#include \n+#ifdef NEEDS_SYS_PARAM_H\n+#include \n+#endif\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#if !defined(NO_POLL_H)\n+#include \n+#elif !defined(NO_SYS_POLL_H)\n+#include \n+#else\n+/* Pull the compat stuff */\n+#include \n+#endif\n+#ifdef HAVE_BSD_SYSCTL\n+#include \n+#endif\n+\n+#if defined(__MINGW32__)\n+#include \"compat/mingw/posix.h\"\n+#elif defined(_MSC_VER)\n+#include \"compat/msvc/posix.h\"\n+#else\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#ifndef NO_SYS_SELECT_H\n+#include \n+#endif\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#ifndef NO_INTTYPES_H\n+#include \n+#else\n+#include \n+#endif\n+#ifdef HAVE_ARC4RANDOM_LIBBSD\n+#include \n+#endif\n+#ifdef HAVE_GETRANDOM\n+#include \n+#endif\n+#ifdef NO_INTPTR_T\n+/*\n+ * On I16LP32, ILP32 and LP64 \"long\" is the safe bet, however\n+ * on LLP86, IL33LLP64 and P64 it needs to be \"long long\",\n+ * while on IP16 and IP16L32 it is \"int\" (resp. \"short\")\n+ * Size needs to match (or exceed) 'sizeof(void *)'.\n+ * We can't take \"long long\" here as not everybody has it.\n+ */\n+typedef long intptr_t;\n+typedef unsigned long uintptr_t;\n+#endif\n+#undef _ALL_SOURCE /* AIX 5.3L defines a struct list with _ALL_SOURCE. */\n+#include \n+#define _ALL_SOURCE 1\n+#endif\n+\n+#ifdef MKDIR_WO_TRAILING_SLASH\n+#define mkdir(a,b) compat_mkdir_wo_trailing_slash((a),(b))\n+int compat_mkdir_wo_trailing_slash(const char*, mode_t);\n+#endif\n+\n+#ifdef time\n+#undef time\n+#endif\n+static inline time_t git_time(time_t *tloc)\n+{\n+\tstruct timeval tv;\n+\n+\t/*\n+\t * Avoid time(NULL), which can disagree with gettimeofday(2)\n+\t * and filesystem timestamps.\n+\t */\n+\tgettimeofday(&tv, NULL);\n+\n+\tif (tloc)\n+\t\t*tloc = tv.tv_sec;\n+\treturn tv.tv_sec;\n+}\n+#define time git_time\n+\n+#ifdef NO_STRUCT_ITIMERVAL\n+struct itimerval {\n+\tstruct timeval it_interval;\n+\tstruct timeval it_value;\n+};\n+#endif\n+\n+#ifdef NO_SETITIMER\n+static inline int git_setitimer(int which UNUSED,\n+\t\t\t\tconst struct itimerval *value UNUSED,\n+\t\t\t\tstruct itimerval *newvalue UNUSED) {\n+\treturn 0; /* pretend success */\n+}\n+#undef setitimer\n+#define setitimer(which,value,ovalue) git_setitimer(which,value,ovalue)\n+#endif\n+\n+#ifndef NO_LIBGEN_H\n+#include \n+#else\n+#define basename gitbasename\n+char *gitbasename(char *);\n+#define dirname gitdirname\n+char *gitdirname(char *);\n+#endif\n+\n+#ifndef NO_ICONV\n+#include \n+#endif\n+\n+/* On most systems would have given us this, but\n+ * not on some systems (e.g. z/OS).\n+ */\n+#ifndef NI_MAXHOST\n+#define NI_MAXHOST 1025\n+#endif\n+\n+#ifndef NI_MAXSERV\n+#define NI_MAXSERV 32\n+#endif\n+\n+/* On most systems would have given us this, but\n+ * not on some systems (e.g. GNU/Hurd).\n+ */\n+#ifndef PATH_MAX\n+#define PATH_MAX 4096\n+#endif\n+\n+#ifndef NAME_MAX\n+#define NAME_MAX 255\n+#endif\n+\n+typedef uintmax_t timestamp_t;\n+#define PRItime PRIuMAX\n+#define parse_timestamp strtoumax\n+#define TIME_MAX UINTMAX_MAX\n+#define TIME_MIN 0\n+\n+int lstat_cache_aware_rmdir(const char *path);\n+#if !defined(__MINGW32__) && !defined(_MSC_VER)\n+#define rmdir lstat_cache_aware_rmdir\n+#endif\n+\n+#if defined(NO_MMAP) || defined(USE_WIN32_MMAP)\n+\n+#ifndef PROT_READ\n+#define PROT_READ 1\n+#define PROT_WRITE 2\n+#define MAP_PRIVATE 1\n+#endif\n+\n+#define mmap git_mmap\n+#define munmap git_munmap\n+void *git_mmap(void *start, size_t length, int prot, int flags, int fd, off_t offset);\n+int git_munmap(void *start, size_t length);\n+\n+#else /* NO_MMAP || USE_WIN32_MMAP */\n+\n+#include \n+\n+#endif /* NO_MMAP || USE_WIN32_MMAP */\n+\n+#ifndef MAP_FAILED\n+#define MAP_FAILED ((void *)-1)\n+#endif\n+\n+#ifdef NEEDS_MODE_TRANSLATION\n+#undef S_IFMT\n+#undef S_IFREG\n+#undef S_IFDIR\n+#undef S_IFLNK\n+#undef S_IFBLK\n+#undef S_IFCHR\n+#undef S_IFIFO\n+#undef S_IFSOCK\n+#define S_IFMT 0170000\n+#define S_IFREG 0100000\n+#define S_IFDIR 0040000\n+#define S_IFLNK 0120000\n+#define S_IFBLK 0060000\n+#define S_IFCHR 0020000\n+#define S_IFIFO 0010000\n+#define S_IFSOCK 0140000\n+#ifdef stat\n+#undef stat\n+#endif\n+#define stat(path, buf) git_stat(path, buf)\n+int git_stat(const char *, struct stat *);\n+#ifdef fstat\n+#undef fstat\n+#endif\n+#define fstat(fd, buf) git_fstat(fd, buf)\n+int git_fstat(int, struct stat *);\n+#ifdef lstat\n+#undef lstat\n+#endif\n+#define lstat(path, buf) git_lstat(path, buf)\n+int git_lstat(const char *, struct stat *);\n+#endif\n+\n+#ifdef NO_PREAD\n+#define pread git_pread\n+ssize_t git_pread(int fd, void *buf, size_t count, off_t offset);\n+#endif\n+\n+#ifdef NO_SETENV\n+#define setenv gitsetenv\n+int gitsetenv(const char *, const char *, int);\n+#endif\n+\n+#ifdef NO_MKDTEMP\n+#define mkdtemp gitmkdtemp\n+char *gitmkdtemp(char *);\n+#endif\n+\n+#ifdef NO_UNSETENV\n+#define unsetenv gitunsetenv\n+int gitunsetenv(const char *);\n+#endif\n+\n+#ifdef NO_STRCASESTR\n+#define strcasestr gitstrcasestr\n+char *gitstrcasestr(const char *haystack, const char *needle);\n+#endif\n+\n+#ifdef NO_STRLCPY\n+#define strlcpy gitstrlcpy\n+size_t gitstrlcpy(char *, const char *, size_t);\n+#endif\n+\n+#ifdef NO_STRTOUMAX\n+#define strtoumax gitstrtoumax\n+uintmax_t gitstrtoumax(const char *, char **, int);\n+#define strtoimax gitstrtoimax\n+intmax_t gitstrtoimax(const char *, char **, int);\n+#endif\n+\n+#ifdef NO_HSTRERROR\n+#define hstrerror githstrerror\n+const char *githstrerror(int herror);\n+#endif\n+\n+#ifdef NO_MEMMEM\n+#define memmem gitmemmem\n+void *gitmemmem(const void *haystack, size_t haystacklen,\n+\t\tconst void *needle, size_t needlelen);\n+#endif\n+\n+#ifdef OVERRIDE_STRDUP\n+#ifdef strdup\n+#undef strdup\n+#endif\n+#define strdup gitstrdup\n+char *gitstrdup(const char *s);\n+#endif\n+\n+#ifdef NO_GETPAGESIZE\n+#define getpagesize() sysconf(_SC_PAGESIZE)\n+#endif\n+\n+#ifndef O_CLOEXEC\n+#define O_CLOEXEC 0\n+#endif\n+\n+#ifdef FREAD_READS_DIRECTORIES\n+# if !defined(SUPPRESS_FOPEN_REDEFINITION)\n+# ifdef fopen\n+# undef fopen\n+# endif\n+# define fopen(a,b) git_fopen(a,b)\n+# endif\n+FILE *git_fopen(const char*, const char*);\n+#endif\n+\n+#ifdef SNPRINTF_RETURNS_BOGUS\n+#ifdef snprintf\n+#undef snprintf\n+#endif\n+#define snprintf git_snprintf\n+int git_snprintf(char *str, size_t maxsize,\n+\t\t const char *format, ...);\n+#ifdef vsnprintf\n+#undef vsnprintf\n+#endif\n+#define vsnprintf git_vsnprintf\n+int git_vsnprintf(char *str, size_t maxsize,\n+\t\t const char *format, va_list ap);\n+#endif\n+\n+#ifdef OPEN_RETURNS_EINTR\n+#undef open\n+#define open git_open_with_retry\n+int git_open_with_retry(const char *path, int flag, ...);\n+#endif\n+\n+#ifdef __GLIBC_PREREQ\n+#if __GLIBC_PREREQ(2, 1)\n+#define HAVE_STRCHRNUL\n+#endif\n+#endif\n+\n+#ifndef HAVE_STRCHRNUL\n+#define strchrnul gitstrchrnul\n+static inline char *gitstrchrnul(const char *s, int c)\n+{\n+\twhile (*s && *s != c)\n+\t\ts++;\n+\treturn (char *)s;\n+}\n+#endif\n+\n+#ifdef NO_INET_PTON\n+int inet_pton(int af, const char *src, void *dst);\n+#endif\n+\n+#ifdef NO_INET_NTOP\n+const char *inet_ntop(int af, const void *src, char *dst, size_t size);\n+#endif\n+\n+#ifdef NO_PTHREADS\n+#define atexit git_atexit\n+int git_atexit(void (*handler)(void));\n+#endif\n+\n+#ifndef HOST_NAME_MAX\n+#define HOST_NAME_MAX 256\n+#endif\n+\n+#include \"sane-ctype.h\"\n+\n+void git_stable_qsort(void *base, size_t nmemb, size_t size,\n+\t\t int(*compar)(const void *, const void *));\n+#ifdef INTERNAL_QSORT\n+#define qsort git_stable_qsort\n+#endif\n+\n+#define QSORT(base, n, compar) sane_qsort((base), (n), sizeof(*(base)), compar)\n+static inline void sane_qsort(void *base, size_t nmemb, size_t size,\n+\t\t\t int(*compar)(const void *, const void *))\n+{\n+\tif (nmemb > 1)\n+\t\tqsort(base, nmemb, size, compar);\n+}\n+\n+#define STABLE_QSORT(base, n, compar) \\\n+\tgit_stable_qsort((base), (n), sizeof(*(base)), compar)\n+\n+#ifndef HAVE_ISO_QSORT_S\n+int git_qsort_s(void *base, size_t nmemb, size_t size,\n+\t\tint (*compar)(const void *, const void *, void *), void *ctx);\n+#define qsort_s git_qsort_s\n+#endif\n+\n+#define QSORT_S(base, n, compar, ctx) do {\t\t\t\\\n+\tif (qsort_s((base), (n), sizeof(*(base)), compar, ctx))\t\\\n+\t\tBUG(\"qsort_s() failed\");\t\t\t\\\n+} while (0)\n+\n+#ifdef NO_NSEC\n+#undef USE_NSEC\n+#define ST_CTIME_NSEC(st) 0\n+#define ST_MTIME_NSEC(st) 0\n+#else\n+#ifdef USE_ST_TIMESPEC\n+#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctimespec.tv_nsec))\n+#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtimespec.tv_nsec))\n+#else\n+#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctim.tv_nsec))\n+#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtim.tv_nsec))\n+#endif\n+#endif\n+\n+#ifndef va_copy\n+/*\n+ * Since an obvious implementation of va_list would be to make it a\n+ * pointer into the stack frame, a simple assignment will work on\n+ * many systems. But let's try to be more portable.\n+ */\n+#ifdef __va_copy\n+#define va_copy(dst, src) __va_copy(dst, src)\n+#else\n+#define va_copy(dst, src) ((dst) = (src))\n+#endif\n+#endif\n+\n+#ifndef _POSIX_THREAD_SAFE_FUNCTIONS\n+static inline void git_flockfile(FILE *fh UNUSED)\n+{\n+\t; /* nothing */\n+}\n+static inline void git_funlockfile(FILE *fh UNUSED)\n+{\n+\t; /* nothing */\n+}\n+#undef flockfile\n+#undef funlockfile\n+#undef getc_unlocked\n+#define flockfile(fh) git_flockfile(fh)\n+#define funlockfile(fh) git_funlockfile(fh)\n+#define getc_unlocked(fh) getc(fh)\n+#endif\n+\n+#ifdef FILENO_IS_A_MACRO\n+int git_fileno(FILE *stream);\n+# ifndef COMPAT_CODE_FILENO\n+# undef fileno\n+# define fileno(p) git_fileno(p)\n+# endif\n+#endif\n+\n+#ifdef NEED_ACCESS_ROOT_HANDLER\n+int git_access(const char *path, int mode);\n+# ifndef COMPAT_CODE_ACCESS\n+# ifdef access\n+# undef access\n+# endif\n+# define access(path, mode) git_access(path, mode)\n+# endif\n+#endif\n+\n+#endif /* COMPAT_POSIX_H */\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex b96fb98e1e..30ad0725fb 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -23,26 +23,9 @@\n #include \n #endif\n \n-struct strbuf;\n-\n-\n-#define _FILE_OFFSET_BITS 64\n+#include \"compat/posix.h\"\n \n-\n-/* Derived from Linux \"Features Test Macro\" header\n- * Convenience macros to test the versions of gcc (or\n- * a compatible compiler).\n- * Use them like this:\n- * #if GIT_GNUC_PREREQ (2,8)\n- * ... code requiring gcc 2.8 or later ...\n- * #endif\n-*/\n-#if defined(__GNUC__) && defined(__GNUC_MINOR__)\n-# define GIT_GNUC_PREREQ(maj, min) \\\n-\t((__GNUC__ << 16) + __GNUC_MINOR__ >= ((maj) << 16) + (min))\n-#else\n- #define GIT_GNUC_PREREQ(maj, min) 0\n-#endif\n+struct strbuf;\n \n #if defined(__GNUC__) || defined(__clang__)\n # define PRAGMA(pragma) _Pragma(#pragma)\n@@ -176,71 +159,6 @@ DISABLE_WARNING(-Wsign-compare)\n /* Approximation of the length of the decimal representation of this type. */\n #define decimal_length(x)\t((int)(sizeof(x) * 2.56 + 0.5) + 1)\n \n-#ifdef __MINGW64__\n-#define _POSIX_C_SOURCE 1\n-#elif defined(__sun__)\n- /*\n- * On Solaris, when _XOPEN_EXTENDED is set, its header file\n- * forces the programs to be XPG4v2, defeating any _XOPEN_SOURCE\n- * setting to say we are XPG5 or XPG6. Also on Solaris,\n- * XPG6 programs must be compiled with a c99 compiler, while\n- * non XPG6 programs must be compiled with a pre-c99 compiler.\n- */\n-# if __STDC_VERSION__ - 0 >= 199901L\n-# define _XOPEN_SOURCE 600\n-# else\n-# define _XOPEN_SOURCE 500\n-# endif\n-#elif !defined(__APPLE__) && !defined(__FreeBSD__) && !defined(__USLC__) && \\\n- !defined(_M_UNIX) && !defined(__sgi) && !defined(__DragonFly__) && \\\n- !defined(__TANDEM) && !defined(__QNX__) && !defined(__MirBSD__) && \\\n- !defined(__CYGWIN__)\n-#define _XOPEN_SOURCE 600 /* glibc2 and AIX 5.3L need 500, OpenBSD needs 600 for S_ISLNK() */\n-#define _XOPEN_SOURCE_EXTENDED 1 /* AIX 5.3L needs this */\n-#endif\n-#define _ALL_SOURCE 1\n-#define _GNU_SOURCE 1\n-#define _BSD_SOURCE 1\n-#define _DEFAULT_SOURCE 1\n-#define _NETBSD_SOURCE 1\n-#define _SGI_SOURCE 1\n-\n-/*\n- * UNUSED marks a function parameter that is always unused. It also\n- * can be used to annotate a function, a variable, or a type that is\n- * always unused.\n- *\n- * A callback interface may dictate that a function accepts a\n- * parameter at that position, but the implementation of the function\n- * may not need to use the parameter. In such a case, mark the parameter\n- * with UNUSED.\n- *\n- * When a parameter may be used or unused, depending on conditional\n- * compilation, consider using MAYBE_UNUSED instead.\n- */\n-#if GIT_GNUC_PREREQ(4, 5)\n-#define UNUSED __attribute__((unused)) \\\n-\t__attribute__((deprecated (\"parameter declared as UNUSED\")))\n-#elif defined(__GNUC__)\n-#define UNUSED __attribute__((unused)) \\\n-\t__attribute__((deprecated))\n-#else\n-#define UNUSED\n-#endif\n-\n-#if defined(WIN32) && !defined(__CYGWIN__) /* Both MinGW and MSVC */\n-# if !defined(_WIN32_WINNT)\n-# define _WIN32_WINNT 0x0600\n-# endif\n-#define WIN32_LEAN_AND_MEAN /* stops windows.h including winsock.h */\n-#include \n-#ifndef NO_UNIX_SOCKETS\n-#include \n-#endif\n-#include \n-#define GIT_WINDOWS_NATIVE\n-#endif\n-\n #if defined(NO_UNIX_SOCKETS) || !defined(GIT_WINDOWS_NATIVE)\n static inline int _have_unix_sockets(void)\n {\n@@ -253,45 +171,6 @@ static inline int _have_unix_sockets(void)\n #define have_unix_sockets _have_unix_sockets\n #endif\n \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#ifdef HAVE_STRINGS_H\n-#include /* for strcasecmp() */\n-#endif\n-#include \n-#include \n-#include \n-#ifdef NEEDS_SYS_PARAM_H\n-#include \n-#endif\n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#if !defined(NO_POLL_H)\n-#include \n-#elif !defined(NO_SYS_POLL_H)\n-#include \n-#else\n-/* Pull the compat stuff */\n-#include \n-#endif\n-#ifdef HAVE_BSD_SYSCTL\n-#include \n-#endif\n-\n /* Used by compat/win32/path-utils.h, and more */\n static inline int is_xplatform_dir_sep(int c)\n {\n@@ -308,48 +187,6 @@ static inline int is_xplatform_dir_sep(int c)\n #elif defined(_MSC_VER)\n #include \"compat/win32/path-utils.h\"\n #include \"compat/msvc/compat-util.h\"\n-#else\n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#ifndef NO_SYS_SELECT_H\n-#include \n-#endif\n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#ifndef NO_INTTYPES_H\n-#include \n-#else\n-#include \n-#endif\n-#ifdef HAVE_ARC4RANDOM_LIBBSD\n-#include \n-#endif\n-#ifdef HAVE_GETRANDOM\n-#include \n-#endif\n-#ifdef NO_INTPTR_T\n-/*\n- * On I16LP32, ILP32 and LP64 \"long\" is the safe bet, however\n- * on LLP86, IL33LLP64 and P64 it needs to be \"long long\",\n- * while on IP16 and IP16L32 it is \"int\" (resp. \"short\")\n- * Size needs to match (or exceed) 'sizeof(void *)'.\n- * We can't take \"long long\" here as not everybody has it.\n- */\n-typedef long intptr_t;\n-typedef unsigned long uintptr_t;\n-#endif\n-#undef _ALL_SOURCE /* AIX 5.3L defines a struct list with _ALL_SOURCE. */\n-#include \n-#define _ALL_SOURCE 1\n #endif\n \n /* used on Mac OS X */\n@@ -370,60 +207,6 @@ static inline const char *precompose_string_if_needed(const char *in)\n #define probe_utf8_pathname_composition()\n #endif\n \n-#ifdef MKDIR_WO_TRAILING_SLASH\n-#define mkdir(a,b) compat_mkdir_wo_trailing_slash((a),(b))\n-int compat_mkdir_wo_trailing_slash(const char*, mode_t);\n-#endif\n-\n-#ifdef time\n-#undef time\n-#endif\n-static inline time_t git_time(time_t *tloc)\n-{\n-\tstruct timeval tv;\n-\n-\t/*\n-\t * Avoid time(NULL), which can disagree with gettimeofday(2)\n-\t * and filesystem timestamps.\n-\t */\n-\tgettimeofday(&tv, NULL);\n-\n-\tif (tloc)\n-\t\t*tloc = tv.tv_sec;\n-\treturn tv.tv_sec;\n-}\n-#define time git_time\n-\n-#ifdef NO_STRUCT_ITIMERVAL\n-struct itimerval {\n-\tstruct timeval it_interval;\n-\tstruct timeval it_value;\n-};\n-#endif\n-\n-#ifdef NO_SETITIMER\n-static inline int git_setitimer(int which UNUSED,\n-\t\t\t\tconst struct itimerval *value UNUSED,\n-\t\t\t\tstruct itimerval *newvalue UNUSED) {\n-\treturn 0; /* pretend success */\n-}\n-#undef setitimer\n-#define setitimer(which,value,ovalue) git_setitimer(which,value,ovalue)\n-#endif\n-\n-#ifndef NO_LIBGEN_H\n-#include \n-#else\n-#define basename gitbasename\n-char *gitbasename(char *);\n-#define dirname gitdirname\n-char *gitdirname(char *);\n-#endif\n-\n-#ifndef NO_ICONV\n-#include \n-#endif\n-\n #ifndef NO_OPENSSL\n #ifdef __APPLE__\n #undef __AVAILABILITY_MACROS_USES_AVAILABILITY\n@@ -441,34 +224,6 @@ char *gitdirname(char *);\n # include \n #endif\n \n-/* On most systems would have given us this, but\n- * not on some systems (e.g. z/OS).\n- */\n-#ifndef NI_MAXHOST\n-#define NI_MAXHOST 1025\n-#endif\n-\n-#ifndef NI_MAXSERV\n-#define NI_MAXSERV 32\n-#endif\n-\n-/* On most systems would have given us this, but\n- * not on some systems (e.g. GNU/Hurd).\n- */\n-#ifndef PATH_MAX\n-#define PATH_MAX 4096\n-#endif\n-\n-#ifndef NAME_MAX\n-#define NAME_MAX 255\n-#endif\n-\n-typedef uintmax_t timestamp_t;\n-#define PRItime PRIuMAX\n-#define parse_timestamp strtoumax\n-#define TIME_MAX UINTMAX_MAX\n-#define TIME_MIN 0\n-\n #ifndef PATH_SEP\n #define PATH_SEP ':'\n #endif\n@@ -492,11 +247,6 @@ static inline int noop_core_config(const char *var UNUSED,\n #define platform_core_config noop_core_config\n #endif\n \n-int lstat_cache_aware_rmdir(const char *path);\n-#if !defined(__MINGW32__) && !defined(_MSC_VER)\n-#define rmdir lstat_cache_aware_rmdir\n-#endif\n-\n #ifndef has_dos_drive_prefix\n static inline int git_has_dos_drive_prefix(const char *path UNUSED)\n {\n@@ -822,25 +572,6 @@ static inline bool strip_suffix(const char *str, const char *suffix,\n \tmemcpy(_swap_b_ptr, _swap_buffer, sizeof(a));\t\t\\\n } while (0)\n \n-#if defined(NO_MMAP) || defined(USE_WIN32_MMAP)\n-\n-#ifndef PROT_READ\n-#define PROT_READ 1\n-#define PROT_WRITE 2\n-#define MAP_PRIVATE 1\n-#endif\n-\n-#define mmap git_mmap\n-#define munmap git_munmap\n-void *git_mmap(void *start, size_t length, int prot, int flags, int fd, off_t offset);\n-int git_munmap(void *start, size_t length);\n-\n-#else /* NO_MMAP || USE_WIN32_MMAP */\n-\n-#include \n-\n-#endif /* NO_MMAP || USE_WIN32_MMAP */\n-\n #ifdef NO_MMAP\n \n /* This value must be multiple of (pagesize * 2) */\n@@ -856,177 +587,15 @@ int git_munmap(void *start, size_t length);\n \n #endif /* NO_MMAP */\n \n-#ifndef MAP_FAILED\n-#define MAP_FAILED ((void *)-1)\n-#endif\n-\n #ifdef NO_ST_BLOCKS_IN_STRUCT_STAT\n #define on_disk_bytes(st) ((st).st_size)\n #else\n #define on_disk_bytes(st) ((st).st_blocks * 512)\n #endif\n \n-#ifdef NEEDS_MODE_TRANSLATION\n-#undef S_IFMT\n-#undef S_IFREG\n-#undef S_IFDIR\n-#undef S_IFLNK\n-#undef S_IFBLK\n-#undef S_IFCHR\n-#undef S_IFIFO\n-#undef S_IFSOCK\n-#define S_IFMT 0170000\n-#define S_IFREG 0100000\n-#define S_IFDIR 0040000\n-#define S_IFLNK 0120000\n-#define S_IFBLK 0060000\n-#define S_IFCHR 0020000\n-#define S_IFIFO 0010000\n-#define S_IFSOCK 0140000\n-#ifdef stat\n-#undef stat\n-#endif\n-#define stat(path, buf) git_stat(path, buf)\n-int git_stat(const char *, struct stat *);\n-#ifdef fstat\n-#undef fstat\n-#endif\n-#define fstat(fd, buf) git_fstat(fd, buf)\n-int git_fstat(int, struct stat *);\n-#ifdef lstat\n-#undef lstat\n-#endif\n-#define lstat(path, buf) git_lstat(path, buf)\n-int git_lstat(const char *, struct stat *);\n-#endif\n-\n #define DEFAULT_PACKED_GIT_LIMIT \\\n \t((1024L * 1024L) * (size_t)(sizeof(void*) >= 8 ? (32 * 1024L * 1024L) : 256))\n \n-#ifdef NO_PREAD\n-#define pread git_pread\n-ssize_t git_pread(int fd, void *buf, size_t count, off_t offset);\n-#endif\n-\n-#ifdef NO_SETENV\n-#define setenv gitsetenv\n-int gitsetenv(const char *, const char *, int);\n-#endif\n-\n-#ifdef NO_MKDTEMP\n-#define mkdtemp gitmkdtemp\n-char *gitmkdtemp(char *);\n-#endif\n-\n-#ifdef NO_UNSETENV\n-#define unsetenv gitunsetenv\n-int gitunsetenv(const char *);\n-#endif\n-\n-#ifdef NO_STRCASESTR\n-#define strcasestr gitstrcasestr\n-char *gitstrcasestr(const char *haystack, const char *needle);\n-#endif\n-\n-#ifdef NO_STRLCPY\n-#define strlcpy gitstrlcpy\n-size_t gitstrlcpy(char *, const char *, size_t);\n-#endif\n-\n-#ifdef NO_STRTOUMAX\n-#define strtoumax gitstrtoumax\n-uintmax_t gitstrtoumax(const char *, char **, int);\n-#define strtoimax gitstrtoimax\n-intmax_t gitstrtoimax(const char *, char **, int);\n-#endif\n-\n-#ifdef NO_HSTRERROR\n-#define hstrerror githstrerror\n-const char *githstrerror(int herror);\n-#endif\n-\n-#ifdef NO_MEMMEM\n-#define memmem gitmemmem\n-void *gitmemmem(const void *haystack, size_t haystacklen,\n-\t\tconst void *needle, size_t needlelen);\n-#endif\n-\n-#ifdef OVERRIDE_STRDUP\n-#ifdef strdup\n-#undef strdup\n-#endif\n-#define strdup gitstrdup\n-char *gitstrdup(const char *s);\n-#endif\n-\n-#ifdef NO_GETPAGESIZE\n-#define getpagesize() sysconf(_SC_PAGESIZE)\n-#endif\n-\n-#ifndef O_CLOEXEC\n-#define O_CLOEXEC 0\n-#endif\n-\n-#ifdef FREAD_READS_DIRECTORIES\n-# if !defined(SUPPRESS_FOPEN_REDEFINITION)\n-# ifdef fopen\n-# undef fopen\n-# endif\n-# define fopen(a,b) git_fopen(a,b)\n-# endif\n-FILE *git_fopen(const char*, const char*);\n-#endif\n-\n-#ifdef SNPRINTF_RETURNS_BOGUS\n-#ifdef snprintf\n-#undef snprintf\n-#endif\n-#define snprintf git_snprintf\n-int git_snprintf(char *str, size_t maxsize,\n-\t\t const char *format, ...);\n-#ifdef vsnprintf\n-#undef vsnprintf\n-#endif\n-#define vsnprintf git_vsnprintf\n-int git_vsnprintf(char *str, size_t maxsize,\n-\t\t const char *format, va_list ap);\n-#endif\n-\n-#ifdef OPEN_RETURNS_EINTR\n-#undef open\n-#define open git_open_with_retry\n-int git_open_with_retry(const char *path, int flag, ...);\n-#endif\n-\n-#ifdef __GLIBC_PREREQ\n-#if __GLIBC_PREREQ(2, 1)\n-#define HAVE_STRCHRNUL\n-#endif\n-#endif\n-\n-#ifndef HAVE_STRCHRNUL\n-#define strchrnul gitstrchrnul\n-static inline char *gitstrchrnul(const char *s, int c)\n-{\n-\twhile (*s && *s != c)\n-\t\ts++;\n-\treturn (char *)s;\n-}\n-#endif\n-\n-#ifdef NO_INET_PTON\n-int inet_pton(int af, const char *src, void *dst);\n-#endif\n-\n-#ifdef NO_INET_NTOP\n-const char *inet_ntop(int af, const void *src, char *dst, size_t size);\n-#endif\n-\n-#ifdef NO_PTHREADS\n-#define atexit git_atexit\n-int git_atexit(void (*handler)(void));\n-#endif\n-\n static inline size_t st_add(size_t a, size_t b)\n {\n \tif (unsigned_add_overflows(a, b))\n@@ -1293,12 +862,6 @@ static inline size_t xsize_t(off_t len)\n \treturn (size_t) len;\n }\n \n-#ifndef HOST_NAME_MAX\n-#define HOST_NAME_MAX 256\n-#endif\n-\n-#include \"sane-ctype.h\"\n-\n /*\n * Like skip_prefix, but compare case-insensitively. Note that the comparison\n * is done via tolower(), so it is strictly ASCII (no multi-byte characters or\n@@ -1364,34 +927,6 @@ static inline int strtol_i(char const *s, int base, int *result)\n \treturn 0;\n }\n \n-void git_stable_qsort(void *base, size_t nmemb, size_t size,\n-\t\t int(*compar)(const void *, const void *));\n-#ifdef INTERNAL_QSORT\n-#define qsort git_stable_qsort\n-#endif\n-\n-#define QSORT(base, n, compar) sane_qsort((base), (n), sizeof(*(base)), compar)\n-static inline void sane_qsort(void *base, size_t nmemb, size_t size,\n-\t\t\t int(*compar)(const void *, const void *))\n-{\n-\tif (nmemb > 1)\n-\t\tqsort(base, nmemb, size, compar);\n-}\n-\n-#define STABLE_QSORT(base, n, compar) \\\n-\tgit_stable_qsort((base), (n), sizeof(*(base)), compar)\n-\n-#ifndef HAVE_ISO_QSORT_S\n-int git_qsort_s(void *base, size_t nmemb, size_t size,\n-\t\tint (*compar)(const void *, const void *, void *), void *ctx);\n-#define qsort_s git_qsort_s\n-#endif\n-\n-#define QSORT_S(base, n, compar, ctx) do {\t\t\t\\\n-\tif (qsort_s((base), (n), sizeof(*(base)), compar, ctx))\t\\\n-\t\tBUG(\"qsort_s() failed\");\t\t\t\\\n-} while (0)\n-\n #ifndef REG_STARTEND\n #error \"Git requires REG_STARTEND support. Compile with NO_REGEX=NeedsStartEnd\"\n #endif\n@@ -1416,39 +951,12 @@ int git_regcomp(regex_t *preg, const char *pattern, int cflags);\n # define FORCE_DIR_SET_GID 0\n #endif\n \n-#ifdef NO_NSEC\n-#undef USE_NSEC\n-#define ST_CTIME_NSEC(st) 0\n-#define ST_MTIME_NSEC(st) 0\n-#else\n-#ifdef USE_ST_TIMESPEC\n-#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctimespec.tv_nsec))\n-#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtimespec.tv_nsec))\n-#else\n-#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctim.tv_nsec))\n-#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtim.tv_nsec))\n-#endif\n-#endif\n-\n #ifdef UNRELIABLE_FSTAT\n #define fstat_is_reliable() 0\n #else\n #define fstat_is_reliable() 1\n #endif\n \n-#ifndef va_copy\n-/*\n- * Since an obvious implementation of va_list would be to make it a\n- * pointer into the stack frame, a simple assignment will work on\n- * many systems. But let's try to be more portable.\n- */\n-#ifdef __va_copy\n-#define va_copy(dst, src) __va_copy(dst, src)\n-#else\n-#define va_copy(dst, src) ((dst) = (src))\n-#endif\n-#endif\n-\n /* usage.c: only to be used for testing BUG() implementation (see test-tool) */\n extern int BUG_exit_code;\n \n@@ -1478,41 +986,6 @@ void bug_fl(const char *file, int line, const char *fmt, ...);\n # define SHELL_PATH \"/bin/sh\"\n #endif\n \n-#ifndef _POSIX_THREAD_SAFE_FUNCTIONS\n-static inline void git_flockfile(FILE *fh UNUSED)\n-{\n-\t; /* nothing */\n-}\n-static inline void git_funlockfile(FILE *fh UNUSED)\n-{\n-\t; /* nothing */\n-}\n-#undef flockfile\n-#undef funlockfile\n-#undef getc_unlocked\n-#define flockfile(fh) git_flockfile(fh)\n-#define funlockfile(fh) git_funlockfile(fh)\n-#define getc_unlocked(fh) getc(fh)\n-#endif\n-\n-#ifdef FILENO_IS_A_MACRO\n-int git_fileno(FILE *stream);\n-# ifndef COMPAT_CODE_FILENO\n-# undef fileno\n-# define fileno(p) git_fileno(p)\n-# endif\n-#endif\n-\n-#ifdef NEED_ACCESS_ROOT_HANDLER\n-int git_access(const char *path, int mode);\n-# ifndef COMPAT_CODE_ACCESS\n-# ifdef access\n-# undef access\n-# endif\n-# define access(path, mode) git_access(path, mode)\n-# endif\n-#endif\n-\n /*\n * Our code often opens a path to an optional file, to work on its\n * contents when we can successfully open it. We can ignore a failure\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511259","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-18-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 18/19] reftable: decouple from Git codebase by pulling in \"compat/posix.h\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:26Z","receivedAt":"2025-01-27T13:04:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library includes \"git-compat-util.h\" in order to get a\nPOSIX-like programming environment that papers over various differences\nbetween platforms. The header also brings with it a couple of helpers\nspecific to the Git codebase though, and over time we have started to\nuse these helpers in the reftable library, as well.\n\nThis makes it very hard to use the reftable library as a standalone\nlibrary without the rest of the Git codebase, so other libraries like\ne.g. libgit2 cannot easily use it. But now that we have removed all\ncalls to Git-specific functionality and have split out \"compat/posix.h\"\nas a separate header we can address this.\n\nStop including \"git-compat-util.h\" and instead include \"compat/posix.h\"\nto finalize the decoupling of the reftable library from the rest of the\nGit codebase. The only bits which remain specific to Git are \"system.h\"\nand \"system.c\", which projects will have to provide.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/system.c | 2 ++\n reftable/system.h | 3 ++-\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/reftable/system.c b/reftable/system.c\nindex e25ccc0da3..1ee268b125 100644\n--- a/reftable/system.c\n+++ b/reftable/system.c\n@@ -1,3 +1,5 @@\n+#include \"../git-compat-util.h\"\n+\n #include \"system.h\"\n #include \"basics.h\"\n #include \"reftable-error.h\"\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 3bd4a4e322..e4a8944a70 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -11,7 +11,8 @@ license that can be found in the LICENSE file or at\n \n /* This header glues the reftable library to the rest of Git */\n \n-#include \"git-compat-util.h\"\n+#include \"compat/posix.h\"\n+#include \n \n /*\n * Return a random 32 bit integer. This function is expected to return\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511260","messageId":"20250127-pks-reftable-drop-git-compat-util-v1-19-6e280a564877@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH 19/19] Makefile: skip reftable library for Coccinelle","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-27T13:04:27Z","receivedAt":"2025-01-27T13:04:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library does not use any of the common helpers that the Git\nproject has. Consequently, most of the rules that we have in Coccinelle\ndo not apply to the library at all and may even generate false positives\nwhen a pattern can be converted to use a Git helper function.\n\nExclude reftable library sources from being checked by Coccinelle to\navoid such false positives.\n\nSigned-off-by: Patrick Steinhardt \n---\n Makefile | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex d3011e30f7..dc2ac32e4a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -946,7 +946,7 @@ FOUND_SOURCE_FILES := $(filter-out $(GENERATED_H),$(shell $(SOURCES_CMD)))\n FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES))\n FOUND_H_SOURCES = $(filter %.h,$(FOUND_SOURCE_FILES))\n \n-COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES))\n+COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES) reftable/%,$(FOUND_C_SOURCES))\n \n LIB_H = $(FOUND_H_SOURCES)\n \n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511282","messageId":"u7hbqd6at3owgka4pitd4vgzc5emgipnx3w4dat5ofpiv5hxxo@lbfj36uhqtsr","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-1-6e280a564877@pks.im","subject":"Re: [PATCH 01/19] reftable/stack: stop using `read_in_full()`","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-01-27T16:57:20Z","receivedAt":"2025-01-27T17:00:13Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/01/27 02:04PM, Patrick Steinhardt wrote:\n> There is a single callsite of `read_in_full()` in the reftable library.\n> Open-code the function to reduce our dependency on the Git library.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n> reftable/stack.c | 18 ++++++++++++++----\n> 1 file changed, 14 insertions(+), 4 deletions(-)\n> \n> diff --git a/reftable/stack.c b/reftable/stack.c\n> index f7c1845e15..9490366795 100644\n> --- a/reftable/stack.c\n> +++ b/reftable/stack.c\n> @@ -115,13 +115,16 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n> \n> static int fd_read_lines(int fd, char ***namesp)\n> {\n> -\toff_t size = lseek(fd, 0, SEEK_END);\n> \tchar *buf = NULL;\n> \tint err = 0;\n> +\toff_t size;\n> +\n> +\tsize = lseek(fd, 0, SEEK_END);\n> \tif (size < 0) {\n> \t\terr = REFTABLE_IO_ERROR;\n> \t\tgoto done;\n> \t}\n> +\n> \terr = lseek(fd, 0, SEEK_SET);\n> \tif (err < 0) {\n> \t\terr = REFTABLE_IO_ERROR;\n> @@ -134,9 +137,16 @@ static int fd_read_lines(int fd, char ***namesp)\n> \t\tgoto done;\n> \t}\n> \n> -\tif (read_in_full(fd, buf, size) != size) {\n> -\t\terr = REFTABLE_IO_ERROR;\n> -\t\tgoto done;\n> +\tfor (size_t total_read = 0; total_read < (size_t) size; ) {\n\nThe cast from off_t -> size_t matches the currect behavior, but is it\nalways safe to do this? In `git-compat-util.h` it looks like we have\n`xsize_t()` to safely handle these conversions. Since this series is\nmoving away from `git-compat-util.h` should ideally something similar be\nimplemented?\n\n> +\t\tssize_t bytes_read = read(fd, buf + total_read, size - total_read);\n> +\t\tif (bytes_read < 0 && (errno == EAGAIN || errno == EINTR))\n\nThe error handling here for EAGAIN doesn't go as far as what `xread()`\ndoes via `handle_nonblock()`. In this scenario is that ok?\n\n> +\t\t\tcontinue;\n> +\t\tif (bytes_read < 0 || !bytes_read) {\n> +\t\t\terr = REFTABLE_IO_ERROR;\n> +\t\t\tgoto done;\n> +\t\t}\n> +\n> +\t\ttotal_read += bytes_read;\n> \t}\n> \tbuf[size] = 0;\n> \n> \n> -- \n> 2.48.1.362.g079036d154.dirty\n> \n> \n"},{"id":"511284","messageId":"v3rkbym7a6tm3co6okuleppiabx6t4zxbcn5iei66iv3dxqkrf@wmqsctl2mmgn","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-5-6e280a564877@pks.im","subject":"Re: [PATCH 05/19] reftable/record: stop using `BUG()` in `reftable_record_init()`","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-01-27T17:36:57Z","receivedAt":"2025-01-27T17:39:49Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/01/27 02:04PM, Patrick Steinhardt wrote:\n> We're aborting the program via `BUG()` in case `reftable_record_init()`\n> was invoked with an unknown record type. This is bad because we may now\n> die in library code, and because it makes us depend on the Git codebase.\n> \n> Refactor the code such that `reftable_record_init()` can return an error\n> code to the caller. Adapt any callers accordingly.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n[snip]\n> diff --git a/reftable/record.c b/reftable/record.c\n> index d1664c47ca..31985bb977 100644\n> --- a/reftable/record.c\n> +++ b/reftable/record.c\n> @@ -1301,7 +1301,7 @@ reftable_record_vtable(struct reftable_record *rec)\n> \tabort();\n> }\n> \n> -void reftable_record_init(struct reftable_record *rec, uint8_t typ)\n> +int reftable_record_init(struct reftable_record *rec, uint8_t typ)\n> {\n> \tmemset(rec, 0, sizeof(*rec));\n> \trec->type = typ;\n> @@ -1310,11 +1310,11 @@ void reftable_record_init(struct reftable_record *rec, uint8_t typ)\n> \tcase BLOCK_TYPE_REF:\n> \tcase BLOCK_TYPE_LOG:\n> \tcase BLOCK_TYPE_OBJ:\n> -\t\treturn;\n> +\t\treturn 0;\n> \tcase BLOCK_TYPE_INDEX:\n> \t\treftable_buf_init(&rec->u.idx.last_key);\n> -\t\treturn;\n> +\t\treturn 0;\n> \tdefault:\n> -\t\tBUG(\"unhandled record type\");\n> +\t\treturn REFTABLE_API_ERROR;\n\n\nI was initially unsure if `REFTABLE_API_ERROR` would be the most\nappropriate error to return here in this situation, but looking at its\ndocumented use case, I would say this fits as a \"misuse of the API\". The\nother option would be to add a more granular error type to indicate the\nunsupported record type, but that seems unnecessary here.\n\n> \t}\n> }\n"},{"id":"511285","messageId":"xmqqtt9krw1z.fsf@gitster.g","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"Re: [PATCH 00/19] reftable: stop using \"git-compat-util.h\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-01-27T17:44:24Z","receivedAt":"2025-01-27T17:44:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> This patch series stops using all kinds of helpers exposed by our\n> \"git-compat-util.h\" header and open-codes them instead. In order to keep\n> us from using these helpers by accident the final step is to pull out\n> POSIX-related bits and pieces into a new \"compat/posix.h\" header, which\n> the reftable library then uses instead of \"git-compat-util.h\".\n\nVery nice.\n\nIs there something we can also do in order to keep reftable from using\nstale version of these helpers that it copied with this series when\nwe make improvements on our side to the original?\n\nI think the answer might be \"then use a common library\nimplementation that is used by both Git and reftable\", but then we\nmight be in the same place as before? I dunno.\n\nWhat do \"libification\" folks think? Anybody?\n\n"},{"id":"511296","messageId":"jvue6ynjjlnto657pvobi2trx4xajnu7nyeqc4vnumy64ej4at@5g4wf3vko6a2","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-6-6e280a564877@pks.im","subject":"Re: [PATCH 06/19] reftable/record: don't `BUG()` in `reftable_record_cmp()`","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-01-27T19:21:55Z","receivedAt":"2025-01-27T19:24:47Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/01/27 02:04PM, Patrick Steinhardt wrote:\n> The refatble library aborts with a bug in case `reftable_record_cmp()`\n\ns/refatble/reftable/\n\n> is invoked with two records of differing types. This would cause the\n> program to die without the caller being able to handle the error, which\n> is not something we want in the context of library code. And it ties us\n> to the Git codebase.\n> \n> Refactor the code such that `reftable_record_cmp()` returns an error\n> code separate from the actual comparison result. This requires us to\n> also adapt some callers up the callchain in a similar fashion.\n> \n> Signed-off-by: Patrick Steinhardt \n"},{"id":"511298","messageId":"qpwuo3av2zvopiip75lxuntugfekym7tncdv4e3aww2n74ivbo@imsgukjwneu6","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-17-6e280a564877@pks.im","subject":"Re: [PATCH 17/19] git-compat-util.h: split out POSIX-emulating bits","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-01-27T19:25:31Z","receivedAt":"2025-01-27T19:28:24Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/01/27 02:04PM, Patrick Steinhardt wrote:\n> The \"git-compat-util.h\" header is a treasure trove of various bits and\n> pieces used throughout the project. It basically mixes two different\n> things into one:\n> \n> - Providing a POSIX-like interface even on platforms that aren't\n> POSIX-compliant.\n> \n> - Providing low-level functionality that is specific to Git.\n> \n> This intermixing is a bit of a problem for the reftable library as we\n> don't want to recreate the POSIX-like interface there. But neither do we\n> want to pull in the Git-specific functionality, as it is otherwise quite\n> easy to start depedning on the Git codebase again.\n\ns/depedning/depending/\n\n> Split out a new header \"compat/posix.h\" that only contains the bits and\n> pieces relevant for the emulation of POSIX, which we will start using in\n> the next commit.\n> \n> Signed-off-by: Patrick Steinhardt \n"},{"id":"511315","messageId":"Z5iQH4C2JqazaJUl@pks.im","threadId":"62862","inReplyTo":"u7hbqd6at3owgka4pitd4vgzc5emgipnx3w4dat5ofpiv5hxxo@lbfj36uhqtsr","subject":"Re: [PATCH 01/19] reftable/stack: stop using `read_in_full()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:06:55Z","receivedAt":"2025-01-28T08:07:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jan 27, 2025 at 10:57:20AM -0600, Justin Tobler wrote:\n> On 25/01/27 02:04PM, Patrick Steinhardt wrote:\n> > diff --git a/reftable/stack.c b/reftable/stack.c\n> > index f7c1845e15..9490366795 100644\n> > --- a/reftable/stack.c\n> > +++ b/reftable/stack.c\n> > @@ -115,13 +115,16 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n> > \n> > static int fd_read_lines(int fd, char ***namesp)\n> > {\n> > -\toff_t size = lseek(fd, 0, SEEK_END);\n> > \tchar *buf = NULL;\n> > \tint err = 0;\n> > +\toff_t size;\n> > +\n> > +\tsize = lseek(fd, 0, SEEK_END);\n> > \tif (size < 0) {\n> > \t\terr = REFTABLE_IO_ERROR;\n> > \t\tgoto done;\n> > \t}\n> > +\n> > \terr = lseek(fd, 0, SEEK_SET);\n> > \tif (err < 0) {\n> > \t\terr = REFTABLE_IO_ERROR;\n> > @@ -134,9 +137,16 @@ static int fd_read_lines(int fd, char ***namesp)\n> > \t\tgoto done;\n> > \t}\n> > \n> > -\tif (read_in_full(fd, buf, size) != size) {\n> > -\t\terr = REFTABLE_IO_ERROR;\n> > -\t\tgoto done;\n> > +\tfor (size_t total_read = 0; total_read < (size_t) size; ) {\n> \n> The cast from off_t -> size_t matches the currect behavior, but is it\n> always safe to do this? In `git-compat-util.h` it looks like we have\n> `xsize_t()` to safely handle these conversions. Since this series is\n> moving away from `git-compat-util.h` should ideally something similar be\n> implemented?\n\nIt is safe, because a couple lines further up we check for `size < 0`\nand error out if that is the case. So we know it's a positive integer,\nand thus it can be represented via `size_t`.\n\n> > +\t\tssize_t bytes_read = read(fd, buf + total_read, size - total_read);\n> > +\t\tif (bytes_read < 0 && (errno == EAGAIN || errno == EINTR))\n> \n> The error handling here for EAGAIN doesn't go as far as what `xread()`\n> does via `handle_nonblock()`. In this scenario is that ok?\n\nYes, because we don't set `O_NONBLOCK` in the reftable library.\n\nI'll note that in the commit message.\n\nPatrick\n"},{"id":"511316","messageId":"Z5iTz69yq-IF9qU0@pks.im","threadId":"62862","inReplyTo":"xmqqtt9krw1z.fsf@gitster.g","subject":"Re: [PATCH 00/19] reftable: stop using \"git-compat-util.h\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:22:39Z","receivedAt":"2025-01-28T08:22:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jan 27, 2025 at 09:44:24AM -0800, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > This patch series stops using all kinds of helpers exposed by our\n> > \"git-compat-util.h\" header and open-codes them instead. In order to keep\n> > us from using these helpers by accident the final step is to pull out\n> > POSIX-related bits and pieces into a new \"compat/posix.h\" header, which\n> > the reftable library then uses instead of \"git-compat-util.h\".\n> \n> Very nice.\n> \n> Is there something we can also do in order to keep reftable from using\n> stale version of these helpers that it copied with this series when\n> we make improvements on our side to the original?\n\nOverall the amount of duplication is quite limited. It looks like a lot\nin this patch series, but many of the changes are really only a couple\nof lines of code that are quite unlikely to grow stale. The subsystems\nthat really are complex (think tempfiles, lockfiles) use the same\nunderlying implementation as Git does via shims in \"reftable/system.c\".\n\nMy hope is that by starting to use the reftable library in libgit2 we'll\nget additional test coverage thereof that helps weed out any issues we\nhave, including issues in the compatibility layer. But I also assume\nthat almost all bugs that we'll find will not be in the low-level code,\nbut rather in the business logic of reftables themselves.\n\n> I think the answer might be \"then use a common library\n> implementation that is used by both Git and reftable\", but then we\n> might be in the same place as before? I dunno.\n\nI don't really have a good idea for how to do this. Using a common\nlibrary would likely push us into the same place as before indeed,\nunless we started to split out standalone files that aren't allowed to\nlink against anything but the \"compat/\" directory. But even if we had\nthat it would be quite painful to take the reftable library and reuse it\nsomewhere else, because now it's more than just \"cp -r reftable/\" and\nreimplementing system-specific bits in \"reftable/system.c\".\n\nI guess time will tell how much of a problem this really is. As said, my\nprediction is that we won't face many bugs in the low-level code, and\nthen the question becomes moot anyway. But if time proves me wrong I'll\nthink a bit more about potential solutions.\n\nPatrick\n"},{"id":"511317","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH v2 00/20] reftable: stop using \"git-compat-util.h\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:01Z","receivedAt":"2025-01-28T08:28:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series is the final step to fully decouple the reftable\nlibrary from the rest of the Git codebase. The goal of this is to make\nthe library reusable by other projects like libgit2 by simply copying\nover the source files, making Git the canonical upstream for reftable\nfunctionality.\n\nThis patch series stops using all kinds of helpers exposed by our\n\"git-compat-util.h\" header and open-codes them instead. In order to keep\nus from using these helpers by accident the final step is to pull out\nPOSIX-related bits and pieces into a new \"compat/posix.h\" header, which\nthe reftable library then uses instead of \"git-compat-util.h\".\n\nThe series is built on top of master at 5f8f7081f7 (The third batch,\n2025-01-23) with ps/reftable-sign-compare at 33319b0976 (reftable:\naddress trivial -Wsign-compare warnings, 2025-01-20) merged into it.\nThere is a trivial merge conflict with ps/zlib-ng that can be solved\nlike this:\n\n diff --cc reftable/system.h\n index e4a8944a70,d02eacea8f..0000000000\n --- a/reftable/system.h\n +++ b/reftable/system.h\n @@@ -11,15 -11,9 +11,15 @@@ https://developers.google.com/open-sour\n \n /* This header glues the reftable library to the rest of Git */\n \n -#include \"git-compat-util.h\"\n +#include \"compat/posix.h\"\n - #include \n + #include \"compat/zlib-compat.h\"\n \n +/*\n + * Return a random 32 bit integer. This function is expected to return\n + * pre-seeded data.\n + */\n +uint32_t reftable_rand(void);\n +\n /*\n * An implementation-specific temporary file. By making this specific to the\n * implementation it becomes possible to tie temporary files into any kind of\n\nChanges in v2:\n - The splitup of Windows headers has broken compilation because some\n of the headers couldn't be found anymore. I've fixed this more\n generally by converting includes in \"compat/\" to always be relative\n to the project source directory, dropping the platform-specific\n `-Icompat/` include.\n - Explain why we don't port over `EWOULDBLOCK` handling.\n - Fix commit message typos.\n - Link to v1: https://lore.kernel.org/r/20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (20):\n reftable/stack: stop using `read_in_full()`\n reftable/stack: stop using `write_in_full()`\n reftable/blocksource: stop using `xmmap()`\n reftable/record: stop using `COPY_ARRAY()`\n reftable/record: stop using `BUG()` in `reftable_record_init()`\n reftable/record: don't `BUG()` in `reftable_record_cmp()`\n reftable: stop using `BUG()` in trivial cases\n reftable/basics: stop using `st_mult()` in array allocators\n reftable/basics: provide wrappers for big endian conversion\n reftable/reader: stop using `ARRAY_SIZE()` macro\n reftable/system: introduce `reftable_rand()`\n reftable/stack: stop using `sleep_millisec()`\n reftable/basics: stop using `SWAP()` macro\n reftable/basics: stop using `UNUSED` annotation\n compat: consistently resolve headers via project root\n compat/mingw: split out POSIX-related bits\n compat/msvc: split out POSIX-related bits\n git-compat-util.h: split out POSIX-emulating bits\n reftable: decouple from Git codebase by pulling in \"compat/posix.h\"\n Makefile: skip reftable library for Coccinelle\n\n Makefile | 2 +-\n compat/access.c | 3 +-\n compat/basename.c | 4 +-\n compat/fileno.c | 2 +-\n compat/fopen.c | 2 +-\n compat/fsmonitor/fsm-health-darwin.c | 2 +-\n compat/fsmonitor/fsm-health-win32.c | 2 +-\n compat/fsmonitor/fsm-listen-darwin.c | 4 +-\n compat/fsmonitor/fsm-listen-win32.c | 2 +-\n compat/inet_ntop.c | 2 +-\n compat/inet_pton.c | 2 +-\n compat/memmem.c | 2 +-\n compat/{mingw.c => mingw/compat-util.c} | 24 +-\n compat/mingw/compat-util.h | 220 +++++++++++++\n compat/{mingw.h => mingw/posix.h} | 216 +-----------\n compat/mkdir.c | 2 +-\n compat/mkdtemp.c | 2 +-\n compat/mmap.c | 2 +-\n compat/msvc.c | 6 -\n compat/msvc/compat-util.c | 6 +\n compat/msvc/compat-util.h | 7 +\n compat/{msvc.h => msvc/posix.h} | 8 +-\n compat/nonblock.c | 4 +-\n compat/obstack.c | 2 +-\n compat/posix.h | 541 +++++++++++++++++++++++++++++++\n compat/pread.c | 4 +-\n compat/precompose_utf8.c | 2 +-\n compat/qsort_s.c | 2 +-\n compat/regcomp_enhanced.c | 2 +-\n compat/setenv.c | 2 +-\n compat/snprintf.c | 2 +-\n compat/strcasestr.c | 2 +-\n compat/strdup.c | 2 +-\n compat/strlcpy.c | 2 +-\n compat/strtoimax.c | 2 +-\n compat/strtoumax.c | 2 +-\n compat/unsetenv.c | 2 +-\n compat/win32/dirent.c | 2 +-\n compat/win32/flush.c | 2 +-\n compat/win32/path-utils.c | 4 +-\n compat/win32/pthread.c | 4 +-\n compat/win32/syslog.c | 2 +-\n compat/win32/trace2_win32_process_info.c | 10 +-\n compat/win32mmap.c | 2 +-\n compat/winansi.c | 6 +-\n config.mak.uname | 10 +-\n contrib/buildsystems/CMakeLists.txt | 2 +-\n git-compat-util.h | 535 +-----------------------------\n meson.build | 8 +-\n reftable/basics.c | 19 --\n reftable/basics.h | 123 ++++++-\n reftable/block.c | 16 +-\n reftable/blocksource.c | 21 +-\n reftable/iter.c | 20 +-\n reftable/merged.c | 27 +-\n reftable/pq.c | 40 ++-\n reftable/pq.h | 2 +-\n reftable/reader.c | 33 +-\n reftable/record.c | 96 ++++--\n reftable/record.h | 6 +-\n reftable/stack.c | 52 ++-\n reftable/system.c | 7 +\n reftable/system.h | 9 +-\n reftable/writer.c | 29 +-\n t/unit-tests/t-reftable-basics.c | 28 +-\n t/unit-tests/t-reftable-pq.c | 22 +-\n t/unit-tests/t-reftable-record.c | 42 ++-\n 67 files changed, 1282 insertions(+), 992 deletions(-)\n\nRange-diff versus v1:\n\n 1: 1dc7341c0d ! 1: a0985a8b32 reftable/stack: stop using `read_in_full()`\n @@ Commit message\n There is a single callsite of `read_in_full()` in the reftable library.\n Open-code the function to reduce our dependency on the Git library.\n \n + Note that we only partially port over the logic from `read_in_full()`\n + and its underlying `xread()` helper. Most importantly, the latter also\n + knows to handle `EWOULDBLOCK` via `handle_nonblock()`. This logic is\n + irrelevant for us though because the reftable library never sets the\n + `O_NONBLOCK` option in the first place.\n +\n Signed-off-by: Patrick Steinhardt \n \n ## reftable/stack.c ##\n 2: a3a8bc7e8f = 2: 09e27dcfaf reftable/stack: stop using `write_in_full()`\n 3: 43b83eb8e8 = 3: 0ecdb7a440 reftable/blocksource: stop using `xmmap()`\n 4: d89df1bac5 = 4: b5e5803ec9 reftable/record: stop using `COPY_ARRAY()`\n 5: 94fd212a82 = 5: 5f479bf84d reftable/record: stop using `BUG()` in `reftable_record_init()`\n 6: c110e452fe ! 6: 9b10ec8f35 reftable/record: don't `BUG()` in `reftable_record_cmp()`\n @@ Metadata\n ## Commit message ##\n reftable/record: don't `BUG()` in `reftable_record_cmp()`\n \n - The refatble library aborts with a bug in case `reftable_record_cmp()`\n + The reftable library aborts with a bug in case `reftable_record_cmp()`\n is invoked with two records of differing types. This would cause the\n program to die without the caller being able to handle the error, which\n is not something we want in the context of library code. And it ties us\n 7: 61165df24e = 7: 0fe9c21db3 reftable: stop using `BUG()` in trivial cases\n 8: a3e2d8f194 = 8: b1f8ed2e92 reftable/basics: stop using `st_mult()` in array allocators\n 9: 75c719d097 = 9: 40a5989b25 reftable/basics: provide wrappers for big endian conversion\n10: 754391b32e = 10: 6c258880f1 reftable/reader: stop using `ARRAY_SIZE()` macro\n11: 4e14362169 = 11: 65a78993fc reftable/system: introduce `reftable_rand()`\n12: 142610e800 = 12: e8a5090382 reftable/stack: stop using `sleep_millisec()`\n13: 2b748b36ee = 13: af72104f1c reftable/basics: stop using `SWAP()` macro\n14: fda87c2ec7 = 14: 4634e83e47 reftable/basics: stop using `UNUSED` annotation\n -: ---------- > 15: 3a4949d74d compat: consistently resolve headers via project root\n15: 34f81f0477 ! 16: 3520bf9252 compat/mingw: split out POSIX-related bits\n @@ compat/mingw/posix.h: char *mingw_query_user_email(void);\n ## compat/msvc.c ##\n @@\n #include \n - #include \"../strbuf.h\"\n + #include \"strbuf.h\"\n \n --#include \"mingw.c\"\n -+#include \"mingw/compat-util.c\"\n +-#include \"compat/mingw.c\"\n ++#include \"compat/mingw/compat-util.c\"\n \n ## compat/msvc.h ##\n @@ compat/msvc.h: typedef int sigset_t;\n @@ config.mak.uname: endif\n \tNO_INITGROUPS = YesPlease\n @@ config.mak.uname: ifeq ($(uname_S),MINGW)\n \tBASIC_LDFLAGS += -municode\n - \tCOMPAT_CFLAGS += -DNOGDI -Icompat -Icompat/win32\n + \tCOMPAT_CFLAGS += -DNOGDI -Icompat/win32\n \tCOMPAT_CFLAGS += -DSTRIP_EXTENSION=\\\".exe\\\"\n -\tCOMPAT_OBJS += compat/mingw.o compat/winansi.o \\\n +\tCOMPAT_OBJS += compat/mingw/compat-util.o compat/winansi.o \\\n16: ed9da5dbf4 = 17: fbb53e8526 compat/msvc: split out POSIX-related bits\n17: f19f95e6d8 ! 18: 6d57bc4b26 git-compat-util.h: split out POSIX-emulating bits\n @@ Commit message\n This intermixing is a bit of a problem for the reftable library as we\n don't want to recreate the POSIX-like interface there. But neither do we\n want to pull in the Git-specific functionality, as it is otherwise quite\n - easy to start depedning on the Git codebase again.\n + easy to start depending on the Git codebase again.\n \n Split out a new header \"compat/posix.h\" that only contains the bits and\n pieces relevant for the emulation of POSIX, which we will start using in\n18: 463c285baa = 19: 746698c8ef reftable: decouple from Git codebase by pulling in \"compat/posix.h\"\n19: 0973de24c0 = 20: 0f34ac6145 Makefile: skip reftable library for Coccinelle\n\n---\nbase-commit: 5f8f7081f7761acdf83d0a4c6819fe3d724f01d7\nchange-id: 20241119-pks-reftable-drop-git-compat-util-470f2bfde562\nprerequisite-message-id: <20250120-b4-pks-reftable-sign-compare-v2-0-b4566d02e4a5@pks.im>\nprerequisite-patch-id: fda53cfaa42008baf9e5e66f9805f1091d5e28f0\nprerequisite-patch-id: 5b9f06f4b50f51c0afb42fbf4a1a678069b38056\nprerequisite-patch-id: 864d6d587b43aae2dedbb6a327a1632eafae51c7\nprerequisite-patch-id: aec82a709396ae48b44c86016533517f2a9983cb\nprerequisite-patch-id: 89087dd9f69dabfaac9b14874a6e8d8c1a42eada\nprerequisite-patch-id: 87ec0aec4990da55346179776f9326725256642f\nprerequisite-patch-id: b19cbfbe4fdd75d9ae319ca0e222ea01902f513e\nprerequisite-patch-id: 1395a747e0005f729ef88a47f0d0dd4cd7c4dca2\nprerequisite-patch-id: d0e48535aa481bb1770753858cd6a093a1ad625b\nprerequisite-patch-id: d11a8b914e70cd56a5280e75fb7158f5de87eb22\n\n"},{"id":"511318","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-1-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 01/20] reftable/stack: stop using `read_in_full()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:02Z","receivedAt":"2025-01-28T08:28:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"There is a single callsite of `read_in_full()` in the reftable library.\nOpen-code the function to reduce our dependency on the Git library.\n\nNote that we only partially port over the logic from `read_in_full()`\nand its underlying `xread()` helper. Most importantly, the latter also\nknows to handle `EWOULDBLOCK` via `handle_nonblock()`. This logic is\nirrelevant for us though because the reftable library never sets the\n`O_NONBLOCK` option in the first place.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 18 ++++++++++++++----\n 1 file changed, 14 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex f7c1845e15..9490366795 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -115,13 +115,16 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \n static int fd_read_lines(int fd, char ***namesp)\n {\n-\toff_t size = lseek(fd, 0, SEEK_END);\n \tchar *buf = NULL;\n \tint err = 0;\n+\toff_t size;\n+\n+\tsize = lseek(fd, 0, SEEK_END);\n \tif (size < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n+\n \terr = lseek(fd, 0, SEEK_SET);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n@@ -134,9 +137,16 @@ static int fd_read_lines(int fd, char ***namesp)\n \t\tgoto done;\n \t}\n \n-\tif (read_in_full(fd, buf, size) != size) {\n-\t\terr = REFTABLE_IO_ERROR;\n-\t\tgoto done;\n+\tfor (size_t total_read = 0; total_read < (size_t) size; ) {\n+\t\tssize_t bytes_read = read(fd, buf + total_read, size - total_read);\n+\t\tif (bytes_read < 0 && (errno == EAGAIN || errno == EINTR))\n+\t\t\tcontinue;\n+\t\tif (bytes_read < 0 || !bytes_read) {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\ttotal_read += bytes_read;\n \t}\n \tbuf[size] = 0;\n \n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511319","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-2-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 02/20] reftable/stack: stop using `write_in_full()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:03Z","receivedAt":"2025-01-28T08:28:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Similar to the preceding commit, drop our use of `write_in_full()` and\nimplement a new wrapper `reftable_write_full()` that handles this logic\nfor us. This is done to reduce our dependency on the Git library.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 28 ++++++++++++++++++++++++----\n 1 file changed, 24 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 9490366795..5f155b344b 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -48,6 +48,25 @@ static int stack_fsync(const struct reftable_write_options *opts, int fd)\n \treturn fsync(fd);\n }\n \n+static ssize_t reftable_write_data(int fd, const void *data, size_t size)\n+{\n+\tsize_t total_written = 0;\n+\tconst char *p = data;\n+\n+\twhile (total_written < size) {\n+\t\tssize_t bytes_written = write(fd, p, size - total_written);\n+\t\tif (bytes_written < 0 && (errno == EAGAIN || errno == EINTR))\n+\t\t\tcontinue;\n+\t\tif (bytes_written < 0)\n+\t\t\treturn REFTABLE_IO_ERROR;\n+\n+\t\ttotal_written += bytes_written;\n+\t\tp += bytes_written;\n+\t}\n+\n+\treturn total_written;\n+}\n+\n struct fd_writer {\n \tconst struct reftable_write_options *opts;\n \tint fd;\n@@ -56,7 +75,7 @@ struct fd_writer {\n static ssize_t fd_writer_write(void *arg, const void *data, size_t sz)\n {\n \tstruct fd_writer *writer = arg;\n-\treturn write_in_full(writer->fd, data, sz);\n+\treturn reftable_write_data(writer->fd, data, sz);\n }\n \n static int fd_writer_flush(void *arg)\n@@ -784,7 +803,8 @@ int reftable_addition_commit(struct reftable_addition *add)\n \t\t\tgoto done;\n \t}\n \n-\terr = write_in_full(add->tables_list_lock.fd, table_list.buf, table_list.len);\n+\terr = reftable_write_data(add->tables_list_lock.fd,\n+\t\t\t\t table_list.buf, table_list.len);\n \treftable_buf_release(&table_list);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n@@ -1468,8 +1488,8 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\t\tgoto done;\n \t}\n \n-\terr = write_in_full(tables_list_lock.fd,\n-\t\t\t tables_list_buf.buf, tables_list_buf.len);\n+\terr = reftable_write_data(tables_list_lock.fd,\n+\t\t\t\t tables_list_buf.buf, tables_list_buf.len);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tunlink(new_table_path.buf);\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511320","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-4-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 04/20] reftable/record: stop using `COPY_ARRAY()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:05Z","receivedAt":"2025-01-28T08:28:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Drop our use of `COPY_ARRAY()`, replacing it with an open-coded variant\nthereof. This is done to reduce our dependency on the Git library.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/record.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 8919df8a4d..d1664c47ca 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -508,7 +508,8 @@ static int reftable_obj_record_copy_from(void *rec, const void *src_rec,\n \tif (!obj->offsets)\n \t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \tobj->offset_len = src->offset_len;\n-\tCOPY_ARRAY(obj->offsets, src->offsets, src->offset_len);\n+\tif (src->offset_len)\n+\t\tmemcpy(obj->offsets, src->offsets, sizeof(*src->offsets) * src->offset_len);\n \n \treturn 0;\n }\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511321","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-3-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 03/20] reftable/blocksource: stop using `xmmap()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:04Z","receivedAt":"2025-01-28T08:28:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We use `xmmap()` to map reftables into memory. This function has two\nproblems:\n\n - It causes us to die in case the mmap fails.\n\n - It ties us to the Git codebase.\n\nRefactor the code to use mmap(3p) instead with manual error checking.\nNote that this function may not be the system-provided mmap(3p), but may\npoint to our `git_mmap()` wrapper that emulates the syscall on systems\nthat do not have mmap(3p) available.\n\nFix `reftable_block_source_from_file()` to properly bubble up the error\ncode in case the map(3p) call fails.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/blocksource.c | 11 ++++++++---\n 1 file changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex bba4a45b98..02972c46f4 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -98,7 +98,7 @@ static struct reftable_block_source_vtable file_vtable = {\n int reftable_block_source_from_file(struct reftable_block_source *bs,\n \t\t\t\t const char *name)\n {\n-\tstruct file_block_source *p;\n+\tstruct file_block_source *p = NULL;\n \tstruct stat st;\n \tint fd, err;\n \n@@ -122,7 +122,12 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,\n \t}\n \n \tp->size = st.st_size;\n-\tp->data = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tp->data = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tif (p->data == MAP_FAILED) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tp->data = NULL;\n+\t\tgoto out;\n+\t}\n \n \tassert(!bs->ops);\n \tbs->ops = &file_vtable;\n@@ -135,5 +140,5 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,\n \t\tclose(fd);\n \tif (err < 0)\n \t\treftable_free(p);\n-\treturn 0;\n+\treturn err;\n }\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511322","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-5-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 05/20] reftable/record: stop using `BUG()` in `reftable_record_init()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:06Z","receivedAt":"2025-01-28T08:28:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're aborting the program via `BUG()` in case `reftable_record_init()`\nwas invoked with an unknown record type. This is bad because we may now\ndie in library code, and because it makes us depend on the Git codebase.\n\nRefactor the code such that `reftable_record_init()` can return an error\ncode to the caller. Adapt any callers accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/block.c | 4 +++-\n reftable/merged.c | 5 ++++-\n reftable/reader.c | 5 ++++-\n reftable/record.c | 8 ++++----\n reftable/record.h | 4 ++--\n t/unit-tests/t-reftable-pq.c | 6 +++---\n t/unit-tests/t-reftable-record.c | 2 +-\n 7 files changed, 21 insertions(+), 13 deletions(-)\n\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 8ac865ce78..255d566854 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -509,7 +509,9 @@ int block_iter_seek_key(struct block_iter *it, const struct block_reader *br,\n \tit->block_len = br->block_len;\n \tit->hash_size = br->hash_size;\n \n-\treftable_record_init(&rec, block_reader_type(br));\n+\terr = reftable_record_init(&rec, block_reader_type(br));\n+\tif (err < 0)\n+\t\tgoto done;\n \n \t/*\n \t * We're looking for the last entry less than the wanted key so that\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex e72b39e178..4156eec07f 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -253,7 +253,10 @@ int merged_table_init_iter(struct reftable_merged_table *mt,\n \t}\n \n \tfor (size_t i = 0; i < mt->readers_len; i++) {\n-\t\treftable_record_init(&subiters[i].rec, typ);\n+\t\tret = reftable_record_init(&subiters[i].rec, typ);\n+\t\tif (ret < 0)\n+\t\t\tgoto out;\n+\n \t\tret = reader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n \t\tif (ret < 0)\n \t\t\tgoto out;\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 3f2e4b2800..de6e6dd932 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -360,7 +360,10 @@ static int table_iter_seek_linear(struct table_iter *ti,\n \tstruct reftable_record rec;\n \tint err;\n \n-\treftable_record_init(&rec, reftable_record_type(want));\n+\terr = reftable_record_init(&rec, reftable_record_type(want));\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = reftable_record_key(want, &want_key);\n \tif (err < 0)\n \t\tgoto done;\ndiff --git a/reftable/record.c b/reftable/record.c\nindex d1664c47ca..31985bb977 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -1301,7 +1301,7 @@ reftable_record_vtable(struct reftable_record *rec)\n \tabort();\n }\n \n-void reftable_record_init(struct reftable_record *rec, uint8_t typ)\n+int reftable_record_init(struct reftable_record *rec, uint8_t typ)\n {\n \tmemset(rec, 0, sizeof(*rec));\n \trec->type = typ;\n@@ -1310,11 +1310,11 @@ void reftable_record_init(struct reftable_record *rec, uint8_t typ)\n \tcase BLOCK_TYPE_REF:\n \tcase BLOCK_TYPE_LOG:\n \tcase BLOCK_TYPE_OBJ:\n-\t\treturn;\n+\t\treturn 0;\n \tcase BLOCK_TYPE_INDEX:\n \t\treftable_buf_init(&rec->u.idx.last_key);\n-\t\treturn;\n+\t\treturn 0;\n \tdefault:\n-\t\tBUG(\"unhandled record type\");\n+\t\treturn REFTABLE_API_ERROR;\n \t}\n }\ndiff --git a/reftable/record.h b/reftable/record.h\nindex c7755a4d75..e1846c294b 100644\n--- a/reftable/record.h\n+++ b/reftable/record.h\n@@ -130,8 +130,8 @@ struct reftable_record {\n \t} u;\n };\n \n-/* Initialize the reftable record for the given type */\n-void reftable_record_init(struct reftable_record *rec, uint8_t typ);\n+/* Initialize the reftable record for the given type. */\n+int reftable_record_init(struct reftable_record *rec, uint8_t typ);\n \n /* see struct record_vtable */\n int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b);\ndiff --git a/t/unit-tests/t-reftable-pq.c b/t/unit-tests/t-reftable-pq.c\nindex f3f8a0cdf3..d8a4c283a1 100644\n--- a/t/unit-tests/t-reftable-pq.c\n+++ b/t/unit-tests/t-reftable-pq.c\n@@ -32,7 +32,7 @@ static void t_pq_record(void)\n \tchar *last = NULL;\n \n \tfor (i = 0; i < N; i++) {\n-\t\treftable_record_init(&recs[i], BLOCK_TYPE_REF);\n+\t\tcheck(!reftable_record_init(&recs[i], BLOCK_TYPE_REF));\n \t\trecs[i].u.ref.refname = xstrfmt(\"%02\"PRIuMAX, (uintmax_t)i);\n \t}\n \n@@ -72,7 +72,7 @@ static void t_pq_index(void)\n \tsize_t N = ARRAY_SIZE(recs), i;\n \n \tfor (i = 0; i < N; i++) {\n-\t\treftable_record_init(&recs[i], BLOCK_TYPE_REF);\n+\t\tcheck(!reftable_record_init(&recs[i], BLOCK_TYPE_REF));\n \t\trecs[i].u.ref.refname = (char *) \"refs/heads/master\";\n \t}\n \n@@ -111,7 +111,7 @@ static void t_merged_iter_pqueue_top(void)\n \tsize_t N = ARRAY_SIZE(recs), i;\n \n \tfor (i = 0; i < N; i++) {\n-\t\treftable_record_init(&recs[i], BLOCK_TYPE_REF);\n+\t\tcheck(!reftable_record_init(&recs[i], BLOCK_TYPE_REF));\n \t\trecs[i].u.ref.refname = (char *) \"refs/heads/master\";\n \t}\n \ndiff --git a/t/unit-tests/t-reftable-record.c b/t/unit-tests/t-reftable-record.c\nindex d49d2a2729..6540bd20e3 100644\n--- a/t/unit-tests/t-reftable-record.c\n+++ b/t/unit-tests/t-reftable-record.c\n@@ -17,7 +17,7 @@ static void t_copy(struct reftable_record *rec)\n \tuint8_t typ;\n \n \ttyp = reftable_record_type(rec);\n-\treftable_record_init(©, typ);\n+\tcheck(!reftable_record_init(©, typ));\n \treftable_record_copy_from(©, rec, REFTABLE_HASH_SIZE_SHA1);\n \t/* do it twice to catch memory leaks */\n \treftable_record_copy_from(©, rec, REFTABLE_HASH_SIZE_SHA1);\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511323","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-6-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 06/20] reftable/record: don't `BUG()` in `reftable_record_cmp()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:07Z","receivedAt":"2025-01-28T08:28:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library aborts with a bug in case `reftable_record_cmp()`\nis invoked with two records of differing types. This would cause the\nprogram to die without the caller being able to handle the error, which\nis not something we want in the context of library code. And it ties us\nto the Git codebase.\n\nRefactor the code such that `reftable_record_cmp()` returns an error\ncode separate from the actual comparison result. This requires us to\nalso adapt some callers up the callchain in a similar fashion.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 20 +++++++++++++++-----\n reftable/pq.c | 36 +++++++++++++++++++++++++++++-------\n reftable/pq.h | 2 +-\n reftable/record.c | 10 ++++++----\n reftable/record.h | 2 +-\n t/unit-tests/t-reftable-pq.c | 16 ++++++++++++----\n t/unit-tests/t-reftable-record.c | 40 ++++++++++++++++++++++++++++------------\n 7 files changed, 92 insertions(+), 34 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 4156eec07f..563864068c 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -66,8 +66,11 @@ static int merged_iter_seek(struct merged_iter *mi, struct reftable_record *want\n \tint err;\n \n \tmi->advance_index = -1;\n-\twhile (!merged_iter_pqueue_is_empty(mi->pq))\n-\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq)) {\n+\t\terr = merged_iter_pqueue_remove(&mi->pq, NULL);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n \n \tfor (size_t i = 0; i < mi->subiters_len; i++) {\n \t\terr = iterator_seek(&mi->subiters[i].iter, want);\n@@ -120,7 +123,9 @@ static int merged_iter_next_entry(struct merged_iter *mi,\n \tif (empty)\n \t\treturn 1;\n \n-\tentry = merged_iter_pqueue_remove(&mi->pq);\n+\terr = merged_iter_pqueue_remove(&mi->pq, &entry);\n+\tif (err < 0)\n+\t\treturn err;\n \n \t/*\n \t One can also use reftable as datacenter-local storage, where the ref\n@@ -134,11 +139,16 @@ static int merged_iter_next_entry(struct merged_iter *mi,\n \t\tstruct pq_entry top = merged_iter_pqueue_top(mi->pq);\n \t\tint cmp;\n \n-\t\tcmp = reftable_record_cmp(top.rec, entry.rec);\n+\t\terr = reftable_record_cmp(top.rec, entry.rec, &cmp);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n \t\tif (cmp > 0)\n \t\t\tbreak;\n \n-\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\t\terr = merged_iter_pqueue_remove(&mi->pq, NULL);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\n \t\terr = merged_iter_advance_subiter(mi, top.index);\n \t\tif (err < 0)\n \t\t\treturn err;\ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex 5591e875e1..ef8035cfd9 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -15,13 +15,18 @@ license that can be found in the LICENSE file or at\n \n int pq_less(struct pq_entry *a, struct pq_entry *b)\n {\n-\tint cmp = reftable_record_cmp(a->rec, b->rec);\n+\tint cmp, err;\n+\n+\terr = reftable_record_cmp(a->rec, b->rec, &cmp);\n+\tif (err < 0)\n+\t\treturn err;\n+\n \tif (cmp == 0)\n \t\treturn a->index > b->index;\n \treturn cmp < 0;\n }\n \n-struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n+int merged_iter_pqueue_remove(struct merged_iter_pqueue *pq, struct pq_entry *out)\n {\n \tsize_t i = 0;\n \tstruct pq_entry e = pq->heap[0];\n@@ -32,17 +37,34 @@ struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n \t\tsize_t min = i;\n \t\tsize_t j = 2 * i + 1;\n \t\tsize_t k = 2 * i + 2;\n-\t\tif (j < pq->len && pq_less(&pq->heap[j], &pq->heap[i]))\n-\t\t\tmin = j;\n-\t\tif (k < pq->len && pq_less(&pq->heap[k], &pq->heap[min]))\n-\t\t\tmin = k;\n+\t\tint cmp;\n+\n+\t\tif (j < pq->len) {\n+\t\t\tcmp = pq_less(&pq->heap[j], &pq->heap[i]);\n+\t\t\tif (cmp < 0)\n+\t\t\t\treturn -1;\n+\t\t\telse if (cmp)\n+\t\t\t\tmin = j;\n+\t\t}\n+\n+\t\tif (k < pq->len) {\n+\t\t\tcmp = pq_less(&pq->heap[k], &pq->heap[min]);\n+\t\t\tif (cmp < 0)\n+\t\t\t\treturn -1;\n+\t\t\telse if (cmp)\n+\t\t\t\tmin = k;\n+\t\t}\n+\n \t\tif (min == i)\n \t\t\tbreak;\n \t\tSWAP(pq->heap[i], pq->heap[min]);\n \t\ti = min;\n \t}\n \n-\treturn e;\n+\tif (out)\n+\t\t*out = e;\n+\n+\treturn 0;\n }\n \n int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e)\ndiff --git a/reftable/pq.h b/reftable/pq.h\nindex 83c062eeca..ff39016445 100644\n--- a/reftable/pq.h\n+++ b/reftable/pq.h\n@@ -22,7 +22,7 @@ struct merged_iter_pqueue {\n \tsize_t cap;\n };\n \n-struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n+int merged_iter_pqueue_remove(struct merged_iter_pqueue *pq, struct pq_entry *out);\n int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e);\n void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n int pq_less(struct pq_entry *a, struct pq_entry *b);\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 31985bb977..da550494a2 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -1190,12 +1190,14 @@ int reftable_record_is_deletion(struct reftable_record *rec)\n \t\treftable_record_data(rec));\n }\n \n-int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b)\n+int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b,\n+\t\t\tint *cmp)\n {\n \tif (a->type != b->type)\n-\t\tBUG(\"cannot compare reftable records of different type\");\n-\treturn reftable_record_vtable(a)->cmp(\n-\t\treftable_record_data(a), reftable_record_data(b));\n+\t\treturn -1;\n+\t*cmp = reftable_record_vtable(a)->cmp(reftable_record_data(a),\n+\t\t\t\t\t reftable_record_data(b));\n+\treturn 0;\n }\n \n int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, uint32_t hash_size)\ndiff --git a/reftable/record.h b/reftable/record.h\nindex e1846c294b..867810a932 100644\n--- a/reftable/record.h\n+++ b/reftable/record.h\n@@ -134,7 +134,7 @@ struct reftable_record {\n int reftable_record_init(struct reftable_record *rec, uint8_t typ);\n \n /* see struct record_vtable */\n-int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b);\n+int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b, int *cmp);\n int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, uint32_t hash_size);\n int reftable_record_key(struct reftable_record *rec, struct reftable_buf *dest);\n int reftable_record_copy_from(struct reftable_record *rec,\ndiff --git a/t/unit-tests/t-reftable-pq.c b/t/unit-tests/t-reftable-pq.c\nindex d8a4c283a1..c128fe8616 100644\n--- a/t/unit-tests/t-reftable-pq.c\n+++ b/t/unit-tests/t-reftable-pq.c\n@@ -21,7 +21,9 @@ static void merged_iter_pqueue_check(const struct merged_iter_pqueue *pq)\n \n static int pq_entry_equal(struct pq_entry *a, struct pq_entry *b)\n {\n-\treturn !reftable_record_cmp(a->rec, b->rec) && (a->index == b->index);\n+\tint cmp;\n+\tcheck(!reftable_record_cmp(a->rec, b->rec, &cmp));\n+\treturn !cmp && (a->index == b->index);\n }\n \n static void t_pq_record(void)\n@@ -49,7 +51,9 @@ static void t_pq_record(void)\n \n \twhile (!merged_iter_pqueue_is_empty(pq)) {\n \t\tstruct pq_entry top = merged_iter_pqueue_top(pq);\n-\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct pq_entry e;\n+\n+\t\tcheck(!merged_iter_pqueue_remove(&pq, &e));\n \t\tmerged_iter_pqueue_check(&pq);\n \n \t\tcheck(pq_entry_equal(&top, &e));\n@@ -90,7 +94,9 @@ static void t_pq_index(void)\n \n \tfor (i = N - 1; i > 0; i--) {\n \t\tstruct pq_entry top = merged_iter_pqueue_top(pq);\n-\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct pq_entry e;\n+\n+\t\tcheck(!merged_iter_pqueue_remove(&pq, &e));\n \t\tmerged_iter_pqueue_check(&pq);\n \n \t\tcheck(pq_entry_equal(&top, &e));\n@@ -129,7 +135,9 @@ static void t_merged_iter_pqueue_top(void)\n \n \tfor (i = N - 1; i > 0; i--) {\n \t\tstruct pq_entry top = merged_iter_pqueue_top(pq);\n-\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct pq_entry e;\n+\n+\t\tcheck(!merged_iter_pqueue_remove(&pq, &e));\n \n \t\tmerged_iter_pqueue_check(&pq);\n \t\tcheck(pq_entry_equal(&top, &e));\ndiff --git a/t/unit-tests/t-reftable-record.c b/t/unit-tests/t-reftable-record.c\nindex 6540bd20e3..5954966373 100644\n--- a/t/unit-tests/t-reftable-record.c\n+++ b/t/unit-tests/t-reftable-record.c\n@@ -100,16 +100,20 @@ static void t_reftable_ref_record_comparison(void)\n \t\t\t.u.ref.value.symref = (char *) \"refs/heads/master\",\n \t\t},\n \t};\n+\tint cmp;\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \n \tin[1].u.ref.value_type = in[0].u.ref.value_type;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n }\n \n static void t_reftable_ref_record_compare_name(void)\n@@ -209,17 +213,20 @@ static void t_reftable_log_record_comparison(void)\n \t\t\t.u.log.update_index = 22,\n \t\t},\n \t};\n+\tint cmp;\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \t/* comparison should be reversed for equal keys, because\n \t * comparison is now performed on the basis of update indices */\n-\tcheck_int(reftable_record_cmp(&in[0], &in[1]), <, 0);\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck_int(cmp, <, 0);\n \n \tin[1].u.log.update_index = in[0].u.log.update_index;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n }\n \n static void t_reftable_log_record_compare_key(void)\n@@ -396,16 +403,20 @@ static void t_reftable_obj_record_comparison(void)\n \t\t\t.u.obj.hash_prefix_len = 5,\n \t\t},\n \t};\n+\tint cmp;\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \n \tin[1].u.obj.offset_len = in[0].u.obj.offset_len;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n }\n \n static void t_reftable_obj_record_roundtrip(void)\n@@ -486,19 +497,24 @@ static void t_reftable_index_record_comparison(void)\n \t\t\t.u.idx.last_key = REFTABLE_BUF_INIT,\n \t\t},\n \t};\n+\tint cmp;\n+\n \tcheck(!reftable_buf_addstr(&in[0].u.idx.last_key, \"refs/heads/master\"));\n \tcheck(!reftable_buf_addstr(&in[1].u.idx.last_key, \"refs/heads/master\"));\n \tcheck(!reftable_buf_addstr(&in[2].u.idx.last_key, \"refs/heads/branch\"));\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \n \tin[1].u.idx.offset = in[0].u.idx.offset;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tfor (size_t i = 0; i < ARRAY_SIZE(in); i++)\n \t\treftable_record_release(&in[i]);\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511324","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-7-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 07/20] reftable: stop using `BUG()` in trivial cases","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:08Z","receivedAt":"2025-01-28T08:28:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Stop using `BUG()` in the remaining trivial cases that we still have in\nthe reftable library. Instead of aborting the program, we'll now bubble\nup a `REFTABLE_API_ERROR` to indicate misuse of the calling conventions.\n\nNote that in both `reftable_reader_{inc,dec}ref()` we simply stop\ncalling `BUG()` altogether. The only situation where the counter should\nbe zero is when the structure has already been free'd anyway, so we\nwould run into undefined behaviour regardless of whether we try to abort\nthe program or not.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/iter.c | 3 +--\n reftable/reader.c | 4 ----\n reftable/writer.c | 5 ++---\n 3 files changed, 3 insertions(+), 9 deletions(-)\n\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex 86e801ca9f..b2ffb09c16 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -146,8 +146,7 @@ static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n static int indexed_table_ref_iter_seek(void *p UNUSED,\n \t\t\t\t struct reftable_record *want UNUSED)\n {\n-\tBUG(\"seeking indexed table is not supported\");\n-\treturn -1;\n+\treturn REFTABLE_API_ERROR;\n }\n \n static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex de6e6dd932..36a5633ede 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -677,8 +677,6 @@ int reftable_reader_new(struct reftable_reader **out,\n \n void reftable_reader_incref(struct reftable_reader *r)\n {\n-\tif (!r->refcount)\n-\t\tBUG(\"cannot increment ref counter of dead reader\");\n \tr->refcount++;\n }\n \n@@ -686,8 +684,6 @@ void reftable_reader_decref(struct reftable_reader *r)\n {\n \tif (!r)\n \t\treturn;\n-\tif (!r->refcount)\n-\t\tBUG(\"cannot decrement ref counter of dead reader\");\n \tif (--r->refcount)\n \t\treturn;\n \tblock_source_close(&r->source);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 91d6629486..155863ee5f 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -158,7 +158,7 @@ int reftable_writer_new(struct reftable_writer **out,\n \t\topts = *_opts;\n \toptions_set_defaults(&opts);\n \tif (opts.block_size >= (1 << 24))\n-\t\tBUG(\"configured block size exceeds 16MB\");\n+\t\treturn REFTABLE_API_ERROR;\n \n \treftable_buf_init(&wp->block_writer_data.last_key);\n \treftable_buf_init(&wp->last_key);\n@@ -289,8 +289,7 @@ static int writer_add_record(struct reftable_writer *w,\n \t}\n \n \tif (block_writer_type(w->block_writer) != reftable_record_type(rec))\n-\t\tBUG(\"record of type %d added to writer of type %d\",\n-\t\t reftable_record_type(rec), block_writer_type(w->block_writer));\n+\t\treturn REFTABLE_API_ERROR;\n \n \t/*\n \t * Try to add the record to the writer. If this succeeds then we're\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511325","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-8-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 08/20] reftable/basics: stop using `st_mult()` in array allocators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:09Z","receivedAt":"2025-01-28T08:28:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're using `st_mult()` as part of our macro helpers that allocate\narrays. This is bad due two two reasons:\n\n - `st_mult()` causes us to die in case the multiplication overflows.\n\n - `st_mult()` ties us to the Git codebase.\n\nRefactor the code to instead detect overflows manually and return an\nerror in such cases.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 36 ++++++++++++++++++++++++++++++++----\n 1 file changed, 32 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex a2a010a0e1..646f8d67f2 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -117,18 +117,46 @@ void reftable_free(void *p);\n void *reftable_calloc(size_t nelem, size_t elsize);\n char *reftable_strdup(const char *str);\n \n-#define REFTABLE_ALLOC_ARRAY(x, alloc) (x) = reftable_malloc(st_mult(sizeof(*(x)), (alloc)))\n+static inline int reftable_alloc_size(size_t nelem, size_t elsize, size_t *out)\n+{\n+\tif (nelem && elsize > SIZE_MAX / nelem)\n+\t\treturn -1;\n+\t*out = nelem * elsize;\n+\treturn 0;\n+}\n+\n+#define REFTABLE_ALLOC_ARRAY(x, alloc) do { \\\n+\t\tsize_t alloc_size; \\\n+\t\tif (reftable_alloc_size(sizeof(*(x)), (alloc), &alloc_size) < 0) { \\\n+\t\t\terrno = ENOMEM; \\\n+\t\t\t(x) = NULL; \\\n+\t\t} else { \\\n+\t\t\t(x) = reftable_malloc(alloc_size); \\\n+\t\t} \\\n+\t} while (0)\n #define REFTABLE_CALLOC_ARRAY(x, alloc) (x) = reftable_calloc((alloc), sizeof(*(x)))\n-#define REFTABLE_REALLOC_ARRAY(x, alloc) (x) = reftable_realloc((x), st_mult(sizeof(*(x)), (alloc)))\n+#define REFTABLE_REALLOC_ARRAY(x, alloc) do { \\\n+\t\tsize_t alloc_size; \\\n+\t\tif (reftable_alloc_size(sizeof(*(x)), (alloc), &alloc_size) < 0) { \\\n+\t\t\terrno = ENOMEM; \\\n+\t\t\t(x) = NULL; \\\n+\t\t} else { \\\n+\t\t\t(x) = reftable_realloc((x), alloc_size); \\\n+\t\t} \\\n+\t} while (0)\n \n static inline void *reftable_alloc_grow(void *p, size_t nelem, size_t elsize,\n \t\t\t\t\tsize_t *allocp)\n {\n \tvoid *new_p;\n-\tsize_t alloc = *allocp * 2 + 1;\n+\tsize_t alloc = *allocp * 2 + 1, alloc_bytes;\n \tif (alloc < nelem)\n \t\talloc = nelem;\n-\tnew_p = reftable_realloc(p, st_mult(elsize, alloc));\n+\tif (reftable_alloc_size(elsize, alloc, &alloc_bytes) < 0) {\n+\t\terrno = ENOMEM;\n+\t\treturn p;\n+\t}\n+\tnew_p = reftable_realloc(p, alloc_bytes);\n \tif (!new_p)\n \t\treturn p;\n \t*allocp = alloc;\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511326","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-9-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 09/20] reftable/basics: provide wrappers for big endian conversion","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:10Z","receivedAt":"2025-01-28T08:28:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're using a mixture of big endian conversion functions provided by\nboth the reftable library, but also by the Git codebase. Refactor the\ncode so that we exclusively use reftable-provided wrappers in order to\nuntangle us from the Git codebase.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 19 ----------\n reftable/basics.h | 76 ++++++++++++++++++++++++++++++++++++++--\n reftable/block.c | 12 +++----\n reftable/reader.c | 22 ++++++------\n reftable/record.c | 8 ++---\n reftable/writer.c | 20 +++++------\n t/unit-tests/t-reftable-basics.c | 28 ++++++++++++---\n 7 files changed, 127 insertions(+), 58 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 3b5ea27bbd..8c4a4433e4 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -147,25 +147,6 @@ char *reftable_buf_detach(struct reftable_buf *buf)\n \treturn result;\n }\n \n-void put_be24(uint8_t *out, uint32_t i)\n-{\n-\tout[0] = (uint8_t)((i >> 16) & 0xff);\n-\tout[1] = (uint8_t)((i >> 8) & 0xff);\n-\tout[2] = (uint8_t)(i & 0xff);\n-}\n-\n-uint32_t get_be24(uint8_t *in)\n-{\n-\treturn (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 |\n-\t (uint32_t)(in[2]);\n-}\n-\n-void put_be16(uint8_t *out, uint16_t i)\n-{\n-\tout[0] = (uint8_t)((i >> 8) & 0xff);\n-\tout[1] = (uint8_t)(i & 0xff);\n-}\n-\n size_t binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n {\n \tsize_t lo = 0;\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 646f8d67f2..c1ddbaec3f 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -76,9 +76,79 @@ char *reftable_buf_detach(struct reftable_buf *buf);\n \n /* Bigendian en/decoding of integers */\n \n-void put_be24(uint8_t *out, uint32_t i);\n-uint32_t get_be24(uint8_t *in);\n-void put_be16(uint8_t *out, uint16_t i);\n+static inline void reftable_put_be16(void *out, uint16_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 8) & 0xff);\n+\tp[1] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline void reftable_put_be24(void *out, uint32_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 16) & 0xff);\n+\tp[1] = (uint8_t)((i >> 8) & 0xff);\n+\tp[2] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline void reftable_put_be32(void *out, uint32_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 24) & 0xff);\n+\tp[1] = (uint8_t)((i >> 16) & 0xff);\n+\tp[2] = (uint8_t)((i >> 8) & 0xff);\n+\tp[3] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline void reftable_put_be64(void *out, uint64_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 56) & 0xff);\n+\tp[1] = (uint8_t)((i >> 48) & 0xff);\n+\tp[2] = (uint8_t)((i >> 40) & 0xff);\n+\tp[3] = (uint8_t)((i >> 32) & 0xff);\n+\tp[4] = (uint8_t)((i >> 24) & 0xff);\n+\tp[5] = (uint8_t)((i >> 16) & 0xff);\n+\tp[6] = (uint8_t)((i >> 8) & 0xff);\n+\tp[7] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline uint16_t reftable_get_be16(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint16_t)(p[0]) << 8 |\n+\t (uint16_t)(p[1]) << 0;\n+}\n+\n+static inline uint32_t reftable_get_be24(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint32_t)(p[0]) << 16 |\n+\t (uint32_t)(p[1]) << 8 |\n+\t (uint32_t)(p[2]) << 0;\n+}\n+\n+static inline uint32_t reftable_get_be32(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint32_t)(p[0]) << 24 |\n+\t (uint32_t)(p[1]) << 16 |\n+\t (uint32_t)(p[2]) << 8|\n+\t (uint32_t)(p[3]) << 0;\n+}\n+\n+static inline uint64_t reftable_get_be64(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint64_t)(p[0]) << 56 |\n+\t (uint64_t)(p[1]) << 48 |\n+\t (uint64_t)(p[2]) << 40 |\n+\t (uint64_t)(p[3]) << 32 |\n+\t (uint64_t)(p[4]) << 24 |\n+\t (uint64_t)(p[5]) << 16 |\n+\t (uint64_t)(p[6]) << 8 |\n+\t (uint64_t)(p[7]) << 0;\n+}\n \n /*\n * find smallest index i in [0, sz) at which `f(i) > 0`, assuming that f is\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 255d566854..373908807e 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -148,13 +148,13 @@ int block_writer_add(struct block_writer *w, struct reftable_record *rec)\n int block_writer_finish(struct block_writer *w)\n {\n \tfor (uint32_t i = 0; i < w->restart_len; i++) {\n-\t\tput_be24(w->block + w->next, w->restarts[i]);\n+\t\treftable_put_be24(w->block + w->next, w->restarts[i]);\n \t\tw->next += 3;\n \t}\n \n-\tput_be16(w->block + w->next, w->restart_len);\n+\treftable_put_be16(w->block + w->next, w->restart_len);\n \tw->next += 2;\n-\tput_be24(w->block + 1 + w->header_off, w->next);\n+\treftable_put_be24(w->block + 1 + w->header_off, w->next);\n \n \t/*\n \t * Log records are stored zlib-compressed. Note that the compression\n@@ -216,7 +216,7 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,\n {\n \tuint32_t full_block_size = table_block_size;\n \tuint8_t typ = block->data[header_off];\n-\tuint32_t sz = get_be24(block->data + header_off + 1);\n+\tuint32_t sz = reftable_get_be24(block->data + header_off + 1);\n \tint err = 0;\n \tuint16_t restart_count = 0;\n \tuint32_t restart_start = 0;\n@@ -300,7 +300,7 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,\n \t\tfull_block_size = sz;\n \t}\n \n-\trestart_count = get_be16(block->data + sz - 2);\n+\trestart_count = reftable_get_be16(block->data + sz - 2);\n \trestart_start = sz - 2 - 3 * restart_count;\n \trestart_bytes = block->data + restart_start;\n \n@@ -355,7 +355,7 @@ int block_reader_first_key(const struct block_reader *br, struct reftable_buf *k\n \n static uint32_t block_reader_restart_offset(const struct block_reader *br, size_t idx)\n {\n-\treturn get_be24(br->restart_bytes + 3 * idx);\n+\treturn reftable_get_be24(br->restart_bytes + 3 * idx);\n }\n \n void block_iter_seek_start(struct block_iter *it, const struct block_reader *br)\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 36a5633ede..bf07a0a586 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -101,18 +101,18 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n \t}\n \n \tf++;\n-\tr->block_size = get_be24(f);\n+\tr->block_size = reftable_get_be24(f);\n \n \tf += 3;\n-\tr->min_update_index = get_be64(f);\n+\tr->min_update_index = reftable_get_be64(f);\n \tf += 8;\n-\tr->max_update_index = get_be64(f);\n+\tr->max_update_index = reftable_get_be64(f);\n \tf += 8;\n \n \tif (r->version == 1) {\n \t\tr->hash_id = REFTABLE_HASH_SHA1;\n \t} else {\n-\t\tswitch (get_be32(f)) {\n+\t\tswitch (reftable_get_be32(f)) {\n \t\tcase REFTABLE_FORMAT_ID_SHA1:\n \t\t\tr->hash_id = REFTABLE_HASH_SHA1;\n \t\t\tbreak;\n@@ -127,24 +127,24 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n \t\tf += 4;\n \t}\n \n-\tr->ref_offsets.index_offset = get_be64(f);\n+\tr->ref_offsets.index_offset = reftable_get_be64(f);\n \tf += 8;\n \n-\tr->obj_offsets.offset = get_be64(f);\n+\tr->obj_offsets.offset = reftable_get_be64(f);\n \tf += 8;\n \n \tr->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1);\n \tr->obj_offsets.offset >>= 5;\n \n-\tr->obj_offsets.index_offset = get_be64(f);\n+\tr->obj_offsets.index_offset = reftable_get_be64(f);\n \tf += 8;\n-\tr->log_offsets.offset = get_be64(f);\n+\tr->log_offsets.offset = reftable_get_be64(f);\n \tf += 8;\n-\tr->log_offsets.index_offset = get_be64(f);\n+\tr->log_offsets.index_offset = reftable_get_be64(f);\n \tf += 8;\n \n \tcomputed_crc = crc32(0, footer, f - footer);\n-\tfile_crc = get_be32(f);\n+\tfile_crc = reftable_get_be32(f);\n \tf += 4;\n \tif (computed_crc != file_crc) {\n \t\terr = REFTABLE_FORMAT_ERROR;\n@@ -214,7 +214,7 @@ static int32_t extract_block_size(uint8_t *data, uint8_t *typ, uint64_t off,\n \n \t*typ = data[0];\n \tif (reftable_is_block_type(*typ)) {\n-\t\tresult = get_be24(data + 1);\n+\t\tresult = reftable_get_be24(data + 1);\n \t}\n \treturn result;\n }\ndiff --git a/reftable/record.c b/reftable/record.c\nindex da550494a2..286659c098 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -684,7 +684,7 @@ static int reftable_log_record_key(const void *r, struct reftable_buf *dest)\n \t\treturn err;\n \n \tts = (~ts) - rec->update_index;\n-\tput_be64(&i64[0], ts);\n+\treftable_put_be64(&i64[0], ts);\n \n \terr = reftable_buf_add(dest, i64, sizeof(i64));\n \tif (err < 0)\n@@ -809,7 +809,7 @@ static int reftable_log_record_encode(const void *rec, struct string_view s,\n \tif (s.len < 2)\n \t\treturn -1;\n \n-\tput_be16(s.buf, r->value.update.tz_offset);\n+\treftable_put_be16(s.buf, r->value.update.tz_offset);\n \tstring_view_consume(&s, 2);\n \n \tn = encode_string(\n@@ -841,7 +841,7 @@ static int reftable_log_record_decode(void *rec, struct reftable_buf key,\n \t}\n \n \tmemcpy(r->refname, key.buf, key.len - 8);\n-\tts = get_be64(key.buf + key.len - 8);\n+\tts = reftable_get_be64((unsigned char *)key.buf + key.len - 8);\n \n \tr->update_index = (~max) - ts;\n \n@@ -932,7 +932,7 @@ static int reftable_log_record_decode(void *rec, struct reftable_buf key,\n \t\tgoto done;\n \t}\n \n-\tr->value.update.tz_offset = get_be16(in.buf);\n+\tr->value.update.tz_offset = reftable_get_be16(in.buf);\n \tstring_view_consume(&in, 2);\n \n \tn = decode_string(scratch, in);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 155863ee5f..5961698311 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -99,9 +99,9 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \n \tdest[4] = writer_version(w);\n \n-\tput_be24(dest + 5, w->opts.block_size);\n-\tput_be64(dest + 8, w->min_update_index);\n-\tput_be64(dest + 16, w->max_update_index);\n+\treftable_put_be24(dest + 5, w->opts.block_size);\n+\treftable_put_be64(dest + 8, w->min_update_index);\n+\treftable_put_be64(dest + 16, w->max_update_index);\n \tif (writer_version(w) == 2) {\n \t\tuint32_t hash_id;\n \n@@ -116,7 +116,7 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \t\t\treturn -1;\n \t\t}\n \n-\t\tput_be32(dest + 24, hash_id);\n+\t\treftable_put_be32(dest + 24, hash_id);\n \t}\n \n \treturn header_size(writer_version(w));\n@@ -717,19 +717,19 @@ int reftable_writer_close(struct reftable_writer *w)\n \t}\n \n \tp += writer_write_header(w, footer);\n-\tput_be64(p, w->stats.ref_stats.index_offset);\n+\treftable_put_be64(p, w->stats.ref_stats.index_offset);\n \tp += 8;\n-\tput_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n+\treftable_put_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n \tp += 8;\n-\tput_be64(p, w->stats.obj_stats.index_offset);\n+\treftable_put_be64(p, w->stats.obj_stats.index_offset);\n \tp += 8;\n \n-\tput_be64(p, w->stats.log_stats.offset);\n+\treftable_put_be64(p, w->stats.log_stats.offset);\n \tp += 8;\n-\tput_be64(p, w->stats.log_stats.index_offset);\n+\treftable_put_be64(p, w->stats.log_stats.index_offset);\n \tp += 8;\n \n-\tput_be32(p, crc32(0, footer, p - footer));\n+\treftable_put_be32(p, crc32(0, footer, p - footer));\n \tp += 4;\n \n \terr = w->flush(w->write_arg);\ndiff --git a/t/unit-tests/t-reftable-basics.c b/t/unit-tests/t-reftable-basics.c\nindex 9ba7eb05ad..c9e751e49e 100644\n--- a/t/unit-tests/t-reftable-basics.c\n+++ b/t/unit-tests/t-reftable-basics.c\n@@ -128,12 +128,30 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \t\treftable_buf_release(&b);\n \t}\n \n-\tif_test (\"put_be24 and get_be24 work\") {\n+\tif_test (\"reftable_put_be64 and reftable_get_be64 work\") {\n+\t\tuint64_t in = 0x1122334455667788;\n+\t\tuint8_t dest[8];\n+\t\tuint64_t out;\n+\t\treftable_put_be64(dest, in);\n+\t\tout = reftable_get_be64(dest);\n+\t\tcheck_int(in, ==, out);\n+\t}\n+\n+\tif_test (\"reftable_put_be32 and reftable_get_be32 work\") {\n+\t\tuint32_t in = 0x11223344;\n+\t\tuint8_t dest[4];\n+\t\tuint32_t out;\n+\t\treftable_put_be32(dest, in);\n+\t\tout = reftable_get_be32(dest);\n+\t\tcheck_int(in, ==, out);\n+\t}\n+\n+\tif_test (\"reftable_put_be24 and reftable_get_be24 work\") {\n \t\tuint32_t in = 0x112233;\n \t\tuint8_t dest[3];\n \t\tuint32_t out;\n-\t\tput_be24(dest, in);\n-\t\tout = get_be24(dest);\n+\t\treftable_put_be24(dest, in);\n+\t\tout = reftable_get_be24(dest);\n \t\tcheck_int(in, ==, out);\n \t}\n \n@@ -141,8 +159,8 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \t\tuint32_t in = 0xfef1;\n \t\tuint8_t dest[3];\n \t\tuint32_t out;\n-\t\tput_be16(dest, in);\n-\t\tout = get_be16(dest);\n+\t\treftable_put_be16(dest, in);\n+\t\tout = reftable_get_be16(dest);\n \t\tcheck_int(in, ==, out);\n \t}\n \n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511327","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-10-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 10/20] reftable/reader: stop using `ARRAY_SIZE()` macro","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:11Z","receivedAt":"2025-01-28T08:28:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have a single user of the `ARRAY_SIZE()` macro in the reftable\nreader. Drop its use to reduce our dependence on the Git codebase.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/reader.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex bf07a0a586..c3a3674665 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -849,7 +849,7 @@ int reftable_reader_print_blocks(const char *tablename)\n \tprintf(\"header:\\n\");\n \tprintf(\" block_size: %d\\n\", r->block_size);\n \n-\tfor (i = 0; i < ARRAY_SIZE(sections); i++) {\n+\tfor (i = 0; i < sizeof(sections) / sizeof(*sections); i++) {\n \t\terr = table_iter_seek_start(&ti, sections[i].type, 0);\n \t\tif (err < 0)\n \t\t\tgoto done;\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511328","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-11-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 11/20] reftable/system: introduce `reftable_rand()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:12Z","receivedAt":"2025-01-28T08:28:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new system-level `reftable_rand()` function that generates a\nsingle unsigned integer for us. The implementation of this function is\nto be provided by the calling codebase, which allows us to more easily\nhook into pre-seeded random number generators.\n\nAdapt the two callsites where we generated random data.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 4 ++--\n reftable/system.c | 5 +++++\n reftable/system.h | 6 ++++++\n 3 files changed, 13 insertions(+), 2 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 5f155b344b..82bdf5169f 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -523,7 +523,7 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\tclose(fd);\n \t\tfd = -1;\n \n-\t\tdelay = delay + (delay * git_rand(CSPRNG_BYTES_INSECURE)) / UINT32_MAX + 1;\n+\t\tdelay = delay + (delay * reftable_rand()) / UINT32_MAX + 1;\n \t\tsleep_millisec(delay);\n \t}\n \n@@ -688,7 +688,7 @@ int reftable_stack_add(struct reftable_stack *st,\n static int format_name(struct reftable_buf *dest, uint64_t min, uint64_t max)\n {\n \tchar buf[100];\n-\tuint32_t rnd = git_rand(CSPRNG_BYTES_INSECURE);\n+\tuint32_t rnd = reftable_rand();\n \tsnprintf(buf, sizeof(buf), \"0x%012\" PRIx64 \"-0x%012\" PRIx64 \"-%08x\",\n \t\t min, max, rnd);\n \treftable_buf_reset(dest);\ndiff --git a/reftable/system.c b/reftable/system.c\nindex adf8e4d30b..e25ccc0da3 100644\n--- a/reftable/system.c\n+++ b/reftable/system.c\n@@ -4,6 +4,11 @@\n #include \"../lockfile.h\"\n #include \"../tempfile.h\"\n \n+uint32_t reftable_rand(void)\n+{\n+\treturn git_rand(CSPRNG_BYTES_INSECURE);\n+}\n+\n int tmpfile_from_pattern(struct reftable_tmpfile *out, const char *pattern)\n {\n \tstruct tempfile *tempfile;\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 7d5f803eeb..3bd4a4e322 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -13,6 +13,12 @@ license that can be found in the LICENSE file or at\n \n #include \"git-compat-util.h\"\n \n+/*\n+ * Return a random 32 bit integer. This function is expected to return\n+ * pre-seeded data.\n+ */\n+uint32_t reftable_rand(void);\n+\n /*\n * An implementation-specific temporary file. By making this specific to the\n * implementation it becomes possible to tie temporary files into any kind of\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511329","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-12-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 12/20] reftable/stack: stop using `sleep_millisec()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:13Z","receivedAt":"2025-01-28T08:28:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor our use of `sleep_millisec()` by open-coding it with poll(3p),\nwhich is the current implementation of this function. Ideally, we'd use\na more direct way to sleep, but there is no equivalent to sleep(3p) that\nwould accept milliseconds as input.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 82bdf5169f..8e9836aef3 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -524,7 +524,7 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\tfd = -1;\n \n \t\tdelay = delay + (delay * reftable_rand()) / UINT32_MAX + 1;\n-\t\tsleep_millisec(delay);\n+\t\tpoll(NULL, 0, delay);\n \t}\n \n out:\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511330","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-13-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 13/20] reftable/basics: stop using `SWAP()` macro","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:14Z","receivedAt":"2025-01-28T08:28:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Stop using `SWAP()` macro in favor of an open-coded variant of it. Note\nthat this also requires us to open-code the build assert that `SWAP()`\nitself uses to verify that the size of both variables matches.\n\nThis is done to reduce our dependency on the Git codebase.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 9 +++++++++\n reftable/merged.c | 2 +-\n reftable/pq.c | 4 ++--\n reftable/record.c | 16 ++++++++--------\n 4 files changed, 20 insertions(+), 11 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex c1ddbaec3f..59000798f0 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -266,6 +266,15 @@ static inline void *reftable_alloc_grow(void *p, size_t nelem, size_t elsize,\n # define strdup(str) REFTABLE_BANNED(strdup)\n #endif\n \n+#define REFTABLE_SWAP(a, b) do {\t\t\t\t\t\t\t\t\\\n+\tvoid *_swap_a_ptr = &(a);\t\t\t\t\t\t\t\t\\\n+\tvoid *_swap_b_ptr = &(b);\t\t\t\t\t\t\t\t\\\n+\tunsigned char _swap_buffer[sizeof(a) - 2 * sizeof(a) * (sizeof(a) != sizeof(b))];\t\\\n+\tmemcpy(_swap_buffer, _swap_a_ptr, sizeof(a));\t\t\t\t\t\t\\\n+\tmemcpy(_swap_a_ptr, _swap_b_ptr, sizeof(a));\t\t\t\t\t\t\\\n+\tmemcpy(_swap_b_ptr, _swap_buffer, sizeof(a));\t\t\t\t\t\t\\\n+} while (0)\n+\n /* Find the longest shared prefix size of `a` and `b` */\n size_t common_prefix_size(struct reftable_buf *a, struct reftable_buf *b);\n \ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 563864068c..4ff1553772 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -155,7 +155,7 @@ static int merged_iter_next_entry(struct merged_iter *mi,\n \t}\n \n \tmi->advance_index = entry.index;\n-\tSWAP(*rec, *entry.rec);\n+\tREFTABLE_SWAP(*rec, *entry.rec);\n \treturn 0;\n }\n \ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex ef8035cfd9..82394a972d 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -57,7 +57,7 @@ int merged_iter_pqueue_remove(struct merged_iter_pqueue *pq, struct pq_entry *ou\n \n \t\tif (min == i)\n \t\t\tbreak;\n-\t\tSWAP(pq->heap[i], pq->heap[min]);\n+\t\tREFTABLE_SWAP(pq->heap[i], pq->heap[min]);\n \t\ti = min;\n \t}\n \n@@ -81,7 +81,7 @@ int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry\n \t\tsize_t j = (i - 1) / 2;\n \t\tif (pq_less(&pq->heap[j], &pq->heap[i]))\n \t\t\tbreak;\n-\t\tSWAP(pq->heap[j], pq->heap[i]);\n+\t\tREFTABLE_SWAP(pq->heap[j], pq->heap[i]);\n \t\ti = j;\n \t}\n \ndiff --git a/reftable/record.c b/reftable/record.c\nindex 286659c098..d86cd65bb4 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -237,11 +237,11 @@ static int reftable_ref_record_copy_from(void *rec, const void *src_rec,\n \tsize_t refname_cap = 0;\n \tint err;\n \n-\tSWAP(refname, ref->refname);\n-\tSWAP(refname_cap, ref->refname_cap);\n+\tREFTABLE_SWAP(refname, ref->refname);\n+\tREFTABLE_SWAP(refname_cap, ref->refname_cap);\n \treftable_ref_record_release(ref);\n-\tSWAP(ref->refname, refname);\n-\tSWAP(ref->refname_cap, refname_cap);\n+\tREFTABLE_SWAP(ref->refname, refname);\n+\tREFTABLE_SWAP(ref->refname_cap, refname_cap);\n \n \tif (src->refname) {\n \t\tsize_t refname_len = strlen(src->refname);\n@@ -376,11 +376,11 @@ static int reftable_ref_record_decode(void *rec, struct reftable_buf key,\n \t\treturn n;\n \tstring_view_consume(&in, n);\n \n-\tSWAP(refname, r->refname);\n-\tSWAP(refname_cap, r->refname_cap);\n+\tREFTABLE_SWAP(refname, r->refname);\n+\tREFTABLE_SWAP(refname_cap, r->refname_cap);\n \treftable_ref_record_release(r);\n-\tSWAP(r->refname, refname);\n-\tSWAP(r->refname_cap, refname_cap);\n+\tREFTABLE_SWAP(r->refname, refname);\n+\tREFTABLE_SWAP(r->refname_cap, refname_cap);\n \n \tREFTABLE_ALLOC_GROW_OR_NULL(r->refname, key.len + 1, r->refname_cap);\n \tif (!r->refname) {\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511331","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-14-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 14/20] reftable/basics: stop using `UNUSED` annotation","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:15Z","receivedAt":"2025-01-28T08:28:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Stop using the `UNUSED` annotation and replace it with a new\n`REFTABLE_UNUSED` macro. The latter is a weaker guarantee compared to\n`UNUSED` as it only suppresses unused parameters without generating a\nwarning in case a parameter marked as unused is in fact used. But it's\ngood enough, and by relaxing the behaviour a bit we avoid having to wire\nup compiler-specific logic.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 2 ++\n reftable/blocksource.c | 10 +++++++---\n reftable/iter.c | 17 ++++++++++++-----\n reftable/record.c | 51 ++++++++++++++++++++++++++++++++++++--------------\n reftable/writer.c | 4 +++-\n 5 files changed, 61 insertions(+), 23 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 59000798f0..4d0645a4e9 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -16,6 +16,8 @@ license that can be found in the LICENSE file or at\n #include \"system.h\"\n #include \"reftable-basics.h\"\n \n+#define REFTABLE_UNUSED(x) (void)(x)\n+\n struct reftable_buf {\n \tsize_t alloc;\n \tsize_t len;\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex 02972c46f4..bfd64b0e48 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -13,15 +13,17 @@ license that can be found in the LICENSE file or at\n #include \"reftable-blocksource.h\"\n #include \"reftable-error.h\"\n \n-static void reftable_buf_return_block(void *b UNUSED, struct reftable_block *dest)\n+static void reftable_buf_return_block(void *b, struct reftable_block *dest)\n {\n+\tREFTABLE_UNUSED(b);\n \tif (dest->len)\n \t\tmemset(dest->data, 0xff, dest->len);\n \treftable_free(dest->data);\n }\n \n-static void reftable_buf_close(void *b UNUSED)\n+static void reftable_buf_close(void *b)\n {\n+\tREFTABLE_UNUSED(b);\n }\n \n static ssize_t reftable_buf_read_block(void *v, struct reftable_block *dest,\n@@ -67,8 +69,10 @@ static uint64_t file_size(void *b)\n \treturn ((struct file_block_source *)b)->size;\n }\n \n-static void file_return_block(void *b UNUSED, struct reftable_block *dest UNUSED)\n+static void file_return_block(void *b, struct reftable_block *dest)\n {\n+\tREFTABLE_UNUSED(b);\n+\tREFTABLE_UNUSED(dest);\n }\n \n static void file_close(void *v)\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex b2ffb09c16..452add2705 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -25,18 +25,23 @@ int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n \treturn it->ops->next(it->iter_arg, rec);\n }\n \n-static int empty_iterator_seek(void *arg UNUSED, struct reftable_record *want UNUSED)\n+static int empty_iterator_seek(void *arg, struct reftable_record *want)\n {\n+\tREFTABLE_UNUSED(arg);\n+\tREFTABLE_UNUSED(want);\n \treturn 0;\n }\n \n-static int empty_iterator_next(void *arg UNUSED, struct reftable_record *rec UNUSED)\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n {\n+\tREFTABLE_UNUSED(arg);\n+\tREFTABLE_UNUSED(rec);\n \treturn 1;\n }\n \n-static void empty_iterator_close(void *arg UNUSED)\n+static void empty_iterator_close(void *arg)\n {\n+\tREFTABLE_UNUSED(arg);\n }\n \n static struct reftable_iterator_vtable empty_vtable = {\n@@ -143,9 +148,11 @@ static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n \treturn 0;\n }\n \n-static int indexed_table_ref_iter_seek(void *p UNUSED,\n-\t\t\t\t struct reftable_record *want UNUSED)\n+static int indexed_table_ref_iter_seek(void *p,\n+\t\t\t\t struct reftable_record *want)\n {\n+\tREFTABLE_UNUSED(p);\n+\tREFTABLE_UNUSED(want);\n \treturn REFTABLE_API_ERROR;\n }\n \ndiff --git a/reftable/record.c b/reftable/record.c\nindex d86cd65bb4..46093ef987 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -490,11 +490,13 @@ static void reftable_obj_record_release(void *rec)\n }\n \n static int reftable_obj_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t\t uint32_t hash_size)\n {\n \tstruct reftable_obj_record *obj = rec;\n \tconst struct reftable_obj_record *src = src_rec;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\n \treftable_obj_record_release(obj);\n \n \tREFTABLE_ALLOC_ARRAY(obj->hash_prefix, src->hash_prefix_len);\n@@ -523,13 +525,16 @@ static uint8_t reftable_obj_record_val_type(const void *rec)\n }\n \n static int reftable_obj_record_encode(const void *rec, struct string_view s,\n-\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t uint32_t hash_size)\n {\n \tconst struct reftable_obj_record *r = rec;\n \tstruct string_view start = s;\n \tint i = 0;\n \tint n = 0;\n \tuint64_t last = 0;\n+\n+\tREFTABLE_UNUSED(hash_size);\n+\n \tif (r->offset_len == 0 || r->offset_len >= 8) {\n \t\tn = put_var_int(&s, r->offset_len);\n \t\tif (n < 0) {\n@@ -558,8 +563,8 @@ static int reftable_obj_record_encode(const void *rec, struct string_view s,\n \n static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n \t\t\t\t uint8_t val_type, struct string_view in,\n-\t\t\t\t uint32_t hash_size UNUSED,\n-\t\t\t\t struct reftable_buf *scratch UNUSED)\n+\t\t\t\t uint32_t hash_size,\n+\t\t\t\t struct reftable_buf *scratch)\n {\n \tstruct string_view start = in;\n \tstruct reftable_obj_record *r = rec;\n@@ -567,6 +572,9 @@ static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n \tint n = 0;\n \tuint64_t last;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\tREFTABLE_UNUSED(scratch);\n+\n \treftable_obj_record_release(r);\n \n \tREFTABLE_ALLOC_ARRAY(r->hash_prefix, key.len);\n@@ -613,17 +621,20 @@ static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n \treturn start.len - in.len;\n }\n \n-static int not_a_deletion(const void *p UNUSED)\n+static int not_a_deletion(const void *p)\n {\n+\tREFTABLE_UNUSED(p);\n \treturn 0;\n }\n \n static int reftable_obj_record_equal_void(const void *a, const void *b,\n-\t\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t\t uint32_t hash_size)\n {\n \tstruct reftable_obj_record *ra = (struct reftable_obj_record *) a;\n \tstruct reftable_obj_record *rb = (struct reftable_obj_record *) b;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\n \tif (ra->hash_prefix_len != rb->hash_prefix_len\n \t || ra->offset_len != rb->offset_len)\n \t\treturn 0;\n@@ -1049,12 +1060,14 @@ static int reftable_index_record_key(const void *r, struct reftable_buf *dest)\n }\n \n static int reftable_index_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t\t uint32_t hash_size)\n {\n \tstruct reftable_index_record *dst = rec;\n \tconst struct reftable_index_record *src = src_rec;\n \tint err;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\n \treftable_buf_reset(&dst->last_key);\n \terr = reftable_buf_add(&dst->last_key, src->last_key.buf, src->last_key.len);\n \tif (err < 0)\n@@ -1070,19 +1083,23 @@ static void reftable_index_record_release(void *rec)\n \treftable_buf_release(&idx->last_key);\n }\n \n-static uint8_t reftable_index_record_val_type(const void *rec UNUSED)\n+static uint8_t reftable_index_record_val_type(const void *rec)\n {\n+\tREFTABLE_UNUSED(rec);\n \treturn 0;\n }\n \n static int reftable_index_record_encode(const void *rec, struct string_view out,\n-\t\t\t\t\tuint32_t hash_size UNUSED)\n+\t\t\t\t\tuint32_t hash_size)\n {\n \tconst struct reftable_index_record *r =\n \t\t(const struct reftable_index_record *)rec;\n \tstruct string_view start = out;\n+\tint n;\n \n-\tint n = put_var_int(&out, r->offset);\n+\tREFTABLE_UNUSED(hash_size);\n+\n+\tn = put_var_int(&out, r->offset);\n \tif (n < 0)\n \t\treturn n;\n \n@@ -1092,15 +1109,19 @@ static int reftable_index_record_encode(const void *rec, struct string_view out,\n }\n \n static int reftable_index_record_decode(void *rec, struct reftable_buf key,\n-\t\t\t\t\tuint8_t val_type UNUSED,\n+\t\t\t\t\tuint8_t val_type,\n \t\t\t\t\tstruct string_view in,\n-\t\t\t\t\tuint32_t hash_size UNUSED,\n-\t\t\t\t\tstruct reftable_buf *scratch UNUSED)\n+\t\t\t\t\tuint32_t hash_size,\n+\t\t\t\t\tstruct reftable_buf *scratch)\n {\n \tstruct string_view start = in;\n \tstruct reftable_index_record *r = rec;\n \tint err, n = 0;\n \n+\tREFTABLE_UNUSED(val_type);\n+\tREFTABLE_UNUSED(hash_size);\n+\tREFTABLE_UNUSED(scratch);\n+\n \treftable_buf_reset(&r->last_key);\n \terr = reftable_buf_add(&r->last_key, key.buf, key.len);\n \tif (err < 0)\n@@ -1115,11 +1136,13 @@ static int reftable_index_record_decode(void *rec, struct reftable_buf key,\n }\n \n static int reftable_index_record_equal(const void *a, const void *b,\n-\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t uint32_t hash_size)\n {\n \tstruct reftable_index_record *ia = (struct reftable_index_record *) a;\n \tstruct reftable_index_record *ib = (struct reftable_index_record *) b;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\n \treturn ia->offset == ib->offset && !reftable_buf_cmp(&ia->last_key, &ib->last_key);\n }\n \ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 5961698311..0040a1b1c4 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -636,10 +636,12 @@ static void write_object_record(void *void_arg, void *key)\n done:;\n }\n \n-static void object_record_free(void *void_arg UNUSED, void *key)\n+static void object_record_free(void *void_arg, void *key)\n {\n \tstruct obj_index_tree_node *entry = key;\n \n+\tREFTABLE_UNUSED(void_arg);\n+\n \tREFTABLE_FREE_AND_NULL(entry->offsets);\n \treftable_buf_release(&entry->hash);\n \treftable_free(entry);\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511332","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-15-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 15/20] compat: consistently resolve headers via project root","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:16Z","receivedAt":"2025-01-28T08:28:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The way we include headers in the \"compat/\" directory is inconsistent:\n\n - Sometimes we use includes relative to the current file.\n\n - Sometimes we use includes relative to the project root directory,\n which is being added via a `-I` compiler switch.\n\n - On Windows we also use includes relative to \"compat/\", which is\n being added via another platform-specific `-I` compiler switch in\n \"compat.mak.uname\".\n\nThis makes it very confusing to figure out which specific file is being\nincluded right now and from what layer that file comes.\n\nStop adding \"compat/\" to the Windows-specific include directories and\nadapt all includes in \"compat/\" to be relative to the projet's root\ndirectory. Despite being more consistent, it also makes it way easier to\nspot where an include comes from.\n\nThis change also prepares for subsequent commits where we split up both\n\"compat/mingw.h\" and \"compat/msvc.h\".\n\nSigned-off-by: Patrick Steinhardt \n---\n compat/access.c | 3 ++-\n compat/basename.c | 4 ++--\n compat/fileno.c | 2 +-\n compat/fopen.c | 2 +-\n compat/fsmonitor/fsm-health-darwin.c | 2 +-\n compat/fsmonitor/fsm-health-win32.c | 2 +-\n compat/fsmonitor/fsm-listen-darwin.c | 4 ++--\n compat/fsmonitor/fsm-listen-win32.c | 2 +-\n compat/inet_ntop.c | 2 +-\n compat/inet_pton.c | 2 +-\n compat/memmem.c | 2 +-\n compat/mingw.c | 24 ++++++++++++------------\n compat/mkdir.c | 2 +-\n compat/mkdtemp.c | 2 +-\n compat/mmap.c | 2 +-\n compat/msvc.c | 8 ++++----\n compat/nonblock.c | 4 ++--\n compat/obstack.c | 2 +-\n compat/pread.c | 4 ++--\n compat/precompose_utf8.c | 2 +-\n compat/qsort_s.c | 2 +-\n compat/regcomp_enhanced.c | 2 +-\n compat/setenv.c | 2 +-\n compat/snprintf.c | 2 +-\n compat/strcasestr.c | 2 +-\n compat/strdup.c | 2 +-\n compat/strlcpy.c | 2 +-\n compat/strtoimax.c | 2 +-\n compat/strtoumax.c | 2 +-\n compat/unsetenv.c | 2 +-\n compat/win32/dirent.c | 2 +-\n compat/win32/flush.c | 2 +-\n compat/win32/path-utils.c | 4 ++--\n compat/win32/pthread.c | 4 ++--\n compat/win32/syslog.c | 2 +-\n compat/win32/trace2_win32_process_info.c | 10 +++++-----\n compat/win32mmap.c | 2 +-\n compat/winansi.c | 6 +++---\n config.mak.uname | 4 ++--\n 39 files changed, 67 insertions(+), 66 deletions(-)\n\ndiff --git a/compat/access.c b/compat/access.c\nindex 19fda3e877..918f51a08b 100644\n--- a/compat/access.c\n+++ b/compat/access.c\n@@ -1,5 +1,6 @@\n #define COMPAT_CODE_ACCESS\n-#include \"../git-compat-util.h\"\n+\n+#include \"git-compat-util.h\"\n \n /* Do the same thing access(2) does, but use the effective uid,\n * and don't make the mistake of telling root that any file is\ndiff --git a/compat/basename.c b/compat/basename.c\nindex c33579ef61..2f59c5475b 100644\n--- a/compat/basename.c\n+++ b/compat/basename.c\n@@ -1,5 +1,5 @@\n-#include \"../git-compat-util.h\"\n-#include \"../strbuf.h\"\n+#include \"git-compat-util.h\"\n+#include \"strbuf.h\"\n \n /* Adapted from libiberty's basename.c. */\n char *gitbasename (char *path)\ndiff --git a/compat/fileno.c b/compat/fileno.c\nindex 8e80ef335d..977d63bf62 100644\n--- a/compat/fileno.c\n+++ b/compat/fileno.c\n@@ -1,5 +1,5 @@\n #define COMPAT_CODE_FILENO\n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n int git_fileno(FILE *stream)\n {\ndiff --git a/compat/fopen.c b/compat/fopen.c\nindex 107b3e8182..4af1f14996 100644\n--- a/compat/fopen.c\n+++ b/compat/fopen.c\n@@ -9,7 +9,7 @@\n * fopen after including git-compat-util.h is inadequate in this case.\n */\n #define SUPPRESS_FOPEN_REDEFINITION\n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n FILE *git_fopen(const char *path, const char *mode)\n {\ndiff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c\nindex c2afcbe6c8..68c9fd7363 100644\n--- a/compat/fsmonitor/fsm-health-darwin.c\n+++ b/compat/fsmonitor/fsm-health-darwin.c\n@@ -1,7 +1,7 @@\n #include \"git-compat-util.h\"\n+#include \"compat/fsmonitor/fsm-health.h\"\n #include \"config.h\"\n #include \"fsmonitor-ll.h\"\n-#include \"fsm-health.h\"\n #include \"fsmonitor--daemon.h\"\n \n int fsm_health__ctor(struct fsmonitor_daemon_state *state UNUSED)\ndiff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c\nindex 2aa8c219ac..16e073383b 100644\n--- a/compat/fsmonitor/fsm-health-win32.c\n+++ b/compat/fsmonitor/fsm-health-win32.c\n@@ -1,7 +1,7 @@\n #include \"git-compat-util.h\"\n+#include \"compat/fsmonitor/fsm-health.h\"\n #include \"config.h\"\n #include \"fsmonitor-ll.h\"\n-#include \"fsm-health.h\"\n #include \"fsmonitor--daemon.h\"\n #include \"gettext.h\"\n #include \"simple-ipc.h\"\ndiff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c\nindex 43c3a915a0..b9414da0d1 100644\n--- a/compat/fsmonitor/fsm-listen-darwin.c\n+++ b/compat/fsmonitor/fsm-listen-darwin.c\n@@ -1,6 +1,6 @@\n #ifndef __clang__\n #include \n-#include \"fsm-darwin-gcc.h\"\n+#include \"compat/fsmonitor/fsm-darwin-gcc.h\"\n #else\n #include \n #include \n@@ -24,8 +24,8 @@\n #endif\n \n #include \"git-compat-util.h\"\n+#include \"compat/fsmonitor/fsm-listen.h\"\n #include \"fsmonitor-ll.h\"\n-#include \"fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n #include \"fsmonitor-path-utils.h\"\n #include \"gettext.h\"\ndiff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c\nindex 9a6efc9bea..3fb5710e21 100644\n--- a/compat/fsmonitor/fsm-listen-win32.c\n+++ b/compat/fsmonitor/fsm-listen-win32.c\n@@ -1,7 +1,7 @@\n #include \"git-compat-util.h\"\n+#include \"compat/fsmonitor/fsm-listen.h\"\n #include \"config.h\"\n #include \"fsmonitor-ll.h\"\n-#include \"fsm-listen.h\"\n #include \"fsmonitor--daemon.h\"\n #include \"gettext.h\"\n #include \"simple-ipc.h\"\ndiff --git a/compat/inet_ntop.c b/compat/inet_ntop.c\nindex 68307262be..3bac8377cc 100644\n--- a/compat/inet_ntop.c\n+++ b/compat/inet_ntop.c\n@@ -15,7 +15,7 @@\n * SOFTWARE.\n */\n \n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n #ifndef NS_INADDRSZ\n #define NS_INADDRSZ\t4\ndiff --git a/compat/inet_pton.c b/compat/inet_pton.c\nindex 2b9a0a4e22..60935a1c83 100644\n--- a/compat/inet_pton.c\n+++ b/compat/inet_pton.c\n@@ -15,7 +15,7 @@\n * WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.\n */\n \n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n #ifndef NS_INT16SZ\n #define NS_INT16SZ 2\ndiff --git a/compat/memmem.c b/compat/memmem.c\nindex 56bcb4277f..2df3e7b7b9 100644\n--- a/compat/memmem.c\n+++ b/compat/memmem.c\n@@ -1,4 +1,4 @@\n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n void *gitmemmem(const void *haystack, size_t haystack_len,\n const void *needle, size_t needle_len)\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex 1d5b211b54..5b0593c59d 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -1,22 +1,22 @@\n #define USE_THE_REPOSITORY_VARIABLE\n #define DISABLE_SIGN_COMPARE_WARNINGS\n \n-#include \"../git-compat-util.h\"\n-#include \"win32.h\"\n+#include \"git-compat-util.h\"\n+#include \"compat/win32.h\"\n #include \n #include \n #include \n #include \n-#include \"../strbuf.h\"\n-#include \"../run-command.h\"\n-#include \"../abspath.h\"\n-#include \"../alloc.h\"\n-#include \"win32/lazyload.h\"\n-#include \"../config.h\"\n-#include \"../environment.h\"\n-#include \"../trace2.h\"\n-#include \"../symlinks.h\"\n-#include \"../wrapper.h\"\n+#include \"strbuf.h\"\n+#include \"run-command.h\"\n+#include \"abspath.h\"\n+#include \"alloc.h\"\n+#include \"compat/win32/lazyload.h\"\n+#include \"config.h\"\n+#include \"environment.h\"\n+#include \"trace2.h\"\n+#include \"symlinks.h\"\n+#include \"wrapper.h\"\n #include \"dir.h\"\n #include \"gettext.h\"\n #define SECURITY_WIN32\ndiff --git a/compat/mkdir.c b/compat/mkdir.c\nindex 02aea3b32e..8bd7c9a927 100644\n--- a/compat/mkdir.c\n+++ b/compat/mkdir.c\n@@ -1,4 +1,4 @@\n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n #undef mkdir\n \n /* for platforms that can't deal with a trailing '/' */\ndiff --git a/compat/mkdtemp.c b/compat/mkdtemp.c\nindex 1136119592..4349e15eee 100644\n--- a/compat/mkdtemp.c\n+++ b/compat/mkdtemp.c\n@@ -1,4 +1,4 @@\n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n char *gitmkdtemp(char *template)\n {\ndiff --git a/compat/mmap.c b/compat/mmap.c\nindex 2fe1c7732e..0a50adc9b9 100644\n--- a/compat/mmap.c\n+++ b/compat/mmap.c\n@@ -1,4 +1,4 @@\n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n void *git_mmap(void *start, size_t length, int prot, int flags, int fd, off_t offset)\n {\ndiff --git a/compat/msvc.c b/compat/msvc.c\nindex 71843d7eef..5aacf3af19 100644\n--- a/compat/msvc.c\n+++ b/compat/msvc.c\n@@ -1,6 +1,6 @@\n-#include \"../git-compat-util.h\"\n-#include \"win32.h\"\n+#include \"git-compat-util.h\"\n+#include \"compat/win32.h\"\n #include \n-#include \"../strbuf.h\"\n+#include \"strbuf.h\"\n \n-#include \"mingw.c\"\n+#include \"compat/mingw.c\"\ndiff --git a/compat/nonblock.c b/compat/nonblock.c\nindex 5b51195c32..99ea76a7a4 100644\n--- a/compat/nonblock.c\n+++ b/compat/nonblock.c\n@@ -1,5 +1,5 @@\n #include \"git-compat-util.h\"\n-#include \"nonblock.h\"\n+#include \"compat/nonblock.h\"\n \n #ifdef O_NONBLOCK\n \n@@ -14,7 +14,7 @@ int enable_pipe_nonblock(int fd)\n \n #elif defined(GIT_WINDOWS_NATIVE)\n \n-#include \"win32.h\"\n+#include \"compat/win32.h\"\n \n int enable_pipe_nonblock(int fd)\n {\ndiff --git a/compat/obstack.c b/compat/obstack.c\nindex 27cd5c1ea1..baab07aca4 100644\n--- a/compat/obstack.c\n+++ b/compat/obstack.c\n@@ -19,7 +19,7 @@\n \n #include \"git-compat-util.h\"\n #include \n-#include \"obstack.h\"\n+#include \"compat/obstack.h\"\n \n /* NOTE BEFORE MODIFYING THIS FILE: This version number must be\n incremented whenever callers compiled using an old obstack.h can no\ndiff --git a/compat/pread.c b/compat/pread.c\nindex 484e6d4c71..82161b5a2a 100644\n--- a/compat/pread.c\n+++ b/compat/pread.c\n@@ -1,5 +1,5 @@\n-#include \"../git-compat-util.h\"\n-#include \"../wrapper.h\"\n+#include \"git-compat-util.h\"\n+#include \"wrapper.h\"\n \n ssize_t git_pread(int fd, void *buf, size_t count, off_t offset)\n {\ndiff --git a/compat/precompose_utf8.c b/compat/precompose_utf8.c\nindex f7cc7b3be5..994a2f1303 100644\n--- a/compat/precompose_utf8.c\n+++ b/compat/precompose_utf8.c\n@@ -7,13 +7,13 @@\n #define USE_THE_REPOSITORY_VARIABLE\n \n #include \"git-compat-util.h\"\n+#include \"compat/precompose_utf8.h\"\n #include \"config.h\"\n #include \"environment.h\"\n #include \"gettext.h\"\n #include \"path.h\"\n #include \"strbuf.h\"\n #include \"utf8.h\"\n-#include \"precompose_utf8.h\"\n \n typedef char *iconv_ibp;\n static const char *repo_encoding = \"UTF-8\";\ndiff --git a/compat/qsort_s.c b/compat/qsort_s.c\nindex 0f7ff30f5f..66cd6e096d 100644\n--- a/compat/qsort_s.c\n+++ b/compat/qsort_s.c\n@@ -1,4 +1,4 @@\n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n /*\n * A merge sort implementation, simplified from the qsort implementation\ndiff --git a/compat/regcomp_enhanced.c b/compat/regcomp_enhanced.c\nindex 84193ce53b..693b63c2c4 100644\n--- a/compat/regcomp_enhanced.c\n+++ b/compat/regcomp_enhanced.c\n@@ -1,4 +1,4 @@\n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n #undef regcomp\n \n int git_regcomp(regex_t *preg, const char *pattern, int cflags)\ndiff --git a/compat/setenv.c b/compat/setenv.c\nindex 7849f258d2..c49df82534 100644\n--- a/compat/setenv.c\n+++ b/compat/setenv.c\n@@ -1,4 +1,4 @@\n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n int gitsetenv(const char *name, const char *value, int replace)\n {\ndiff --git a/compat/snprintf.c b/compat/snprintf.c\nindex 0b11688537..91a63cf3f7 100644\n--- a/compat/snprintf.c\n+++ b/compat/snprintf.c\n@@ -1,4 +1,4 @@\n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n /*\n * The size parameter specifies the available space, i.e. includes\ndiff --git a/compat/strcasestr.c b/compat/strcasestr.c\nindex 26896deca6..d66609361b 100644\n--- a/compat/strcasestr.c\n+++ b/compat/strcasestr.c\n@@ -1,4 +1,4 @@\n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n char *gitstrcasestr(const char *haystack, const char *needle)\n {\ndiff --git a/compat/strdup.c b/compat/strdup.c\nindex f3fb978eb3..9ef99f88f2 100644\n--- a/compat/strdup.c\n+++ b/compat/strdup.c\n@@ -1,4 +1,4 @@\n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n char *gitstrdup(const char *s1)\n {\ndiff --git a/compat/strlcpy.c b/compat/strlcpy.c\nindex 4024c36030..780b39ff82 100644\n--- a/compat/strlcpy.c\n+++ b/compat/strlcpy.c\n@@ -1,4 +1,4 @@\n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n size_t gitstrlcpy(char *dest, const char *src, size_t size)\n {\ndiff --git a/compat/strtoimax.c b/compat/strtoimax.c\nindex ac09ed89e7..c9ed79f0d1 100644\n--- a/compat/strtoimax.c\n+++ b/compat/strtoimax.c\n@@ -1,4 +1,4 @@\n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n intmax_t gitstrtoimax (const char *nptr, char **endptr, int base)\n {\ndiff --git a/compat/strtoumax.c b/compat/strtoumax.c\nindex 5541353a77..50fe26d746 100644\n--- a/compat/strtoumax.c\n+++ b/compat/strtoumax.c\n@@ -1,4 +1,4 @@\n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n uintmax_t gitstrtoumax (const char *nptr, char **endptr, int base)\n {\ndiff --git a/compat/unsetenv.c b/compat/unsetenv.c\nindex b9d34af613..d96e0d7bcc 100644\n--- a/compat/unsetenv.c\n+++ b/compat/unsetenv.c\n@@ -1,4 +1,4 @@\n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n int gitunsetenv(const char *name)\n {\ndiff --git a/compat/win32/dirent.c b/compat/win32/dirent.c\nindex 52420ec7d4..362622746d 100644\n--- a/compat/win32/dirent.c\n+++ b/compat/win32/dirent.c\n@@ -1,4 +1,4 @@\n-#include \"../../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n struct DIR {\n \tstruct dirent dd_dir; /* includes d_type */\ndiff --git a/compat/win32/flush.c b/compat/win32/flush.c\nindex 291f90ea94..bcad557eb4 100644\n--- a/compat/win32/flush.c\n+++ b/compat/win32/flush.c\n@@ -1,6 +1,6 @@\n #include \"git-compat-util.h\"\n #include \n-#include \"lazyload.h\"\n+#include \"compat/win32/lazyload.h\"\n \n int win32_fsync_no_flush(int fd)\n {\ndiff --git a/compat/win32/path-utils.c b/compat/win32/path-utils.c\nindex 966ef779b9..20e9a69f56 100644\n--- a/compat/win32/path-utils.c\n+++ b/compat/win32/path-utils.c\n@@ -1,7 +1,7 @@\n #define USE_THE_REPOSITORY_VARIABLE\n \n-#include \"../../git-compat-util.h\"\n-#include \"../../environment.h\"\n+#include \"git-compat-util.h\"\n+#include \"environment.h\"\n \n int win32_has_dos_drive_prefix(const char *path)\n {\ndiff --git a/compat/win32/pthread.c b/compat/win32/pthread.c\nindex 58980a529c..86199d67ba 100644\n--- a/compat/win32/pthread.c\n+++ b/compat/win32/pthread.c\n@@ -7,8 +7,8 @@\n * no need for double-checking.\n */\n \n-#include \"../../git-compat-util.h\"\n-#include \"pthread.h\"\n+#include \"git-compat-util.h\"\n+#include \"compat/win32/pthread.h\"\n \n #include \n #include \ndiff --git a/compat/win32/syslog.c b/compat/win32/syslog.c\nindex 4e4794743a..587fb99ba9 100644\n--- a/compat/win32/syslog.c\n+++ b/compat/win32/syslog.c\n@@ -1,4 +1,4 @@\n-#include \"../../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n static HANDLE ms_eventlog;\n \ndiff --git a/compat/win32/trace2_win32_process_info.c b/compat/win32/trace2_win32_process_info.c\nindex f147da706a..55e482f100 100644\n--- a/compat/win32/trace2_win32_process_info.c\n+++ b/compat/win32/trace2_win32_process_info.c\n@@ -1,10 +1,10 @@\n #define USE_THE_REPOSITORY_VARIABLE\n \n-#include \"../../git-compat-util.h\"\n-#include \"../../json-writer.h\"\n-#include \"../../repository.h\"\n-#include \"../../trace2.h\"\n-#include \"lazyload.h\"\n+#include \"git-compat-util.h\"\n+#include \"compat/win32/lazyload.h\"\n+#include \"json-writer.h\"\n+#include \"repository.h\"\n+#include \"trace2.h\"\n #include \n #include \n \ndiff --git a/compat/win32mmap.c b/compat/win32mmap.c\nindex e951934316..aff1768acb 100644\n--- a/compat/win32mmap.c\n+++ b/compat/win32mmap.c\n@@ -1,6 +1,6 @@\n #define DISABLE_SIGN_COMPARE_WARNINGS\n \n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n \n void *git_mmap(void *start, size_t length, int prot, int flags, int fd, off_t offset)\n {\ndiff --git a/compat/winansi.c b/compat/winansi.c\nindex ac2ffb7869..143685e5ab 100644\n--- a/compat/winansi.c\n+++ b/compat/winansi.c\n@@ -6,11 +6,11 @@\n \n #define DISABLE_SIGN_COMPARE_WARNINGS\n \n-#include \"../git-compat-util.h\"\n+#include \"git-compat-util.h\"\n #include \n #include \n-#include \"win32.h\"\n-#include \"win32/lazyload.h\"\n+#include \"compat/win32.h\"\n+#include \"compat/win32/lazyload.h\"\n \n static int fd_is_interactive[3] = { 0, 0, 0 };\n #define FD_CONSOLE 0x1\ndiff --git a/config.mak.uname b/config.mak.uname\nindex b12d4e168a..cd9535a65c 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -501,7 +501,7 @@ endif\n \t\tcompat/win32/pthread.o compat/win32/syslog.o \\\n \t\tcompat/win32/trace2_win32_process_info.o \\\n \t\tcompat/win32/dirent.o\n-\tCOMPAT_CFLAGS = -D__USE_MINGW_ACCESS -DDETECT_MSYS_TTY -DNOGDI -DHAVE_STRING_H -Icompat -Icompat/regex -Icompat/win32 -DSTRIP_EXTENSION=\\\".exe\\\"\n+\tCOMPAT_CFLAGS = -D__USE_MINGW_ACCESS -DDETECT_MSYS_TTY -DNOGDI -DHAVE_STRING_H -Icompat/regex -Icompat/win32 -DSTRIP_EXTENSION=\\\".exe\\\"\n \tBASIC_LDFLAGS = -IGNORE:4217 -IGNORE:4049 -NOLOGO -ENTRY:wmainCRTStartup -SUBSYSTEM:CONSOLE\n \t# invalidcontinue.obj allows Git's source code to close the same file\n \t# handle twice, or to access the osfhandle of an already-closed stdout\n@@ -692,7 +692,7 @@ ifeq ($(uname_S),MINGW)\n \tHAVE_PLATFORM_PROCINFO = YesPlease\n \tCSPRNG_METHOD = rtlgenrandom\n \tBASIC_LDFLAGS += -municode\n-\tCOMPAT_CFLAGS += -DNOGDI -Icompat -Icompat/win32\n+\tCOMPAT_CFLAGS += -DNOGDI -Icompat/win32\n \tCOMPAT_CFLAGS += -DSTRIP_EXTENSION=\\\".exe\\\"\n \tCOMPAT_OBJS += compat/mingw.o compat/winansi.o \\\n \t\tcompat/win32/trace2_win32_process_info.o \\\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511333","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-16-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 16/20] compat/mingw: split out POSIX-related bits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:17Z","receivedAt":"2025-01-28T08:28:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Split out the POSIX-related bits from \"compat/mingw.h\". This is in\npreparation for splitting up \"git-compat-utils.h\" into a header that\nprovides POSIX-compatibility and a header that provides common wrappers\nused by the Git project.\n\nSigned-off-by: Patrick Steinhardt \n---\n compat/{mingw.c => mingw/compat-util.c} | 0\n compat/mingw/compat-util.h | 220 ++++++++++++++++++++++++++++++++\n compat/{mingw.h => mingw/posix.h} | 216 +------------------------------\n compat/msvc.c | 2 +-\n compat/msvc.h | 2 +-\n config.mak.uname | 4 +-\n contrib/buildsystems/CMakeLists.txt | 2 +-\n git-compat-util.h | 4 +-\n meson.build | 6 +-\n 9 files changed, 234 insertions(+), 222 deletions(-)\n\ndiff --git a/compat/mingw.c b/compat/mingw/compat-util.c\nsimilarity index 100%\nrename from compat/mingw.c\nrename to compat/mingw/compat-util.c\ndiff --git a/compat/mingw/compat-util.h b/compat/mingw/compat-util.h\nnew file mode 100644\nindex 0000000000..b433762526\n--- /dev/null\n+++ b/compat/mingw/compat-util.h\n@@ -0,0 +1,220 @@\n+#ifndef COMPAT_MINGW_COMPAT_UTIL_H\n+#define COMPAT_MINGW_COMPAT_UTIL_H\n+\n+#include \"compat/mingw/posix.h\"\n+\n+struct config_context;\n+int mingw_core_config(const char *var, const char *value,\n+\t\t const struct config_context *ctx, void *cb);\n+#define platform_core_config mingw_core_config\n+\n+#ifndef NO_OPENSSL\n+#include \n+static inline int mingw_SSL_set_fd(SSL *ssl, int fd)\n+{\n+\treturn SSL_set_fd(ssl, _get_osfhandle(fd));\n+}\n+#define SSL_set_fd mingw_SSL_set_fd\n+\n+static inline int mingw_SSL_set_rfd(SSL *ssl, int fd)\n+{\n+\treturn SSL_set_rfd(ssl, _get_osfhandle(fd));\n+}\n+#define SSL_set_rfd mingw_SSL_set_rfd\n+\n+static inline int mingw_SSL_set_wfd(SSL *ssl, int fd)\n+{\n+\treturn SSL_set_wfd(ssl, _get_osfhandle(fd));\n+}\n+#define SSL_set_wfd mingw_SSL_set_wfd\n+#endif\n+\n+/*\n+ * git specific compatibility\n+ */\n+\n+static inline void convert_slashes(char *path)\n+{\n+\tfor (; *path; path++)\n+\t\tif (*path == '\\\\')\n+\t\t\t*path = '/';\n+}\n+#define PATH_SEP ';'\n+char *mingw_query_user_email(void);\n+#define query_user_email mingw_query_user_email\n+\n+/**\n+ * Verifies that the specified path is owned by the user running the\n+ * current process.\n+ */\n+int is_path_owned_by_current_sid(const char *path, struct strbuf *report);\n+#define is_path_owned_by_current_user is_path_owned_by_current_sid\n+\n+/**\n+ * Verifies that the given path is a valid one on Windows.\n+ *\n+ * In particular, path segments are disallowed which\n+ *\n+ * - end in a period or a space (except the special directories `.` and `..`).\n+ *\n+ * - contain any of the reserved characters, e.g. `:`, `;`, `*`, etc\n+ *\n+ * - correspond to reserved names (such as `AUX`, `PRN`, etc)\n+ *\n+ * The `allow_literal_nul` parameter controls whether the path `NUL` should\n+ * be considered valid (this makes sense e.g. before opening files, as it is\n+ * perfectly legitimate to open `NUL` on Windows, just as it is to open\n+ * `/dev/null` on Unix/Linux).\n+ *\n+ * Returns 1 upon success, otherwise 0.\n+ */\n+int is_valid_win32_path(const char *path, int allow_literal_nul);\n+#define is_valid_path(path) is_valid_win32_path(path, 0)\n+\n+/**\n+ * Converts UTF-8 encoded string to UTF-16LE.\n+ *\n+ * To support repositories with legacy-encoded file names, invalid UTF-8 bytes\n+ * 0xa0 - 0xff are converted to corresponding printable Unicode chars \\u00a0 -\n+ * \\u00ff, and invalid UTF-8 bytes 0x80 - 0x9f (which would make non-printable\n+ * Unicode) are converted to hex-code.\n+ *\n+ * Lead-bytes not followed by an appropriate number of trail-bytes, over-long\n+ * encodings and 4-byte encodings > \\u10ffff are detected as invalid UTF-8.\n+ *\n+ * Maximum space requirement for the target buffer is two wide chars per UTF-8\n+ * char (((strlen(utf) * 2) + 1) [* sizeof(wchar_t)]).\n+ *\n+ * The maximum space is needed only if the entire input string consists of\n+ * invalid UTF-8 bytes in range 0x80-0x9f, as per the following table:\n+ *\n+ * | | UTF-8 | UTF-16 |\n+ * Code point | UTF-8 sequence | bytes | words | ratio\n+ * --------------+-------------------+-------+--------+-------\n+ * 000000-00007f | 0-7f | 1 | 1 | 1\n+ * 000080-0007ff | c2-df + 80-bf | 2 | 1 | 0.5\n+ * 000800-00ffff | e0-ef + 2 * 80-bf | 3 | 1 | 0.33\n+ * 010000-10ffff | f0-f4 + 3 * 80-bf | 4 | 2 (a) | 0.5\n+ * invalid | 80-9f | 1 | 2 (b) | 2\n+ * invalid | a0-ff | 1 | 1 | 1\n+ *\n+ * (a) encoded as UTF-16 surrogate pair\n+ * (b) encoded as two hex digits\n+ *\n+ * Note that, while the UTF-8 encoding scheme can be extended to 5-byte, 6-byte\n+ * or even indefinite-byte sequences, the largest valid code point \\u10ffff\n+ * encodes as only 4 UTF-8 bytes.\n+ *\n+ * Parameters:\n+ * wcs: wide char target buffer\n+ * utf: string to convert\n+ * wcslen: size of target buffer (in wchar_t's)\n+ * utflen: size of string to convert, or -1 if 0-terminated\n+ *\n+ * Returns:\n+ * length of converted string (_wcslen(wcs)), or -1 on failure\n+ *\n+ * Errors:\n+ * EINVAL: one of the input parameters is invalid (e.g. NULL)\n+ * ERANGE: the output buffer is too small\n+ */\n+int xutftowcsn(wchar_t *wcs, const char *utf, size_t wcslen, int utflen);\n+\n+/**\n+ * Simplified variant of xutftowcsn, assumes input string is \\0-terminated.\n+ */\n+static inline int xutftowcs(wchar_t *wcs, const char *utf, size_t wcslen)\n+{\n+\treturn xutftowcsn(wcs, utf, wcslen, -1);\n+}\n+\n+/**\n+ * Simplified file system specific variant of xutftowcsn, assumes output\n+ * buffer size is MAX_PATH wide chars and input string is \\0-terminated,\n+ * fails with ENAMETOOLONG if input string is too long.\n+ */\n+static inline int xutftowcs_path(wchar_t *wcs, const char *utf)\n+{\n+\tint result = xutftowcsn(wcs, utf, MAX_PATH, -1);\n+\tif (result < 0 && errno == ERANGE)\n+\t\terrno = ENAMETOOLONG;\n+\treturn result;\n+}\n+\n+/**\n+ * Converts UTF-16LE encoded string to UTF-8.\n+ *\n+ * Maximum space requirement for the target buffer is three UTF-8 chars per\n+ * wide char ((_wcslen(wcs) * 3) + 1).\n+ *\n+ * The maximum space is needed only if the entire input string consists of\n+ * UTF-16 words in range 0x0800-0xd7ff or 0xe000-0xffff (i.e. \\u0800-\\uffff\n+ * modulo surrogate pairs), as per the following table:\n+ *\n+ * | | UTF-16 | UTF-8 |\n+ * Code point | UTF-16 sequence | words | bytes | ratio\n+ * --------------+-----------------------+--------+-------+-------\n+ * 000000-00007f | 0000-007f | 1 | 1 | 1\n+ * 000080-0007ff | 0080-07ff | 1 | 2 | 2\n+ * 000800-00ffff | 0800-d7ff / e000-ffff | 1 | 3 | 3\n+ * 010000-10ffff | d800-dbff + dc00-dfff | 2 | 4 | 2\n+ *\n+ * Note that invalid code points > 10ffff cannot be represented in UTF-16.\n+ *\n+ * Parameters:\n+ * utf: target buffer\n+ * wcs: wide string to convert\n+ * utflen: size of target buffer\n+ *\n+ * Returns:\n+ * length of converted string, or -1 on failure\n+ *\n+ * Errors:\n+ * EINVAL: one of the input parameters is invalid (e.g. NULL)\n+ * ERANGE: the output buffer is too small\n+ */\n+int xwcstoutf(char *utf, const wchar_t *wcs, size_t utflen);\n+\n+/*\n+ * A critical section used in the implementation of the spawn\n+ * functions (mingw_spawnv[p]e()) and waitpid(). Initialised in\n+ * the replacement main() macro below.\n+ */\n+extern CRITICAL_SECTION pinfo_cs;\n+\n+/*\n+ * Git, like most portable C applications, implements a main() function. On\n+ * Windows, this main() function would receive parameters encoded in the\n+ * current locale, but Git for Windows would prefer UTF-8 encoded parameters.\n+ *\n+ * To make that happen, we still declare main() here, and then declare and\n+ * implement wmain() (which is the Unicode variant of main()) and compile with\n+ * -municode. This wmain() function reencodes the parameters from UTF-16 to\n+ * UTF-8 format, sets up a couple of other things as required on Windows, and\n+ * then hands off to the main() function.\n+ */\n+int wmain(int argc, const wchar_t **w_argv);\n+int main(int argc, const char **argv);\n+\n+/*\n+ * For debugging: if a problem occurs, say, in a Git process that is spawned\n+ * from another Git process which in turn is spawned from yet another Git\n+ * process, it can be quite daunting to figure out what is going on.\n+ *\n+ * Call this function to open a new MinTTY (this assumes you are in Git for\n+ * Windows' SDK) with a GDB that attaches to the current process right away.\n+ */\n+void open_in_gdb(void);\n+\n+/*\n+ * Used by Pthread API implementation for Windows\n+ */\n+int err_win_to_posix(DWORD winerr);\n+\n+#ifndef NO_UNIX_SOCKETS\n+int mingw_have_unix_sockets(void);\n+#undef have_unix_sockets\n+#define have_unix_sockets mingw_have_unix_sockets\n+#endif\n+\n+#endif /* COMPAT_MINGW_COMPAT_UTIL_H */\ndiff --git a/compat/mingw.h b/compat/mingw/posix.h\nsimilarity index 59%\nrename from compat/mingw.h\nrename to compat/mingw/posix.h\nindex ebfb8ba423..8dddfa818d 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw/posix.h\n@@ -1,3 +1,6 @@\n+#ifndef COMPAT_MINGW_POSIX_H\n+#define COMPAT_MINGW_POSIX_H\n+\n #ifdef __MINGW64_VERSION_MAJOR\n #include \n #include \n@@ -11,11 +14,6 @@ typedef _sigset_t sigset_t;\n #undef _POSIX_THREAD_SAFE_FUNCTIONS\n #endif\n \n-struct config_context;\n-int mingw_core_config(const char *var, const char *value,\n-\t\t const struct config_context *ctx, void *cb);\n-#define platform_core_config mingw_core_config\n-\n /*\n * things that are not available in header files\n */\n@@ -180,27 +178,6 @@ int mingw_kill(pid_t pid, int sig);\n #define locate_in_PATH mingw_locate_in_PATH\n char *mingw_locate_in_PATH(const char *cmd);\n \n-#ifndef NO_OPENSSL\n-#include \n-static inline int mingw_SSL_set_fd(SSL *ssl, int fd)\n-{\n-\treturn SSL_set_fd(ssl, _get_osfhandle(fd));\n-}\n-#define SSL_set_fd mingw_SSL_set_fd\n-\n-static inline int mingw_SSL_set_rfd(SSL *ssl, int fd)\n-{\n-\treturn SSL_set_rfd(ssl, _get_osfhandle(fd));\n-}\n-#define SSL_set_rfd mingw_SSL_set_rfd\n-\n-static inline int mingw_SSL_set_wfd(SSL *ssl, int fd)\n-{\n-\treturn SSL_set_wfd(ssl, _get_osfhandle(fd));\n-}\n-#define SSL_set_wfd mingw_SSL_set_wfd\n-#endif\n-\n /*\n * implementations of missing functions\n */\n@@ -444,19 +421,6 @@ int winansi_dup2(int oldfd, int newfd);\n void winansi_init(void);\n HANDLE winansi_get_osfhandle(int fd);\n \n-/*\n- * git specific compatibility\n- */\n-\n-static inline void convert_slashes(char *path)\n-{\n-\tfor (; *path; path++)\n-\t\tif (*path == '\\\\')\n-\t\t\t*path = '/';\n-}\n-#define PATH_SEP ';'\n-char *mingw_query_user_email(void);\n-#define query_user_email mingw_query_user_email\n #if !defined(__MINGW64_VERSION_MAJOR) && (!defined(_MSC_VER) || _MSC_VER < 1800)\n #define PRIuMAX \"I64u\"\n #define PRId64 \"I64d\"\n@@ -464,176 +428,4 @@ char *mingw_query_user_email(void);\n #include \n #endif\n \n-/**\n- * Verifies that the specified path is owned by the user running the\n- * current process.\n- */\n-int is_path_owned_by_current_sid(const char *path, struct strbuf *report);\n-#define is_path_owned_by_current_user is_path_owned_by_current_sid\n-\n-/**\n- * Verifies that the given path is a valid one on Windows.\n- *\n- * In particular, path segments are disallowed which\n- *\n- * - end in a period or a space (except the special directories `.` and `..`).\n- *\n- * - contain any of the reserved characters, e.g. `:`, `;`, `*`, etc\n- *\n- * - correspond to reserved names (such as `AUX`, `PRN`, etc)\n- *\n- * The `allow_literal_nul` parameter controls whether the path `NUL` should\n- * be considered valid (this makes sense e.g. before opening files, as it is\n- * perfectly legitimate to open `NUL` on Windows, just as it is to open\n- * `/dev/null` on Unix/Linux).\n- *\n- * Returns 1 upon success, otherwise 0.\n- */\n-int is_valid_win32_path(const char *path, int allow_literal_nul);\n-#define is_valid_path(path) is_valid_win32_path(path, 0)\n-\n-/**\n- * Converts UTF-8 encoded string to UTF-16LE.\n- *\n- * To support repositories with legacy-encoded file names, invalid UTF-8 bytes\n- * 0xa0 - 0xff are converted to corresponding printable Unicode chars \\u00a0 -\n- * \\u00ff, and invalid UTF-8 bytes 0x80 - 0x9f (which would make non-printable\n- * Unicode) are converted to hex-code.\n- *\n- * Lead-bytes not followed by an appropriate number of trail-bytes, over-long\n- * encodings and 4-byte encodings > \\u10ffff are detected as invalid UTF-8.\n- *\n- * Maximum space requirement for the target buffer is two wide chars per UTF-8\n- * char (((strlen(utf) * 2) + 1) [* sizeof(wchar_t)]).\n- *\n- * The maximum space is needed only if the entire input string consists of\n- * invalid UTF-8 bytes in range 0x80-0x9f, as per the following table:\n- *\n- * | | UTF-8 | UTF-16 |\n- * Code point | UTF-8 sequence | bytes | words | ratio\n- * --------------+-------------------+-------+--------+-------\n- * 000000-00007f | 0-7f | 1 | 1 | 1\n- * 000080-0007ff | c2-df + 80-bf | 2 | 1 | 0.5\n- * 000800-00ffff | e0-ef + 2 * 80-bf | 3 | 1 | 0.33\n- * 010000-10ffff | f0-f4 + 3 * 80-bf | 4 | 2 (a) | 0.5\n- * invalid | 80-9f | 1 | 2 (b) | 2\n- * invalid | a0-ff | 1 | 1 | 1\n- *\n- * (a) encoded as UTF-16 surrogate pair\n- * (b) encoded as two hex digits\n- *\n- * Note that, while the UTF-8 encoding scheme can be extended to 5-byte, 6-byte\n- * or even indefinite-byte sequences, the largest valid code point \\u10ffff\n- * encodes as only 4 UTF-8 bytes.\n- *\n- * Parameters:\n- * wcs: wide char target buffer\n- * utf: string to convert\n- * wcslen: size of target buffer (in wchar_t's)\n- * utflen: size of string to convert, or -1 if 0-terminated\n- *\n- * Returns:\n- * length of converted string (_wcslen(wcs)), or -1 on failure\n- *\n- * Errors:\n- * EINVAL: one of the input parameters is invalid (e.g. NULL)\n- * ERANGE: the output buffer is too small\n- */\n-int xutftowcsn(wchar_t *wcs, const char *utf, size_t wcslen, int utflen);\n-\n-/**\n- * Simplified variant of xutftowcsn, assumes input string is \\0-terminated.\n- */\n-static inline int xutftowcs(wchar_t *wcs, const char *utf, size_t wcslen)\n-{\n-\treturn xutftowcsn(wcs, utf, wcslen, -1);\n-}\n-\n-/**\n- * Simplified file system specific variant of xutftowcsn, assumes output\n- * buffer size is MAX_PATH wide chars and input string is \\0-terminated,\n- * fails with ENAMETOOLONG if input string is too long.\n- */\n-static inline int xutftowcs_path(wchar_t *wcs, const char *utf)\n-{\n-\tint result = xutftowcsn(wcs, utf, MAX_PATH, -1);\n-\tif (result < 0 && errno == ERANGE)\n-\t\terrno = ENAMETOOLONG;\n-\treturn result;\n-}\n-\n-/**\n- * Converts UTF-16LE encoded string to UTF-8.\n- *\n- * Maximum space requirement for the target buffer is three UTF-8 chars per\n- * wide char ((_wcslen(wcs) * 3) + 1).\n- *\n- * The maximum space is needed only if the entire input string consists of\n- * UTF-16 words in range 0x0800-0xd7ff or 0xe000-0xffff (i.e. \\u0800-\\uffff\n- * modulo surrogate pairs), as per the following table:\n- *\n- * | | UTF-16 | UTF-8 |\n- * Code point | UTF-16 sequence | words | bytes | ratio\n- * --------------+-----------------------+--------+-------+-------\n- * 000000-00007f | 0000-007f | 1 | 1 | 1\n- * 000080-0007ff | 0080-07ff | 1 | 2 | 2\n- * 000800-00ffff | 0800-d7ff / e000-ffff | 1 | 3 | 3\n- * 010000-10ffff | d800-dbff + dc00-dfff | 2 | 4 | 2\n- *\n- * Note that invalid code points > 10ffff cannot be represented in UTF-16.\n- *\n- * Parameters:\n- * utf: target buffer\n- * wcs: wide string to convert\n- * utflen: size of target buffer\n- *\n- * Returns:\n- * length of converted string, or -1 on failure\n- *\n- * Errors:\n- * EINVAL: one of the input parameters is invalid (e.g. NULL)\n- * ERANGE: the output buffer is too small\n- */\n-int xwcstoutf(char *utf, const wchar_t *wcs, size_t utflen);\n-\n-/*\n- * A critical section used in the implementation of the spawn\n- * functions (mingw_spawnv[p]e()) and waitpid(). Initialised in\n- * the replacement main() macro below.\n- */\n-extern CRITICAL_SECTION pinfo_cs;\n-\n-/*\n- * Git, like most portable C applications, implements a main() function. On\n- * Windows, this main() function would receive parameters encoded in the\n- * current locale, but Git for Windows would prefer UTF-8 encoded parameters.\n- *\n- * To make that happen, we still declare main() here, and then declare and\n- * implement wmain() (which is the Unicode variant of main()) and compile with\n- * -municode. This wmain() function reencodes the parameters from UTF-16 to\n- * UTF-8 format, sets up a couple of other things as required on Windows, and\n- * then hands off to the main() function.\n- */\n-int wmain(int argc, const wchar_t **w_argv);\n-int main(int argc, const char **argv);\n-\n-/*\n- * For debugging: if a problem occurs, say, in a Git process that is spawned\n- * from another Git process which in turn is spawned from yet another Git\n- * process, it can be quite daunting to figure out what is going on.\n- *\n- * Call this function to open a new MinTTY (this assumes you are in Git for\n- * Windows' SDK) with a GDB that attaches to the current process right away.\n- */\n-void open_in_gdb(void);\n-\n-/*\n- * Used by Pthread API implementation for Windows\n- */\n-int err_win_to_posix(DWORD winerr);\n-\n-#ifndef NO_UNIX_SOCKETS\n-int mingw_have_unix_sockets(void);\n-#undef have_unix_sockets\n-#define have_unix_sockets mingw_have_unix_sockets\n-#endif\n+#endif /* COMPAT_MINGW_POSIX_H */\ndiff --git a/compat/msvc.c b/compat/msvc.c\nindex 5aacf3af19..7b6b8ddc71 100644\n--- a/compat/msvc.c\n+++ b/compat/msvc.c\n@@ -3,4 +3,4 @@\n #include \n #include \"strbuf.h\"\n \n-#include \"compat/mingw.c\"\n+#include \"compat/mingw/compat-util.c\"\ndiff --git a/compat/msvc.h b/compat/msvc.h\nindex 1d7a8c6145..df39978f4f 100644\n--- a/compat/msvc.h\n+++ b/compat/msvc.h\n@@ -28,6 +28,6 @@ typedef int sigset_t;\n /* open for reading, writing, or both (not in fcntl.h) */\n #define O_ACCMODE (_O_RDONLY | _O_WRONLY | _O_RDWR)\n \n-#include \"compat/mingw.h\"\n+#include \"compat/mingw/compat-util.h\"\n \n #endif\ndiff --git a/config.mak.uname b/config.mak.uname\nindex cd9535a65c..38a774de39 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -541,7 +541,7 @@ endif\n \n \tEXTRA_PROGRAMS += headless-git$X\n \n-compat/msvc.o: compat/msvc.c compat/mingw.c GIT-CFLAGS\n+compat/msvc.o: compat/msvc.c compat/mingw/compat-util.c GIT-CFLAGS\n endif\n ifeq ($(uname_S),Interix)\n \tNO_INITGROUPS = YesPlease\n@@ -694,7 +694,7 @@ ifeq ($(uname_S),MINGW)\n \tBASIC_LDFLAGS += -municode\n \tCOMPAT_CFLAGS += -DNOGDI -Icompat/win32\n \tCOMPAT_CFLAGS += -DSTRIP_EXTENSION=\\\".exe\\\"\n-\tCOMPAT_OBJS += compat/mingw.o compat/winansi.o \\\n+\tCOMPAT_OBJS += compat/mingw/compat-util.o compat/winansi.o \\\n \t\tcompat/win32/trace2_win32_process_info.o \\\n \t\tcompat/win32/flush.o \\\n \t\tcompat/win32/path-utils.o \\\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 10dc54fdcb..65ce8705f0 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -258,7 +258,7 @@ if(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\t\t\tUSE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP\n \t\t\t\tHAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET HAVE_RTLGENRANDOM)\n \tlist(APPEND compat_SOURCES\n-\t\tcompat/mingw.c\n+\t\tcompat/mingw/compat-util.c\n \t\tcompat/winansi.c\n \t\tcompat/win32/flush.c\n \t\tcompat/win32/path-utils.c\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex e283c46c6f..b96fb98e1e 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -304,10 +304,10 @@ static inline int is_xplatform_dir_sep(int c)\n #if defined(__MINGW32__)\n /* pull in Windows compatibility stuff */\n #include \"compat/win32/path-utils.h\"\n-#include \"compat/mingw.h\"\n+#include \"compat/mingw/compat-util.h\"\n #elif defined(_MSC_VER)\n #include \"compat/win32/path-utils.h\"\n-#include \"compat/msvc.h\"\n+#include \"compat/msvc/compat-util.h\"\n #else\n #include \n #include \ndiff --git a/meson.build b/meson.build\nindex 2297f5e9dd..32eb0a86d5 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -1054,7 +1054,7 @@ if host_machine.system() == 'cygwin'\n ]\n elif host_machine.system() == 'windows'\n libgit_sources += [\n- 'compat/mingw.c',\n+ 'compat/mingw/compat-util.c',\n 'compat/winansi.c',\n 'compat/win32/flush.c',\n 'compat/win32/path-utils.c',\n@@ -1157,7 +1157,7 @@ else\n error('Native regex support requested but not found')\n endif\n \n-# setitimer and friends are provided by compat/mingw.c.\n+# setitimer and friends are provided by compat/mingw/compat-util.c.\n if host_machine.system() != 'windows'\n if not compiler.compiles('''\n #include \n@@ -1237,7 +1237,7 @@ if not compiler.has_function('qsort')\n endif\n libgit_sources += 'compat/qsort_s.c'\n \n-# unsetenv is provided by compat/mingw.c.\n+# unsetenv is provided by compat/mingw/compat-util.c.\n if host_machine.system() != 'windows' and not compiler.has_function('unsetenv')\n libgit_c_args += '-DNO_UNSETENV'\n libgit_sources += 'compat/unsetenv.c'\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511334","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-17-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 17/20] compat/msvc: split out POSIX-related bits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:18Z","receivedAt":"2025-01-28T08:28:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Split out the POSIX-related bits from \"compat/msvc.h\". This is in\npreparation for splitting up \"git-compat-utils.h\" into a header that\nprovides POSIX-compatibility and a header that provides common wrappers\nused by the Git project.\n\nSigned-off-by: Patrick Steinhardt \n---\n compat/{msvc.c => msvc/compat-util.c} | 0\n compat/msvc/compat-util.h | 7 +++++++\n compat/{msvc.h => msvc/posix.h} | 8 ++++----\n config.mak.uname | 4 ++--\n meson.build | 4 +++-\n 5 files changed, 16 insertions(+), 7 deletions(-)\n\ndiff --git a/compat/msvc.c b/compat/msvc/compat-util.c\nsimilarity index 100%\nrename from compat/msvc.c\nrename to compat/msvc/compat-util.c\ndiff --git a/compat/msvc/compat-util.h b/compat/msvc/compat-util.h\nnew file mode 100644\nindex 0000000000..e409e9910c\n--- /dev/null\n+++ b/compat/msvc/compat-util.h\n@@ -0,0 +1,7 @@\n+#ifndef COMPAT_MSVC_COMPAT_UTIL_H\n+#define COMPAT_MSVC_COMPAT_UTIL_H\n+\n+#include \"compat/msvc/posix.h\"\n+#include \"compat/mingw/compat-util.h\"\n+\n+#endif /* COMPAT_MSVC_COMPAT_UTIL_H */\ndiff --git a/compat/msvc.h b/compat/msvc/posix.h\nsimilarity index 86%\nrename from compat/msvc.h\nrename to compat/msvc/posix.h\nindex df39978f4f..47062293cb 100644\n--- a/compat/msvc.h\n+++ b/compat/msvc/posix.h\n@@ -1,5 +1,5 @@\n-#ifndef __MSVC__HEAD\n-#define __MSVC__HEAD\n+#ifndef COMPAT_MSVC_POSIX_H\n+#define COMPAT_MSVC_POSIX_H\n \n #include \n #include \n@@ -28,6 +28,6 @@ typedef int sigset_t;\n /* open for reading, writing, or both (not in fcntl.h) */\n #define O_ACCMODE (_O_RDONLY | _O_WRONLY | _O_RDWR)\n \n-#include \"compat/mingw/compat-util.h\"\n+#include \"compat/mingw/posix.h\"\n \n-#endif\n+#endif /* COMPAT_MSVC_POSIX_H */\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 38a774de39..8407efd262 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -495,7 +495,7 @@ endif\n \tAR = compat/vcbuild/scripts/lib.pl\n \tCFLAGS =\n \tBASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE\n-\tCOMPAT_OBJS = compat/msvc.o compat/winansi.o \\\n+\tCOMPAT_OBJS = compat/msvc/compat-util.o compat/winansi.o \\\n \t\tcompat/win32/flush.o \\\n \t\tcompat/win32/path-utils.o \\\n \t\tcompat/win32/pthread.o compat/win32/syslog.o \\\n@@ -541,7 +541,7 @@ endif\n \n \tEXTRA_PROGRAMS += headless-git$X\n \n-compat/msvc.o: compat/msvc.c compat/mingw/compat-util.c GIT-CFLAGS\n+compat/msvc/compat-util.o: compat/msvc/compat-util.c compat/mingw/compat-util.c GIT-CFLAGS\n endif\n ifeq ($(uname_S),Interix)\n \tNO_INITGROUPS = YesPlease\ndiff --git a/meson.build b/meson.build\nindex 32eb0a86d5..e8abf798f5 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -1054,7 +1054,6 @@ if host_machine.system() == 'cygwin'\n ]\n elif host_machine.system() == 'windows'\n libgit_sources += [\n- 'compat/mingw/compat-util.c',\n 'compat/winansi.c',\n 'compat/win32/flush.c',\n 'compat/win32/path-utils.c',\n@@ -1081,6 +1080,9 @@ elif host_machine.system() == 'windows'\n libgit_include_directories += 'compat/win32'\n if compiler.get_id() == 'msvc'\n libgit_include_directories += 'compat/vcbuild/include'\n+ libgit_sources += 'compat/msvc/compat-util.c'\n+ else\n+ libgit_sources += 'compat/mingw/compat-util.c'\n endif\n endif\n \n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511336","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-18-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 18/20] git-compat-util.h: split out POSIX-emulating bits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:19Z","receivedAt":"2025-01-28T08:28:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"git-compat-util.h\" header is a treasure trove of various bits and\npieces used throughout the project. It basically mixes two different\nthings into one:\n\n - Providing a POSIX-like interface even on platforms that aren't\n POSIX-compliant.\n\n - Providing low-level functionality that is specific to Git.\n\nThis intermixing is a bit of a problem for the reftable library as we\ndon't want to recreate the POSIX-like interface there. But neither do we\nwant to pull in the Git-specific functionality, as it is otherwise quite\neasy to start depending on the Git codebase again.\n\nSplit out a new header \"compat/posix.h\" that only contains the bits and\npieces relevant for the emulation of POSIX, which we will start using in\nthe next commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n compat/posix.h | 541 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n git-compat-util.h | 531 +----------------------------------------------------\n 2 files changed, 543 insertions(+), 529 deletions(-)\n\ndiff --git a/compat/posix.h b/compat/posix.h\nnew file mode 100644\nindex 0000000000..2b2da97965\n--- /dev/null\n+++ b/compat/posix.h\n@@ -0,0 +1,541 @@\n+#ifndef COMPAT_POSIX_H\n+#define COMPAT_POSIX_H\n+\n+#define _FILE_OFFSET_BITS 64\n+\n+/*\n+ * Derived from Linux \"Features Test Macro\" header\n+ * Convenience macros to test the versions of gcc (or\n+ * a compatible compiler).\n+ * Use them like this:\n+ * #if GIT_GNUC_PREREQ (2,8)\n+ * ... code requiring gcc 2.8 or later ...\n+ * #endif\n+ *\n+ * This macro of course is not part of POSIX, but we need it for the UNUSED\n+ * macro which is used by some of our POSIX compatibility wrappers.\n+*/\n+#if defined(__GNUC__) && defined(__GNUC_MINOR__)\n+# define GIT_GNUC_PREREQ(maj, min) \\\n+\t((__GNUC__ << 16) + __GNUC_MINOR__ >= ((maj) << 16) + (min))\n+#else\n+ #define GIT_GNUC_PREREQ(maj, min) 0\n+#endif\n+\n+/*\n+ * UNUSED marks a function parameter that is always unused. It also\n+ * can be used to annotate a function, a variable, or a type that is\n+ * always unused.\n+ *\n+ * A callback interface may dictate that a function accepts a\n+ * parameter at that position, but the implementation of the function\n+ * may not need to use the parameter. In such a case, mark the parameter\n+ * with UNUSED.\n+ *\n+ * When a parameter may be used or unused, depending on conditional\n+ * compilation, consider using MAYBE_UNUSED instead.\n+ */\n+#if GIT_GNUC_PREREQ(4, 5)\n+#define UNUSED __attribute__((unused)) \\\n+\t__attribute__((deprecated (\"parameter declared as UNUSED\")))\n+#elif defined(__GNUC__)\n+#define UNUSED __attribute__((unused)) \\\n+\t__attribute__((deprecated))\n+#else\n+#define UNUSED\n+#endif\n+\n+#ifdef __MINGW64__\n+#define _POSIX_C_SOURCE 1\n+#elif defined(__sun__)\n+ /*\n+ * On Solaris, when _XOPEN_EXTENDED is set, its header file\n+ * forces the programs to be XPG4v2, defeating any _XOPEN_SOURCE\n+ * setting to say we are XPG5 or XPG6. Also on Solaris,\n+ * XPG6 programs must be compiled with a c99 compiler, while\n+ * non XPG6 programs must be compiled with a pre-c99 compiler.\n+ */\n+# if __STDC_VERSION__ - 0 >= 199901L\n+# define _XOPEN_SOURCE 600\n+# else\n+# define _XOPEN_SOURCE 500\n+# endif\n+#elif !defined(__APPLE__) && !defined(__FreeBSD__) && !defined(__USLC__) && \\\n+ !defined(_M_UNIX) && !defined(__sgi) && !defined(__DragonFly__) && \\\n+ !defined(__TANDEM) && !defined(__QNX__) && !defined(__MirBSD__) && \\\n+ !defined(__CYGWIN__)\n+#define _XOPEN_SOURCE 600 /* glibc2 and AIX 5.3L need 500, OpenBSD needs 600 for S_ISLNK() */\n+#define _XOPEN_SOURCE_EXTENDED 1 /* AIX 5.3L needs this */\n+#endif\n+#define _ALL_SOURCE 1\n+#define _GNU_SOURCE 1\n+#define _BSD_SOURCE 1\n+#define _DEFAULT_SOURCE 1\n+#define _NETBSD_SOURCE 1\n+#define _SGI_SOURCE 1\n+\n+#if defined(WIN32) && !defined(__CYGWIN__) /* Both MinGW and MSVC */\n+# if !defined(_WIN32_WINNT)\n+# define _WIN32_WINNT 0x0600\n+# endif\n+#define WIN32_LEAN_AND_MEAN /* stops windows.h including winsock.h */\n+#include \n+#ifndef NO_UNIX_SOCKETS\n+#include \n+#endif\n+#include \n+#define GIT_WINDOWS_NATIVE\n+#endif\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#ifdef HAVE_STRINGS_H\n+#include /* for strcasecmp() */\n+#endif\n+#include \n+#include \n+#include \n+#ifdef NEEDS_SYS_PARAM_H\n+#include \n+#endif\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#if !defined(NO_POLL_H)\n+#include \n+#elif !defined(NO_SYS_POLL_H)\n+#include \n+#else\n+/* Pull the compat stuff */\n+#include \n+#endif\n+#ifdef HAVE_BSD_SYSCTL\n+#include \n+#endif\n+\n+#if defined(__MINGW32__)\n+#include \"compat/mingw/posix.h\"\n+#elif defined(_MSC_VER)\n+#include \"compat/msvc/posix.h\"\n+#else\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#ifndef NO_SYS_SELECT_H\n+#include \n+#endif\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#ifndef NO_INTTYPES_H\n+#include \n+#else\n+#include \n+#endif\n+#ifdef HAVE_ARC4RANDOM_LIBBSD\n+#include \n+#endif\n+#ifdef HAVE_GETRANDOM\n+#include \n+#endif\n+#ifdef NO_INTPTR_T\n+/*\n+ * On I16LP32, ILP32 and LP64 \"long\" is the safe bet, however\n+ * on LLP86, IL33LLP64 and P64 it needs to be \"long long\",\n+ * while on IP16 and IP16L32 it is \"int\" (resp. \"short\")\n+ * Size needs to match (or exceed) 'sizeof(void *)'.\n+ * We can't take \"long long\" here as not everybody has it.\n+ */\n+typedef long intptr_t;\n+typedef unsigned long uintptr_t;\n+#endif\n+#undef _ALL_SOURCE /* AIX 5.3L defines a struct list with _ALL_SOURCE. */\n+#include \n+#define _ALL_SOURCE 1\n+#endif\n+\n+#ifdef MKDIR_WO_TRAILING_SLASH\n+#define mkdir(a,b) compat_mkdir_wo_trailing_slash((a),(b))\n+int compat_mkdir_wo_trailing_slash(const char*, mode_t);\n+#endif\n+\n+#ifdef time\n+#undef time\n+#endif\n+static inline time_t git_time(time_t *tloc)\n+{\n+\tstruct timeval tv;\n+\n+\t/*\n+\t * Avoid time(NULL), which can disagree with gettimeofday(2)\n+\t * and filesystem timestamps.\n+\t */\n+\tgettimeofday(&tv, NULL);\n+\n+\tif (tloc)\n+\t\t*tloc = tv.tv_sec;\n+\treturn tv.tv_sec;\n+}\n+#define time git_time\n+\n+#ifdef NO_STRUCT_ITIMERVAL\n+struct itimerval {\n+\tstruct timeval it_interval;\n+\tstruct timeval it_value;\n+};\n+#endif\n+\n+#ifdef NO_SETITIMER\n+static inline int git_setitimer(int which UNUSED,\n+\t\t\t\tconst struct itimerval *value UNUSED,\n+\t\t\t\tstruct itimerval *newvalue UNUSED) {\n+\treturn 0; /* pretend success */\n+}\n+#undef setitimer\n+#define setitimer(which,value,ovalue) git_setitimer(which,value,ovalue)\n+#endif\n+\n+#ifndef NO_LIBGEN_H\n+#include \n+#else\n+#define basename gitbasename\n+char *gitbasename(char *);\n+#define dirname gitdirname\n+char *gitdirname(char *);\n+#endif\n+\n+#ifndef NO_ICONV\n+#include \n+#endif\n+\n+/* On most systems would have given us this, but\n+ * not on some systems (e.g. z/OS).\n+ */\n+#ifndef NI_MAXHOST\n+#define NI_MAXHOST 1025\n+#endif\n+\n+#ifndef NI_MAXSERV\n+#define NI_MAXSERV 32\n+#endif\n+\n+/* On most systems would have given us this, but\n+ * not on some systems (e.g. GNU/Hurd).\n+ */\n+#ifndef PATH_MAX\n+#define PATH_MAX 4096\n+#endif\n+\n+#ifndef NAME_MAX\n+#define NAME_MAX 255\n+#endif\n+\n+typedef uintmax_t timestamp_t;\n+#define PRItime PRIuMAX\n+#define parse_timestamp strtoumax\n+#define TIME_MAX UINTMAX_MAX\n+#define TIME_MIN 0\n+\n+int lstat_cache_aware_rmdir(const char *path);\n+#if !defined(__MINGW32__) && !defined(_MSC_VER)\n+#define rmdir lstat_cache_aware_rmdir\n+#endif\n+\n+#if defined(NO_MMAP) || defined(USE_WIN32_MMAP)\n+\n+#ifndef PROT_READ\n+#define PROT_READ 1\n+#define PROT_WRITE 2\n+#define MAP_PRIVATE 1\n+#endif\n+\n+#define mmap git_mmap\n+#define munmap git_munmap\n+void *git_mmap(void *start, size_t length, int prot, int flags, int fd, off_t offset);\n+int git_munmap(void *start, size_t length);\n+\n+#else /* NO_MMAP || USE_WIN32_MMAP */\n+\n+#include \n+\n+#endif /* NO_MMAP || USE_WIN32_MMAP */\n+\n+#ifndef MAP_FAILED\n+#define MAP_FAILED ((void *)-1)\n+#endif\n+\n+#ifdef NEEDS_MODE_TRANSLATION\n+#undef S_IFMT\n+#undef S_IFREG\n+#undef S_IFDIR\n+#undef S_IFLNK\n+#undef S_IFBLK\n+#undef S_IFCHR\n+#undef S_IFIFO\n+#undef S_IFSOCK\n+#define S_IFMT 0170000\n+#define S_IFREG 0100000\n+#define S_IFDIR 0040000\n+#define S_IFLNK 0120000\n+#define S_IFBLK 0060000\n+#define S_IFCHR 0020000\n+#define S_IFIFO 0010000\n+#define S_IFSOCK 0140000\n+#ifdef stat\n+#undef stat\n+#endif\n+#define stat(path, buf) git_stat(path, buf)\n+int git_stat(const char *, struct stat *);\n+#ifdef fstat\n+#undef fstat\n+#endif\n+#define fstat(fd, buf) git_fstat(fd, buf)\n+int git_fstat(int, struct stat *);\n+#ifdef lstat\n+#undef lstat\n+#endif\n+#define lstat(path, buf) git_lstat(path, buf)\n+int git_lstat(const char *, struct stat *);\n+#endif\n+\n+#ifdef NO_PREAD\n+#define pread git_pread\n+ssize_t git_pread(int fd, void *buf, size_t count, off_t offset);\n+#endif\n+\n+#ifdef NO_SETENV\n+#define setenv gitsetenv\n+int gitsetenv(const char *, const char *, int);\n+#endif\n+\n+#ifdef NO_MKDTEMP\n+#define mkdtemp gitmkdtemp\n+char *gitmkdtemp(char *);\n+#endif\n+\n+#ifdef NO_UNSETENV\n+#define unsetenv gitunsetenv\n+int gitunsetenv(const char *);\n+#endif\n+\n+#ifdef NO_STRCASESTR\n+#define strcasestr gitstrcasestr\n+char *gitstrcasestr(const char *haystack, const char *needle);\n+#endif\n+\n+#ifdef NO_STRLCPY\n+#define strlcpy gitstrlcpy\n+size_t gitstrlcpy(char *, const char *, size_t);\n+#endif\n+\n+#ifdef NO_STRTOUMAX\n+#define strtoumax gitstrtoumax\n+uintmax_t gitstrtoumax(const char *, char **, int);\n+#define strtoimax gitstrtoimax\n+intmax_t gitstrtoimax(const char *, char **, int);\n+#endif\n+\n+#ifdef NO_HSTRERROR\n+#define hstrerror githstrerror\n+const char *githstrerror(int herror);\n+#endif\n+\n+#ifdef NO_MEMMEM\n+#define memmem gitmemmem\n+void *gitmemmem(const void *haystack, size_t haystacklen,\n+\t\tconst void *needle, size_t needlelen);\n+#endif\n+\n+#ifdef OVERRIDE_STRDUP\n+#ifdef strdup\n+#undef strdup\n+#endif\n+#define strdup gitstrdup\n+char *gitstrdup(const char *s);\n+#endif\n+\n+#ifdef NO_GETPAGESIZE\n+#define getpagesize() sysconf(_SC_PAGESIZE)\n+#endif\n+\n+#ifndef O_CLOEXEC\n+#define O_CLOEXEC 0\n+#endif\n+\n+#ifdef FREAD_READS_DIRECTORIES\n+# if !defined(SUPPRESS_FOPEN_REDEFINITION)\n+# ifdef fopen\n+# undef fopen\n+# endif\n+# define fopen(a,b) git_fopen(a,b)\n+# endif\n+FILE *git_fopen(const char*, const char*);\n+#endif\n+\n+#ifdef SNPRINTF_RETURNS_BOGUS\n+#ifdef snprintf\n+#undef snprintf\n+#endif\n+#define snprintf git_snprintf\n+int git_snprintf(char *str, size_t maxsize,\n+\t\t const char *format, ...);\n+#ifdef vsnprintf\n+#undef vsnprintf\n+#endif\n+#define vsnprintf git_vsnprintf\n+int git_vsnprintf(char *str, size_t maxsize,\n+\t\t const char *format, va_list ap);\n+#endif\n+\n+#ifdef OPEN_RETURNS_EINTR\n+#undef open\n+#define open git_open_with_retry\n+int git_open_with_retry(const char *path, int flag, ...);\n+#endif\n+\n+#ifdef __GLIBC_PREREQ\n+#if __GLIBC_PREREQ(2, 1)\n+#define HAVE_STRCHRNUL\n+#endif\n+#endif\n+\n+#ifndef HAVE_STRCHRNUL\n+#define strchrnul gitstrchrnul\n+static inline char *gitstrchrnul(const char *s, int c)\n+{\n+\twhile (*s && *s != c)\n+\t\ts++;\n+\treturn (char *)s;\n+}\n+#endif\n+\n+#ifdef NO_INET_PTON\n+int inet_pton(int af, const char *src, void *dst);\n+#endif\n+\n+#ifdef NO_INET_NTOP\n+const char *inet_ntop(int af, const void *src, char *dst, size_t size);\n+#endif\n+\n+#ifdef NO_PTHREADS\n+#define atexit git_atexit\n+int git_atexit(void (*handler)(void));\n+#endif\n+\n+#ifndef HOST_NAME_MAX\n+#define HOST_NAME_MAX 256\n+#endif\n+\n+#include \"sane-ctype.h\"\n+\n+void git_stable_qsort(void *base, size_t nmemb, size_t size,\n+\t\t int(*compar)(const void *, const void *));\n+#ifdef INTERNAL_QSORT\n+#define qsort git_stable_qsort\n+#endif\n+\n+#define QSORT(base, n, compar) sane_qsort((base), (n), sizeof(*(base)), compar)\n+static inline void sane_qsort(void *base, size_t nmemb, size_t size,\n+\t\t\t int(*compar)(const void *, const void *))\n+{\n+\tif (nmemb > 1)\n+\t\tqsort(base, nmemb, size, compar);\n+}\n+\n+#define STABLE_QSORT(base, n, compar) \\\n+\tgit_stable_qsort((base), (n), sizeof(*(base)), compar)\n+\n+#ifndef HAVE_ISO_QSORT_S\n+int git_qsort_s(void *base, size_t nmemb, size_t size,\n+\t\tint (*compar)(const void *, const void *, void *), void *ctx);\n+#define qsort_s git_qsort_s\n+#endif\n+\n+#define QSORT_S(base, n, compar, ctx) do {\t\t\t\\\n+\tif (qsort_s((base), (n), sizeof(*(base)), compar, ctx))\t\\\n+\t\tBUG(\"qsort_s() failed\");\t\t\t\\\n+} while (0)\n+\n+#ifdef NO_NSEC\n+#undef USE_NSEC\n+#define ST_CTIME_NSEC(st) 0\n+#define ST_MTIME_NSEC(st) 0\n+#else\n+#ifdef USE_ST_TIMESPEC\n+#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctimespec.tv_nsec))\n+#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtimespec.tv_nsec))\n+#else\n+#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctim.tv_nsec))\n+#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtim.tv_nsec))\n+#endif\n+#endif\n+\n+#ifndef va_copy\n+/*\n+ * Since an obvious implementation of va_list would be to make it a\n+ * pointer into the stack frame, a simple assignment will work on\n+ * many systems. But let's try to be more portable.\n+ */\n+#ifdef __va_copy\n+#define va_copy(dst, src) __va_copy(dst, src)\n+#else\n+#define va_copy(dst, src) ((dst) = (src))\n+#endif\n+#endif\n+\n+#ifndef _POSIX_THREAD_SAFE_FUNCTIONS\n+static inline void git_flockfile(FILE *fh UNUSED)\n+{\n+\t; /* nothing */\n+}\n+static inline void git_funlockfile(FILE *fh UNUSED)\n+{\n+\t; /* nothing */\n+}\n+#undef flockfile\n+#undef funlockfile\n+#undef getc_unlocked\n+#define flockfile(fh) git_flockfile(fh)\n+#define funlockfile(fh) git_funlockfile(fh)\n+#define getc_unlocked(fh) getc(fh)\n+#endif\n+\n+#ifdef FILENO_IS_A_MACRO\n+int git_fileno(FILE *stream);\n+# ifndef COMPAT_CODE_FILENO\n+# undef fileno\n+# define fileno(p) git_fileno(p)\n+# endif\n+#endif\n+\n+#ifdef NEED_ACCESS_ROOT_HANDLER\n+int git_access(const char *path, int mode);\n+# ifndef COMPAT_CODE_ACCESS\n+# ifdef access\n+# undef access\n+# endif\n+# define access(path, mode) git_access(path, mode)\n+# endif\n+#endif\n+\n+#endif /* COMPAT_POSIX_H */\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex b96fb98e1e..30ad0725fb 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -23,26 +23,9 @@\n #include \n #endif\n \n-struct strbuf;\n-\n-\n-#define _FILE_OFFSET_BITS 64\n+#include \"compat/posix.h\"\n \n-\n-/* Derived from Linux \"Features Test Macro\" header\n- * Convenience macros to test the versions of gcc (or\n- * a compatible compiler).\n- * Use them like this:\n- * #if GIT_GNUC_PREREQ (2,8)\n- * ... code requiring gcc 2.8 or later ...\n- * #endif\n-*/\n-#if defined(__GNUC__) && defined(__GNUC_MINOR__)\n-# define GIT_GNUC_PREREQ(maj, min) \\\n-\t((__GNUC__ << 16) + __GNUC_MINOR__ >= ((maj) << 16) + (min))\n-#else\n- #define GIT_GNUC_PREREQ(maj, min) 0\n-#endif\n+struct strbuf;\n \n #if defined(__GNUC__) || defined(__clang__)\n # define PRAGMA(pragma) _Pragma(#pragma)\n@@ -176,71 +159,6 @@ DISABLE_WARNING(-Wsign-compare)\n /* Approximation of the length of the decimal representation of this type. */\n #define decimal_length(x)\t((int)(sizeof(x) * 2.56 + 0.5) + 1)\n \n-#ifdef __MINGW64__\n-#define _POSIX_C_SOURCE 1\n-#elif defined(__sun__)\n- /*\n- * On Solaris, when _XOPEN_EXTENDED is set, its header file\n- * forces the programs to be XPG4v2, defeating any _XOPEN_SOURCE\n- * setting to say we are XPG5 or XPG6. Also on Solaris,\n- * XPG6 programs must be compiled with a c99 compiler, while\n- * non XPG6 programs must be compiled with a pre-c99 compiler.\n- */\n-# if __STDC_VERSION__ - 0 >= 199901L\n-# define _XOPEN_SOURCE 600\n-# else\n-# define _XOPEN_SOURCE 500\n-# endif\n-#elif !defined(__APPLE__) && !defined(__FreeBSD__) && !defined(__USLC__) && \\\n- !defined(_M_UNIX) && !defined(__sgi) && !defined(__DragonFly__) && \\\n- !defined(__TANDEM) && !defined(__QNX__) && !defined(__MirBSD__) && \\\n- !defined(__CYGWIN__)\n-#define _XOPEN_SOURCE 600 /* glibc2 and AIX 5.3L need 500, OpenBSD needs 600 for S_ISLNK() */\n-#define _XOPEN_SOURCE_EXTENDED 1 /* AIX 5.3L needs this */\n-#endif\n-#define _ALL_SOURCE 1\n-#define _GNU_SOURCE 1\n-#define _BSD_SOURCE 1\n-#define _DEFAULT_SOURCE 1\n-#define _NETBSD_SOURCE 1\n-#define _SGI_SOURCE 1\n-\n-/*\n- * UNUSED marks a function parameter that is always unused. It also\n- * can be used to annotate a function, a variable, or a type that is\n- * always unused.\n- *\n- * A callback interface may dictate that a function accepts a\n- * parameter at that position, but the implementation of the function\n- * may not need to use the parameter. In such a case, mark the parameter\n- * with UNUSED.\n- *\n- * When a parameter may be used or unused, depending on conditional\n- * compilation, consider using MAYBE_UNUSED instead.\n- */\n-#if GIT_GNUC_PREREQ(4, 5)\n-#define UNUSED __attribute__((unused)) \\\n-\t__attribute__((deprecated (\"parameter declared as UNUSED\")))\n-#elif defined(__GNUC__)\n-#define UNUSED __attribute__((unused)) \\\n-\t__attribute__((deprecated))\n-#else\n-#define UNUSED\n-#endif\n-\n-#if defined(WIN32) && !defined(__CYGWIN__) /* Both MinGW and MSVC */\n-# if !defined(_WIN32_WINNT)\n-# define _WIN32_WINNT 0x0600\n-# endif\n-#define WIN32_LEAN_AND_MEAN /* stops windows.h including winsock.h */\n-#include \n-#ifndef NO_UNIX_SOCKETS\n-#include \n-#endif\n-#include \n-#define GIT_WINDOWS_NATIVE\n-#endif\n-\n #if defined(NO_UNIX_SOCKETS) || !defined(GIT_WINDOWS_NATIVE)\n static inline int _have_unix_sockets(void)\n {\n@@ -253,45 +171,6 @@ static inline int _have_unix_sockets(void)\n #define have_unix_sockets _have_unix_sockets\n #endif\n \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#ifdef HAVE_STRINGS_H\n-#include /* for strcasecmp() */\n-#endif\n-#include \n-#include \n-#include \n-#ifdef NEEDS_SYS_PARAM_H\n-#include \n-#endif\n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#if !defined(NO_POLL_H)\n-#include \n-#elif !defined(NO_SYS_POLL_H)\n-#include \n-#else\n-/* Pull the compat stuff */\n-#include \n-#endif\n-#ifdef HAVE_BSD_SYSCTL\n-#include \n-#endif\n-\n /* Used by compat/win32/path-utils.h, and more */\n static inline int is_xplatform_dir_sep(int c)\n {\n@@ -308,48 +187,6 @@ static inline int is_xplatform_dir_sep(int c)\n #elif defined(_MSC_VER)\n #include \"compat/win32/path-utils.h\"\n #include \"compat/msvc/compat-util.h\"\n-#else\n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#ifndef NO_SYS_SELECT_H\n-#include \n-#endif\n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#ifndef NO_INTTYPES_H\n-#include \n-#else\n-#include \n-#endif\n-#ifdef HAVE_ARC4RANDOM_LIBBSD\n-#include \n-#endif\n-#ifdef HAVE_GETRANDOM\n-#include \n-#endif\n-#ifdef NO_INTPTR_T\n-/*\n- * On I16LP32, ILP32 and LP64 \"long\" is the safe bet, however\n- * on LLP86, IL33LLP64 and P64 it needs to be \"long long\",\n- * while on IP16 and IP16L32 it is \"int\" (resp. \"short\")\n- * Size needs to match (or exceed) 'sizeof(void *)'.\n- * We can't take \"long long\" here as not everybody has it.\n- */\n-typedef long intptr_t;\n-typedef unsigned long uintptr_t;\n-#endif\n-#undef _ALL_SOURCE /* AIX 5.3L defines a struct list with _ALL_SOURCE. */\n-#include \n-#define _ALL_SOURCE 1\n #endif\n \n /* used on Mac OS X */\n@@ -370,60 +207,6 @@ static inline const char *precompose_string_if_needed(const char *in)\n #define probe_utf8_pathname_composition()\n #endif\n \n-#ifdef MKDIR_WO_TRAILING_SLASH\n-#define mkdir(a,b) compat_mkdir_wo_trailing_slash((a),(b))\n-int compat_mkdir_wo_trailing_slash(const char*, mode_t);\n-#endif\n-\n-#ifdef time\n-#undef time\n-#endif\n-static inline time_t git_time(time_t *tloc)\n-{\n-\tstruct timeval tv;\n-\n-\t/*\n-\t * Avoid time(NULL), which can disagree with gettimeofday(2)\n-\t * and filesystem timestamps.\n-\t */\n-\tgettimeofday(&tv, NULL);\n-\n-\tif (tloc)\n-\t\t*tloc = tv.tv_sec;\n-\treturn tv.tv_sec;\n-}\n-#define time git_time\n-\n-#ifdef NO_STRUCT_ITIMERVAL\n-struct itimerval {\n-\tstruct timeval it_interval;\n-\tstruct timeval it_value;\n-};\n-#endif\n-\n-#ifdef NO_SETITIMER\n-static inline int git_setitimer(int which UNUSED,\n-\t\t\t\tconst struct itimerval *value UNUSED,\n-\t\t\t\tstruct itimerval *newvalue UNUSED) {\n-\treturn 0; /* pretend success */\n-}\n-#undef setitimer\n-#define setitimer(which,value,ovalue) git_setitimer(which,value,ovalue)\n-#endif\n-\n-#ifndef NO_LIBGEN_H\n-#include \n-#else\n-#define basename gitbasename\n-char *gitbasename(char *);\n-#define dirname gitdirname\n-char *gitdirname(char *);\n-#endif\n-\n-#ifndef NO_ICONV\n-#include \n-#endif\n-\n #ifndef NO_OPENSSL\n #ifdef __APPLE__\n #undef __AVAILABILITY_MACROS_USES_AVAILABILITY\n@@ -441,34 +224,6 @@ char *gitdirname(char *);\n # include \n #endif\n \n-/* On most systems would have given us this, but\n- * not on some systems (e.g. z/OS).\n- */\n-#ifndef NI_MAXHOST\n-#define NI_MAXHOST 1025\n-#endif\n-\n-#ifndef NI_MAXSERV\n-#define NI_MAXSERV 32\n-#endif\n-\n-/* On most systems would have given us this, but\n- * not on some systems (e.g. GNU/Hurd).\n- */\n-#ifndef PATH_MAX\n-#define PATH_MAX 4096\n-#endif\n-\n-#ifndef NAME_MAX\n-#define NAME_MAX 255\n-#endif\n-\n-typedef uintmax_t timestamp_t;\n-#define PRItime PRIuMAX\n-#define parse_timestamp strtoumax\n-#define TIME_MAX UINTMAX_MAX\n-#define TIME_MIN 0\n-\n #ifndef PATH_SEP\n #define PATH_SEP ':'\n #endif\n@@ -492,11 +247,6 @@ static inline int noop_core_config(const char *var UNUSED,\n #define platform_core_config noop_core_config\n #endif\n \n-int lstat_cache_aware_rmdir(const char *path);\n-#if !defined(__MINGW32__) && !defined(_MSC_VER)\n-#define rmdir lstat_cache_aware_rmdir\n-#endif\n-\n #ifndef has_dos_drive_prefix\n static inline int git_has_dos_drive_prefix(const char *path UNUSED)\n {\n@@ -822,25 +572,6 @@ static inline bool strip_suffix(const char *str, const char *suffix,\n \tmemcpy(_swap_b_ptr, _swap_buffer, sizeof(a));\t\t\\\n } while (0)\n \n-#if defined(NO_MMAP) || defined(USE_WIN32_MMAP)\n-\n-#ifndef PROT_READ\n-#define PROT_READ 1\n-#define PROT_WRITE 2\n-#define MAP_PRIVATE 1\n-#endif\n-\n-#define mmap git_mmap\n-#define munmap git_munmap\n-void *git_mmap(void *start, size_t length, int prot, int flags, int fd, off_t offset);\n-int git_munmap(void *start, size_t length);\n-\n-#else /* NO_MMAP || USE_WIN32_MMAP */\n-\n-#include \n-\n-#endif /* NO_MMAP || USE_WIN32_MMAP */\n-\n #ifdef NO_MMAP\n \n /* This value must be multiple of (pagesize * 2) */\n@@ -856,177 +587,15 @@ int git_munmap(void *start, size_t length);\n \n #endif /* NO_MMAP */\n \n-#ifndef MAP_FAILED\n-#define MAP_FAILED ((void *)-1)\n-#endif\n-\n #ifdef NO_ST_BLOCKS_IN_STRUCT_STAT\n #define on_disk_bytes(st) ((st).st_size)\n #else\n #define on_disk_bytes(st) ((st).st_blocks * 512)\n #endif\n \n-#ifdef NEEDS_MODE_TRANSLATION\n-#undef S_IFMT\n-#undef S_IFREG\n-#undef S_IFDIR\n-#undef S_IFLNK\n-#undef S_IFBLK\n-#undef S_IFCHR\n-#undef S_IFIFO\n-#undef S_IFSOCK\n-#define S_IFMT 0170000\n-#define S_IFREG 0100000\n-#define S_IFDIR 0040000\n-#define S_IFLNK 0120000\n-#define S_IFBLK 0060000\n-#define S_IFCHR 0020000\n-#define S_IFIFO 0010000\n-#define S_IFSOCK 0140000\n-#ifdef stat\n-#undef stat\n-#endif\n-#define stat(path, buf) git_stat(path, buf)\n-int git_stat(const char *, struct stat *);\n-#ifdef fstat\n-#undef fstat\n-#endif\n-#define fstat(fd, buf) git_fstat(fd, buf)\n-int git_fstat(int, struct stat *);\n-#ifdef lstat\n-#undef lstat\n-#endif\n-#define lstat(path, buf) git_lstat(path, buf)\n-int git_lstat(const char *, struct stat *);\n-#endif\n-\n #define DEFAULT_PACKED_GIT_LIMIT \\\n \t((1024L * 1024L) * (size_t)(sizeof(void*) >= 8 ? (32 * 1024L * 1024L) : 256))\n \n-#ifdef NO_PREAD\n-#define pread git_pread\n-ssize_t git_pread(int fd, void *buf, size_t count, off_t offset);\n-#endif\n-\n-#ifdef NO_SETENV\n-#define setenv gitsetenv\n-int gitsetenv(const char *, const char *, int);\n-#endif\n-\n-#ifdef NO_MKDTEMP\n-#define mkdtemp gitmkdtemp\n-char *gitmkdtemp(char *);\n-#endif\n-\n-#ifdef NO_UNSETENV\n-#define unsetenv gitunsetenv\n-int gitunsetenv(const char *);\n-#endif\n-\n-#ifdef NO_STRCASESTR\n-#define strcasestr gitstrcasestr\n-char *gitstrcasestr(const char *haystack, const char *needle);\n-#endif\n-\n-#ifdef NO_STRLCPY\n-#define strlcpy gitstrlcpy\n-size_t gitstrlcpy(char *, const char *, size_t);\n-#endif\n-\n-#ifdef NO_STRTOUMAX\n-#define strtoumax gitstrtoumax\n-uintmax_t gitstrtoumax(const char *, char **, int);\n-#define strtoimax gitstrtoimax\n-intmax_t gitstrtoimax(const char *, char **, int);\n-#endif\n-\n-#ifdef NO_HSTRERROR\n-#define hstrerror githstrerror\n-const char *githstrerror(int herror);\n-#endif\n-\n-#ifdef NO_MEMMEM\n-#define memmem gitmemmem\n-void *gitmemmem(const void *haystack, size_t haystacklen,\n-\t\tconst void *needle, size_t needlelen);\n-#endif\n-\n-#ifdef OVERRIDE_STRDUP\n-#ifdef strdup\n-#undef strdup\n-#endif\n-#define strdup gitstrdup\n-char *gitstrdup(const char *s);\n-#endif\n-\n-#ifdef NO_GETPAGESIZE\n-#define getpagesize() sysconf(_SC_PAGESIZE)\n-#endif\n-\n-#ifndef O_CLOEXEC\n-#define O_CLOEXEC 0\n-#endif\n-\n-#ifdef FREAD_READS_DIRECTORIES\n-# if !defined(SUPPRESS_FOPEN_REDEFINITION)\n-# ifdef fopen\n-# undef fopen\n-# endif\n-# define fopen(a,b) git_fopen(a,b)\n-# endif\n-FILE *git_fopen(const char*, const char*);\n-#endif\n-\n-#ifdef SNPRINTF_RETURNS_BOGUS\n-#ifdef snprintf\n-#undef snprintf\n-#endif\n-#define snprintf git_snprintf\n-int git_snprintf(char *str, size_t maxsize,\n-\t\t const char *format, ...);\n-#ifdef vsnprintf\n-#undef vsnprintf\n-#endif\n-#define vsnprintf git_vsnprintf\n-int git_vsnprintf(char *str, size_t maxsize,\n-\t\t const char *format, va_list ap);\n-#endif\n-\n-#ifdef OPEN_RETURNS_EINTR\n-#undef open\n-#define open git_open_with_retry\n-int git_open_with_retry(const char *path, int flag, ...);\n-#endif\n-\n-#ifdef __GLIBC_PREREQ\n-#if __GLIBC_PREREQ(2, 1)\n-#define HAVE_STRCHRNUL\n-#endif\n-#endif\n-\n-#ifndef HAVE_STRCHRNUL\n-#define strchrnul gitstrchrnul\n-static inline char *gitstrchrnul(const char *s, int c)\n-{\n-\twhile (*s && *s != c)\n-\t\ts++;\n-\treturn (char *)s;\n-}\n-#endif\n-\n-#ifdef NO_INET_PTON\n-int inet_pton(int af, const char *src, void *dst);\n-#endif\n-\n-#ifdef NO_INET_NTOP\n-const char *inet_ntop(int af, const void *src, char *dst, size_t size);\n-#endif\n-\n-#ifdef NO_PTHREADS\n-#define atexit git_atexit\n-int git_atexit(void (*handler)(void));\n-#endif\n-\n static inline size_t st_add(size_t a, size_t b)\n {\n \tif (unsigned_add_overflows(a, b))\n@@ -1293,12 +862,6 @@ static inline size_t xsize_t(off_t len)\n \treturn (size_t) len;\n }\n \n-#ifndef HOST_NAME_MAX\n-#define HOST_NAME_MAX 256\n-#endif\n-\n-#include \"sane-ctype.h\"\n-\n /*\n * Like skip_prefix, but compare case-insensitively. Note that the comparison\n * is done via tolower(), so it is strictly ASCII (no multi-byte characters or\n@@ -1364,34 +927,6 @@ static inline int strtol_i(char const *s, int base, int *result)\n \treturn 0;\n }\n \n-void git_stable_qsort(void *base, size_t nmemb, size_t size,\n-\t\t int(*compar)(const void *, const void *));\n-#ifdef INTERNAL_QSORT\n-#define qsort git_stable_qsort\n-#endif\n-\n-#define QSORT(base, n, compar) sane_qsort((base), (n), sizeof(*(base)), compar)\n-static inline void sane_qsort(void *base, size_t nmemb, size_t size,\n-\t\t\t int(*compar)(const void *, const void *))\n-{\n-\tif (nmemb > 1)\n-\t\tqsort(base, nmemb, size, compar);\n-}\n-\n-#define STABLE_QSORT(base, n, compar) \\\n-\tgit_stable_qsort((base), (n), sizeof(*(base)), compar)\n-\n-#ifndef HAVE_ISO_QSORT_S\n-int git_qsort_s(void *base, size_t nmemb, size_t size,\n-\t\tint (*compar)(const void *, const void *, void *), void *ctx);\n-#define qsort_s git_qsort_s\n-#endif\n-\n-#define QSORT_S(base, n, compar, ctx) do {\t\t\t\\\n-\tif (qsort_s((base), (n), sizeof(*(base)), compar, ctx))\t\\\n-\t\tBUG(\"qsort_s() failed\");\t\t\t\\\n-} while (0)\n-\n #ifndef REG_STARTEND\n #error \"Git requires REG_STARTEND support. Compile with NO_REGEX=NeedsStartEnd\"\n #endif\n@@ -1416,39 +951,12 @@ int git_regcomp(regex_t *preg, const char *pattern, int cflags);\n # define FORCE_DIR_SET_GID 0\n #endif\n \n-#ifdef NO_NSEC\n-#undef USE_NSEC\n-#define ST_CTIME_NSEC(st) 0\n-#define ST_MTIME_NSEC(st) 0\n-#else\n-#ifdef USE_ST_TIMESPEC\n-#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctimespec.tv_nsec))\n-#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtimespec.tv_nsec))\n-#else\n-#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctim.tv_nsec))\n-#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtim.tv_nsec))\n-#endif\n-#endif\n-\n #ifdef UNRELIABLE_FSTAT\n #define fstat_is_reliable() 0\n #else\n #define fstat_is_reliable() 1\n #endif\n \n-#ifndef va_copy\n-/*\n- * Since an obvious implementation of va_list would be to make it a\n- * pointer into the stack frame, a simple assignment will work on\n- * many systems. But let's try to be more portable.\n- */\n-#ifdef __va_copy\n-#define va_copy(dst, src) __va_copy(dst, src)\n-#else\n-#define va_copy(dst, src) ((dst) = (src))\n-#endif\n-#endif\n-\n /* usage.c: only to be used for testing BUG() implementation (see test-tool) */\n extern int BUG_exit_code;\n \n@@ -1478,41 +986,6 @@ void bug_fl(const char *file, int line, const char *fmt, ...);\n # define SHELL_PATH \"/bin/sh\"\n #endif\n \n-#ifndef _POSIX_THREAD_SAFE_FUNCTIONS\n-static inline void git_flockfile(FILE *fh UNUSED)\n-{\n-\t; /* nothing */\n-}\n-static inline void git_funlockfile(FILE *fh UNUSED)\n-{\n-\t; /* nothing */\n-}\n-#undef flockfile\n-#undef funlockfile\n-#undef getc_unlocked\n-#define flockfile(fh) git_flockfile(fh)\n-#define funlockfile(fh) git_funlockfile(fh)\n-#define getc_unlocked(fh) getc(fh)\n-#endif\n-\n-#ifdef FILENO_IS_A_MACRO\n-int git_fileno(FILE *stream);\n-# ifndef COMPAT_CODE_FILENO\n-# undef fileno\n-# define fileno(p) git_fileno(p)\n-# endif\n-#endif\n-\n-#ifdef NEED_ACCESS_ROOT_HANDLER\n-int git_access(const char *path, int mode);\n-# ifndef COMPAT_CODE_ACCESS\n-# ifdef access\n-# undef access\n-# endif\n-# define access(path, mode) git_access(path, mode)\n-# endif\n-#endif\n-\n /*\n * Our code often opens a path to an optional file, to work on its\n * contents when we can successfully open it. We can ignore a failure\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511335","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-19-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 19/20] reftable: decouple from Git codebase by pulling in \"compat/posix.h\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:20Z","receivedAt":"2025-01-28T08:28:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library includes \"git-compat-util.h\" in order to get a\nPOSIX-like programming environment that papers over various differences\nbetween platforms. The header also brings with it a couple of helpers\nspecific to the Git codebase though, and over time we have started to\nuse these helpers in the reftable library, as well.\n\nThis makes it very hard to use the reftable library as a standalone\nlibrary without the rest of the Git codebase, so other libraries like\ne.g. libgit2 cannot easily use it. But now that we have removed all\ncalls to Git-specific functionality and have split out \"compat/posix.h\"\nas a separate header we can address this.\n\nStop including \"git-compat-util.h\" and instead include \"compat/posix.h\"\nto finalize the decoupling of the reftable library from the rest of the\nGit codebase. The only bits which remain specific to Git are \"system.h\"\nand \"system.c\", which projects will have to provide.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/system.c | 2 ++\n reftable/system.h | 3 ++-\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/reftable/system.c b/reftable/system.c\nindex e25ccc0da3..1ee268b125 100644\n--- a/reftable/system.c\n+++ b/reftable/system.c\n@@ -1,3 +1,5 @@\n+#include \"../git-compat-util.h\"\n+\n #include \"system.h\"\n #include \"basics.h\"\n #include \"reftable-error.h\"\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 3bd4a4e322..e4a8944a70 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -11,7 +11,8 @@ license that can be found in the LICENSE file or at\n \n /* This header glues the reftable library to the rest of Git */\n \n-#include \"git-compat-util.h\"\n+#include \"compat/posix.h\"\n+#include \n \n /*\n * Return a random 32 bit integer. This function is expected to return\n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511337","messageId":"20250128-pks-reftable-drop-git-compat-util-v2-20-c85c20336317@pks.im","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"[PATCH v2 20/20] Makefile: skip reftable library for Coccinelle","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-28T08:28:21Z","receivedAt":"2025-01-28T08:28:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library does not use any of the common helpers that the Git\nproject has. Consequently, most of the rules that we have in Coccinelle\ndo not apply to the library at all and may even generate false positives\nwhen a pattern can be converted to use a Git helper function.\n\nExclude reftable library sources from being checked by Coccinelle to\navoid such false positives.\n\nSigned-off-by: Patrick Steinhardt \n---\n Makefile | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex d3011e30f7..dc2ac32e4a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -946,7 +946,7 @@ FOUND_SOURCE_FILES := $(filter-out $(GENERATED_H),$(shell $(SOURCES_CMD)))\n FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES))\n FOUND_H_SOURCES = $(filter %.h,$(FOUND_SOURCE_FILES))\n \n-COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES))\n+COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES) reftable/%,$(FOUND_C_SOURCES))\n \n LIB_H = $(FOUND_H_SOURCES)\n \n\n-- \n2.48.1.362.g079036d154.dirty\n\n"},{"id":"511357","messageId":"xmqqsep2oolz.fsf@gitster.g","threadId":"62862","inReplyTo":"Z5iQH4C2JqazaJUl@pks.im","subject":"Re: [PATCH 01/19] reftable/stack: stop using `read_in_full()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-01-28T17:05:44Z","receivedAt":"2025-01-28T17:05:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> The cast from off_t -> size_t matches the currect behavior, but is it\n>> always safe to do this? In `git-compat-util.h` it looks like we have\n>> `xsize_t()` to safely handle these conversions. Since this series is\n>> moving away from `git-compat-util.h` should ideally something similar be\n>> implemented?\n>\n> It is safe, because a couple lines further up we check for `size < 0`\n> and error out if that is the case. So we know it's a positive integer,\n> and thus it can be represented via `size_t`.\n\nEven where off_t (which measures on-disk file in bytes) may be wider\nthan size_t (which measures in-core piece of memory in bytes)?\n\n>> > +\t\tssize_t bytes_read = read(fd, buf + total_read, size - total_read);\n>> > +\t\tif (bytes_read < 0 && (errno == EAGAIN || errno == EINTR))\n>> \n>> The error handling here for EAGAIN doesn't go as far as what `xread()`\n>> does via `handle_nonblock()`. In this scenario is that ok?\n>\n> Yes, because we don't set `O_NONBLOCK` in the reftable library.\n>\n> I'll note that in the commit message.\n\nGood. Thanks.\n"},{"id":"511359","messageId":"xmqqo6zqoncp.fsf@gitster.g","threadId":"62862","inReplyTo":"Z5iTz69yq-IF9qU0@pks.im","subject":"Re: [PATCH 00/19] reftable: stop using \"git-compat-util.h\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-01-28T17:32:54Z","receivedAt":"2025-01-28T17:32:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> I guess time will tell how much of a problem this really is. As said, my\n> prediction is that we won't face many bugs in the low-level code, and\n> then the question becomes moot anyway. But if time proves me wrong I'll\n> think a bit more about potential solutions.\n\nSure. I am OK with the approach of playing by the ear.\n\nThanks.\n"},{"id":"511378","messageId":"xmqq5xlymu5x.fsf@gitster.g","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im","subject":"Re: [PATCH v2 00/20] reftable: stop using \"git-compat-util.h\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-01-28T22:48:42Z","receivedAt":"2025-01-28T22:48:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> Changes in v2:\n> - The splitup of Windows headers has broken compilation because some\n> of the headers couldn't be found anymore. I've fixed this more\n> generally by converting includes in \"compat/\" to always be relative\n> to the project source directory, dropping the platform-specific\n> `-Icompat/` include.\n\nNice.\n\nUse of -Icompat/plat/ would make more sense if our sources include\na header file using the standard name of a file that usually appear\nin /usr/include/ on a platform that either lacks /usr/include/foo.h\nor whose /usr/include/foo.h is inadequate by giving a replacement\ndefinition in compat/plat/foo.h we ship. But that is not how we use\nthem, so I very much like this change.\n\n> -: ---------- > 15: 3a4949d74d compat: consistently resolve headers via project root\n\nWill take a look and replace.\n\nThanks.\n"},{"id":"511398","messageId":"Z5nX2BEoF0YxYKTg@pks.im","threadId":"62862","inReplyTo":"xmqq5xlymu5x.fsf@gitster.g","subject":"Re: [PATCH v2 00/20] reftable: stop using \"git-compat-util.h\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-29T07:25:12Z","receivedAt":"2025-01-29T07:25:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Jan 28, 2025 at 02:48:42PM -0800, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > Changes in v2:\n> > - The splitup of Windows headers has broken compilation because some\n> > of the headers couldn't be found anymore. I've fixed this more\n> > generally by converting includes in \"compat/\" to always be relative\n> > to the project source directory, dropping the platform-specific\n> > `-Icompat/` include.\n> \n> Nice.\n> \n> Use of -Icompat/plat/ would make more sense if our sources include\n> a header file using the standard name of a file that usually appear\n> in /usr/include/ on a platform that either lacks /usr/include/foo.h\n> or whose /usr/include/foo.h is inadequate by giving a replacement\n> definition in compat/plat/foo.h we ship. But that is not how we use\n> them, so I very much like this change.\n\nWe still do that for drop-in replacements like \"compat/regex\" or\n\"compat/poll\", but for everything else we stop doing it.\n\nPatrick\n"},{"id":"511399","messageId":"Z5nY1ryenuFuNpM6@pks.im","threadId":"62862","inReplyTo":"xmqqsep2oolz.fsf@gitster.g","subject":"Re: [PATCH 01/19] reftable/stack: stop using `read_in_full()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-29T07:29:26Z","receivedAt":"2025-01-29T07:29:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Jan 28, 2025 at 09:05:44AM -0800, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> >> The cast from off_t -> size_t matches the currect behavior, but is it\n> >> always safe to do this? In `git-compat-util.h` it looks like we have\n> >> `xsize_t()` to safely handle these conversions. Since this series is\n> >> moving away from `git-compat-util.h` should ideally something similar be\n> >> implemented?\n> >\n> > It is safe, because a couple lines further up we check for `size < 0`\n> > and error out if that is the case. So we know it's a positive integer,\n> > and thus it can be represented via `size_t`.\n> \n> Even where off_t (which measures on-disk file in bytes) may be wider\n> than size_t (which measures in-core piece of memory in bytes)?\n\nWait, can that actually happen? Hm. I assume it can, for example on 32\nbit systems with large-file support enabled. There I assume that `off_t`\nwould be a 64 bit signed integer, whereas `size_t` may be a 32 bit\nunsigned integer.\n\nWill address in v3.\n\nPatrick\n"},{"id":"511401","messageId":"c0ee472a-5ce6-4e2e-8922-65927f07029c@kdbg.org","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-15-c85c20336317@pks.im","subject":"Re: [PATCH v2 15/20] compat: consistently resolve headers via project root","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2025-01-29T07:50:24Z","receivedAt":"2025-01-29T07:50:40Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 28.01.25 um 09:28 schrieb Patrick Steinhardt:\n> diff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c\n> index c2afcbe6c8..68c9fd7363 100644\n> --- a/compat/fsmonitor/fsm-health-darwin.c\n> +++ b/compat/fsmonitor/fsm-health-darwin.c\n> @@ -1,7 +1,7 @@\n> #include \"git-compat-util.h\"\n> +#include \"compat/fsmonitor/fsm-health.h\"\n> #include \"config.h\"\n> #include \"fsmonitor-ll.h\"\n> -#include \"fsm-health.h\"\n> #include \"fsmonitor--daemon.h\"\n> \n> int fsm_health__ctor(struct fsmonitor_daemon_state *state UNUSED)\n\nThe use of double-quotes\n\n #include \"fsm-health.h\"\n\nvery clearly indicates \"this file is expected to be in the same\ndirectory as the file that is including it\". On the other hand,\n\n #include \n\nwould state \"this file is to be found somewhere on the include paths,\nbut not in the directory of the including file\".\n\nTherefore, changes like this one, where the new path only points back to\nthe same directory where the including file is, are very much\nundesirable. The unpatched form is preferable.\n\nDepending on the desired goal, is it perhaps better to use <> brackets\nmore liberally?\n\n-- Hannes\n\n"},{"id":"511409","messageId":"xmqqh65hloeo.fsf@gitster.g","threadId":"62862","inReplyTo":"Z5nX2BEoF0YxYKTg@pks.im","subject":"Re: [PATCH v2 00/20] reftable: stop using \"git-compat-util.h\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-01-29T13:50:39Z","receivedAt":"2025-01-29T13:50:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Tue, Jan 28, 2025 at 02:48:42PM -0800, Junio C Hamano wrote:\n>> Patrick Steinhardt writes:\n>> \n>> > Changes in v2:\n>> > - The splitup of Windows headers has broken compilation because some\n>> > of the headers couldn't be found anymore. I've fixed this more\n>> > generally by converting includes in \"compat/\" to always be relative\n>> > to the project source directory, dropping the platform-specific\n>> > `-Icompat/` include.\n>> \n>> Nice.\n>> \n>> Use of -Icompat/plat/ would make more sense if our sources include\n>> a header file using the standard name of a file that usually appear\n>> in /usr/include/ on a platform that either lacks /usr/include/foo.h\n>> or whose /usr/include/foo.h is inadequate by giving a replacement\n>> definition in compat/plat/foo.h we ship. But that is not how we use\n>> them, so I very much like this change.\n>\n> We still do that for drop-in replacements like \"compat/regex\" or\n> \"compat/poll\", but for everything else we stop doing it.\n\nYup.\n"},{"id":"511411","messageId":"xmqqldutk8bb.fsf@gitster.g","threadId":"62862","inReplyTo":"c0ee472a-5ce6-4e2e-8922-65927f07029c@kdbg.org","subject":"Re: [PATCH v2 15/20] compat: consistently resolve headers via project root","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-01-29T14:23:36Z","receivedAt":"2025-01-29T14:23:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Sixt writes:\n\n> Depending on the desired goal, is it perhaps better to use <> brackets\n> more liberally?\n\nI'd be in general in favor of that direction, as the semantics of <>\nis more clear than \"\" from the point of view of the standard ;-)\n\nThanks.\n"},{"id":"511414","messageId":"ctx7men3stwyasz2xnoaxpp66yn6yfggnfzbawzly57puarazc@p3hpzfn2a7rk","threadId":"62862","inReplyTo":"20250128-pks-reftable-drop-git-compat-util-v2-4-c85c20336317@pks.im","subject":"Re: [PATCH v2 04/20] reftable/record: stop using `COPY_ARRAY()`","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-01-29T15:46:54Z","receivedAt":"2025-01-29T15:49:48Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/01/28 09:28AM, Patrick Steinhardt wrote:\n> Drop our use of `COPY_ARRAY()`, replacing it with an open-coded variant\n> thereof. This is done to reduce our dependency on the Git library.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n> reftable/record.c | 3 ++-\n> 1 file changed, 2 insertions(+), 1 deletion(-)\n> \n> diff --git a/reftable/record.c b/reftable/record.c\n> index 8919df8a4d..d1664c47ca 100644\n> --- a/reftable/record.c\n> +++ b/reftable/record.c\n> @@ -508,7 +508,8 @@ static int reftable_obj_record_copy_from(void *rec, const void *src_rec,\n> \tif (!obj->offsets)\n> \t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n> \tobj->offset_len = src->offset_len;\n> -\tCOPY_ARRAY(obj->offsets, src->offsets, src->offset_len);\n> +\tif (src->offset_len)\n> +\t\tmemcpy(obj->offsets, src->offsets, sizeof(*src->offsets) * src->offset_len);\n\nThe `COPY_ARRAY` version uses `st_mutl()` to protect against potential\noverflows of the size parameter. Does this variant need to account for\nsuch situations as well?\n\n> \treturn 0;\n> }\n> \n> -- \n> 2.48.1.362.g079036d154.dirty\n> \n"},{"id":"511695","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH v3 00/18] reftable: stop using \"git-compat-util.h\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:31Z","receivedAt":"2025-02-03T08:03:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series is the final step to fully decouple the reftable\nlibrary from the rest of the Git codebase. The goal of this is to make\nthe library reusable by other projects like libgit2 by simply copying\nover the source files, making Git the canonical upstream for reftable\nfunctionality.\n\nThis patch series stops using all kinds of helpers exposed by our\n\"git-compat-util.h\" header and open-codes them instead. In order to keep\nus from using these helpers by accident the final step is to pull out\nPOSIX-related bits and pieces into a new \"compat/posix.h\" header, which\nthe reftable library then uses instead of \"git-compat-util.h\".\n\nThe series is built on top of master at 5f8f7081f7 (The third batch,\n2025-01-23) with ps/reftable-sign-compare at 33319b0976 (reftable:\naddress trivial -Wsign-compare warnings, 2025-01-20) merged into it.\nThere is a trivial merge conflict with ps/zlib-ng that can be solved\nlike this:\n\n diff --cc reftable/system.h\n index e4a8944a70,d02eacea8f..0000000000\n --- a/reftable/system.h\n +++ b/reftable/system.h\n @@@ -11,15 -11,9 +11,15 @@@ https://developers.google.com/open-sour\n \n /* This header glues the reftable library to the rest of Git */\n \n -#include \"git-compat-util.h\"\n +#include \"compat/posix.h\"\n - #include \n + #include \"compat/zlib-compat.h\"\n \n +/*\n + * Return a random 32 bit integer. This function is expected to return\n + * pre-seeded data.\n + */\n +uint32_t reftable_rand(void);\n +\n /*\n * An implementation-specific temporary file. By making this specific to the\n * implementation it becomes possible to tie temporary files into any kind of\n\nChanges in v2:\n - The splitup of Windows headers has broken compilation because some\n of the headers couldn't be found anymore. I've fixed this more\n generally by converting includes in \"compat/\" to always be relative\n to the project source directory, dropping the platform-specific\n `-Icompat/` include.\n - Explain why we don't port over `EWOULDBLOCK` handling.\n - Fix commit message typos.\n - Link to v1: https://lore.kernel.org/r/20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im\n\nChanges in v3:\n - Fix type of `total_read` variable used to track how many bytes we\n have read in `fd_read_lines()`.\n - Drop the patch use root-relative includes again. Let's rather\n discuss this outside of the scope of this series.\n - Link to v2: https://lore.kernel.org/r/20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (18):\n reftable/stack: stop using `read_in_full()`\n reftable/stack: stop using `write_in_full()`\n reftable/blocksource: stop using `xmmap()`\n reftable/record: stop using `COPY_ARRAY()`\n reftable/record: stop using `BUG()` in `reftable_record_init()`\n reftable/record: don't `BUG()` in `reftable_record_cmp()`\n reftable: stop using `BUG()` in trivial cases\n reftable/basics: stop using `st_mult()` in array allocators\n reftable/basics: provide wrappers for big endian conversion\n reftable/reader: stop using `ARRAY_SIZE()` macro\n reftable/system: introduce `reftable_rand()`\n reftable/stack: stop using `sleep_millisec()`\n reftable/basics: stop using `SWAP()` macro\n reftable/basics: stop using `UNUSED` annotation\n compat/mingw: split out POSIX-related bits\n git-compat-util.h: split out POSIX-emulating bits\n reftable: decouple from Git codebase by pulling in \"compat/posix.h\"\n Makefile: skip reftable library for Coccinelle\n\n Makefile | 2 +-\n compat/{mingw.c => mingw/compat-util.c} | 0\n compat/mingw/compat-util.h | 220 +++++++++++++\n compat/{mingw.h => mingw/posix.h} | 216 +------------\n compat/{msvc.c => msvc/compat-util.c} | 2 +-\n compat/msvc/compat-util.h | 7 +\n compat/{msvc.h => msvc/posix.h} | 8 +-\n compat/posix.h | 541 ++++++++++++++++++++++++++++++++\n config.mak.uname | 6 +-\n contrib/buildsystems/CMakeLists.txt | 2 +-\n git-compat-util.h | 535 +------------------------------\n meson.build | 8 +-\n reftable/basics.c | 19 --\n reftable/basics.h | 123 +++++++-\n reftable/block.c | 16 +-\n reftable/blocksource.c | 21 +-\n reftable/iter.c | 20 +-\n reftable/merged.c | 27 +-\n reftable/pq.c | 40 ++-\n reftable/pq.h | 2 +-\n reftable/reader.c | 33 +-\n reftable/record.c | 109 ++++---\n reftable/record.h | 6 +-\n reftable/stack.c | 52 ++-\n reftable/system.c | 7 +\n reftable/system.h | 9 +-\n reftable/writer.c | 29 +-\n t/unit-tests/t-reftable-basics.c | 28 +-\n t/unit-tests/t-reftable-pq.c | 22 +-\n t/unit-tests/t-reftable-record.c | 42 ++-\n 30 files changed, 1223 insertions(+), 929 deletions(-)\n\nRange-diff versus v2:\n\n 1: e9364a8d05 ! 1: 8488db3ac9 reftable/stack: stop using `read_in_full()`\n @@ reftable/stack.c: static int fd_read_lines(int fd, char ***namesp)\n -\tif (read_in_full(fd, buf, size) != size) {\n -\t\terr = REFTABLE_IO_ERROR;\n -\t\tgoto done;\n -+\tfor (size_t total_read = 0; total_read < (size_t) size; ) {\n ++\tfor (off_t total_read = 0; total_read < size; ) {\n +\t\tssize_t bytes_read = read(fd, buf + total_read, size - total_read);\n +\t\tif (bytes_read < 0 && (errno == EAGAIN || errno == EINTR))\n +\t\t\tcontinue;\n 2: 41e411c00d = 2: 19db682279 reftable/stack: stop using `write_in_full()`\n 3: 1abf9a77e4 = 3: 17fa559a68 reftable/blocksource: stop using `xmmap()`\n 4: 458837e511 < -: ---------- reftable/record: stop using `COPY_ARRAY()`\n -: ---------- > 4: 1667a37d38 reftable/record: stop using `COPY_ARRAY()`\n 5: a620950cb4 = 5: 1fe9563380 reftable/record: stop using `BUG()` in `reftable_record_init()`\n 6: 0590780f55 = 6: 12c1fdc125 reftable/record: don't `BUG()` in `reftable_record_cmp()`\n 7: adcba48303 = 7: 8cfb77ce2c reftable: stop using `BUG()` in trivial cases\n 8: 353d958e38 = 8: c7227123fa reftable/basics: stop using `st_mult()` in array allocators\n 9: c6abf6ca1e = 9: 204a7ffed7 reftable/basics: provide wrappers for big endian conversion\n10: 095c6805d1 = 10: 533f2cdbd5 reftable/reader: stop using `ARRAY_SIZE()` macro\n11: 816a984361 = 11: f5a9edc360 reftable/system: introduce `reftable_rand()`\n12: eb09c1d79d = 12: 4b066d5be8 reftable/stack: stop using `sleep_millisec()`\n13: 1595e4d771 = 13: 8580d01ee2 reftable/basics: stop using `SWAP()` macro\n14: 448ec74479 = 14: 72847f6d69 reftable/basics: stop using `UNUSED` annotation\n15: 838f2bcd3a < -: ---------- compat: consistently resolve headers via project root\n16: cf6b915daf ! 15: 0348eed321 compat/mingw: split out POSIX-related bits\n @@ compat/mingw/compat-util.h (new)\n +#ifndef COMPAT_MINGW_COMPAT_UTIL_H\n +#define COMPAT_MINGW_COMPAT_UTIL_H\n +\n -+#include \"compat/mingw/posix.h\"\n ++#include \"posix.h\"\n +\n +struct config_context;\n +int mingw_core_config(const char *var, const char *value,\n @@ compat/mingw/posix.h: char *mingw_query_user_email(void);\n -#endif\n +#endif /* COMPAT_MINGW_POSIX_H */\n \n - ## compat/msvc.c ##\n + ## compat/msvc.c => compat/msvc/compat-util.c ##\n @@\n #include \n - #include \"strbuf.h\"\n + #include \"../strbuf.h\"\n \n --#include \"compat/mingw.c\"\n -+#include \"compat/mingw/compat-util.c\"\n +-#include \"mingw.c\"\n ++#include \"mingw/compat-util.c\"\n \n - ## compat/msvc.h ##\n -@@ compat/msvc.h: typedef int sigset_t;\n + ## compat/msvc/compat-util.h (new) ##\n +@@\n ++#ifndef COMPAT_MSVC_COMPAT_UTIL_H\n ++#define COMPAT_MSVC_COMPAT_UTIL_H\n ++\n ++#include \"posix.h\"\n ++#include \"../mingw/compat-util.h\"\n ++\n ++#endif /* COMPAT_MSVC_COMPAT_UTIL_H */\n +\n + ## compat/msvc.h => compat/msvc/posix.h ##\n +@@\n +-#ifndef __MSVC__HEAD\n +-#define __MSVC__HEAD\n ++#ifndef COMPAT_MSVC_POSIX_H\n ++#define COMPAT_MSVC_POSIX_H\n + \n + #include \n + #include \n +@@ compat/msvc/posix.h: typedef int sigset_t;\n /* open for reading, writing, or both (not in fcntl.h) */\n #define O_ACCMODE (_O_RDONLY | _O_WRONLY | _O_RDWR)\n \n -#include \"compat/mingw.h\"\n -+#include \"compat/mingw/compat-util.h\"\n ++#include \"../mingw/posix.h\"\n \n - #endif\n +-#endif\n ++#endif /* COMPAT_MSVC_POSIX_H */\n \n ## config.mak.uname ##\n @@ config.mak.uname: endif\n + \tAR = compat/vcbuild/scripts/lib.pl\n + \tCFLAGS =\n + \tBASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE\n +-\tCOMPAT_OBJS = compat/msvc.o compat/winansi.o \\\n ++\tCOMPAT_OBJS = compat/msvc/compat-util.o compat/winansi.o \\\n + \t\tcompat/win32/flush.o \\\n + \t\tcompat/win32/path-utils.o \\\n + \t\tcompat/win32/pthread.o compat/win32/syslog.o \\\n +@@ config.mak.uname: endif\n \n \tEXTRA_PROGRAMS += headless-git$X\n \n -compat/msvc.o: compat/msvc.c compat/mingw.c GIT-CFLAGS\n -+compat/msvc.o: compat/msvc.c compat/mingw/compat-util.c GIT-CFLAGS\n ++compat/msvc/compat-util.o: compat/msvc/compat-util.c compat/mingw/compat-util.c GIT-CFLAGS\n endif\n ifeq ($(uname_S),Interix)\n \tNO_INITGROUPS = YesPlease\n @@ config.mak.uname: ifeq ($(uname_S),MINGW)\n \tBASIC_LDFLAGS += -municode\n - \tCOMPAT_CFLAGS += -DNOGDI -Icompat/win32\n + \tCOMPAT_CFLAGS += -DNOGDI -Icompat -Icompat/win32\n \tCOMPAT_CFLAGS += -DSTRIP_EXTENSION=\\\".exe\\\"\n -\tCOMPAT_OBJS += compat/mingw.o compat/winansi.o \\\n +\tCOMPAT_OBJS += compat/mingw/compat-util.o compat/winansi.o \\\n @@ meson.build: if host_machine.system() == 'cygwin'\n elif host_machine.system() == 'windows'\n libgit_sources += [\n - 'compat/mingw.c',\n -+ 'compat/mingw/compat-util.c',\n 'compat/winansi.c',\n 'compat/win32/flush.c',\n 'compat/win32/path-utils.c',\n +@@ meson.build: elif host_machine.system() == 'windows'\n + libgit_include_directories += 'compat/win32'\n + if compiler.get_id() == 'msvc'\n + libgit_include_directories += 'compat/vcbuild/include'\n ++ libgit_sources += 'compat/msvc/compat-util.c'\n ++ else\n ++ libgit_sources += 'compat/mingw/compat-util.c'\n + endif\n + endif\n + \n @@ meson.build: else\n error('Native regex support requested but not found')\n endif\n17: ae57e3f391 < -: ---------- compat/msvc: split out POSIX-related bits\n18: c32f39b777 ! 16: 141fd6a7cd git-compat-util.h: split out POSIX-emulating bits\n @@ compat/posix.h (new)\n +#endif\n +\n +#if defined(__MINGW32__)\n -+#include \"compat/mingw/posix.h\"\n ++#include \"mingw/posix.h\"\n +#elif defined(_MSC_VER)\n -+#include \"compat/msvc/posix.h\"\n ++#include \"msvc/posix.h\"\n +#else\n +#include \n +#include \n19: 496d30ee27 ! 17: 640f5e6376 reftable: decouple from Git codebase by pulling in \"compat/posix.h\"\n @@ reftable/system.h: license that can be found in the LICENSE file or at\n /* This header glues the reftable library to the rest of Git */\n \n -#include \"git-compat-util.h\"\n -+#include \"compat/posix.h\"\n ++#include \"../compat/posix.h\"\n +#include \n \n /*\n20: d257172bf7 = 18: 2f17a1963d Makefile: skip reftable library for Coccinelle\n\n---\nbase-commit: 8047765d092881ec4aef7dfc57772161eee7f0f5\nchange-id: 20241119-pks-reftable-drop-git-compat-util-470f2bfde562\n\n"},{"id":"511696","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-1-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"[PATCH v3 01/18] reftable/stack: stop using `read_in_full()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:32Z","receivedAt":"2025-02-03T08:03:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"There is a single callsite of `read_in_full()` in the reftable library.\nOpen-code the function to reduce our dependency on the Git library.\n\nNote that we only partially port over the logic from `read_in_full()`\nand its underlying `xread()` helper. Most importantly, the latter also\nknows to handle `EWOULDBLOCK` via `handle_nonblock()`. This logic is\nirrelevant for us though because the reftable library never sets the\n`O_NONBLOCK` option in the first place.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 18 ++++++++++++++----\n 1 file changed, 14 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex f7c1845e15..d57acd9540 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -115,13 +115,16 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \n static int fd_read_lines(int fd, char ***namesp)\n {\n-\toff_t size = lseek(fd, 0, SEEK_END);\n \tchar *buf = NULL;\n \tint err = 0;\n+\toff_t size;\n+\n+\tsize = lseek(fd, 0, SEEK_END);\n \tif (size < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n+\n \terr = lseek(fd, 0, SEEK_SET);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n@@ -134,9 +137,16 @@ static int fd_read_lines(int fd, char ***namesp)\n \t\tgoto done;\n \t}\n \n-\tif (read_in_full(fd, buf, size) != size) {\n-\t\terr = REFTABLE_IO_ERROR;\n-\t\tgoto done;\n+\tfor (off_t total_read = 0; total_read < size; ) {\n+\t\tssize_t bytes_read = read(fd, buf + total_read, size - total_read);\n+\t\tif (bytes_read < 0 && (errno == EAGAIN || errno == EINTR))\n+\t\t\tcontinue;\n+\t\tif (bytes_read < 0 || !bytes_read) {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\ttotal_read += bytes_read;\n \t}\n \tbuf[size] = 0;\n \n\n-- \n2.48.1.502.g6dc24dfdaf.dirty\n\n"},{"id":"511697","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-2-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"[PATCH v3 02/18] reftable/stack: stop using `write_in_full()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:33Z","receivedAt":"2025-02-03T08:03:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Similar to the preceding commit, drop our use of `write_in_full()` and\nimplement a new wrapper `reftable_write_full()` that handles this logic\nfor us. This is done to reduce our dependency on the Git library.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 28 ++++++++++++++++++++++++----\n 1 file changed, 24 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex d57acd9540..46abf51ec5 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -48,6 +48,25 @@ static int stack_fsync(const struct reftable_write_options *opts, int fd)\n \treturn fsync(fd);\n }\n \n+static ssize_t reftable_write_data(int fd, const void *data, size_t size)\n+{\n+\tsize_t total_written = 0;\n+\tconst char *p = data;\n+\n+\twhile (total_written < size) {\n+\t\tssize_t bytes_written = write(fd, p, size - total_written);\n+\t\tif (bytes_written < 0 && (errno == EAGAIN || errno == EINTR))\n+\t\t\tcontinue;\n+\t\tif (bytes_written < 0)\n+\t\t\treturn REFTABLE_IO_ERROR;\n+\n+\t\ttotal_written += bytes_written;\n+\t\tp += bytes_written;\n+\t}\n+\n+\treturn total_written;\n+}\n+\n struct fd_writer {\n \tconst struct reftable_write_options *opts;\n \tint fd;\n@@ -56,7 +75,7 @@ struct fd_writer {\n static ssize_t fd_writer_write(void *arg, const void *data, size_t sz)\n {\n \tstruct fd_writer *writer = arg;\n-\treturn write_in_full(writer->fd, data, sz);\n+\treturn reftable_write_data(writer->fd, data, sz);\n }\n \n static int fd_writer_flush(void *arg)\n@@ -784,7 +803,8 @@ int reftable_addition_commit(struct reftable_addition *add)\n \t\t\tgoto done;\n \t}\n \n-\terr = write_in_full(add->tables_list_lock.fd, table_list.buf, table_list.len);\n+\terr = reftable_write_data(add->tables_list_lock.fd,\n+\t\t\t\t table_list.buf, table_list.len);\n \treftable_buf_release(&table_list);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n@@ -1468,8 +1488,8 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\t\tgoto done;\n \t}\n \n-\terr = write_in_full(tables_list_lock.fd,\n-\t\t\t tables_list_buf.buf, tables_list_buf.len);\n+\terr = reftable_write_data(tables_list_lock.fd,\n+\t\t\t\t tables_list_buf.buf, tables_list_buf.len);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tunlink(new_table_path.buf);\n\n-- \n2.48.1.502.g6dc24dfdaf.dirty\n\n"},{"id":"511698","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-3-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"[PATCH v3 03/18] reftable/blocksource: stop using `xmmap()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:34Z","receivedAt":"2025-02-03T08:03:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We use `xmmap()` to map reftables into memory. This function has two\nproblems:\n\n - It causes us to die in case the mmap fails.\n\n - It ties us to the Git codebase.\n\nRefactor the code to use mmap(3p) instead with manual error checking.\nNote that this function may not be the system-provided mmap(3p), but may\npoint to our `git_mmap()` wrapper that emulates the syscall on systems\nthat do not have mmap(3p) available.\n\nFix `reftable_block_source_from_file()` to properly bubble up the error\ncode in case the map(3p) call fails.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/blocksource.c | 11 ++++++++---\n 1 file changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex bba4a45b98..02972c46f4 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -98,7 +98,7 @@ static struct reftable_block_source_vtable file_vtable = {\n int reftable_block_source_from_file(struct reftable_block_source *bs,\n \t\t\t\t const char *name)\n {\n-\tstruct file_block_source *p;\n+\tstruct file_block_source *p = NULL;\n \tstruct stat st;\n \tint fd, err;\n \n@@ -122,7 +122,12 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,\n \t}\n \n \tp->size = st.st_size;\n-\tp->data = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tp->data = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tif (p->data == MAP_FAILED) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tp->data = NULL;\n+\t\tgoto out;\n+\t}\n \n \tassert(!bs->ops);\n \tbs->ops = &file_vtable;\n@@ -135,5 +140,5 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,\n \t\tclose(fd);\n \tif (err < 0)\n \t\treftable_free(p);\n-\treturn 0;\n+\treturn err;\n }\n\n-- \n2.48.1.502.g6dc24dfdaf.dirty\n\n"},{"id":"511699","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-4-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"[PATCH v3 04/18] reftable/record: stop using `COPY_ARRAY()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:35Z","receivedAt":"2025-02-03T08:03:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Drop our use of `COPY_ARRAY()`, replacing it with an open-coded variant\nthereof. This is done to reduce our dependency on the Git library.\n\nWhile at it, guard the whole array copy logic so that we only copy it in\ncase there actually is anything to be copied. Otherwise, we may end up\ntrying to allocate a zero-sized array, which will return a NULL pointer\nand thus cause us to return an `REFTABLE_OUT_OF_MEMORY_ERROR`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/record.c | 16 +++++++++++-----\n 1 file changed, 11 insertions(+), 5 deletions(-)\n\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 8919df8a4d..2c0cc32cbd 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -504,11 +504,17 @@ static int reftable_obj_record_copy_from(void *rec, const void *src_rec,\n \tif (src->hash_prefix_len)\n \t\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n \n-\tREFTABLE_ALLOC_ARRAY(obj->offsets, src->offset_len);\n-\tif (!obj->offsets)\n-\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n-\tobj->offset_len = src->offset_len;\n-\tCOPY_ARRAY(obj->offsets, src->offsets, src->offset_len);\n+\tif (src->offset_len) {\n+\t\tif (sizeof(*src->offsets) > SIZE_MAX / src->offset_len)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n+\t\tREFTABLE_ALLOC_ARRAY(obj->offsets, src->offset_len);\n+\t\tif (!obj->offsets)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n+\t\tmemcpy(obj->offsets, src->offsets, sizeof(*src->offsets) * src->offset_len);\n+\t\tobj->offset_len = src->offset_len;\n+\t}\n \n \treturn 0;\n }\n\n-- \n2.48.1.502.g6dc24dfdaf.dirty\n\n"},{"id":"511700","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-5-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"[PATCH v3 05/18] reftable/record: stop using `BUG()` in `reftable_record_init()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:36Z","receivedAt":"2025-02-03T08:03:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're aborting the program via `BUG()` in case `reftable_record_init()`\nwas invoked with an unknown record type. This is bad because we may now\ndie in library code, and because it makes us depend on the Git codebase.\n\nRefactor the code such that `reftable_record_init()` can return an error\ncode to the caller. Adapt any callers accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/block.c | 4 +++-\n reftable/merged.c | 5 ++++-\n reftable/reader.c | 5 ++++-\n reftable/record.c | 8 ++++----\n reftable/record.h | 4 ++--\n t/unit-tests/t-reftable-pq.c | 6 +++---\n t/unit-tests/t-reftable-record.c | 2 +-\n 7 files changed, 21 insertions(+), 13 deletions(-)\n\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 8ac865ce78..255d566854 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -509,7 +509,9 @@ int block_iter_seek_key(struct block_iter *it, const struct block_reader *br,\n \tit->block_len = br->block_len;\n \tit->hash_size = br->hash_size;\n \n-\treftable_record_init(&rec, block_reader_type(br));\n+\terr = reftable_record_init(&rec, block_reader_type(br));\n+\tif (err < 0)\n+\t\tgoto done;\n \n \t/*\n \t * We're looking for the last entry less than the wanted key so that\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex e72b39e178..4156eec07f 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -253,7 +253,10 @@ int merged_table_init_iter(struct reftable_merged_table *mt,\n \t}\n \n \tfor (size_t i = 0; i < mt->readers_len; i++) {\n-\t\treftable_record_init(&subiters[i].rec, typ);\n+\t\tret = reftable_record_init(&subiters[i].rec, typ);\n+\t\tif (ret < 0)\n+\t\t\tgoto out;\n+\n \t\tret = reader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n \t\tif (ret < 0)\n \t\t\tgoto out;\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 3f2e4b2800..de6e6dd932 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -360,7 +360,10 @@ static int table_iter_seek_linear(struct table_iter *ti,\n \tstruct reftable_record rec;\n \tint err;\n \n-\treftable_record_init(&rec, reftable_record_type(want));\n+\terr = reftable_record_init(&rec, reftable_record_type(want));\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = reftable_record_key(want, &want_key);\n \tif (err < 0)\n \t\tgoto done;\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 2c0cc32cbd..1e18f8dffb 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -1306,7 +1306,7 @@ reftable_record_vtable(struct reftable_record *rec)\n \tabort();\n }\n \n-void reftable_record_init(struct reftable_record *rec, uint8_t typ)\n+int reftable_record_init(struct reftable_record *rec, uint8_t typ)\n {\n \tmemset(rec, 0, sizeof(*rec));\n \trec->type = typ;\n@@ -1315,11 +1315,11 @@ void reftable_record_init(struct reftable_record *rec, uint8_t typ)\n \tcase BLOCK_TYPE_REF:\n \tcase BLOCK_TYPE_LOG:\n \tcase BLOCK_TYPE_OBJ:\n-\t\treturn;\n+\t\treturn 0;\n \tcase BLOCK_TYPE_INDEX:\n \t\treftable_buf_init(&rec->u.idx.last_key);\n-\t\treturn;\n+\t\treturn 0;\n \tdefault:\n-\t\tBUG(\"unhandled record type\");\n+\t\treturn REFTABLE_API_ERROR;\n \t}\n }\ndiff --git a/reftable/record.h b/reftable/record.h\nindex c7755a4d75..e1846c294b 100644\n--- a/reftable/record.h\n+++ b/reftable/record.h\n@@ -130,8 +130,8 @@ struct reftable_record {\n \t} u;\n };\n \n-/* Initialize the reftable record for the given type */\n-void reftable_record_init(struct reftable_record *rec, uint8_t typ);\n+/* Initialize the reftable record for the given type. */\n+int reftable_record_init(struct reftable_record *rec, uint8_t typ);\n \n /* see struct record_vtable */\n int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b);\ndiff --git a/t/unit-tests/t-reftable-pq.c b/t/unit-tests/t-reftable-pq.c\nindex f3f8a0cdf3..d8a4c283a1 100644\n--- a/t/unit-tests/t-reftable-pq.c\n+++ b/t/unit-tests/t-reftable-pq.c\n@@ -32,7 +32,7 @@ static void t_pq_record(void)\n \tchar *last = NULL;\n \n \tfor (i = 0; i < N; i++) {\n-\t\treftable_record_init(&recs[i], BLOCK_TYPE_REF);\n+\t\tcheck(!reftable_record_init(&recs[i], BLOCK_TYPE_REF));\n \t\trecs[i].u.ref.refname = xstrfmt(\"%02\"PRIuMAX, (uintmax_t)i);\n \t}\n \n@@ -72,7 +72,7 @@ static void t_pq_index(void)\n \tsize_t N = ARRAY_SIZE(recs), i;\n \n \tfor (i = 0; i < N; i++) {\n-\t\treftable_record_init(&recs[i], BLOCK_TYPE_REF);\n+\t\tcheck(!reftable_record_init(&recs[i], BLOCK_TYPE_REF));\n \t\trecs[i].u.ref.refname = (char *) \"refs/heads/master\";\n \t}\n \n@@ -111,7 +111,7 @@ static void t_merged_iter_pqueue_top(void)\n \tsize_t N = ARRAY_SIZE(recs), i;\n \n \tfor (i = 0; i < N; i++) {\n-\t\treftable_record_init(&recs[i], BLOCK_TYPE_REF);\n+\t\tcheck(!reftable_record_init(&recs[i], BLOCK_TYPE_REF));\n \t\trecs[i].u.ref.refname = (char *) \"refs/heads/master\";\n \t}\n \ndiff --git a/t/unit-tests/t-reftable-record.c b/t/unit-tests/t-reftable-record.c\nindex d49d2a2729..6540bd20e3 100644\n--- a/t/unit-tests/t-reftable-record.c\n+++ b/t/unit-tests/t-reftable-record.c\n@@ -17,7 +17,7 @@ static void t_copy(struct reftable_record *rec)\n \tuint8_t typ;\n \n \ttyp = reftable_record_type(rec);\n-\treftable_record_init(©, typ);\n+\tcheck(!reftable_record_init(©, typ));\n \treftable_record_copy_from(©, rec, REFTABLE_HASH_SIZE_SHA1);\n \t/* do it twice to catch memory leaks */\n \treftable_record_copy_from(©, rec, REFTABLE_HASH_SIZE_SHA1);\n\n-- \n2.48.1.502.g6dc24dfdaf.dirty\n\n"},{"id":"511701","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-6-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"[PATCH v3 06/18] reftable/record: don't `BUG()` in `reftable_record_cmp()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:37Z","receivedAt":"2025-02-03T08:03:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library aborts with a bug in case `reftable_record_cmp()`\nis invoked with two records of differing types. This would cause the\nprogram to die without the caller being able to handle the error, which\nis not something we want in the context of library code. And it ties us\nto the Git codebase.\n\nRefactor the code such that `reftable_record_cmp()` returns an error\ncode separate from the actual comparison result. This requires us to\nalso adapt some callers up the callchain in a similar fashion.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 20 +++++++++++++++-----\n reftable/pq.c | 36 +++++++++++++++++++++++++++++-------\n reftable/pq.h | 2 +-\n reftable/record.c | 10 ++++++----\n reftable/record.h | 2 +-\n t/unit-tests/t-reftable-pq.c | 16 ++++++++++++----\n t/unit-tests/t-reftable-record.c | 40 ++++++++++++++++++++++++++++------------\n 7 files changed, 92 insertions(+), 34 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 4156eec07f..563864068c 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -66,8 +66,11 @@ static int merged_iter_seek(struct merged_iter *mi, struct reftable_record *want\n \tint err;\n \n \tmi->advance_index = -1;\n-\twhile (!merged_iter_pqueue_is_empty(mi->pq))\n-\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq)) {\n+\t\terr = merged_iter_pqueue_remove(&mi->pq, NULL);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n \n \tfor (size_t i = 0; i < mi->subiters_len; i++) {\n \t\terr = iterator_seek(&mi->subiters[i].iter, want);\n@@ -120,7 +123,9 @@ static int merged_iter_next_entry(struct merged_iter *mi,\n \tif (empty)\n \t\treturn 1;\n \n-\tentry = merged_iter_pqueue_remove(&mi->pq);\n+\terr = merged_iter_pqueue_remove(&mi->pq, &entry);\n+\tif (err < 0)\n+\t\treturn err;\n \n \t/*\n \t One can also use reftable as datacenter-local storage, where the ref\n@@ -134,11 +139,16 @@ static int merged_iter_next_entry(struct merged_iter *mi,\n \t\tstruct pq_entry top = merged_iter_pqueue_top(mi->pq);\n \t\tint cmp;\n \n-\t\tcmp = reftable_record_cmp(top.rec, entry.rec);\n+\t\terr = reftable_record_cmp(top.rec, entry.rec, &cmp);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n \t\tif (cmp > 0)\n \t\t\tbreak;\n \n-\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\t\terr = merged_iter_pqueue_remove(&mi->pq, NULL);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\n \t\terr = merged_iter_advance_subiter(mi, top.index);\n \t\tif (err < 0)\n \t\t\treturn err;\ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex 5591e875e1..ef8035cfd9 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -15,13 +15,18 @@ license that can be found in the LICENSE file or at\n \n int pq_less(struct pq_entry *a, struct pq_entry *b)\n {\n-\tint cmp = reftable_record_cmp(a->rec, b->rec);\n+\tint cmp, err;\n+\n+\terr = reftable_record_cmp(a->rec, b->rec, &cmp);\n+\tif (err < 0)\n+\t\treturn err;\n+\n \tif (cmp == 0)\n \t\treturn a->index > b->index;\n \treturn cmp < 0;\n }\n \n-struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n+int merged_iter_pqueue_remove(struct merged_iter_pqueue *pq, struct pq_entry *out)\n {\n \tsize_t i = 0;\n \tstruct pq_entry e = pq->heap[0];\n@@ -32,17 +37,34 @@ struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n \t\tsize_t min = i;\n \t\tsize_t j = 2 * i + 1;\n \t\tsize_t k = 2 * i + 2;\n-\t\tif (j < pq->len && pq_less(&pq->heap[j], &pq->heap[i]))\n-\t\t\tmin = j;\n-\t\tif (k < pq->len && pq_less(&pq->heap[k], &pq->heap[min]))\n-\t\t\tmin = k;\n+\t\tint cmp;\n+\n+\t\tif (j < pq->len) {\n+\t\t\tcmp = pq_less(&pq->heap[j], &pq->heap[i]);\n+\t\t\tif (cmp < 0)\n+\t\t\t\treturn -1;\n+\t\t\telse if (cmp)\n+\t\t\t\tmin = j;\n+\t\t}\n+\n+\t\tif (k < pq->len) {\n+\t\t\tcmp = pq_less(&pq->heap[k], &pq->heap[min]);\n+\t\t\tif (cmp < 0)\n+\t\t\t\treturn -1;\n+\t\t\telse if (cmp)\n+\t\t\t\tmin = k;\n+\t\t}\n+\n \t\tif (min == i)\n \t\t\tbreak;\n \t\tSWAP(pq->heap[i], pq->heap[min]);\n \t\ti = min;\n \t}\n \n-\treturn e;\n+\tif (out)\n+\t\t*out = e;\n+\n+\treturn 0;\n }\n \n int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e)\ndiff --git a/reftable/pq.h b/reftable/pq.h\nindex 83c062eeca..ff39016445 100644\n--- a/reftable/pq.h\n+++ b/reftable/pq.h\n@@ -22,7 +22,7 @@ struct merged_iter_pqueue {\n \tsize_t cap;\n };\n \n-struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n+int merged_iter_pqueue_remove(struct merged_iter_pqueue *pq, struct pq_entry *out);\n int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e);\n void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n int pq_less(struct pq_entry *a, struct pq_entry *b);\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 1e18f8dffb..b39d99fcc7 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -1195,12 +1195,14 @@ int reftable_record_is_deletion(struct reftable_record *rec)\n \t\treftable_record_data(rec));\n }\n \n-int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b)\n+int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b,\n+\t\t\tint *cmp)\n {\n \tif (a->type != b->type)\n-\t\tBUG(\"cannot compare reftable records of different type\");\n-\treturn reftable_record_vtable(a)->cmp(\n-\t\treftable_record_data(a), reftable_record_data(b));\n+\t\treturn -1;\n+\t*cmp = reftable_record_vtable(a)->cmp(reftable_record_data(a),\n+\t\t\t\t\t reftable_record_data(b));\n+\treturn 0;\n }\n \n int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, uint32_t hash_size)\ndiff --git a/reftable/record.h b/reftable/record.h\nindex e1846c294b..867810a932 100644\n--- a/reftable/record.h\n+++ b/reftable/record.h\n@@ -134,7 +134,7 @@ struct reftable_record {\n int reftable_record_init(struct reftable_record *rec, uint8_t typ);\n \n /* see struct record_vtable */\n-int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b);\n+int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b, int *cmp);\n int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, uint32_t hash_size);\n int reftable_record_key(struct reftable_record *rec, struct reftable_buf *dest);\n int reftable_record_copy_from(struct reftable_record *rec,\ndiff --git a/t/unit-tests/t-reftable-pq.c b/t/unit-tests/t-reftable-pq.c\nindex d8a4c283a1..c128fe8616 100644\n--- a/t/unit-tests/t-reftable-pq.c\n+++ b/t/unit-tests/t-reftable-pq.c\n@@ -21,7 +21,9 @@ static void merged_iter_pqueue_check(const struct merged_iter_pqueue *pq)\n \n static int pq_entry_equal(struct pq_entry *a, struct pq_entry *b)\n {\n-\treturn !reftable_record_cmp(a->rec, b->rec) && (a->index == b->index);\n+\tint cmp;\n+\tcheck(!reftable_record_cmp(a->rec, b->rec, &cmp));\n+\treturn !cmp && (a->index == b->index);\n }\n \n static void t_pq_record(void)\n@@ -49,7 +51,9 @@ static void t_pq_record(void)\n \n \twhile (!merged_iter_pqueue_is_empty(pq)) {\n \t\tstruct pq_entry top = merged_iter_pqueue_top(pq);\n-\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct pq_entry e;\n+\n+\t\tcheck(!merged_iter_pqueue_remove(&pq, &e));\n \t\tmerged_iter_pqueue_check(&pq);\n \n \t\tcheck(pq_entry_equal(&top, &e));\n@@ -90,7 +94,9 @@ static void t_pq_index(void)\n \n \tfor (i = N - 1; i > 0; i--) {\n \t\tstruct pq_entry top = merged_iter_pqueue_top(pq);\n-\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct pq_entry e;\n+\n+\t\tcheck(!merged_iter_pqueue_remove(&pq, &e));\n \t\tmerged_iter_pqueue_check(&pq);\n \n \t\tcheck(pq_entry_equal(&top, &e));\n@@ -129,7 +135,9 @@ static void t_merged_iter_pqueue_top(void)\n \n \tfor (i = N - 1; i > 0; i--) {\n \t\tstruct pq_entry top = merged_iter_pqueue_top(pq);\n-\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct pq_entry e;\n+\n+\t\tcheck(!merged_iter_pqueue_remove(&pq, &e));\n \n \t\tmerged_iter_pqueue_check(&pq);\n \t\tcheck(pq_entry_equal(&top, &e));\ndiff --git a/t/unit-tests/t-reftable-record.c b/t/unit-tests/t-reftable-record.c\nindex 6540bd20e3..5954966373 100644\n--- a/t/unit-tests/t-reftable-record.c\n+++ b/t/unit-tests/t-reftable-record.c\n@@ -100,16 +100,20 @@ static void t_reftable_ref_record_comparison(void)\n \t\t\t.u.ref.value.symref = (char *) \"refs/heads/master\",\n \t\t},\n \t};\n+\tint cmp;\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \n \tin[1].u.ref.value_type = in[0].u.ref.value_type;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n }\n \n static void t_reftable_ref_record_compare_name(void)\n@@ -209,17 +213,20 @@ static void t_reftable_log_record_comparison(void)\n \t\t\t.u.log.update_index = 22,\n \t\t},\n \t};\n+\tint cmp;\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \t/* comparison should be reversed for equal keys, because\n \t * comparison is now performed on the basis of update indices */\n-\tcheck_int(reftable_record_cmp(&in[0], &in[1]), <, 0);\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck_int(cmp, <, 0);\n \n \tin[1].u.log.update_index = in[0].u.log.update_index;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n }\n \n static void t_reftable_log_record_compare_key(void)\n@@ -396,16 +403,20 @@ static void t_reftable_obj_record_comparison(void)\n \t\t\t.u.obj.hash_prefix_len = 5,\n \t\t},\n \t};\n+\tint cmp;\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \n \tin[1].u.obj.offset_len = in[0].u.obj.offset_len;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n }\n \n static void t_reftable_obj_record_roundtrip(void)\n@@ -486,19 +497,24 @@ static void t_reftable_index_record_comparison(void)\n \t\t\t.u.idx.last_key = REFTABLE_BUF_INIT,\n \t\t},\n \t};\n+\tint cmp;\n+\n \tcheck(!reftable_buf_addstr(&in[0].u.idx.last_key, \"refs/heads/master\"));\n \tcheck(!reftable_buf_addstr(&in[1].u.idx.last_key, \"refs/heads/master\"));\n \tcheck(!reftable_buf_addstr(&in[2].u.idx.last_key, \"refs/heads/branch\"));\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \n \tin[1].u.idx.offset = in[0].u.idx.offset;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tfor (size_t i = 0; i < ARRAY_SIZE(in); i++)\n \t\treftable_record_release(&in[i]);\n\n-- \n2.48.1.502.g6dc24dfdaf.dirty\n\n"},{"id":"511702","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-7-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"[PATCH v3 07/18] reftable: stop using `BUG()` in trivial cases","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:38Z","receivedAt":"2025-02-03T08:03:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Stop using `BUG()` in the remaining trivial cases that we still have in\nthe reftable library. Instead of aborting the program, we'll now bubble\nup a `REFTABLE_API_ERROR` to indicate misuse of the calling conventions.\n\nNote that in both `reftable_reader_{inc,dec}ref()` we simply stop\ncalling `BUG()` altogether. The only situation where the counter should\nbe zero is when the structure has already been free'd anyway, so we\nwould run into undefined behaviour regardless of whether we try to abort\nthe program or not.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/iter.c | 3 +--\n reftable/reader.c | 4 ----\n reftable/writer.c | 5 ++---\n 3 files changed, 3 insertions(+), 9 deletions(-)\n\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex 86e801ca9f..b2ffb09c16 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -146,8 +146,7 @@ static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n static int indexed_table_ref_iter_seek(void *p UNUSED,\n \t\t\t\t struct reftable_record *want UNUSED)\n {\n-\tBUG(\"seeking indexed table is not supported\");\n-\treturn -1;\n+\treturn REFTABLE_API_ERROR;\n }\n \n static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex de6e6dd932..36a5633ede 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -677,8 +677,6 @@ int reftable_reader_new(struct reftable_reader **out,\n \n void reftable_reader_incref(struct reftable_reader *r)\n {\n-\tif (!r->refcount)\n-\t\tBUG(\"cannot increment ref counter of dead reader\");\n \tr->refcount++;\n }\n \n@@ -686,8 +684,6 @@ void reftable_reader_decref(struct reftable_reader *r)\n {\n \tif (!r)\n \t\treturn;\n-\tif (!r->refcount)\n-\t\tBUG(\"cannot decrement ref counter of dead reader\");\n \tif (--r->refcount)\n \t\treturn;\n \tblock_source_close(&r->source);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 91d6629486..155863ee5f 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -158,7 +158,7 @@ int reftable_writer_new(struct reftable_writer **out,\n \t\topts = *_opts;\n \toptions_set_defaults(&opts);\n \tif (opts.block_size >= (1 << 24))\n-\t\tBUG(\"configured block size exceeds 16MB\");\n+\t\treturn REFTABLE_API_ERROR;\n \n \treftable_buf_init(&wp->block_writer_data.last_key);\n \treftable_buf_init(&wp->last_key);\n@@ -289,8 +289,7 @@ static int writer_add_record(struct reftable_writer *w,\n \t}\n \n \tif (block_writer_type(w->block_writer) != reftable_record_type(rec))\n-\t\tBUG(\"record of type %d added to writer of type %d\",\n-\t\t reftable_record_type(rec), block_writer_type(w->block_writer));\n+\t\treturn REFTABLE_API_ERROR;\n \n \t/*\n \t * Try to add the record to the writer. If this succeeds then we're\n\n-- \n2.48.1.502.g6dc24dfdaf.dirty\n\n"},{"id":"511703","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-8-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"[PATCH v3 08/18] reftable/basics: stop using `st_mult()` in array allocators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:39Z","receivedAt":"2025-02-03T08:03:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're using `st_mult()` as part of our macro helpers that allocate\narrays. This is bad due two two reasons:\n\n - `st_mult()` causes us to die in case the multiplication overflows.\n\n - `st_mult()` ties us to the Git codebase.\n\nRefactor the code to instead detect overflows manually and return an\nerror in such cases.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 36 ++++++++++++++++++++++++++++++++----\n 1 file changed, 32 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex a2a010a0e1..646f8d67f2 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -117,18 +117,46 @@ void reftable_free(void *p);\n void *reftable_calloc(size_t nelem, size_t elsize);\n char *reftable_strdup(const char *str);\n \n-#define REFTABLE_ALLOC_ARRAY(x, alloc) (x) = reftable_malloc(st_mult(sizeof(*(x)), (alloc)))\n+static inline int reftable_alloc_size(size_t nelem, size_t elsize, size_t *out)\n+{\n+\tif (nelem && elsize > SIZE_MAX / nelem)\n+\t\treturn -1;\n+\t*out = nelem * elsize;\n+\treturn 0;\n+}\n+\n+#define REFTABLE_ALLOC_ARRAY(x, alloc) do { \\\n+\t\tsize_t alloc_size; \\\n+\t\tif (reftable_alloc_size(sizeof(*(x)), (alloc), &alloc_size) < 0) { \\\n+\t\t\terrno = ENOMEM; \\\n+\t\t\t(x) = NULL; \\\n+\t\t} else { \\\n+\t\t\t(x) = reftable_malloc(alloc_size); \\\n+\t\t} \\\n+\t} while (0)\n #define REFTABLE_CALLOC_ARRAY(x, alloc) (x) = reftable_calloc((alloc), sizeof(*(x)))\n-#define REFTABLE_REALLOC_ARRAY(x, alloc) (x) = reftable_realloc((x), st_mult(sizeof(*(x)), (alloc)))\n+#define REFTABLE_REALLOC_ARRAY(x, alloc) do { \\\n+\t\tsize_t alloc_size; \\\n+\t\tif (reftable_alloc_size(sizeof(*(x)), (alloc), &alloc_size) < 0) { \\\n+\t\t\terrno = ENOMEM; \\\n+\t\t\t(x) = NULL; \\\n+\t\t} else { \\\n+\t\t\t(x) = reftable_realloc((x), alloc_size); \\\n+\t\t} \\\n+\t} while (0)\n \n static inline void *reftable_alloc_grow(void *p, size_t nelem, size_t elsize,\n \t\t\t\t\tsize_t *allocp)\n {\n \tvoid *new_p;\n-\tsize_t alloc = *allocp * 2 + 1;\n+\tsize_t alloc = *allocp * 2 + 1, alloc_bytes;\n \tif (alloc < nelem)\n \t\talloc = nelem;\n-\tnew_p = reftable_realloc(p, st_mult(elsize, alloc));\n+\tif (reftable_alloc_size(elsize, alloc, &alloc_bytes) < 0) {\n+\t\terrno = ENOMEM;\n+\t\treturn p;\n+\t}\n+\tnew_p = reftable_realloc(p, alloc_bytes);\n \tif (!new_p)\n \t\treturn p;\n \t*allocp = alloc;\n\n-- \n2.48.1.502.g6dc24dfdaf.dirty\n\n"},{"id":"511704","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-9-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"[PATCH v3 09/18] reftable/basics: provide wrappers for big endian conversion","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:40Z","receivedAt":"2025-02-03T08:03:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're using a mixture of big endian conversion functions provided by\nboth the reftable library, but also by the Git codebase. Refactor the\ncode so that we exclusively use reftable-provided wrappers in order to\nuntangle us from the Git codebase.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 19 ----------\n reftable/basics.h | 76 ++++++++++++++++++++++++++++++++++++++--\n reftable/block.c | 12 +++----\n reftable/reader.c | 22 ++++++------\n reftable/record.c | 8 ++---\n reftable/writer.c | 20 +++++------\n t/unit-tests/t-reftable-basics.c | 28 ++++++++++++---\n 7 files changed, 127 insertions(+), 58 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 3b5ea27bbd..8c4a4433e4 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -147,25 +147,6 @@ char *reftable_buf_detach(struct reftable_buf *buf)\n \treturn result;\n }\n \n-void put_be24(uint8_t *out, uint32_t i)\n-{\n-\tout[0] = (uint8_t)((i >> 16) & 0xff);\n-\tout[1] = (uint8_t)((i >> 8) & 0xff);\n-\tout[2] = (uint8_t)(i & 0xff);\n-}\n-\n-uint32_t get_be24(uint8_t *in)\n-{\n-\treturn (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 |\n-\t (uint32_t)(in[2]);\n-}\n-\n-void put_be16(uint8_t *out, uint16_t i)\n-{\n-\tout[0] = (uint8_t)((i >> 8) & 0xff);\n-\tout[1] = (uint8_t)(i & 0xff);\n-}\n-\n size_t binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n {\n \tsize_t lo = 0;\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 646f8d67f2..c1ddbaec3f 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -76,9 +76,79 @@ char *reftable_buf_detach(struct reftable_buf *buf);\n \n /* Bigendian en/decoding of integers */\n \n-void put_be24(uint8_t *out, uint32_t i);\n-uint32_t get_be24(uint8_t *in);\n-void put_be16(uint8_t *out, uint16_t i);\n+static inline void reftable_put_be16(void *out, uint16_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 8) & 0xff);\n+\tp[1] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline void reftable_put_be24(void *out, uint32_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 16) & 0xff);\n+\tp[1] = (uint8_t)((i >> 8) & 0xff);\n+\tp[2] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline void reftable_put_be32(void *out, uint32_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 24) & 0xff);\n+\tp[1] = (uint8_t)((i >> 16) & 0xff);\n+\tp[2] = (uint8_t)((i >> 8) & 0xff);\n+\tp[3] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline void reftable_put_be64(void *out, uint64_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 56) & 0xff);\n+\tp[1] = (uint8_t)((i >> 48) & 0xff);\n+\tp[2] = (uint8_t)((i >> 40) & 0xff);\n+\tp[3] = (uint8_t)((i >> 32) & 0xff);\n+\tp[4] = (uint8_t)((i >> 24) & 0xff);\n+\tp[5] = (uint8_t)((i >> 16) & 0xff);\n+\tp[6] = (uint8_t)((i >> 8) & 0xff);\n+\tp[7] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline uint16_t reftable_get_be16(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint16_t)(p[0]) << 8 |\n+\t (uint16_t)(p[1]) << 0;\n+}\n+\n+static inline uint32_t reftable_get_be24(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint32_t)(p[0]) << 16 |\n+\t (uint32_t)(p[1]) << 8 |\n+\t (uint32_t)(p[2]) << 0;\n+}\n+\n+static inline uint32_t reftable_get_be32(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint32_t)(p[0]) << 24 |\n+\t (uint32_t)(p[1]) << 16 |\n+\t (uint32_t)(p[2]) << 8|\n+\t (uint32_t)(p[3]) << 0;\n+}\n+\n+static inline uint64_t reftable_get_be64(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint64_t)(p[0]) << 56 |\n+\t (uint64_t)(p[1]) << 48 |\n+\t (uint64_t)(p[2]) << 40 |\n+\t (uint64_t)(p[3]) << 32 |\n+\t (uint64_t)(p[4]) << 24 |\n+\t (uint64_t)(p[5]) << 16 |\n+\t (uint64_t)(p[6]) << 8 |\n+\t (uint64_t)(p[7]) << 0;\n+}\n \n /*\n * find smallest index i in [0, sz) at which `f(i) > 0`, assuming that f is\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 255d566854..373908807e 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -148,13 +148,13 @@ int block_writer_add(struct block_writer *w, struct reftable_record *rec)\n int block_writer_finish(struct block_writer *w)\n {\n \tfor (uint32_t i = 0; i < w->restart_len; i++) {\n-\t\tput_be24(w->block + w->next, w->restarts[i]);\n+\t\treftable_put_be24(w->block + w->next, w->restarts[i]);\n \t\tw->next += 3;\n \t}\n \n-\tput_be16(w->block + w->next, w->restart_len);\n+\treftable_put_be16(w->block + w->next, w->restart_len);\n \tw->next += 2;\n-\tput_be24(w->block + 1 + w->header_off, w->next);\n+\treftable_put_be24(w->block + 1 + w->header_off, w->next);\n \n \t/*\n \t * Log records are stored zlib-compressed. Note that the compression\n@@ -216,7 +216,7 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,\n {\n \tuint32_t full_block_size = table_block_size;\n \tuint8_t typ = block->data[header_off];\n-\tuint32_t sz = get_be24(block->data + header_off + 1);\n+\tuint32_t sz = reftable_get_be24(block->data + header_off + 1);\n \tint err = 0;\n \tuint16_t restart_count = 0;\n \tuint32_t restart_start = 0;\n@@ -300,7 +300,7 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,\n \t\tfull_block_size = sz;\n \t}\n \n-\trestart_count = get_be16(block->data + sz - 2);\n+\trestart_count = reftable_get_be16(block->data + sz - 2);\n \trestart_start = sz - 2 - 3 * restart_count;\n \trestart_bytes = block->data + restart_start;\n \n@@ -355,7 +355,7 @@ int block_reader_first_key(const struct block_reader *br, struct reftable_buf *k\n \n static uint32_t block_reader_restart_offset(const struct block_reader *br, size_t idx)\n {\n-\treturn get_be24(br->restart_bytes + 3 * idx);\n+\treturn reftable_get_be24(br->restart_bytes + 3 * idx);\n }\n \n void block_iter_seek_start(struct block_iter *it, const struct block_reader *br)\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 36a5633ede..bf07a0a586 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -101,18 +101,18 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n \t}\n \n \tf++;\n-\tr->block_size = get_be24(f);\n+\tr->block_size = reftable_get_be24(f);\n \n \tf += 3;\n-\tr->min_update_index = get_be64(f);\n+\tr->min_update_index = reftable_get_be64(f);\n \tf += 8;\n-\tr->max_update_index = get_be64(f);\n+\tr->max_update_index = reftable_get_be64(f);\n \tf += 8;\n \n \tif (r->version == 1) {\n \t\tr->hash_id = REFTABLE_HASH_SHA1;\n \t} else {\n-\t\tswitch (get_be32(f)) {\n+\t\tswitch (reftable_get_be32(f)) {\n \t\tcase REFTABLE_FORMAT_ID_SHA1:\n \t\t\tr->hash_id = REFTABLE_HASH_SHA1;\n \t\t\tbreak;\n@@ -127,24 +127,24 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n \t\tf += 4;\n \t}\n \n-\tr->ref_offsets.index_offset = get_be64(f);\n+\tr->ref_offsets.index_offset = reftable_get_be64(f);\n \tf += 8;\n \n-\tr->obj_offsets.offset = get_be64(f);\n+\tr->obj_offsets.offset = reftable_get_be64(f);\n \tf += 8;\n \n \tr->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1);\n \tr->obj_offsets.offset >>= 5;\n \n-\tr->obj_offsets.index_offset = get_be64(f);\n+\tr->obj_offsets.index_offset = reftable_get_be64(f);\n \tf += 8;\n-\tr->log_offsets.offset = get_be64(f);\n+\tr->log_offsets.offset = reftable_get_be64(f);\n \tf += 8;\n-\tr->log_offsets.index_offset = get_be64(f);\n+\tr->log_offsets.index_offset = reftable_get_be64(f);\n \tf += 8;\n \n \tcomputed_crc = crc32(0, footer, f - footer);\n-\tfile_crc = get_be32(f);\n+\tfile_crc = reftable_get_be32(f);\n \tf += 4;\n \tif (computed_crc != file_crc) {\n \t\terr = REFTABLE_FORMAT_ERROR;\n@@ -214,7 +214,7 @@ static int32_t extract_block_size(uint8_t *data, uint8_t *typ, uint64_t off,\n \n \t*typ = data[0];\n \tif (reftable_is_block_type(*typ)) {\n-\t\tresult = get_be24(data + 1);\n+\t\tresult = reftable_get_be24(data + 1);\n \t}\n \treturn result;\n }\ndiff --git a/reftable/record.c b/reftable/record.c\nindex b39d99fcc7..3552bafa99 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -689,7 +689,7 @@ static int reftable_log_record_key(const void *r, struct reftable_buf *dest)\n \t\treturn err;\n \n \tts = (~ts) - rec->update_index;\n-\tput_be64(&i64[0], ts);\n+\treftable_put_be64(&i64[0], ts);\n \n \terr = reftable_buf_add(dest, i64, sizeof(i64));\n \tif (err < 0)\n@@ -814,7 +814,7 @@ static int reftable_log_record_encode(const void *rec, struct string_view s,\n \tif (s.len < 2)\n \t\treturn -1;\n \n-\tput_be16(s.buf, r->value.update.tz_offset);\n+\treftable_put_be16(s.buf, r->value.update.tz_offset);\n \tstring_view_consume(&s, 2);\n \n \tn = encode_string(\n@@ -846,7 +846,7 @@ static int reftable_log_record_decode(void *rec, struct reftable_buf key,\n \t}\n \n \tmemcpy(r->refname, key.buf, key.len - 8);\n-\tts = get_be64(key.buf + key.len - 8);\n+\tts = reftable_get_be64((unsigned char *)key.buf + key.len - 8);\n \n \tr->update_index = (~max) - ts;\n \n@@ -937,7 +937,7 @@ static int reftable_log_record_decode(void *rec, struct reftable_buf key,\n \t\tgoto done;\n \t}\n \n-\tr->value.update.tz_offset = get_be16(in.buf);\n+\tr->value.update.tz_offset = reftable_get_be16(in.buf);\n \tstring_view_consume(&in, 2);\n \n \tn = decode_string(scratch, in);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 155863ee5f..5961698311 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -99,9 +99,9 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \n \tdest[4] = writer_version(w);\n \n-\tput_be24(dest + 5, w->opts.block_size);\n-\tput_be64(dest + 8, w->min_update_index);\n-\tput_be64(dest + 16, w->max_update_index);\n+\treftable_put_be24(dest + 5, w->opts.block_size);\n+\treftable_put_be64(dest + 8, w->min_update_index);\n+\treftable_put_be64(dest + 16, w->max_update_index);\n \tif (writer_version(w) == 2) {\n \t\tuint32_t hash_id;\n \n@@ -116,7 +116,7 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \t\t\treturn -1;\n \t\t}\n \n-\t\tput_be32(dest + 24, hash_id);\n+\t\treftable_put_be32(dest + 24, hash_id);\n \t}\n \n \treturn header_size(writer_version(w));\n@@ -717,19 +717,19 @@ int reftable_writer_close(struct reftable_writer *w)\n \t}\n \n \tp += writer_write_header(w, footer);\n-\tput_be64(p, w->stats.ref_stats.index_offset);\n+\treftable_put_be64(p, w->stats.ref_stats.index_offset);\n \tp += 8;\n-\tput_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n+\treftable_put_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n \tp += 8;\n-\tput_be64(p, w->stats.obj_stats.index_offset);\n+\treftable_put_be64(p, w->stats.obj_stats.index_offset);\n \tp += 8;\n \n-\tput_be64(p, w->stats.log_stats.offset);\n+\treftable_put_be64(p, w->stats.log_stats.offset);\n \tp += 8;\n-\tput_be64(p, w->stats.log_stats.index_offset);\n+\treftable_put_be64(p, w->stats.log_stats.index_offset);\n \tp += 8;\n \n-\tput_be32(p, crc32(0, footer, p - footer));\n+\treftable_put_be32(p, crc32(0, footer, p - footer));\n \tp += 4;\n \n \terr = w->flush(w->write_arg);\ndiff --git a/t/unit-tests/t-reftable-basics.c b/t/unit-tests/t-reftable-basics.c\nindex 9ba7eb05ad..c9e751e49e 100644\n--- a/t/unit-tests/t-reftable-basics.c\n+++ b/t/unit-tests/t-reftable-basics.c\n@@ -128,12 +128,30 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \t\treftable_buf_release(&b);\n \t}\n \n-\tif_test (\"put_be24 and get_be24 work\") {\n+\tif_test (\"reftable_put_be64 and reftable_get_be64 work\") {\n+\t\tuint64_t in = 0x1122334455667788;\n+\t\tuint8_t dest[8];\n+\t\tuint64_t out;\n+\t\treftable_put_be64(dest, in);\n+\t\tout = reftable_get_be64(dest);\n+\t\tcheck_int(in, ==, out);\n+\t}\n+\n+\tif_test (\"reftable_put_be32 and reftable_get_be32 work\") {\n+\t\tuint32_t in = 0x11223344;\n+\t\tuint8_t dest[4];\n+\t\tuint32_t out;\n+\t\treftable_put_be32(dest, in);\n+\t\tout = reftable_get_be32(dest);\n+\t\tcheck_int(in, ==, out);\n+\t}\n+\n+\tif_test (\"reftable_put_be24 and reftable_get_be24 work\") {\n \t\tuint32_t in = 0x112233;\n \t\tuint8_t dest[3];\n \t\tuint32_t out;\n-\t\tput_be24(dest, in);\n-\t\tout = get_be24(dest);\n+\t\treftable_put_be24(dest, in);\n+\t\tout = reftable_get_be24(dest);\n \t\tcheck_int(in, ==, out);\n \t}\n \n@@ -141,8 +159,8 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \t\tuint32_t in = 0xfef1;\n \t\tuint8_t dest[3];\n \t\tuint32_t out;\n-\t\tput_be16(dest, in);\n-\t\tout = get_be16(dest);\n+\t\treftable_put_be16(dest, in);\n+\t\tout = reftable_get_be16(dest);\n \t\tcheck_int(in, ==, out);\n \t}\n \n\n-- \n2.48.1.502.g6dc24dfdaf.dirty\n\n"},{"id":"511705","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-10-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"[PATCH v3 10/18] reftable/reader: stop using `ARRAY_SIZE()` macro","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:41Z","receivedAt":"2025-02-03T08:03:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have a single user of the `ARRAY_SIZE()` macro in the reftable\nreader. Drop its use to reduce our dependence on the Git codebase.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/reader.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex bf07a0a586..c3a3674665 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -849,7 +849,7 @@ int reftable_reader_print_blocks(const char *tablename)\n \tprintf(\"header:\\n\");\n \tprintf(\" block_size: %d\\n\", r->block_size);\n \n-\tfor (i = 0; i < ARRAY_SIZE(sections); i++) {\n+\tfor (i = 0; i < sizeof(sections) / sizeof(*sections); i++) {\n \t\terr = table_iter_seek_start(&ti, sections[i].type, 0);\n \t\tif (err < 0)\n \t\t\tgoto done;\n\n-- \n2.48.1.502.g6dc24dfdaf.dirty\n\n"},{"id":"511706","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-11-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"[PATCH v3 11/18] reftable/system: introduce `reftable_rand()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:42Z","receivedAt":"2025-02-03T08:03:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new system-level `reftable_rand()` function that generates a\nsingle unsigned integer for us. The implementation of this function is\nto be provided by the calling codebase, which allows us to more easily\nhook into pre-seeded random number generators.\n\nAdapt the two callsites where we generated random data.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 4 ++--\n reftable/system.c | 5 +++++\n reftable/system.h | 6 ++++++\n 3 files changed, 13 insertions(+), 2 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 46abf51ec5..e22577ec77 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -523,7 +523,7 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\tclose(fd);\n \t\tfd = -1;\n \n-\t\tdelay = delay + (delay * git_rand(CSPRNG_BYTES_INSECURE)) / UINT32_MAX + 1;\n+\t\tdelay = delay + (delay * reftable_rand()) / UINT32_MAX + 1;\n \t\tsleep_millisec(delay);\n \t}\n \n@@ -688,7 +688,7 @@ int reftable_stack_add(struct reftable_stack *st,\n static int format_name(struct reftable_buf *dest, uint64_t min, uint64_t max)\n {\n \tchar buf[100];\n-\tuint32_t rnd = git_rand(CSPRNG_BYTES_INSECURE);\n+\tuint32_t rnd = reftable_rand();\n \tsnprintf(buf, sizeof(buf), \"0x%012\" PRIx64 \"-0x%012\" PRIx64 \"-%08x\",\n \t\t min, max, rnd);\n \treftable_buf_reset(dest);\ndiff --git a/reftable/system.c b/reftable/system.c\nindex adf8e4d30b..e25ccc0da3 100644\n--- a/reftable/system.c\n+++ b/reftable/system.c\n@@ -4,6 +4,11 @@\n #include \"../lockfile.h\"\n #include \"../tempfile.h\"\n \n+uint32_t reftable_rand(void)\n+{\n+\treturn git_rand(CSPRNG_BYTES_INSECURE);\n+}\n+\n int tmpfile_from_pattern(struct reftable_tmpfile *out, const char *pattern)\n {\n \tstruct tempfile *tempfile;\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 7d5f803eeb..3bd4a4e322 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -13,6 +13,12 @@ license that can be found in the LICENSE file or at\n \n #include \"git-compat-util.h\"\n \n+/*\n+ * Return a random 32 bit integer. This function is expected to return\n+ * pre-seeded data.\n+ */\n+uint32_t reftable_rand(void);\n+\n /*\n * An implementation-specific temporary file. By making this specific to the\n * implementation it becomes possible to tie temporary files into any kind of\n\n-- \n2.48.1.502.g6dc24dfdaf.dirty\n\n"},{"id":"511707","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-12-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"[PATCH v3 12/18] reftable/stack: stop using `sleep_millisec()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:43Z","receivedAt":"2025-02-03T08:03:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor our use of `sleep_millisec()` by open-coding it with poll(3p),\nwhich is the current implementation of this function. Ideally, we'd use\na more direct way to sleep, but there is no equivalent to sleep(3p) that\nwould accept milliseconds as input.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex e22577ec77..7b0b10cd38 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -524,7 +524,7 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\tfd = -1;\n \n \t\tdelay = delay + (delay * reftable_rand()) / UINT32_MAX + 1;\n-\t\tsleep_millisec(delay);\n+\t\tpoll(NULL, 0, delay);\n \t}\n \n out:\n\n-- \n2.48.1.502.g6dc24dfdaf.dirty\n\n"},{"id":"511708","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-13-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"[PATCH v3 13/18] reftable/basics: stop using `SWAP()` macro","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:44Z","receivedAt":"2025-02-03T08:03:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Stop using `SWAP()` macro in favor of an open-coded variant of it. Note\nthat this also requires us to open-code the build assert that `SWAP()`\nitself uses to verify that the size of both variables matches.\n\nThis is done to reduce our dependency on the Git codebase.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 9 +++++++++\n reftable/merged.c | 2 +-\n reftable/pq.c | 4 ++--\n reftable/record.c | 16 ++++++++--------\n 4 files changed, 20 insertions(+), 11 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex c1ddbaec3f..59000798f0 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -266,6 +266,15 @@ static inline void *reftable_alloc_grow(void *p, size_t nelem, size_t elsize,\n # define strdup(str) REFTABLE_BANNED(strdup)\n #endif\n \n+#define REFTABLE_SWAP(a, b) do {\t\t\t\t\t\t\t\t\\\n+\tvoid *_swap_a_ptr = &(a);\t\t\t\t\t\t\t\t\\\n+\tvoid *_swap_b_ptr = &(b);\t\t\t\t\t\t\t\t\\\n+\tunsigned char _swap_buffer[sizeof(a) - 2 * sizeof(a) * (sizeof(a) != sizeof(b))];\t\\\n+\tmemcpy(_swap_buffer, _swap_a_ptr, sizeof(a));\t\t\t\t\t\t\\\n+\tmemcpy(_swap_a_ptr, _swap_b_ptr, sizeof(a));\t\t\t\t\t\t\\\n+\tmemcpy(_swap_b_ptr, _swap_buffer, sizeof(a));\t\t\t\t\t\t\\\n+} while (0)\n+\n /* Find the longest shared prefix size of `a` and `b` */\n size_t common_prefix_size(struct reftable_buf *a, struct reftable_buf *b);\n \ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 563864068c..4ff1553772 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -155,7 +155,7 @@ static int merged_iter_next_entry(struct merged_iter *mi,\n \t}\n \n \tmi->advance_index = entry.index;\n-\tSWAP(*rec, *entry.rec);\n+\tREFTABLE_SWAP(*rec, *entry.rec);\n \treturn 0;\n }\n \ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex ef8035cfd9..82394a972d 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -57,7 +57,7 @@ int merged_iter_pqueue_remove(struct merged_iter_pqueue *pq, struct pq_entry *ou\n \n \t\tif (min == i)\n \t\t\tbreak;\n-\t\tSWAP(pq->heap[i], pq->heap[min]);\n+\t\tREFTABLE_SWAP(pq->heap[i], pq->heap[min]);\n \t\ti = min;\n \t}\n \n@@ -81,7 +81,7 @@ int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry\n \t\tsize_t j = (i - 1) / 2;\n \t\tif (pq_less(&pq->heap[j], &pq->heap[i]))\n \t\t\tbreak;\n-\t\tSWAP(pq->heap[j], pq->heap[i]);\n+\t\tREFTABLE_SWAP(pq->heap[j], pq->heap[i]);\n \t\ti = j;\n \t}\n \ndiff --git a/reftable/record.c b/reftable/record.c\nindex 3552bafa99..9a1edf39a0 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -237,11 +237,11 @@ static int reftable_ref_record_copy_from(void *rec, const void *src_rec,\n \tsize_t refname_cap = 0;\n \tint err;\n \n-\tSWAP(refname, ref->refname);\n-\tSWAP(refname_cap, ref->refname_cap);\n+\tREFTABLE_SWAP(refname, ref->refname);\n+\tREFTABLE_SWAP(refname_cap, ref->refname_cap);\n \treftable_ref_record_release(ref);\n-\tSWAP(ref->refname, refname);\n-\tSWAP(ref->refname_cap, refname_cap);\n+\tREFTABLE_SWAP(ref->refname, refname);\n+\tREFTABLE_SWAP(ref->refname_cap, refname_cap);\n \n \tif (src->refname) {\n \t\tsize_t refname_len = strlen(src->refname);\n@@ -376,11 +376,11 @@ static int reftable_ref_record_decode(void *rec, struct reftable_buf key,\n \t\treturn n;\n \tstring_view_consume(&in, n);\n \n-\tSWAP(refname, r->refname);\n-\tSWAP(refname_cap, r->refname_cap);\n+\tREFTABLE_SWAP(refname, r->refname);\n+\tREFTABLE_SWAP(refname_cap, r->refname_cap);\n \treftable_ref_record_release(r);\n-\tSWAP(r->refname, refname);\n-\tSWAP(r->refname_cap, refname_cap);\n+\tREFTABLE_SWAP(r->refname, refname);\n+\tREFTABLE_SWAP(r->refname_cap, refname_cap);\n \n \tREFTABLE_ALLOC_GROW_OR_NULL(r->refname, key.len + 1, r->refname_cap);\n \tif (!r->refname) {\n\n-- \n2.48.1.502.g6dc24dfdaf.dirty\n\n"},{"id":"511709","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-14-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"[PATCH v3 14/18] reftable/basics: stop using `UNUSED` annotation","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:45Z","receivedAt":"2025-02-03T08:03:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Stop using the `UNUSED` annotation and replace it with a new\n`REFTABLE_UNUSED` macro. The latter is a weaker guarantee compared to\n`UNUSED` as it only suppresses unused parameters without generating a\nwarning in case a parameter marked as unused is in fact used. But it's\ngood enough, and by relaxing the behaviour a bit we avoid having to wire\nup compiler-specific logic.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 2 ++\n reftable/blocksource.c | 10 +++++++---\n reftable/iter.c | 17 ++++++++++++-----\n reftable/record.c | 51 ++++++++++++++++++++++++++++++++++++--------------\n reftable/writer.c | 4 +++-\n 5 files changed, 61 insertions(+), 23 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 59000798f0..4d0645a4e9 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -16,6 +16,8 @@ license that can be found in the LICENSE file or at\n #include \"system.h\"\n #include \"reftable-basics.h\"\n \n+#define REFTABLE_UNUSED(x) (void)(x)\n+\n struct reftable_buf {\n \tsize_t alloc;\n \tsize_t len;\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex 02972c46f4..bfd64b0e48 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -13,15 +13,17 @@ license that can be found in the LICENSE file or at\n #include \"reftable-blocksource.h\"\n #include \"reftable-error.h\"\n \n-static void reftable_buf_return_block(void *b UNUSED, struct reftable_block *dest)\n+static void reftable_buf_return_block(void *b, struct reftable_block *dest)\n {\n+\tREFTABLE_UNUSED(b);\n \tif (dest->len)\n \t\tmemset(dest->data, 0xff, dest->len);\n \treftable_free(dest->data);\n }\n \n-static void reftable_buf_close(void *b UNUSED)\n+static void reftable_buf_close(void *b)\n {\n+\tREFTABLE_UNUSED(b);\n }\n \n static ssize_t reftable_buf_read_block(void *v, struct reftable_block *dest,\n@@ -67,8 +69,10 @@ static uint64_t file_size(void *b)\n \treturn ((struct file_block_source *)b)->size;\n }\n \n-static void file_return_block(void *b UNUSED, struct reftable_block *dest UNUSED)\n+static void file_return_block(void *b, struct reftable_block *dest)\n {\n+\tREFTABLE_UNUSED(b);\n+\tREFTABLE_UNUSED(dest);\n }\n \n static void file_close(void *v)\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex b2ffb09c16..452add2705 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -25,18 +25,23 @@ int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n \treturn it->ops->next(it->iter_arg, rec);\n }\n \n-static int empty_iterator_seek(void *arg UNUSED, struct reftable_record *want UNUSED)\n+static int empty_iterator_seek(void *arg, struct reftable_record *want)\n {\n+\tREFTABLE_UNUSED(arg);\n+\tREFTABLE_UNUSED(want);\n \treturn 0;\n }\n \n-static int empty_iterator_next(void *arg UNUSED, struct reftable_record *rec UNUSED)\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n {\n+\tREFTABLE_UNUSED(arg);\n+\tREFTABLE_UNUSED(rec);\n \treturn 1;\n }\n \n-static void empty_iterator_close(void *arg UNUSED)\n+static void empty_iterator_close(void *arg)\n {\n+\tREFTABLE_UNUSED(arg);\n }\n \n static struct reftable_iterator_vtable empty_vtable = {\n@@ -143,9 +148,11 @@ static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n \treturn 0;\n }\n \n-static int indexed_table_ref_iter_seek(void *p UNUSED,\n-\t\t\t\t struct reftable_record *want UNUSED)\n+static int indexed_table_ref_iter_seek(void *p,\n+\t\t\t\t struct reftable_record *want)\n {\n+\tREFTABLE_UNUSED(p);\n+\tREFTABLE_UNUSED(want);\n \treturn REFTABLE_API_ERROR;\n }\n \ndiff --git a/reftable/record.c b/reftable/record.c\nindex 9a1edf39a0..5ee2fe44a7 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -490,11 +490,13 @@ static void reftable_obj_record_release(void *rec)\n }\n \n static int reftable_obj_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t\t uint32_t hash_size)\n {\n \tstruct reftable_obj_record *obj = rec;\n \tconst struct reftable_obj_record *src = src_rec;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\n \treftable_obj_record_release(obj);\n \n \tREFTABLE_ALLOC_ARRAY(obj->hash_prefix, src->hash_prefix_len);\n@@ -528,13 +530,16 @@ static uint8_t reftable_obj_record_val_type(const void *rec)\n }\n \n static int reftable_obj_record_encode(const void *rec, struct string_view s,\n-\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t uint32_t hash_size)\n {\n \tconst struct reftable_obj_record *r = rec;\n \tstruct string_view start = s;\n \tint i = 0;\n \tint n = 0;\n \tuint64_t last = 0;\n+\n+\tREFTABLE_UNUSED(hash_size);\n+\n \tif (r->offset_len == 0 || r->offset_len >= 8) {\n \t\tn = put_var_int(&s, r->offset_len);\n \t\tif (n < 0) {\n@@ -563,8 +568,8 @@ static int reftable_obj_record_encode(const void *rec, struct string_view s,\n \n static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n \t\t\t\t uint8_t val_type, struct string_view in,\n-\t\t\t\t uint32_t hash_size UNUSED,\n-\t\t\t\t struct reftable_buf *scratch UNUSED)\n+\t\t\t\t uint32_t hash_size,\n+\t\t\t\t struct reftable_buf *scratch)\n {\n \tstruct string_view start = in;\n \tstruct reftable_obj_record *r = rec;\n@@ -572,6 +577,9 @@ static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n \tint n = 0;\n \tuint64_t last;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\tREFTABLE_UNUSED(scratch);\n+\n \treftable_obj_record_release(r);\n \n \tREFTABLE_ALLOC_ARRAY(r->hash_prefix, key.len);\n@@ -618,17 +626,20 @@ static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n \treturn start.len - in.len;\n }\n \n-static int not_a_deletion(const void *p UNUSED)\n+static int not_a_deletion(const void *p)\n {\n+\tREFTABLE_UNUSED(p);\n \treturn 0;\n }\n \n static int reftable_obj_record_equal_void(const void *a, const void *b,\n-\t\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t\t uint32_t hash_size)\n {\n \tstruct reftable_obj_record *ra = (struct reftable_obj_record *) a;\n \tstruct reftable_obj_record *rb = (struct reftable_obj_record *) b;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\n \tif (ra->hash_prefix_len != rb->hash_prefix_len\n \t || ra->offset_len != rb->offset_len)\n \t\treturn 0;\n@@ -1054,12 +1065,14 @@ static int reftable_index_record_key(const void *r, struct reftable_buf *dest)\n }\n \n static int reftable_index_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t\t uint32_t hash_size)\n {\n \tstruct reftable_index_record *dst = rec;\n \tconst struct reftable_index_record *src = src_rec;\n \tint err;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\n \treftable_buf_reset(&dst->last_key);\n \terr = reftable_buf_add(&dst->last_key, src->last_key.buf, src->last_key.len);\n \tif (err < 0)\n@@ -1075,19 +1088,23 @@ static void reftable_index_record_release(void *rec)\n \treftable_buf_release(&idx->last_key);\n }\n \n-static uint8_t reftable_index_record_val_type(const void *rec UNUSED)\n+static uint8_t reftable_index_record_val_type(const void *rec)\n {\n+\tREFTABLE_UNUSED(rec);\n \treturn 0;\n }\n \n static int reftable_index_record_encode(const void *rec, struct string_view out,\n-\t\t\t\t\tuint32_t hash_size UNUSED)\n+\t\t\t\t\tuint32_t hash_size)\n {\n \tconst struct reftable_index_record *r =\n \t\t(const struct reftable_index_record *)rec;\n \tstruct string_view start = out;\n+\tint n;\n \n-\tint n = put_var_int(&out, r->offset);\n+\tREFTABLE_UNUSED(hash_size);\n+\n+\tn = put_var_int(&out, r->offset);\n \tif (n < 0)\n \t\treturn n;\n \n@@ -1097,15 +1114,19 @@ static int reftable_index_record_encode(const void *rec, struct string_view out,\n }\n \n static int reftable_index_record_decode(void *rec, struct reftable_buf key,\n-\t\t\t\t\tuint8_t val_type UNUSED,\n+\t\t\t\t\tuint8_t val_type,\n \t\t\t\t\tstruct string_view in,\n-\t\t\t\t\tuint32_t hash_size UNUSED,\n-\t\t\t\t\tstruct reftable_buf *scratch UNUSED)\n+\t\t\t\t\tuint32_t hash_size,\n+\t\t\t\t\tstruct reftable_buf *scratch)\n {\n \tstruct string_view start = in;\n \tstruct reftable_index_record *r = rec;\n \tint err, n = 0;\n \n+\tREFTABLE_UNUSED(val_type);\n+\tREFTABLE_UNUSED(hash_size);\n+\tREFTABLE_UNUSED(scratch);\n+\n \treftable_buf_reset(&r->last_key);\n \terr = reftable_buf_add(&r->last_key, key.buf, key.len);\n \tif (err < 0)\n@@ -1120,11 +1141,13 @@ static int reftable_index_record_decode(void *rec, struct reftable_buf key,\n }\n \n static int reftable_index_record_equal(const void *a, const void *b,\n-\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t uint32_t hash_size)\n {\n \tstruct reftable_index_record *ia = (struct reftable_index_record *) a;\n \tstruct reftable_index_record *ib = (struct reftable_index_record *) b;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\n \treturn ia->offset == ib->offset && !reftable_buf_cmp(&ia->last_key, &ib->last_key);\n }\n \ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 5961698311..0040a1b1c4 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -636,10 +636,12 @@ static void write_object_record(void *void_arg, void *key)\n done:;\n }\n \n-static void object_record_free(void *void_arg UNUSED, void *key)\n+static void object_record_free(void *void_arg, void *key)\n {\n \tstruct obj_index_tree_node *entry = key;\n \n+\tREFTABLE_UNUSED(void_arg);\n+\n \tREFTABLE_FREE_AND_NULL(entry->offsets);\n \treftable_buf_release(&entry->hash);\n \treftable_free(entry);\n\n-- \n2.48.1.502.g6dc24dfdaf.dirty\n\n"},{"id":"511710","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-15-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"[PATCH v3 15/18] compat/mingw: split out POSIX-related bits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:46Z","receivedAt":"2025-02-03T08:03:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Split out the POSIX-related bits from \"compat/mingw.h\". This is in\npreparation for splitting up \"git-compat-utils.h\" into a header that\nprovides POSIX-compatibility and a header that provides common wrappers\nused by the Git project.\n\nSigned-off-by: Patrick Steinhardt \n---\n compat/{mingw.c => mingw/compat-util.c} | 0\n compat/mingw/compat-util.h | 220 ++++++++++++++++++++++++++++++++\n compat/{mingw.h => mingw/posix.h} | 216 +------------------------------\n compat/{msvc.c => msvc/compat-util.c} | 2 +-\n compat/msvc/compat-util.h | 7 +\n compat/{msvc.h => msvc/posix.h} | 8 +-\n config.mak.uname | 6 +-\n contrib/buildsystems/CMakeLists.txt | 2 +-\n git-compat-util.h | 4 +-\n meson.build | 8 +-\n 10 files changed, 247 insertions(+), 226 deletions(-)\n\ndiff --git a/compat/mingw.c b/compat/mingw/compat-util.c\nsimilarity index 100%\nrename from compat/mingw.c\nrename to compat/mingw/compat-util.c\ndiff --git a/compat/mingw/compat-util.h b/compat/mingw/compat-util.h\nnew file mode 100644\nindex 0000000000..470542d309\n--- /dev/null\n+++ b/compat/mingw/compat-util.h\n@@ -0,0 +1,220 @@\n+#ifndef COMPAT_MINGW_COMPAT_UTIL_H\n+#define COMPAT_MINGW_COMPAT_UTIL_H\n+\n+#include \"posix.h\"\n+\n+struct config_context;\n+int mingw_core_config(const char *var, const char *value,\n+\t\t const struct config_context *ctx, void *cb);\n+#define platform_core_config mingw_core_config\n+\n+#ifndef NO_OPENSSL\n+#include \n+static inline int mingw_SSL_set_fd(SSL *ssl, int fd)\n+{\n+\treturn SSL_set_fd(ssl, _get_osfhandle(fd));\n+}\n+#define SSL_set_fd mingw_SSL_set_fd\n+\n+static inline int mingw_SSL_set_rfd(SSL *ssl, int fd)\n+{\n+\treturn SSL_set_rfd(ssl, _get_osfhandle(fd));\n+}\n+#define SSL_set_rfd mingw_SSL_set_rfd\n+\n+static inline int mingw_SSL_set_wfd(SSL *ssl, int fd)\n+{\n+\treturn SSL_set_wfd(ssl, _get_osfhandle(fd));\n+}\n+#define SSL_set_wfd mingw_SSL_set_wfd\n+#endif\n+\n+/*\n+ * git specific compatibility\n+ */\n+\n+static inline void convert_slashes(char *path)\n+{\n+\tfor (; *path; path++)\n+\t\tif (*path == '\\\\')\n+\t\t\t*path = '/';\n+}\n+#define PATH_SEP ';'\n+char *mingw_query_user_email(void);\n+#define query_user_email mingw_query_user_email\n+\n+/**\n+ * Verifies that the specified path is owned by the user running the\n+ * current process.\n+ */\n+int is_path_owned_by_current_sid(const char *path, struct strbuf *report);\n+#define is_path_owned_by_current_user is_path_owned_by_current_sid\n+\n+/**\n+ * Verifies that the given path is a valid one on Windows.\n+ *\n+ * In particular, path segments are disallowed which\n+ *\n+ * - end in a period or a space (except the special directories `.` and `..`).\n+ *\n+ * - contain any of the reserved characters, e.g. `:`, `;`, `*`, etc\n+ *\n+ * - correspond to reserved names (such as `AUX`, `PRN`, etc)\n+ *\n+ * The `allow_literal_nul` parameter controls whether the path `NUL` should\n+ * be considered valid (this makes sense e.g. before opening files, as it is\n+ * perfectly legitimate to open `NUL` on Windows, just as it is to open\n+ * `/dev/null` on Unix/Linux).\n+ *\n+ * Returns 1 upon success, otherwise 0.\n+ */\n+int is_valid_win32_path(const char *path, int allow_literal_nul);\n+#define is_valid_path(path) is_valid_win32_path(path, 0)\n+\n+/**\n+ * Converts UTF-8 encoded string to UTF-16LE.\n+ *\n+ * To support repositories with legacy-encoded file names, invalid UTF-8 bytes\n+ * 0xa0 - 0xff are converted to corresponding printable Unicode chars \\u00a0 -\n+ * \\u00ff, and invalid UTF-8 bytes 0x80 - 0x9f (which would make non-printable\n+ * Unicode) are converted to hex-code.\n+ *\n+ * Lead-bytes not followed by an appropriate number of trail-bytes, over-long\n+ * encodings and 4-byte encodings > \\u10ffff are detected as invalid UTF-8.\n+ *\n+ * Maximum space requirement for the target buffer is two wide chars per UTF-8\n+ * char (((strlen(utf) * 2) + 1) [* sizeof(wchar_t)]).\n+ *\n+ * The maximum space is needed only if the entire input string consists of\n+ * invalid UTF-8 bytes in range 0x80-0x9f, as per the following table:\n+ *\n+ * | | UTF-8 | UTF-16 |\n+ * Code point | UTF-8 sequence | bytes | words | ratio\n+ * --------------+-------------------+-------+--------+-------\n+ * 000000-00007f | 0-7f | 1 | 1 | 1\n+ * 000080-0007ff | c2-df + 80-bf | 2 | 1 | 0.5\n+ * 000800-00ffff | e0-ef + 2 * 80-bf | 3 | 1 | 0.33\n+ * 010000-10ffff | f0-f4 + 3 * 80-bf | 4 | 2 (a) | 0.5\n+ * invalid | 80-9f | 1 | 2 (b) | 2\n+ * invalid | a0-ff | 1 | 1 | 1\n+ *\n+ * (a) encoded as UTF-16 surrogate pair\n+ * (b) encoded as two hex digits\n+ *\n+ * Note that, while the UTF-8 encoding scheme can be extended to 5-byte, 6-byte\n+ * or even indefinite-byte sequences, the largest valid code point \\u10ffff\n+ * encodes as only 4 UTF-8 bytes.\n+ *\n+ * Parameters:\n+ * wcs: wide char target buffer\n+ * utf: string to convert\n+ * wcslen: size of target buffer (in wchar_t's)\n+ * utflen: size of string to convert, or -1 if 0-terminated\n+ *\n+ * Returns:\n+ * length of converted string (_wcslen(wcs)), or -1 on failure\n+ *\n+ * Errors:\n+ * EINVAL: one of the input parameters is invalid (e.g. NULL)\n+ * ERANGE: the output buffer is too small\n+ */\n+int xutftowcsn(wchar_t *wcs, const char *utf, size_t wcslen, int utflen);\n+\n+/**\n+ * Simplified variant of xutftowcsn, assumes input string is \\0-terminated.\n+ */\n+static inline int xutftowcs(wchar_t *wcs, const char *utf, size_t wcslen)\n+{\n+\treturn xutftowcsn(wcs, utf, wcslen, -1);\n+}\n+\n+/**\n+ * Simplified file system specific variant of xutftowcsn, assumes output\n+ * buffer size is MAX_PATH wide chars and input string is \\0-terminated,\n+ * fails with ENAMETOOLONG if input string is too long.\n+ */\n+static inline int xutftowcs_path(wchar_t *wcs, const char *utf)\n+{\n+\tint result = xutftowcsn(wcs, utf, MAX_PATH, -1);\n+\tif (result < 0 && errno == ERANGE)\n+\t\terrno = ENAMETOOLONG;\n+\treturn result;\n+}\n+\n+/**\n+ * Converts UTF-16LE encoded string to UTF-8.\n+ *\n+ * Maximum space requirement for the target buffer is three UTF-8 chars per\n+ * wide char ((_wcslen(wcs) * 3) + 1).\n+ *\n+ * The maximum space is needed only if the entire input string consists of\n+ * UTF-16 words in range 0x0800-0xd7ff or 0xe000-0xffff (i.e. \\u0800-\\uffff\n+ * modulo surrogate pairs), as per the following table:\n+ *\n+ * | | UTF-16 | UTF-8 |\n+ * Code point | UTF-16 sequence | words | bytes | ratio\n+ * --------------+-----------------------+--------+-------+-------\n+ * 000000-00007f | 0000-007f | 1 | 1 | 1\n+ * 000080-0007ff | 0080-07ff | 1 | 2 | 2\n+ * 000800-00ffff | 0800-d7ff / e000-ffff | 1 | 3 | 3\n+ * 010000-10ffff | d800-dbff + dc00-dfff | 2 | 4 | 2\n+ *\n+ * Note that invalid code points > 10ffff cannot be represented in UTF-16.\n+ *\n+ * Parameters:\n+ * utf: target buffer\n+ * wcs: wide string to convert\n+ * utflen: size of target buffer\n+ *\n+ * Returns:\n+ * length of converted string, or -1 on failure\n+ *\n+ * Errors:\n+ * EINVAL: one of the input parameters is invalid (e.g. NULL)\n+ * ERANGE: the output buffer is too small\n+ */\n+int xwcstoutf(char *utf, const wchar_t *wcs, size_t utflen);\n+\n+/*\n+ * A critical section used in the implementation of the spawn\n+ * functions (mingw_spawnv[p]e()) and waitpid(). Initialised in\n+ * the replacement main() macro below.\n+ */\n+extern CRITICAL_SECTION pinfo_cs;\n+\n+/*\n+ * Git, like most portable C applications, implements a main() function. On\n+ * Windows, this main() function would receive parameters encoded in the\n+ * current locale, but Git for Windows would prefer UTF-8 encoded parameters.\n+ *\n+ * To make that happen, we still declare main() here, and then declare and\n+ * implement wmain() (which is the Unicode variant of main()) and compile with\n+ * -municode. This wmain() function reencodes the parameters from UTF-16 to\n+ * UTF-8 format, sets up a couple of other things as required on Windows, and\n+ * then hands off to the main() function.\n+ */\n+int wmain(int argc, const wchar_t **w_argv);\n+int main(int argc, const char **argv);\n+\n+/*\n+ * For debugging: if a problem occurs, say, in a Git process that is spawned\n+ * from another Git process which in turn is spawned from yet another Git\n+ * process, it can be quite daunting to figure out what is going on.\n+ *\n+ * Call this function to open a new MinTTY (this assumes you are in Git for\n+ * Windows' SDK) with a GDB that attaches to the current process right away.\n+ */\n+void open_in_gdb(void);\n+\n+/*\n+ * Used by Pthread API implementation for Windows\n+ */\n+int err_win_to_posix(DWORD winerr);\n+\n+#ifndef NO_UNIX_SOCKETS\n+int mingw_have_unix_sockets(void);\n+#undef have_unix_sockets\n+#define have_unix_sockets mingw_have_unix_sockets\n+#endif\n+\n+#endif /* COMPAT_MINGW_COMPAT_UTIL_H */\ndiff --git a/compat/mingw.h b/compat/mingw/posix.h\nsimilarity index 59%\nrename from compat/mingw.h\nrename to compat/mingw/posix.h\nindex ebfb8ba423..8dddfa818d 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw/posix.h\n@@ -1,3 +1,6 @@\n+#ifndef COMPAT_MINGW_POSIX_H\n+#define COMPAT_MINGW_POSIX_H\n+\n #ifdef __MINGW64_VERSION_MAJOR\n #include \n #include \n@@ -11,11 +14,6 @@ typedef _sigset_t sigset_t;\n #undef _POSIX_THREAD_SAFE_FUNCTIONS\n #endif\n \n-struct config_context;\n-int mingw_core_config(const char *var, const char *value,\n-\t\t const struct config_context *ctx, void *cb);\n-#define platform_core_config mingw_core_config\n-\n /*\n * things that are not available in header files\n */\n@@ -180,27 +178,6 @@ int mingw_kill(pid_t pid, int sig);\n #define locate_in_PATH mingw_locate_in_PATH\n char *mingw_locate_in_PATH(const char *cmd);\n \n-#ifndef NO_OPENSSL\n-#include \n-static inline int mingw_SSL_set_fd(SSL *ssl, int fd)\n-{\n-\treturn SSL_set_fd(ssl, _get_osfhandle(fd));\n-}\n-#define SSL_set_fd mingw_SSL_set_fd\n-\n-static inline int mingw_SSL_set_rfd(SSL *ssl, int fd)\n-{\n-\treturn SSL_set_rfd(ssl, _get_osfhandle(fd));\n-}\n-#define SSL_set_rfd mingw_SSL_set_rfd\n-\n-static inline int mingw_SSL_set_wfd(SSL *ssl, int fd)\n-{\n-\treturn SSL_set_wfd(ssl, _get_osfhandle(fd));\n-}\n-#define SSL_set_wfd mingw_SSL_set_wfd\n-#endif\n-\n /*\n * implementations of missing functions\n */\n@@ -444,19 +421,6 @@ int winansi_dup2(int oldfd, int newfd);\n void winansi_init(void);\n HANDLE winansi_get_osfhandle(int fd);\n \n-/*\n- * git specific compatibility\n- */\n-\n-static inline void convert_slashes(char *path)\n-{\n-\tfor (; *path; path++)\n-\t\tif (*path == '\\\\')\n-\t\t\t*path = '/';\n-}\n-#define PATH_SEP ';'\n-char *mingw_query_user_email(void);\n-#define query_user_email mingw_query_user_email\n #if !defined(__MINGW64_VERSION_MAJOR) && (!defined(_MSC_VER) || _MSC_VER < 1800)\n #define PRIuMAX \"I64u\"\n #define PRId64 \"I64d\"\n@@ -464,176 +428,4 @@ char *mingw_query_user_email(void);\n #include \n #endif\n \n-/**\n- * Verifies that the specified path is owned by the user running the\n- * current process.\n- */\n-int is_path_owned_by_current_sid(const char *path, struct strbuf *report);\n-#define is_path_owned_by_current_user is_path_owned_by_current_sid\n-\n-/**\n- * Verifies that the given path is a valid one on Windows.\n- *\n- * In particular, path segments are disallowed which\n- *\n- * - end in a period or a space (except the special directories `.` and `..`).\n- *\n- * - contain any of the reserved characters, e.g. `:`, `;`, `*`, etc\n- *\n- * - correspond to reserved names (such as `AUX`, `PRN`, etc)\n- *\n- * The `allow_literal_nul` parameter controls whether the path `NUL` should\n- * be considered valid (this makes sense e.g. before opening files, as it is\n- * perfectly legitimate to open `NUL` on Windows, just as it is to open\n- * `/dev/null` on Unix/Linux).\n- *\n- * Returns 1 upon success, otherwise 0.\n- */\n-int is_valid_win32_path(const char *path, int allow_literal_nul);\n-#define is_valid_path(path) is_valid_win32_path(path, 0)\n-\n-/**\n- * Converts UTF-8 encoded string to UTF-16LE.\n- *\n- * To support repositories with legacy-encoded file names, invalid UTF-8 bytes\n- * 0xa0 - 0xff are converted to corresponding printable Unicode chars \\u00a0 -\n- * \\u00ff, and invalid UTF-8 bytes 0x80 - 0x9f (which would make non-printable\n- * Unicode) are converted to hex-code.\n- *\n- * Lead-bytes not followed by an appropriate number of trail-bytes, over-long\n- * encodings and 4-byte encodings > \\u10ffff are detected as invalid UTF-8.\n- *\n- * Maximum space requirement for the target buffer is two wide chars per UTF-8\n- * char (((strlen(utf) * 2) + 1) [* sizeof(wchar_t)]).\n- *\n- * The maximum space is needed only if the entire input string consists of\n- * invalid UTF-8 bytes in range 0x80-0x9f, as per the following table:\n- *\n- * | | UTF-8 | UTF-16 |\n- * Code point | UTF-8 sequence | bytes | words | ratio\n- * --------------+-------------------+-------+--------+-------\n- * 000000-00007f | 0-7f | 1 | 1 | 1\n- * 000080-0007ff | c2-df + 80-bf | 2 | 1 | 0.5\n- * 000800-00ffff | e0-ef + 2 * 80-bf | 3 | 1 | 0.33\n- * 010000-10ffff | f0-f4 + 3 * 80-bf | 4 | 2 (a) | 0.5\n- * invalid | 80-9f | 1 | 2 (b) | 2\n- * invalid | a0-ff | 1 | 1 | 1\n- *\n- * (a) encoded as UTF-16 surrogate pair\n- * (b) encoded as two hex digits\n- *\n- * Note that, while the UTF-8 encoding scheme can be extended to 5-byte, 6-byte\n- * or even indefinite-byte sequences, the largest valid code point \\u10ffff\n- * encodes as only 4 UTF-8 bytes.\n- *\n- * Parameters:\n- * wcs: wide char target buffer\n- * utf: string to convert\n- * wcslen: size of target buffer (in wchar_t's)\n- * utflen: size of string to convert, or -1 if 0-terminated\n- *\n- * Returns:\n- * length of converted string (_wcslen(wcs)), or -1 on failure\n- *\n- * Errors:\n- * EINVAL: one of the input parameters is invalid (e.g. NULL)\n- * ERANGE: the output buffer is too small\n- */\n-int xutftowcsn(wchar_t *wcs, const char *utf, size_t wcslen, int utflen);\n-\n-/**\n- * Simplified variant of xutftowcsn, assumes input string is \\0-terminated.\n- */\n-static inline int xutftowcs(wchar_t *wcs, const char *utf, size_t wcslen)\n-{\n-\treturn xutftowcsn(wcs, utf, wcslen, -1);\n-}\n-\n-/**\n- * Simplified file system specific variant of xutftowcsn, assumes output\n- * buffer size is MAX_PATH wide chars and input string is \\0-terminated,\n- * fails with ENAMETOOLONG if input string is too long.\n- */\n-static inline int xutftowcs_path(wchar_t *wcs, const char *utf)\n-{\n-\tint result = xutftowcsn(wcs, utf, MAX_PATH, -1);\n-\tif (result < 0 && errno == ERANGE)\n-\t\terrno = ENAMETOOLONG;\n-\treturn result;\n-}\n-\n-/**\n- * Converts UTF-16LE encoded string to UTF-8.\n- *\n- * Maximum space requirement for the target buffer is three UTF-8 chars per\n- * wide char ((_wcslen(wcs) * 3) + 1).\n- *\n- * The maximum space is needed only if the entire input string consists of\n- * UTF-16 words in range 0x0800-0xd7ff or 0xe000-0xffff (i.e. \\u0800-\\uffff\n- * modulo surrogate pairs), as per the following table:\n- *\n- * | | UTF-16 | UTF-8 |\n- * Code point | UTF-16 sequence | words | bytes | ratio\n- * --------------+-----------------------+--------+-------+-------\n- * 000000-00007f | 0000-007f | 1 | 1 | 1\n- * 000080-0007ff | 0080-07ff | 1 | 2 | 2\n- * 000800-00ffff | 0800-d7ff / e000-ffff | 1 | 3 | 3\n- * 010000-10ffff | d800-dbff + dc00-dfff | 2 | 4 | 2\n- *\n- * Note that invalid code points > 10ffff cannot be represented in UTF-16.\n- *\n- * Parameters:\n- * utf: target buffer\n- * wcs: wide string to convert\n- * utflen: size of target buffer\n- *\n- * Returns:\n- * length of converted string, or -1 on failure\n- *\n- * Errors:\n- * EINVAL: one of the input parameters is invalid (e.g. NULL)\n- * ERANGE: the output buffer is too small\n- */\n-int xwcstoutf(char *utf, const wchar_t *wcs, size_t utflen);\n-\n-/*\n- * A critical section used in the implementation of the spawn\n- * functions (mingw_spawnv[p]e()) and waitpid(). Initialised in\n- * the replacement main() macro below.\n- */\n-extern CRITICAL_SECTION pinfo_cs;\n-\n-/*\n- * Git, like most portable C applications, implements a main() function. On\n- * Windows, this main() function would receive parameters encoded in the\n- * current locale, but Git for Windows would prefer UTF-8 encoded parameters.\n- *\n- * To make that happen, we still declare main() here, and then declare and\n- * implement wmain() (which is the Unicode variant of main()) and compile with\n- * -municode. This wmain() function reencodes the parameters from UTF-16 to\n- * UTF-8 format, sets up a couple of other things as required on Windows, and\n- * then hands off to the main() function.\n- */\n-int wmain(int argc, const wchar_t **w_argv);\n-int main(int argc, const char **argv);\n-\n-/*\n- * For debugging: if a problem occurs, say, in a Git process that is spawned\n- * from another Git process which in turn is spawned from yet another Git\n- * process, it can be quite daunting to figure out what is going on.\n- *\n- * Call this function to open a new MinTTY (this assumes you are in Git for\n- * Windows' SDK) with a GDB that attaches to the current process right away.\n- */\n-void open_in_gdb(void);\n-\n-/*\n- * Used by Pthread API implementation for Windows\n- */\n-int err_win_to_posix(DWORD winerr);\n-\n-#ifndef NO_UNIX_SOCKETS\n-int mingw_have_unix_sockets(void);\n-#undef have_unix_sockets\n-#define have_unix_sockets mingw_have_unix_sockets\n-#endif\n+#endif /* COMPAT_MINGW_POSIX_H */\ndiff --git a/compat/msvc.c b/compat/msvc/compat-util.c\nsimilarity index 75%\nrename from compat/msvc.c\nrename to compat/msvc/compat-util.c\nindex 71843d7eef..7f228a6399 100644\n--- a/compat/msvc.c\n+++ b/compat/msvc/compat-util.c\n@@ -3,4 +3,4 @@\n #include \n #include \"../strbuf.h\"\n \n-#include \"mingw.c\"\n+#include \"mingw/compat-util.c\"\ndiff --git a/compat/msvc/compat-util.h b/compat/msvc/compat-util.h\nnew file mode 100644\nindex 0000000000..404af236b5\n--- /dev/null\n+++ b/compat/msvc/compat-util.h\n@@ -0,0 +1,7 @@\n+#ifndef COMPAT_MSVC_COMPAT_UTIL_H\n+#define COMPAT_MSVC_COMPAT_UTIL_H\n+\n+#include \"posix.h\"\n+#include \"../mingw/compat-util.h\"\n+\n+#endif /* COMPAT_MSVC_COMPAT_UTIL_H */\ndiff --git a/compat/msvc.h b/compat/msvc/posix.h\nsimilarity index 86%\nrename from compat/msvc.h\nrename to compat/msvc/posix.h\nindex 1d7a8c6145..703a7c2067 100644\n--- a/compat/msvc.h\n+++ b/compat/msvc/posix.h\n@@ -1,5 +1,5 @@\n-#ifndef __MSVC__HEAD\n-#define __MSVC__HEAD\n+#ifndef COMPAT_MSVC_POSIX_H\n+#define COMPAT_MSVC_POSIX_H\n \n #include \n #include \n@@ -28,6 +28,6 @@ typedef int sigset_t;\n /* open for reading, writing, or both (not in fcntl.h) */\n #define O_ACCMODE (_O_RDONLY | _O_WRONLY | _O_RDWR)\n \n-#include \"compat/mingw.h\"\n+#include \"../mingw/posix.h\"\n \n-#endif\n+#endif /* COMPAT_MSVC_POSIX_H */\ndiff --git a/config.mak.uname b/config.mak.uname\nindex b12d4e168a..dbd05adb5d 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -495,7 +495,7 @@ endif\n \tAR = compat/vcbuild/scripts/lib.pl\n \tCFLAGS =\n \tBASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE\n-\tCOMPAT_OBJS = compat/msvc.o compat/winansi.o \\\n+\tCOMPAT_OBJS = compat/msvc/compat-util.o compat/winansi.o \\\n \t\tcompat/win32/flush.o \\\n \t\tcompat/win32/path-utils.o \\\n \t\tcompat/win32/pthread.o compat/win32/syslog.o \\\n@@ -541,7 +541,7 @@ endif\n \n \tEXTRA_PROGRAMS += headless-git$X\n \n-compat/msvc.o: compat/msvc.c compat/mingw.c GIT-CFLAGS\n+compat/msvc/compat-util.o: compat/msvc/compat-util.c compat/mingw/compat-util.c GIT-CFLAGS\n endif\n ifeq ($(uname_S),Interix)\n \tNO_INITGROUPS = YesPlease\n@@ -694,7 +694,7 @@ ifeq ($(uname_S),MINGW)\n \tBASIC_LDFLAGS += -municode\n \tCOMPAT_CFLAGS += -DNOGDI -Icompat -Icompat/win32\n \tCOMPAT_CFLAGS += -DSTRIP_EXTENSION=\\\".exe\\\"\n-\tCOMPAT_OBJS += compat/mingw.o compat/winansi.o \\\n+\tCOMPAT_OBJS += compat/mingw/compat-util.o compat/winansi.o \\\n \t\tcompat/win32/trace2_win32_process_info.o \\\n \t\tcompat/win32/flush.o \\\n \t\tcompat/win32/path-utils.o \\\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 10dc54fdcb..65ce8705f0 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -258,7 +258,7 @@ if(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\t\t\tUSE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP\n \t\t\t\tHAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET HAVE_RTLGENRANDOM)\n \tlist(APPEND compat_SOURCES\n-\t\tcompat/mingw.c\n+\t\tcompat/mingw/compat-util.c\n \t\tcompat/winansi.c\n \t\tcompat/win32/flush.c\n \t\tcompat/win32/path-utils.c\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex e283c46c6f..b96fb98e1e 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -304,10 +304,10 @@ static inline int is_xplatform_dir_sep(int c)\n #if defined(__MINGW32__)\n /* pull in Windows compatibility stuff */\n #include \"compat/win32/path-utils.h\"\n-#include \"compat/mingw.h\"\n+#include \"compat/mingw/compat-util.h\"\n #elif defined(_MSC_VER)\n #include \"compat/win32/path-utils.h\"\n-#include \"compat/msvc.h\"\n+#include \"compat/msvc/compat-util.h\"\n #else\n #include \n #include \ndiff --git a/meson.build b/meson.build\nindex 2297f5e9dd..e8abf798f5 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -1054,7 +1054,6 @@ if host_machine.system() == 'cygwin'\n ]\n elif host_machine.system() == 'windows'\n libgit_sources += [\n- 'compat/mingw.c',\n 'compat/winansi.c',\n 'compat/win32/flush.c',\n 'compat/win32/path-utils.c',\n@@ -1081,6 +1080,9 @@ elif host_machine.system() == 'windows'\n libgit_include_directories += 'compat/win32'\n if compiler.get_id() == 'msvc'\n libgit_include_directories += 'compat/vcbuild/include'\n+ libgit_sources += 'compat/msvc/compat-util.c'\n+ else\n+ libgit_sources += 'compat/mingw/compat-util.c'\n endif\n endif\n \n@@ -1157,7 +1159,7 @@ else\n error('Native regex support requested but not found')\n endif\n \n-# setitimer and friends are provided by compat/mingw.c.\n+# setitimer and friends are provided by compat/mingw/compat-util.c.\n if host_machine.system() != 'windows'\n if not compiler.compiles('''\n #include \n@@ -1237,7 +1239,7 @@ if not compiler.has_function('qsort')\n endif\n libgit_sources += 'compat/qsort_s.c'\n \n-# unsetenv is provided by compat/mingw.c.\n+# unsetenv is provided by compat/mingw/compat-util.c.\n if host_machine.system() != 'windows' and not compiler.has_function('unsetenv')\n libgit_c_args += '-DNO_UNSETENV'\n libgit_sources += 'compat/unsetenv.c'\n\n-- \n2.48.1.502.g6dc24dfdaf.dirty\n\n"},{"id":"511711","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-16-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"[PATCH v3 16/18] git-compat-util.h: split out POSIX-emulating bits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:47Z","receivedAt":"2025-02-03T08:04:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"git-compat-util.h\" header is a treasure trove of various bits and\npieces used throughout the project. It basically mixes two different\nthings into one:\n\n - Providing a POSIX-like interface even on platforms that aren't\n POSIX-compliant.\n\n - Providing low-level functionality that is specific to Git.\n\nThis intermixing is a bit of a problem for the reftable library as we\ndon't want to recreate the POSIX-like interface there. But neither do we\nwant to pull in the Git-specific functionality, as it is otherwise quite\neasy to start depending on the Git codebase again.\n\nSplit out a new header \"compat/posix.h\" that only contains the bits and\npieces relevant for the emulation of POSIX, which we will start using in\nthe next commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n compat/posix.h | 541 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n git-compat-util.h | 531 +----------------------------------------------------\n 2 files changed, 543 insertions(+), 529 deletions(-)\n\ndiff --git a/compat/posix.h b/compat/posix.h\nnew file mode 100644\nindex 0000000000..8aa41f7183\n--- /dev/null\n+++ b/compat/posix.h\n@@ -0,0 +1,541 @@\n+#ifndef COMPAT_POSIX_H\n+#define COMPAT_POSIX_H\n+\n+#define _FILE_OFFSET_BITS 64\n+\n+/*\n+ * Derived from Linux \"Features Test Macro\" header\n+ * Convenience macros to test the versions of gcc (or\n+ * a compatible compiler).\n+ * Use them like this:\n+ * #if GIT_GNUC_PREREQ (2,8)\n+ * ... code requiring gcc 2.8 or later ...\n+ * #endif\n+ *\n+ * This macro of course is not part of POSIX, but we need it for the UNUSED\n+ * macro which is used by some of our POSIX compatibility wrappers.\n+*/\n+#if defined(__GNUC__) && defined(__GNUC_MINOR__)\n+# define GIT_GNUC_PREREQ(maj, min) \\\n+\t((__GNUC__ << 16) + __GNUC_MINOR__ >= ((maj) << 16) + (min))\n+#else\n+ #define GIT_GNUC_PREREQ(maj, min) 0\n+#endif\n+\n+/*\n+ * UNUSED marks a function parameter that is always unused. It also\n+ * can be used to annotate a function, a variable, or a type that is\n+ * always unused.\n+ *\n+ * A callback interface may dictate that a function accepts a\n+ * parameter at that position, but the implementation of the function\n+ * may not need to use the parameter. In such a case, mark the parameter\n+ * with UNUSED.\n+ *\n+ * When a parameter may be used or unused, depending on conditional\n+ * compilation, consider using MAYBE_UNUSED instead.\n+ */\n+#if GIT_GNUC_PREREQ(4, 5)\n+#define UNUSED __attribute__((unused)) \\\n+\t__attribute__((deprecated (\"parameter declared as UNUSED\")))\n+#elif defined(__GNUC__)\n+#define UNUSED __attribute__((unused)) \\\n+\t__attribute__((deprecated))\n+#else\n+#define UNUSED\n+#endif\n+\n+#ifdef __MINGW64__\n+#define _POSIX_C_SOURCE 1\n+#elif defined(__sun__)\n+ /*\n+ * On Solaris, when _XOPEN_EXTENDED is set, its header file\n+ * forces the programs to be XPG4v2, defeating any _XOPEN_SOURCE\n+ * setting to say we are XPG5 or XPG6. Also on Solaris,\n+ * XPG6 programs must be compiled with a c99 compiler, while\n+ * non XPG6 programs must be compiled with a pre-c99 compiler.\n+ */\n+# if __STDC_VERSION__ - 0 >= 199901L\n+# define _XOPEN_SOURCE 600\n+# else\n+# define _XOPEN_SOURCE 500\n+# endif\n+#elif !defined(__APPLE__) && !defined(__FreeBSD__) && !defined(__USLC__) && \\\n+ !defined(_M_UNIX) && !defined(__sgi) && !defined(__DragonFly__) && \\\n+ !defined(__TANDEM) && !defined(__QNX__) && !defined(__MirBSD__) && \\\n+ !defined(__CYGWIN__)\n+#define _XOPEN_SOURCE 600 /* glibc2 and AIX 5.3L need 500, OpenBSD needs 600 for S_ISLNK() */\n+#define _XOPEN_SOURCE_EXTENDED 1 /* AIX 5.3L needs this */\n+#endif\n+#define _ALL_SOURCE 1\n+#define _GNU_SOURCE 1\n+#define _BSD_SOURCE 1\n+#define _DEFAULT_SOURCE 1\n+#define _NETBSD_SOURCE 1\n+#define _SGI_SOURCE 1\n+\n+#if defined(WIN32) && !defined(__CYGWIN__) /* Both MinGW and MSVC */\n+# if !defined(_WIN32_WINNT)\n+# define _WIN32_WINNT 0x0600\n+# endif\n+#define WIN32_LEAN_AND_MEAN /* stops windows.h including winsock.h */\n+#include \n+#ifndef NO_UNIX_SOCKETS\n+#include \n+#endif\n+#include \n+#define GIT_WINDOWS_NATIVE\n+#endif\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#ifdef HAVE_STRINGS_H\n+#include /* for strcasecmp() */\n+#endif\n+#include \n+#include \n+#include \n+#ifdef NEEDS_SYS_PARAM_H\n+#include \n+#endif\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#if !defined(NO_POLL_H)\n+#include \n+#elif !defined(NO_SYS_POLL_H)\n+#include \n+#else\n+/* Pull the compat stuff */\n+#include \n+#endif\n+#ifdef HAVE_BSD_SYSCTL\n+#include \n+#endif\n+\n+#if defined(__MINGW32__)\n+#include \"mingw/posix.h\"\n+#elif defined(_MSC_VER)\n+#include \"msvc/posix.h\"\n+#else\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#ifndef NO_SYS_SELECT_H\n+#include \n+#endif\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#ifndef NO_INTTYPES_H\n+#include \n+#else\n+#include \n+#endif\n+#ifdef HAVE_ARC4RANDOM_LIBBSD\n+#include \n+#endif\n+#ifdef HAVE_GETRANDOM\n+#include \n+#endif\n+#ifdef NO_INTPTR_T\n+/*\n+ * On I16LP32, ILP32 and LP64 \"long\" is the safe bet, however\n+ * on LLP86, IL33LLP64 and P64 it needs to be \"long long\",\n+ * while on IP16 and IP16L32 it is \"int\" (resp. \"short\")\n+ * Size needs to match (or exceed) 'sizeof(void *)'.\n+ * We can't take \"long long\" here as not everybody has it.\n+ */\n+typedef long intptr_t;\n+typedef unsigned long uintptr_t;\n+#endif\n+#undef _ALL_SOURCE /* AIX 5.3L defines a struct list with _ALL_SOURCE. */\n+#include \n+#define _ALL_SOURCE 1\n+#endif\n+\n+#ifdef MKDIR_WO_TRAILING_SLASH\n+#define mkdir(a,b) compat_mkdir_wo_trailing_slash((a),(b))\n+int compat_mkdir_wo_trailing_slash(const char*, mode_t);\n+#endif\n+\n+#ifdef time\n+#undef time\n+#endif\n+static inline time_t git_time(time_t *tloc)\n+{\n+\tstruct timeval tv;\n+\n+\t/*\n+\t * Avoid time(NULL), which can disagree with gettimeofday(2)\n+\t * and filesystem timestamps.\n+\t */\n+\tgettimeofday(&tv, NULL);\n+\n+\tif (tloc)\n+\t\t*tloc = tv.tv_sec;\n+\treturn tv.tv_sec;\n+}\n+#define time git_time\n+\n+#ifdef NO_STRUCT_ITIMERVAL\n+struct itimerval {\n+\tstruct timeval it_interval;\n+\tstruct timeval it_value;\n+};\n+#endif\n+\n+#ifdef NO_SETITIMER\n+static inline int git_setitimer(int which UNUSED,\n+\t\t\t\tconst struct itimerval *value UNUSED,\n+\t\t\t\tstruct itimerval *newvalue UNUSED) {\n+\treturn 0; /* pretend success */\n+}\n+#undef setitimer\n+#define setitimer(which,value,ovalue) git_setitimer(which,value,ovalue)\n+#endif\n+\n+#ifndef NO_LIBGEN_H\n+#include \n+#else\n+#define basename gitbasename\n+char *gitbasename(char *);\n+#define dirname gitdirname\n+char *gitdirname(char *);\n+#endif\n+\n+#ifndef NO_ICONV\n+#include \n+#endif\n+\n+/* On most systems would have given us this, but\n+ * not on some systems (e.g. z/OS).\n+ */\n+#ifndef NI_MAXHOST\n+#define NI_MAXHOST 1025\n+#endif\n+\n+#ifndef NI_MAXSERV\n+#define NI_MAXSERV 32\n+#endif\n+\n+/* On most systems would have given us this, but\n+ * not on some systems (e.g. GNU/Hurd).\n+ */\n+#ifndef PATH_MAX\n+#define PATH_MAX 4096\n+#endif\n+\n+#ifndef NAME_MAX\n+#define NAME_MAX 255\n+#endif\n+\n+typedef uintmax_t timestamp_t;\n+#define PRItime PRIuMAX\n+#define parse_timestamp strtoumax\n+#define TIME_MAX UINTMAX_MAX\n+#define TIME_MIN 0\n+\n+int lstat_cache_aware_rmdir(const char *path);\n+#if !defined(__MINGW32__) && !defined(_MSC_VER)\n+#define rmdir lstat_cache_aware_rmdir\n+#endif\n+\n+#if defined(NO_MMAP) || defined(USE_WIN32_MMAP)\n+\n+#ifndef PROT_READ\n+#define PROT_READ 1\n+#define PROT_WRITE 2\n+#define MAP_PRIVATE 1\n+#endif\n+\n+#define mmap git_mmap\n+#define munmap git_munmap\n+void *git_mmap(void *start, size_t length, int prot, int flags, int fd, off_t offset);\n+int git_munmap(void *start, size_t length);\n+\n+#else /* NO_MMAP || USE_WIN32_MMAP */\n+\n+#include \n+\n+#endif /* NO_MMAP || USE_WIN32_MMAP */\n+\n+#ifndef MAP_FAILED\n+#define MAP_FAILED ((void *)-1)\n+#endif\n+\n+#ifdef NEEDS_MODE_TRANSLATION\n+#undef S_IFMT\n+#undef S_IFREG\n+#undef S_IFDIR\n+#undef S_IFLNK\n+#undef S_IFBLK\n+#undef S_IFCHR\n+#undef S_IFIFO\n+#undef S_IFSOCK\n+#define S_IFMT 0170000\n+#define S_IFREG 0100000\n+#define S_IFDIR 0040000\n+#define S_IFLNK 0120000\n+#define S_IFBLK 0060000\n+#define S_IFCHR 0020000\n+#define S_IFIFO 0010000\n+#define S_IFSOCK 0140000\n+#ifdef stat\n+#undef stat\n+#endif\n+#define stat(path, buf) git_stat(path, buf)\n+int git_stat(const char *, struct stat *);\n+#ifdef fstat\n+#undef fstat\n+#endif\n+#define fstat(fd, buf) git_fstat(fd, buf)\n+int git_fstat(int, struct stat *);\n+#ifdef lstat\n+#undef lstat\n+#endif\n+#define lstat(path, buf) git_lstat(path, buf)\n+int git_lstat(const char *, struct stat *);\n+#endif\n+\n+#ifdef NO_PREAD\n+#define pread git_pread\n+ssize_t git_pread(int fd, void *buf, size_t count, off_t offset);\n+#endif\n+\n+#ifdef NO_SETENV\n+#define setenv gitsetenv\n+int gitsetenv(const char *, const char *, int);\n+#endif\n+\n+#ifdef NO_MKDTEMP\n+#define mkdtemp gitmkdtemp\n+char *gitmkdtemp(char *);\n+#endif\n+\n+#ifdef NO_UNSETENV\n+#define unsetenv gitunsetenv\n+int gitunsetenv(const char *);\n+#endif\n+\n+#ifdef NO_STRCASESTR\n+#define strcasestr gitstrcasestr\n+char *gitstrcasestr(const char *haystack, const char *needle);\n+#endif\n+\n+#ifdef NO_STRLCPY\n+#define strlcpy gitstrlcpy\n+size_t gitstrlcpy(char *, const char *, size_t);\n+#endif\n+\n+#ifdef NO_STRTOUMAX\n+#define strtoumax gitstrtoumax\n+uintmax_t gitstrtoumax(const char *, char **, int);\n+#define strtoimax gitstrtoimax\n+intmax_t gitstrtoimax(const char *, char **, int);\n+#endif\n+\n+#ifdef NO_HSTRERROR\n+#define hstrerror githstrerror\n+const char *githstrerror(int herror);\n+#endif\n+\n+#ifdef NO_MEMMEM\n+#define memmem gitmemmem\n+void *gitmemmem(const void *haystack, size_t haystacklen,\n+\t\tconst void *needle, size_t needlelen);\n+#endif\n+\n+#ifdef OVERRIDE_STRDUP\n+#ifdef strdup\n+#undef strdup\n+#endif\n+#define strdup gitstrdup\n+char *gitstrdup(const char *s);\n+#endif\n+\n+#ifdef NO_GETPAGESIZE\n+#define getpagesize() sysconf(_SC_PAGESIZE)\n+#endif\n+\n+#ifndef O_CLOEXEC\n+#define O_CLOEXEC 0\n+#endif\n+\n+#ifdef FREAD_READS_DIRECTORIES\n+# if !defined(SUPPRESS_FOPEN_REDEFINITION)\n+# ifdef fopen\n+# undef fopen\n+# endif\n+# define fopen(a,b) git_fopen(a,b)\n+# endif\n+FILE *git_fopen(const char*, const char*);\n+#endif\n+\n+#ifdef SNPRINTF_RETURNS_BOGUS\n+#ifdef snprintf\n+#undef snprintf\n+#endif\n+#define snprintf git_snprintf\n+int git_snprintf(char *str, size_t maxsize,\n+\t\t const char *format, ...);\n+#ifdef vsnprintf\n+#undef vsnprintf\n+#endif\n+#define vsnprintf git_vsnprintf\n+int git_vsnprintf(char *str, size_t maxsize,\n+\t\t const char *format, va_list ap);\n+#endif\n+\n+#ifdef OPEN_RETURNS_EINTR\n+#undef open\n+#define open git_open_with_retry\n+int git_open_with_retry(const char *path, int flag, ...);\n+#endif\n+\n+#ifdef __GLIBC_PREREQ\n+#if __GLIBC_PREREQ(2, 1)\n+#define HAVE_STRCHRNUL\n+#endif\n+#endif\n+\n+#ifndef HAVE_STRCHRNUL\n+#define strchrnul gitstrchrnul\n+static inline char *gitstrchrnul(const char *s, int c)\n+{\n+\twhile (*s && *s != c)\n+\t\ts++;\n+\treturn (char *)s;\n+}\n+#endif\n+\n+#ifdef NO_INET_PTON\n+int inet_pton(int af, const char *src, void *dst);\n+#endif\n+\n+#ifdef NO_INET_NTOP\n+const char *inet_ntop(int af, const void *src, char *dst, size_t size);\n+#endif\n+\n+#ifdef NO_PTHREADS\n+#define atexit git_atexit\n+int git_atexit(void (*handler)(void));\n+#endif\n+\n+#ifndef HOST_NAME_MAX\n+#define HOST_NAME_MAX 256\n+#endif\n+\n+#include \"sane-ctype.h\"\n+\n+void git_stable_qsort(void *base, size_t nmemb, size_t size,\n+\t\t int(*compar)(const void *, const void *));\n+#ifdef INTERNAL_QSORT\n+#define qsort git_stable_qsort\n+#endif\n+\n+#define QSORT(base, n, compar) sane_qsort((base), (n), sizeof(*(base)), compar)\n+static inline void sane_qsort(void *base, size_t nmemb, size_t size,\n+\t\t\t int(*compar)(const void *, const void *))\n+{\n+\tif (nmemb > 1)\n+\t\tqsort(base, nmemb, size, compar);\n+}\n+\n+#define STABLE_QSORT(base, n, compar) \\\n+\tgit_stable_qsort((base), (n), sizeof(*(base)), compar)\n+\n+#ifndef HAVE_ISO_QSORT_S\n+int git_qsort_s(void *base, size_t nmemb, size_t size,\n+\t\tint (*compar)(const void *, const void *, void *), void *ctx);\n+#define qsort_s git_qsort_s\n+#endif\n+\n+#define QSORT_S(base, n, compar, ctx) do {\t\t\t\\\n+\tif (qsort_s((base), (n), sizeof(*(base)), compar, ctx))\t\\\n+\t\tBUG(\"qsort_s() failed\");\t\t\t\\\n+} while (0)\n+\n+#ifdef NO_NSEC\n+#undef USE_NSEC\n+#define ST_CTIME_NSEC(st) 0\n+#define ST_MTIME_NSEC(st) 0\n+#else\n+#ifdef USE_ST_TIMESPEC\n+#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctimespec.tv_nsec))\n+#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtimespec.tv_nsec))\n+#else\n+#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctim.tv_nsec))\n+#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtim.tv_nsec))\n+#endif\n+#endif\n+\n+#ifndef va_copy\n+/*\n+ * Since an obvious implementation of va_list would be to make it a\n+ * pointer into the stack frame, a simple assignment will work on\n+ * many systems. But let's try to be more portable.\n+ */\n+#ifdef __va_copy\n+#define va_copy(dst, src) __va_copy(dst, src)\n+#else\n+#define va_copy(dst, src) ((dst) = (src))\n+#endif\n+#endif\n+\n+#ifndef _POSIX_THREAD_SAFE_FUNCTIONS\n+static inline void git_flockfile(FILE *fh UNUSED)\n+{\n+\t; /* nothing */\n+}\n+static inline void git_funlockfile(FILE *fh UNUSED)\n+{\n+\t; /* nothing */\n+}\n+#undef flockfile\n+#undef funlockfile\n+#undef getc_unlocked\n+#define flockfile(fh) git_flockfile(fh)\n+#define funlockfile(fh) git_funlockfile(fh)\n+#define getc_unlocked(fh) getc(fh)\n+#endif\n+\n+#ifdef FILENO_IS_A_MACRO\n+int git_fileno(FILE *stream);\n+# ifndef COMPAT_CODE_FILENO\n+# undef fileno\n+# define fileno(p) git_fileno(p)\n+# endif\n+#endif\n+\n+#ifdef NEED_ACCESS_ROOT_HANDLER\n+int git_access(const char *path, int mode);\n+# ifndef COMPAT_CODE_ACCESS\n+# ifdef access\n+# undef access\n+# endif\n+# define access(path, mode) git_access(path, mode)\n+# endif\n+#endif\n+\n+#endif /* COMPAT_POSIX_H */\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex b96fb98e1e..30ad0725fb 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -23,26 +23,9 @@\n #include \n #endif\n \n-struct strbuf;\n-\n-\n-#define _FILE_OFFSET_BITS 64\n+#include \"compat/posix.h\"\n \n-\n-/* Derived from Linux \"Features Test Macro\" header\n- * Convenience macros to test the versions of gcc (or\n- * a compatible compiler).\n- * Use them like this:\n- * #if GIT_GNUC_PREREQ (2,8)\n- * ... code requiring gcc 2.8 or later ...\n- * #endif\n-*/\n-#if defined(__GNUC__) && defined(__GNUC_MINOR__)\n-# define GIT_GNUC_PREREQ(maj, min) \\\n-\t((__GNUC__ << 16) + __GNUC_MINOR__ >= ((maj) << 16) + (min))\n-#else\n- #define GIT_GNUC_PREREQ(maj, min) 0\n-#endif\n+struct strbuf;\n \n #if defined(__GNUC__) || defined(__clang__)\n # define PRAGMA(pragma) _Pragma(#pragma)\n@@ -176,71 +159,6 @@ DISABLE_WARNING(-Wsign-compare)\n /* Approximation of the length of the decimal representation of this type. */\n #define decimal_length(x)\t((int)(sizeof(x) * 2.56 + 0.5) + 1)\n \n-#ifdef __MINGW64__\n-#define _POSIX_C_SOURCE 1\n-#elif defined(__sun__)\n- /*\n- * On Solaris, when _XOPEN_EXTENDED is set, its header file\n- * forces the programs to be XPG4v2, defeating any _XOPEN_SOURCE\n- * setting to say we are XPG5 or XPG6. Also on Solaris,\n- * XPG6 programs must be compiled with a c99 compiler, while\n- * non XPG6 programs must be compiled with a pre-c99 compiler.\n- */\n-# if __STDC_VERSION__ - 0 >= 199901L\n-# define _XOPEN_SOURCE 600\n-# else\n-# define _XOPEN_SOURCE 500\n-# endif\n-#elif !defined(__APPLE__) && !defined(__FreeBSD__) && !defined(__USLC__) && \\\n- !defined(_M_UNIX) && !defined(__sgi) && !defined(__DragonFly__) && \\\n- !defined(__TANDEM) && !defined(__QNX__) && !defined(__MirBSD__) && \\\n- !defined(__CYGWIN__)\n-#define _XOPEN_SOURCE 600 /* glibc2 and AIX 5.3L need 500, OpenBSD needs 600 for S_ISLNK() */\n-#define _XOPEN_SOURCE_EXTENDED 1 /* AIX 5.3L needs this */\n-#endif\n-#define _ALL_SOURCE 1\n-#define _GNU_SOURCE 1\n-#define _BSD_SOURCE 1\n-#define _DEFAULT_SOURCE 1\n-#define _NETBSD_SOURCE 1\n-#define _SGI_SOURCE 1\n-\n-/*\n- * UNUSED marks a function parameter that is always unused. It also\n- * can be used to annotate a function, a variable, or a type that is\n- * always unused.\n- *\n- * A callback interface may dictate that a function accepts a\n- * parameter at that position, but the implementation of the function\n- * may not need to use the parameter. In such a case, mark the parameter\n- * with UNUSED.\n- *\n- * When a parameter may be used or unused, depending on conditional\n- * compilation, consider using MAYBE_UNUSED instead.\n- */\n-#if GIT_GNUC_PREREQ(4, 5)\n-#define UNUSED __attribute__((unused)) \\\n-\t__attribute__((deprecated (\"parameter declared as UNUSED\")))\n-#elif defined(__GNUC__)\n-#define UNUSED __attribute__((unused)) \\\n-\t__attribute__((deprecated))\n-#else\n-#define UNUSED\n-#endif\n-\n-#if defined(WIN32) && !defined(__CYGWIN__) /* Both MinGW and MSVC */\n-# if !defined(_WIN32_WINNT)\n-# define _WIN32_WINNT 0x0600\n-# endif\n-#define WIN32_LEAN_AND_MEAN /* stops windows.h including winsock.h */\n-#include \n-#ifndef NO_UNIX_SOCKETS\n-#include \n-#endif\n-#include \n-#define GIT_WINDOWS_NATIVE\n-#endif\n-\n #if defined(NO_UNIX_SOCKETS) || !defined(GIT_WINDOWS_NATIVE)\n static inline int _have_unix_sockets(void)\n {\n@@ -253,45 +171,6 @@ static inline int _have_unix_sockets(void)\n #define have_unix_sockets _have_unix_sockets\n #endif\n \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#ifdef HAVE_STRINGS_H\n-#include /* for strcasecmp() */\n-#endif\n-#include \n-#include \n-#include \n-#ifdef NEEDS_SYS_PARAM_H\n-#include \n-#endif\n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#if !defined(NO_POLL_H)\n-#include \n-#elif !defined(NO_SYS_POLL_H)\n-#include \n-#else\n-/* Pull the compat stuff */\n-#include \n-#endif\n-#ifdef HAVE_BSD_SYSCTL\n-#include \n-#endif\n-\n /* Used by compat/win32/path-utils.h, and more */\n static inline int is_xplatform_dir_sep(int c)\n {\n@@ -308,48 +187,6 @@ static inline int is_xplatform_dir_sep(int c)\n #elif defined(_MSC_VER)\n #include \"compat/win32/path-utils.h\"\n #include \"compat/msvc/compat-util.h\"\n-#else\n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#ifndef NO_SYS_SELECT_H\n-#include \n-#endif\n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#ifndef NO_INTTYPES_H\n-#include \n-#else\n-#include \n-#endif\n-#ifdef HAVE_ARC4RANDOM_LIBBSD\n-#include \n-#endif\n-#ifdef HAVE_GETRANDOM\n-#include \n-#endif\n-#ifdef NO_INTPTR_T\n-/*\n- * On I16LP32, ILP32 and LP64 \"long\" is the safe bet, however\n- * on LLP86, IL33LLP64 and P64 it needs to be \"long long\",\n- * while on IP16 and IP16L32 it is \"int\" (resp. \"short\")\n- * Size needs to match (or exceed) 'sizeof(void *)'.\n- * We can't take \"long long\" here as not everybody has it.\n- */\n-typedef long intptr_t;\n-typedef unsigned long uintptr_t;\n-#endif\n-#undef _ALL_SOURCE /* AIX 5.3L defines a struct list with _ALL_SOURCE. */\n-#include \n-#define _ALL_SOURCE 1\n #endif\n \n /* used on Mac OS X */\n@@ -370,60 +207,6 @@ static inline const char *precompose_string_if_needed(const char *in)\n #define probe_utf8_pathname_composition()\n #endif\n \n-#ifdef MKDIR_WO_TRAILING_SLASH\n-#define mkdir(a,b) compat_mkdir_wo_trailing_slash((a),(b))\n-int compat_mkdir_wo_trailing_slash(const char*, mode_t);\n-#endif\n-\n-#ifdef time\n-#undef time\n-#endif\n-static inline time_t git_time(time_t *tloc)\n-{\n-\tstruct timeval tv;\n-\n-\t/*\n-\t * Avoid time(NULL), which can disagree with gettimeofday(2)\n-\t * and filesystem timestamps.\n-\t */\n-\tgettimeofday(&tv, NULL);\n-\n-\tif (tloc)\n-\t\t*tloc = tv.tv_sec;\n-\treturn tv.tv_sec;\n-}\n-#define time git_time\n-\n-#ifdef NO_STRUCT_ITIMERVAL\n-struct itimerval {\n-\tstruct timeval it_interval;\n-\tstruct timeval it_value;\n-};\n-#endif\n-\n-#ifdef NO_SETITIMER\n-static inline int git_setitimer(int which UNUSED,\n-\t\t\t\tconst struct itimerval *value UNUSED,\n-\t\t\t\tstruct itimerval *newvalue UNUSED) {\n-\treturn 0; /* pretend success */\n-}\n-#undef setitimer\n-#define setitimer(which,value,ovalue) git_setitimer(which,value,ovalue)\n-#endif\n-\n-#ifndef NO_LIBGEN_H\n-#include \n-#else\n-#define basename gitbasename\n-char *gitbasename(char *);\n-#define dirname gitdirname\n-char *gitdirname(char *);\n-#endif\n-\n-#ifndef NO_ICONV\n-#include \n-#endif\n-\n #ifndef NO_OPENSSL\n #ifdef __APPLE__\n #undef __AVAILABILITY_MACROS_USES_AVAILABILITY\n@@ -441,34 +224,6 @@ char *gitdirname(char *);\n # include \n #endif\n \n-/* On most systems would have given us this, but\n- * not on some systems (e.g. z/OS).\n- */\n-#ifndef NI_MAXHOST\n-#define NI_MAXHOST 1025\n-#endif\n-\n-#ifndef NI_MAXSERV\n-#define NI_MAXSERV 32\n-#endif\n-\n-/* On most systems would have given us this, but\n- * not on some systems (e.g. GNU/Hurd).\n- */\n-#ifndef PATH_MAX\n-#define PATH_MAX 4096\n-#endif\n-\n-#ifndef NAME_MAX\n-#define NAME_MAX 255\n-#endif\n-\n-typedef uintmax_t timestamp_t;\n-#define PRItime PRIuMAX\n-#define parse_timestamp strtoumax\n-#define TIME_MAX UINTMAX_MAX\n-#define TIME_MIN 0\n-\n #ifndef PATH_SEP\n #define PATH_SEP ':'\n #endif\n@@ -492,11 +247,6 @@ static inline int noop_core_config(const char *var UNUSED,\n #define platform_core_config noop_core_config\n #endif\n \n-int lstat_cache_aware_rmdir(const char *path);\n-#if !defined(__MINGW32__) && !defined(_MSC_VER)\n-#define rmdir lstat_cache_aware_rmdir\n-#endif\n-\n #ifndef has_dos_drive_prefix\n static inline int git_has_dos_drive_prefix(const char *path UNUSED)\n {\n@@ -822,25 +572,6 @@ static inline bool strip_suffix(const char *str, const char *suffix,\n \tmemcpy(_swap_b_ptr, _swap_buffer, sizeof(a));\t\t\\\n } while (0)\n \n-#if defined(NO_MMAP) || defined(USE_WIN32_MMAP)\n-\n-#ifndef PROT_READ\n-#define PROT_READ 1\n-#define PROT_WRITE 2\n-#define MAP_PRIVATE 1\n-#endif\n-\n-#define mmap git_mmap\n-#define munmap git_munmap\n-void *git_mmap(void *start, size_t length, int prot, int flags, int fd, off_t offset);\n-int git_munmap(void *start, size_t length);\n-\n-#else /* NO_MMAP || USE_WIN32_MMAP */\n-\n-#include \n-\n-#endif /* NO_MMAP || USE_WIN32_MMAP */\n-\n #ifdef NO_MMAP\n \n /* This value must be multiple of (pagesize * 2) */\n@@ -856,177 +587,15 @@ int git_munmap(void *start, size_t length);\n \n #endif /* NO_MMAP */\n \n-#ifndef MAP_FAILED\n-#define MAP_FAILED ((void *)-1)\n-#endif\n-\n #ifdef NO_ST_BLOCKS_IN_STRUCT_STAT\n #define on_disk_bytes(st) ((st).st_size)\n #else\n #define on_disk_bytes(st) ((st).st_blocks * 512)\n #endif\n \n-#ifdef NEEDS_MODE_TRANSLATION\n-#undef S_IFMT\n-#undef S_IFREG\n-#undef S_IFDIR\n-#undef S_IFLNK\n-#undef S_IFBLK\n-#undef S_IFCHR\n-#undef S_IFIFO\n-#undef S_IFSOCK\n-#define S_IFMT 0170000\n-#define S_IFREG 0100000\n-#define S_IFDIR 0040000\n-#define S_IFLNK 0120000\n-#define S_IFBLK 0060000\n-#define S_IFCHR 0020000\n-#define S_IFIFO 0010000\n-#define S_IFSOCK 0140000\n-#ifdef stat\n-#undef stat\n-#endif\n-#define stat(path, buf) git_stat(path, buf)\n-int git_stat(const char *, struct stat *);\n-#ifdef fstat\n-#undef fstat\n-#endif\n-#define fstat(fd, buf) git_fstat(fd, buf)\n-int git_fstat(int, struct stat *);\n-#ifdef lstat\n-#undef lstat\n-#endif\n-#define lstat(path, buf) git_lstat(path, buf)\n-int git_lstat(const char *, struct stat *);\n-#endif\n-\n #define DEFAULT_PACKED_GIT_LIMIT \\\n \t((1024L * 1024L) * (size_t)(sizeof(void*) >= 8 ? (32 * 1024L * 1024L) : 256))\n \n-#ifdef NO_PREAD\n-#define pread git_pread\n-ssize_t git_pread(int fd, void *buf, size_t count, off_t offset);\n-#endif\n-\n-#ifdef NO_SETENV\n-#define setenv gitsetenv\n-int gitsetenv(const char *, const char *, int);\n-#endif\n-\n-#ifdef NO_MKDTEMP\n-#define mkdtemp gitmkdtemp\n-char *gitmkdtemp(char *);\n-#endif\n-\n-#ifdef NO_UNSETENV\n-#define unsetenv gitunsetenv\n-int gitunsetenv(const char *);\n-#endif\n-\n-#ifdef NO_STRCASESTR\n-#define strcasestr gitstrcasestr\n-char *gitstrcasestr(const char *haystack, const char *needle);\n-#endif\n-\n-#ifdef NO_STRLCPY\n-#define strlcpy gitstrlcpy\n-size_t gitstrlcpy(char *, const char *, size_t);\n-#endif\n-\n-#ifdef NO_STRTOUMAX\n-#define strtoumax gitstrtoumax\n-uintmax_t gitstrtoumax(const char *, char **, int);\n-#define strtoimax gitstrtoimax\n-intmax_t gitstrtoimax(const char *, char **, int);\n-#endif\n-\n-#ifdef NO_HSTRERROR\n-#define hstrerror githstrerror\n-const char *githstrerror(int herror);\n-#endif\n-\n-#ifdef NO_MEMMEM\n-#define memmem gitmemmem\n-void *gitmemmem(const void *haystack, size_t haystacklen,\n-\t\tconst void *needle, size_t needlelen);\n-#endif\n-\n-#ifdef OVERRIDE_STRDUP\n-#ifdef strdup\n-#undef strdup\n-#endif\n-#define strdup gitstrdup\n-char *gitstrdup(const char *s);\n-#endif\n-\n-#ifdef NO_GETPAGESIZE\n-#define getpagesize() sysconf(_SC_PAGESIZE)\n-#endif\n-\n-#ifndef O_CLOEXEC\n-#define O_CLOEXEC 0\n-#endif\n-\n-#ifdef FREAD_READS_DIRECTORIES\n-# if !defined(SUPPRESS_FOPEN_REDEFINITION)\n-# ifdef fopen\n-# undef fopen\n-# endif\n-# define fopen(a,b) git_fopen(a,b)\n-# endif\n-FILE *git_fopen(const char*, const char*);\n-#endif\n-\n-#ifdef SNPRINTF_RETURNS_BOGUS\n-#ifdef snprintf\n-#undef snprintf\n-#endif\n-#define snprintf git_snprintf\n-int git_snprintf(char *str, size_t maxsize,\n-\t\t const char *format, ...);\n-#ifdef vsnprintf\n-#undef vsnprintf\n-#endif\n-#define vsnprintf git_vsnprintf\n-int git_vsnprintf(char *str, size_t maxsize,\n-\t\t const char *format, va_list ap);\n-#endif\n-\n-#ifdef OPEN_RETURNS_EINTR\n-#undef open\n-#define open git_open_with_retry\n-int git_open_with_retry(const char *path, int flag, ...);\n-#endif\n-\n-#ifdef __GLIBC_PREREQ\n-#if __GLIBC_PREREQ(2, 1)\n-#define HAVE_STRCHRNUL\n-#endif\n-#endif\n-\n-#ifndef HAVE_STRCHRNUL\n-#define strchrnul gitstrchrnul\n-static inline char *gitstrchrnul(const char *s, int c)\n-{\n-\twhile (*s && *s != c)\n-\t\ts++;\n-\treturn (char *)s;\n-}\n-#endif\n-\n-#ifdef NO_INET_PTON\n-int inet_pton(int af, const char *src, void *dst);\n-#endif\n-\n-#ifdef NO_INET_NTOP\n-const char *inet_ntop(int af, const void *src, char *dst, size_t size);\n-#endif\n-\n-#ifdef NO_PTHREADS\n-#define atexit git_atexit\n-int git_atexit(void (*handler)(void));\n-#endif\n-\n static inline size_t st_add(size_t a, size_t b)\n {\n \tif (unsigned_add_overflows(a, b))\n@@ -1293,12 +862,6 @@ static inline size_t xsize_t(off_t len)\n \treturn (size_t) len;\n }\n \n-#ifndef HOST_NAME_MAX\n-#define HOST_NAME_MAX 256\n-#endif\n-\n-#include \"sane-ctype.h\"\n-\n /*\n * Like skip_prefix, but compare case-insensitively. Note that the comparison\n * is done via tolower(), so it is strictly ASCII (no multi-byte characters or\n@@ -1364,34 +927,6 @@ static inline int strtol_i(char const *s, int base, int *result)\n \treturn 0;\n }\n \n-void git_stable_qsort(void *base, size_t nmemb, size_t size,\n-\t\t int(*compar)(const void *, const void *));\n-#ifdef INTERNAL_QSORT\n-#define qsort git_stable_qsort\n-#endif\n-\n-#define QSORT(base, n, compar) sane_qsort((base), (n), sizeof(*(base)), compar)\n-static inline void sane_qsort(void *base, size_t nmemb, size_t size,\n-\t\t\t int(*compar)(const void *, const void *))\n-{\n-\tif (nmemb > 1)\n-\t\tqsort(base, nmemb, size, compar);\n-}\n-\n-#define STABLE_QSORT(base, n, compar) \\\n-\tgit_stable_qsort((base), (n), sizeof(*(base)), compar)\n-\n-#ifndef HAVE_ISO_QSORT_S\n-int git_qsort_s(void *base, size_t nmemb, size_t size,\n-\t\tint (*compar)(const void *, const void *, void *), void *ctx);\n-#define qsort_s git_qsort_s\n-#endif\n-\n-#define QSORT_S(base, n, compar, ctx) do {\t\t\t\\\n-\tif (qsort_s((base), (n), sizeof(*(base)), compar, ctx))\t\\\n-\t\tBUG(\"qsort_s() failed\");\t\t\t\\\n-} while (0)\n-\n #ifndef REG_STARTEND\n #error \"Git requires REG_STARTEND support. Compile with NO_REGEX=NeedsStartEnd\"\n #endif\n@@ -1416,39 +951,12 @@ int git_regcomp(regex_t *preg, const char *pattern, int cflags);\n # define FORCE_DIR_SET_GID 0\n #endif\n \n-#ifdef NO_NSEC\n-#undef USE_NSEC\n-#define ST_CTIME_NSEC(st) 0\n-#define ST_MTIME_NSEC(st) 0\n-#else\n-#ifdef USE_ST_TIMESPEC\n-#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctimespec.tv_nsec))\n-#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtimespec.tv_nsec))\n-#else\n-#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctim.tv_nsec))\n-#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtim.tv_nsec))\n-#endif\n-#endif\n-\n #ifdef UNRELIABLE_FSTAT\n #define fstat_is_reliable() 0\n #else\n #define fstat_is_reliable() 1\n #endif\n \n-#ifndef va_copy\n-/*\n- * Since an obvious implementation of va_list would be to make it a\n- * pointer into the stack frame, a simple assignment will work on\n- * many systems. But let's try to be more portable.\n- */\n-#ifdef __va_copy\n-#define va_copy(dst, src) __va_copy(dst, src)\n-#else\n-#define va_copy(dst, src) ((dst) = (src))\n-#endif\n-#endif\n-\n /* usage.c: only to be used for testing BUG() implementation (see test-tool) */\n extern int BUG_exit_code;\n \n@@ -1478,41 +986,6 @@ void bug_fl(const char *file, int line, const char *fmt, ...);\n # define SHELL_PATH \"/bin/sh\"\n #endif\n \n-#ifndef _POSIX_THREAD_SAFE_FUNCTIONS\n-static inline void git_flockfile(FILE *fh UNUSED)\n-{\n-\t; /* nothing */\n-}\n-static inline void git_funlockfile(FILE *fh UNUSED)\n-{\n-\t; /* nothing */\n-}\n-#undef flockfile\n-#undef funlockfile\n-#undef getc_unlocked\n-#define flockfile(fh) git_flockfile(fh)\n-#define funlockfile(fh) git_funlockfile(fh)\n-#define getc_unlocked(fh) getc(fh)\n-#endif\n-\n-#ifdef FILENO_IS_A_MACRO\n-int git_fileno(FILE *stream);\n-# ifndef COMPAT_CODE_FILENO\n-# undef fileno\n-# define fileno(p) git_fileno(p)\n-# endif\n-#endif\n-\n-#ifdef NEED_ACCESS_ROOT_HANDLER\n-int git_access(const char *path, int mode);\n-# ifndef COMPAT_CODE_ACCESS\n-# ifdef access\n-# undef access\n-# endif\n-# define access(path, mode) git_access(path, mode)\n-# endif\n-#endif\n-\n /*\n * Our code often opens a path to an optional file, to work on its\n * contents when we can successfully open it. We can ignore a failure\n\n-- \n2.48.1.502.g6dc24dfdaf.dirty\n\n"},{"id":"511712","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-17-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"[PATCH v3 17/18] reftable: decouple from Git codebase by pulling in \"compat/posix.h\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:48Z","receivedAt":"2025-02-03T08:04:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library includes \"git-compat-util.h\" in order to get a\nPOSIX-like programming environment that papers over various differences\nbetween platforms. The header also brings with it a couple of helpers\nspecific to the Git codebase though, and over time we have started to\nuse these helpers in the reftable library, as well.\n\nThis makes it very hard to use the reftable library as a standalone\nlibrary without the rest of the Git codebase, so other libraries like\ne.g. libgit2 cannot easily use it. But now that we have removed all\ncalls to Git-specific functionality and have split out \"compat/posix.h\"\nas a separate header we can address this.\n\nStop including \"git-compat-util.h\" and instead include \"compat/posix.h\"\nto finalize the decoupling of the reftable library from the rest of the\nGit codebase. The only bits which remain specific to Git are \"system.h\"\nand \"system.c\", which projects will have to provide.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/system.c | 2 ++\n reftable/system.h | 3 ++-\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/reftable/system.c b/reftable/system.c\nindex e25ccc0da3..1ee268b125 100644\n--- a/reftable/system.c\n+++ b/reftable/system.c\n@@ -1,3 +1,5 @@\n+#include \"../git-compat-util.h\"\n+\n #include \"system.h\"\n #include \"basics.h\"\n #include \"reftable-error.h\"\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 3bd4a4e322..dccdf11f76 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -11,7 +11,8 @@ license that can be found in the LICENSE file or at\n \n /* This header glues the reftable library to the rest of Git */\n \n-#include \"git-compat-util.h\"\n+#include \"../compat/posix.h\"\n+#include \n \n /*\n * Return a random 32 bit integer. This function is expected to return\n\n-- \n2.48.1.502.g6dc24dfdaf.dirty\n\n"},{"id":"511713","messageId":"20250203-pks-reftable-drop-git-compat-util-v3-18-446c9ed4ee9e@pks.im","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"[PATCH v3 18/18] Makefile: skip reftable library for Coccinelle","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:03:49Z","receivedAt":"2025-02-03T08:04:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library does not use any of the common helpers that the Git\nproject has. Consequently, most of the rules that we have in Coccinelle\ndo not apply to the library at all and may even generate false positives\nwhen a pattern can be converted to use a Git helper function.\n\nExclude reftable library sources from being checked by Coccinelle to\navoid such false positives.\n\nSigned-off-by: Patrick Steinhardt \n---\n Makefile | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex d3011e30f7..dc2ac32e4a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -946,7 +946,7 @@ FOUND_SOURCE_FILES := $(filter-out $(GENERATED_H),$(shell $(SOURCES_CMD)))\n FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES))\n FOUND_H_SOURCES = $(filter %.h,$(FOUND_SOURCE_FILES))\n \n-COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES))\n+COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES) reftable/%,$(FOUND_C_SOURCES))\n \n LIB_H = $(FOUND_H_SOURCES)\n \n\n-- \n2.48.1.502.g6dc24dfdaf.dirty\n\n"},{"id":"511720","messageId":"Z6CBD9oXQz6oKiCD@pks.im","threadId":"62862","inReplyTo":"ctx7men3stwyasz2xnoaxpp66yn6yfggnfzbawzly57puarazc@p3hpzfn2a7rk","subject":"Re: [PATCH v2 04/20] reftable/record: stop using `COPY_ARRAY()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:40:47Z","receivedAt":"2025-02-03T08:40:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Jan 29, 2025 at 09:46:54AM -0600, Justin Tobler wrote:\n> On 25/01/28 09:28AM, Patrick Steinhardt wrote:\n> > Drop our use of `COPY_ARRAY()`, replacing it with an open-coded variant\n> > thereof. This is done to reduce our dependency on the Git library.\n> > \n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > reftable/record.c | 3 ++-\n> > 1 file changed, 2 insertions(+), 1 deletion(-)\n> > \n> > diff --git a/reftable/record.c b/reftable/record.c\n> > index 8919df8a4d..d1664c47ca 100644\n> > --- a/reftable/record.c\n> > +++ b/reftable/record.c\n> > @@ -508,7 +508,8 @@ static int reftable_obj_record_copy_from(void *rec, const void *src_rec,\n> > \tif (!obj->offsets)\n> > \t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n> > \tobj->offset_len = src->offset_len;\n> > -\tCOPY_ARRAY(obj->offsets, src->offsets, src->offset_len);\n> > +\tif (src->offset_len)\n> > +\t\tmemcpy(obj->offsets, src->offsets, sizeof(*src->offsets) * src->offset_len);\n> \n> The `COPY_ARRAY` version uses `st_mutl()` to protect against potential\n> overflows of the size parameter. Does this variant need to account for\n> such situations as well?\n\nIt shouldn't be needed as we already know that the allocation was\nsuccessful in the source record, and we're basically just copying things\nover. But better be safe than sorry, so I'll add it in.\n\nPatrick\n"},{"id":"511721","messageId":"Z6CBEsI1E_TNnMvq@pks.im","threadId":"62862","inReplyTo":"xmqqldutk8bb.fsf@gitster.g","subject":"Re: [PATCH v2 15/20] compat: consistently resolve headers via project root","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-03T08:40:50Z","receivedAt":"2025-02-03T08:40:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Jan 29, 2025 at 06:23:36AM -0800, Junio C Hamano wrote:\n> Johannes Sixt writes:\n> \n> > Depending on the desired goal, is it perhaps better to use <> brackets\n> > more liberally?\n> \n> I'd be in general in favor of that direction, as the semantics of <>\n> is more clear than \"\" from the point of view of the standard ;-)\n\nHm. Semantics are indeed clearer when using <>, but I think that in this\ncontext \"\" still conveys more about where a specific header comes from.\nI know it's not the whole truth, but \"\" is typically taken to mean \"the\nheader comes from our own sources\", whereas <> is typically taken to\nmean that it is provided by the system.\n\nSo if we now e.g. started to `#include ` I wonder whether\nthis would really result in code that is easier to understand, even if\nit is technically correct.\n\nI dunno -- I think I'll punt on the issue for now and drop the commit,\nbecause I'd rather want to not start discussing this in the context of\nthis series here. I'm happy to discuss this as a subsequent step though.\n\nPatrick\n"},{"id":"511922","messageId":"6kamtfoc5gsltrreu75n5we5qwylsnic2b5waxjcgq4hvtopl4@6j2dswje3i4k","threadId":"62862","inReplyTo":"20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im","subject":"Re: [PATCH v3 00/18] reftable: stop using \"git-compat-util.h\"","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-06T04:04:12Z","receivedAt":"2025-02-06T04:07:11Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/02/03 09:03AM, Patrick Steinhardt wrote:\n> Changes in v3:\n> - Fix type of `total_read` variable used to track how many bytes we\n> have read in `fd_read_lines()`.\n> - Drop the patch use root-relative includes again. Let's rather\n> discuss this outside of the scope of this series.\n> - Link to v2: https://lore.kernel.org/r/20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im\n\nThanks Patrick. This version looks good to me!\n\n-Justin\n"},{"id":"511946","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH v4 00/18] reftable: stop using \"git-compat-util.h\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:02Z","receivedAt":"2025-02-06T07:52:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series is the final step to fully decouple the reftable\nlibrary from the rest of the Git codebase. The goal of this is to make\nthe library reusable by other projects like libgit2 by simply copying\nover the source files, making Git the canonical upstream for reftable\nfunctionality.\n\nThis patch series stops using all kinds of helpers exposed by our\n\"git-compat-util.h\" header and open-codes them instead. In order to keep\nus from using these helpers by accident the final step is to pull out\nPOSIX-related bits and pieces into a new \"compat/posix.h\" header, which\nthe reftable library then uses instead of \"git-compat-util.h\".\n\nThe series is built on top of master at 5f8f7081f7 (The third batch,\n2025-01-23) with ps/reftable-sign-compare at 33319b0976 (reftable:\naddress trivial -Wsign-compare warnings, 2025-01-20) merged into it.\nThere is a trivial merge conflict with ps/zlib-ng that can be solved\nlike this:\n\n diff --cc reftable/system.h\n index e4a8944a70,d02eacea8f..0000000000\n --- a/reftable/system.h\n +++ b/reftable/system.h\n @@@ -11,15 -11,9 +11,15 @@@ https://developers.google.com/open-sour\n \n /* This header glues the reftable library to the rest of Git */\n \n -#include \"git-compat-util.h\"\n +#include \"compat/posix.h\"\n - #include \n + #include \"compat/zlib-compat.h\"\n \n +/*\n + * Return a random 32 bit integer. This function is expected to return\n + * pre-seeded data.\n + */\n +uint32_t reftable_rand(void);\n +\n /*\n * An implementation-specific temporary file. By making this specific to the\n * implementation it becomes possible to tie temporary files into any kind of\n\nChanges in v2:\n - The splitup of Windows headers has broken compilation because some\n of the headers couldn't be found anymore. I've fixed this more\n generally by converting includes in \"compat/\" to always be relative\n to the project source directory, dropping the platform-specific\n `-Icompat/` include.\n - Explain why we don't port over `EWOULDBLOCK` handling.\n - Fix commit message typos.\n - Link to v1: https://lore.kernel.org/r/20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im\n\nChanges in v3:\n - Fix type of `total_read` variable used to track how many bytes we\n have read in `fd_read_lines()`.\n - Drop the patch use root-relative includes again. Let's rather\n discuss this outside of the scope of this series.\n - Link to v2: https://lore.kernel.org/r/20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im\n\nChanges in v4:\n - Fix a couple of now-invalid relative includes that I missed. This\n fixes the build issue in \"seen\" with Meson.\n - Link to v3: https://lore.kernel.org/r/20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (18):\n reftable/stack: stop using `read_in_full()`\n reftable/stack: stop using `write_in_full()`\n reftable/blocksource: stop using `xmmap()`\n reftable/record: stop using `COPY_ARRAY()`\n reftable/record: stop using `BUG()` in `reftable_record_init()`\n reftable/record: don't `BUG()` in `reftable_record_cmp()`\n reftable: stop using `BUG()` in trivial cases\n reftable/basics: stop using `st_mult()` in array allocators\n reftable/basics: provide wrappers for big endian conversion\n reftable/reader: stop using `ARRAY_SIZE()` macro\n reftable/system: introduce `reftable_rand()`\n reftable/stack: stop using `sleep_millisec()`\n reftable/basics: stop using `SWAP()` macro\n reftable/basics: stop using `UNUSED` annotation\n compat/mingw: split out POSIX-related bits\n git-compat-util.h: split out POSIX-emulating bits\n reftable: decouple from Git codebase by pulling in \"compat/posix.h\"\n Makefile: skip reftable library for Coccinelle\n\n Makefile | 2 +-\n compat/{mingw.c => mingw/compat-util.c} | 28 +-\n compat/mingw/compat-util.h | 220 +++++++++++++\n compat/{mingw.h => mingw/posix.h} | 216 +------------\n compat/msvc.c | 6 -\n compat/msvc/compat-util.c | 6 +\n compat/msvc/compat-util.h | 7 +\n compat/{msvc.h => msvc/posix.h} | 8 +-\n compat/posix.h | 541 ++++++++++++++++++++++++++++++++\n config.mak.uname | 6 +-\n contrib/buildsystems/CMakeLists.txt | 2 +-\n git-compat-util.h | 535 +------------------------------\n meson.build | 8 +-\n reftable/basics.c | 19 --\n reftable/basics.h | 123 +++++++-\n reftable/block.c | 16 +-\n reftable/blocksource.c | 21 +-\n reftable/iter.c | 20 +-\n reftable/merged.c | 27 +-\n reftable/pq.c | 40 ++-\n reftable/pq.h | 2 +-\n reftable/reader.c | 33 +-\n reftable/record.c | 109 ++++---\n reftable/record.h | 6 +-\n reftable/stack.c | 52 ++-\n reftable/system.c | 7 +\n reftable/system.h | 9 +-\n reftable/writer.c | 29 +-\n t/unit-tests/t-reftable-basics.c | 28 +-\n t/unit-tests/t-reftable-pq.c | 22 +-\n t/unit-tests/t-reftable-record.c | 42 ++-\n 31 files changed, 1242 insertions(+), 948 deletions(-)\n\nRange-diff versus v3:\n\n 1: 801e7bb30c = 1: 07f44927f1 reftable/stack: stop using `read_in_full()`\n 2: 9b1b778a28 = 2: c62404d62e reftable/stack: stop using `write_in_full()`\n 3: 336b5ca3e1 = 3: f936b06d4b reftable/blocksource: stop using `xmmap()`\n 4: ec74201d8d = 4: ea203b9060 reftable/record: stop using `COPY_ARRAY()`\n 5: 853740bd10 = 5: c13aaeca16 reftable/record: stop using `BUG()` in `reftable_record_init()`\n 6: c6003aa731 = 6: 397b5321bf reftable/record: don't `BUG()` in `reftable_record_cmp()`\n 7: afbb02bdc7 = 7: 921d4ba9f0 reftable: stop using `BUG()` in trivial cases\n 8: fe34e5e425 = 8: 0c4396cc16 reftable/basics: stop using `st_mult()` in array allocators\n 9: 405b6bdac7 = 9: af50906b6d reftable/basics: provide wrappers for big endian conversion\n10: f0ca7a5f13 = 10: 1e931c747b reftable/reader: stop using `ARRAY_SIZE()` macro\n11: 2e8152c509 = 11: f66900df08 reftable/system: introduce `reftable_rand()`\n12: bdb708349d = 12: 99af9a50da reftable/stack: stop using `sleep_millisec()`\n13: 6e59462638 = 13: bdcd8744df reftable/basics: stop using `SWAP()` macro\n14: 7c4e2eeb9b = 14: 41f637bd3f reftable/basics: stop using `UNUSED` annotation\n15: 942eeca014 ! 15: 4dbbc71e11 compat/mingw: split out POSIX-related bits\n @@ Commit message\n Signed-off-by: Patrick Steinhardt \n \n ## compat/mingw.c => compat/mingw/compat-util.c ##\n +@@\n + #define USE_THE_REPOSITORY_VARIABLE\n + #define DISABLE_SIGN_COMPARE_WARNINGS\n + \n +-#include \"../git-compat-util.h\"\n +-#include \"win32.h\"\n ++#include \"../../git-compat-util.h\"\n ++#include \"../win32.h\"\n + #include \n + #include \n + #include \n + #include \n +-#include \"../strbuf.h\"\n +-#include \"../run-command.h\"\n +-#include \"../abspath.h\"\n +-#include \"../alloc.h\"\n +-#include \"win32/lazyload.h\"\n +-#include \"../config.h\"\n +-#include \"../environment.h\"\n +-#include \"../trace2.h\"\n +-#include \"../symlinks.h\"\n +-#include \"../wrapper.h\"\n +-#include \"dir.h\"\n +-#include \"gettext.h\"\n ++#include \"../../strbuf.h\"\n ++#include \"../../run-command.h\"\n ++#include \"../../abspath.h\"\n ++#include \"../../alloc.h\"\n ++#include \"../win32/lazyload.h\"\n ++#include \"../../config.h\"\n ++#include \"../../environment.h\"\n ++#include \"../../trace2.h\"\n ++#include \"../../symlinks.h\"\n ++#include \"../../wrapper.h\"\n ++#include \"../../dir.h\"\n ++#include \"../../gettext.h\"\n + #define SECURITY_WIN32\n + #include \n + \n \n ## compat/mingw/compat-util.h (new) ##\n @@\n @@ compat/mingw/posix.h: char *mingw_query_user_email(void);\n -#endif\n +#endif /* COMPAT_MINGW_POSIX_H */\n \n - ## compat/msvc.c => compat/msvc/compat-util.c ##\n + ## compat/msvc.c (deleted) ##\n @@\n - #include \n - #include \"../strbuf.h\"\n - \n +-#include \"../git-compat-util.h\"\n +-#include \"win32.h\"\n +-#include \n +-#include \"../strbuf.h\"\n +-\n -#include \"mingw.c\"\n -+#include \"mingw/compat-util.c\"\n +\n + ## compat/msvc/compat-util.c (new) ##\n +@@\n ++#include \"../../git-compat-util.h\"\n ++#include \"../win32.h\"\n ++#include \n ++#include \"../../strbuf.h\"\n ++\n ++#include \"../mingw/compat-util.c\"\n \n ## compat/msvc/compat-util.h (new) ##\n @@\n16: 53151b2649 ! 16: f0e9c6d3ee git-compat-util.h: split out POSIX-emulating bits\n @@ compat/posix.h (new)\n +#define HOST_NAME_MAX 256\n +#endif\n +\n -+#include \"sane-ctype.h\"\n ++#include \"../sane-ctype.h\"\n +\n +void git_stable_qsort(void *base, size_t nmemb, size_t size,\n +\t\t int(*compar)(const void *, const void *));\n17: b63a798513 = 17: 35a975dfe2 reftable: decouple from Git codebase by pulling in \"compat/posix.h\"\n18: f9edb54708 = 18: 1dde84b32c Makefile: skip reftable library for Coccinelle\n\n---\nbase-commit: 8047765d092881ec4aef7dfc57772161eee7f0f5\nchange-id: 20241119-pks-reftable-drop-git-compat-util-470f2bfde562\n\n"},{"id":"511947","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-2-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","subject":"[PATCH v4 02/18] reftable/stack: stop using `write_in_full()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:04Z","receivedAt":"2025-02-06T07:52:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Similar to the preceding commit, drop our use of `write_in_full()` and\nimplement a new wrapper `reftable_write_full()` that handles this logic\nfor us. This is done to reduce our dependency on the Git library.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 28 ++++++++++++++++++++++++----\n 1 file changed, 24 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex d57acd9540..46abf51ec5 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -48,6 +48,25 @@ static int stack_fsync(const struct reftable_write_options *opts, int fd)\n \treturn fsync(fd);\n }\n \n+static ssize_t reftable_write_data(int fd, const void *data, size_t size)\n+{\n+\tsize_t total_written = 0;\n+\tconst char *p = data;\n+\n+\twhile (total_written < size) {\n+\t\tssize_t bytes_written = write(fd, p, size - total_written);\n+\t\tif (bytes_written < 0 && (errno == EAGAIN || errno == EINTR))\n+\t\t\tcontinue;\n+\t\tif (bytes_written < 0)\n+\t\t\treturn REFTABLE_IO_ERROR;\n+\n+\t\ttotal_written += bytes_written;\n+\t\tp += bytes_written;\n+\t}\n+\n+\treturn total_written;\n+}\n+\n struct fd_writer {\n \tconst struct reftable_write_options *opts;\n \tint fd;\n@@ -56,7 +75,7 @@ struct fd_writer {\n static ssize_t fd_writer_write(void *arg, const void *data, size_t sz)\n {\n \tstruct fd_writer *writer = arg;\n-\treturn write_in_full(writer->fd, data, sz);\n+\treturn reftable_write_data(writer->fd, data, sz);\n }\n \n static int fd_writer_flush(void *arg)\n@@ -784,7 +803,8 @@ int reftable_addition_commit(struct reftable_addition *add)\n \t\t\tgoto done;\n \t}\n \n-\terr = write_in_full(add->tables_list_lock.fd, table_list.buf, table_list.len);\n+\terr = reftable_write_data(add->tables_list_lock.fd,\n+\t\t\t\t table_list.buf, table_list.len);\n \treftable_buf_release(&table_list);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n@@ -1468,8 +1488,8 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\t\tgoto done;\n \t}\n \n-\terr = write_in_full(tables_list_lock.fd,\n-\t\t\t tables_list_buf.buf, tables_list_buf.len);\n+\terr = reftable_write_data(tables_list_lock.fd,\n+\t\t\t\t tables_list_buf.buf, tables_list_buf.len);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tunlink(new_table_path.buf);\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"511948","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-1-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","subject":"[PATCH v4 01/18] reftable/stack: stop using `read_in_full()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:03Z","receivedAt":"2025-02-06T07:52:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"There is a single callsite of `read_in_full()` in the reftable library.\nOpen-code the function to reduce our dependency on the Git library.\n\nNote that we only partially port over the logic from `read_in_full()`\nand its underlying `xread()` helper. Most importantly, the latter also\nknows to handle `EWOULDBLOCK` via `handle_nonblock()`. This logic is\nirrelevant for us though because the reftable library never sets the\n`O_NONBLOCK` option in the first place.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 18 ++++++++++++++----\n 1 file changed, 14 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex f7c1845e15..d57acd9540 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -115,13 +115,16 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \n static int fd_read_lines(int fd, char ***namesp)\n {\n-\toff_t size = lseek(fd, 0, SEEK_END);\n \tchar *buf = NULL;\n \tint err = 0;\n+\toff_t size;\n+\n+\tsize = lseek(fd, 0, SEEK_END);\n \tif (size < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n+\n \terr = lseek(fd, 0, SEEK_SET);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n@@ -134,9 +137,16 @@ static int fd_read_lines(int fd, char ***namesp)\n \t\tgoto done;\n \t}\n \n-\tif (read_in_full(fd, buf, size) != size) {\n-\t\terr = REFTABLE_IO_ERROR;\n-\t\tgoto done;\n+\tfor (off_t total_read = 0; total_read < size; ) {\n+\t\tssize_t bytes_read = read(fd, buf + total_read, size - total_read);\n+\t\tif (bytes_read < 0 && (errno == EAGAIN || errno == EINTR))\n+\t\t\tcontinue;\n+\t\tif (bytes_read < 0 || !bytes_read) {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\ttotal_read += bytes_read;\n \t}\n \tbuf[size] = 0;\n \n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"511949","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-3-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","subject":"[PATCH v4 03/18] reftable/blocksource: stop using `xmmap()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:05Z","receivedAt":"2025-02-06T07:52:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We use `xmmap()` to map reftables into memory. This function has two\nproblems:\n\n - It causes us to die in case the mmap fails.\n\n - It ties us to the Git codebase.\n\nRefactor the code to use mmap(3p) instead with manual error checking.\nNote that this function may not be the system-provided mmap(3p), but may\npoint to our `git_mmap()` wrapper that emulates the syscall on systems\nthat do not have mmap(3p) available.\n\nFix `reftable_block_source_from_file()` to properly bubble up the error\ncode in case the map(3p) call fails.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/blocksource.c | 11 ++++++++---\n 1 file changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex bba4a45b98..02972c46f4 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -98,7 +98,7 @@ static struct reftable_block_source_vtable file_vtable = {\n int reftable_block_source_from_file(struct reftable_block_source *bs,\n \t\t\t\t const char *name)\n {\n-\tstruct file_block_source *p;\n+\tstruct file_block_source *p = NULL;\n \tstruct stat st;\n \tint fd, err;\n \n@@ -122,7 +122,12 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,\n \t}\n \n \tp->size = st.st_size;\n-\tp->data = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tp->data = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tif (p->data == MAP_FAILED) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tp->data = NULL;\n+\t\tgoto out;\n+\t}\n \n \tassert(!bs->ops);\n \tbs->ops = &file_vtable;\n@@ -135,5 +140,5 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,\n \t\tclose(fd);\n \tif (err < 0)\n \t\treftable_free(p);\n-\treturn 0;\n+\treturn err;\n }\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"511950","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-4-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","subject":"[PATCH v4 04/18] reftable/record: stop using `COPY_ARRAY()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:06Z","receivedAt":"2025-02-06T07:52:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Drop our use of `COPY_ARRAY()`, replacing it with an open-coded variant\nthereof. This is done to reduce our dependency on the Git library.\n\nWhile at it, guard the whole array copy logic so that we only copy it in\ncase there actually is anything to be copied. Otherwise, we may end up\ntrying to allocate a zero-sized array, which will return a NULL pointer\nand thus cause us to return an `REFTABLE_OUT_OF_MEMORY_ERROR`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/record.c | 16 +++++++++++-----\n 1 file changed, 11 insertions(+), 5 deletions(-)\n\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 8919df8a4d..2c0cc32cbd 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -504,11 +504,17 @@ static int reftable_obj_record_copy_from(void *rec, const void *src_rec,\n \tif (src->hash_prefix_len)\n \t\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n \n-\tREFTABLE_ALLOC_ARRAY(obj->offsets, src->offset_len);\n-\tif (!obj->offsets)\n-\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n-\tobj->offset_len = src->offset_len;\n-\tCOPY_ARRAY(obj->offsets, src->offsets, src->offset_len);\n+\tif (src->offset_len) {\n+\t\tif (sizeof(*src->offsets) > SIZE_MAX / src->offset_len)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n+\t\tREFTABLE_ALLOC_ARRAY(obj->offsets, src->offset_len);\n+\t\tif (!obj->offsets)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n+\t\tmemcpy(obj->offsets, src->offsets, sizeof(*src->offsets) * src->offset_len);\n+\t\tobj->offset_len = src->offset_len;\n+\t}\n \n \treturn 0;\n }\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"511951","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-5-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","subject":"[PATCH v4 05/18] reftable/record: stop using `BUG()` in `reftable_record_init()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:07Z","receivedAt":"2025-02-06T07:52:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're aborting the program via `BUG()` in case `reftable_record_init()`\nwas invoked with an unknown record type. This is bad because we may now\ndie in library code, and because it makes us depend on the Git codebase.\n\nRefactor the code such that `reftable_record_init()` can return an error\ncode to the caller. Adapt any callers accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/block.c | 4 +++-\n reftable/merged.c | 5 ++++-\n reftable/reader.c | 5 ++++-\n reftable/record.c | 8 ++++----\n reftable/record.h | 4 ++--\n t/unit-tests/t-reftable-pq.c | 6 +++---\n t/unit-tests/t-reftable-record.c | 2 +-\n 7 files changed, 21 insertions(+), 13 deletions(-)\n\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 8ac865ce78..255d566854 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -509,7 +509,9 @@ int block_iter_seek_key(struct block_iter *it, const struct block_reader *br,\n \tit->block_len = br->block_len;\n \tit->hash_size = br->hash_size;\n \n-\treftable_record_init(&rec, block_reader_type(br));\n+\terr = reftable_record_init(&rec, block_reader_type(br));\n+\tif (err < 0)\n+\t\tgoto done;\n \n \t/*\n \t * We're looking for the last entry less than the wanted key so that\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex e72b39e178..4156eec07f 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -253,7 +253,10 @@ int merged_table_init_iter(struct reftable_merged_table *mt,\n \t}\n \n \tfor (size_t i = 0; i < mt->readers_len; i++) {\n-\t\treftable_record_init(&subiters[i].rec, typ);\n+\t\tret = reftable_record_init(&subiters[i].rec, typ);\n+\t\tif (ret < 0)\n+\t\t\tgoto out;\n+\n \t\tret = reader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n \t\tif (ret < 0)\n \t\t\tgoto out;\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 3f2e4b2800..de6e6dd932 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -360,7 +360,10 @@ static int table_iter_seek_linear(struct table_iter *ti,\n \tstruct reftable_record rec;\n \tint err;\n \n-\treftable_record_init(&rec, reftable_record_type(want));\n+\terr = reftable_record_init(&rec, reftable_record_type(want));\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = reftable_record_key(want, &want_key);\n \tif (err < 0)\n \t\tgoto done;\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 2c0cc32cbd..1e18f8dffb 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -1306,7 +1306,7 @@ reftable_record_vtable(struct reftable_record *rec)\n \tabort();\n }\n \n-void reftable_record_init(struct reftable_record *rec, uint8_t typ)\n+int reftable_record_init(struct reftable_record *rec, uint8_t typ)\n {\n \tmemset(rec, 0, sizeof(*rec));\n \trec->type = typ;\n@@ -1315,11 +1315,11 @@ void reftable_record_init(struct reftable_record *rec, uint8_t typ)\n \tcase BLOCK_TYPE_REF:\n \tcase BLOCK_TYPE_LOG:\n \tcase BLOCK_TYPE_OBJ:\n-\t\treturn;\n+\t\treturn 0;\n \tcase BLOCK_TYPE_INDEX:\n \t\treftable_buf_init(&rec->u.idx.last_key);\n-\t\treturn;\n+\t\treturn 0;\n \tdefault:\n-\t\tBUG(\"unhandled record type\");\n+\t\treturn REFTABLE_API_ERROR;\n \t}\n }\ndiff --git a/reftable/record.h b/reftable/record.h\nindex c7755a4d75..e1846c294b 100644\n--- a/reftable/record.h\n+++ b/reftable/record.h\n@@ -130,8 +130,8 @@ struct reftable_record {\n \t} u;\n };\n \n-/* Initialize the reftable record for the given type */\n-void reftable_record_init(struct reftable_record *rec, uint8_t typ);\n+/* Initialize the reftable record for the given type. */\n+int reftable_record_init(struct reftable_record *rec, uint8_t typ);\n \n /* see struct record_vtable */\n int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b);\ndiff --git a/t/unit-tests/t-reftable-pq.c b/t/unit-tests/t-reftable-pq.c\nindex f3f8a0cdf3..d8a4c283a1 100644\n--- a/t/unit-tests/t-reftable-pq.c\n+++ b/t/unit-tests/t-reftable-pq.c\n@@ -32,7 +32,7 @@ static void t_pq_record(void)\n \tchar *last = NULL;\n \n \tfor (i = 0; i < N; i++) {\n-\t\treftable_record_init(&recs[i], BLOCK_TYPE_REF);\n+\t\tcheck(!reftable_record_init(&recs[i], BLOCK_TYPE_REF));\n \t\trecs[i].u.ref.refname = xstrfmt(\"%02\"PRIuMAX, (uintmax_t)i);\n \t}\n \n@@ -72,7 +72,7 @@ static void t_pq_index(void)\n \tsize_t N = ARRAY_SIZE(recs), i;\n \n \tfor (i = 0; i < N; i++) {\n-\t\treftable_record_init(&recs[i], BLOCK_TYPE_REF);\n+\t\tcheck(!reftable_record_init(&recs[i], BLOCK_TYPE_REF));\n \t\trecs[i].u.ref.refname = (char *) \"refs/heads/master\";\n \t}\n \n@@ -111,7 +111,7 @@ static void t_merged_iter_pqueue_top(void)\n \tsize_t N = ARRAY_SIZE(recs), i;\n \n \tfor (i = 0; i < N; i++) {\n-\t\treftable_record_init(&recs[i], BLOCK_TYPE_REF);\n+\t\tcheck(!reftable_record_init(&recs[i], BLOCK_TYPE_REF));\n \t\trecs[i].u.ref.refname = (char *) \"refs/heads/master\";\n \t}\n \ndiff --git a/t/unit-tests/t-reftable-record.c b/t/unit-tests/t-reftable-record.c\nindex d49d2a2729..6540bd20e3 100644\n--- a/t/unit-tests/t-reftable-record.c\n+++ b/t/unit-tests/t-reftable-record.c\n@@ -17,7 +17,7 @@ static void t_copy(struct reftable_record *rec)\n \tuint8_t typ;\n \n \ttyp = reftable_record_type(rec);\n-\treftable_record_init(©, typ);\n+\tcheck(!reftable_record_init(©, typ));\n \treftable_record_copy_from(©, rec, REFTABLE_HASH_SIZE_SHA1);\n \t/* do it twice to catch memory leaks */\n \treftable_record_copy_from(©, rec, REFTABLE_HASH_SIZE_SHA1);\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"511952","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-6-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","subject":"[PATCH v4 06/18] reftable/record: don't `BUG()` in `reftable_record_cmp()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:08Z","receivedAt":"2025-02-06T07:52:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library aborts with a bug in case `reftable_record_cmp()`\nis invoked with two records of differing types. This would cause the\nprogram to die without the caller being able to handle the error, which\nis not something we want in the context of library code. And it ties us\nto the Git codebase.\n\nRefactor the code such that `reftable_record_cmp()` returns an error\ncode separate from the actual comparison result. This requires us to\nalso adapt some callers up the callchain in a similar fashion.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 20 +++++++++++++++-----\n reftable/pq.c | 36 +++++++++++++++++++++++++++++-------\n reftable/pq.h | 2 +-\n reftable/record.c | 10 ++++++----\n reftable/record.h | 2 +-\n t/unit-tests/t-reftable-pq.c | 16 ++++++++++++----\n t/unit-tests/t-reftable-record.c | 40 ++++++++++++++++++++++++++++------------\n 7 files changed, 92 insertions(+), 34 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 4156eec07f..563864068c 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -66,8 +66,11 @@ static int merged_iter_seek(struct merged_iter *mi, struct reftable_record *want\n \tint err;\n \n \tmi->advance_index = -1;\n-\twhile (!merged_iter_pqueue_is_empty(mi->pq))\n-\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq)) {\n+\t\terr = merged_iter_pqueue_remove(&mi->pq, NULL);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n \n \tfor (size_t i = 0; i < mi->subiters_len; i++) {\n \t\terr = iterator_seek(&mi->subiters[i].iter, want);\n@@ -120,7 +123,9 @@ static int merged_iter_next_entry(struct merged_iter *mi,\n \tif (empty)\n \t\treturn 1;\n \n-\tentry = merged_iter_pqueue_remove(&mi->pq);\n+\terr = merged_iter_pqueue_remove(&mi->pq, &entry);\n+\tif (err < 0)\n+\t\treturn err;\n \n \t/*\n \t One can also use reftable as datacenter-local storage, where the ref\n@@ -134,11 +139,16 @@ static int merged_iter_next_entry(struct merged_iter *mi,\n \t\tstruct pq_entry top = merged_iter_pqueue_top(mi->pq);\n \t\tint cmp;\n \n-\t\tcmp = reftable_record_cmp(top.rec, entry.rec);\n+\t\terr = reftable_record_cmp(top.rec, entry.rec, &cmp);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n \t\tif (cmp > 0)\n \t\t\tbreak;\n \n-\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\t\terr = merged_iter_pqueue_remove(&mi->pq, NULL);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\n \t\terr = merged_iter_advance_subiter(mi, top.index);\n \t\tif (err < 0)\n \t\t\treturn err;\ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex 5591e875e1..ef8035cfd9 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -15,13 +15,18 @@ license that can be found in the LICENSE file or at\n \n int pq_less(struct pq_entry *a, struct pq_entry *b)\n {\n-\tint cmp = reftable_record_cmp(a->rec, b->rec);\n+\tint cmp, err;\n+\n+\terr = reftable_record_cmp(a->rec, b->rec, &cmp);\n+\tif (err < 0)\n+\t\treturn err;\n+\n \tif (cmp == 0)\n \t\treturn a->index > b->index;\n \treturn cmp < 0;\n }\n \n-struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n+int merged_iter_pqueue_remove(struct merged_iter_pqueue *pq, struct pq_entry *out)\n {\n \tsize_t i = 0;\n \tstruct pq_entry e = pq->heap[0];\n@@ -32,17 +37,34 @@ struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n \t\tsize_t min = i;\n \t\tsize_t j = 2 * i + 1;\n \t\tsize_t k = 2 * i + 2;\n-\t\tif (j < pq->len && pq_less(&pq->heap[j], &pq->heap[i]))\n-\t\t\tmin = j;\n-\t\tif (k < pq->len && pq_less(&pq->heap[k], &pq->heap[min]))\n-\t\t\tmin = k;\n+\t\tint cmp;\n+\n+\t\tif (j < pq->len) {\n+\t\t\tcmp = pq_less(&pq->heap[j], &pq->heap[i]);\n+\t\t\tif (cmp < 0)\n+\t\t\t\treturn -1;\n+\t\t\telse if (cmp)\n+\t\t\t\tmin = j;\n+\t\t}\n+\n+\t\tif (k < pq->len) {\n+\t\t\tcmp = pq_less(&pq->heap[k], &pq->heap[min]);\n+\t\t\tif (cmp < 0)\n+\t\t\t\treturn -1;\n+\t\t\telse if (cmp)\n+\t\t\t\tmin = k;\n+\t\t}\n+\n \t\tif (min == i)\n \t\t\tbreak;\n \t\tSWAP(pq->heap[i], pq->heap[min]);\n \t\ti = min;\n \t}\n \n-\treturn e;\n+\tif (out)\n+\t\t*out = e;\n+\n+\treturn 0;\n }\n \n int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e)\ndiff --git a/reftable/pq.h b/reftable/pq.h\nindex 83c062eeca..ff39016445 100644\n--- a/reftable/pq.h\n+++ b/reftable/pq.h\n@@ -22,7 +22,7 @@ struct merged_iter_pqueue {\n \tsize_t cap;\n };\n \n-struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n+int merged_iter_pqueue_remove(struct merged_iter_pqueue *pq, struct pq_entry *out);\n int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e);\n void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n int pq_less(struct pq_entry *a, struct pq_entry *b);\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 1e18f8dffb..b39d99fcc7 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -1195,12 +1195,14 @@ int reftable_record_is_deletion(struct reftable_record *rec)\n \t\treftable_record_data(rec));\n }\n \n-int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b)\n+int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b,\n+\t\t\tint *cmp)\n {\n \tif (a->type != b->type)\n-\t\tBUG(\"cannot compare reftable records of different type\");\n-\treturn reftable_record_vtable(a)->cmp(\n-\t\treftable_record_data(a), reftable_record_data(b));\n+\t\treturn -1;\n+\t*cmp = reftable_record_vtable(a)->cmp(reftable_record_data(a),\n+\t\t\t\t\t reftable_record_data(b));\n+\treturn 0;\n }\n \n int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, uint32_t hash_size)\ndiff --git a/reftable/record.h b/reftable/record.h\nindex e1846c294b..867810a932 100644\n--- a/reftable/record.h\n+++ b/reftable/record.h\n@@ -134,7 +134,7 @@ struct reftable_record {\n int reftable_record_init(struct reftable_record *rec, uint8_t typ);\n \n /* see struct record_vtable */\n-int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b);\n+int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b, int *cmp);\n int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, uint32_t hash_size);\n int reftable_record_key(struct reftable_record *rec, struct reftable_buf *dest);\n int reftable_record_copy_from(struct reftable_record *rec,\ndiff --git a/t/unit-tests/t-reftable-pq.c b/t/unit-tests/t-reftable-pq.c\nindex d8a4c283a1..c128fe8616 100644\n--- a/t/unit-tests/t-reftable-pq.c\n+++ b/t/unit-tests/t-reftable-pq.c\n@@ -21,7 +21,9 @@ static void merged_iter_pqueue_check(const struct merged_iter_pqueue *pq)\n \n static int pq_entry_equal(struct pq_entry *a, struct pq_entry *b)\n {\n-\treturn !reftable_record_cmp(a->rec, b->rec) && (a->index == b->index);\n+\tint cmp;\n+\tcheck(!reftable_record_cmp(a->rec, b->rec, &cmp));\n+\treturn !cmp && (a->index == b->index);\n }\n \n static void t_pq_record(void)\n@@ -49,7 +51,9 @@ static void t_pq_record(void)\n \n \twhile (!merged_iter_pqueue_is_empty(pq)) {\n \t\tstruct pq_entry top = merged_iter_pqueue_top(pq);\n-\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct pq_entry e;\n+\n+\t\tcheck(!merged_iter_pqueue_remove(&pq, &e));\n \t\tmerged_iter_pqueue_check(&pq);\n \n \t\tcheck(pq_entry_equal(&top, &e));\n@@ -90,7 +94,9 @@ static void t_pq_index(void)\n \n \tfor (i = N - 1; i > 0; i--) {\n \t\tstruct pq_entry top = merged_iter_pqueue_top(pq);\n-\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct pq_entry e;\n+\n+\t\tcheck(!merged_iter_pqueue_remove(&pq, &e));\n \t\tmerged_iter_pqueue_check(&pq);\n \n \t\tcheck(pq_entry_equal(&top, &e));\n@@ -129,7 +135,9 @@ static void t_merged_iter_pqueue_top(void)\n \n \tfor (i = N - 1; i > 0; i--) {\n \t\tstruct pq_entry top = merged_iter_pqueue_top(pq);\n-\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct pq_entry e;\n+\n+\t\tcheck(!merged_iter_pqueue_remove(&pq, &e));\n \n \t\tmerged_iter_pqueue_check(&pq);\n \t\tcheck(pq_entry_equal(&top, &e));\ndiff --git a/t/unit-tests/t-reftable-record.c b/t/unit-tests/t-reftable-record.c\nindex 6540bd20e3..5954966373 100644\n--- a/t/unit-tests/t-reftable-record.c\n+++ b/t/unit-tests/t-reftable-record.c\n@@ -100,16 +100,20 @@ static void t_reftable_ref_record_comparison(void)\n \t\t\t.u.ref.value.symref = (char *) \"refs/heads/master\",\n \t\t},\n \t};\n+\tint cmp;\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \n \tin[1].u.ref.value_type = in[0].u.ref.value_type;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n }\n \n static void t_reftable_ref_record_compare_name(void)\n@@ -209,17 +213,20 @@ static void t_reftable_log_record_comparison(void)\n \t\t\t.u.log.update_index = 22,\n \t\t},\n \t};\n+\tint cmp;\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \t/* comparison should be reversed for equal keys, because\n \t * comparison is now performed on the basis of update indices */\n-\tcheck_int(reftable_record_cmp(&in[0], &in[1]), <, 0);\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck_int(cmp, <, 0);\n \n \tin[1].u.log.update_index = in[0].u.log.update_index;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n }\n \n static void t_reftable_log_record_compare_key(void)\n@@ -396,16 +403,20 @@ static void t_reftable_obj_record_comparison(void)\n \t\t\t.u.obj.hash_prefix_len = 5,\n \t\t},\n \t};\n+\tint cmp;\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \n \tin[1].u.obj.offset_len = in[0].u.obj.offset_len;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n }\n \n static void t_reftable_obj_record_roundtrip(void)\n@@ -486,19 +497,24 @@ static void t_reftable_index_record_comparison(void)\n \t\t\t.u.idx.last_key = REFTABLE_BUF_INIT,\n \t\t},\n \t};\n+\tint cmp;\n+\n \tcheck(!reftable_buf_addstr(&in[0].u.idx.last_key, \"refs/heads/master\"));\n \tcheck(!reftable_buf_addstr(&in[1].u.idx.last_key, \"refs/heads/master\"));\n \tcheck(!reftable_buf_addstr(&in[2].u.idx.last_key, \"refs/heads/branch\"));\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \n \tin[1].u.idx.offset = in[0].u.idx.offset;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tfor (size_t i = 0; i < ARRAY_SIZE(in); i++)\n \t\treftable_record_release(&in[i]);\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"511953","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-7-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","subject":"[PATCH v4 07/18] reftable: stop using `BUG()` in trivial cases","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:09Z","receivedAt":"2025-02-06T07:52:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Stop using `BUG()` in the remaining trivial cases that we still have in\nthe reftable library. Instead of aborting the program, we'll now bubble\nup a `REFTABLE_API_ERROR` to indicate misuse of the calling conventions.\n\nNote that in both `reftable_reader_{inc,dec}ref()` we simply stop\ncalling `BUG()` altogether. The only situation where the counter should\nbe zero is when the structure has already been free'd anyway, so we\nwould run into undefined behaviour regardless of whether we try to abort\nthe program or not.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/iter.c | 3 +--\n reftable/reader.c | 4 ----\n reftable/writer.c | 5 ++---\n 3 files changed, 3 insertions(+), 9 deletions(-)\n\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex 86e801ca9f..b2ffb09c16 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -146,8 +146,7 @@ static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n static int indexed_table_ref_iter_seek(void *p UNUSED,\n \t\t\t\t struct reftable_record *want UNUSED)\n {\n-\tBUG(\"seeking indexed table is not supported\");\n-\treturn -1;\n+\treturn REFTABLE_API_ERROR;\n }\n \n static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex de6e6dd932..36a5633ede 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -677,8 +677,6 @@ int reftable_reader_new(struct reftable_reader **out,\n \n void reftable_reader_incref(struct reftable_reader *r)\n {\n-\tif (!r->refcount)\n-\t\tBUG(\"cannot increment ref counter of dead reader\");\n \tr->refcount++;\n }\n \n@@ -686,8 +684,6 @@ void reftable_reader_decref(struct reftable_reader *r)\n {\n \tif (!r)\n \t\treturn;\n-\tif (!r->refcount)\n-\t\tBUG(\"cannot decrement ref counter of dead reader\");\n \tif (--r->refcount)\n \t\treturn;\n \tblock_source_close(&r->source);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 91d6629486..155863ee5f 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -158,7 +158,7 @@ int reftable_writer_new(struct reftable_writer **out,\n \t\topts = *_opts;\n \toptions_set_defaults(&opts);\n \tif (opts.block_size >= (1 << 24))\n-\t\tBUG(\"configured block size exceeds 16MB\");\n+\t\treturn REFTABLE_API_ERROR;\n \n \treftable_buf_init(&wp->block_writer_data.last_key);\n \treftable_buf_init(&wp->last_key);\n@@ -289,8 +289,7 @@ static int writer_add_record(struct reftable_writer *w,\n \t}\n \n \tif (block_writer_type(w->block_writer) != reftable_record_type(rec))\n-\t\tBUG(\"record of type %d added to writer of type %d\",\n-\t\t reftable_record_type(rec), block_writer_type(w->block_writer));\n+\t\treturn REFTABLE_API_ERROR;\n \n \t/*\n \t * Try to add the record to the writer. If this succeeds then we're\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"511954","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-8-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","subject":"[PATCH v4 08/18] reftable/basics: stop using `st_mult()` in array allocators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:10Z","receivedAt":"2025-02-06T07:52:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're using `st_mult()` as part of our macro helpers that allocate\narrays. This is bad due two two reasons:\n\n - `st_mult()` causes us to die in case the multiplication overflows.\n\n - `st_mult()` ties us to the Git codebase.\n\nRefactor the code to instead detect overflows manually and return an\nerror in such cases.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 36 ++++++++++++++++++++++++++++++++----\n 1 file changed, 32 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex a2a010a0e1..646f8d67f2 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -117,18 +117,46 @@ void reftable_free(void *p);\n void *reftable_calloc(size_t nelem, size_t elsize);\n char *reftable_strdup(const char *str);\n \n-#define REFTABLE_ALLOC_ARRAY(x, alloc) (x) = reftable_malloc(st_mult(sizeof(*(x)), (alloc)))\n+static inline int reftable_alloc_size(size_t nelem, size_t elsize, size_t *out)\n+{\n+\tif (nelem && elsize > SIZE_MAX / nelem)\n+\t\treturn -1;\n+\t*out = nelem * elsize;\n+\treturn 0;\n+}\n+\n+#define REFTABLE_ALLOC_ARRAY(x, alloc) do { \\\n+\t\tsize_t alloc_size; \\\n+\t\tif (reftable_alloc_size(sizeof(*(x)), (alloc), &alloc_size) < 0) { \\\n+\t\t\terrno = ENOMEM; \\\n+\t\t\t(x) = NULL; \\\n+\t\t} else { \\\n+\t\t\t(x) = reftable_malloc(alloc_size); \\\n+\t\t} \\\n+\t} while (0)\n #define REFTABLE_CALLOC_ARRAY(x, alloc) (x) = reftable_calloc((alloc), sizeof(*(x)))\n-#define REFTABLE_REALLOC_ARRAY(x, alloc) (x) = reftable_realloc((x), st_mult(sizeof(*(x)), (alloc)))\n+#define REFTABLE_REALLOC_ARRAY(x, alloc) do { \\\n+\t\tsize_t alloc_size; \\\n+\t\tif (reftable_alloc_size(sizeof(*(x)), (alloc), &alloc_size) < 0) { \\\n+\t\t\terrno = ENOMEM; \\\n+\t\t\t(x) = NULL; \\\n+\t\t} else { \\\n+\t\t\t(x) = reftable_realloc((x), alloc_size); \\\n+\t\t} \\\n+\t} while (0)\n \n static inline void *reftable_alloc_grow(void *p, size_t nelem, size_t elsize,\n \t\t\t\t\tsize_t *allocp)\n {\n \tvoid *new_p;\n-\tsize_t alloc = *allocp * 2 + 1;\n+\tsize_t alloc = *allocp * 2 + 1, alloc_bytes;\n \tif (alloc < nelem)\n \t\talloc = nelem;\n-\tnew_p = reftable_realloc(p, st_mult(elsize, alloc));\n+\tif (reftable_alloc_size(elsize, alloc, &alloc_bytes) < 0) {\n+\t\terrno = ENOMEM;\n+\t\treturn p;\n+\t}\n+\tnew_p = reftable_realloc(p, alloc_bytes);\n \tif (!new_p)\n \t\treturn p;\n \t*allocp = alloc;\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"511955","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-10-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","subject":"[PATCH v4 10/18] reftable/reader: stop using `ARRAY_SIZE()` macro","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:12Z","receivedAt":"2025-02-06T07:52:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have a single user of the `ARRAY_SIZE()` macro in the reftable\nreader. Drop its use to reduce our dependence on the Git codebase.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/reader.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex bf07a0a586..c3a3674665 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -849,7 +849,7 @@ int reftable_reader_print_blocks(const char *tablename)\n \tprintf(\"header:\\n\");\n \tprintf(\" block_size: %d\\n\", r->block_size);\n \n-\tfor (i = 0; i < ARRAY_SIZE(sections); i++) {\n+\tfor (i = 0; i < sizeof(sections) / sizeof(*sections); i++) {\n \t\terr = table_iter_seek_start(&ti, sections[i].type, 0);\n \t\tif (err < 0)\n \t\t\tgoto done;\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"511956","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-9-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","subject":"[PATCH v4 09/18] reftable/basics: provide wrappers for big endian conversion","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:11Z","receivedAt":"2025-02-06T07:52:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're using a mixture of big endian conversion functions provided by\nboth the reftable library, but also by the Git codebase. Refactor the\ncode so that we exclusively use reftable-provided wrappers in order to\nuntangle us from the Git codebase.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 19 ----------\n reftable/basics.h | 76 ++++++++++++++++++++++++++++++++++++++--\n reftable/block.c | 12 +++----\n reftable/reader.c | 22 ++++++------\n reftable/record.c | 8 ++---\n reftable/writer.c | 20 +++++------\n t/unit-tests/t-reftable-basics.c | 28 ++++++++++++---\n 7 files changed, 127 insertions(+), 58 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 3b5ea27bbd..8c4a4433e4 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -147,25 +147,6 @@ char *reftable_buf_detach(struct reftable_buf *buf)\n \treturn result;\n }\n \n-void put_be24(uint8_t *out, uint32_t i)\n-{\n-\tout[0] = (uint8_t)((i >> 16) & 0xff);\n-\tout[1] = (uint8_t)((i >> 8) & 0xff);\n-\tout[2] = (uint8_t)(i & 0xff);\n-}\n-\n-uint32_t get_be24(uint8_t *in)\n-{\n-\treturn (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 |\n-\t (uint32_t)(in[2]);\n-}\n-\n-void put_be16(uint8_t *out, uint16_t i)\n-{\n-\tout[0] = (uint8_t)((i >> 8) & 0xff);\n-\tout[1] = (uint8_t)(i & 0xff);\n-}\n-\n size_t binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n {\n \tsize_t lo = 0;\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 646f8d67f2..c1ddbaec3f 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -76,9 +76,79 @@ char *reftable_buf_detach(struct reftable_buf *buf);\n \n /* Bigendian en/decoding of integers */\n \n-void put_be24(uint8_t *out, uint32_t i);\n-uint32_t get_be24(uint8_t *in);\n-void put_be16(uint8_t *out, uint16_t i);\n+static inline void reftable_put_be16(void *out, uint16_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 8) & 0xff);\n+\tp[1] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline void reftable_put_be24(void *out, uint32_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 16) & 0xff);\n+\tp[1] = (uint8_t)((i >> 8) & 0xff);\n+\tp[2] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline void reftable_put_be32(void *out, uint32_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 24) & 0xff);\n+\tp[1] = (uint8_t)((i >> 16) & 0xff);\n+\tp[2] = (uint8_t)((i >> 8) & 0xff);\n+\tp[3] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline void reftable_put_be64(void *out, uint64_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 56) & 0xff);\n+\tp[1] = (uint8_t)((i >> 48) & 0xff);\n+\tp[2] = (uint8_t)((i >> 40) & 0xff);\n+\tp[3] = (uint8_t)((i >> 32) & 0xff);\n+\tp[4] = (uint8_t)((i >> 24) & 0xff);\n+\tp[5] = (uint8_t)((i >> 16) & 0xff);\n+\tp[6] = (uint8_t)((i >> 8) & 0xff);\n+\tp[7] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline uint16_t reftable_get_be16(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint16_t)(p[0]) << 8 |\n+\t (uint16_t)(p[1]) << 0;\n+}\n+\n+static inline uint32_t reftable_get_be24(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint32_t)(p[0]) << 16 |\n+\t (uint32_t)(p[1]) << 8 |\n+\t (uint32_t)(p[2]) << 0;\n+}\n+\n+static inline uint32_t reftable_get_be32(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint32_t)(p[0]) << 24 |\n+\t (uint32_t)(p[1]) << 16 |\n+\t (uint32_t)(p[2]) << 8|\n+\t (uint32_t)(p[3]) << 0;\n+}\n+\n+static inline uint64_t reftable_get_be64(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint64_t)(p[0]) << 56 |\n+\t (uint64_t)(p[1]) << 48 |\n+\t (uint64_t)(p[2]) << 40 |\n+\t (uint64_t)(p[3]) << 32 |\n+\t (uint64_t)(p[4]) << 24 |\n+\t (uint64_t)(p[5]) << 16 |\n+\t (uint64_t)(p[6]) << 8 |\n+\t (uint64_t)(p[7]) << 0;\n+}\n \n /*\n * find smallest index i in [0, sz) at which `f(i) > 0`, assuming that f is\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 255d566854..373908807e 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -148,13 +148,13 @@ int block_writer_add(struct block_writer *w, struct reftable_record *rec)\n int block_writer_finish(struct block_writer *w)\n {\n \tfor (uint32_t i = 0; i < w->restart_len; i++) {\n-\t\tput_be24(w->block + w->next, w->restarts[i]);\n+\t\treftable_put_be24(w->block + w->next, w->restarts[i]);\n \t\tw->next += 3;\n \t}\n \n-\tput_be16(w->block + w->next, w->restart_len);\n+\treftable_put_be16(w->block + w->next, w->restart_len);\n \tw->next += 2;\n-\tput_be24(w->block + 1 + w->header_off, w->next);\n+\treftable_put_be24(w->block + 1 + w->header_off, w->next);\n \n \t/*\n \t * Log records are stored zlib-compressed. Note that the compression\n@@ -216,7 +216,7 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,\n {\n \tuint32_t full_block_size = table_block_size;\n \tuint8_t typ = block->data[header_off];\n-\tuint32_t sz = get_be24(block->data + header_off + 1);\n+\tuint32_t sz = reftable_get_be24(block->data + header_off + 1);\n \tint err = 0;\n \tuint16_t restart_count = 0;\n \tuint32_t restart_start = 0;\n@@ -300,7 +300,7 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,\n \t\tfull_block_size = sz;\n \t}\n \n-\trestart_count = get_be16(block->data + sz - 2);\n+\trestart_count = reftable_get_be16(block->data + sz - 2);\n \trestart_start = sz - 2 - 3 * restart_count;\n \trestart_bytes = block->data + restart_start;\n \n@@ -355,7 +355,7 @@ int block_reader_first_key(const struct block_reader *br, struct reftable_buf *k\n \n static uint32_t block_reader_restart_offset(const struct block_reader *br, size_t idx)\n {\n-\treturn get_be24(br->restart_bytes + 3 * idx);\n+\treturn reftable_get_be24(br->restart_bytes + 3 * idx);\n }\n \n void block_iter_seek_start(struct block_iter *it, const struct block_reader *br)\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 36a5633ede..bf07a0a586 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -101,18 +101,18 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n \t}\n \n \tf++;\n-\tr->block_size = get_be24(f);\n+\tr->block_size = reftable_get_be24(f);\n \n \tf += 3;\n-\tr->min_update_index = get_be64(f);\n+\tr->min_update_index = reftable_get_be64(f);\n \tf += 8;\n-\tr->max_update_index = get_be64(f);\n+\tr->max_update_index = reftable_get_be64(f);\n \tf += 8;\n \n \tif (r->version == 1) {\n \t\tr->hash_id = REFTABLE_HASH_SHA1;\n \t} else {\n-\t\tswitch (get_be32(f)) {\n+\t\tswitch (reftable_get_be32(f)) {\n \t\tcase REFTABLE_FORMAT_ID_SHA1:\n \t\t\tr->hash_id = REFTABLE_HASH_SHA1;\n \t\t\tbreak;\n@@ -127,24 +127,24 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n \t\tf += 4;\n \t}\n \n-\tr->ref_offsets.index_offset = get_be64(f);\n+\tr->ref_offsets.index_offset = reftable_get_be64(f);\n \tf += 8;\n \n-\tr->obj_offsets.offset = get_be64(f);\n+\tr->obj_offsets.offset = reftable_get_be64(f);\n \tf += 8;\n \n \tr->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1);\n \tr->obj_offsets.offset >>= 5;\n \n-\tr->obj_offsets.index_offset = get_be64(f);\n+\tr->obj_offsets.index_offset = reftable_get_be64(f);\n \tf += 8;\n-\tr->log_offsets.offset = get_be64(f);\n+\tr->log_offsets.offset = reftable_get_be64(f);\n \tf += 8;\n-\tr->log_offsets.index_offset = get_be64(f);\n+\tr->log_offsets.index_offset = reftable_get_be64(f);\n \tf += 8;\n \n \tcomputed_crc = crc32(0, footer, f - footer);\n-\tfile_crc = get_be32(f);\n+\tfile_crc = reftable_get_be32(f);\n \tf += 4;\n \tif (computed_crc != file_crc) {\n \t\terr = REFTABLE_FORMAT_ERROR;\n@@ -214,7 +214,7 @@ static int32_t extract_block_size(uint8_t *data, uint8_t *typ, uint64_t off,\n \n \t*typ = data[0];\n \tif (reftable_is_block_type(*typ)) {\n-\t\tresult = get_be24(data + 1);\n+\t\tresult = reftable_get_be24(data + 1);\n \t}\n \treturn result;\n }\ndiff --git a/reftable/record.c b/reftable/record.c\nindex b39d99fcc7..3552bafa99 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -689,7 +689,7 @@ static int reftable_log_record_key(const void *r, struct reftable_buf *dest)\n \t\treturn err;\n \n \tts = (~ts) - rec->update_index;\n-\tput_be64(&i64[0], ts);\n+\treftable_put_be64(&i64[0], ts);\n \n \terr = reftable_buf_add(dest, i64, sizeof(i64));\n \tif (err < 0)\n@@ -814,7 +814,7 @@ static int reftable_log_record_encode(const void *rec, struct string_view s,\n \tif (s.len < 2)\n \t\treturn -1;\n \n-\tput_be16(s.buf, r->value.update.tz_offset);\n+\treftable_put_be16(s.buf, r->value.update.tz_offset);\n \tstring_view_consume(&s, 2);\n \n \tn = encode_string(\n@@ -846,7 +846,7 @@ static int reftable_log_record_decode(void *rec, struct reftable_buf key,\n \t}\n \n \tmemcpy(r->refname, key.buf, key.len - 8);\n-\tts = get_be64(key.buf + key.len - 8);\n+\tts = reftable_get_be64((unsigned char *)key.buf + key.len - 8);\n \n \tr->update_index = (~max) - ts;\n \n@@ -937,7 +937,7 @@ static int reftable_log_record_decode(void *rec, struct reftable_buf key,\n \t\tgoto done;\n \t}\n \n-\tr->value.update.tz_offset = get_be16(in.buf);\n+\tr->value.update.tz_offset = reftable_get_be16(in.buf);\n \tstring_view_consume(&in, 2);\n \n \tn = decode_string(scratch, in);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 155863ee5f..5961698311 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -99,9 +99,9 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \n \tdest[4] = writer_version(w);\n \n-\tput_be24(dest + 5, w->opts.block_size);\n-\tput_be64(dest + 8, w->min_update_index);\n-\tput_be64(dest + 16, w->max_update_index);\n+\treftable_put_be24(dest + 5, w->opts.block_size);\n+\treftable_put_be64(dest + 8, w->min_update_index);\n+\treftable_put_be64(dest + 16, w->max_update_index);\n \tif (writer_version(w) == 2) {\n \t\tuint32_t hash_id;\n \n@@ -116,7 +116,7 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \t\t\treturn -1;\n \t\t}\n \n-\t\tput_be32(dest + 24, hash_id);\n+\t\treftable_put_be32(dest + 24, hash_id);\n \t}\n \n \treturn header_size(writer_version(w));\n@@ -717,19 +717,19 @@ int reftable_writer_close(struct reftable_writer *w)\n \t}\n \n \tp += writer_write_header(w, footer);\n-\tput_be64(p, w->stats.ref_stats.index_offset);\n+\treftable_put_be64(p, w->stats.ref_stats.index_offset);\n \tp += 8;\n-\tput_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n+\treftable_put_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n \tp += 8;\n-\tput_be64(p, w->stats.obj_stats.index_offset);\n+\treftable_put_be64(p, w->stats.obj_stats.index_offset);\n \tp += 8;\n \n-\tput_be64(p, w->stats.log_stats.offset);\n+\treftable_put_be64(p, w->stats.log_stats.offset);\n \tp += 8;\n-\tput_be64(p, w->stats.log_stats.index_offset);\n+\treftable_put_be64(p, w->stats.log_stats.index_offset);\n \tp += 8;\n \n-\tput_be32(p, crc32(0, footer, p - footer));\n+\treftable_put_be32(p, crc32(0, footer, p - footer));\n \tp += 4;\n \n \terr = w->flush(w->write_arg);\ndiff --git a/t/unit-tests/t-reftable-basics.c b/t/unit-tests/t-reftable-basics.c\nindex 9ba7eb05ad..c9e751e49e 100644\n--- a/t/unit-tests/t-reftable-basics.c\n+++ b/t/unit-tests/t-reftable-basics.c\n@@ -128,12 +128,30 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \t\treftable_buf_release(&b);\n \t}\n \n-\tif_test (\"put_be24 and get_be24 work\") {\n+\tif_test (\"reftable_put_be64 and reftable_get_be64 work\") {\n+\t\tuint64_t in = 0x1122334455667788;\n+\t\tuint8_t dest[8];\n+\t\tuint64_t out;\n+\t\treftable_put_be64(dest, in);\n+\t\tout = reftable_get_be64(dest);\n+\t\tcheck_int(in, ==, out);\n+\t}\n+\n+\tif_test (\"reftable_put_be32 and reftable_get_be32 work\") {\n+\t\tuint32_t in = 0x11223344;\n+\t\tuint8_t dest[4];\n+\t\tuint32_t out;\n+\t\treftable_put_be32(dest, in);\n+\t\tout = reftable_get_be32(dest);\n+\t\tcheck_int(in, ==, out);\n+\t}\n+\n+\tif_test (\"reftable_put_be24 and reftable_get_be24 work\") {\n \t\tuint32_t in = 0x112233;\n \t\tuint8_t dest[3];\n \t\tuint32_t out;\n-\t\tput_be24(dest, in);\n-\t\tout = get_be24(dest);\n+\t\treftable_put_be24(dest, in);\n+\t\tout = reftable_get_be24(dest);\n \t\tcheck_int(in, ==, out);\n \t}\n \n@@ -141,8 +159,8 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \t\tuint32_t in = 0xfef1;\n \t\tuint8_t dest[3];\n \t\tuint32_t out;\n-\t\tput_be16(dest, in);\n-\t\tout = get_be16(dest);\n+\t\treftable_put_be16(dest, in);\n+\t\tout = reftable_get_be16(dest);\n \t\tcheck_int(in, ==, out);\n \t}\n \n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"511957","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-11-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","subject":"[PATCH v4 11/18] reftable/system: introduce `reftable_rand()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:13Z","receivedAt":"2025-02-06T07:52:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new system-level `reftable_rand()` function that generates a\nsingle unsigned integer for us. The implementation of this function is\nto be provided by the calling codebase, which allows us to more easily\nhook into pre-seeded random number generators.\n\nAdapt the two callsites where we generated random data.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 4 ++--\n reftable/system.c | 5 +++++\n reftable/system.h | 6 ++++++\n 3 files changed, 13 insertions(+), 2 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 46abf51ec5..e22577ec77 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -523,7 +523,7 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\tclose(fd);\n \t\tfd = -1;\n \n-\t\tdelay = delay + (delay * git_rand(CSPRNG_BYTES_INSECURE)) / UINT32_MAX + 1;\n+\t\tdelay = delay + (delay * reftable_rand()) / UINT32_MAX + 1;\n \t\tsleep_millisec(delay);\n \t}\n \n@@ -688,7 +688,7 @@ int reftable_stack_add(struct reftable_stack *st,\n static int format_name(struct reftable_buf *dest, uint64_t min, uint64_t max)\n {\n \tchar buf[100];\n-\tuint32_t rnd = git_rand(CSPRNG_BYTES_INSECURE);\n+\tuint32_t rnd = reftable_rand();\n \tsnprintf(buf, sizeof(buf), \"0x%012\" PRIx64 \"-0x%012\" PRIx64 \"-%08x\",\n \t\t min, max, rnd);\n \treftable_buf_reset(dest);\ndiff --git a/reftable/system.c b/reftable/system.c\nindex adf8e4d30b..e25ccc0da3 100644\n--- a/reftable/system.c\n+++ b/reftable/system.c\n@@ -4,6 +4,11 @@\n #include \"../lockfile.h\"\n #include \"../tempfile.h\"\n \n+uint32_t reftable_rand(void)\n+{\n+\treturn git_rand(CSPRNG_BYTES_INSECURE);\n+}\n+\n int tmpfile_from_pattern(struct reftable_tmpfile *out, const char *pattern)\n {\n \tstruct tempfile *tempfile;\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 7d5f803eeb..3bd4a4e322 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -13,6 +13,12 @@ license that can be found in the LICENSE file or at\n \n #include \"git-compat-util.h\"\n \n+/*\n+ * Return a random 32 bit integer. This function is expected to return\n+ * pre-seeded data.\n+ */\n+uint32_t reftable_rand(void);\n+\n /*\n * An implementation-specific temporary file. By making this specific to the\n * implementation it becomes possible to tie temporary files into any kind of\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"511958","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-12-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","subject":"[PATCH v4 12/18] reftable/stack: stop using `sleep_millisec()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:14Z","receivedAt":"2025-02-06T07:52:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor our use of `sleep_millisec()` by open-coding it with poll(3p),\nwhich is the current implementation of this function. Ideally, we'd use\na more direct way to sleep, but there is no equivalent to sleep(3p) that\nwould accept milliseconds as input.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex e22577ec77..7b0b10cd38 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -524,7 +524,7 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\tfd = -1;\n \n \t\tdelay = delay + (delay * reftable_rand()) / UINT32_MAX + 1;\n-\t\tsleep_millisec(delay);\n+\t\tpoll(NULL, 0, delay);\n \t}\n \n out:\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"511959","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-13-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","subject":"[PATCH v4 13/18] reftable/basics: stop using `SWAP()` macro","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:15Z","receivedAt":"2025-02-06T07:52:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Stop using `SWAP()` macro in favor of an open-coded variant of it. Note\nthat this also requires us to open-code the build assert that `SWAP()`\nitself uses to verify that the size of both variables matches.\n\nThis is done to reduce our dependency on the Git codebase.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 9 +++++++++\n reftable/merged.c | 2 +-\n reftable/pq.c | 4 ++--\n reftable/record.c | 16 ++++++++--------\n 4 files changed, 20 insertions(+), 11 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex c1ddbaec3f..59000798f0 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -266,6 +266,15 @@ static inline void *reftable_alloc_grow(void *p, size_t nelem, size_t elsize,\n # define strdup(str) REFTABLE_BANNED(strdup)\n #endif\n \n+#define REFTABLE_SWAP(a, b) do {\t\t\t\t\t\t\t\t\\\n+\tvoid *_swap_a_ptr = &(a);\t\t\t\t\t\t\t\t\\\n+\tvoid *_swap_b_ptr = &(b);\t\t\t\t\t\t\t\t\\\n+\tunsigned char _swap_buffer[sizeof(a) - 2 * sizeof(a) * (sizeof(a) != sizeof(b))];\t\\\n+\tmemcpy(_swap_buffer, _swap_a_ptr, sizeof(a));\t\t\t\t\t\t\\\n+\tmemcpy(_swap_a_ptr, _swap_b_ptr, sizeof(a));\t\t\t\t\t\t\\\n+\tmemcpy(_swap_b_ptr, _swap_buffer, sizeof(a));\t\t\t\t\t\t\\\n+} while (0)\n+\n /* Find the longest shared prefix size of `a` and `b` */\n size_t common_prefix_size(struct reftable_buf *a, struct reftable_buf *b);\n \ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 563864068c..4ff1553772 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -155,7 +155,7 @@ static int merged_iter_next_entry(struct merged_iter *mi,\n \t}\n \n \tmi->advance_index = entry.index;\n-\tSWAP(*rec, *entry.rec);\n+\tREFTABLE_SWAP(*rec, *entry.rec);\n \treturn 0;\n }\n \ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex ef8035cfd9..82394a972d 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -57,7 +57,7 @@ int merged_iter_pqueue_remove(struct merged_iter_pqueue *pq, struct pq_entry *ou\n \n \t\tif (min == i)\n \t\t\tbreak;\n-\t\tSWAP(pq->heap[i], pq->heap[min]);\n+\t\tREFTABLE_SWAP(pq->heap[i], pq->heap[min]);\n \t\ti = min;\n \t}\n \n@@ -81,7 +81,7 @@ int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry\n \t\tsize_t j = (i - 1) / 2;\n \t\tif (pq_less(&pq->heap[j], &pq->heap[i]))\n \t\t\tbreak;\n-\t\tSWAP(pq->heap[j], pq->heap[i]);\n+\t\tREFTABLE_SWAP(pq->heap[j], pq->heap[i]);\n \t\ti = j;\n \t}\n \ndiff --git a/reftable/record.c b/reftable/record.c\nindex 3552bafa99..9a1edf39a0 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -237,11 +237,11 @@ static int reftable_ref_record_copy_from(void *rec, const void *src_rec,\n \tsize_t refname_cap = 0;\n \tint err;\n \n-\tSWAP(refname, ref->refname);\n-\tSWAP(refname_cap, ref->refname_cap);\n+\tREFTABLE_SWAP(refname, ref->refname);\n+\tREFTABLE_SWAP(refname_cap, ref->refname_cap);\n \treftable_ref_record_release(ref);\n-\tSWAP(ref->refname, refname);\n-\tSWAP(ref->refname_cap, refname_cap);\n+\tREFTABLE_SWAP(ref->refname, refname);\n+\tREFTABLE_SWAP(ref->refname_cap, refname_cap);\n \n \tif (src->refname) {\n \t\tsize_t refname_len = strlen(src->refname);\n@@ -376,11 +376,11 @@ static int reftable_ref_record_decode(void *rec, struct reftable_buf key,\n \t\treturn n;\n \tstring_view_consume(&in, n);\n \n-\tSWAP(refname, r->refname);\n-\tSWAP(refname_cap, r->refname_cap);\n+\tREFTABLE_SWAP(refname, r->refname);\n+\tREFTABLE_SWAP(refname_cap, r->refname_cap);\n \treftable_ref_record_release(r);\n-\tSWAP(r->refname, refname);\n-\tSWAP(r->refname_cap, refname_cap);\n+\tREFTABLE_SWAP(r->refname, refname);\n+\tREFTABLE_SWAP(r->refname_cap, refname_cap);\n \n \tREFTABLE_ALLOC_GROW_OR_NULL(r->refname, key.len + 1, r->refname_cap);\n \tif (!r->refname) {\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"511960","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-14-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","subject":"[PATCH v4 14/18] reftable/basics: stop using `UNUSED` annotation","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:16Z","receivedAt":"2025-02-06T07:52:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Stop using the `UNUSED` annotation and replace it with a new\n`REFTABLE_UNUSED` macro. The latter is a weaker guarantee compared to\n`UNUSED` as it only suppresses unused parameters without generating a\nwarning in case a parameter marked as unused is in fact used. But it's\ngood enough, and by relaxing the behaviour a bit we avoid having to wire\nup compiler-specific logic.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 2 ++\n reftable/blocksource.c | 10 +++++++---\n reftable/iter.c | 17 ++++++++++++-----\n reftable/record.c | 51 ++++++++++++++++++++++++++++++++++++--------------\n reftable/writer.c | 4 +++-\n 5 files changed, 61 insertions(+), 23 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 59000798f0..4d0645a4e9 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -16,6 +16,8 @@ license that can be found in the LICENSE file or at\n #include \"system.h\"\n #include \"reftable-basics.h\"\n \n+#define REFTABLE_UNUSED(x) (void)(x)\n+\n struct reftable_buf {\n \tsize_t alloc;\n \tsize_t len;\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex 02972c46f4..bfd64b0e48 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -13,15 +13,17 @@ license that can be found in the LICENSE file or at\n #include \"reftable-blocksource.h\"\n #include \"reftable-error.h\"\n \n-static void reftable_buf_return_block(void *b UNUSED, struct reftable_block *dest)\n+static void reftable_buf_return_block(void *b, struct reftable_block *dest)\n {\n+\tREFTABLE_UNUSED(b);\n \tif (dest->len)\n \t\tmemset(dest->data, 0xff, dest->len);\n \treftable_free(dest->data);\n }\n \n-static void reftable_buf_close(void *b UNUSED)\n+static void reftable_buf_close(void *b)\n {\n+\tREFTABLE_UNUSED(b);\n }\n \n static ssize_t reftable_buf_read_block(void *v, struct reftable_block *dest,\n@@ -67,8 +69,10 @@ static uint64_t file_size(void *b)\n \treturn ((struct file_block_source *)b)->size;\n }\n \n-static void file_return_block(void *b UNUSED, struct reftable_block *dest UNUSED)\n+static void file_return_block(void *b, struct reftable_block *dest)\n {\n+\tREFTABLE_UNUSED(b);\n+\tREFTABLE_UNUSED(dest);\n }\n \n static void file_close(void *v)\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex b2ffb09c16..452add2705 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -25,18 +25,23 @@ int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n \treturn it->ops->next(it->iter_arg, rec);\n }\n \n-static int empty_iterator_seek(void *arg UNUSED, struct reftable_record *want UNUSED)\n+static int empty_iterator_seek(void *arg, struct reftable_record *want)\n {\n+\tREFTABLE_UNUSED(arg);\n+\tREFTABLE_UNUSED(want);\n \treturn 0;\n }\n \n-static int empty_iterator_next(void *arg UNUSED, struct reftable_record *rec UNUSED)\n+static int empty_iterator_next(void *arg, struct reftable_record *rec)\n {\n+\tREFTABLE_UNUSED(arg);\n+\tREFTABLE_UNUSED(rec);\n \treturn 1;\n }\n \n-static void empty_iterator_close(void *arg UNUSED)\n+static void empty_iterator_close(void *arg)\n {\n+\tREFTABLE_UNUSED(arg);\n }\n \n static struct reftable_iterator_vtable empty_vtable = {\n@@ -143,9 +148,11 @@ static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n \treturn 0;\n }\n \n-static int indexed_table_ref_iter_seek(void *p UNUSED,\n-\t\t\t\t struct reftable_record *want UNUSED)\n+static int indexed_table_ref_iter_seek(void *p,\n+\t\t\t\t struct reftable_record *want)\n {\n+\tREFTABLE_UNUSED(p);\n+\tREFTABLE_UNUSED(want);\n \treturn REFTABLE_API_ERROR;\n }\n \ndiff --git a/reftable/record.c b/reftable/record.c\nindex 9a1edf39a0..5ee2fe44a7 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -490,11 +490,13 @@ static void reftable_obj_record_release(void *rec)\n }\n \n static int reftable_obj_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t\t uint32_t hash_size)\n {\n \tstruct reftable_obj_record *obj = rec;\n \tconst struct reftable_obj_record *src = src_rec;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\n \treftable_obj_record_release(obj);\n \n \tREFTABLE_ALLOC_ARRAY(obj->hash_prefix, src->hash_prefix_len);\n@@ -528,13 +530,16 @@ static uint8_t reftable_obj_record_val_type(const void *rec)\n }\n \n static int reftable_obj_record_encode(const void *rec, struct string_view s,\n-\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t uint32_t hash_size)\n {\n \tconst struct reftable_obj_record *r = rec;\n \tstruct string_view start = s;\n \tint i = 0;\n \tint n = 0;\n \tuint64_t last = 0;\n+\n+\tREFTABLE_UNUSED(hash_size);\n+\n \tif (r->offset_len == 0 || r->offset_len >= 8) {\n \t\tn = put_var_int(&s, r->offset_len);\n \t\tif (n < 0) {\n@@ -563,8 +568,8 @@ static int reftable_obj_record_encode(const void *rec, struct string_view s,\n \n static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n \t\t\t\t uint8_t val_type, struct string_view in,\n-\t\t\t\t uint32_t hash_size UNUSED,\n-\t\t\t\t struct reftable_buf *scratch UNUSED)\n+\t\t\t\t uint32_t hash_size,\n+\t\t\t\t struct reftable_buf *scratch)\n {\n \tstruct string_view start = in;\n \tstruct reftable_obj_record *r = rec;\n@@ -572,6 +577,9 @@ static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n \tint n = 0;\n \tuint64_t last;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\tREFTABLE_UNUSED(scratch);\n+\n \treftable_obj_record_release(r);\n \n \tREFTABLE_ALLOC_ARRAY(r->hash_prefix, key.len);\n@@ -618,17 +626,20 @@ static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n \treturn start.len - in.len;\n }\n \n-static int not_a_deletion(const void *p UNUSED)\n+static int not_a_deletion(const void *p)\n {\n+\tREFTABLE_UNUSED(p);\n \treturn 0;\n }\n \n static int reftable_obj_record_equal_void(const void *a, const void *b,\n-\t\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t\t uint32_t hash_size)\n {\n \tstruct reftable_obj_record *ra = (struct reftable_obj_record *) a;\n \tstruct reftable_obj_record *rb = (struct reftable_obj_record *) b;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\n \tif (ra->hash_prefix_len != rb->hash_prefix_len\n \t || ra->offset_len != rb->offset_len)\n \t\treturn 0;\n@@ -1054,12 +1065,14 @@ static int reftable_index_record_key(const void *r, struct reftable_buf *dest)\n }\n \n static int reftable_index_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t\t uint32_t hash_size)\n {\n \tstruct reftable_index_record *dst = rec;\n \tconst struct reftable_index_record *src = src_rec;\n \tint err;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\n \treftable_buf_reset(&dst->last_key);\n \terr = reftable_buf_add(&dst->last_key, src->last_key.buf, src->last_key.len);\n \tif (err < 0)\n@@ -1075,19 +1088,23 @@ static void reftable_index_record_release(void *rec)\n \treftable_buf_release(&idx->last_key);\n }\n \n-static uint8_t reftable_index_record_val_type(const void *rec UNUSED)\n+static uint8_t reftable_index_record_val_type(const void *rec)\n {\n+\tREFTABLE_UNUSED(rec);\n \treturn 0;\n }\n \n static int reftable_index_record_encode(const void *rec, struct string_view out,\n-\t\t\t\t\tuint32_t hash_size UNUSED)\n+\t\t\t\t\tuint32_t hash_size)\n {\n \tconst struct reftable_index_record *r =\n \t\t(const struct reftable_index_record *)rec;\n \tstruct string_view start = out;\n+\tint n;\n \n-\tint n = put_var_int(&out, r->offset);\n+\tREFTABLE_UNUSED(hash_size);\n+\n+\tn = put_var_int(&out, r->offset);\n \tif (n < 0)\n \t\treturn n;\n \n@@ -1097,15 +1114,19 @@ static int reftable_index_record_encode(const void *rec, struct string_view out,\n }\n \n static int reftable_index_record_decode(void *rec, struct reftable_buf key,\n-\t\t\t\t\tuint8_t val_type UNUSED,\n+\t\t\t\t\tuint8_t val_type,\n \t\t\t\t\tstruct string_view in,\n-\t\t\t\t\tuint32_t hash_size UNUSED,\n-\t\t\t\t\tstruct reftable_buf *scratch UNUSED)\n+\t\t\t\t\tuint32_t hash_size,\n+\t\t\t\t\tstruct reftable_buf *scratch)\n {\n \tstruct string_view start = in;\n \tstruct reftable_index_record *r = rec;\n \tint err, n = 0;\n \n+\tREFTABLE_UNUSED(val_type);\n+\tREFTABLE_UNUSED(hash_size);\n+\tREFTABLE_UNUSED(scratch);\n+\n \treftable_buf_reset(&r->last_key);\n \terr = reftable_buf_add(&r->last_key, key.buf, key.len);\n \tif (err < 0)\n@@ -1120,11 +1141,13 @@ static int reftable_index_record_decode(void *rec, struct reftable_buf key,\n }\n \n static int reftable_index_record_equal(const void *a, const void *b,\n-\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t uint32_t hash_size)\n {\n \tstruct reftable_index_record *ia = (struct reftable_index_record *) a;\n \tstruct reftable_index_record *ib = (struct reftable_index_record *) b;\n \n+\tREFTABLE_UNUSED(hash_size);\n+\n \treturn ia->offset == ib->offset && !reftable_buf_cmp(&ia->last_key, &ib->last_key);\n }\n \ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 5961698311..0040a1b1c4 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -636,10 +636,12 @@ static void write_object_record(void *void_arg, void *key)\n done:;\n }\n \n-static void object_record_free(void *void_arg UNUSED, void *key)\n+static void object_record_free(void *void_arg, void *key)\n {\n \tstruct obj_index_tree_node *entry = key;\n \n+\tREFTABLE_UNUSED(void_arg);\n+\n \tREFTABLE_FREE_AND_NULL(entry->offsets);\n \treftable_buf_release(&entry->hash);\n \treftable_free(entry);\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"511961","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-15-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","subject":"[PATCH v4 15/18] compat/mingw: split out POSIX-related bits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:17Z","receivedAt":"2025-02-06T07:52:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Split out the POSIX-related bits from \"compat/mingw.h\". This is in\npreparation for splitting up \"git-compat-utils.h\" into a header that\nprovides POSIX-compatibility and a header that provides common wrappers\nused by the Git project.\n\nSigned-off-by: Patrick Steinhardt \n---\n compat/{mingw.c => mingw/compat-util.c} | 28 ++--\n compat/mingw/compat-util.h | 220 ++++++++++++++++++++++++++++++++\n compat/{mingw.h => mingw/posix.h} | 216 +------------------------------\n compat/msvc.c | 6 -\n compat/msvc/compat-util.c | 6 +\n compat/msvc/compat-util.h | 7 +\n compat/{msvc.h => msvc/posix.h} | 8 +-\n config.mak.uname | 6 +-\n contrib/buildsystems/CMakeLists.txt | 2 +-\n git-compat-util.h | 4 +-\n meson.build | 8 +-\n 11 files changed, 266 insertions(+), 245 deletions(-)\n\ndiff --git a/compat/mingw.c b/compat/mingw/compat-util.c\nsimilarity index 99%\nrename from compat/mingw.c\nrename to compat/mingw/compat-util.c\nindex 1d5b211b54..aeb569bcea 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw/compat-util.c\n@@ -1,24 +1,24 @@\n #define USE_THE_REPOSITORY_VARIABLE\n #define DISABLE_SIGN_COMPARE_WARNINGS\n \n-#include \"../git-compat-util.h\"\n-#include \"win32.h\"\n+#include \"../../git-compat-util.h\"\n+#include \"../win32.h\"\n #include \n #include \n #include \n #include \n-#include \"../strbuf.h\"\n-#include \"../run-command.h\"\n-#include \"../abspath.h\"\n-#include \"../alloc.h\"\n-#include \"win32/lazyload.h\"\n-#include \"../config.h\"\n-#include \"../environment.h\"\n-#include \"../trace2.h\"\n-#include \"../symlinks.h\"\n-#include \"../wrapper.h\"\n-#include \"dir.h\"\n-#include \"gettext.h\"\n+#include \"../../strbuf.h\"\n+#include \"../../run-command.h\"\n+#include \"../../abspath.h\"\n+#include \"../../alloc.h\"\n+#include \"../win32/lazyload.h\"\n+#include \"../../config.h\"\n+#include \"../../environment.h\"\n+#include \"../../trace2.h\"\n+#include \"../../symlinks.h\"\n+#include \"../../wrapper.h\"\n+#include \"../../dir.h\"\n+#include \"../../gettext.h\"\n #define SECURITY_WIN32\n #include \n \ndiff --git a/compat/mingw/compat-util.h b/compat/mingw/compat-util.h\nnew file mode 100644\nindex 0000000000..470542d309\n--- /dev/null\n+++ b/compat/mingw/compat-util.h\n@@ -0,0 +1,220 @@\n+#ifndef COMPAT_MINGW_COMPAT_UTIL_H\n+#define COMPAT_MINGW_COMPAT_UTIL_H\n+\n+#include \"posix.h\"\n+\n+struct config_context;\n+int mingw_core_config(const char *var, const char *value,\n+\t\t const struct config_context *ctx, void *cb);\n+#define platform_core_config mingw_core_config\n+\n+#ifndef NO_OPENSSL\n+#include \n+static inline int mingw_SSL_set_fd(SSL *ssl, int fd)\n+{\n+\treturn SSL_set_fd(ssl, _get_osfhandle(fd));\n+}\n+#define SSL_set_fd mingw_SSL_set_fd\n+\n+static inline int mingw_SSL_set_rfd(SSL *ssl, int fd)\n+{\n+\treturn SSL_set_rfd(ssl, _get_osfhandle(fd));\n+}\n+#define SSL_set_rfd mingw_SSL_set_rfd\n+\n+static inline int mingw_SSL_set_wfd(SSL *ssl, int fd)\n+{\n+\treturn SSL_set_wfd(ssl, _get_osfhandle(fd));\n+}\n+#define SSL_set_wfd mingw_SSL_set_wfd\n+#endif\n+\n+/*\n+ * git specific compatibility\n+ */\n+\n+static inline void convert_slashes(char *path)\n+{\n+\tfor (; *path; path++)\n+\t\tif (*path == '\\\\')\n+\t\t\t*path = '/';\n+}\n+#define PATH_SEP ';'\n+char *mingw_query_user_email(void);\n+#define query_user_email mingw_query_user_email\n+\n+/**\n+ * Verifies that the specified path is owned by the user running the\n+ * current process.\n+ */\n+int is_path_owned_by_current_sid(const char *path, struct strbuf *report);\n+#define is_path_owned_by_current_user is_path_owned_by_current_sid\n+\n+/**\n+ * Verifies that the given path is a valid one on Windows.\n+ *\n+ * In particular, path segments are disallowed which\n+ *\n+ * - end in a period or a space (except the special directories `.` and `..`).\n+ *\n+ * - contain any of the reserved characters, e.g. `:`, `;`, `*`, etc\n+ *\n+ * - correspond to reserved names (such as `AUX`, `PRN`, etc)\n+ *\n+ * The `allow_literal_nul` parameter controls whether the path `NUL` should\n+ * be considered valid (this makes sense e.g. before opening files, as it is\n+ * perfectly legitimate to open `NUL` on Windows, just as it is to open\n+ * `/dev/null` on Unix/Linux).\n+ *\n+ * Returns 1 upon success, otherwise 0.\n+ */\n+int is_valid_win32_path(const char *path, int allow_literal_nul);\n+#define is_valid_path(path) is_valid_win32_path(path, 0)\n+\n+/**\n+ * Converts UTF-8 encoded string to UTF-16LE.\n+ *\n+ * To support repositories with legacy-encoded file names, invalid UTF-8 bytes\n+ * 0xa0 - 0xff are converted to corresponding printable Unicode chars \\u00a0 -\n+ * \\u00ff, and invalid UTF-8 bytes 0x80 - 0x9f (which would make non-printable\n+ * Unicode) are converted to hex-code.\n+ *\n+ * Lead-bytes not followed by an appropriate number of trail-bytes, over-long\n+ * encodings and 4-byte encodings > \\u10ffff are detected as invalid UTF-8.\n+ *\n+ * Maximum space requirement for the target buffer is two wide chars per UTF-8\n+ * char (((strlen(utf) * 2) + 1) [* sizeof(wchar_t)]).\n+ *\n+ * The maximum space is needed only if the entire input string consists of\n+ * invalid UTF-8 bytes in range 0x80-0x9f, as per the following table:\n+ *\n+ * | | UTF-8 | UTF-16 |\n+ * Code point | UTF-8 sequence | bytes | words | ratio\n+ * --------------+-------------------+-------+--------+-------\n+ * 000000-00007f | 0-7f | 1 | 1 | 1\n+ * 000080-0007ff | c2-df + 80-bf | 2 | 1 | 0.5\n+ * 000800-00ffff | e0-ef + 2 * 80-bf | 3 | 1 | 0.33\n+ * 010000-10ffff | f0-f4 + 3 * 80-bf | 4 | 2 (a) | 0.5\n+ * invalid | 80-9f | 1 | 2 (b) | 2\n+ * invalid | a0-ff | 1 | 1 | 1\n+ *\n+ * (a) encoded as UTF-16 surrogate pair\n+ * (b) encoded as two hex digits\n+ *\n+ * Note that, while the UTF-8 encoding scheme can be extended to 5-byte, 6-byte\n+ * or even indefinite-byte sequences, the largest valid code point \\u10ffff\n+ * encodes as only 4 UTF-8 bytes.\n+ *\n+ * Parameters:\n+ * wcs: wide char target buffer\n+ * utf: string to convert\n+ * wcslen: size of target buffer (in wchar_t's)\n+ * utflen: size of string to convert, or -1 if 0-terminated\n+ *\n+ * Returns:\n+ * length of converted string (_wcslen(wcs)), or -1 on failure\n+ *\n+ * Errors:\n+ * EINVAL: one of the input parameters is invalid (e.g. NULL)\n+ * ERANGE: the output buffer is too small\n+ */\n+int xutftowcsn(wchar_t *wcs, const char *utf, size_t wcslen, int utflen);\n+\n+/**\n+ * Simplified variant of xutftowcsn, assumes input string is \\0-terminated.\n+ */\n+static inline int xutftowcs(wchar_t *wcs, const char *utf, size_t wcslen)\n+{\n+\treturn xutftowcsn(wcs, utf, wcslen, -1);\n+}\n+\n+/**\n+ * Simplified file system specific variant of xutftowcsn, assumes output\n+ * buffer size is MAX_PATH wide chars and input string is \\0-terminated,\n+ * fails with ENAMETOOLONG if input string is too long.\n+ */\n+static inline int xutftowcs_path(wchar_t *wcs, const char *utf)\n+{\n+\tint result = xutftowcsn(wcs, utf, MAX_PATH, -1);\n+\tif (result < 0 && errno == ERANGE)\n+\t\terrno = ENAMETOOLONG;\n+\treturn result;\n+}\n+\n+/**\n+ * Converts UTF-16LE encoded string to UTF-8.\n+ *\n+ * Maximum space requirement for the target buffer is three UTF-8 chars per\n+ * wide char ((_wcslen(wcs) * 3) + 1).\n+ *\n+ * The maximum space is needed only if the entire input string consists of\n+ * UTF-16 words in range 0x0800-0xd7ff or 0xe000-0xffff (i.e. \\u0800-\\uffff\n+ * modulo surrogate pairs), as per the following table:\n+ *\n+ * | | UTF-16 | UTF-8 |\n+ * Code point | UTF-16 sequence | words | bytes | ratio\n+ * --------------+-----------------------+--------+-------+-------\n+ * 000000-00007f | 0000-007f | 1 | 1 | 1\n+ * 000080-0007ff | 0080-07ff | 1 | 2 | 2\n+ * 000800-00ffff | 0800-d7ff / e000-ffff | 1 | 3 | 3\n+ * 010000-10ffff | d800-dbff + dc00-dfff | 2 | 4 | 2\n+ *\n+ * Note that invalid code points > 10ffff cannot be represented in UTF-16.\n+ *\n+ * Parameters:\n+ * utf: target buffer\n+ * wcs: wide string to convert\n+ * utflen: size of target buffer\n+ *\n+ * Returns:\n+ * length of converted string, or -1 on failure\n+ *\n+ * Errors:\n+ * EINVAL: one of the input parameters is invalid (e.g. NULL)\n+ * ERANGE: the output buffer is too small\n+ */\n+int xwcstoutf(char *utf, const wchar_t *wcs, size_t utflen);\n+\n+/*\n+ * A critical section used in the implementation of the spawn\n+ * functions (mingw_spawnv[p]e()) and waitpid(). Initialised in\n+ * the replacement main() macro below.\n+ */\n+extern CRITICAL_SECTION pinfo_cs;\n+\n+/*\n+ * Git, like most portable C applications, implements a main() function. On\n+ * Windows, this main() function would receive parameters encoded in the\n+ * current locale, but Git for Windows would prefer UTF-8 encoded parameters.\n+ *\n+ * To make that happen, we still declare main() here, and then declare and\n+ * implement wmain() (which is the Unicode variant of main()) and compile with\n+ * -municode. This wmain() function reencodes the parameters from UTF-16 to\n+ * UTF-8 format, sets up a couple of other things as required on Windows, and\n+ * then hands off to the main() function.\n+ */\n+int wmain(int argc, const wchar_t **w_argv);\n+int main(int argc, const char **argv);\n+\n+/*\n+ * For debugging: if a problem occurs, say, in a Git process that is spawned\n+ * from another Git process which in turn is spawned from yet another Git\n+ * process, it can be quite daunting to figure out what is going on.\n+ *\n+ * Call this function to open a new MinTTY (this assumes you are in Git for\n+ * Windows' SDK) with a GDB that attaches to the current process right away.\n+ */\n+void open_in_gdb(void);\n+\n+/*\n+ * Used by Pthread API implementation for Windows\n+ */\n+int err_win_to_posix(DWORD winerr);\n+\n+#ifndef NO_UNIX_SOCKETS\n+int mingw_have_unix_sockets(void);\n+#undef have_unix_sockets\n+#define have_unix_sockets mingw_have_unix_sockets\n+#endif\n+\n+#endif /* COMPAT_MINGW_COMPAT_UTIL_H */\ndiff --git a/compat/mingw.h b/compat/mingw/posix.h\nsimilarity index 59%\nrename from compat/mingw.h\nrename to compat/mingw/posix.h\nindex ebfb8ba423..8dddfa818d 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw/posix.h\n@@ -1,3 +1,6 @@\n+#ifndef COMPAT_MINGW_POSIX_H\n+#define COMPAT_MINGW_POSIX_H\n+\n #ifdef __MINGW64_VERSION_MAJOR\n #include \n #include \n@@ -11,11 +14,6 @@ typedef _sigset_t sigset_t;\n #undef _POSIX_THREAD_SAFE_FUNCTIONS\n #endif\n \n-struct config_context;\n-int mingw_core_config(const char *var, const char *value,\n-\t\t const struct config_context *ctx, void *cb);\n-#define platform_core_config mingw_core_config\n-\n /*\n * things that are not available in header files\n */\n@@ -180,27 +178,6 @@ int mingw_kill(pid_t pid, int sig);\n #define locate_in_PATH mingw_locate_in_PATH\n char *mingw_locate_in_PATH(const char *cmd);\n \n-#ifndef NO_OPENSSL\n-#include \n-static inline int mingw_SSL_set_fd(SSL *ssl, int fd)\n-{\n-\treturn SSL_set_fd(ssl, _get_osfhandle(fd));\n-}\n-#define SSL_set_fd mingw_SSL_set_fd\n-\n-static inline int mingw_SSL_set_rfd(SSL *ssl, int fd)\n-{\n-\treturn SSL_set_rfd(ssl, _get_osfhandle(fd));\n-}\n-#define SSL_set_rfd mingw_SSL_set_rfd\n-\n-static inline int mingw_SSL_set_wfd(SSL *ssl, int fd)\n-{\n-\treturn SSL_set_wfd(ssl, _get_osfhandle(fd));\n-}\n-#define SSL_set_wfd mingw_SSL_set_wfd\n-#endif\n-\n /*\n * implementations of missing functions\n */\n@@ -444,19 +421,6 @@ int winansi_dup2(int oldfd, int newfd);\n void winansi_init(void);\n HANDLE winansi_get_osfhandle(int fd);\n \n-/*\n- * git specific compatibility\n- */\n-\n-static inline void convert_slashes(char *path)\n-{\n-\tfor (; *path; path++)\n-\t\tif (*path == '\\\\')\n-\t\t\t*path = '/';\n-}\n-#define PATH_SEP ';'\n-char *mingw_query_user_email(void);\n-#define query_user_email mingw_query_user_email\n #if !defined(__MINGW64_VERSION_MAJOR) && (!defined(_MSC_VER) || _MSC_VER < 1800)\n #define PRIuMAX \"I64u\"\n #define PRId64 \"I64d\"\n@@ -464,176 +428,4 @@ char *mingw_query_user_email(void);\n #include \n #endif\n \n-/**\n- * Verifies that the specified path is owned by the user running the\n- * current process.\n- */\n-int is_path_owned_by_current_sid(const char *path, struct strbuf *report);\n-#define is_path_owned_by_current_user is_path_owned_by_current_sid\n-\n-/**\n- * Verifies that the given path is a valid one on Windows.\n- *\n- * In particular, path segments are disallowed which\n- *\n- * - end in a period or a space (except the special directories `.` and `..`).\n- *\n- * - contain any of the reserved characters, e.g. `:`, `;`, `*`, etc\n- *\n- * - correspond to reserved names (such as `AUX`, `PRN`, etc)\n- *\n- * The `allow_literal_nul` parameter controls whether the path `NUL` should\n- * be considered valid (this makes sense e.g. before opening files, as it is\n- * perfectly legitimate to open `NUL` on Windows, just as it is to open\n- * `/dev/null` on Unix/Linux).\n- *\n- * Returns 1 upon success, otherwise 0.\n- */\n-int is_valid_win32_path(const char *path, int allow_literal_nul);\n-#define is_valid_path(path) is_valid_win32_path(path, 0)\n-\n-/**\n- * Converts UTF-8 encoded string to UTF-16LE.\n- *\n- * To support repositories with legacy-encoded file names, invalid UTF-8 bytes\n- * 0xa0 - 0xff are converted to corresponding printable Unicode chars \\u00a0 -\n- * \\u00ff, and invalid UTF-8 bytes 0x80 - 0x9f (which would make non-printable\n- * Unicode) are converted to hex-code.\n- *\n- * Lead-bytes not followed by an appropriate number of trail-bytes, over-long\n- * encodings and 4-byte encodings > \\u10ffff are detected as invalid UTF-8.\n- *\n- * Maximum space requirement for the target buffer is two wide chars per UTF-8\n- * char (((strlen(utf) * 2) + 1) [* sizeof(wchar_t)]).\n- *\n- * The maximum space is needed only if the entire input string consists of\n- * invalid UTF-8 bytes in range 0x80-0x9f, as per the following table:\n- *\n- * | | UTF-8 | UTF-16 |\n- * Code point | UTF-8 sequence | bytes | words | ratio\n- * --------------+-------------------+-------+--------+-------\n- * 000000-00007f | 0-7f | 1 | 1 | 1\n- * 000080-0007ff | c2-df + 80-bf | 2 | 1 | 0.5\n- * 000800-00ffff | e0-ef + 2 * 80-bf | 3 | 1 | 0.33\n- * 010000-10ffff | f0-f4 + 3 * 80-bf | 4 | 2 (a) | 0.5\n- * invalid | 80-9f | 1 | 2 (b) | 2\n- * invalid | a0-ff | 1 | 1 | 1\n- *\n- * (a) encoded as UTF-16 surrogate pair\n- * (b) encoded as two hex digits\n- *\n- * Note that, while the UTF-8 encoding scheme can be extended to 5-byte, 6-byte\n- * or even indefinite-byte sequences, the largest valid code point \\u10ffff\n- * encodes as only 4 UTF-8 bytes.\n- *\n- * Parameters:\n- * wcs: wide char target buffer\n- * utf: string to convert\n- * wcslen: size of target buffer (in wchar_t's)\n- * utflen: size of string to convert, or -1 if 0-terminated\n- *\n- * Returns:\n- * length of converted string (_wcslen(wcs)), or -1 on failure\n- *\n- * Errors:\n- * EINVAL: one of the input parameters is invalid (e.g. NULL)\n- * ERANGE: the output buffer is too small\n- */\n-int xutftowcsn(wchar_t *wcs, const char *utf, size_t wcslen, int utflen);\n-\n-/**\n- * Simplified variant of xutftowcsn, assumes input string is \\0-terminated.\n- */\n-static inline int xutftowcs(wchar_t *wcs, const char *utf, size_t wcslen)\n-{\n-\treturn xutftowcsn(wcs, utf, wcslen, -1);\n-}\n-\n-/**\n- * Simplified file system specific variant of xutftowcsn, assumes output\n- * buffer size is MAX_PATH wide chars and input string is \\0-terminated,\n- * fails with ENAMETOOLONG if input string is too long.\n- */\n-static inline int xutftowcs_path(wchar_t *wcs, const char *utf)\n-{\n-\tint result = xutftowcsn(wcs, utf, MAX_PATH, -1);\n-\tif (result < 0 && errno == ERANGE)\n-\t\terrno = ENAMETOOLONG;\n-\treturn result;\n-}\n-\n-/**\n- * Converts UTF-16LE encoded string to UTF-8.\n- *\n- * Maximum space requirement for the target buffer is three UTF-8 chars per\n- * wide char ((_wcslen(wcs) * 3) + 1).\n- *\n- * The maximum space is needed only if the entire input string consists of\n- * UTF-16 words in range 0x0800-0xd7ff or 0xe000-0xffff (i.e. \\u0800-\\uffff\n- * modulo surrogate pairs), as per the following table:\n- *\n- * | | UTF-16 | UTF-8 |\n- * Code point | UTF-16 sequence | words | bytes | ratio\n- * --------------+-----------------------+--------+-------+-------\n- * 000000-00007f | 0000-007f | 1 | 1 | 1\n- * 000080-0007ff | 0080-07ff | 1 | 2 | 2\n- * 000800-00ffff | 0800-d7ff / e000-ffff | 1 | 3 | 3\n- * 010000-10ffff | d800-dbff + dc00-dfff | 2 | 4 | 2\n- *\n- * Note that invalid code points > 10ffff cannot be represented in UTF-16.\n- *\n- * Parameters:\n- * utf: target buffer\n- * wcs: wide string to convert\n- * utflen: size of target buffer\n- *\n- * Returns:\n- * length of converted string, or -1 on failure\n- *\n- * Errors:\n- * EINVAL: one of the input parameters is invalid (e.g. NULL)\n- * ERANGE: the output buffer is too small\n- */\n-int xwcstoutf(char *utf, const wchar_t *wcs, size_t utflen);\n-\n-/*\n- * A critical section used in the implementation of the spawn\n- * functions (mingw_spawnv[p]e()) and waitpid(). Initialised in\n- * the replacement main() macro below.\n- */\n-extern CRITICAL_SECTION pinfo_cs;\n-\n-/*\n- * Git, like most portable C applications, implements a main() function. On\n- * Windows, this main() function would receive parameters encoded in the\n- * current locale, but Git for Windows would prefer UTF-8 encoded parameters.\n- *\n- * To make that happen, we still declare main() here, and then declare and\n- * implement wmain() (which is the Unicode variant of main()) and compile with\n- * -municode. This wmain() function reencodes the parameters from UTF-16 to\n- * UTF-8 format, sets up a couple of other things as required on Windows, and\n- * then hands off to the main() function.\n- */\n-int wmain(int argc, const wchar_t **w_argv);\n-int main(int argc, const char **argv);\n-\n-/*\n- * For debugging: if a problem occurs, say, in a Git process that is spawned\n- * from another Git process which in turn is spawned from yet another Git\n- * process, it can be quite daunting to figure out what is going on.\n- *\n- * Call this function to open a new MinTTY (this assumes you are in Git for\n- * Windows' SDK) with a GDB that attaches to the current process right away.\n- */\n-void open_in_gdb(void);\n-\n-/*\n- * Used by Pthread API implementation for Windows\n- */\n-int err_win_to_posix(DWORD winerr);\n-\n-#ifndef NO_UNIX_SOCKETS\n-int mingw_have_unix_sockets(void);\n-#undef have_unix_sockets\n-#define have_unix_sockets mingw_have_unix_sockets\n-#endif\n+#endif /* COMPAT_MINGW_POSIX_H */\ndiff --git a/compat/msvc.c b/compat/msvc.c\ndeleted file mode 100644\nindex 71843d7eef..0000000000\n--- a/compat/msvc.c\n+++ /dev/null\n@@ -1,6 +0,0 @@\n-#include \"../git-compat-util.h\"\n-#include \"win32.h\"\n-#include \n-#include \"../strbuf.h\"\n-\n-#include \"mingw.c\"\ndiff --git a/compat/msvc/compat-util.c b/compat/msvc/compat-util.c\nnew file mode 100644\nindex 0000000000..b92e52b8b8\n--- /dev/null\n+++ b/compat/msvc/compat-util.c\n@@ -0,0 +1,6 @@\n+#include \"../../git-compat-util.h\"\n+#include \"../win32.h\"\n+#include \n+#include \"../../strbuf.h\"\n+\n+#include \"../mingw/compat-util.c\"\ndiff --git a/compat/msvc/compat-util.h b/compat/msvc/compat-util.h\nnew file mode 100644\nindex 0000000000..404af236b5\n--- /dev/null\n+++ b/compat/msvc/compat-util.h\n@@ -0,0 +1,7 @@\n+#ifndef COMPAT_MSVC_COMPAT_UTIL_H\n+#define COMPAT_MSVC_COMPAT_UTIL_H\n+\n+#include \"posix.h\"\n+#include \"../mingw/compat-util.h\"\n+\n+#endif /* COMPAT_MSVC_COMPAT_UTIL_H */\ndiff --git a/compat/msvc.h b/compat/msvc/posix.h\nsimilarity index 86%\nrename from compat/msvc.h\nrename to compat/msvc/posix.h\nindex 1d7a8c6145..703a7c2067 100644\n--- a/compat/msvc.h\n+++ b/compat/msvc/posix.h\n@@ -1,5 +1,5 @@\n-#ifndef __MSVC__HEAD\n-#define __MSVC__HEAD\n+#ifndef COMPAT_MSVC_POSIX_H\n+#define COMPAT_MSVC_POSIX_H\n \n #include \n #include \n@@ -28,6 +28,6 @@ typedef int sigset_t;\n /* open for reading, writing, or both (not in fcntl.h) */\n #define O_ACCMODE (_O_RDONLY | _O_WRONLY | _O_RDWR)\n \n-#include \"compat/mingw.h\"\n+#include \"../mingw/posix.h\"\n \n-#endif\n+#endif /* COMPAT_MSVC_POSIX_H */\ndiff --git a/config.mak.uname b/config.mak.uname\nindex b12d4e168a..dbd05adb5d 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -495,7 +495,7 @@ endif\n \tAR = compat/vcbuild/scripts/lib.pl\n \tCFLAGS =\n \tBASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE\n-\tCOMPAT_OBJS = compat/msvc.o compat/winansi.o \\\n+\tCOMPAT_OBJS = compat/msvc/compat-util.o compat/winansi.o \\\n \t\tcompat/win32/flush.o \\\n \t\tcompat/win32/path-utils.o \\\n \t\tcompat/win32/pthread.o compat/win32/syslog.o \\\n@@ -541,7 +541,7 @@ endif\n \n \tEXTRA_PROGRAMS += headless-git$X\n \n-compat/msvc.o: compat/msvc.c compat/mingw.c GIT-CFLAGS\n+compat/msvc/compat-util.o: compat/msvc/compat-util.c compat/mingw/compat-util.c GIT-CFLAGS\n endif\n ifeq ($(uname_S),Interix)\n \tNO_INITGROUPS = YesPlease\n@@ -694,7 +694,7 @@ ifeq ($(uname_S),MINGW)\n \tBASIC_LDFLAGS += -municode\n \tCOMPAT_CFLAGS += -DNOGDI -Icompat -Icompat/win32\n \tCOMPAT_CFLAGS += -DSTRIP_EXTENSION=\\\".exe\\\"\n-\tCOMPAT_OBJS += compat/mingw.o compat/winansi.o \\\n+\tCOMPAT_OBJS += compat/mingw/compat-util.o compat/winansi.o \\\n \t\tcompat/win32/trace2_win32_process_info.o \\\n \t\tcompat/win32/flush.o \\\n \t\tcompat/win32/path-utils.o \\\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 10dc54fdcb..65ce8705f0 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -258,7 +258,7 @@ if(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\t\t\tUSE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP\n \t\t\t\tHAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET HAVE_RTLGENRANDOM)\n \tlist(APPEND compat_SOURCES\n-\t\tcompat/mingw.c\n+\t\tcompat/mingw/compat-util.c\n \t\tcompat/winansi.c\n \t\tcompat/win32/flush.c\n \t\tcompat/win32/path-utils.c\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex e283c46c6f..b96fb98e1e 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -304,10 +304,10 @@ static inline int is_xplatform_dir_sep(int c)\n #if defined(__MINGW32__)\n /* pull in Windows compatibility stuff */\n #include \"compat/win32/path-utils.h\"\n-#include \"compat/mingw.h\"\n+#include \"compat/mingw/compat-util.h\"\n #elif defined(_MSC_VER)\n #include \"compat/win32/path-utils.h\"\n-#include \"compat/msvc.h\"\n+#include \"compat/msvc/compat-util.h\"\n #else\n #include \n #include \ndiff --git a/meson.build b/meson.build\nindex 2297f5e9dd..e8abf798f5 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -1054,7 +1054,6 @@ if host_machine.system() == 'cygwin'\n ]\n elif host_machine.system() == 'windows'\n libgit_sources += [\n- 'compat/mingw.c',\n 'compat/winansi.c',\n 'compat/win32/flush.c',\n 'compat/win32/path-utils.c',\n@@ -1081,6 +1080,9 @@ elif host_machine.system() == 'windows'\n libgit_include_directories += 'compat/win32'\n if compiler.get_id() == 'msvc'\n libgit_include_directories += 'compat/vcbuild/include'\n+ libgit_sources += 'compat/msvc/compat-util.c'\n+ else\n+ libgit_sources += 'compat/mingw/compat-util.c'\n endif\n endif\n \n@@ -1157,7 +1159,7 @@ else\n error('Native regex support requested but not found')\n endif\n \n-# setitimer and friends are provided by compat/mingw.c.\n+# setitimer and friends are provided by compat/mingw/compat-util.c.\n if host_machine.system() != 'windows'\n if not compiler.compiles('''\n #include \n@@ -1237,7 +1239,7 @@ if not compiler.has_function('qsort')\n endif\n libgit_sources += 'compat/qsort_s.c'\n \n-# unsetenv is provided by compat/mingw.c.\n+# unsetenv is provided by compat/mingw/compat-util.c.\n if host_machine.system() != 'windows' and not compiler.has_function('unsetenv')\n libgit_c_args += '-DNO_UNSETENV'\n libgit_sources += 'compat/unsetenv.c'\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"511962","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-16-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","subject":"[PATCH v4 16/18] git-compat-util.h: split out POSIX-emulating bits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:18Z","receivedAt":"2025-02-06T07:52:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"git-compat-util.h\" header is a treasure trove of various bits and\npieces used throughout the project. It basically mixes two different\nthings into one:\n\n - Providing a POSIX-like interface even on platforms that aren't\n POSIX-compliant.\n\n - Providing low-level functionality that is specific to Git.\n\nThis intermixing is a bit of a problem for the reftable library as we\ndon't want to recreate the POSIX-like interface there. But neither do we\nwant to pull in the Git-specific functionality, as it is otherwise quite\neasy to start depending on the Git codebase again.\n\nSplit out a new header \"compat/posix.h\" that only contains the bits and\npieces relevant for the emulation of POSIX, which we will start using in\nthe next commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n compat/posix.h | 541 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n git-compat-util.h | 531 +----------------------------------------------------\n 2 files changed, 543 insertions(+), 529 deletions(-)\n\ndiff --git a/compat/posix.h b/compat/posix.h\nnew file mode 100644\nindex 0000000000..7e6c4c1a1f\n--- /dev/null\n+++ b/compat/posix.h\n@@ -0,0 +1,541 @@\n+#ifndef COMPAT_POSIX_H\n+#define COMPAT_POSIX_H\n+\n+#define _FILE_OFFSET_BITS 64\n+\n+/*\n+ * Derived from Linux \"Features Test Macro\" header\n+ * Convenience macros to test the versions of gcc (or\n+ * a compatible compiler).\n+ * Use them like this:\n+ * #if GIT_GNUC_PREREQ (2,8)\n+ * ... code requiring gcc 2.8 or later ...\n+ * #endif\n+ *\n+ * This macro of course is not part of POSIX, but we need it for the UNUSED\n+ * macro which is used by some of our POSIX compatibility wrappers.\n+*/\n+#if defined(__GNUC__) && defined(__GNUC_MINOR__)\n+# define GIT_GNUC_PREREQ(maj, min) \\\n+\t((__GNUC__ << 16) + __GNUC_MINOR__ >= ((maj) << 16) + (min))\n+#else\n+ #define GIT_GNUC_PREREQ(maj, min) 0\n+#endif\n+\n+/*\n+ * UNUSED marks a function parameter that is always unused. It also\n+ * can be used to annotate a function, a variable, or a type that is\n+ * always unused.\n+ *\n+ * A callback interface may dictate that a function accepts a\n+ * parameter at that position, but the implementation of the function\n+ * may not need to use the parameter. In such a case, mark the parameter\n+ * with UNUSED.\n+ *\n+ * When a parameter may be used or unused, depending on conditional\n+ * compilation, consider using MAYBE_UNUSED instead.\n+ */\n+#if GIT_GNUC_PREREQ(4, 5)\n+#define UNUSED __attribute__((unused)) \\\n+\t__attribute__((deprecated (\"parameter declared as UNUSED\")))\n+#elif defined(__GNUC__)\n+#define UNUSED __attribute__((unused)) \\\n+\t__attribute__((deprecated))\n+#else\n+#define UNUSED\n+#endif\n+\n+#ifdef __MINGW64__\n+#define _POSIX_C_SOURCE 1\n+#elif defined(__sun__)\n+ /*\n+ * On Solaris, when _XOPEN_EXTENDED is set, its header file\n+ * forces the programs to be XPG4v2, defeating any _XOPEN_SOURCE\n+ * setting to say we are XPG5 or XPG6. Also on Solaris,\n+ * XPG6 programs must be compiled with a c99 compiler, while\n+ * non XPG6 programs must be compiled with a pre-c99 compiler.\n+ */\n+# if __STDC_VERSION__ - 0 >= 199901L\n+# define _XOPEN_SOURCE 600\n+# else\n+# define _XOPEN_SOURCE 500\n+# endif\n+#elif !defined(__APPLE__) && !defined(__FreeBSD__) && !defined(__USLC__) && \\\n+ !defined(_M_UNIX) && !defined(__sgi) && !defined(__DragonFly__) && \\\n+ !defined(__TANDEM) && !defined(__QNX__) && !defined(__MirBSD__) && \\\n+ !defined(__CYGWIN__)\n+#define _XOPEN_SOURCE 600 /* glibc2 and AIX 5.3L need 500, OpenBSD needs 600 for S_ISLNK() */\n+#define _XOPEN_SOURCE_EXTENDED 1 /* AIX 5.3L needs this */\n+#endif\n+#define _ALL_SOURCE 1\n+#define _GNU_SOURCE 1\n+#define _BSD_SOURCE 1\n+#define _DEFAULT_SOURCE 1\n+#define _NETBSD_SOURCE 1\n+#define _SGI_SOURCE 1\n+\n+#if defined(WIN32) && !defined(__CYGWIN__) /* Both MinGW and MSVC */\n+# if !defined(_WIN32_WINNT)\n+# define _WIN32_WINNT 0x0600\n+# endif\n+#define WIN32_LEAN_AND_MEAN /* stops windows.h including winsock.h */\n+#include \n+#ifndef NO_UNIX_SOCKETS\n+#include \n+#endif\n+#include \n+#define GIT_WINDOWS_NATIVE\n+#endif\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#ifdef HAVE_STRINGS_H\n+#include /* for strcasecmp() */\n+#endif\n+#include \n+#include \n+#include \n+#ifdef NEEDS_SYS_PARAM_H\n+#include \n+#endif\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#if !defined(NO_POLL_H)\n+#include \n+#elif !defined(NO_SYS_POLL_H)\n+#include \n+#else\n+/* Pull the compat stuff */\n+#include \n+#endif\n+#ifdef HAVE_BSD_SYSCTL\n+#include \n+#endif\n+\n+#if defined(__MINGW32__)\n+#include \"mingw/posix.h\"\n+#elif defined(_MSC_VER)\n+#include \"msvc/posix.h\"\n+#else\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#ifndef NO_SYS_SELECT_H\n+#include \n+#endif\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#ifndef NO_INTTYPES_H\n+#include \n+#else\n+#include \n+#endif\n+#ifdef HAVE_ARC4RANDOM_LIBBSD\n+#include \n+#endif\n+#ifdef HAVE_GETRANDOM\n+#include \n+#endif\n+#ifdef NO_INTPTR_T\n+/*\n+ * On I16LP32, ILP32 and LP64 \"long\" is the safe bet, however\n+ * on LLP86, IL33LLP64 and P64 it needs to be \"long long\",\n+ * while on IP16 and IP16L32 it is \"int\" (resp. \"short\")\n+ * Size needs to match (or exceed) 'sizeof(void *)'.\n+ * We can't take \"long long\" here as not everybody has it.\n+ */\n+typedef long intptr_t;\n+typedef unsigned long uintptr_t;\n+#endif\n+#undef _ALL_SOURCE /* AIX 5.3L defines a struct list with _ALL_SOURCE. */\n+#include \n+#define _ALL_SOURCE 1\n+#endif\n+\n+#ifdef MKDIR_WO_TRAILING_SLASH\n+#define mkdir(a,b) compat_mkdir_wo_trailing_slash((a),(b))\n+int compat_mkdir_wo_trailing_slash(const char*, mode_t);\n+#endif\n+\n+#ifdef time\n+#undef time\n+#endif\n+static inline time_t git_time(time_t *tloc)\n+{\n+\tstruct timeval tv;\n+\n+\t/*\n+\t * Avoid time(NULL), which can disagree with gettimeofday(2)\n+\t * and filesystem timestamps.\n+\t */\n+\tgettimeofday(&tv, NULL);\n+\n+\tif (tloc)\n+\t\t*tloc = tv.tv_sec;\n+\treturn tv.tv_sec;\n+}\n+#define time git_time\n+\n+#ifdef NO_STRUCT_ITIMERVAL\n+struct itimerval {\n+\tstruct timeval it_interval;\n+\tstruct timeval it_value;\n+};\n+#endif\n+\n+#ifdef NO_SETITIMER\n+static inline int git_setitimer(int which UNUSED,\n+\t\t\t\tconst struct itimerval *value UNUSED,\n+\t\t\t\tstruct itimerval *newvalue UNUSED) {\n+\treturn 0; /* pretend success */\n+}\n+#undef setitimer\n+#define setitimer(which,value,ovalue) git_setitimer(which,value,ovalue)\n+#endif\n+\n+#ifndef NO_LIBGEN_H\n+#include \n+#else\n+#define basename gitbasename\n+char *gitbasename(char *);\n+#define dirname gitdirname\n+char *gitdirname(char *);\n+#endif\n+\n+#ifndef NO_ICONV\n+#include \n+#endif\n+\n+/* On most systems would have given us this, but\n+ * not on some systems (e.g. z/OS).\n+ */\n+#ifndef NI_MAXHOST\n+#define NI_MAXHOST 1025\n+#endif\n+\n+#ifndef NI_MAXSERV\n+#define NI_MAXSERV 32\n+#endif\n+\n+/* On most systems would have given us this, but\n+ * not on some systems (e.g. GNU/Hurd).\n+ */\n+#ifndef PATH_MAX\n+#define PATH_MAX 4096\n+#endif\n+\n+#ifndef NAME_MAX\n+#define NAME_MAX 255\n+#endif\n+\n+typedef uintmax_t timestamp_t;\n+#define PRItime PRIuMAX\n+#define parse_timestamp strtoumax\n+#define TIME_MAX UINTMAX_MAX\n+#define TIME_MIN 0\n+\n+int lstat_cache_aware_rmdir(const char *path);\n+#if !defined(__MINGW32__) && !defined(_MSC_VER)\n+#define rmdir lstat_cache_aware_rmdir\n+#endif\n+\n+#if defined(NO_MMAP) || defined(USE_WIN32_MMAP)\n+\n+#ifndef PROT_READ\n+#define PROT_READ 1\n+#define PROT_WRITE 2\n+#define MAP_PRIVATE 1\n+#endif\n+\n+#define mmap git_mmap\n+#define munmap git_munmap\n+void *git_mmap(void *start, size_t length, int prot, int flags, int fd, off_t offset);\n+int git_munmap(void *start, size_t length);\n+\n+#else /* NO_MMAP || USE_WIN32_MMAP */\n+\n+#include \n+\n+#endif /* NO_MMAP || USE_WIN32_MMAP */\n+\n+#ifndef MAP_FAILED\n+#define MAP_FAILED ((void *)-1)\n+#endif\n+\n+#ifdef NEEDS_MODE_TRANSLATION\n+#undef S_IFMT\n+#undef S_IFREG\n+#undef S_IFDIR\n+#undef S_IFLNK\n+#undef S_IFBLK\n+#undef S_IFCHR\n+#undef S_IFIFO\n+#undef S_IFSOCK\n+#define S_IFMT 0170000\n+#define S_IFREG 0100000\n+#define S_IFDIR 0040000\n+#define S_IFLNK 0120000\n+#define S_IFBLK 0060000\n+#define S_IFCHR 0020000\n+#define S_IFIFO 0010000\n+#define S_IFSOCK 0140000\n+#ifdef stat\n+#undef stat\n+#endif\n+#define stat(path, buf) git_stat(path, buf)\n+int git_stat(const char *, struct stat *);\n+#ifdef fstat\n+#undef fstat\n+#endif\n+#define fstat(fd, buf) git_fstat(fd, buf)\n+int git_fstat(int, struct stat *);\n+#ifdef lstat\n+#undef lstat\n+#endif\n+#define lstat(path, buf) git_lstat(path, buf)\n+int git_lstat(const char *, struct stat *);\n+#endif\n+\n+#ifdef NO_PREAD\n+#define pread git_pread\n+ssize_t git_pread(int fd, void *buf, size_t count, off_t offset);\n+#endif\n+\n+#ifdef NO_SETENV\n+#define setenv gitsetenv\n+int gitsetenv(const char *, const char *, int);\n+#endif\n+\n+#ifdef NO_MKDTEMP\n+#define mkdtemp gitmkdtemp\n+char *gitmkdtemp(char *);\n+#endif\n+\n+#ifdef NO_UNSETENV\n+#define unsetenv gitunsetenv\n+int gitunsetenv(const char *);\n+#endif\n+\n+#ifdef NO_STRCASESTR\n+#define strcasestr gitstrcasestr\n+char *gitstrcasestr(const char *haystack, const char *needle);\n+#endif\n+\n+#ifdef NO_STRLCPY\n+#define strlcpy gitstrlcpy\n+size_t gitstrlcpy(char *, const char *, size_t);\n+#endif\n+\n+#ifdef NO_STRTOUMAX\n+#define strtoumax gitstrtoumax\n+uintmax_t gitstrtoumax(const char *, char **, int);\n+#define strtoimax gitstrtoimax\n+intmax_t gitstrtoimax(const char *, char **, int);\n+#endif\n+\n+#ifdef NO_HSTRERROR\n+#define hstrerror githstrerror\n+const char *githstrerror(int herror);\n+#endif\n+\n+#ifdef NO_MEMMEM\n+#define memmem gitmemmem\n+void *gitmemmem(const void *haystack, size_t haystacklen,\n+\t\tconst void *needle, size_t needlelen);\n+#endif\n+\n+#ifdef OVERRIDE_STRDUP\n+#ifdef strdup\n+#undef strdup\n+#endif\n+#define strdup gitstrdup\n+char *gitstrdup(const char *s);\n+#endif\n+\n+#ifdef NO_GETPAGESIZE\n+#define getpagesize() sysconf(_SC_PAGESIZE)\n+#endif\n+\n+#ifndef O_CLOEXEC\n+#define O_CLOEXEC 0\n+#endif\n+\n+#ifdef FREAD_READS_DIRECTORIES\n+# if !defined(SUPPRESS_FOPEN_REDEFINITION)\n+# ifdef fopen\n+# undef fopen\n+# endif\n+# define fopen(a,b) git_fopen(a,b)\n+# endif\n+FILE *git_fopen(const char*, const char*);\n+#endif\n+\n+#ifdef SNPRINTF_RETURNS_BOGUS\n+#ifdef snprintf\n+#undef snprintf\n+#endif\n+#define snprintf git_snprintf\n+int git_snprintf(char *str, size_t maxsize,\n+\t\t const char *format, ...);\n+#ifdef vsnprintf\n+#undef vsnprintf\n+#endif\n+#define vsnprintf git_vsnprintf\n+int git_vsnprintf(char *str, size_t maxsize,\n+\t\t const char *format, va_list ap);\n+#endif\n+\n+#ifdef OPEN_RETURNS_EINTR\n+#undef open\n+#define open git_open_with_retry\n+int git_open_with_retry(const char *path, int flag, ...);\n+#endif\n+\n+#ifdef __GLIBC_PREREQ\n+#if __GLIBC_PREREQ(2, 1)\n+#define HAVE_STRCHRNUL\n+#endif\n+#endif\n+\n+#ifndef HAVE_STRCHRNUL\n+#define strchrnul gitstrchrnul\n+static inline char *gitstrchrnul(const char *s, int c)\n+{\n+\twhile (*s && *s != c)\n+\t\ts++;\n+\treturn (char *)s;\n+}\n+#endif\n+\n+#ifdef NO_INET_PTON\n+int inet_pton(int af, const char *src, void *dst);\n+#endif\n+\n+#ifdef NO_INET_NTOP\n+const char *inet_ntop(int af, const void *src, char *dst, size_t size);\n+#endif\n+\n+#ifdef NO_PTHREADS\n+#define atexit git_atexit\n+int git_atexit(void (*handler)(void));\n+#endif\n+\n+#ifndef HOST_NAME_MAX\n+#define HOST_NAME_MAX 256\n+#endif\n+\n+#include \"../sane-ctype.h\"\n+\n+void git_stable_qsort(void *base, size_t nmemb, size_t size,\n+\t\t int(*compar)(const void *, const void *));\n+#ifdef INTERNAL_QSORT\n+#define qsort git_stable_qsort\n+#endif\n+\n+#define QSORT(base, n, compar) sane_qsort((base), (n), sizeof(*(base)), compar)\n+static inline void sane_qsort(void *base, size_t nmemb, size_t size,\n+\t\t\t int(*compar)(const void *, const void *))\n+{\n+\tif (nmemb > 1)\n+\t\tqsort(base, nmemb, size, compar);\n+}\n+\n+#define STABLE_QSORT(base, n, compar) \\\n+\tgit_stable_qsort((base), (n), sizeof(*(base)), compar)\n+\n+#ifndef HAVE_ISO_QSORT_S\n+int git_qsort_s(void *base, size_t nmemb, size_t size,\n+\t\tint (*compar)(const void *, const void *, void *), void *ctx);\n+#define qsort_s git_qsort_s\n+#endif\n+\n+#define QSORT_S(base, n, compar, ctx) do {\t\t\t\\\n+\tif (qsort_s((base), (n), sizeof(*(base)), compar, ctx))\t\\\n+\t\tBUG(\"qsort_s() failed\");\t\t\t\\\n+} while (0)\n+\n+#ifdef NO_NSEC\n+#undef USE_NSEC\n+#define ST_CTIME_NSEC(st) 0\n+#define ST_MTIME_NSEC(st) 0\n+#else\n+#ifdef USE_ST_TIMESPEC\n+#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctimespec.tv_nsec))\n+#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtimespec.tv_nsec))\n+#else\n+#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctim.tv_nsec))\n+#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtim.tv_nsec))\n+#endif\n+#endif\n+\n+#ifndef va_copy\n+/*\n+ * Since an obvious implementation of va_list would be to make it a\n+ * pointer into the stack frame, a simple assignment will work on\n+ * many systems. But let's try to be more portable.\n+ */\n+#ifdef __va_copy\n+#define va_copy(dst, src) __va_copy(dst, src)\n+#else\n+#define va_copy(dst, src) ((dst) = (src))\n+#endif\n+#endif\n+\n+#ifndef _POSIX_THREAD_SAFE_FUNCTIONS\n+static inline void git_flockfile(FILE *fh UNUSED)\n+{\n+\t; /* nothing */\n+}\n+static inline void git_funlockfile(FILE *fh UNUSED)\n+{\n+\t; /* nothing */\n+}\n+#undef flockfile\n+#undef funlockfile\n+#undef getc_unlocked\n+#define flockfile(fh) git_flockfile(fh)\n+#define funlockfile(fh) git_funlockfile(fh)\n+#define getc_unlocked(fh) getc(fh)\n+#endif\n+\n+#ifdef FILENO_IS_A_MACRO\n+int git_fileno(FILE *stream);\n+# ifndef COMPAT_CODE_FILENO\n+# undef fileno\n+# define fileno(p) git_fileno(p)\n+# endif\n+#endif\n+\n+#ifdef NEED_ACCESS_ROOT_HANDLER\n+int git_access(const char *path, int mode);\n+# ifndef COMPAT_CODE_ACCESS\n+# ifdef access\n+# undef access\n+# endif\n+# define access(path, mode) git_access(path, mode)\n+# endif\n+#endif\n+\n+#endif /* COMPAT_POSIX_H */\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex b96fb98e1e..30ad0725fb 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -23,26 +23,9 @@\n #include \n #endif\n \n-struct strbuf;\n-\n-\n-#define _FILE_OFFSET_BITS 64\n+#include \"compat/posix.h\"\n \n-\n-/* Derived from Linux \"Features Test Macro\" header\n- * Convenience macros to test the versions of gcc (or\n- * a compatible compiler).\n- * Use them like this:\n- * #if GIT_GNUC_PREREQ (2,8)\n- * ... code requiring gcc 2.8 or later ...\n- * #endif\n-*/\n-#if defined(__GNUC__) && defined(__GNUC_MINOR__)\n-# define GIT_GNUC_PREREQ(maj, min) \\\n-\t((__GNUC__ << 16) + __GNUC_MINOR__ >= ((maj) << 16) + (min))\n-#else\n- #define GIT_GNUC_PREREQ(maj, min) 0\n-#endif\n+struct strbuf;\n \n #if defined(__GNUC__) || defined(__clang__)\n # define PRAGMA(pragma) _Pragma(#pragma)\n@@ -176,71 +159,6 @@ DISABLE_WARNING(-Wsign-compare)\n /* Approximation of the length of the decimal representation of this type. */\n #define decimal_length(x)\t((int)(sizeof(x) * 2.56 + 0.5) + 1)\n \n-#ifdef __MINGW64__\n-#define _POSIX_C_SOURCE 1\n-#elif defined(__sun__)\n- /*\n- * On Solaris, when _XOPEN_EXTENDED is set, its header file\n- * forces the programs to be XPG4v2, defeating any _XOPEN_SOURCE\n- * setting to say we are XPG5 or XPG6. Also on Solaris,\n- * XPG6 programs must be compiled with a c99 compiler, while\n- * non XPG6 programs must be compiled with a pre-c99 compiler.\n- */\n-# if __STDC_VERSION__ - 0 >= 199901L\n-# define _XOPEN_SOURCE 600\n-# else\n-# define _XOPEN_SOURCE 500\n-# endif\n-#elif !defined(__APPLE__) && !defined(__FreeBSD__) && !defined(__USLC__) && \\\n- !defined(_M_UNIX) && !defined(__sgi) && !defined(__DragonFly__) && \\\n- !defined(__TANDEM) && !defined(__QNX__) && !defined(__MirBSD__) && \\\n- !defined(__CYGWIN__)\n-#define _XOPEN_SOURCE 600 /* glibc2 and AIX 5.3L need 500, OpenBSD needs 600 for S_ISLNK() */\n-#define _XOPEN_SOURCE_EXTENDED 1 /* AIX 5.3L needs this */\n-#endif\n-#define _ALL_SOURCE 1\n-#define _GNU_SOURCE 1\n-#define _BSD_SOURCE 1\n-#define _DEFAULT_SOURCE 1\n-#define _NETBSD_SOURCE 1\n-#define _SGI_SOURCE 1\n-\n-/*\n- * UNUSED marks a function parameter that is always unused. It also\n- * can be used to annotate a function, a variable, or a type that is\n- * always unused.\n- *\n- * A callback interface may dictate that a function accepts a\n- * parameter at that position, but the implementation of the function\n- * may not need to use the parameter. In such a case, mark the parameter\n- * with UNUSED.\n- *\n- * When a parameter may be used or unused, depending on conditional\n- * compilation, consider using MAYBE_UNUSED instead.\n- */\n-#if GIT_GNUC_PREREQ(4, 5)\n-#define UNUSED __attribute__((unused)) \\\n-\t__attribute__((deprecated (\"parameter declared as UNUSED\")))\n-#elif defined(__GNUC__)\n-#define UNUSED __attribute__((unused)) \\\n-\t__attribute__((deprecated))\n-#else\n-#define UNUSED\n-#endif\n-\n-#if defined(WIN32) && !defined(__CYGWIN__) /* Both MinGW and MSVC */\n-# if !defined(_WIN32_WINNT)\n-# define _WIN32_WINNT 0x0600\n-# endif\n-#define WIN32_LEAN_AND_MEAN /* stops windows.h including winsock.h */\n-#include \n-#ifndef NO_UNIX_SOCKETS\n-#include \n-#endif\n-#include \n-#define GIT_WINDOWS_NATIVE\n-#endif\n-\n #if defined(NO_UNIX_SOCKETS) || !defined(GIT_WINDOWS_NATIVE)\n static inline int _have_unix_sockets(void)\n {\n@@ -253,45 +171,6 @@ static inline int _have_unix_sockets(void)\n #define have_unix_sockets _have_unix_sockets\n #endif\n \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#ifdef HAVE_STRINGS_H\n-#include /* for strcasecmp() */\n-#endif\n-#include \n-#include \n-#include \n-#ifdef NEEDS_SYS_PARAM_H\n-#include \n-#endif\n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#if !defined(NO_POLL_H)\n-#include \n-#elif !defined(NO_SYS_POLL_H)\n-#include \n-#else\n-/* Pull the compat stuff */\n-#include \n-#endif\n-#ifdef HAVE_BSD_SYSCTL\n-#include \n-#endif\n-\n /* Used by compat/win32/path-utils.h, and more */\n static inline int is_xplatform_dir_sep(int c)\n {\n@@ -308,48 +187,6 @@ static inline int is_xplatform_dir_sep(int c)\n #elif defined(_MSC_VER)\n #include \"compat/win32/path-utils.h\"\n #include \"compat/msvc/compat-util.h\"\n-#else\n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#ifndef NO_SYS_SELECT_H\n-#include \n-#endif\n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#ifndef NO_INTTYPES_H\n-#include \n-#else\n-#include \n-#endif\n-#ifdef HAVE_ARC4RANDOM_LIBBSD\n-#include \n-#endif\n-#ifdef HAVE_GETRANDOM\n-#include \n-#endif\n-#ifdef NO_INTPTR_T\n-/*\n- * On I16LP32, ILP32 and LP64 \"long\" is the safe bet, however\n- * on LLP86, IL33LLP64 and P64 it needs to be \"long long\",\n- * while on IP16 and IP16L32 it is \"int\" (resp. \"short\")\n- * Size needs to match (or exceed) 'sizeof(void *)'.\n- * We can't take \"long long\" here as not everybody has it.\n- */\n-typedef long intptr_t;\n-typedef unsigned long uintptr_t;\n-#endif\n-#undef _ALL_SOURCE /* AIX 5.3L defines a struct list with _ALL_SOURCE. */\n-#include \n-#define _ALL_SOURCE 1\n #endif\n \n /* used on Mac OS X */\n@@ -370,60 +207,6 @@ static inline const char *precompose_string_if_needed(const char *in)\n #define probe_utf8_pathname_composition()\n #endif\n \n-#ifdef MKDIR_WO_TRAILING_SLASH\n-#define mkdir(a,b) compat_mkdir_wo_trailing_slash((a),(b))\n-int compat_mkdir_wo_trailing_slash(const char*, mode_t);\n-#endif\n-\n-#ifdef time\n-#undef time\n-#endif\n-static inline time_t git_time(time_t *tloc)\n-{\n-\tstruct timeval tv;\n-\n-\t/*\n-\t * Avoid time(NULL), which can disagree with gettimeofday(2)\n-\t * and filesystem timestamps.\n-\t */\n-\tgettimeofday(&tv, NULL);\n-\n-\tif (tloc)\n-\t\t*tloc = tv.tv_sec;\n-\treturn tv.tv_sec;\n-}\n-#define time git_time\n-\n-#ifdef NO_STRUCT_ITIMERVAL\n-struct itimerval {\n-\tstruct timeval it_interval;\n-\tstruct timeval it_value;\n-};\n-#endif\n-\n-#ifdef NO_SETITIMER\n-static inline int git_setitimer(int which UNUSED,\n-\t\t\t\tconst struct itimerval *value UNUSED,\n-\t\t\t\tstruct itimerval *newvalue UNUSED) {\n-\treturn 0; /* pretend success */\n-}\n-#undef setitimer\n-#define setitimer(which,value,ovalue) git_setitimer(which,value,ovalue)\n-#endif\n-\n-#ifndef NO_LIBGEN_H\n-#include \n-#else\n-#define basename gitbasename\n-char *gitbasename(char *);\n-#define dirname gitdirname\n-char *gitdirname(char *);\n-#endif\n-\n-#ifndef NO_ICONV\n-#include \n-#endif\n-\n #ifndef NO_OPENSSL\n #ifdef __APPLE__\n #undef __AVAILABILITY_MACROS_USES_AVAILABILITY\n@@ -441,34 +224,6 @@ char *gitdirname(char *);\n # include \n #endif\n \n-/* On most systems would have given us this, but\n- * not on some systems (e.g. z/OS).\n- */\n-#ifndef NI_MAXHOST\n-#define NI_MAXHOST 1025\n-#endif\n-\n-#ifndef NI_MAXSERV\n-#define NI_MAXSERV 32\n-#endif\n-\n-/* On most systems would have given us this, but\n- * not on some systems (e.g. GNU/Hurd).\n- */\n-#ifndef PATH_MAX\n-#define PATH_MAX 4096\n-#endif\n-\n-#ifndef NAME_MAX\n-#define NAME_MAX 255\n-#endif\n-\n-typedef uintmax_t timestamp_t;\n-#define PRItime PRIuMAX\n-#define parse_timestamp strtoumax\n-#define TIME_MAX UINTMAX_MAX\n-#define TIME_MIN 0\n-\n #ifndef PATH_SEP\n #define PATH_SEP ':'\n #endif\n@@ -492,11 +247,6 @@ static inline int noop_core_config(const char *var UNUSED,\n #define platform_core_config noop_core_config\n #endif\n \n-int lstat_cache_aware_rmdir(const char *path);\n-#if !defined(__MINGW32__) && !defined(_MSC_VER)\n-#define rmdir lstat_cache_aware_rmdir\n-#endif\n-\n #ifndef has_dos_drive_prefix\n static inline int git_has_dos_drive_prefix(const char *path UNUSED)\n {\n@@ -822,25 +572,6 @@ static inline bool strip_suffix(const char *str, const char *suffix,\n \tmemcpy(_swap_b_ptr, _swap_buffer, sizeof(a));\t\t\\\n } while (0)\n \n-#if defined(NO_MMAP) || defined(USE_WIN32_MMAP)\n-\n-#ifndef PROT_READ\n-#define PROT_READ 1\n-#define PROT_WRITE 2\n-#define MAP_PRIVATE 1\n-#endif\n-\n-#define mmap git_mmap\n-#define munmap git_munmap\n-void *git_mmap(void *start, size_t length, int prot, int flags, int fd, off_t offset);\n-int git_munmap(void *start, size_t length);\n-\n-#else /* NO_MMAP || USE_WIN32_MMAP */\n-\n-#include \n-\n-#endif /* NO_MMAP || USE_WIN32_MMAP */\n-\n #ifdef NO_MMAP\n \n /* This value must be multiple of (pagesize * 2) */\n@@ -856,177 +587,15 @@ int git_munmap(void *start, size_t length);\n \n #endif /* NO_MMAP */\n \n-#ifndef MAP_FAILED\n-#define MAP_FAILED ((void *)-1)\n-#endif\n-\n #ifdef NO_ST_BLOCKS_IN_STRUCT_STAT\n #define on_disk_bytes(st) ((st).st_size)\n #else\n #define on_disk_bytes(st) ((st).st_blocks * 512)\n #endif\n \n-#ifdef NEEDS_MODE_TRANSLATION\n-#undef S_IFMT\n-#undef S_IFREG\n-#undef S_IFDIR\n-#undef S_IFLNK\n-#undef S_IFBLK\n-#undef S_IFCHR\n-#undef S_IFIFO\n-#undef S_IFSOCK\n-#define S_IFMT 0170000\n-#define S_IFREG 0100000\n-#define S_IFDIR 0040000\n-#define S_IFLNK 0120000\n-#define S_IFBLK 0060000\n-#define S_IFCHR 0020000\n-#define S_IFIFO 0010000\n-#define S_IFSOCK 0140000\n-#ifdef stat\n-#undef stat\n-#endif\n-#define stat(path, buf) git_stat(path, buf)\n-int git_stat(const char *, struct stat *);\n-#ifdef fstat\n-#undef fstat\n-#endif\n-#define fstat(fd, buf) git_fstat(fd, buf)\n-int git_fstat(int, struct stat *);\n-#ifdef lstat\n-#undef lstat\n-#endif\n-#define lstat(path, buf) git_lstat(path, buf)\n-int git_lstat(const char *, struct stat *);\n-#endif\n-\n #define DEFAULT_PACKED_GIT_LIMIT \\\n \t((1024L * 1024L) * (size_t)(sizeof(void*) >= 8 ? (32 * 1024L * 1024L) : 256))\n \n-#ifdef NO_PREAD\n-#define pread git_pread\n-ssize_t git_pread(int fd, void *buf, size_t count, off_t offset);\n-#endif\n-\n-#ifdef NO_SETENV\n-#define setenv gitsetenv\n-int gitsetenv(const char *, const char *, int);\n-#endif\n-\n-#ifdef NO_MKDTEMP\n-#define mkdtemp gitmkdtemp\n-char *gitmkdtemp(char *);\n-#endif\n-\n-#ifdef NO_UNSETENV\n-#define unsetenv gitunsetenv\n-int gitunsetenv(const char *);\n-#endif\n-\n-#ifdef NO_STRCASESTR\n-#define strcasestr gitstrcasestr\n-char *gitstrcasestr(const char *haystack, const char *needle);\n-#endif\n-\n-#ifdef NO_STRLCPY\n-#define strlcpy gitstrlcpy\n-size_t gitstrlcpy(char *, const char *, size_t);\n-#endif\n-\n-#ifdef NO_STRTOUMAX\n-#define strtoumax gitstrtoumax\n-uintmax_t gitstrtoumax(const char *, char **, int);\n-#define strtoimax gitstrtoimax\n-intmax_t gitstrtoimax(const char *, char **, int);\n-#endif\n-\n-#ifdef NO_HSTRERROR\n-#define hstrerror githstrerror\n-const char *githstrerror(int herror);\n-#endif\n-\n-#ifdef NO_MEMMEM\n-#define memmem gitmemmem\n-void *gitmemmem(const void *haystack, size_t haystacklen,\n-\t\tconst void *needle, size_t needlelen);\n-#endif\n-\n-#ifdef OVERRIDE_STRDUP\n-#ifdef strdup\n-#undef strdup\n-#endif\n-#define strdup gitstrdup\n-char *gitstrdup(const char *s);\n-#endif\n-\n-#ifdef NO_GETPAGESIZE\n-#define getpagesize() sysconf(_SC_PAGESIZE)\n-#endif\n-\n-#ifndef O_CLOEXEC\n-#define O_CLOEXEC 0\n-#endif\n-\n-#ifdef FREAD_READS_DIRECTORIES\n-# if !defined(SUPPRESS_FOPEN_REDEFINITION)\n-# ifdef fopen\n-# undef fopen\n-# endif\n-# define fopen(a,b) git_fopen(a,b)\n-# endif\n-FILE *git_fopen(const char*, const char*);\n-#endif\n-\n-#ifdef SNPRINTF_RETURNS_BOGUS\n-#ifdef snprintf\n-#undef snprintf\n-#endif\n-#define snprintf git_snprintf\n-int git_snprintf(char *str, size_t maxsize,\n-\t\t const char *format, ...);\n-#ifdef vsnprintf\n-#undef vsnprintf\n-#endif\n-#define vsnprintf git_vsnprintf\n-int git_vsnprintf(char *str, size_t maxsize,\n-\t\t const char *format, va_list ap);\n-#endif\n-\n-#ifdef OPEN_RETURNS_EINTR\n-#undef open\n-#define open git_open_with_retry\n-int git_open_with_retry(const char *path, int flag, ...);\n-#endif\n-\n-#ifdef __GLIBC_PREREQ\n-#if __GLIBC_PREREQ(2, 1)\n-#define HAVE_STRCHRNUL\n-#endif\n-#endif\n-\n-#ifndef HAVE_STRCHRNUL\n-#define strchrnul gitstrchrnul\n-static inline char *gitstrchrnul(const char *s, int c)\n-{\n-\twhile (*s && *s != c)\n-\t\ts++;\n-\treturn (char *)s;\n-}\n-#endif\n-\n-#ifdef NO_INET_PTON\n-int inet_pton(int af, const char *src, void *dst);\n-#endif\n-\n-#ifdef NO_INET_NTOP\n-const char *inet_ntop(int af, const void *src, char *dst, size_t size);\n-#endif\n-\n-#ifdef NO_PTHREADS\n-#define atexit git_atexit\n-int git_atexit(void (*handler)(void));\n-#endif\n-\n static inline size_t st_add(size_t a, size_t b)\n {\n \tif (unsigned_add_overflows(a, b))\n@@ -1293,12 +862,6 @@ static inline size_t xsize_t(off_t len)\n \treturn (size_t) len;\n }\n \n-#ifndef HOST_NAME_MAX\n-#define HOST_NAME_MAX 256\n-#endif\n-\n-#include \"sane-ctype.h\"\n-\n /*\n * Like skip_prefix, but compare case-insensitively. Note that the comparison\n * is done via tolower(), so it is strictly ASCII (no multi-byte characters or\n@@ -1364,34 +927,6 @@ static inline int strtol_i(char const *s, int base, int *result)\n \treturn 0;\n }\n \n-void git_stable_qsort(void *base, size_t nmemb, size_t size,\n-\t\t int(*compar)(const void *, const void *));\n-#ifdef INTERNAL_QSORT\n-#define qsort git_stable_qsort\n-#endif\n-\n-#define QSORT(base, n, compar) sane_qsort((base), (n), sizeof(*(base)), compar)\n-static inline void sane_qsort(void *base, size_t nmemb, size_t size,\n-\t\t\t int(*compar)(const void *, const void *))\n-{\n-\tif (nmemb > 1)\n-\t\tqsort(base, nmemb, size, compar);\n-}\n-\n-#define STABLE_QSORT(base, n, compar) \\\n-\tgit_stable_qsort((base), (n), sizeof(*(base)), compar)\n-\n-#ifndef HAVE_ISO_QSORT_S\n-int git_qsort_s(void *base, size_t nmemb, size_t size,\n-\t\tint (*compar)(const void *, const void *, void *), void *ctx);\n-#define qsort_s git_qsort_s\n-#endif\n-\n-#define QSORT_S(base, n, compar, ctx) do {\t\t\t\\\n-\tif (qsort_s((base), (n), sizeof(*(base)), compar, ctx))\t\\\n-\t\tBUG(\"qsort_s() failed\");\t\t\t\\\n-} while (0)\n-\n #ifndef REG_STARTEND\n #error \"Git requires REG_STARTEND support. Compile with NO_REGEX=NeedsStartEnd\"\n #endif\n@@ -1416,39 +951,12 @@ int git_regcomp(regex_t *preg, const char *pattern, int cflags);\n # define FORCE_DIR_SET_GID 0\n #endif\n \n-#ifdef NO_NSEC\n-#undef USE_NSEC\n-#define ST_CTIME_NSEC(st) 0\n-#define ST_MTIME_NSEC(st) 0\n-#else\n-#ifdef USE_ST_TIMESPEC\n-#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctimespec.tv_nsec))\n-#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtimespec.tv_nsec))\n-#else\n-#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctim.tv_nsec))\n-#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtim.tv_nsec))\n-#endif\n-#endif\n-\n #ifdef UNRELIABLE_FSTAT\n #define fstat_is_reliable() 0\n #else\n #define fstat_is_reliable() 1\n #endif\n \n-#ifndef va_copy\n-/*\n- * Since an obvious implementation of va_list would be to make it a\n- * pointer into the stack frame, a simple assignment will work on\n- * many systems. But let's try to be more portable.\n- */\n-#ifdef __va_copy\n-#define va_copy(dst, src) __va_copy(dst, src)\n-#else\n-#define va_copy(dst, src) ((dst) = (src))\n-#endif\n-#endif\n-\n /* usage.c: only to be used for testing BUG() implementation (see test-tool) */\n extern int BUG_exit_code;\n \n@@ -1478,41 +986,6 @@ void bug_fl(const char *file, int line, const char *fmt, ...);\n # define SHELL_PATH \"/bin/sh\"\n #endif\n \n-#ifndef _POSIX_THREAD_SAFE_FUNCTIONS\n-static inline void git_flockfile(FILE *fh UNUSED)\n-{\n-\t; /* nothing */\n-}\n-static inline void git_funlockfile(FILE *fh UNUSED)\n-{\n-\t; /* nothing */\n-}\n-#undef flockfile\n-#undef funlockfile\n-#undef getc_unlocked\n-#define flockfile(fh) git_flockfile(fh)\n-#define funlockfile(fh) git_funlockfile(fh)\n-#define getc_unlocked(fh) getc(fh)\n-#endif\n-\n-#ifdef FILENO_IS_A_MACRO\n-int git_fileno(FILE *stream);\n-# ifndef COMPAT_CODE_FILENO\n-# undef fileno\n-# define fileno(p) git_fileno(p)\n-# endif\n-#endif\n-\n-#ifdef NEED_ACCESS_ROOT_HANDLER\n-int git_access(const char *path, int mode);\n-# ifndef COMPAT_CODE_ACCESS\n-# ifdef access\n-# undef access\n-# endif\n-# define access(path, mode) git_access(path, mode)\n-# endif\n-#endif\n-\n /*\n * Our code often opens a path to an optional file, to work on its\n * contents when we can successfully open it. We can ignore a failure\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"511963","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-17-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","subject":"[PATCH v4 17/18] reftable: decouple from Git codebase by pulling in \"compat/posix.h\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:19Z","receivedAt":"2025-02-06T07:52:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library includes \"git-compat-util.h\" in order to get a\nPOSIX-like programming environment that papers over various differences\nbetween platforms. The header also brings with it a couple of helpers\nspecific to the Git codebase though, and over time we have started to\nuse these helpers in the reftable library, as well.\n\nThis makes it very hard to use the reftable library as a standalone\nlibrary without the rest of the Git codebase, so other libraries like\ne.g. libgit2 cannot easily use it. But now that we have removed all\ncalls to Git-specific functionality and have split out \"compat/posix.h\"\nas a separate header we can address this.\n\nStop including \"git-compat-util.h\" and instead include \"compat/posix.h\"\nto finalize the decoupling of the reftable library from the rest of the\nGit codebase. The only bits which remain specific to Git are \"system.h\"\nand \"system.c\", which projects will have to provide.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/system.c | 2 ++\n reftable/system.h | 3 ++-\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/reftable/system.c b/reftable/system.c\nindex e25ccc0da3..1ee268b125 100644\n--- a/reftable/system.c\n+++ b/reftable/system.c\n@@ -1,3 +1,5 @@\n+#include \"../git-compat-util.h\"\n+\n #include \"system.h\"\n #include \"basics.h\"\n #include \"reftable-error.h\"\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 3bd4a4e322..dccdf11f76 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -11,7 +11,8 @@ license that can be found in the LICENSE file or at\n \n /* This header glues the reftable library to the rest of Git */\n \n-#include \"git-compat-util.h\"\n+#include \"../compat/posix.h\"\n+#include \n \n /*\n * Return a random 32 bit integer. This function is expected to return\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"511964","messageId":"20250206-pks-reftable-drop-git-compat-util-v4-18-603d276d5f95@pks.im","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im","subject":"[PATCH v4 18/18] Makefile: skip reftable library for Coccinelle","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-06T07:52:20Z","receivedAt":"2025-02-06T07:52:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library does not use any of the common helpers that the Git\nproject has. Consequently, most of the rules that we have in Coccinelle\ndo not apply to the library at all and may even generate false positives\nwhen a pattern can be converted to use a Git helper function.\n\nExclude reftable library sources from being checked by Coccinelle to\navoid such false positives.\n\nSigned-off-by: Patrick Steinhardt \n---\n Makefile | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex d3011e30f7..dc2ac32e4a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -946,7 +946,7 @@ FOUND_SOURCE_FILES := $(filter-out $(GENERATED_H),$(shell $(SOURCES_CMD)))\n FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES))\n FOUND_H_SOURCES = $(filter %.h,$(FOUND_SOURCE_FILES))\n \n-COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES))\n+COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES) reftable/%,$(FOUND_C_SOURCES))\n \n LIB_H = $(FOUND_H_SOURCES)\n \n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512048","messageId":"87bjve3wd8.fsf@iotcl.com","threadId":"62862","inReplyTo":"20250206-pks-reftable-drop-git-compat-util-v4-14-603d276d5f95@pks.im","subject":"Re: [PATCH v4 14/18] reftable/basics: stop using `UNUSED` annotation","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2025-02-07T10:03:15Z","receivedAt":"2025-02-07T10:03:35Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Patrick Steinhardt writes:\n\n> Stop using the `UNUSED` annotation and replace it with a new\n> `REFTABLE_UNUSED` macro. The latter is a weaker guarantee compared to\n> `UNUSED` as it only suppresses unused parameters without generating a\n> warning in case a parameter marked as unused is in fact used. But it's\n> good enough, and by relaxing the behaviour a bit we avoid having to wire\n> up compiler-specific logic.\n\nI see MAYBE_UNUSED is defined as `__attribute__((__unused__))`, which as\nfar as I can tell is independent on the compiler, and has the same\neffect as the implementation in this patch. Would it make sense to use\nthat defition instead? Or did you intentionally choose a statement that\nwill sit on a separate line so it's more obviously that line needs to be\ndeleted whenever the parameter is put in use?\n\n-- \nToon\n"},{"id":"512074","messageId":"Z6Xze8aEWbia8etK@pks.im","threadId":"62862","inReplyTo":"87bjve3wd8.fsf@iotcl.com","subject":"Re: [PATCH v4 14/18] reftable/basics: stop using `UNUSED` annotation","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:50:19Z","receivedAt":"2025-02-07T11:50:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Feb 07, 2025 at 11:03:15AM +0100, Toon Claes wrote:\n> Patrick Steinhardt writes:\n> \n> > Stop using the `UNUSED` annotation and replace it with a new\n> > `REFTABLE_UNUSED` macro. The latter is a weaker guarantee compared to\n> > `UNUSED` as it only suppresses unused parameters without generating a\n> > warning in case a parameter marked as unused is in fact used. But it's\n> > good enough, and by relaxing the behaviour a bit we avoid having to wire\n> > up compiler-specific logic.\n> \n> I see MAYBE_UNUSED is defined as `__attribute__((__unused__))`, which as\n> far as I can tell is independent on the compiler, and has the same\n> effect as the implementation in this patch. Would it make sense to use\n> that defition instead? Or did you intentionally choose a statement that\n> will sit on a separate line so it's more obviously that line needs to be\n> deleted whenever the parameter is put in use?\n\nInteresting. I was checking the definition of `UNUSED`, which depends on\nthe actual compiler we're using. But you're right, `MAYBE_UNUSED` is\ndefined unconditionally, so by that reasoning we should be able to use\nit on all supported platforms indeed.\n\nI'll revise this patch, thanks for noticing!\n\nPatrick\n"},{"id":"512075","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im","subject":"[PATCH v5 00/18] reftable: stop using \"git-compat-util.h\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:51:55Z","receivedAt":"2025-02-07T11:52:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series is the final step to fully decouple the reftable\nlibrary from the rest of the Git codebase. The goal of this is to make\nthe library reusable by other projects like libgit2 by simply copying\nover the source files, making Git the canonical upstream for reftable\nfunctionality.\n\nThis patch series stops using all kinds of helpers exposed by our\n\"git-compat-util.h\" header and open-codes them instead. In order to keep\nus from using these helpers by accident the final step is to pull out\nPOSIX-related bits and pieces into a new \"compat/posix.h\" header, which\nthe reftable library then uses instead of \"git-compat-util.h\".\n\nThe series is built on top of master at 5f8f7081f7 (The third batch,\n2025-01-23) with ps/reftable-sign-compare at 33319b0976 (reftable:\naddress trivial -Wsign-compare warnings, 2025-01-20) merged into it.\nThere is a trivial merge conflict with ps/zlib-ng that can be solved\nlike this:\n\n diff --cc reftable/system.h\n index e4a8944a70,d02eacea8f..0000000000\n --- a/reftable/system.h\n +++ b/reftable/system.h\n @@@ -11,15 -11,9 +11,15 @@@ https://developers.google.com/open-sour\n \n /* This header glues the reftable library to the rest of Git */\n \n -#include \"git-compat-util.h\"\n +#include \"compat/posix.h\"\n - #include \n + #include \"compat/zlib-compat.h\"\n \n +/*\n + * Return a random 32 bit integer. This function is expected to return\n + * pre-seeded data.\n + */\n +uint32_t reftable_rand(void);\n +\n /*\n * An implementation-specific temporary file. By making this specific to the\n * implementation it becomes possible to tie temporary files into any kind of\n\nChanges in v2:\n - The splitup of Windows headers has broken compilation because some\n of the headers couldn't be found anymore. I've fixed this more\n generally by converting includes in \"compat/\" to always be relative\n to the project source directory, dropping the platform-specific\n `-Icompat/` include.\n - Explain why we don't port over `EWOULDBLOCK` handling.\n - Fix commit message typos.\n - Link to v1: https://lore.kernel.org/r/20250127-pks-reftable-drop-git-compat-util-v1-0-6e280a564877@pks.im\n\nChanges in v3:\n - Fix type of `total_read` variable used to track how many bytes we\n have read in `fd_read_lines()`.\n - Drop the patch use root-relative includes again. Let's rather\n discuss this outside of the scope of this series.\n - Link to v2: https://lore.kernel.org/r/20250128-pks-reftable-drop-git-compat-util-v2-0-c85c20336317@pks.im\n\nChanges in v4:\n - Fix a couple of now-invalid relative includes that I missed. This\n fixes the build issue in \"seen\" with Meson.\n - Link to v3: https://lore.kernel.org/r/20250203-pks-reftable-drop-git-compat-util-v3-0-446c9ed4ee9e@pks.im\n\nChanges in v5:\n - Rework the implementation of `REFTABLE_UNUSED` to use\n `__attribute__((__unused__))`.\n - Link to v4: https://lore.kernel.org/r/20250206-pks-reftable-drop-git-compat-util-v4-0-603d276d5f95@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (18):\n reftable/stack: stop using `read_in_full()`\n reftable/stack: stop using `write_in_full()`\n reftable/blocksource: stop using `xmmap()`\n reftable/record: stop using `COPY_ARRAY()`\n reftable/record: stop using `BUG()` in `reftable_record_init()`\n reftable/record: don't `BUG()` in `reftable_record_cmp()`\n reftable: stop using `BUG()` in trivial cases\n reftable/basics: stop using `st_mult()` in array allocators\n reftable/basics: provide wrappers for big endian conversion\n reftable/reader: stop using `ARRAY_SIZE()` macro\n reftable/system: introduce `reftable_rand()`\n reftable/stack: stop using `sleep_millisec()`\n reftable/basics: stop using `SWAP()` macro\n reftable/basics: introduce `REFTABLE_UNUSED` annotation\n compat/mingw: split out POSIX-related bits\n git-compat-util.h: split out POSIX-emulating bits\n reftable: decouple from Git codebase by pulling in \"compat/posix.h\"\n Makefile: skip reftable library for Coccinelle\n\n Makefile | 2 +-\n compat/{mingw.c => mingw/compat-util.c} | 28 +-\n compat/mingw/compat-util.h | 220 +++++++++++++\n compat/{mingw.h => mingw/posix.h} | 216 +------------\n compat/msvc.c | 6 -\n compat/msvc/compat-util.c | 6 +\n compat/msvc/compat-util.h | 7 +\n compat/{msvc.h => msvc/posix.h} | 8 +-\n compat/posix.h | 541 ++++++++++++++++++++++++++++++++\n config.mak.uname | 6 +-\n contrib/buildsystems/CMakeLists.txt | 2 +-\n git-compat-util.h | 535 +------------------------------\n meson.build | 8 +-\n reftable/basics.c | 19 --\n reftable/basics.h | 123 +++++++-\n reftable/block.c | 16 +-\n reftable/blocksource.c | 17 +-\n reftable/iter.c | 13 +-\n reftable/merged.c | 27 +-\n reftable/pq.c | 40 ++-\n reftable/pq.h | 2 +-\n reftable/reader.c | 33 +-\n reftable/record.c | 84 ++---\n reftable/record.h | 6 +-\n reftable/stack.c | 52 ++-\n reftable/system.c | 7 +\n reftable/system.h | 9 +-\n reftable/writer.c | 27 +-\n t/unit-tests/t-reftable-basics.c | 28 +-\n t/unit-tests/t-reftable-pq.c | 22 +-\n t/unit-tests/t-reftable-record.c | 42 ++-\n 31 files changed, 1205 insertions(+), 947 deletions(-)\n\nRange-diff versus v4:\n\n 1: 7db7704c24 = 1: 74f576c975 reftable/stack: stop using `read_in_full()`\n 2: 3ddd26284d = 2: acd5854f56 reftable/stack: stop using `write_in_full()`\n 3: 5f4a747da9 = 3: 25aab5acc1 reftable/blocksource: stop using `xmmap()`\n 4: 64b87dd35f = 4: 5ea4f2d8ea reftable/record: stop using `COPY_ARRAY()`\n 5: 49ab3789ac = 5: 2ca248c1cc reftable/record: stop using `BUG()` in `reftable_record_init()`\n 6: 47aacb4064 = 6: 590b9097e3 reftable/record: don't `BUG()` in `reftable_record_cmp()`\n 7: 4a777111bb = 7: 461cb30e1c reftable: stop using `BUG()` in trivial cases\n 8: 743e70374d = 8: 72f672b731 reftable/basics: stop using `st_mult()` in array allocators\n 9: 2a7cba7ec7 = 9: c597bd1621 reftable/basics: provide wrappers for big endian conversion\n10: fd741883d7 = 10: 643b70b67e reftable/reader: stop using `ARRAY_SIZE()` macro\n11: 2a4e188e96 = 11: dc809eefbf reftable/system: introduce `reftable_rand()`\n12: 57014b3de7 = 12: bf59b53b60 reftable/stack: stop using `sleep_millisec()`\n13: 7daddddd0d = 13: 68ffa9306f reftable/basics: stop using `SWAP()` macro\n14: e7556a7d53 < -: ---------- reftable/basics: stop using `UNUSED` annotation\n -: ---------- > 14: b444c64faf reftable/basics: introduce `REFTABLE_UNUSED` annotation\n15: 4628c8a98e = 15: 2043277fb1 compat/mingw: split out POSIX-related bits\n16: 055b865078 = 16: f496f30aa2 git-compat-util.h: split out POSIX-emulating bits\n17: ae323e3b8d = 17: 3e4564d927 reftable: decouple from Git codebase by pulling in \"compat/posix.h\"\n18: 6c3360b76a = 18: 63d85e0ace Makefile: skip reftable library for Coccinelle\n\n---\nbase-commit: 8047765d092881ec4aef7dfc57772161eee7f0f5\nchange-id: 20241119-pks-reftable-drop-git-compat-util-470f2bfde562\n\n"},{"id":"512076","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-1-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","subject":"[PATCH v5 01/18] reftable/stack: stop using `read_in_full()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:51:56Z","receivedAt":"2025-02-07T11:52:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"There is a single callsite of `read_in_full()` in the reftable library.\nOpen-code the function to reduce our dependency on the Git library.\n\nNote that we only partially port over the logic from `read_in_full()`\nand its underlying `xread()` helper. Most importantly, the latter also\nknows to handle `EWOULDBLOCK` via `handle_nonblock()`. This logic is\nirrelevant for us though because the reftable library never sets the\n`O_NONBLOCK` option in the first place.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 18 ++++++++++++++----\n 1 file changed, 14 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex f7c1845e15..d57acd9540 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -115,13 +115,16 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \n static int fd_read_lines(int fd, char ***namesp)\n {\n-\toff_t size = lseek(fd, 0, SEEK_END);\n \tchar *buf = NULL;\n \tint err = 0;\n+\toff_t size;\n+\n+\tsize = lseek(fd, 0, SEEK_END);\n \tif (size < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n+\n \terr = lseek(fd, 0, SEEK_SET);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n@@ -134,9 +137,16 @@ static int fd_read_lines(int fd, char ***namesp)\n \t\tgoto done;\n \t}\n \n-\tif (read_in_full(fd, buf, size) != size) {\n-\t\terr = REFTABLE_IO_ERROR;\n-\t\tgoto done;\n+\tfor (off_t total_read = 0; total_read < size; ) {\n+\t\tssize_t bytes_read = read(fd, buf + total_read, size - total_read);\n+\t\tif (bytes_read < 0 && (errno == EAGAIN || errno == EINTR))\n+\t\t\tcontinue;\n+\t\tif (bytes_read < 0 || !bytes_read) {\n+\t\t\terr = REFTABLE_IO_ERROR;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\ttotal_read += bytes_read;\n \t}\n \tbuf[size] = 0;\n \n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512077","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-2-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","subject":"[PATCH v5 02/18] reftable/stack: stop using `write_in_full()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:51:57Z","receivedAt":"2025-02-07T11:52:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Similar to the preceding commit, drop our use of `write_in_full()` and\nimplement a new wrapper `reftable_write_full()` that handles this logic\nfor us. This is done to reduce our dependency on the Git library.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 28 ++++++++++++++++++++++++----\n 1 file changed, 24 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex d57acd9540..46abf51ec5 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -48,6 +48,25 @@ static int stack_fsync(const struct reftable_write_options *opts, int fd)\n \treturn fsync(fd);\n }\n \n+static ssize_t reftable_write_data(int fd, const void *data, size_t size)\n+{\n+\tsize_t total_written = 0;\n+\tconst char *p = data;\n+\n+\twhile (total_written < size) {\n+\t\tssize_t bytes_written = write(fd, p, size - total_written);\n+\t\tif (bytes_written < 0 && (errno == EAGAIN || errno == EINTR))\n+\t\t\tcontinue;\n+\t\tif (bytes_written < 0)\n+\t\t\treturn REFTABLE_IO_ERROR;\n+\n+\t\ttotal_written += bytes_written;\n+\t\tp += bytes_written;\n+\t}\n+\n+\treturn total_written;\n+}\n+\n struct fd_writer {\n \tconst struct reftable_write_options *opts;\n \tint fd;\n@@ -56,7 +75,7 @@ struct fd_writer {\n static ssize_t fd_writer_write(void *arg, const void *data, size_t sz)\n {\n \tstruct fd_writer *writer = arg;\n-\treturn write_in_full(writer->fd, data, sz);\n+\treturn reftable_write_data(writer->fd, data, sz);\n }\n \n static int fd_writer_flush(void *arg)\n@@ -784,7 +803,8 @@ int reftable_addition_commit(struct reftable_addition *add)\n \t\t\tgoto done;\n \t}\n \n-\terr = write_in_full(add->tables_list_lock.fd, table_list.buf, table_list.len);\n+\terr = reftable_write_data(add->tables_list_lock.fd,\n+\t\t\t\t table_list.buf, table_list.len);\n \treftable_buf_release(&table_list);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n@@ -1468,8 +1488,8 @@ static int stack_compact_range(struct reftable_stack *st,\n \t\t\tgoto done;\n \t}\n \n-\terr = write_in_full(tables_list_lock.fd,\n-\t\t\t tables_list_buf.buf, tables_list_buf.len);\n+\terr = reftable_write_data(tables_list_lock.fd,\n+\t\t\t\t tables_list_buf.buf, tables_list_buf.len);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tunlink(new_table_path.buf);\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512078","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-3-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","subject":"[PATCH v5 03/18] reftable/blocksource: stop using `xmmap()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:51:58Z","receivedAt":"2025-02-07T11:52:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We use `xmmap()` to map reftables into memory. This function has two\nproblems:\n\n - It causes us to die in case the mmap fails.\n\n - It ties us to the Git codebase.\n\nRefactor the code to use mmap(3p) instead with manual error checking.\nNote that this function may not be the system-provided mmap(3p), but may\npoint to our `git_mmap()` wrapper that emulates the syscall on systems\nthat do not have mmap(3p) available.\n\nFix `reftable_block_source_from_file()` to properly bubble up the error\ncode in case the map(3p) call fails.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/blocksource.c | 11 ++++++++---\n 1 file changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex bba4a45b98..02972c46f4 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -98,7 +98,7 @@ static struct reftable_block_source_vtable file_vtable = {\n int reftable_block_source_from_file(struct reftable_block_source *bs,\n \t\t\t\t const char *name)\n {\n-\tstruct file_block_source *p;\n+\tstruct file_block_source *p = NULL;\n \tstruct stat st;\n \tint fd, err;\n \n@@ -122,7 +122,12 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,\n \t}\n \n \tp->size = st.st_size;\n-\tp->data = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tp->data = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tif (p->data == MAP_FAILED) {\n+\t\terr = REFTABLE_IO_ERROR;\n+\t\tp->data = NULL;\n+\t\tgoto out;\n+\t}\n \n \tassert(!bs->ops);\n \tbs->ops = &file_vtable;\n@@ -135,5 +140,5 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,\n \t\tclose(fd);\n \tif (err < 0)\n \t\treftable_free(p);\n-\treturn 0;\n+\treturn err;\n }\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512079","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-4-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","subject":"[PATCH v5 04/18] reftable/record: stop using `COPY_ARRAY()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:51:59Z","receivedAt":"2025-02-07T11:52:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Drop our use of `COPY_ARRAY()`, replacing it with an open-coded variant\nthereof. This is done to reduce our dependency on the Git library.\n\nWhile at it, guard the whole array copy logic so that we only copy it in\ncase there actually is anything to be copied. Otherwise, we may end up\ntrying to allocate a zero-sized array, which will return a NULL pointer\nand thus cause us to return an `REFTABLE_OUT_OF_MEMORY_ERROR`.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/record.c | 16 +++++++++++-----\n 1 file changed, 11 insertions(+), 5 deletions(-)\n\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 8919df8a4d..2c0cc32cbd 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -504,11 +504,17 @@ static int reftable_obj_record_copy_from(void *rec, const void *src_rec,\n \tif (src->hash_prefix_len)\n \t\tmemcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);\n \n-\tREFTABLE_ALLOC_ARRAY(obj->offsets, src->offset_len);\n-\tif (!obj->offsets)\n-\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n-\tobj->offset_len = src->offset_len;\n-\tCOPY_ARRAY(obj->offsets, src->offsets, src->offset_len);\n+\tif (src->offset_len) {\n+\t\tif (sizeof(*src->offsets) > SIZE_MAX / src->offset_len)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n+\t\tREFTABLE_ALLOC_ARRAY(obj->offsets, src->offset_len);\n+\t\tif (!obj->offsets)\n+\t\t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n+\n+\t\tmemcpy(obj->offsets, src->offsets, sizeof(*src->offsets) * src->offset_len);\n+\t\tobj->offset_len = src->offset_len;\n+\t}\n \n \treturn 0;\n }\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512080","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-5-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","subject":"[PATCH v5 05/18] reftable/record: stop using `BUG()` in `reftable_record_init()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:52:00Z","receivedAt":"2025-02-07T11:52:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're aborting the program via `BUG()` in case `reftable_record_init()`\nwas invoked with an unknown record type. This is bad because we may now\ndie in library code, and because it makes us depend on the Git codebase.\n\nRefactor the code such that `reftable_record_init()` can return an error\ncode to the caller. Adapt any callers accordingly.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/block.c | 4 +++-\n reftable/merged.c | 5 ++++-\n reftable/reader.c | 5 ++++-\n reftable/record.c | 8 ++++----\n reftable/record.h | 4 ++--\n t/unit-tests/t-reftable-pq.c | 6 +++---\n t/unit-tests/t-reftable-record.c | 2 +-\n 7 files changed, 21 insertions(+), 13 deletions(-)\n\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 8ac865ce78..255d566854 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -509,7 +509,9 @@ int block_iter_seek_key(struct block_iter *it, const struct block_reader *br,\n \tit->block_len = br->block_len;\n \tit->hash_size = br->hash_size;\n \n-\treftable_record_init(&rec, block_reader_type(br));\n+\terr = reftable_record_init(&rec, block_reader_type(br));\n+\tif (err < 0)\n+\t\tgoto done;\n \n \t/*\n \t * We're looking for the last entry less than the wanted key so that\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex e72b39e178..4156eec07f 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -253,7 +253,10 @@ int merged_table_init_iter(struct reftable_merged_table *mt,\n \t}\n \n \tfor (size_t i = 0; i < mt->readers_len; i++) {\n-\t\treftable_record_init(&subiters[i].rec, typ);\n+\t\tret = reftable_record_init(&subiters[i].rec, typ);\n+\t\tif (ret < 0)\n+\t\t\tgoto out;\n+\n \t\tret = reader_init_iter(mt->readers[i], &subiters[i].iter, typ);\n \t\tif (ret < 0)\n \t\t\tgoto out;\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 3f2e4b2800..de6e6dd932 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -360,7 +360,10 @@ static int table_iter_seek_linear(struct table_iter *ti,\n \tstruct reftable_record rec;\n \tint err;\n \n-\treftable_record_init(&rec, reftable_record_type(want));\n+\terr = reftable_record_init(&rec, reftable_record_type(want));\n+\tif (err < 0)\n+\t\tgoto done;\n+\n \terr = reftable_record_key(want, &want_key);\n \tif (err < 0)\n \t\tgoto done;\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 2c0cc32cbd..1e18f8dffb 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -1306,7 +1306,7 @@ reftable_record_vtable(struct reftable_record *rec)\n \tabort();\n }\n \n-void reftable_record_init(struct reftable_record *rec, uint8_t typ)\n+int reftable_record_init(struct reftable_record *rec, uint8_t typ)\n {\n \tmemset(rec, 0, sizeof(*rec));\n \trec->type = typ;\n@@ -1315,11 +1315,11 @@ void reftable_record_init(struct reftable_record *rec, uint8_t typ)\n \tcase BLOCK_TYPE_REF:\n \tcase BLOCK_TYPE_LOG:\n \tcase BLOCK_TYPE_OBJ:\n-\t\treturn;\n+\t\treturn 0;\n \tcase BLOCK_TYPE_INDEX:\n \t\treftable_buf_init(&rec->u.idx.last_key);\n-\t\treturn;\n+\t\treturn 0;\n \tdefault:\n-\t\tBUG(\"unhandled record type\");\n+\t\treturn REFTABLE_API_ERROR;\n \t}\n }\ndiff --git a/reftable/record.h b/reftable/record.h\nindex c7755a4d75..e1846c294b 100644\n--- a/reftable/record.h\n+++ b/reftable/record.h\n@@ -130,8 +130,8 @@ struct reftable_record {\n \t} u;\n };\n \n-/* Initialize the reftable record for the given type */\n-void reftable_record_init(struct reftable_record *rec, uint8_t typ);\n+/* Initialize the reftable record for the given type. */\n+int reftable_record_init(struct reftable_record *rec, uint8_t typ);\n \n /* see struct record_vtable */\n int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b);\ndiff --git a/t/unit-tests/t-reftable-pq.c b/t/unit-tests/t-reftable-pq.c\nindex f3f8a0cdf3..d8a4c283a1 100644\n--- a/t/unit-tests/t-reftable-pq.c\n+++ b/t/unit-tests/t-reftable-pq.c\n@@ -32,7 +32,7 @@ static void t_pq_record(void)\n \tchar *last = NULL;\n \n \tfor (i = 0; i < N; i++) {\n-\t\treftable_record_init(&recs[i], BLOCK_TYPE_REF);\n+\t\tcheck(!reftable_record_init(&recs[i], BLOCK_TYPE_REF));\n \t\trecs[i].u.ref.refname = xstrfmt(\"%02\"PRIuMAX, (uintmax_t)i);\n \t}\n \n@@ -72,7 +72,7 @@ static void t_pq_index(void)\n \tsize_t N = ARRAY_SIZE(recs), i;\n \n \tfor (i = 0; i < N; i++) {\n-\t\treftable_record_init(&recs[i], BLOCK_TYPE_REF);\n+\t\tcheck(!reftable_record_init(&recs[i], BLOCK_TYPE_REF));\n \t\trecs[i].u.ref.refname = (char *) \"refs/heads/master\";\n \t}\n \n@@ -111,7 +111,7 @@ static void t_merged_iter_pqueue_top(void)\n \tsize_t N = ARRAY_SIZE(recs), i;\n \n \tfor (i = 0; i < N; i++) {\n-\t\treftable_record_init(&recs[i], BLOCK_TYPE_REF);\n+\t\tcheck(!reftable_record_init(&recs[i], BLOCK_TYPE_REF));\n \t\trecs[i].u.ref.refname = (char *) \"refs/heads/master\";\n \t}\n \ndiff --git a/t/unit-tests/t-reftable-record.c b/t/unit-tests/t-reftable-record.c\nindex d49d2a2729..6540bd20e3 100644\n--- a/t/unit-tests/t-reftable-record.c\n+++ b/t/unit-tests/t-reftable-record.c\n@@ -17,7 +17,7 @@ static void t_copy(struct reftable_record *rec)\n \tuint8_t typ;\n \n \ttyp = reftable_record_type(rec);\n-\treftable_record_init(©, typ);\n+\tcheck(!reftable_record_init(©, typ));\n \treftable_record_copy_from(©, rec, REFTABLE_HASH_SIZE_SHA1);\n \t/* do it twice to catch memory leaks */\n \treftable_record_copy_from(©, rec, REFTABLE_HASH_SIZE_SHA1);\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512081","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-6-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","subject":"[PATCH v5 06/18] reftable/record: don't `BUG()` in `reftable_record_cmp()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:52:01Z","receivedAt":"2025-02-07T11:52:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library aborts with a bug in case `reftable_record_cmp()`\nis invoked with two records of differing types. This would cause the\nprogram to die without the caller being able to handle the error, which\nis not something we want in the context of library code. And it ties us\nto the Git codebase.\n\nRefactor the code such that `reftable_record_cmp()` returns an error\ncode separate from the actual comparison result. This requires us to\nalso adapt some callers up the callchain in a similar fashion.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/merged.c | 20 +++++++++++++++-----\n reftable/pq.c | 36 +++++++++++++++++++++++++++++-------\n reftable/pq.h | 2 +-\n reftable/record.c | 10 ++++++----\n reftable/record.h | 2 +-\n t/unit-tests/t-reftable-pq.c | 16 ++++++++++++----\n t/unit-tests/t-reftable-record.c | 40 ++++++++++++++++++++++++++++------------\n 7 files changed, 92 insertions(+), 34 deletions(-)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 4156eec07f..563864068c 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -66,8 +66,11 @@ static int merged_iter_seek(struct merged_iter *mi, struct reftable_record *want\n \tint err;\n \n \tmi->advance_index = -1;\n-\twhile (!merged_iter_pqueue_is_empty(mi->pq))\n-\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq)) {\n+\t\terr = merged_iter_pqueue_remove(&mi->pq, NULL);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\t}\n \n \tfor (size_t i = 0; i < mi->subiters_len; i++) {\n \t\terr = iterator_seek(&mi->subiters[i].iter, want);\n@@ -120,7 +123,9 @@ static int merged_iter_next_entry(struct merged_iter *mi,\n \tif (empty)\n \t\treturn 1;\n \n-\tentry = merged_iter_pqueue_remove(&mi->pq);\n+\terr = merged_iter_pqueue_remove(&mi->pq, &entry);\n+\tif (err < 0)\n+\t\treturn err;\n \n \t/*\n \t One can also use reftable as datacenter-local storage, where the ref\n@@ -134,11 +139,16 @@ static int merged_iter_next_entry(struct merged_iter *mi,\n \t\tstruct pq_entry top = merged_iter_pqueue_top(mi->pq);\n \t\tint cmp;\n \n-\t\tcmp = reftable_record_cmp(top.rec, entry.rec);\n+\t\terr = reftable_record_cmp(top.rec, entry.rec, &cmp);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n \t\tif (cmp > 0)\n \t\t\tbreak;\n \n-\t\tmerged_iter_pqueue_remove(&mi->pq);\n+\t\terr = merged_iter_pqueue_remove(&mi->pq, NULL);\n+\t\tif (err < 0)\n+\t\t\treturn err;\n+\n \t\terr = merged_iter_advance_subiter(mi, top.index);\n \t\tif (err < 0)\n \t\t\treturn err;\ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex 5591e875e1..ef8035cfd9 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -15,13 +15,18 @@ license that can be found in the LICENSE file or at\n \n int pq_less(struct pq_entry *a, struct pq_entry *b)\n {\n-\tint cmp = reftable_record_cmp(a->rec, b->rec);\n+\tint cmp, err;\n+\n+\terr = reftable_record_cmp(a->rec, b->rec, &cmp);\n+\tif (err < 0)\n+\t\treturn err;\n+\n \tif (cmp == 0)\n \t\treturn a->index > b->index;\n \treturn cmp < 0;\n }\n \n-struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n+int merged_iter_pqueue_remove(struct merged_iter_pqueue *pq, struct pq_entry *out)\n {\n \tsize_t i = 0;\n \tstruct pq_entry e = pq->heap[0];\n@@ -32,17 +37,34 @@ struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)\n \t\tsize_t min = i;\n \t\tsize_t j = 2 * i + 1;\n \t\tsize_t k = 2 * i + 2;\n-\t\tif (j < pq->len && pq_less(&pq->heap[j], &pq->heap[i]))\n-\t\t\tmin = j;\n-\t\tif (k < pq->len && pq_less(&pq->heap[k], &pq->heap[min]))\n-\t\t\tmin = k;\n+\t\tint cmp;\n+\n+\t\tif (j < pq->len) {\n+\t\t\tcmp = pq_less(&pq->heap[j], &pq->heap[i]);\n+\t\t\tif (cmp < 0)\n+\t\t\t\treturn -1;\n+\t\t\telse if (cmp)\n+\t\t\t\tmin = j;\n+\t\t}\n+\n+\t\tif (k < pq->len) {\n+\t\t\tcmp = pq_less(&pq->heap[k], &pq->heap[min]);\n+\t\t\tif (cmp < 0)\n+\t\t\t\treturn -1;\n+\t\t\telse if (cmp)\n+\t\t\t\tmin = k;\n+\t\t}\n+\n \t\tif (min == i)\n \t\t\tbreak;\n \t\tSWAP(pq->heap[i], pq->heap[min]);\n \t\ti = min;\n \t}\n \n-\treturn e;\n+\tif (out)\n+\t\t*out = e;\n+\n+\treturn 0;\n }\n \n int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e)\ndiff --git a/reftable/pq.h b/reftable/pq.h\nindex 83c062eeca..ff39016445 100644\n--- a/reftable/pq.h\n+++ b/reftable/pq.h\n@@ -22,7 +22,7 @@ struct merged_iter_pqueue {\n \tsize_t cap;\n };\n \n-struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);\n+int merged_iter_pqueue_remove(struct merged_iter_pqueue *pq, struct pq_entry *out);\n int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e);\n void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);\n int pq_less(struct pq_entry *a, struct pq_entry *b);\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 1e18f8dffb..b39d99fcc7 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -1195,12 +1195,14 @@ int reftable_record_is_deletion(struct reftable_record *rec)\n \t\treftable_record_data(rec));\n }\n \n-int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b)\n+int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b,\n+\t\t\tint *cmp)\n {\n \tif (a->type != b->type)\n-\t\tBUG(\"cannot compare reftable records of different type\");\n-\treturn reftable_record_vtable(a)->cmp(\n-\t\treftable_record_data(a), reftable_record_data(b));\n+\t\treturn -1;\n+\t*cmp = reftable_record_vtable(a)->cmp(reftable_record_data(a),\n+\t\t\t\t\t reftable_record_data(b));\n+\treturn 0;\n }\n \n int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, uint32_t hash_size)\ndiff --git a/reftable/record.h b/reftable/record.h\nindex e1846c294b..867810a932 100644\n--- a/reftable/record.h\n+++ b/reftable/record.h\n@@ -134,7 +134,7 @@ struct reftable_record {\n int reftable_record_init(struct reftable_record *rec, uint8_t typ);\n \n /* see struct record_vtable */\n-int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b);\n+int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b, int *cmp);\n int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, uint32_t hash_size);\n int reftable_record_key(struct reftable_record *rec, struct reftable_buf *dest);\n int reftable_record_copy_from(struct reftable_record *rec,\ndiff --git a/t/unit-tests/t-reftable-pq.c b/t/unit-tests/t-reftable-pq.c\nindex d8a4c283a1..c128fe8616 100644\n--- a/t/unit-tests/t-reftable-pq.c\n+++ b/t/unit-tests/t-reftable-pq.c\n@@ -21,7 +21,9 @@ static void merged_iter_pqueue_check(const struct merged_iter_pqueue *pq)\n \n static int pq_entry_equal(struct pq_entry *a, struct pq_entry *b)\n {\n-\treturn !reftable_record_cmp(a->rec, b->rec) && (a->index == b->index);\n+\tint cmp;\n+\tcheck(!reftable_record_cmp(a->rec, b->rec, &cmp));\n+\treturn !cmp && (a->index == b->index);\n }\n \n static void t_pq_record(void)\n@@ -49,7 +51,9 @@ static void t_pq_record(void)\n \n \twhile (!merged_iter_pqueue_is_empty(pq)) {\n \t\tstruct pq_entry top = merged_iter_pqueue_top(pq);\n-\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct pq_entry e;\n+\n+\t\tcheck(!merged_iter_pqueue_remove(&pq, &e));\n \t\tmerged_iter_pqueue_check(&pq);\n \n \t\tcheck(pq_entry_equal(&top, &e));\n@@ -90,7 +94,9 @@ static void t_pq_index(void)\n \n \tfor (i = N - 1; i > 0; i--) {\n \t\tstruct pq_entry top = merged_iter_pqueue_top(pq);\n-\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct pq_entry e;\n+\n+\t\tcheck(!merged_iter_pqueue_remove(&pq, &e));\n \t\tmerged_iter_pqueue_check(&pq);\n \n \t\tcheck(pq_entry_equal(&top, &e));\n@@ -129,7 +135,9 @@ static void t_merged_iter_pqueue_top(void)\n \n \tfor (i = N - 1; i > 0; i--) {\n \t\tstruct pq_entry top = merged_iter_pqueue_top(pq);\n-\t\tstruct pq_entry e = merged_iter_pqueue_remove(&pq);\n+\t\tstruct pq_entry e;\n+\n+\t\tcheck(!merged_iter_pqueue_remove(&pq, &e));\n \n \t\tmerged_iter_pqueue_check(&pq);\n \t\tcheck(pq_entry_equal(&top, &e));\ndiff --git a/t/unit-tests/t-reftable-record.c b/t/unit-tests/t-reftable-record.c\nindex 6540bd20e3..5954966373 100644\n--- a/t/unit-tests/t-reftable-record.c\n+++ b/t/unit-tests/t-reftable-record.c\n@@ -100,16 +100,20 @@ static void t_reftable_ref_record_comparison(void)\n \t\t\t.u.ref.value.symref = (char *) \"refs/heads/master\",\n \t\t},\n \t};\n+\tint cmp;\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \n \tin[1].u.ref.value_type = in[0].u.ref.value_type;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n }\n \n static void t_reftable_ref_record_compare_name(void)\n@@ -209,17 +213,20 @@ static void t_reftable_log_record_comparison(void)\n \t\t\t.u.log.update_index = 22,\n \t\t},\n \t};\n+\tint cmp;\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \t/* comparison should be reversed for equal keys, because\n \t * comparison is now performed on the basis of update indices */\n-\tcheck_int(reftable_record_cmp(&in[0], &in[1]), <, 0);\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck_int(cmp, <, 0);\n \n \tin[1].u.log.update_index = in[0].u.log.update_index;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n }\n \n static void t_reftable_log_record_compare_key(void)\n@@ -396,16 +403,20 @@ static void t_reftable_obj_record_comparison(void)\n \t\t\t.u.obj.hash_prefix_len = 5,\n \t\t},\n \t};\n+\tint cmp;\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \n \tin[1].u.obj.offset_len = in[0].u.obj.offset_len;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n }\n \n static void t_reftable_obj_record_roundtrip(void)\n@@ -486,19 +497,24 @@ static void t_reftable_index_record_comparison(void)\n \t\t\t.u.idx.last_key = REFTABLE_BUF_INIT,\n \t\t},\n \t};\n+\tint cmp;\n+\n \tcheck(!reftable_buf_addstr(&in[0].u.idx.last_key, \"refs/heads/master\"));\n \tcheck(!reftable_buf_addstr(&in[1].u.idx.last_key, \"refs/heads/master\"));\n \tcheck(!reftable_buf_addstr(&in[2].u.idx.last_key, \"refs/heads/branch\"));\n \n \tcheck(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tcheck(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck_int(reftable_record_cmp(&in[1], &in[2]), >, 0);\n+\tcheck(!reftable_record_cmp(&in[1], &in[2], &cmp));\n+\tcheck_int(cmp, >, 0);\n \n \tin[1].u.idx.offset = in[0].u.idx.offset;\n \tcheck(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1));\n-\tcheck(!reftable_record_cmp(&in[0], &in[1]));\n+\tcheck(!reftable_record_cmp(&in[0], &in[1], &cmp));\n+\tcheck(!cmp);\n \n \tfor (size_t i = 0; i < ARRAY_SIZE(in); i++)\n \t\treftable_record_release(&in[i]);\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512082","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-8-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","subject":"[PATCH v5 08/18] reftable/basics: stop using `st_mult()` in array allocators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:52:03Z","receivedAt":"2025-02-07T11:52:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're using `st_mult()` as part of our macro helpers that allocate\narrays. This is bad due two two reasons:\n\n - `st_mult()` causes us to die in case the multiplication overflows.\n\n - `st_mult()` ties us to the Git codebase.\n\nRefactor the code to instead detect overflows manually and return an\nerror in such cases.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 36 ++++++++++++++++++++++++++++++++----\n 1 file changed, 32 insertions(+), 4 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex a2a010a0e1..646f8d67f2 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -117,18 +117,46 @@ void reftable_free(void *p);\n void *reftable_calloc(size_t nelem, size_t elsize);\n char *reftable_strdup(const char *str);\n \n-#define REFTABLE_ALLOC_ARRAY(x, alloc) (x) = reftable_malloc(st_mult(sizeof(*(x)), (alloc)))\n+static inline int reftable_alloc_size(size_t nelem, size_t elsize, size_t *out)\n+{\n+\tif (nelem && elsize > SIZE_MAX / nelem)\n+\t\treturn -1;\n+\t*out = nelem * elsize;\n+\treturn 0;\n+}\n+\n+#define REFTABLE_ALLOC_ARRAY(x, alloc) do { \\\n+\t\tsize_t alloc_size; \\\n+\t\tif (reftable_alloc_size(sizeof(*(x)), (alloc), &alloc_size) < 0) { \\\n+\t\t\terrno = ENOMEM; \\\n+\t\t\t(x) = NULL; \\\n+\t\t} else { \\\n+\t\t\t(x) = reftable_malloc(alloc_size); \\\n+\t\t} \\\n+\t} while (0)\n #define REFTABLE_CALLOC_ARRAY(x, alloc) (x) = reftable_calloc((alloc), sizeof(*(x)))\n-#define REFTABLE_REALLOC_ARRAY(x, alloc) (x) = reftable_realloc((x), st_mult(sizeof(*(x)), (alloc)))\n+#define REFTABLE_REALLOC_ARRAY(x, alloc) do { \\\n+\t\tsize_t alloc_size; \\\n+\t\tif (reftable_alloc_size(sizeof(*(x)), (alloc), &alloc_size) < 0) { \\\n+\t\t\terrno = ENOMEM; \\\n+\t\t\t(x) = NULL; \\\n+\t\t} else { \\\n+\t\t\t(x) = reftable_realloc((x), alloc_size); \\\n+\t\t} \\\n+\t} while (0)\n \n static inline void *reftable_alloc_grow(void *p, size_t nelem, size_t elsize,\n \t\t\t\t\tsize_t *allocp)\n {\n \tvoid *new_p;\n-\tsize_t alloc = *allocp * 2 + 1;\n+\tsize_t alloc = *allocp * 2 + 1, alloc_bytes;\n \tif (alloc < nelem)\n \t\talloc = nelem;\n-\tnew_p = reftable_realloc(p, st_mult(elsize, alloc));\n+\tif (reftable_alloc_size(elsize, alloc, &alloc_bytes) < 0) {\n+\t\terrno = ENOMEM;\n+\t\treturn p;\n+\t}\n+\tnew_p = reftable_realloc(p, alloc_bytes);\n \tif (!new_p)\n \t\treturn p;\n \t*allocp = alloc;\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512084","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-7-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","subject":"[PATCH v5 07/18] reftable: stop using `BUG()` in trivial cases","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:52:02Z","receivedAt":"2025-02-07T11:52:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Stop using `BUG()` in the remaining trivial cases that we still have in\nthe reftable library. Instead of aborting the program, we'll now bubble\nup a `REFTABLE_API_ERROR` to indicate misuse of the calling conventions.\n\nNote that in both `reftable_reader_{inc,dec}ref()` we simply stop\ncalling `BUG()` altogether. The only situation where the counter should\nbe zero is when the structure has already been free'd anyway, so we\nwould run into undefined behaviour regardless of whether we try to abort\nthe program or not.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/iter.c | 3 +--\n reftable/reader.c | 4 ----\n reftable/writer.c | 5 ++---\n 3 files changed, 3 insertions(+), 9 deletions(-)\n\ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex 86e801ca9f..b2ffb09c16 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -146,8 +146,7 @@ static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n static int indexed_table_ref_iter_seek(void *p UNUSED,\n \t\t\t\t struct reftable_record *want UNUSED)\n {\n-\tBUG(\"seeking indexed table is not supported\");\n-\treturn -1;\n+\treturn REFTABLE_API_ERROR;\n }\n \n static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex de6e6dd932..36a5633ede 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -677,8 +677,6 @@ int reftable_reader_new(struct reftable_reader **out,\n \n void reftable_reader_incref(struct reftable_reader *r)\n {\n-\tif (!r->refcount)\n-\t\tBUG(\"cannot increment ref counter of dead reader\");\n \tr->refcount++;\n }\n \n@@ -686,8 +684,6 @@ void reftable_reader_decref(struct reftable_reader *r)\n {\n \tif (!r)\n \t\treturn;\n-\tif (!r->refcount)\n-\t\tBUG(\"cannot decrement ref counter of dead reader\");\n \tif (--r->refcount)\n \t\treturn;\n \tblock_source_close(&r->source);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 91d6629486..155863ee5f 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -158,7 +158,7 @@ int reftable_writer_new(struct reftable_writer **out,\n \t\topts = *_opts;\n \toptions_set_defaults(&opts);\n \tif (opts.block_size >= (1 << 24))\n-\t\tBUG(\"configured block size exceeds 16MB\");\n+\t\treturn REFTABLE_API_ERROR;\n \n \treftable_buf_init(&wp->block_writer_data.last_key);\n \treftable_buf_init(&wp->last_key);\n@@ -289,8 +289,7 @@ static int writer_add_record(struct reftable_writer *w,\n \t}\n \n \tif (block_writer_type(w->block_writer) != reftable_record_type(rec))\n-\t\tBUG(\"record of type %d added to writer of type %d\",\n-\t\t reftable_record_type(rec), block_writer_type(w->block_writer));\n+\t\treturn REFTABLE_API_ERROR;\n \n \t/*\n \t * Try to add the record to the writer. If this succeeds then we're\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512083","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-9-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","subject":"[PATCH v5 09/18] reftable/basics: provide wrappers for big endian conversion","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:52:04Z","receivedAt":"2025-02-07T11:52:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're using a mixture of big endian conversion functions provided by\nboth the reftable library, but also by the Git codebase. Refactor the\ncode so that we exclusively use reftable-provided wrappers in order to\nuntangle us from the Git codebase.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.c | 19 ----------\n reftable/basics.h | 76 ++++++++++++++++++++++++++++++++++++++--\n reftable/block.c | 12 +++----\n reftable/reader.c | 22 ++++++------\n reftable/record.c | 8 ++---\n reftable/writer.c | 20 +++++------\n t/unit-tests/t-reftable-basics.c | 28 ++++++++++++---\n 7 files changed, 127 insertions(+), 58 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex 3b5ea27bbd..8c4a4433e4 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -147,25 +147,6 @@ char *reftable_buf_detach(struct reftable_buf *buf)\n \treturn result;\n }\n \n-void put_be24(uint8_t *out, uint32_t i)\n-{\n-\tout[0] = (uint8_t)((i >> 16) & 0xff);\n-\tout[1] = (uint8_t)((i >> 8) & 0xff);\n-\tout[2] = (uint8_t)(i & 0xff);\n-}\n-\n-uint32_t get_be24(uint8_t *in)\n-{\n-\treturn (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 |\n-\t (uint32_t)(in[2]);\n-}\n-\n-void put_be16(uint8_t *out, uint16_t i)\n-{\n-\tout[0] = (uint8_t)((i >> 8) & 0xff);\n-\tout[1] = (uint8_t)(i & 0xff);\n-}\n-\n size_t binsearch(size_t sz, int (*f)(size_t k, void *args), void *args)\n {\n \tsize_t lo = 0;\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 646f8d67f2..c1ddbaec3f 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -76,9 +76,79 @@ char *reftable_buf_detach(struct reftable_buf *buf);\n \n /* Bigendian en/decoding of integers */\n \n-void put_be24(uint8_t *out, uint32_t i);\n-uint32_t get_be24(uint8_t *in);\n-void put_be16(uint8_t *out, uint16_t i);\n+static inline void reftable_put_be16(void *out, uint16_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 8) & 0xff);\n+\tp[1] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline void reftable_put_be24(void *out, uint32_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 16) & 0xff);\n+\tp[1] = (uint8_t)((i >> 8) & 0xff);\n+\tp[2] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline void reftable_put_be32(void *out, uint32_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 24) & 0xff);\n+\tp[1] = (uint8_t)((i >> 16) & 0xff);\n+\tp[2] = (uint8_t)((i >> 8) & 0xff);\n+\tp[3] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline void reftable_put_be64(void *out, uint64_t i)\n+{\n+\tunsigned char *p = out;\n+\tp[0] = (uint8_t)((i >> 56) & 0xff);\n+\tp[1] = (uint8_t)((i >> 48) & 0xff);\n+\tp[2] = (uint8_t)((i >> 40) & 0xff);\n+\tp[3] = (uint8_t)((i >> 32) & 0xff);\n+\tp[4] = (uint8_t)((i >> 24) & 0xff);\n+\tp[5] = (uint8_t)((i >> 16) & 0xff);\n+\tp[6] = (uint8_t)((i >> 8) & 0xff);\n+\tp[7] = (uint8_t)((i >> 0) & 0xff);\n+}\n+\n+static inline uint16_t reftable_get_be16(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint16_t)(p[0]) << 8 |\n+\t (uint16_t)(p[1]) << 0;\n+}\n+\n+static inline uint32_t reftable_get_be24(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint32_t)(p[0]) << 16 |\n+\t (uint32_t)(p[1]) << 8 |\n+\t (uint32_t)(p[2]) << 0;\n+}\n+\n+static inline uint32_t reftable_get_be32(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint32_t)(p[0]) << 24 |\n+\t (uint32_t)(p[1]) << 16 |\n+\t (uint32_t)(p[2]) << 8|\n+\t (uint32_t)(p[3]) << 0;\n+}\n+\n+static inline uint64_t reftable_get_be64(const void *in)\n+{\n+\tconst unsigned char *p = in;\n+\treturn (uint64_t)(p[0]) << 56 |\n+\t (uint64_t)(p[1]) << 48 |\n+\t (uint64_t)(p[2]) << 40 |\n+\t (uint64_t)(p[3]) << 32 |\n+\t (uint64_t)(p[4]) << 24 |\n+\t (uint64_t)(p[5]) << 16 |\n+\t (uint64_t)(p[6]) << 8 |\n+\t (uint64_t)(p[7]) << 0;\n+}\n \n /*\n * find smallest index i in [0, sz) at which `f(i) > 0`, assuming that f is\ndiff --git a/reftable/block.c b/reftable/block.c\nindex 255d566854..373908807e 100644\n--- a/reftable/block.c\n+++ b/reftable/block.c\n@@ -148,13 +148,13 @@ int block_writer_add(struct block_writer *w, struct reftable_record *rec)\n int block_writer_finish(struct block_writer *w)\n {\n \tfor (uint32_t i = 0; i < w->restart_len; i++) {\n-\t\tput_be24(w->block + w->next, w->restarts[i]);\n+\t\treftable_put_be24(w->block + w->next, w->restarts[i]);\n \t\tw->next += 3;\n \t}\n \n-\tput_be16(w->block + w->next, w->restart_len);\n+\treftable_put_be16(w->block + w->next, w->restart_len);\n \tw->next += 2;\n-\tput_be24(w->block + 1 + w->header_off, w->next);\n+\treftable_put_be24(w->block + 1 + w->header_off, w->next);\n \n \t/*\n \t * Log records are stored zlib-compressed. Note that the compression\n@@ -216,7 +216,7 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,\n {\n \tuint32_t full_block_size = table_block_size;\n \tuint8_t typ = block->data[header_off];\n-\tuint32_t sz = get_be24(block->data + header_off + 1);\n+\tuint32_t sz = reftable_get_be24(block->data + header_off + 1);\n \tint err = 0;\n \tuint16_t restart_count = 0;\n \tuint32_t restart_start = 0;\n@@ -300,7 +300,7 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,\n \t\tfull_block_size = sz;\n \t}\n \n-\trestart_count = get_be16(block->data + sz - 2);\n+\trestart_count = reftable_get_be16(block->data + sz - 2);\n \trestart_start = sz - 2 - 3 * restart_count;\n \trestart_bytes = block->data + restart_start;\n \n@@ -355,7 +355,7 @@ int block_reader_first_key(const struct block_reader *br, struct reftable_buf *k\n \n static uint32_t block_reader_restart_offset(const struct block_reader *br, size_t idx)\n {\n-\treturn get_be24(br->restart_bytes + 3 * idx);\n+\treturn reftable_get_be24(br->restart_bytes + 3 * idx);\n }\n \n void block_iter_seek_start(struct block_iter *it, const struct block_reader *br)\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex 36a5633ede..bf07a0a586 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -101,18 +101,18 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n \t}\n \n \tf++;\n-\tr->block_size = get_be24(f);\n+\tr->block_size = reftable_get_be24(f);\n \n \tf += 3;\n-\tr->min_update_index = get_be64(f);\n+\tr->min_update_index = reftable_get_be64(f);\n \tf += 8;\n-\tr->max_update_index = get_be64(f);\n+\tr->max_update_index = reftable_get_be64(f);\n \tf += 8;\n \n \tif (r->version == 1) {\n \t\tr->hash_id = REFTABLE_HASH_SHA1;\n \t} else {\n-\t\tswitch (get_be32(f)) {\n+\t\tswitch (reftable_get_be32(f)) {\n \t\tcase REFTABLE_FORMAT_ID_SHA1:\n \t\t\tr->hash_id = REFTABLE_HASH_SHA1;\n \t\t\tbreak;\n@@ -127,24 +127,24 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,\n \t\tf += 4;\n \t}\n \n-\tr->ref_offsets.index_offset = get_be64(f);\n+\tr->ref_offsets.index_offset = reftable_get_be64(f);\n \tf += 8;\n \n-\tr->obj_offsets.offset = get_be64(f);\n+\tr->obj_offsets.offset = reftable_get_be64(f);\n \tf += 8;\n \n \tr->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1);\n \tr->obj_offsets.offset >>= 5;\n \n-\tr->obj_offsets.index_offset = get_be64(f);\n+\tr->obj_offsets.index_offset = reftable_get_be64(f);\n \tf += 8;\n-\tr->log_offsets.offset = get_be64(f);\n+\tr->log_offsets.offset = reftable_get_be64(f);\n \tf += 8;\n-\tr->log_offsets.index_offset = get_be64(f);\n+\tr->log_offsets.index_offset = reftable_get_be64(f);\n \tf += 8;\n \n \tcomputed_crc = crc32(0, footer, f - footer);\n-\tfile_crc = get_be32(f);\n+\tfile_crc = reftable_get_be32(f);\n \tf += 4;\n \tif (computed_crc != file_crc) {\n \t\terr = REFTABLE_FORMAT_ERROR;\n@@ -214,7 +214,7 @@ static int32_t extract_block_size(uint8_t *data, uint8_t *typ, uint64_t off,\n \n \t*typ = data[0];\n \tif (reftable_is_block_type(*typ)) {\n-\t\tresult = get_be24(data + 1);\n+\t\tresult = reftable_get_be24(data + 1);\n \t}\n \treturn result;\n }\ndiff --git a/reftable/record.c b/reftable/record.c\nindex b39d99fcc7..3552bafa99 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -689,7 +689,7 @@ static int reftable_log_record_key(const void *r, struct reftable_buf *dest)\n \t\treturn err;\n \n \tts = (~ts) - rec->update_index;\n-\tput_be64(&i64[0], ts);\n+\treftable_put_be64(&i64[0], ts);\n \n \terr = reftable_buf_add(dest, i64, sizeof(i64));\n \tif (err < 0)\n@@ -814,7 +814,7 @@ static int reftable_log_record_encode(const void *rec, struct string_view s,\n \tif (s.len < 2)\n \t\treturn -1;\n \n-\tput_be16(s.buf, r->value.update.tz_offset);\n+\treftable_put_be16(s.buf, r->value.update.tz_offset);\n \tstring_view_consume(&s, 2);\n \n \tn = encode_string(\n@@ -846,7 +846,7 @@ static int reftable_log_record_decode(void *rec, struct reftable_buf key,\n \t}\n \n \tmemcpy(r->refname, key.buf, key.len - 8);\n-\tts = get_be64(key.buf + key.len - 8);\n+\tts = reftable_get_be64((unsigned char *)key.buf + key.len - 8);\n \n \tr->update_index = (~max) - ts;\n \n@@ -937,7 +937,7 @@ static int reftable_log_record_decode(void *rec, struct reftable_buf key,\n \t\tgoto done;\n \t}\n \n-\tr->value.update.tz_offset = get_be16(in.buf);\n+\tr->value.update.tz_offset = reftable_get_be16(in.buf);\n \tstring_view_consume(&in, 2);\n \n \tn = decode_string(scratch, in);\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 155863ee5f..5961698311 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -99,9 +99,9 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \n \tdest[4] = writer_version(w);\n \n-\tput_be24(dest + 5, w->opts.block_size);\n-\tput_be64(dest + 8, w->min_update_index);\n-\tput_be64(dest + 16, w->max_update_index);\n+\treftable_put_be24(dest + 5, w->opts.block_size);\n+\treftable_put_be64(dest + 8, w->min_update_index);\n+\treftable_put_be64(dest + 16, w->max_update_index);\n \tif (writer_version(w) == 2) {\n \t\tuint32_t hash_id;\n \n@@ -116,7 +116,7 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)\n \t\t\treturn -1;\n \t\t}\n \n-\t\tput_be32(dest + 24, hash_id);\n+\t\treftable_put_be32(dest + 24, hash_id);\n \t}\n \n \treturn header_size(writer_version(w));\n@@ -717,19 +717,19 @@ int reftable_writer_close(struct reftable_writer *w)\n \t}\n \n \tp += writer_write_header(w, footer);\n-\tput_be64(p, w->stats.ref_stats.index_offset);\n+\treftable_put_be64(p, w->stats.ref_stats.index_offset);\n \tp += 8;\n-\tput_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n+\treftable_put_be64(p, (w->stats.obj_stats.offset) << 5 | w->stats.object_id_len);\n \tp += 8;\n-\tput_be64(p, w->stats.obj_stats.index_offset);\n+\treftable_put_be64(p, w->stats.obj_stats.index_offset);\n \tp += 8;\n \n-\tput_be64(p, w->stats.log_stats.offset);\n+\treftable_put_be64(p, w->stats.log_stats.offset);\n \tp += 8;\n-\tput_be64(p, w->stats.log_stats.index_offset);\n+\treftable_put_be64(p, w->stats.log_stats.index_offset);\n \tp += 8;\n \n-\tput_be32(p, crc32(0, footer, p - footer));\n+\treftable_put_be32(p, crc32(0, footer, p - footer));\n \tp += 4;\n \n \terr = w->flush(w->write_arg);\ndiff --git a/t/unit-tests/t-reftable-basics.c b/t/unit-tests/t-reftable-basics.c\nindex 9ba7eb05ad..c9e751e49e 100644\n--- a/t/unit-tests/t-reftable-basics.c\n+++ b/t/unit-tests/t-reftable-basics.c\n@@ -128,12 +128,30 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \t\treftable_buf_release(&b);\n \t}\n \n-\tif_test (\"put_be24 and get_be24 work\") {\n+\tif_test (\"reftable_put_be64 and reftable_get_be64 work\") {\n+\t\tuint64_t in = 0x1122334455667788;\n+\t\tuint8_t dest[8];\n+\t\tuint64_t out;\n+\t\treftable_put_be64(dest, in);\n+\t\tout = reftable_get_be64(dest);\n+\t\tcheck_int(in, ==, out);\n+\t}\n+\n+\tif_test (\"reftable_put_be32 and reftable_get_be32 work\") {\n+\t\tuint32_t in = 0x11223344;\n+\t\tuint8_t dest[4];\n+\t\tuint32_t out;\n+\t\treftable_put_be32(dest, in);\n+\t\tout = reftable_get_be32(dest);\n+\t\tcheck_int(in, ==, out);\n+\t}\n+\n+\tif_test (\"reftable_put_be24 and reftable_get_be24 work\") {\n \t\tuint32_t in = 0x112233;\n \t\tuint8_t dest[3];\n \t\tuint32_t out;\n-\t\tput_be24(dest, in);\n-\t\tout = get_be24(dest);\n+\t\treftable_put_be24(dest, in);\n+\t\tout = reftable_get_be24(dest);\n \t\tcheck_int(in, ==, out);\n \t}\n \n@@ -141,8 +159,8 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \t\tuint32_t in = 0xfef1;\n \t\tuint8_t dest[3];\n \t\tuint32_t out;\n-\t\tput_be16(dest, in);\n-\t\tout = get_be16(dest);\n+\t\treftable_put_be16(dest, in);\n+\t\tout = reftable_get_be16(dest);\n \t\tcheck_int(in, ==, out);\n \t}\n \n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512085","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-10-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","subject":"[PATCH v5 10/18] reftable/reader: stop using `ARRAY_SIZE()` macro","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:52:05Z","receivedAt":"2025-02-07T11:52:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have a single user of the `ARRAY_SIZE()` macro in the reftable\nreader. Drop its use to reduce our dependence on the Git codebase.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/reader.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/reftable/reader.c b/reftable/reader.c\nindex bf07a0a586..c3a3674665 100644\n--- a/reftable/reader.c\n+++ b/reftable/reader.c\n@@ -849,7 +849,7 @@ int reftable_reader_print_blocks(const char *tablename)\n \tprintf(\"header:\\n\");\n \tprintf(\" block_size: %d\\n\", r->block_size);\n \n-\tfor (i = 0; i < ARRAY_SIZE(sections); i++) {\n+\tfor (i = 0; i < sizeof(sections) / sizeof(*sections); i++) {\n \t\terr = table_iter_seek_start(&ti, sections[i].type, 0);\n \t\tif (err < 0)\n \t\t\tgoto done;\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512086","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-11-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","subject":"[PATCH v5 11/18] reftable/system: introduce `reftable_rand()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:52:06Z","receivedAt":"2025-02-07T11:52:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new system-level `reftable_rand()` function that generates a\nsingle unsigned integer for us. The implementation of this function is\nto be provided by the calling codebase, which allows us to more easily\nhook into pre-seeded random number generators.\n\nAdapt the two callsites where we generated random data.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 4 ++--\n reftable/system.c | 5 +++++\n reftable/system.h | 6 ++++++\n 3 files changed, 13 insertions(+), 2 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 46abf51ec5..e22577ec77 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -523,7 +523,7 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\tclose(fd);\n \t\tfd = -1;\n \n-\t\tdelay = delay + (delay * git_rand(CSPRNG_BYTES_INSECURE)) / UINT32_MAX + 1;\n+\t\tdelay = delay + (delay * reftable_rand()) / UINT32_MAX + 1;\n \t\tsleep_millisec(delay);\n \t}\n \n@@ -688,7 +688,7 @@ int reftable_stack_add(struct reftable_stack *st,\n static int format_name(struct reftable_buf *dest, uint64_t min, uint64_t max)\n {\n \tchar buf[100];\n-\tuint32_t rnd = git_rand(CSPRNG_BYTES_INSECURE);\n+\tuint32_t rnd = reftable_rand();\n \tsnprintf(buf, sizeof(buf), \"0x%012\" PRIx64 \"-0x%012\" PRIx64 \"-%08x\",\n \t\t min, max, rnd);\n \treftable_buf_reset(dest);\ndiff --git a/reftable/system.c b/reftable/system.c\nindex adf8e4d30b..e25ccc0da3 100644\n--- a/reftable/system.c\n+++ b/reftable/system.c\n@@ -4,6 +4,11 @@\n #include \"../lockfile.h\"\n #include \"../tempfile.h\"\n \n+uint32_t reftable_rand(void)\n+{\n+\treturn git_rand(CSPRNG_BYTES_INSECURE);\n+}\n+\n int tmpfile_from_pattern(struct reftable_tmpfile *out, const char *pattern)\n {\n \tstruct tempfile *tempfile;\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 7d5f803eeb..3bd4a4e322 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -13,6 +13,12 @@ license that can be found in the LICENSE file or at\n \n #include \"git-compat-util.h\"\n \n+/*\n+ * Return a random 32 bit integer. This function is expected to return\n+ * pre-seeded data.\n+ */\n+uint32_t reftable_rand(void);\n+\n /*\n * An implementation-specific temporary file. By making this specific to the\n * implementation it becomes possible to tie temporary files into any kind of\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512088","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-12-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","subject":"[PATCH v5 12/18] reftable/stack: stop using `sleep_millisec()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:52:07Z","receivedAt":"2025-02-07T11:52:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor our use of `sleep_millisec()` by open-coding it with poll(3p),\nwhich is the current implementation of this function. Ideally, we'd use\na more direct way to sleep, but there is no equivalent to sleep(3p) that\nwould accept milliseconds as input.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/stack.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex e22577ec77..7b0b10cd38 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -524,7 +524,7 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\tfd = -1;\n \n \t\tdelay = delay + (delay * reftable_rand()) / UINT32_MAX + 1;\n-\t\tsleep_millisec(delay);\n+\t\tpoll(NULL, 0, delay);\n \t}\n \n out:\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512087","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-13-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","subject":"[PATCH v5 13/18] reftable/basics: stop using `SWAP()` macro","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:52:08Z","receivedAt":"2025-02-07T11:52:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Stop using `SWAP()` macro in favor of an open-coded variant of it. Note\nthat this also requires us to open-code the build assert that `SWAP()`\nitself uses to verify that the size of both variables matches.\n\nThis is done to reduce our dependency on the Git codebase.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 9 +++++++++\n reftable/merged.c | 2 +-\n reftable/pq.c | 4 ++--\n reftable/record.c | 16 ++++++++--------\n 4 files changed, 20 insertions(+), 11 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex c1ddbaec3f..59000798f0 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -266,6 +266,15 @@ static inline void *reftable_alloc_grow(void *p, size_t nelem, size_t elsize,\n # define strdup(str) REFTABLE_BANNED(strdup)\n #endif\n \n+#define REFTABLE_SWAP(a, b) do {\t\t\t\t\t\t\t\t\\\n+\tvoid *_swap_a_ptr = &(a);\t\t\t\t\t\t\t\t\\\n+\tvoid *_swap_b_ptr = &(b);\t\t\t\t\t\t\t\t\\\n+\tunsigned char _swap_buffer[sizeof(a) - 2 * sizeof(a) * (sizeof(a) != sizeof(b))];\t\\\n+\tmemcpy(_swap_buffer, _swap_a_ptr, sizeof(a));\t\t\t\t\t\t\\\n+\tmemcpy(_swap_a_ptr, _swap_b_ptr, sizeof(a));\t\t\t\t\t\t\\\n+\tmemcpy(_swap_b_ptr, _swap_buffer, sizeof(a));\t\t\t\t\t\t\\\n+} while (0)\n+\n /* Find the longest shared prefix size of `a` and `b` */\n size_t common_prefix_size(struct reftable_buf *a, struct reftable_buf *b);\n \ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 563864068c..4ff1553772 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -155,7 +155,7 @@ static int merged_iter_next_entry(struct merged_iter *mi,\n \t}\n \n \tmi->advance_index = entry.index;\n-\tSWAP(*rec, *entry.rec);\n+\tREFTABLE_SWAP(*rec, *entry.rec);\n \treturn 0;\n }\n \ndiff --git a/reftable/pq.c b/reftable/pq.c\nindex ef8035cfd9..82394a972d 100644\n--- a/reftable/pq.c\n+++ b/reftable/pq.c\n@@ -57,7 +57,7 @@ int merged_iter_pqueue_remove(struct merged_iter_pqueue *pq, struct pq_entry *ou\n \n \t\tif (min == i)\n \t\t\tbreak;\n-\t\tSWAP(pq->heap[i], pq->heap[min]);\n+\t\tREFTABLE_SWAP(pq->heap[i], pq->heap[min]);\n \t\ti = min;\n \t}\n \n@@ -81,7 +81,7 @@ int merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry\n \t\tsize_t j = (i - 1) / 2;\n \t\tif (pq_less(&pq->heap[j], &pq->heap[i]))\n \t\t\tbreak;\n-\t\tSWAP(pq->heap[j], pq->heap[i]);\n+\t\tREFTABLE_SWAP(pq->heap[j], pq->heap[i]);\n \t\ti = j;\n \t}\n \ndiff --git a/reftable/record.c b/reftable/record.c\nindex 3552bafa99..9a1edf39a0 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -237,11 +237,11 @@ static int reftable_ref_record_copy_from(void *rec, const void *src_rec,\n \tsize_t refname_cap = 0;\n \tint err;\n \n-\tSWAP(refname, ref->refname);\n-\tSWAP(refname_cap, ref->refname_cap);\n+\tREFTABLE_SWAP(refname, ref->refname);\n+\tREFTABLE_SWAP(refname_cap, ref->refname_cap);\n \treftable_ref_record_release(ref);\n-\tSWAP(ref->refname, refname);\n-\tSWAP(ref->refname_cap, refname_cap);\n+\tREFTABLE_SWAP(ref->refname, refname);\n+\tREFTABLE_SWAP(ref->refname_cap, refname_cap);\n \n \tif (src->refname) {\n \t\tsize_t refname_len = strlen(src->refname);\n@@ -376,11 +376,11 @@ static int reftable_ref_record_decode(void *rec, struct reftable_buf key,\n \t\treturn n;\n \tstring_view_consume(&in, n);\n \n-\tSWAP(refname, r->refname);\n-\tSWAP(refname_cap, r->refname_cap);\n+\tREFTABLE_SWAP(refname, r->refname);\n+\tREFTABLE_SWAP(refname_cap, r->refname_cap);\n \treftable_ref_record_release(r);\n-\tSWAP(r->refname, refname);\n-\tSWAP(r->refname_cap, refname_cap);\n+\tREFTABLE_SWAP(r->refname, refname);\n+\tREFTABLE_SWAP(r->refname_cap, refname_cap);\n \n \tREFTABLE_ALLOC_GROW_OR_NULL(r->refname, key.len + 1, r->refname_cap);\n \tif (!r->refname) {\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512089","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-14-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","subject":"[PATCH v5 14/18] reftable/basics: introduce `REFTABLE_UNUSED` annotation","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:52:09Z","receivedAt":"2025-02-07T11:52:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce the `REFTABLE_UNUSED` annotation and replace all existing\nusers of `UNUSED` in the reftable library to use the new macro instead.\n\nNote that we unconditionally define `MAYBE_UNUSED` in the exact same\nway, so doing so unconditionally for `REFTABLE_UNUSED` should be fine,\ntoo.\n\nSuggested-by: Toon Claes \nSigned-off-by: Patrick Steinhardt \n---\n reftable/basics.h | 2 ++\n reftable/blocksource.c | 6 +++---\n reftable/iter.c | 10 +++++-----\n reftable/record.c | 26 +++++++++++++-------------\n reftable/writer.c | 2 +-\n 5 files changed, 24 insertions(+), 22 deletions(-)\n\ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex 59000798f0..fd59cbb772 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -16,6 +16,8 @@ license that can be found in the LICENSE file or at\n #include \"system.h\"\n #include \"reftable-basics.h\"\n \n+#define REFTABLE_UNUSED __attribute__((__unused__))\n+\n struct reftable_buf {\n \tsize_t alloc;\n \tsize_t len;\ndiff --git a/reftable/blocksource.c b/reftable/blocksource.c\nindex 02972c46f4..78c1be2337 100644\n--- a/reftable/blocksource.c\n+++ b/reftable/blocksource.c\n@@ -13,14 +13,14 @@ license that can be found in the LICENSE file or at\n #include \"reftable-blocksource.h\"\n #include \"reftable-error.h\"\n \n-static void reftable_buf_return_block(void *b UNUSED, struct reftable_block *dest)\n+static void reftable_buf_return_block(void *b REFTABLE_UNUSED, struct reftable_block *dest)\n {\n \tif (dest->len)\n \t\tmemset(dest->data, 0xff, dest->len);\n \treftable_free(dest->data);\n }\n \n-static void reftable_buf_close(void *b UNUSED)\n+static void reftable_buf_close(void *b REFTABLE_UNUSED)\n {\n }\n \n@@ -67,7 +67,7 @@ static uint64_t file_size(void *b)\n \treturn ((struct file_block_source *)b)->size;\n }\n \n-static void file_return_block(void *b UNUSED, struct reftable_block *dest UNUSED)\n+static void file_return_block(void *b REFTABLE_UNUSED, struct reftable_block *dest REFTABLE_UNUSED)\n {\n }\n \ndiff --git a/reftable/iter.c b/reftable/iter.c\nindex b2ffb09c16..f520382e70 100644\n--- a/reftable/iter.c\n+++ b/reftable/iter.c\n@@ -25,17 +25,17 @@ int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)\n \treturn it->ops->next(it->iter_arg, rec);\n }\n \n-static int empty_iterator_seek(void *arg UNUSED, struct reftable_record *want UNUSED)\n+static int empty_iterator_seek(void *arg REFTABLE_UNUSED, struct reftable_record *want REFTABLE_UNUSED)\n {\n \treturn 0;\n }\n \n-static int empty_iterator_next(void *arg UNUSED, struct reftable_record *rec UNUSED)\n+static int empty_iterator_next(void *arg REFTABLE_UNUSED, struct reftable_record *rec REFTABLE_UNUSED)\n {\n \treturn 1;\n }\n \n-static void empty_iterator_close(void *arg UNUSED)\n+static void empty_iterator_close(void *arg REFTABLE_UNUSED)\n {\n }\n \n@@ -143,8 +143,8 @@ static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)\n \treturn 0;\n }\n \n-static int indexed_table_ref_iter_seek(void *p UNUSED,\n-\t\t\t\t struct reftable_record *want UNUSED)\n+static int indexed_table_ref_iter_seek(void *p REFTABLE_UNUSED,\n+\t\t\t\t struct reftable_record *want REFTABLE_UNUSED)\n {\n \treturn REFTABLE_API_ERROR;\n }\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 9a1edf39a0..142853d507 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -490,7 +490,7 @@ static void reftable_obj_record_release(void *rec)\n }\n \n static int reftable_obj_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t\t uint32_t hash_size REFTABLE_UNUSED)\n {\n \tstruct reftable_obj_record *obj = rec;\n \tconst struct reftable_obj_record *src = src_rec;\n@@ -528,7 +528,7 @@ static uint8_t reftable_obj_record_val_type(const void *rec)\n }\n \n static int reftable_obj_record_encode(const void *rec, struct string_view s,\n-\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t uint32_t hash_size REFTABLE_UNUSED)\n {\n \tconst struct reftable_obj_record *r = rec;\n \tstruct string_view start = s;\n@@ -563,8 +563,8 @@ static int reftable_obj_record_encode(const void *rec, struct string_view s,\n \n static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n \t\t\t\t uint8_t val_type, struct string_view in,\n-\t\t\t\t uint32_t hash_size UNUSED,\n-\t\t\t\t struct reftable_buf *scratch UNUSED)\n+\t\t\t\t uint32_t hash_size REFTABLE_UNUSED,\n+\t\t\t\t struct reftable_buf *scratch REFTABLE_UNUSED)\n {\n \tstruct string_view start = in;\n \tstruct reftable_obj_record *r = rec;\n@@ -618,13 +618,13 @@ static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n \treturn start.len - in.len;\n }\n \n-static int not_a_deletion(const void *p UNUSED)\n+static int not_a_deletion(const void *p REFTABLE_UNUSED)\n {\n \treturn 0;\n }\n \n static int reftable_obj_record_equal_void(const void *a, const void *b,\n-\t\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t\t uint32_t hash_size REFTABLE_UNUSED)\n {\n \tstruct reftable_obj_record *ra = (struct reftable_obj_record *) a;\n \tstruct reftable_obj_record *rb = (struct reftable_obj_record *) b;\n@@ -1054,7 +1054,7 @@ static int reftable_index_record_key(const void *r, struct reftable_buf *dest)\n }\n \n static int reftable_index_record_copy_from(void *rec, const void *src_rec,\n-\t\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t\t uint32_t hash_size REFTABLE_UNUSED)\n {\n \tstruct reftable_index_record *dst = rec;\n \tconst struct reftable_index_record *src = src_rec;\n@@ -1075,13 +1075,13 @@ static void reftable_index_record_release(void *rec)\n \treftable_buf_release(&idx->last_key);\n }\n \n-static uint8_t reftable_index_record_val_type(const void *rec UNUSED)\n+static uint8_t reftable_index_record_val_type(const void *rec REFTABLE_UNUSED)\n {\n \treturn 0;\n }\n \n static int reftable_index_record_encode(const void *rec, struct string_view out,\n-\t\t\t\t\tuint32_t hash_size UNUSED)\n+\t\t\t\t\tuint32_t hash_size REFTABLE_UNUSED)\n {\n \tconst struct reftable_index_record *r =\n \t\t(const struct reftable_index_record *)rec;\n@@ -1097,10 +1097,10 @@ static int reftable_index_record_encode(const void *rec, struct string_view out,\n }\n \n static int reftable_index_record_decode(void *rec, struct reftable_buf key,\n-\t\t\t\t\tuint8_t val_type UNUSED,\n+\t\t\t\t\tuint8_t val_type REFTABLE_UNUSED,\n \t\t\t\t\tstruct string_view in,\n-\t\t\t\t\tuint32_t hash_size UNUSED,\n-\t\t\t\t\tstruct reftable_buf *scratch UNUSED)\n+\t\t\t\t\tuint32_t hash_size REFTABLE_UNUSED,\n+\t\t\t\t\tstruct reftable_buf *scratch REFTABLE_UNUSED)\n {\n \tstruct string_view start = in;\n \tstruct reftable_index_record *r = rec;\n@@ -1120,7 +1120,7 @@ static int reftable_index_record_decode(void *rec, struct reftable_buf key,\n }\n \n static int reftable_index_record_equal(const void *a, const void *b,\n-\t\t\t\t uint32_t hash_size UNUSED)\n+\t\t\t\t uint32_t hash_size REFTABLE_UNUSED)\n {\n \tstruct reftable_index_record *ia = (struct reftable_index_record *) a;\n \tstruct reftable_index_record *ib = (struct reftable_index_record *) b;\ndiff --git a/reftable/writer.c b/reftable/writer.c\nindex 5961698311..bb00e89f53 100644\n--- a/reftable/writer.c\n+++ b/reftable/writer.c\n@@ -636,7 +636,7 @@ static void write_object_record(void *void_arg, void *key)\n done:;\n }\n \n-static void object_record_free(void *void_arg UNUSED, void *key)\n+static void object_record_free(void *void_arg REFTABLE_UNUSED, void *key)\n {\n \tstruct obj_index_tree_node *entry = key;\n \n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512091","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-15-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","subject":"[PATCH v5 15/18] compat/mingw: split out POSIX-related bits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:52:10Z","receivedAt":"2025-02-07T11:52:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Split out the POSIX-related bits from \"compat/mingw.h\". This is in\npreparation for splitting up \"git-compat-utils.h\" into a header that\nprovides POSIX-compatibility and a header that provides common wrappers\nused by the Git project.\n\nSigned-off-by: Patrick Steinhardt \n---\n compat/{mingw.c => mingw/compat-util.c} | 28 ++--\n compat/mingw/compat-util.h | 220 ++++++++++++++++++++++++++++++++\n compat/{mingw.h => mingw/posix.h} | 216 +------------------------------\n compat/msvc.c | 6 -\n compat/msvc/compat-util.c | 6 +\n compat/msvc/compat-util.h | 7 +\n compat/{msvc.h => msvc/posix.h} | 8 +-\n config.mak.uname | 6 +-\n contrib/buildsystems/CMakeLists.txt | 2 +-\n git-compat-util.h | 4 +-\n meson.build | 8 +-\n 11 files changed, 266 insertions(+), 245 deletions(-)\n\ndiff --git a/compat/mingw.c b/compat/mingw/compat-util.c\nsimilarity index 99%\nrename from compat/mingw.c\nrename to compat/mingw/compat-util.c\nindex 1d5b211b54..aeb569bcea 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw/compat-util.c\n@@ -1,24 +1,24 @@\n #define USE_THE_REPOSITORY_VARIABLE\n #define DISABLE_SIGN_COMPARE_WARNINGS\n \n-#include \"../git-compat-util.h\"\n-#include \"win32.h\"\n+#include \"../../git-compat-util.h\"\n+#include \"../win32.h\"\n #include \n #include \n #include \n #include \n-#include \"../strbuf.h\"\n-#include \"../run-command.h\"\n-#include \"../abspath.h\"\n-#include \"../alloc.h\"\n-#include \"win32/lazyload.h\"\n-#include \"../config.h\"\n-#include \"../environment.h\"\n-#include \"../trace2.h\"\n-#include \"../symlinks.h\"\n-#include \"../wrapper.h\"\n-#include \"dir.h\"\n-#include \"gettext.h\"\n+#include \"../../strbuf.h\"\n+#include \"../../run-command.h\"\n+#include \"../../abspath.h\"\n+#include \"../../alloc.h\"\n+#include \"../win32/lazyload.h\"\n+#include \"../../config.h\"\n+#include \"../../environment.h\"\n+#include \"../../trace2.h\"\n+#include \"../../symlinks.h\"\n+#include \"../../wrapper.h\"\n+#include \"../../dir.h\"\n+#include \"../../gettext.h\"\n #define SECURITY_WIN32\n #include \n \ndiff --git a/compat/mingw/compat-util.h b/compat/mingw/compat-util.h\nnew file mode 100644\nindex 0000000000..470542d309\n--- /dev/null\n+++ b/compat/mingw/compat-util.h\n@@ -0,0 +1,220 @@\n+#ifndef COMPAT_MINGW_COMPAT_UTIL_H\n+#define COMPAT_MINGW_COMPAT_UTIL_H\n+\n+#include \"posix.h\"\n+\n+struct config_context;\n+int mingw_core_config(const char *var, const char *value,\n+\t\t const struct config_context *ctx, void *cb);\n+#define platform_core_config mingw_core_config\n+\n+#ifndef NO_OPENSSL\n+#include \n+static inline int mingw_SSL_set_fd(SSL *ssl, int fd)\n+{\n+\treturn SSL_set_fd(ssl, _get_osfhandle(fd));\n+}\n+#define SSL_set_fd mingw_SSL_set_fd\n+\n+static inline int mingw_SSL_set_rfd(SSL *ssl, int fd)\n+{\n+\treturn SSL_set_rfd(ssl, _get_osfhandle(fd));\n+}\n+#define SSL_set_rfd mingw_SSL_set_rfd\n+\n+static inline int mingw_SSL_set_wfd(SSL *ssl, int fd)\n+{\n+\treturn SSL_set_wfd(ssl, _get_osfhandle(fd));\n+}\n+#define SSL_set_wfd mingw_SSL_set_wfd\n+#endif\n+\n+/*\n+ * git specific compatibility\n+ */\n+\n+static inline void convert_slashes(char *path)\n+{\n+\tfor (; *path; path++)\n+\t\tif (*path == '\\\\')\n+\t\t\t*path = '/';\n+}\n+#define PATH_SEP ';'\n+char *mingw_query_user_email(void);\n+#define query_user_email mingw_query_user_email\n+\n+/**\n+ * Verifies that the specified path is owned by the user running the\n+ * current process.\n+ */\n+int is_path_owned_by_current_sid(const char *path, struct strbuf *report);\n+#define is_path_owned_by_current_user is_path_owned_by_current_sid\n+\n+/**\n+ * Verifies that the given path is a valid one on Windows.\n+ *\n+ * In particular, path segments are disallowed which\n+ *\n+ * - end in a period or a space (except the special directories `.` and `..`).\n+ *\n+ * - contain any of the reserved characters, e.g. `:`, `;`, `*`, etc\n+ *\n+ * - correspond to reserved names (such as `AUX`, `PRN`, etc)\n+ *\n+ * The `allow_literal_nul` parameter controls whether the path `NUL` should\n+ * be considered valid (this makes sense e.g. before opening files, as it is\n+ * perfectly legitimate to open `NUL` on Windows, just as it is to open\n+ * `/dev/null` on Unix/Linux).\n+ *\n+ * Returns 1 upon success, otherwise 0.\n+ */\n+int is_valid_win32_path(const char *path, int allow_literal_nul);\n+#define is_valid_path(path) is_valid_win32_path(path, 0)\n+\n+/**\n+ * Converts UTF-8 encoded string to UTF-16LE.\n+ *\n+ * To support repositories with legacy-encoded file names, invalid UTF-8 bytes\n+ * 0xa0 - 0xff are converted to corresponding printable Unicode chars \\u00a0 -\n+ * \\u00ff, and invalid UTF-8 bytes 0x80 - 0x9f (which would make non-printable\n+ * Unicode) are converted to hex-code.\n+ *\n+ * Lead-bytes not followed by an appropriate number of trail-bytes, over-long\n+ * encodings and 4-byte encodings > \\u10ffff are detected as invalid UTF-8.\n+ *\n+ * Maximum space requirement for the target buffer is two wide chars per UTF-8\n+ * char (((strlen(utf) * 2) + 1) [* sizeof(wchar_t)]).\n+ *\n+ * The maximum space is needed only if the entire input string consists of\n+ * invalid UTF-8 bytes in range 0x80-0x9f, as per the following table:\n+ *\n+ * | | UTF-8 | UTF-16 |\n+ * Code point | UTF-8 sequence | bytes | words | ratio\n+ * --------------+-------------------+-------+--------+-------\n+ * 000000-00007f | 0-7f | 1 | 1 | 1\n+ * 000080-0007ff | c2-df + 80-bf | 2 | 1 | 0.5\n+ * 000800-00ffff | e0-ef + 2 * 80-bf | 3 | 1 | 0.33\n+ * 010000-10ffff | f0-f4 + 3 * 80-bf | 4 | 2 (a) | 0.5\n+ * invalid | 80-9f | 1 | 2 (b) | 2\n+ * invalid | a0-ff | 1 | 1 | 1\n+ *\n+ * (a) encoded as UTF-16 surrogate pair\n+ * (b) encoded as two hex digits\n+ *\n+ * Note that, while the UTF-8 encoding scheme can be extended to 5-byte, 6-byte\n+ * or even indefinite-byte sequences, the largest valid code point \\u10ffff\n+ * encodes as only 4 UTF-8 bytes.\n+ *\n+ * Parameters:\n+ * wcs: wide char target buffer\n+ * utf: string to convert\n+ * wcslen: size of target buffer (in wchar_t's)\n+ * utflen: size of string to convert, or -1 if 0-terminated\n+ *\n+ * Returns:\n+ * length of converted string (_wcslen(wcs)), or -1 on failure\n+ *\n+ * Errors:\n+ * EINVAL: one of the input parameters is invalid (e.g. NULL)\n+ * ERANGE: the output buffer is too small\n+ */\n+int xutftowcsn(wchar_t *wcs, const char *utf, size_t wcslen, int utflen);\n+\n+/**\n+ * Simplified variant of xutftowcsn, assumes input string is \\0-terminated.\n+ */\n+static inline int xutftowcs(wchar_t *wcs, const char *utf, size_t wcslen)\n+{\n+\treturn xutftowcsn(wcs, utf, wcslen, -1);\n+}\n+\n+/**\n+ * Simplified file system specific variant of xutftowcsn, assumes output\n+ * buffer size is MAX_PATH wide chars and input string is \\0-terminated,\n+ * fails with ENAMETOOLONG if input string is too long.\n+ */\n+static inline int xutftowcs_path(wchar_t *wcs, const char *utf)\n+{\n+\tint result = xutftowcsn(wcs, utf, MAX_PATH, -1);\n+\tif (result < 0 && errno == ERANGE)\n+\t\terrno = ENAMETOOLONG;\n+\treturn result;\n+}\n+\n+/**\n+ * Converts UTF-16LE encoded string to UTF-8.\n+ *\n+ * Maximum space requirement for the target buffer is three UTF-8 chars per\n+ * wide char ((_wcslen(wcs) * 3) + 1).\n+ *\n+ * The maximum space is needed only if the entire input string consists of\n+ * UTF-16 words in range 0x0800-0xd7ff or 0xe000-0xffff (i.e. \\u0800-\\uffff\n+ * modulo surrogate pairs), as per the following table:\n+ *\n+ * | | UTF-16 | UTF-8 |\n+ * Code point | UTF-16 sequence | words | bytes | ratio\n+ * --------------+-----------------------+--------+-------+-------\n+ * 000000-00007f | 0000-007f | 1 | 1 | 1\n+ * 000080-0007ff | 0080-07ff | 1 | 2 | 2\n+ * 000800-00ffff | 0800-d7ff / e000-ffff | 1 | 3 | 3\n+ * 010000-10ffff | d800-dbff + dc00-dfff | 2 | 4 | 2\n+ *\n+ * Note that invalid code points > 10ffff cannot be represented in UTF-16.\n+ *\n+ * Parameters:\n+ * utf: target buffer\n+ * wcs: wide string to convert\n+ * utflen: size of target buffer\n+ *\n+ * Returns:\n+ * length of converted string, or -1 on failure\n+ *\n+ * Errors:\n+ * EINVAL: one of the input parameters is invalid (e.g. NULL)\n+ * ERANGE: the output buffer is too small\n+ */\n+int xwcstoutf(char *utf, const wchar_t *wcs, size_t utflen);\n+\n+/*\n+ * A critical section used in the implementation of the spawn\n+ * functions (mingw_spawnv[p]e()) and waitpid(). Initialised in\n+ * the replacement main() macro below.\n+ */\n+extern CRITICAL_SECTION pinfo_cs;\n+\n+/*\n+ * Git, like most portable C applications, implements a main() function. On\n+ * Windows, this main() function would receive parameters encoded in the\n+ * current locale, but Git for Windows would prefer UTF-8 encoded parameters.\n+ *\n+ * To make that happen, we still declare main() here, and then declare and\n+ * implement wmain() (which is the Unicode variant of main()) and compile with\n+ * -municode. This wmain() function reencodes the parameters from UTF-16 to\n+ * UTF-8 format, sets up a couple of other things as required on Windows, and\n+ * then hands off to the main() function.\n+ */\n+int wmain(int argc, const wchar_t **w_argv);\n+int main(int argc, const char **argv);\n+\n+/*\n+ * For debugging: if a problem occurs, say, in a Git process that is spawned\n+ * from another Git process which in turn is spawned from yet another Git\n+ * process, it can be quite daunting to figure out what is going on.\n+ *\n+ * Call this function to open a new MinTTY (this assumes you are in Git for\n+ * Windows' SDK) with a GDB that attaches to the current process right away.\n+ */\n+void open_in_gdb(void);\n+\n+/*\n+ * Used by Pthread API implementation for Windows\n+ */\n+int err_win_to_posix(DWORD winerr);\n+\n+#ifndef NO_UNIX_SOCKETS\n+int mingw_have_unix_sockets(void);\n+#undef have_unix_sockets\n+#define have_unix_sockets mingw_have_unix_sockets\n+#endif\n+\n+#endif /* COMPAT_MINGW_COMPAT_UTIL_H */\ndiff --git a/compat/mingw.h b/compat/mingw/posix.h\nsimilarity index 59%\nrename from compat/mingw.h\nrename to compat/mingw/posix.h\nindex ebfb8ba423..8dddfa818d 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw/posix.h\n@@ -1,3 +1,6 @@\n+#ifndef COMPAT_MINGW_POSIX_H\n+#define COMPAT_MINGW_POSIX_H\n+\n #ifdef __MINGW64_VERSION_MAJOR\n #include \n #include \n@@ -11,11 +14,6 @@ typedef _sigset_t sigset_t;\n #undef _POSIX_THREAD_SAFE_FUNCTIONS\n #endif\n \n-struct config_context;\n-int mingw_core_config(const char *var, const char *value,\n-\t\t const struct config_context *ctx, void *cb);\n-#define platform_core_config mingw_core_config\n-\n /*\n * things that are not available in header files\n */\n@@ -180,27 +178,6 @@ int mingw_kill(pid_t pid, int sig);\n #define locate_in_PATH mingw_locate_in_PATH\n char *mingw_locate_in_PATH(const char *cmd);\n \n-#ifndef NO_OPENSSL\n-#include \n-static inline int mingw_SSL_set_fd(SSL *ssl, int fd)\n-{\n-\treturn SSL_set_fd(ssl, _get_osfhandle(fd));\n-}\n-#define SSL_set_fd mingw_SSL_set_fd\n-\n-static inline int mingw_SSL_set_rfd(SSL *ssl, int fd)\n-{\n-\treturn SSL_set_rfd(ssl, _get_osfhandle(fd));\n-}\n-#define SSL_set_rfd mingw_SSL_set_rfd\n-\n-static inline int mingw_SSL_set_wfd(SSL *ssl, int fd)\n-{\n-\treturn SSL_set_wfd(ssl, _get_osfhandle(fd));\n-}\n-#define SSL_set_wfd mingw_SSL_set_wfd\n-#endif\n-\n /*\n * implementations of missing functions\n */\n@@ -444,19 +421,6 @@ int winansi_dup2(int oldfd, int newfd);\n void winansi_init(void);\n HANDLE winansi_get_osfhandle(int fd);\n \n-/*\n- * git specific compatibility\n- */\n-\n-static inline void convert_slashes(char *path)\n-{\n-\tfor (; *path; path++)\n-\t\tif (*path == '\\\\')\n-\t\t\t*path = '/';\n-}\n-#define PATH_SEP ';'\n-char *mingw_query_user_email(void);\n-#define query_user_email mingw_query_user_email\n #if !defined(__MINGW64_VERSION_MAJOR) && (!defined(_MSC_VER) || _MSC_VER < 1800)\n #define PRIuMAX \"I64u\"\n #define PRId64 \"I64d\"\n@@ -464,176 +428,4 @@ char *mingw_query_user_email(void);\n #include \n #endif\n \n-/**\n- * Verifies that the specified path is owned by the user running the\n- * current process.\n- */\n-int is_path_owned_by_current_sid(const char *path, struct strbuf *report);\n-#define is_path_owned_by_current_user is_path_owned_by_current_sid\n-\n-/**\n- * Verifies that the given path is a valid one on Windows.\n- *\n- * In particular, path segments are disallowed which\n- *\n- * - end in a period or a space (except the special directories `.` and `..`).\n- *\n- * - contain any of the reserved characters, e.g. `:`, `;`, `*`, etc\n- *\n- * - correspond to reserved names (such as `AUX`, `PRN`, etc)\n- *\n- * The `allow_literal_nul` parameter controls whether the path `NUL` should\n- * be considered valid (this makes sense e.g. before opening files, as it is\n- * perfectly legitimate to open `NUL` on Windows, just as it is to open\n- * `/dev/null` on Unix/Linux).\n- *\n- * Returns 1 upon success, otherwise 0.\n- */\n-int is_valid_win32_path(const char *path, int allow_literal_nul);\n-#define is_valid_path(path) is_valid_win32_path(path, 0)\n-\n-/**\n- * Converts UTF-8 encoded string to UTF-16LE.\n- *\n- * To support repositories with legacy-encoded file names, invalid UTF-8 bytes\n- * 0xa0 - 0xff are converted to corresponding printable Unicode chars \\u00a0 -\n- * \\u00ff, and invalid UTF-8 bytes 0x80 - 0x9f (which would make non-printable\n- * Unicode) are converted to hex-code.\n- *\n- * Lead-bytes not followed by an appropriate number of trail-bytes, over-long\n- * encodings and 4-byte encodings > \\u10ffff are detected as invalid UTF-8.\n- *\n- * Maximum space requirement for the target buffer is two wide chars per UTF-8\n- * char (((strlen(utf) * 2) + 1) [* sizeof(wchar_t)]).\n- *\n- * The maximum space is needed only if the entire input string consists of\n- * invalid UTF-8 bytes in range 0x80-0x9f, as per the following table:\n- *\n- * | | UTF-8 | UTF-16 |\n- * Code point | UTF-8 sequence | bytes | words | ratio\n- * --------------+-------------------+-------+--------+-------\n- * 000000-00007f | 0-7f | 1 | 1 | 1\n- * 000080-0007ff | c2-df + 80-bf | 2 | 1 | 0.5\n- * 000800-00ffff | e0-ef + 2 * 80-bf | 3 | 1 | 0.33\n- * 010000-10ffff | f0-f4 + 3 * 80-bf | 4 | 2 (a) | 0.5\n- * invalid | 80-9f | 1 | 2 (b) | 2\n- * invalid | a0-ff | 1 | 1 | 1\n- *\n- * (a) encoded as UTF-16 surrogate pair\n- * (b) encoded as two hex digits\n- *\n- * Note that, while the UTF-8 encoding scheme can be extended to 5-byte, 6-byte\n- * or even indefinite-byte sequences, the largest valid code point \\u10ffff\n- * encodes as only 4 UTF-8 bytes.\n- *\n- * Parameters:\n- * wcs: wide char target buffer\n- * utf: string to convert\n- * wcslen: size of target buffer (in wchar_t's)\n- * utflen: size of string to convert, or -1 if 0-terminated\n- *\n- * Returns:\n- * length of converted string (_wcslen(wcs)), or -1 on failure\n- *\n- * Errors:\n- * EINVAL: one of the input parameters is invalid (e.g. NULL)\n- * ERANGE: the output buffer is too small\n- */\n-int xutftowcsn(wchar_t *wcs, const char *utf, size_t wcslen, int utflen);\n-\n-/**\n- * Simplified variant of xutftowcsn, assumes input string is \\0-terminated.\n- */\n-static inline int xutftowcs(wchar_t *wcs, const char *utf, size_t wcslen)\n-{\n-\treturn xutftowcsn(wcs, utf, wcslen, -1);\n-}\n-\n-/**\n- * Simplified file system specific variant of xutftowcsn, assumes output\n- * buffer size is MAX_PATH wide chars and input string is \\0-terminated,\n- * fails with ENAMETOOLONG if input string is too long.\n- */\n-static inline int xutftowcs_path(wchar_t *wcs, const char *utf)\n-{\n-\tint result = xutftowcsn(wcs, utf, MAX_PATH, -1);\n-\tif (result < 0 && errno == ERANGE)\n-\t\terrno = ENAMETOOLONG;\n-\treturn result;\n-}\n-\n-/**\n- * Converts UTF-16LE encoded string to UTF-8.\n- *\n- * Maximum space requirement for the target buffer is three UTF-8 chars per\n- * wide char ((_wcslen(wcs) * 3) + 1).\n- *\n- * The maximum space is needed only if the entire input string consists of\n- * UTF-16 words in range 0x0800-0xd7ff or 0xe000-0xffff (i.e. \\u0800-\\uffff\n- * modulo surrogate pairs), as per the following table:\n- *\n- * | | UTF-16 | UTF-8 |\n- * Code point | UTF-16 sequence | words | bytes | ratio\n- * --------------+-----------------------+--------+-------+-------\n- * 000000-00007f | 0000-007f | 1 | 1 | 1\n- * 000080-0007ff | 0080-07ff | 1 | 2 | 2\n- * 000800-00ffff | 0800-d7ff / e000-ffff | 1 | 3 | 3\n- * 010000-10ffff | d800-dbff + dc00-dfff | 2 | 4 | 2\n- *\n- * Note that invalid code points > 10ffff cannot be represented in UTF-16.\n- *\n- * Parameters:\n- * utf: target buffer\n- * wcs: wide string to convert\n- * utflen: size of target buffer\n- *\n- * Returns:\n- * length of converted string, or -1 on failure\n- *\n- * Errors:\n- * EINVAL: one of the input parameters is invalid (e.g. NULL)\n- * ERANGE: the output buffer is too small\n- */\n-int xwcstoutf(char *utf, const wchar_t *wcs, size_t utflen);\n-\n-/*\n- * A critical section used in the implementation of the spawn\n- * functions (mingw_spawnv[p]e()) and waitpid(). Initialised in\n- * the replacement main() macro below.\n- */\n-extern CRITICAL_SECTION pinfo_cs;\n-\n-/*\n- * Git, like most portable C applications, implements a main() function. On\n- * Windows, this main() function would receive parameters encoded in the\n- * current locale, but Git for Windows would prefer UTF-8 encoded parameters.\n- *\n- * To make that happen, we still declare main() here, and then declare and\n- * implement wmain() (which is the Unicode variant of main()) and compile with\n- * -municode. This wmain() function reencodes the parameters from UTF-16 to\n- * UTF-8 format, sets up a couple of other things as required on Windows, and\n- * then hands off to the main() function.\n- */\n-int wmain(int argc, const wchar_t **w_argv);\n-int main(int argc, const char **argv);\n-\n-/*\n- * For debugging: if a problem occurs, say, in a Git process that is spawned\n- * from another Git process which in turn is spawned from yet another Git\n- * process, it can be quite daunting to figure out what is going on.\n- *\n- * Call this function to open a new MinTTY (this assumes you are in Git for\n- * Windows' SDK) with a GDB that attaches to the current process right away.\n- */\n-void open_in_gdb(void);\n-\n-/*\n- * Used by Pthread API implementation for Windows\n- */\n-int err_win_to_posix(DWORD winerr);\n-\n-#ifndef NO_UNIX_SOCKETS\n-int mingw_have_unix_sockets(void);\n-#undef have_unix_sockets\n-#define have_unix_sockets mingw_have_unix_sockets\n-#endif\n+#endif /* COMPAT_MINGW_POSIX_H */\ndiff --git a/compat/msvc.c b/compat/msvc.c\ndeleted file mode 100644\nindex 71843d7eef..0000000000\n--- a/compat/msvc.c\n+++ /dev/null\n@@ -1,6 +0,0 @@\n-#include \"../git-compat-util.h\"\n-#include \"win32.h\"\n-#include \n-#include \"../strbuf.h\"\n-\n-#include \"mingw.c\"\ndiff --git a/compat/msvc/compat-util.c b/compat/msvc/compat-util.c\nnew file mode 100644\nindex 0000000000..b92e52b8b8\n--- /dev/null\n+++ b/compat/msvc/compat-util.c\n@@ -0,0 +1,6 @@\n+#include \"../../git-compat-util.h\"\n+#include \"../win32.h\"\n+#include \n+#include \"../../strbuf.h\"\n+\n+#include \"../mingw/compat-util.c\"\ndiff --git a/compat/msvc/compat-util.h b/compat/msvc/compat-util.h\nnew file mode 100644\nindex 0000000000..404af236b5\n--- /dev/null\n+++ b/compat/msvc/compat-util.h\n@@ -0,0 +1,7 @@\n+#ifndef COMPAT_MSVC_COMPAT_UTIL_H\n+#define COMPAT_MSVC_COMPAT_UTIL_H\n+\n+#include \"posix.h\"\n+#include \"../mingw/compat-util.h\"\n+\n+#endif /* COMPAT_MSVC_COMPAT_UTIL_H */\ndiff --git a/compat/msvc.h b/compat/msvc/posix.h\nsimilarity index 86%\nrename from compat/msvc.h\nrename to compat/msvc/posix.h\nindex 1d7a8c6145..703a7c2067 100644\n--- a/compat/msvc.h\n+++ b/compat/msvc/posix.h\n@@ -1,5 +1,5 @@\n-#ifndef __MSVC__HEAD\n-#define __MSVC__HEAD\n+#ifndef COMPAT_MSVC_POSIX_H\n+#define COMPAT_MSVC_POSIX_H\n \n #include \n #include \n@@ -28,6 +28,6 @@ typedef int sigset_t;\n /* open for reading, writing, or both (not in fcntl.h) */\n #define O_ACCMODE (_O_RDONLY | _O_WRONLY | _O_RDWR)\n \n-#include \"compat/mingw.h\"\n+#include \"../mingw/posix.h\"\n \n-#endif\n+#endif /* COMPAT_MSVC_POSIX_H */\ndiff --git a/config.mak.uname b/config.mak.uname\nindex b12d4e168a..dbd05adb5d 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -495,7 +495,7 @@ endif\n \tAR = compat/vcbuild/scripts/lib.pl\n \tCFLAGS =\n \tBASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE\n-\tCOMPAT_OBJS = compat/msvc.o compat/winansi.o \\\n+\tCOMPAT_OBJS = compat/msvc/compat-util.o compat/winansi.o \\\n \t\tcompat/win32/flush.o \\\n \t\tcompat/win32/path-utils.o \\\n \t\tcompat/win32/pthread.o compat/win32/syslog.o \\\n@@ -541,7 +541,7 @@ endif\n \n \tEXTRA_PROGRAMS += headless-git$X\n \n-compat/msvc.o: compat/msvc.c compat/mingw.c GIT-CFLAGS\n+compat/msvc/compat-util.o: compat/msvc/compat-util.c compat/mingw/compat-util.c GIT-CFLAGS\n endif\n ifeq ($(uname_S),Interix)\n \tNO_INITGROUPS = YesPlease\n@@ -694,7 +694,7 @@ ifeq ($(uname_S),MINGW)\n \tBASIC_LDFLAGS += -municode\n \tCOMPAT_CFLAGS += -DNOGDI -Icompat -Icompat/win32\n \tCOMPAT_CFLAGS += -DSTRIP_EXTENSION=\\\".exe\\\"\n-\tCOMPAT_OBJS += compat/mingw.o compat/winansi.o \\\n+\tCOMPAT_OBJS += compat/mingw/compat-util.o compat/winansi.o \\\n \t\tcompat/win32/trace2_win32_process_info.o \\\n \t\tcompat/win32/flush.o \\\n \t\tcompat/win32/path-utils.o \\\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 10dc54fdcb..65ce8705f0 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -258,7 +258,7 @@ if(CMAKE_SYSTEM_NAME STREQUAL \"Windows\")\n \t\t\t\tUSE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP\n \t\t\t\tHAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET HAVE_RTLGENRANDOM)\n \tlist(APPEND compat_SOURCES\n-\t\tcompat/mingw.c\n+\t\tcompat/mingw/compat-util.c\n \t\tcompat/winansi.c\n \t\tcompat/win32/flush.c\n \t\tcompat/win32/path-utils.c\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex e283c46c6f..b96fb98e1e 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -304,10 +304,10 @@ static inline int is_xplatform_dir_sep(int c)\n #if defined(__MINGW32__)\n /* pull in Windows compatibility stuff */\n #include \"compat/win32/path-utils.h\"\n-#include \"compat/mingw.h\"\n+#include \"compat/mingw/compat-util.h\"\n #elif defined(_MSC_VER)\n #include \"compat/win32/path-utils.h\"\n-#include \"compat/msvc.h\"\n+#include \"compat/msvc/compat-util.h\"\n #else\n #include \n #include \ndiff --git a/meson.build b/meson.build\nindex 2297f5e9dd..e8abf798f5 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -1054,7 +1054,6 @@ if host_machine.system() == 'cygwin'\n ]\n elif host_machine.system() == 'windows'\n libgit_sources += [\n- 'compat/mingw.c',\n 'compat/winansi.c',\n 'compat/win32/flush.c',\n 'compat/win32/path-utils.c',\n@@ -1081,6 +1080,9 @@ elif host_machine.system() == 'windows'\n libgit_include_directories += 'compat/win32'\n if compiler.get_id() == 'msvc'\n libgit_include_directories += 'compat/vcbuild/include'\n+ libgit_sources += 'compat/msvc/compat-util.c'\n+ else\n+ libgit_sources += 'compat/mingw/compat-util.c'\n endif\n endif\n \n@@ -1157,7 +1159,7 @@ else\n error('Native regex support requested but not found')\n endif\n \n-# setitimer and friends are provided by compat/mingw.c.\n+# setitimer and friends are provided by compat/mingw/compat-util.c.\n if host_machine.system() != 'windows'\n if not compiler.compiles('''\n #include \n@@ -1237,7 +1239,7 @@ if not compiler.has_function('qsort')\n endif\n libgit_sources += 'compat/qsort_s.c'\n \n-# unsetenv is provided by compat/mingw.c.\n+# unsetenv is provided by compat/mingw/compat-util.c.\n if host_machine.system() != 'windows' and not compiler.has_function('unsetenv')\n libgit_c_args += '-DNO_UNSETENV'\n libgit_sources += 'compat/unsetenv.c'\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512092","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-16-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","subject":"[PATCH v5 16/18] git-compat-util.h: split out POSIX-emulating bits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:52:11Z","receivedAt":"2025-02-07T11:52:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"git-compat-util.h\" header is a treasure trove of various bits and\npieces used throughout the project. It basically mixes two different\nthings into one:\n\n - Providing a POSIX-like interface even on platforms that aren't\n POSIX-compliant.\n\n - Providing low-level functionality that is specific to Git.\n\nThis intermixing is a bit of a problem for the reftable library as we\ndon't want to recreate the POSIX-like interface there. But neither do we\nwant to pull in the Git-specific functionality, as it is otherwise quite\neasy to start depending on the Git codebase again.\n\nSplit out a new header \"compat/posix.h\" that only contains the bits and\npieces relevant for the emulation of POSIX, which we will start using in\nthe next commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n compat/posix.h | 541 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n git-compat-util.h | 531 +----------------------------------------------------\n 2 files changed, 543 insertions(+), 529 deletions(-)\n\ndiff --git a/compat/posix.h b/compat/posix.h\nnew file mode 100644\nindex 0000000000..7e6c4c1a1f\n--- /dev/null\n+++ b/compat/posix.h\n@@ -0,0 +1,541 @@\n+#ifndef COMPAT_POSIX_H\n+#define COMPAT_POSIX_H\n+\n+#define _FILE_OFFSET_BITS 64\n+\n+/*\n+ * Derived from Linux \"Features Test Macro\" header\n+ * Convenience macros to test the versions of gcc (or\n+ * a compatible compiler).\n+ * Use them like this:\n+ * #if GIT_GNUC_PREREQ (2,8)\n+ * ... code requiring gcc 2.8 or later ...\n+ * #endif\n+ *\n+ * This macro of course is not part of POSIX, but we need it for the UNUSED\n+ * macro which is used by some of our POSIX compatibility wrappers.\n+*/\n+#if defined(__GNUC__) && defined(__GNUC_MINOR__)\n+# define GIT_GNUC_PREREQ(maj, min) \\\n+\t((__GNUC__ << 16) + __GNUC_MINOR__ >= ((maj) << 16) + (min))\n+#else\n+ #define GIT_GNUC_PREREQ(maj, min) 0\n+#endif\n+\n+/*\n+ * UNUSED marks a function parameter that is always unused. It also\n+ * can be used to annotate a function, a variable, or a type that is\n+ * always unused.\n+ *\n+ * A callback interface may dictate that a function accepts a\n+ * parameter at that position, but the implementation of the function\n+ * may not need to use the parameter. In such a case, mark the parameter\n+ * with UNUSED.\n+ *\n+ * When a parameter may be used or unused, depending on conditional\n+ * compilation, consider using MAYBE_UNUSED instead.\n+ */\n+#if GIT_GNUC_PREREQ(4, 5)\n+#define UNUSED __attribute__((unused)) \\\n+\t__attribute__((deprecated (\"parameter declared as UNUSED\")))\n+#elif defined(__GNUC__)\n+#define UNUSED __attribute__((unused)) \\\n+\t__attribute__((deprecated))\n+#else\n+#define UNUSED\n+#endif\n+\n+#ifdef __MINGW64__\n+#define _POSIX_C_SOURCE 1\n+#elif defined(__sun__)\n+ /*\n+ * On Solaris, when _XOPEN_EXTENDED is set, its header file\n+ * forces the programs to be XPG4v2, defeating any _XOPEN_SOURCE\n+ * setting to say we are XPG5 or XPG6. Also on Solaris,\n+ * XPG6 programs must be compiled with a c99 compiler, while\n+ * non XPG6 programs must be compiled with a pre-c99 compiler.\n+ */\n+# if __STDC_VERSION__ - 0 >= 199901L\n+# define _XOPEN_SOURCE 600\n+# else\n+# define _XOPEN_SOURCE 500\n+# endif\n+#elif !defined(__APPLE__) && !defined(__FreeBSD__) && !defined(__USLC__) && \\\n+ !defined(_M_UNIX) && !defined(__sgi) && !defined(__DragonFly__) && \\\n+ !defined(__TANDEM) && !defined(__QNX__) && !defined(__MirBSD__) && \\\n+ !defined(__CYGWIN__)\n+#define _XOPEN_SOURCE 600 /* glibc2 and AIX 5.3L need 500, OpenBSD needs 600 for S_ISLNK() */\n+#define _XOPEN_SOURCE_EXTENDED 1 /* AIX 5.3L needs this */\n+#endif\n+#define _ALL_SOURCE 1\n+#define _GNU_SOURCE 1\n+#define _BSD_SOURCE 1\n+#define _DEFAULT_SOURCE 1\n+#define _NETBSD_SOURCE 1\n+#define _SGI_SOURCE 1\n+\n+#if defined(WIN32) && !defined(__CYGWIN__) /* Both MinGW and MSVC */\n+# if !defined(_WIN32_WINNT)\n+# define _WIN32_WINNT 0x0600\n+# endif\n+#define WIN32_LEAN_AND_MEAN /* stops windows.h including winsock.h */\n+#include \n+#ifndef NO_UNIX_SOCKETS\n+#include \n+#endif\n+#include \n+#define GIT_WINDOWS_NATIVE\n+#endif\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#ifdef HAVE_STRINGS_H\n+#include /* for strcasecmp() */\n+#endif\n+#include \n+#include \n+#include \n+#ifdef NEEDS_SYS_PARAM_H\n+#include \n+#endif\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#if !defined(NO_POLL_H)\n+#include \n+#elif !defined(NO_SYS_POLL_H)\n+#include \n+#else\n+/* Pull the compat stuff */\n+#include \n+#endif\n+#ifdef HAVE_BSD_SYSCTL\n+#include \n+#endif\n+\n+#if defined(__MINGW32__)\n+#include \"mingw/posix.h\"\n+#elif defined(_MSC_VER)\n+#include \"msvc/posix.h\"\n+#else\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#ifndef NO_SYS_SELECT_H\n+#include \n+#endif\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#ifndef NO_INTTYPES_H\n+#include \n+#else\n+#include \n+#endif\n+#ifdef HAVE_ARC4RANDOM_LIBBSD\n+#include \n+#endif\n+#ifdef HAVE_GETRANDOM\n+#include \n+#endif\n+#ifdef NO_INTPTR_T\n+/*\n+ * On I16LP32, ILP32 and LP64 \"long\" is the safe bet, however\n+ * on LLP86, IL33LLP64 and P64 it needs to be \"long long\",\n+ * while on IP16 and IP16L32 it is \"int\" (resp. \"short\")\n+ * Size needs to match (or exceed) 'sizeof(void *)'.\n+ * We can't take \"long long\" here as not everybody has it.\n+ */\n+typedef long intptr_t;\n+typedef unsigned long uintptr_t;\n+#endif\n+#undef _ALL_SOURCE /* AIX 5.3L defines a struct list with _ALL_SOURCE. */\n+#include \n+#define _ALL_SOURCE 1\n+#endif\n+\n+#ifdef MKDIR_WO_TRAILING_SLASH\n+#define mkdir(a,b) compat_mkdir_wo_trailing_slash((a),(b))\n+int compat_mkdir_wo_trailing_slash(const char*, mode_t);\n+#endif\n+\n+#ifdef time\n+#undef time\n+#endif\n+static inline time_t git_time(time_t *tloc)\n+{\n+\tstruct timeval tv;\n+\n+\t/*\n+\t * Avoid time(NULL), which can disagree with gettimeofday(2)\n+\t * and filesystem timestamps.\n+\t */\n+\tgettimeofday(&tv, NULL);\n+\n+\tif (tloc)\n+\t\t*tloc = tv.tv_sec;\n+\treturn tv.tv_sec;\n+}\n+#define time git_time\n+\n+#ifdef NO_STRUCT_ITIMERVAL\n+struct itimerval {\n+\tstruct timeval it_interval;\n+\tstruct timeval it_value;\n+};\n+#endif\n+\n+#ifdef NO_SETITIMER\n+static inline int git_setitimer(int which UNUSED,\n+\t\t\t\tconst struct itimerval *value UNUSED,\n+\t\t\t\tstruct itimerval *newvalue UNUSED) {\n+\treturn 0; /* pretend success */\n+}\n+#undef setitimer\n+#define setitimer(which,value,ovalue) git_setitimer(which,value,ovalue)\n+#endif\n+\n+#ifndef NO_LIBGEN_H\n+#include \n+#else\n+#define basename gitbasename\n+char *gitbasename(char *);\n+#define dirname gitdirname\n+char *gitdirname(char *);\n+#endif\n+\n+#ifndef NO_ICONV\n+#include \n+#endif\n+\n+/* On most systems would have given us this, but\n+ * not on some systems (e.g. z/OS).\n+ */\n+#ifndef NI_MAXHOST\n+#define NI_MAXHOST 1025\n+#endif\n+\n+#ifndef NI_MAXSERV\n+#define NI_MAXSERV 32\n+#endif\n+\n+/* On most systems would have given us this, but\n+ * not on some systems (e.g. GNU/Hurd).\n+ */\n+#ifndef PATH_MAX\n+#define PATH_MAX 4096\n+#endif\n+\n+#ifndef NAME_MAX\n+#define NAME_MAX 255\n+#endif\n+\n+typedef uintmax_t timestamp_t;\n+#define PRItime PRIuMAX\n+#define parse_timestamp strtoumax\n+#define TIME_MAX UINTMAX_MAX\n+#define TIME_MIN 0\n+\n+int lstat_cache_aware_rmdir(const char *path);\n+#if !defined(__MINGW32__) && !defined(_MSC_VER)\n+#define rmdir lstat_cache_aware_rmdir\n+#endif\n+\n+#if defined(NO_MMAP) || defined(USE_WIN32_MMAP)\n+\n+#ifndef PROT_READ\n+#define PROT_READ 1\n+#define PROT_WRITE 2\n+#define MAP_PRIVATE 1\n+#endif\n+\n+#define mmap git_mmap\n+#define munmap git_munmap\n+void *git_mmap(void *start, size_t length, int prot, int flags, int fd, off_t offset);\n+int git_munmap(void *start, size_t length);\n+\n+#else /* NO_MMAP || USE_WIN32_MMAP */\n+\n+#include \n+\n+#endif /* NO_MMAP || USE_WIN32_MMAP */\n+\n+#ifndef MAP_FAILED\n+#define MAP_FAILED ((void *)-1)\n+#endif\n+\n+#ifdef NEEDS_MODE_TRANSLATION\n+#undef S_IFMT\n+#undef S_IFREG\n+#undef S_IFDIR\n+#undef S_IFLNK\n+#undef S_IFBLK\n+#undef S_IFCHR\n+#undef S_IFIFO\n+#undef S_IFSOCK\n+#define S_IFMT 0170000\n+#define S_IFREG 0100000\n+#define S_IFDIR 0040000\n+#define S_IFLNK 0120000\n+#define S_IFBLK 0060000\n+#define S_IFCHR 0020000\n+#define S_IFIFO 0010000\n+#define S_IFSOCK 0140000\n+#ifdef stat\n+#undef stat\n+#endif\n+#define stat(path, buf) git_stat(path, buf)\n+int git_stat(const char *, struct stat *);\n+#ifdef fstat\n+#undef fstat\n+#endif\n+#define fstat(fd, buf) git_fstat(fd, buf)\n+int git_fstat(int, struct stat *);\n+#ifdef lstat\n+#undef lstat\n+#endif\n+#define lstat(path, buf) git_lstat(path, buf)\n+int git_lstat(const char *, struct stat *);\n+#endif\n+\n+#ifdef NO_PREAD\n+#define pread git_pread\n+ssize_t git_pread(int fd, void *buf, size_t count, off_t offset);\n+#endif\n+\n+#ifdef NO_SETENV\n+#define setenv gitsetenv\n+int gitsetenv(const char *, const char *, int);\n+#endif\n+\n+#ifdef NO_MKDTEMP\n+#define mkdtemp gitmkdtemp\n+char *gitmkdtemp(char *);\n+#endif\n+\n+#ifdef NO_UNSETENV\n+#define unsetenv gitunsetenv\n+int gitunsetenv(const char *);\n+#endif\n+\n+#ifdef NO_STRCASESTR\n+#define strcasestr gitstrcasestr\n+char *gitstrcasestr(const char *haystack, const char *needle);\n+#endif\n+\n+#ifdef NO_STRLCPY\n+#define strlcpy gitstrlcpy\n+size_t gitstrlcpy(char *, const char *, size_t);\n+#endif\n+\n+#ifdef NO_STRTOUMAX\n+#define strtoumax gitstrtoumax\n+uintmax_t gitstrtoumax(const char *, char **, int);\n+#define strtoimax gitstrtoimax\n+intmax_t gitstrtoimax(const char *, char **, int);\n+#endif\n+\n+#ifdef NO_HSTRERROR\n+#define hstrerror githstrerror\n+const char *githstrerror(int herror);\n+#endif\n+\n+#ifdef NO_MEMMEM\n+#define memmem gitmemmem\n+void *gitmemmem(const void *haystack, size_t haystacklen,\n+\t\tconst void *needle, size_t needlelen);\n+#endif\n+\n+#ifdef OVERRIDE_STRDUP\n+#ifdef strdup\n+#undef strdup\n+#endif\n+#define strdup gitstrdup\n+char *gitstrdup(const char *s);\n+#endif\n+\n+#ifdef NO_GETPAGESIZE\n+#define getpagesize() sysconf(_SC_PAGESIZE)\n+#endif\n+\n+#ifndef O_CLOEXEC\n+#define O_CLOEXEC 0\n+#endif\n+\n+#ifdef FREAD_READS_DIRECTORIES\n+# if !defined(SUPPRESS_FOPEN_REDEFINITION)\n+# ifdef fopen\n+# undef fopen\n+# endif\n+# define fopen(a,b) git_fopen(a,b)\n+# endif\n+FILE *git_fopen(const char*, const char*);\n+#endif\n+\n+#ifdef SNPRINTF_RETURNS_BOGUS\n+#ifdef snprintf\n+#undef snprintf\n+#endif\n+#define snprintf git_snprintf\n+int git_snprintf(char *str, size_t maxsize,\n+\t\t const char *format, ...);\n+#ifdef vsnprintf\n+#undef vsnprintf\n+#endif\n+#define vsnprintf git_vsnprintf\n+int git_vsnprintf(char *str, size_t maxsize,\n+\t\t const char *format, va_list ap);\n+#endif\n+\n+#ifdef OPEN_RETURNS_EINTR\n+#undef open\n+#define open git_open_with_retry\n+int git_open_with_retry(const char *path, int flag, ...);\n+#endif\n+\n+#ifdef __GLIBC_PREREQ\n+#if __GLIBC_PREREQ(2, 1)\n+#define HAVE_STRCHRNUL\n+#endif\n+#endif\n+\n+#ifndef HAVE_STRCHRNUL\n+#define strchrnul gitstrchrnul\n+static inline char *gitstrchrnul(const char *s, int c)\n+{\n+\twhile (*s && *s != c)\n+\t\ts++;\n+\treturn (char *)s;\n+}\n+#endif\n+\n+#ifdef NO_INET_PTON\n+int inet_pton(int af, const char *src, void *dst);\n+#endif\n+\n+#ifdef NO_INET_NTOP\n+const char *inet_ntop(int af, const void *src, char *dst, size_t size);\n+#endif\n+\n+#ifdef NO_PTHREADS\n+#define atexit git_atexit\n+int git_atexit(void (*handler)(void));\n+#endif\n+\n+#ifndef HOST_NAME_MAX\n+#define HOST_NAME_MAX 256\n+#endif\n+\n+#include \"../sane-ctype.h\"\n+\n+void git_stable_qsort(void *base, size_t nmemb, size_t size,\n+\t\t int(*compar)(const void *, const void *));\n+#ifdef INTERNAL_QSORT\n+#define qsort git_stable_qsort\n+#endif\n+\n+#define QSORT(base, n, compar) sane_qsort((base), (n), sizeof(*(base)), compar)\n+static inline void sane_qsort(void *base, size_t nmemb, size_t size,\n+\t\t\t int(*compar)(const void *, const void *))\n+{\n+\tif (nmemb > 1)\n+\t\tqsort(base, nmemb, size, compar);\n+}\n+\n+#define STABLE_QSORT(base, n, compar) \\\n+\tgit_stable_qsort((base), (n), sizeof(*(base)), compar)\n+\n+#ifndef HAVE_ISO_QSORT_S\n+int git_qsort_s(void *base, size_t nmemb, size_t size,\n+\t\tint (*compar)(const void *, const void *, void *), void *ctx);\n+#define qsort_s git_qsort_s\n+#endif\n+\n+#define QSORT_S(base, n, compar, ctx) do {\t\t\t\\\n+\tif (qsort_s((base), (n), sizeof(*(base)), compar, ctx))\t\\\n+\t\tBUG(\"qsort_s() failed\");\t\t\t\\\n+} while (0)\n+\n+#ifdef NO_NSEC\n+#undef USE_NSEC\n+#define ST_CTIME_NSEC(st) 0\n+#define ST_MTIME_NSEC(st) 0\n+#else\n+#ifdef USE_ST_TIMESPEC\n+#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctimespec.tv_nsec))\n+#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtimespec.tv_nsec))\n+#else\n+#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctim.tv_nsec))\n+#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtim.tv_nsec))\n+#endif\n+#endif\n+\n+#ifndef va_copy\n+/*\n+ * Since an obvious implementation of va_list would be to make it a\n+ * pointer into the stack frame, a simple assignment will work on\n+ * many systems. But let's try to be more portable.\n+ */\n+#ifdef __va_copy\n+#define va_copy(dst, src) __va_copy(dst, src)\n+#else\n+#define va_copy(dst, src) ((dst) = (src))\n+#endif\n+#endif\n+\n+#ifndef _POSIX_THREAD_SAFE_FUNCTIONS\n+static inline void git_flockfile(FILE *fh UNUSED)\n+{\n+\t; /* nothing */\n+}\n+static inline void git_funlockfile(FILE *fh UNUSED)\n+{\n+\t; /* nothing */\n+}\n+#undef flockfile\n+#undef funlockfile\n+#undef getc_unlocked\n+#define flockfile(fh) git_flockfile(fh)\n+#define funlockfile(fh) git_funlockfile(fh)\n+#define getc_unlocked(fh) getc(fh)\n+#endif\n+\n+#ifdef FILENO_IS_A_MACRO\n+int git_fileno(FILE *stream);\n+# ifndef COMPAT_CODE_FILENO\n+# undef fileno\n+# define fileno(p) git_fileno(p)\n+# endif\n+#endif\n+\n+#ifdef NEED_ACCESS_ROOT_HANDLER\n+int git_access(const char *path, int mode);\n+# ifndef COMPAT_CODE_ACCESS\n+# ifdef access\n+# undef access\n+# endif\n+# define access(path, mode) git_access(path, mode)\n+# endif\n+#endif\n+\n+#endif /* COMPAT_POSIX_H */\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex b96fb98e1e..30ad0725fb 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -23,26 +23,9 @@\n #include \n #endif\n \n-struct strbuf;\n-\n-\n-#define _FILE_OFFSET_BITS 64\n+#include \"compat/posix.h\"\n \n-\n-/* Derived from Linux \"Features Test Macro\" header\n- * Convenience macros to test the versions of gcc (or\n- * a compatible compiler).\n- * Use them like this:\n- * #if GIT_GNUC_PREREQ (2,8)\n- * ... code requiring gcc 2.8 or later ...\n- * #endif\n-*/\n-#if defined(__GNUC__) && defined(__GNUC_MINOR__)\n-# define GIT_GNUC_PREREQ(maj, min) \\\n-\t((__GNUC__ << 16) + __GNUC_MINOR__ >= ((maj) << 16) + (min))\n-#else\n- #define GIT_GNUC_PREREQ(maj, min) 0\n-#endif\n+struct strbuf;\n \n #if defined(__GNUC__) || defined(__clang__)\n # define PRAGMA(pragma) _Pragma(#pragma)\n@@ -176,71 +159,6 @@ DISABLE_WARNING(-Wsign-compare)\n /* Approximation of the length of the decimal representation of this type. */\n #define decimal_length(x)\t((int)(sizeof(x) * 2.56 + 0.5) + 1)\n \n-#ifdef __MINGW64__\n-#define _POSIX_C_SOURCE 1\n-#elif defined(__sun__)\n- /*\n- * On Solaris, when _XOPEN_EXTENDED is set, its header file\n- * forces the programs to be XPG4v2, defeating any _XOPEN_SOURCE\n- * setting to say we are XPG5 or XPG6. Also on Solaris,\n- * XPG6 programs must be compiled with a c99 compiler, while\n- * non XPG6 programs must be compiled with a pre-c99 compiler.\n- */\n-# if __STDC_VERSION__ - 0 >= 199901L\n-# define _XOPEN_SOURCE 600\n-# else\n-# define _XOPEN_SOURCE 500\n-# endif\n-#elif !defined(__APPLE__) && !defined(__FreeBSD__) && !defined(__USLC__) && \\\n- !defined(_M_UNIX) && !defined(__sgi) && !defined(__DragonFly__) && \\\n- !defined(__TANDEM) && !defined(__QNX__) && !defined(__MirBSD__) && \\\n- !defined(__CYGWIN__)\n-#define _XOPEN_SOURCE 600 /* glibc2 and AIX 5.3L need 500, OpenBSD needs 600 for S_ISLNK() */\n-#define _XOPEN_SOURCE_EXTENDED 1 /* AIX 5.3L needs this */\n-#endif\n-#define _ALL_SOURCE 1\n-#define _GNU_SOURCE 1\n-#define _BSD_SOURCE 1\n-#define _DEFAULT_SOURCE 1\n-#define _NETBSD_SOURCE 1\n-#define _SGI_SOURCE 1\n-\n-/*\n- * UNUSED marks a function parameter that is always unused. It also\n- * can be used to annotate a function, a variable, or a type that is\n- * always unused.\n- *\n- * A callback interface may dictate that a function accepts a\n- * parameter at that position, but the implementation of the function\n- * may not need to use the parameter. In such a case, mark the parameter\n- * with UNUSED.\n- *\n- * When a parameter may be used or unused, depending on conditional\n- * compilation, consider using MAYBE_UNUSED instead.\n- */\n-#if GIT_GNUC_PREREQ(4, 5)\n-#define UNUSED __attribute__((unused)) \\\n-\t__attribute__((deprecated (\"parameter declared as UNUSED\")))\n-#elif defined(__GNUC__)\n-#define UNUSED __attribute__((unused)) \\\n-\t__attribute__((deprecated))\n-#else\n-#define UNUSED\n-#endif\n-\n-#if defined(WIN32) && !defined(__CYGWIN__) /* Both MinGW and MSVC */\n-# if !defined(_WIN32_WINNT)\n-# define _WIN32_WINNT 0x0600\n-# endif\n-#define WIN32_LEAN_AND_MEAN /* stops windows.h including winsock.h */\n-#include \n-#ifndef NO_UNIX_SOCKETS\n-#include \n-#endif\n-#include \n-#define GIT_WINDOWS_NATIVE\n-#endif\n-\n #if defined(NO_UNIX_SOCKETS) || !defined(GIT_WINDOWS_NATIVE)\n static inline int _have_unix_sockets(void)\n {\n@@ -253,45 +171,6 @@ static inline int _have_unix_sockets(void)\n #define have_unix_sockets _have_unix_sockets\n #endif\n \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#ifdef HAVE_STRINGS_H\n-#include /* for strcasecmp() */\n-#endif\n-#include \n-#include \n-#include \n-#ifdef NEEDS_SYS_PARAM_H\n-#include \n-#endif\n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#if !defined(NO_POLL_H)\n-#include \n-#elif !defined(NO_SYS_POLL_H)\n-#include \n-#else\n-/* Pull the compat stuff */\n-#include \n-#endif\n-#ifdef HAVE_BSD_SYSCTL\n-#include \n-#endif\n-\n /* Used by compat/win32/path-utils.h, and more */\n static inline int is_xplatform_dir_sep(int c)\n {\n@@ -308,48 +187,6 @@ static inline int is_xplatform_dir_sep(int c)\n #elif defined(_MSC_VER)\n #include \"compat/win32/path-utils.h\"\n #include \"compat/msvc/compat-util.h\"\n-#else\n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#ifndef NO_SYS_SELECT_H\n-#include \n-#endif\n-#include \n-#include \n-#include \n-#include \n-#include \n-#include \n-#ifndef NO_INTTYPES_H\n-#include \n-#else\n-#include \n-#endif\n-#ifdef HAVE_ARC4RANDOM_LIBBSD\n-#include \n-#endif\n-#ifdef HAVE_GETRANDOM\n-#include \n-#endif\n-#ifdef NO_INTPTR_T\n-/*\n- * On I16LP32, ILP32 and LP64 \"long\" is the safe bet, however\n- * on LLP86, IL33LLP64 and P64 it needs to be \"long long\",\n- * while on IP16 and IP16L32 it is \"int\" (resp. \"short\")\n- * Size needs to match (or exceed) 'sizeof(void *)'.\n- * We can't take \"long long\" here as not everybody has it.\n- */\n-typedef long intptr_t;\n-typedef unsigned long uintptr_t;\n-#endif\n-#undef _ALL_SOURCE /* AIX 5.3L defines a struct list with _ALL_SOURCE. */\n-#include \n-#define _ALL_SOURCE 1\n #endif\n \n /* used on Mac OS X */\n@@ -370,60 +207,6 @@ static inline const char *precompose_string_if_needed(const char *in)\n #define probe_utf8_pathname_composition()\n #endif\n \n-#ifdef MKDIR_WO_TRAILING_SLASH\n-#define mkdir(a,b) compat_mkdir_wo_trailing_slash((a),(b))\n-int compat_mkdir_wo_trailing_slash(const char*, mode_t);\n-#endif\n-\n-#ifdef time\n-#undef time\n-#endif\n-static inline time_t git_time(time_t *tloc)\n-{\n-\tstruct timeval tv;\n-\n-\t/*\n-\t * Avoid time(NULL), which can disagree with gettimeofday(2)\n-\t * and filesystem timestamps.\n-\t */\n-\tgettimeofday(&tv, NULL);\n-\n-\tif (tloc)\n-\t\t*tloc = tv.tv_sec;\n-\treturn tv.tv_sec;\n-}\n-#define time git_time\n-\n-#ifdef NO_STRUCT_ITIMERVAL\n-struct itimerval {\n-\tstruct timeval it_interval;\n-\tstruct timeval it_value;\n-};\n-#endif\n-\n-#ifdef NO_SETITIMER\n-static inline int git_setitimer(int which UNUSED,\n-\t\t\t\tconst struct itimerval *value UNUSED,\n-\t\t\t\tstruct itimerval *newvalue UNUSED) {\n-\treturn 0; /* pretend success */\n-}\n-#undef setitimer\n-#define setitimer(which,value,ovalue) git_setitimer(which,value,ovalue)\n-#endif\n-\n-#ifndef NO_LIBGEN_H\n-#include \n-#else\n-#define basename gitbasename\n-char *gitbasename(char *);\n-#define dirname gitdirname\n-char *gitdirname(char *);\n-#endif\n-\n-#ifndef NO_ICONV\n-#include \n-#endif\n-\n #ifndef NO_OPENSSL\n #ifdef __APPLE__\n #undef __AVAILABILITY_MACROS_USES_AVAILABILITY\n@@ -441,34 +224,6 @@ char *gitdirname(char *);\n # include \n #endif\n \n-/* On most systems would have given us this, but\n- * not on some systems (e.g. z/OS).\n- */\n-#ifndef NI_MAXHOST\n-#define NI_MAXHOST 1025\n-#endif\n-\n-#ifndef NI_MAXSERV\n-#define NI_MAXSERV 32\n-#endif\n-\n-/* On most systems would have given us this, but\n- * not on some systems (e.g. GNU/Hurd).\n- */\n-#ifndef PATH_MAX\n-#define PATH_MAX 4096\n-#endif\n-\n-#ifndef NAME_MAX\n-#define NAME_MAX 255\n-#endif\n-\n-typedef uintmax_t timestamp_t;\n-#define PRItime PRIuMAX\n-#define parse_timestamp strtoumax\n-#define TIME_MAX UINTMAX_MAX\n-#define TIME_MIN 0\n-\n #ifndef PATH_SEP\n #define PATH_SEP ':'\n #endif\n@@ -492,11 +247,6 @@ static inline int noop_core_config(const char *var UNUSED,\n #define platform_core_config noop_core_config\n #endif\n \n-int lstat_cache_aware_rmdir(const char *path);\n-#if !defined(__MINGW32__) && !defined(_MSC_VER)\n-#define rmdir lstat_cache_aware_rmdir\n-#endif\n-\n #ifndef has_dos_drive_prefix\n static inline int git_has_dos_drive_prefix(const char *path UNUSED)\n {\n@@ -822,25 +572,6 @@ static inline bool strip_suffix(const char *str, const char *suffix,\n \tmemcpy(_swap_b_ptr, _swap_buffer, sizeof(a));\t\t\\\n } while (0)\n \n-#if defined(NO_MMAP) || defined(USE_WIN32_MMAP)\n-\n-#ifndef PROT_READ\n-#define PROT_READ 1\n-#define PROT_WRITE 2\n-#define MAP_PRIVATE 1\n-#endif\n-\n-#define mmap git_mmap\n-#define munmap git_munmap\n-void *git_mmap(void *start, size_t length, int prot, int flags, int fd, off_t offset);\n-int git_munmap(void *start, size_t length);\n-\n-#else /* NO_MMAP || USE_WIN32_MMAP */\n-\n-#include \n-\n-#endif /* NO_MMAP || USE_WIN32_MMAP */\n-\n #ifdef NO_MMAP\n \n /* This value must be multiple of (pagesize * 2) */\n@@ -856,177 +587,15 @@ int git_munmap(void *start, size_t length);\n \n #endif /* NO_MMAP */\n \n-#ifndef MAP_FAILED\n-#define MAP_FAILED ((void *)-1)\n-#endif\n-\n #ifdef NO_ST_BLOCKS_IN_STRUCT_STAT\n #define on_disk_bytes(st) ((st).st_size)\n #else\n #define on_disk_bytes(st) ((st).st_blocks * 512)\n #endif\n \n-#ifdef NEEDS_MODE_TRANSLATION\n-#undef S_IFMT\n-#undef S_IFREG\n-#undef S_IFDIR\n-#undef S_IFLNK\n-#undef S_IFBLK\n-#undef S_IFCHR\n-#undef S_IFIFO\n-#undef S_IFSOCK\n-#define S_IFMT 0170000\n-#define S_IFREG 0100000\n-#define S_IFDIR 0040000\n-#define S_IFLNK 0120000\n-#define S_IFBLK 0060000\n-#define S_IFCHR 0020000\n-#define S_IFIFO 0010000\n-#define S_IFSOCK 0140000\n-#ifdef stat\n-#undef stat\n-#endif\n-#define stat(path, buf) git_stat(path, buf)\n-int git_stat(const char *, struct stat *);\n-#ifdef fstat\n-#undef fstat\n-#endif\n-#define fstat(fd, buf) git_fstat(fd, buf)\n-int git_fstat(int, struct stat *);\n-#ifdef lstat\n-#undef lstat\n-#endif\n-#define lstat(path, buf) git_lstat(path, buf)\n-int git_lstat(const char *, struct stat *);\n-#endif\n-\n #define DEFAULT_PACKED_GIT_LIMIT \\\n \t((1024L * 1024L) * (size_t)(sizeof(void*) >= 8 ? (32 * 1024L * 1024L) : 256))\n \n-#ifdef NO_PREAD\n-#define pread git_pread\n-ssize_t git_pread(int fd, void *buf, size_t count, off_t offset);\n-#endif\n-\n-#ifdef NO_SETENV\n-#define setenv gitsetenv\n-int gitsetenv(const char *, const char *, int);\n-#endif\n-\n-#ifdef NO_MKDTEMP\n-#define mkdtemp gitmkdtemp\n-char *gitmkdtemp(char *);\n-#endif\n-\n-#ifdef NO_UNSETENV\n-#define unsetenv gitunsetenv\n-int gitunsetenv(const char *);\n-#endif\n-\n-#ifdef NO_STRCASESTR\n-#define strcasestr gitstrcasestr\n-char *gitstrcasestr(const char *haystack, const char *needle);\n-#endif\n-\n-#ifdef NO_STRLCPY\n-#define strlcpy gitstrlcpy\n-size_t gitstrlcpy(char *, const char *, size_t);\n-#endif\n-\n-#ifdef NO_STRTOUMAX\n-#define strtoumax gitstrtoumax\n-uintmax_t gitstrtoumax(const char *, char **, int);\n-#define strtoimax gitstrtoimax\n-intmax_t gitstrtoimax(const char *, char **, int);\n-#endif\n-\n-#ifdef NO_HSTRERROR\n-#define hstrerror githstrerror\n-const char *githstrerror(int herror);\n-#endif\n-\n-#ifdef NO_MEMMEM\n-#define memmem gitmemmem\n-void *gitmemmem(const void *haystack, size_t haystacklen,\n-\t\tconst void *needle, size_t needlelen);\n-#endif\n-\n-#ifdef OVERRIDE_STRDUP\n-#ifdef strdup\n-#undef strdup\n-#endif\n-#define strdup gitstrdup\n-char *gitstrdup(const char *s);\n-#endif\n-\n-#ifdef NO_GETPAGESIZE\n-#define getpagesize() sysconf(_SC_PAGESIZE)\n-#endif\n-\n-#ifndef O_CLOEXEC\n-#define O_CLOEXEC 0\n-#endif\n-\n-#ifdef FREAD_READS_DIRECTORIES\n-# if !defined(SUPPRESS_FOPEN_REDEFINITION)\n-# ifdef fopen\n-# undef fopen\n-# endif\n-# define fopen(a,b) git_fopen(a,b)\n-# endif\n-FILE *git_fopen(const char*, const char*);\n-#endif\n-\n-#ifdef SNPRINTF_RETURNS_BOGUS\n-#ifdef snprintf\n-#undef snprintf\n-#endif\n-#define snprintf git_snprintf\n-int git_snprintf(char *str, size_t maxsize,\n-\t\t const char *format, ...);\n-#ifdef vsnprintf\n-#undef vsnprintf\n-#endif\n-#define vsnprintf git_vsnprintf\n-int git_vsnprintf(char *str, size_t maxsize,\n-\t\t const char *format, va_list ap);\n-#endif\n-\n-#ifdef OPEN_RETURNS_EINTR\n-#undef open\n-#define open git_open_with_retry\n-int git_open_with_retry(const char *path, int flag, ...);\n-#endif\n-\n-#ifdef __GLIBC_PREREQ\n-#if __GLIBC_PREREQ(2, 1)\n-#define HAVE_STRCHRNUL\n-#endif\n-#endif\n-\n-#ifndef HAVE_STRCHRNUL\n-#define strchrnul gitstrchrnul\n-static inline char *gitstrchrnul(const char *s, int c)\n-{\n-\twhile (*s && *s != c)\n-\t\ts++;\n-\treturn (char *)s;\n-}\n-#endif\n-\n-#ifdef NO_INET_PTON\n-int inet_pton(int af, const char *src, void *dst);\n-#endif\n-\n-#ifdef NO_INET_NTOP\n-const char *inet_ntop(int af, const void *src, char *dst, size_t size);\n-#endif\n-\n-#ifdef NO_PTHREADS\n-#define atexit git_atexit\n-int git_atexit(void (*handler)(void));\n-#endif\n-\n static inline size_t st_add(size_t a, size_t b)\n {\n \tif (unsigned_add_overflows(a, b))\n@@ -1293,12 +862,6 @@ static inline size_t xsize_t(off_t len)\n \treturn (size_t) len;\n }\n \n-#ifndef HOST_NAME_MAX\n-#define HOST_NAME_MAX 256\n-#endif\n-\n-#include \"sane-ctype.h\"\n-\n /*\n * Like skip_prefix, but compare case-insensitively. Note that the comparison\n * is done via tolower(), so it is strictly ASCII (no multi-byte characters or\n@@ -1364,34 +927,6 @@ static inline int strtol_i(char const *s, int base, int *result)\n \treturn 0;\n }\n \n-void git_stable_qsort(void *base, size_t nmemb, size_t size,\n-\t\t int(*compar)(const void *, const void *));\n-#ifdef INTERNAL_QSORT\n-#define qsort git_stable_qsort\n-#endif\n-\n-#define QSORT(base, n, compar) sane_qsort((base), (n), sizeof(*(base)), compar)\n-static inline void sane_qsort(void *base, size_t nmemb, size_t size,\n-\t\t\t int(*compar)(const void *, const void *))\n-{\n-\tif (nmemb > 1)\n-\t\tqsort(base, nmemb, size, compar);\n-}\n-\n-#define STABLE_QSORT(base, n, compar) \\\n-\tgit_stable_qsort((base), (n), sizeof(*(base)), compar)\n-\n-#ifndef HAVE_ISO_QSORT_S\n-int git_qsort_s(void *base, size_t nmemb, size_t size,\n-\t\tint (*compar)(const void *, const void *, void *), void *ctx);\n-#define qsort_s git_qsort_s\n-#endif\n-\n-#define QSORT_S(base, n, compar, ctx) do {\t\t\t\\\n-\tif (qsort_s((base), (n), sizeof(*(base)), compar, ctx))\t\\\n-\t\tBUG(\"qsort_s() failed\");\t\t\t\\\n-} while (0)\n-\n #ifndef REG_STARTEND\n #error \"Git requires REG_STARTEND support. Compile with NO_REGEX=NeedsStartEnd\"\n #endif\n@@ -1416,39 +951,12 @@ int git_regcomp(regex_t *preg, const char *pattern, int cflags);\n # define FORCE_DIR_SET_GID 0\n #endif\n \n-#ifdef NO_NSEC\n-#undef USE_NSEC\n-#define ST_CTIME_NSEC(st) 0\n-#define ST_MTIME_NSEC(st) 0\n-#else\n-#ifdef USE_ST_TIMESPEC\n-#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctimespec.tv_nsec))\n-#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtimespec.tv_nsec))\n-#else\n-#define ST_CTIME_NSEC(st) ((unsigned int)((st).st_ctim.tv_nsec))\n-#define ST_MTIME_NSEC(st) ((unsigned int)((st).st_mtim.tv_nsec))\n-#endif\n-#endif\n-\n #ifdef UNRELIABLE_FSTAT\n #define fstat_is_reliable() 0\n #else\n #define fstat_is_reliable() 1\n #endif\n \n-#ifndef va_copy\n-/*\n- * Since an obvious implementation of va_list would be to make it a\n- * pointer into the stack frame, a simple assignment will work on\n- * many systems. But let's try to be more portable.\n- */\n-#ifdef __va_copy\n-#define va_copy(dst, src) __va_copy(dst, src)\n-#else\n-#define va_copy(dst, src) ((dst) = (src))\n-#endif\n-#endif\n-\n /* usage.c: only to be used for testing BUG() implementation (see test-tool) */\n extern int BUG_exit_code;\n \n@@ -1478,41 +986,6 @@ void bug_fl(const char *file, int line, const char *fmt, ...);\n # define SHELL_PATH \"/bin/sh\"\n #endif\n \n-#ifndef _POSIX_THREAD_SAFE_FUNCTIONS\n-static inline void git_flockfile(FILE *fh UNUSED)\n-{\n-\t; /* nothing */\n-}\n-static inline void git_funlockfile(FILE *fh UNUSED)\n-{\n-\t; /* nothing */\n-}\n-#undef flockfile\n-#undef funlockfile\n-#undef getc_unlocked\n-#define flockfile(fh) git_flockfile(fh)\n-#define funlockfile(fh) git_funlockfile(fh)\n-#define getc_unlocked(fh) getc(fh)\n-#endif\n-\n-#ifdef FILENO_IS_A_MACRO\n-int git_fileno(FILE *stream);\n-# ifndef COMPAT_CODE_FILENO\n-# undef fileno\n-# define fileno(p) git_fileno(p)\n-# endif\n-#endif\n-\n-#ifdef NEED_ACCESS_ROOT_HANDLER\n-int git_access(const char *path, int mode);\n-# ifndef COMPAT_CODE_ACCESS\n-# ifdef access\n-# undef access\n-# endif\n-# define access(path, mode) git_access(path, mode)\n-# endif\n-#endif\n-\n /*\n * Our code often opens a path to an optional file, to work on its\n * contents when we can successfully open it. We can ignore a failure\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512090","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-17-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","subject":"[PATCH v5 17/18] reftable: decouple from Git codebase by pulling in \"compat/posix.h\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:52:12Z","receivedAt":"2025-02-07T11:52:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library includes \"git-compat-util.h\" in order to get a\nPOSIX-like programming environment that papers over various differences\nbetween platforms. The header also brings with it a couple of helpers\nspecific to the Git codebase though, and over time we have started to\nuse these helpers in the reftable library, as well.\n\nThis makes it very hard to use the reftable library as a standalone\nlibrary without the rest of the Git codebase, so other libraries like\ne.g. libgit2 cannot easily use it. But now that we have removed all\ncalls to Git-specific functionality and have split out \"compat/posix.h\"\nas a separate header we can address this.\n\nStop including \"git-compat-util.h\" and instead include \"compat/posix.h\"\nto finalize the decoupling of the reftable library from the rest of the\nGit codebase. The only bits which remain specific to Git are \"system.h\"\nand \"system.c\", which projects will have to provide.\n\nSigned-off-by: Patrick Steinhardt \n---\n reftable/system.c | 2 ++\n reftable/system.h | 3 ++-\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/reftable/system.c b/reftable/system.c\nindex e25ccc0da3..1ee268b125 100644\n--- a/reftable/system.c\n+++ b/reftable/system.c\n@@ -1,3 +1,5 @@\n+#include \"../git-compat-util.h\"\n+\n #include \"system.h\"\n #include \"basics.h\"\n #include \"reftable-error.h\"\ndiff --git a/reftable/system.h b/reftable/system.h\nindex 3bd4a4e322..dccdf11f76 100644\n--- a/reftable/system.h\n+++ b/reftable/system.h\n@@ -11,7 +11,8 @@ license that can be found in the LICENSE file or at\n \n /* This header glues the reftable library to the rest of Git */\n \n-#include \"git-compat-util.h\"\n+#include \"../compat/posix.h\"\n+#include \n \n /*\n * Return a random 32 bit integer. This function is expected to return\n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512093","messageId":"20250207-pks-reftable-drop-git-compat-util-v5-18-ba2adc79110f@pks.im","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-0-ba2adc79110f@pks.im","subject":"[PATCH v5 18/18] Makefile: skip reftable library for Coccinelle","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-07T11:52:13Z","receivedAt":"2025-02-07T11:52:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable library does not use any of the common helpers that the Git\nproject has. Consequently, most of the rules that we have in Coccinelle\ndo not apply to the library at all and may even generate false positives\nwhen a pattern can be converted to use a Git helper function.\n\nExclude reftable library sources from being checked by Coccinelle to\navoid such false positives.\n\nSigned-off-by: Patrick Steinhardt \n---\n Makefile | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Makefile b/Makefile\nindex d3011e30f7..dc2ac32e4a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -946,7 +946,7 @@ FOUND_SOURCE_FILES := $(filter-out $(GENERATED_H),$(shell $(SOURCES_CMD)))\n FOUND_C_SOURCES = $(filter %.c,$(FOUND_SOURCE_FILES))\n FOUND_H_SOURCES = $(filter %.h,$(FOUND_SOURCE_FILES))\n \n-COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES))\n+COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES) reftable/%,$(FOUND_C_SOURCES))\n \n LIB_H = $(FOUND_H_SOURCES)\n \n\n-- \n2.48.1.538.gc4cfc42d60.dirty\n\n"},{"id":"512162","messageId":"c48e8fa2-cca4-4abc-95d2-234fa5c69b8d@kdbg.org","threadId":"62862","inReplyTo":"20250207-pks-reftable-drop-git-compat-util-v5-15-ba2adc79110f@pks.im","subject":"Re: [PATCH v5 15/18] compat/mingw: split out POSIX-related bits","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2025-02-09T13:14:37Z","receivedAt":"2025-02-09T13:14:52Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 07.02.25 um 12:52 schrieb Patrick Steinhardt:\n> Split out the POSIX-related bits from \"compat/mingw.h\". This is in\n> preparation for splitting up \"git-compat-utils.h\" into a header that\n> provides POSIX-compatibility and a header that provides common wrappers\n> used by the Git project.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n> compat/{mingw.c => mingw/compat-util.c} | 28 ++--\n> compat/mingw/compat-util.h | 220 ++++++++++++++++++++++++++++++++\n> compat/{mingw.h => mingw/posix.h} | 216 +------------------------------\n> compat/msvc.c | 6 -\n> compat/msvc/compat-util.c | 6 +\n> compat/msvc/compat-util.h | 7 +\n> compat/{msvc.h => msvc/posix.h} | 8 +-\n> config.mak.uname | 6 +-\n> contrib/buildsystems/CMakeLists.txt | 2 +-\n> git-compat-util.h | 4 +-\n> meson.build | 8 +-\n> 11 files changed, 266 insertions(+), 245 deletions(-)\n\nI can agree with this rearrangement of compat/mingw.c. But the\nGit-for-Windows repository has much higher stakes with all the\nadditional changes. I put Dscho on Cc.\n\n-- Hannes\n\n"},{"id":"512177","messageId":"xmqq1pw5er3q.fsf@gitster.g","threadId":"62862","inReplyTo":"c48e8fa2-cca4-4abc-95d2-234fa5c69b8d@kdbg.org","subject":"Re: [PATCH v5 15/18] compat/mingw: split out POSIX-related bits","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-10T15:50:33Z","receivedAt":"2025-02-10T15:50:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Sixt writes:\n\n> Am 07.02.25 um 12:52 schrieb Patrick Steinhardt:\n>> Split out the POSIX-related bits from \"compat/mingw.h\". This is in\n>> preparation for splitting up \"git-compat-utils.h\" into a header that\n>> provides POSIX-compatibility and a header that provides common wrappers\n>> used by the Git project.\n>> \n>> Signed-off-by: Patrick Steinhardt \n>> ---\n>> compat/{mingw.c => mingw/compat-util.c} | 28 ++--\n>> compat/mingw/compat-util.h | 220 ++++++++++++++++++++++++++++++++\n>> compat/{mingw.h => mingw/posix.h} | 216 +------------------------------\n>> compat/msvc.c | 6 -\n>> compat/msvc/compat-util.c | 6 +\n>> compat/msvc/compat-util.h | 7 +\n>> compat/{msvc.h => msvc/posix.h} | 8 +-\n>> config.mak.uname | 6 +-\n>> contrib/buildsystems/CMakeLists.txt | 2 +-\n>> git-compat-util.h | 4 +-\n>> meson.build | 8 +-\n>> 11 files changed, 266 insertions(+), 245 deletions(-)\n>\n> I can agree with this rearrangement of compat/mingw.c. But the\n> Git-for-Windows repository has much higher stakes with all the\n> additional changes. I put Dscho on Cc.\n\nThanks. I agree that it is a very prudent and sensible thing to do\nto ask for input from Dscho.\n"},{"id":"512370","messageId":"e202a870-921b-8f38-58af-3d44e94c447d@gmx.de","threadId":"62862","inReplyTo":"c48e8fa2-cca4-4abc-95d2-234fa5c69b8d@kdbg.org","subject":"Re: [PATCH v5 15/18] compat/mingw: split out POSIX-related bits","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2025-02-13T18:22:57Z","receivedAt":"2025-02-13T18:23:03Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Hannes,\n\nthank you for looping me in.\n\nOn Sun, 9 Feb 2025, Johannes Sixt wrote:\n\n> Am 07.02.25 um 12:52 schrieb Patrick Steinhardt:\n> > Split out the POSIX-related bits from \"compat/mingw.h\". This is in\n> > preparation for splitting up \"git-compat-utils.h\" into a header that\n> > provides POSIX-compatibility and a header that provides common wrappers\n> > used by the Git project.\n> >\n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > compat/{mingw.c => mingw/compat-util.c} | 28 ++--\n> > compat/mingw/compat-util.h | 220 ++++++++++++++++++++++++++++++++\n> > compat/{mingw.h => mingw/posix.h} | 216 +------------------------------\n> > compat/msvc.c | 6 -\n> > compat/msvc/compat-util.c | 6 +\n> > compat/msvc/compat-util.h | 7 +\n> > compat/{msvc.h => msvc/posix.h} | 8 +-\n> > config.mak.uname | 6 +-\n> > contrib/buildsystems/CMakeLists.txt | 2 +-\n> > git-compat-util.h | 4 +-\n> > meson.build | 8 +-\n> > 11 files changed, 266 insertions(+), 245 deletions(-)\n>\n> I can agree with this rearrangement of compat/mingw.c. But the\n> Git-for-Windows repository has much higher stakes with all the\n> additional changes. I put Dscho on Cc.\n\nThis indeed causes massive friction. See for example\nhttps://github.com/git-for-windows/git/actions/workflows/main.yml?query=branch%3Ashears%2Fseen,\nwhich should ideally be updated with green builds as frequently as `seen`\nis updated. It's dark red, and there are tons of gaps (which means that I\nwas not able to finish adapting to these changes).\n\nAnother problem introduced by this series is that now the same filename is\nused in multiple directories, which makes it unnecessarily hard e.g. when\nsetting breakpoints in `gdb`, or when trying to follow `#include`s.\n\nGranted, the problem already exists, e.g. with range-diff.c vs\nbuiltin/range-diff.c, but I would hope that Git's source code would be\nchanged away from duplicate filenames instead of adding more.\n\nCiao,\nJohannes\n"},{"id":"512510","messageId":"Z7MvzH3AVe4_waUH@pks.im","threadId":"62862","inReplyTo":"e202a870-921b-8f38-58af-3d44e94c447d@gmx.de","subject":"Re: [PATCH v5 15/18] compat/mingw: split out POSIX-related bits","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-17T12:47:08Z","receivedAt":"2025-02-17T12:47:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Feb 13, 2025 at 07:22:57PM +0100, Johannes Schindelin wrote:\n> Hi Hannes,\n> \n> thank you for looping me in.\n> \n> On Sun, 9 Feb 2025, Johannes Sixt wrote:\n> \n> > Am 07.02.25 um 12:52 schrieb Patrick Steinhardt:\n> > > Split out the POSIX-related bits from \"compat/mingw.h\". This is in\n> > > preparation for splitting up \"git-compat-utils.h\" into a header that\n> > > provides POSIX-compatibility and a header that provides common wrappers\n> > > used by the Git project.\n> > >\n> > > Signed-off-by: Patrick Steinhardt \n> > > ---\n> > > compat/{mingw.c => mingw/compat-util.c} | 28 ++--\n> > > compat/mingw/compat-util.h | 220 ++++++++++++++++++++++++++++++++\n> > > compat/{mingw.h => mingw/posix.h} | 216 +------------------------------\n> > > compat/msvc.c | 6 -\n> > > compat/msvc/compat-util.c | 6 +\n> > > compat/msvc/compat-util.h | 7 +\n> > > compat/{msvc.h => msvc/posix.h} | 8 +-\n> > > config.mak.uname | 6 +-\n> > > contrib/buildsystems/CMakeLists.txt | 2 +-\n> > > git-compat-util.h | 4 +-\n> > > meson.build | 8 +-\n> > > 11 files changed, 266 insertions(+), 245 deletions(-)\n> >\n> > I can agree with this rearrangement of compat/mingw.c. But the\n> > Git-for-Windows repository has much higher stakes with all the\n> > additional changes. I put Dscho on Cc.\n> \n> This indeed causes massive friction. See for example\n> https://github.com/git-for-windows/git/actions/workflows/main.yml?query=branch%3Ashears%2Fseen,\n> which should ideally be updated with green builds as frequently as `seen`\n> is updated. It's dark red, and there are tons of gaps (which means that I\n> was not able to finish adapting to these changes).\n> \n> Another problem introduced by this series is that now the same filename is\n> used in multiple directories, which makes it unnecessarily hard e.g. when\n> setting breakpoints in `gdb`, or when trying to follow `#include`s.\n> \n> Granted, the problem already exists, e.g. with range-diff.c vs\n> builtin/range-diff.c, but I would hope that Git's source code would be\n> changed away from duplicate filenames instead of adding more.\n\nThe question is what ways we have to improve. Filenames being the same\ncan be avoided, e.g. by naming the split-out files e.g. \"mingw-posix.h\"\ninstead of \"mingw/posix.h\". But what alternatives do we have when it\ncomes to splitting out the logic in the first place to reduce the\nconflicts on your side?\n\nA couple of alternatives I see:\n\n - Use `ifdef`s to conditionally only make POSIX bits available. I\n think this would be quite awkward.\n\n - Just ignore the issue on Windows altogether and expose non-POSIX\n bits there. I don't think this is great either, as it means that\n Windows becomes even more of an outlier than it already is.\n\nDo you have any other ideas?\n\nI can send a version that avoids moving stuff around and really only\nsplits out relevant functions into \"compat/{mingw,msvc}-posix.h\". But\nI'm unsure whether that's enough of a help to you.\n\nPatrick\n"}]}